A multi-resolution approach for massively-parallel hardware-friendly optical flow estimation

This paper presents a novel hardware-friendly motion estimation for real-time applications such as robotics or autonomous navigation. Our approach is based on the well-known Lucas & Kanade local algorithm, whose main problem is the unreliability of its estimations for large-range displacements. This disadvantage is solved in the literature by adding the sequential multiscale-with-warping extension, although it dramatically increases the computational cost. Our choice is the implementation of a multiresolution scheme that avoids the warping computation and allows the estimation of large-range motion. This alternative allows the parallel computation of the scale-by-scale motion estimation which makes the whole computation lighter and significantly reduces the processing time compared with the multiscale-with-warping approach. Furthermore, this last fact also means reducing the hardware resource cost for its potential implementation in digital hardware devices such as GPUs, ASICs, or FPGAs. In the discussion, we analyze the speedup of the multiresolution approach compared to the multiscale-with-warping scheme. For an FPGA implementation, we obtain a reduction of latency between 40% and 50% and a resource reduction of 30%. The final solution copes with large-range motion estimations with a simplified architecture very well-suited for customized digital hardware datapath implementations as well as current multicore architectures.     

Occlusion-aware optical flow estimation

Optical flow can be reliably estimated between areas visible in two images, but not in occlusion areas. If optical flow is needed in the whole image domain, one approach is to use additional views of the same scene. If such views are unavailable, an often-used alternative is to extrapolate optical flow in occlusion areas. Since the location of such areas is usually unknown prior to optical flow estimation, this is usually performed in three steps. First, occlusion-ignorant optical flow is estimated, then occlusion areas are identified using the estimated (unreliable) optical flow, and, finally, the optical flow is corrected using the computed occlusion areas. This approach, however, does not permit interaction between optical flow and occlusion estimates. In this paper, we permit such interaction by proposing a variational formulation that jointly computes optical flow, implicitly detects occlusions and extrapolates optical flow in occlusion areas. The extrapolation mechanism is based on anisotropic diffusion and uses the underlying image gradient to preserve structure, such as optical flow discontinuities. Our results show significant improvements in the computed optical flow fields over other approaches, both qualitatively and quantitatively.     

Evaluation of differential optical flow techniques on synthesizedecho images

The performance of three methods for evaluation of motion on synthesized 2-D echo image sequences with features similar to real ones are examined. The selected techniques based on the computation of optical flow are of the differential type and assume that the image brightness pattern is constant over time. They differ in the choice of the smoothing term and in the local or global treatment of the domain. The images were synthesized by simulating the process of echo formation, considering the interaction between ultrasonic fields and human tissues. Moreover, two different approaches were followed to generate the sequences: (1) a known motion field was applied to the intensity distribution of the synthesized images; (2) a known motion field was applied directly to the point scatterer distribution of the tissue. Favorable results were obtained by applying Lucas-Kanade and Horn-Schunck techniques to the sequences of the first type, while all the techniques produced large errors when applied to the other type of sequences. A discussion about the suitability of the above-mentioned techniques for evaluation of motion on real echocardiographic images is also presented together with some results     

Errors-in-variables modeling in optical flow estimation

Gradient-based optical flow estimation methods typically do not take into account errors in the spatial derivative estimates. The presence of these errors causes an errors-in-variables (EIV) problem. Moreover, the use of finite difference methods to calculate these derivatives ensures that the errors are strongly correlated between pixels. Total least squares (TLS) has often been used to address this EIV problem. However, its application in this context is flawed as TLS implicitly assumes that the errors between neighborhood pixels are independent. In this paper, a new optical flow estimation method (EIVM) is formulated to properly treat the EIV problem in optical flow. EIVM is based on Sprent's (1966) procedure which allows the incorporation of a general EIV model in the estimation process. In EIVM, the neighborhood size acts as a smoothing parameter. Due to the weights in the EIVM objective function, the effect of changing the neighborhood size is more complex than in other local model methods such as Lucas and Kanade (1981). These weights, which are functions of the flow estimate, can alter the effective size and orientation of the neighborhood. In this paper, we also present a data-driven method for choosing the neighborhood size based on Stein's unbiased risk estimators (SURE).     

Measuring confidence in optical flow estimation

The performance of a number of techniques proposed for estimating the quality of optical flow estimates formed using differential methods is considered. The existence of a bias in the optical flow estimates computed using numerical derivatives is demonstrated, as well as the correlation between observed errors and predictions made by using a variety of published confidence estimates     

Fast technique for phase-based disparity estimation with noexplicit calculation of phase

A simpler and faster technique for depth estimation, based on phase measurements of disparity, is presented. The technique provides direct evaluations of phase differences, avoids explicit calculations of single phases and the attendant problem of phase wrapping, and is suitable for efficient software and hardware implementations     

Optimal brightness functions for optical flow estimation ofdeformable motion

Estimation accuracy of Horn and Schunck's (1981) classical optical flow algorithm depends on many factors including the brightness pattern of the measured images. Since some applications can select brightness functions with which to "paint" the object, it is desirable to know what patterns will lead to the best motion estimates. The paper presents a method for determining this pattern a priori using mild assumptions about the velocity field and imaging process. The method is based on formulating Horn and Schunck's algorithm as a linear smoother and rigorously deriving an expression for the corresponding error covariance function. The authors then specify a scalar performance measure and develop an approach to select an optimal brightness function which minimizes this performance measure from within a parametrized class. Conditions for existence of an optimal brightness function are also given. The resulting optimal performance is demonstrated using simulations, and a discussion of these results and potential future research is given.     

Wavelet-based compression algorithm for still omnidirectional 3d integral images

Three dimensional (3D) integral imaging is a method that allows the display of full colour images with continuous parallax within a wide viewing zone. Due to the significant quantity of data required to represent a captured 3D integral image with high resolution, image compression becomes mandatory for the storage and transmission of integral images. This paper investigates the use of 2D discrete wavelet transform (2D-DWT) for the compression of omnidirectional 3D integral images (OII). The method requires the extraction of different viewpoint images from the integral image. A single viewpoint image is constructed by extracting one pixel from each microlens, then each viewpoint image is decomposed using a 2D-DWT. The resulting array of coefficients contains several frequency bands. The lower frequency bands of the viewpoint images are assembled and compressed using a 3D discrete cosine transform (3D-DCT) followed by Huffman coding. Whereas, the remaining higher bands are fed directly into a quantisation process followed by arithmetic coding. Simulations are performed on a set of several grey level 3D OII using a uniform scalar quantizer with deadzone. It was found that the algorithm achieves better rate-distortion performance and reconstructs the images with much better image quality at very low bit rates than previously reported 3D-DCT-based scheme.     

Scale space analysis and active contours for omnidirectional images.

A new generation of optical devices that generate images covering a larger part of the field of view than conventional cameras, namely catadioptric cameras, is slowly emerging. These omnidirectional images will most probably deeply impact computer vision in the forthcoming years, provided that the necessary algorithmic background stands strong. In this paper, we propose a general framework that helps define various computer vision primitives. We show that geometry, which plays a central role in the formation of omnidirectional images, must be carefully taken into account while performing such simple tasks as smoothing or edge detection. Partial differential equations (PDEs) offer a very versatile tool that is well suited to cope with geometrical constraints. We derive new energy functionals and PDEs for segmenting images obtained from catadioptric cameras and show that they can be implemented robustly using classical finite difference schemes. Various experimental results illustrate the potential of these new methods on both synthetic and natural images.     

基于光流估计的整帧恢复算法

探讨了在无线环境中可能出现的整帧丢失情况，并利用光流估计的思想，提出了一种有效的双向估计恢复算法。实验结果证明，提出的改进优化算法能较好地提高恢复图像性能，并具有一定的实用价值。     

STC-Flow: Spatio-temporal context-aware optical flow estimation

In this paper, we propose a spatio-temporal contextual network, STC-Flow, for optical flow estimation. Unlike previous optical flow estimation approaches with local pyramid feature extraction and multi-level correlation, we propose a contextual relation exploration architecture by capturing rich long-range dependencies in spatial and temporal dimensions. Specifically, STC-Flow contains three key context modules, i.e., pyramidal spatial context module, temporal context correlation module and recurrent residual contextual upsampling module for the effect of feature extraction, correlation, and flow reconstruction, respectively. Experimental results demonstrate that the proposed scheme achieves the state-of-the-art performance of two-frame based methods on Sintel and KITTI datasets.     

B-spline image model for energy minimization-based optical flow estimation.

Robust estimation of the optical flow is addressed through a multiresolution energy minimization. It involves repeated evaluation of spatial and temporal gradients of image intensity which rely usually on bilinear interpolation and image filtering. We propose to base both computations on a single pyramidal cubic B-spline model of image intensity. We show empirically improvements in convergence speed and estimation error and validate the resulting algorithm on real test sequences.     

基于三维立体线阵的全方位DOA估计方法

本文基于3D立体线阵以及累量的灵活定义,实现了信号DOA的全方位估计,该算法不依赖于信号的具体结构,并构造了三个累量域DOA矩阵,通过对DOA矩阵的特征分解得到配对特征向量,从而获得信号DOA的全方位估计,该方法避免了2D或1D的谱峰搜索,仿真实验结果表明了这种方法的正确性。     

采用光流估计的数字相机自动对焦算法

自动对焦技术对于数字相机至关重要,它是获取清晰图像的重要手段。针对复杂环境下多目标场景图像,提出了一种基于光流场估计的自动对焦算法。通过计算输入图像序列的光流场,对场景中的运动目标进行检测,根据目标运动属性准确判断出感兴趣目标。改进了Brenner清晰度评价方法,利用目标的二维边缘梯度信息建立评价函数,并且通过非线性增益提高评价函数的灵敏度,减小了噪声对评价值的影响。实验证明,该算法能够在主辅目标景深比达50倍的情况下分辨出感兴趣主目标,并在方差为0.02的随机噪声干扰下能有效地评价图像的清晰度;此算法将Brenner等评价函数的峰值稳定余量提高了1至4倍,对于不同图像具有良好的鲁棒性,易于硬件实现。     

A harmonic retrieval framework for discontinuous motion estimation

Motion discontinuities arise when there are occlusions or multiple moving objects in the scene that is imaged. Conventional regularization techniques use smoothness constraints but are not applicable to motion discontinuities. In this paper, we show that discontinuous (or multiple) motion estimation can be viewed as a multicomponent harmonic retrieval problem. From this viewpoint, a number of established techniques for harmonic retrieval ran be applied to solve the challenging problem of discontinuous (or multiple) motion. Compared with existing techniques, the resulting algorithm is not iterative, which not only implies computational efficiency but also obviates concerns regarding convergence or local minima. It also adds flexibility to spatio-temporal techniques which have suffered from lack of explicit modeling of discontinuous motion. Experimental verification of our framework on both synthetic data as well as real image data is provided.     

Robust optical flow estimation via edge preserving filtering

It is known that optical flow estimation techniques suffer from the issues of ill-defined edges and boundaries of the moving objects. Traditional variational methods for optical flow estimation are not robust to handle these issues since the local filters in these methods do not hold the robustness near the edges. In this paper, we propose a non-local total variation NLTV-$L^1$ optical flow estimation method based on robust weighted guided filtering. Specifically, first, the robust weighted guided filtering objective function is proposed to preserve motion edges. The proposed objective function is based on the linear model which is computationally efficient and edge-preserving in complex natural scenarios. Second, the proposed weighted guided filtering objective function is incorporated into the non-local total variation NLTV-$L^1$ energy function. Finally, the novel NLTV-$L^1$ optical flow method is performed using the coarse-to-fine process. Additionally, we modify some state-of-the-art variational optical flow estimation methods by the robust weighted guided filtering objective function to verify the performance on Middlebury, MPI-Sintel, and Foggy Zurich sequences. Experimental results show that the proposed method can preserve edges and improve the accuracy of optical flow estimation compared with several state-of-the-art methods.     

A framework for optical burst switching network design

We analyze optical burst switching (OBS) systems. The analysis leads to a framework which provides guidelines for OBS design. We identify conditions for OBS feasibility and the relationship between burst size, or equivalently burst assembly delay, and throughput, taking into consideration control packet processing and the number of available wavelengths per fiber     

Robust optical flow estimation based on a sparse motion trajectory set

This paper presents an approach to the problem of estimating a dense optical flow field. The approach is based on a multiframe, irregularly spaced motion trajectory set, where each trajectory describes the motion of a given point as a function of time. From this motion trajectory set a dense flow field is estimated using a process of interpolation. A set of localized motion models are estimated, with each pixel labeled as belonging to one of the motion models. A Markov random field framework is adopted, allowing the incorporation of contextual constraints to encourage region-like structures. The approach is compared with a number of conventional optical flow estimation algorithms taken over a number of real and synthetic sequences. Results indicate that the method produces more accurate results for sequences with known ground truth flow. Also, applying the method to real sequences with unknown flow results in lower DFD, for all of the sequences tested.     

A comprehensive cardiac motion estimation framework using both untagged and 3-D tagged MR images based on nonrigid registration

In this paper, we present a novel technique based on nonrigid image registration for myocardial motion estimation using both untagged and 3-D tagged MR images. The novel aspect of our technique is its simultaneous usage of complementary information from both untagged and 3-D tagged MR images. To estimate the motion within the myocardium, we register a sequence of tagged and untagged MR images during the cardiac cycle to a set of reference tagged and untagged MR images at end-diastole. The similarity measure is spatially weighted to maximize the utility of information from both images. In addition, the proposed approach integrates a valve plane tracker and adaptive incompressibility into the framework. We have evaluated the proposed approach on 12 subjects. Our results show a clear improvement in terms of accuracy compared to approaches that use either 3-D tagged or untagged MR image information alone. The relative error compared to manually tracked landmarks is less than 15% throughout the cardiac cycle. Finally, we demonstrate the automatic analysis of cardiac function from the myocardial deformation fields.