^{*}

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

Optical flow algorithms offer a way to estimate motion from a sequence of images. The computation of optical flow plays a key-role in several computer vision applications, including motion detection and segmentation, frame interpolation, three-dimensional scene reconstruction, robot navigation and video compression. In the case of gradient based optical flow implementation, the pre-filtering step plays a vital role, not only for accurate computation of optical flow, but also for the improvement of performance. Generally, in optical flow computation, filtering is used at the initial level on original input images and afterwards, the images are resized. In this paper, we propose an image filtering approach as a pre-processing step for the Lucas-Kanade pyramidal optical flow algorithm. Based on a study of different types of filtering methods and applied on the Iterative Refined Lucas-Kanade, we have concluded on the best filtering practice. As the Gaussian smoothing filter was selected, an empirical approach for the Gaussian variance estimation was introduced. Tested on the Middlebury image sequences, a correlation between the image intensity value and the standard deviation value of the Gaussian function was established. Finally, we have found that our selection method offers a better performance for the Lucas-Kanade optical flow algorithm.

Unlike the processing of static images, much broader information can be extracted from time varying image sequences, this being one of the primary functions of a computer vision system. Obtaining motion information is a challenging task for machines, however, several techniques have been developed in order to obtain the requested motion field. By definition, the optical flow is the pattern of apparent motion of objects, surfaces and edges in a visual scene, caused by the relative motion between the observer and the scene.

In 1981, two differential-based optical flow algorithms were proposed, now considered as classics: one by Horn and Schunck [

While introducing different optical flow methods, it was also necessary to evaluate the proposed methods. Barron, Fleet, and Beauchemin [

In 2000, Christmas [

The design of optimal spatio-temporal filters, especially the ones proposed by Simoncelli [

In this paper, we focused on improving the accuracy of optical flow estimation by using the appropriate filtering method. As image filtering is essential in many applications, including smoothing, noise removal or edge detection, in the case of optical flow, we have investigated the filtering technique as a required preprocessing step. Also, in Section 3 we have analyzed different filtering methods in order to select the most suitable one. Section 4 presents a novel method for the selection of the appropriate Gaussian filter parameter, as discussed and sumarized in Section 5.

We focused our investigation on the Lucas-Kanade optical flow determination. This gradient-based approach uses the constraint of pixel intensities constancy:

The optical flow constraint equation derived from the Taylor expansion of _{x}_{y}_{t}_{x}_{y}

This problem cannot be solved as there are two unknowns in one equation, but if a small region is supposed to have the same velocity, the problem has a solution. Thus,

The optical flow equation is assumed to be used for all pixels within a window centered on pixel _{x}, _{y}

The equation system

This system has more equations than unknowns and thus it is usually over-determined. The Lucas-Kanade method obtains a compromise solution using the least square technique. In consequence, it solves the 2 × 2 system:

The Lucas-Kanade approach is a local optimization problem that cannot perform properly if the object movements are too large. As the gradient information is obtained by neighboring pixels, the real object motion cannot extend beyond the considered region. Also, the local neighborhood taken into account for the least squares approach is finite and there are few chances to correctly determine large movements. Therefore, it is common to use a pyramidal implementation. The input images are resized to a lower resolution, first by filtering with a low pass filter and then subsampled by a factor of 2, technique called coarse-to-fine approach, as shown in

Bouguet describes in [

In this section, we present and discuss the results of our investigation. We have examined the performance of iterative Lucas-Kanade pyramidal optical flow algorithm together with different filtering techniques using well-known image sequences, provided with ground truth optical flow. The experimental were performed on a MATLAB R2010 platform using the standard available toolbox functions.

In the aim of experimental evaluation, we have employed the Middlebury dataset [

We have measured the performance of the estimated optical flow using both average angular error (AAE) and average endpoint error (AEE). The first error metric is the angle difference between the correct and estimated flow vectors defined by:

We have also evaluated the results by means of an absolute error, the flow endpoint error (EE) defined by:
_{GT}_{GT}

At the very first stage of our evaluation, we have used the Pyramidal Iterative Lucas-Kanade algorithm with three pyramidal levels, combined with different filtering techniques on the eight data sets of the Middlebury benchmark. For the preprocessing step of the optical flow estimation, eight different filtering techniques were employed, namely the Gaussian, Median, Mean, High Boost, Laplacian, LOG, Bilateral and Adaptive Noise Removal filtering. Bilateral filtering is a nonlinear filtering method first proposed by Tomasi

In the case of the Gaussian filter, a standard deviation σ = 1 was used, the median filter had a 3 × 3 kernel, the LOG filter had a size of 5 × 5 and standard deviation σ = 0.5. The Mean filter had a 3 × 3 size, the Laplacian filter a value of alpha = 1 and for the case of Bilateral filter, a spatial-domain standard deviation of 0.1 and intensity-domain standard deviation of 0.1 were employed. We have also tested on all image sets, the “High Boost Filter” with 5 × 5 mask window and all pass factor weight ≥1 and the “Adaptive Noise Removal” filter that use neighborhoods of 3 × 3 to estimate the local image mean and standard deviation. The previous mentioned parameters were obtained after a large set of tests in which the parameters of each filter were varied and selected based on the minimum error (AAE and AEE). Thus, we have divided our experimental research into three sections, as presented in the following subsections.

At the earlier stage of our investigation, the goal was to find the appropriate method for filtering in Lucas-Kanade optical flow estimation. Consequently, we have divided our experiment into three parts (

Filtering applied on input images for pyramidal optical flow computation

Filtering applied on all resized input images for pyramidal optical flow computation

Comparison between case 1 and 2

In this section, we present the experimental results for the pyramidal implementation of Lucas-Kanade, in the case where the filtering techniques have been applied before the optical flow estimation. We have concentrated our attention on the Average Angular Error and Average Endpoint Error estimated using eight different filtering techniques. We have found that AAE follows the same pattern as AEEs.

Gaussian, Mean, Median, Adaptive Noise Removal and Bilateral filtering result are comparatively better than the other one tested

Smoothing filter increases significantly the accuracy of the detected flow field

Therefore, selecting the five best performing filters, we have varied the filters parameters in order to obtain the smallest AEE, as listed in

In the case of the pyramidal implementation of Lucas-Kanade, the input images are resized at each level to a lower resolution. Average angular error and average endpoint error are presented in

From the presented results and graphics, we can conclude that:

Gaussian, Mean, Median, Adaptive Noise Removal and Bilateral filters result are comparatively better than the others

Smoothing filters performs better than sharpening filters

We recommended that for the Lucas-Kanade optical flow calculations it is better to use smoothing filters

In order to decide which method performs better, we made a checklist and an average ranking of the different filtering techniques. As a general conclusion from the experiments presented in Sections 3.2.1 and 3.2.2, in the case of pyramidal Lucas-Kanade optical flow, smoothing filters are recommended as the accuracy is improved. In order to decide which the best performing filter is and when it has to be applied, a comparison has been carried out, as shown in the following subsection.

As several filtering methods are considered, a checklist and an average ranking are presented in

Examining the values obtained in the above investigation, we can conclude that:

filtering at all pyramidal levels is better than filtering only the initial images

among all filtering methods, the Gaussian filter is optimal for computing Lucas-Kanade optical flow, as error is decreasing

As the Gaussian filter performs better then any other considered filter, we have focused our investigation on finding the standard deviation parameter optimum value and drawing its dependency with error. Values presented in

From the graphs in

Therefore, we have tried to find the correlation between the image contents and the σ value, by computing a general measure, as the average intensity, for each image and for the entire dataset.

Based on several empirical tests and observations, we are proposing an algorithm for the estimation of the optimal filtering parameter:

compute the mean intensity from input images

find the reference point of Gaussian function

using the values collected above, take a decision about the optimal parameter after a series of comparisons

The mean intensity of the test sequences was estimated using the values computed for two of the images and averageing of it at the end. For instance, in the case of the Dimetrodon image set, frame1 has an average image intensity of 0.3564, frame2 an average image intensity of 0.3567 and the average estimated intensity for the set was 0.3564. In

Examining the plots in

After the estimation of above mentioned values we have compared for each image set the reference point collected from the Gaussian function and the average image intensity. The standard deviation value of the Gaussian filter according to the image characteristics, was established after performing two types of comparisons. In the first case, we have just checked which value is greater than the other:

Once completing this step for all image sets, we have obtained the results in

From

Based on several analyses, we have also suggested another method of choosing the standard deviation, as the ratio between the considered reference point and the image mean intensity value:

After computing the proposed ratio for all the benchmarks, the obtained values are presented in

Examining the obtained value, one can observe that its mean intensity is two times lower than the highest Gaussian value. Therefore, we have specified that the σ should be in the range of [1.62, 2]. As a generalization, the obtained ratio should be the lower bound for the optimal Gaussian filter parameter and the ceiling value of the ratio, the upper bound.

To confirm the above statement, we have tested three synthetic sets of images (

In this paper, we have presented an investigation on image filtering as a preprocessing level for the Lucas-Kanade optical flow computation framework. We have concluded not only on the fact that an optimal filtering must be performed at every pyramidal level, but also introduced a novel method for according the filter to the processed context. Also, from our study we have found that the Gaussian filter performs considerably better among different other filters.

Generally, in pyramidal optical flow computation, the input images are filtered only at the beginning and at the following levels, the images are being resized from that base. Our first experiment concerned the proper use of filtering, not only at the initial level, but subsequently at each pyramidal level. As several test sequences, together with the reference ground truth were available, an improvement of the error was obtained.

Since in our extensive research on the subject, we couldn't find any specifications regarding the optimal type of filter, we have considered the most referenced 2D ones, as Gaussian, Mean, Median, Bilateral, Adaptive Noise Removal or the High Boost filter. From the experimental results we have concluded that the Gaussian filtering is the most suitable in this regard, on the basis of computed average angular error and average endpoint error.

As the Gaussian filter was the most appropriate for pre-filtering the input images, we have investigated the relation between the standard deviation values of the Gaussian function and the image contents. From the plotted results, we have observed that there is no fixed σ value achieving the lowest error for any input sequence. Based on empirical observations, as the variation of error with standard deviation, we have established a correlation. Our novel method for selecting the σ value consists in observing the shape of the Gaussian function using a standard deviation of 1 and extracting the highest value. Comparing this reference point with the image average intensity can give an indication on the suitable value to be used. Also, we have found that the ratio between the image intensity and the highest Gaussian value can give an indication on the proper σ value. Finally, we concluded on the fact that computing the filter standard deviation from image characteristic offers a more accurate optical flow computation.

Optical flow constraint line.

Intersection of (

Coarse-to-fine optical flow estimation.

Coarse-to-fine optical flow estimation, (

Average angular errors using different filters only on input images.

Average endpoint errors using different filters only on input images.

Average angular errors obtained using different filters on all resized images.

Average endpoint errors obtained using different filters on all resized images.

Average angular error for different σ values.

Average angular error measures of iterative pyramidal L-K optical flow by using Gaussian Filtering on all resized input images.

Lowest average endpoint error for best performing filters.

| ||||||||
---|---|---|---|---|---|---|---|---|

Gaussian | 4.7025 | 8.7112 | 8.3686 | 8.5187 | 3.9922 | 14.7064 | 11.5392 | 10.7046 |

smoothing σ | σ = 0.6 | σ = 0.3 | σ = 0.4 | σ = 1 | σ = 0.7 | σ = 4.8 | σ = 3.6 | σ = 0.3 |

Mean | 5.7210 | 10.143 | 9.3132 | 8.6578 | 4.2231 | 16.5652 | 13.1789 | 11.8439 |

Median | 7.8426 | 10.5369 | 9.1439 | 9.7582 | 5.8926 | 19.5492 | 22.5063 | 12.9823 |

Adaptive Noise Removal | 5.8196 | 10.1214 | 9.169 | 9.169 | 4.3921 | 17.1897 | 14.5003 | 11.9682 |

Bilateral σ = [0.1, 0.1] | 17.4394 | 10.5945 | 8.716 | 10.3823 | 8.3447 | 19.2384 | 22.1881 | 14.3764 |

Comparison between filtering methods at initial level and all levels on pyramidal Lucas-Kanade optical flow.

| ||||||||
---|---|---|---|---|---|---|---|---|

Gaussian Smooth Filtering | x | X | x | - | - | - | - | - |

Median Filtering | x | X | x | - | - | - | - | x |

LOG filtering | - | X | - | - | - | - | - | x |

Mean Filtering | x | X | x | - | - | - | - | - |

High Boost Filtering | - | - | - | - | - | x | X | - |

Laplacian Filtering | - | X | - | - | - | - | - | x |

Adaptive Noise Removal filtering | x | X | x | - | - | - | - | x |

Bilateral Filtering | - | - | - | - | - | - | - | - |

Where x—recommended for initial filtering and X—recommended for all levels.

Comparison between filtering methods at initial level and all pyramidal levels using average ranking.

| ||||
---|---|---|---|---|

Gaussian smooth | 9.809825 | 1.156813 | ||

Median | 12.2765 | 1.351225 | ||

LOG | 20.6046875 | 2.27355 | ||

Mean | 9.9558875 | 1.17025 | ||

High Boost | 16.1461625 | 1.893188 | ||

Laplacian | 20.8631375 | 2.688675 | ||

Adaptive Noise Removal | 10.29116 | 1.141913 | ||

Bilateral | 13.909975 | 1.353625 |

Average intensity value of Middlebury dataset.

Dimetrodon | |

RubberWhale | |

Hydrangea | |

Grove3 | |

Grove2 | |

Urban2 | 0.21684 |

Urban3 | 0.2504 |

Venus |

Large values of Gauss function.

−3 | −1.83 | −0.6667 | 0.5 | 1.667 | 2.8333 | |

0.0044 | 0.0743 | 0.3194 | 0.3521 | 0.0995 | 0.0072 |

Results of first comparison.

Dimetrodon | |

RubberWhale | |

Hydrangea | |

Grove3 | |

Grove2 | |

Urban2 | 0.21684 |

Urban3 | 0.2504 |

Venus |

Result for the second comparison step.

Dimetrodon | |

RubberWhale | |

Hydrangea | |

Grove3 | |

Grove2 | |

Urban2 | |

Urban3 | |

Venus |

Average of image intensity values.

Creats | 0.4886 |

Sponza_1 | 0.33605 |

Sponaza_2 | 0.5741 |