An Investigation of Smooth TV-Like Regularization in the Context of the Optical Flow Problem

Abstract

Total variation (TV) is widely used in many image processing problems including the regularization of optical flow estimation. In order to deal with non differentiability of the TV regularization term, smooth approximations have been considered in the literature. In this paper, we investigate the use of three known smooth TV approximations, namely: the Charbonnier, Huber and Green functions. We establish the maximum theoretical error of these approximations and discuss their performance evaluation when applied to the optical flow problem.

1. Introduction

TV-based optical flow can be cast into the following form:

$$\underset{w}{\min }D\left(w\right)+\alpha {∥\nabla w∥}_1,$$

where $w\in {\mathbb{R}}^N$ is the optical flow, $D:{\mathbb{R}}^N\to \mathbb{R}$ is a data energy function, ${∥.∥}_1$ is the $l_1$ norm, and $\alpha >0$ is a regularization parameter weighting the relative importance of data and smoothing terms. Although the TV semi-norm has been useful for performing edge-preserving regularization [1,2,3,4,5,6,7,8,9], it is known to be numerically difficult to handle. Despite its convexity, it is not linear, quadratic or even everywhere-differentiable. Thus, the non-smoothness of this term prevents a straightforward application of gradient based optimization methods.

To remedy the problem of non-differentiability of the $l_1$ norm function $x\mapsto {∥x∥}_1$, there are two solutions. First, we can split $x=y-z$ where $y_i=\max \{x_i,0\}$ and $z_i=\max \{-x_i,0\}$ for $i=1,\dots ,N$. The $l_1$ norm of x will be then equal to ${\sum }_{i=1}^N(y_i+z_i)$, which will remove non-differentiability at zero but unfortunately the problem dimension will be doubled and the optimization will become constrained since y and z should be positive. The second remedy, which we are investigating in this paper, is to replace the $l_1$ norm by a smooth approximation.

Any smooth TV regularization should address two issues: (1) It should remove the singularities that are caused by the use of TV regularization; (2) It should maintain the preservation of motion boundaries. Several smooth approximations of the $l_1$ norm have been established in the literature for the regularization of a wide variety of problems [10,11,12,13,14,15,16,17], as well as for the optical flow problem [2,3,4,5].

To our knowledge, there is no theory that establishes optimality of any of these approximations; the best choice is application dependent. For instance, Nikolova and Ng [17] have considered different smooth TV approximations in the context of restoration and reconstruction of images and signals using half quadratic minimization. Our objective in this paper is to investigate the use of three known smooth TV approximations, namely: the Charbonnier, Huber and Green functions for the case of optical flow computation.

The outline of this paper is as follows. Section 2 describes the variational formulation of the optical flow problem. In particular, we present the TV regularization model for dense optical flow estimation. In Section 3, we consider three smooth approximations of the TV regularization term and discuss their maximum theoretical error of approximation. Section 4 concerns the performance evaluation of the three approximations in terms of the quality of the estimated optical flow and the speed of convergence by using the Middlebury datasets. Finally, we conclude our work in Section 5.

2. Variational Formulation

Let us consider a sequence of gray level images $I(t,x,y)$, $t\in [0,T]$, $(x,y)\in \mathsf{\Omega }$, where $[0,T]$ is the temporal domain and $\mathsf{\Omega }$ denotes the image spatial domain. We will use both continuous and disc in time at frame numbers $t_k=k\Delta t$, $k=0,\dots ,K$ and in space at pixel coordinates $p=(m,n)$, with m (respectively n) corresponds to the discrete column (respectively row) of the image, being the coordinate origin located in the top-left corner of the image. With this notation, $I[k,m,n]$ denotes a discrete representation of $I(t,x,y)$. We will also use both continuous and quantized image intensities and the same symbol I will be used for both of them.

Assuming that the gray level of a point does not change over time we may write the constraint

$$I(t,x(t),y(t\left)\right)=I(0,x,y),$$

(1)

where $\left(x\right(t),y(t\left)\right)$ is the apparent trajectory of the point $\left(x\right(0),y(0\left)\right)=(x,y)$. Taking the derivative with respect to t and denoting $(u(t,x,y),v(t,x,y))=(x^{\prime }\left(t\right),y^{\prime }\left(t\right))$, we obtain the linear optical flow constraint

$$I^t+I^{x}u+I^{y}v=0.$$

(2)

The vector field $w(t,x,y):=\left(u\right(t,x,y),v(t,x,y\left)\right)$ is called optical flow and $I^t$, $\nabla I=(I^x,I^y)$ denote the temporal and spatial partial derivatives of I, which are computed using high-pass gradient filters for discrete images. Clearly, the single constraint (2) is not sufficient to uniquely compute the two components $(u,v)$ of the optical flow (this is called the aperture problem) and only gives the component of the flow normal to the image gradient, i.e., to the level lines of the image. As it is usual, in order to recover a unique flow field, some prior knowledge about it should be added. For that, we assume that the optical flow varies smoothly in space, or better, that is piecewise smooth in $\mathsf{\Omega }$. This can be achieved by including a smoothness term of the form

$$R\left(w\right):={\int }_{\mathsf{\Omega }}G(\nabla I,\nabla w)dxdy,$$

(3)

where $G:{\mathbb{R}}^2\times {\mathbb{R}}^4\to \mathbb{R}$ is a suitable function.

Both data attachment (1) and regularization term (3) can be combined into a single energy functional

$$E\left(w\right)=D\left(w\right)+\alpha R\left(w\right),$$

(4)

where the data functional D is either equal to the linear term

$${\int }_{\mathsf{\Omega }}{(I^t+u{I}^x+v{I}^y)}^{2}dxdy$$

(5)

or the nonlinear term

$${\int }_{\mathsf{\Omega }}{(I(k+1,x+u,y+v)-I(k,x,y))}^{2}dxdy,$$

(6)

and $\alpha >0$ is a regularization parameter.

When using the linear data term (5), the case $G(\nabla I,\nabla w)={\parallel \nabla w\parallel }^2={\parallel \nabla u\parallel }^2+{\parallel \nabla v\parallel }^2$ corresponds to the Horn-Schunck model [18] and the case $G(\nabla I,\nabla u,\nabla v)=\mathrm{trace}({(\nabla w)}^{T}M(\nabla I)\nabla w)$, where $M(\nabla I)=\frac{1}{{∥\nabla I∥}^2+\delta }\left({(\nabla I)}^{\perp }{(\nabla I)}^{\perp T}+\delta 1_d\right)\mathrm{and}\delta \ge 0$, corresponds to the Nagel-Enkelmann model [19]. On the other hand, the TV regularization [1] became the most used in image processing because it allows for discontinuities preserving. In this case, for $w\in L^1\left(\mathsf{\Omega }\right)$, we have

$$R\left(w\right)=\mathrm{TV}\left(w\right):=\underset{\mathsf{\Phi }\in C_c^{\infty }(\mathsf{\Omega },{\mathbb{R}}^4),{\parallel \mathsf{\Phi }\parallel }_{\infty }\le 1}{\sup }{\int }_{\mathsf{\Omega }}<w,\mathrm{div}\mathsf{\Phi }>dxdy,$$

where $C_c^{\infty }(\mathsf{\Omega },{\mathbb{R}}^4)$ is the space of infinitely differentiable vector-valued functions with compact support. Note that when $\mathrm{TV}\left(w\right)<\infty$, the distributional derivative $\mathrm{d}w$ of w is a vector-valued Radon measure with total variation $\left|\mathrm{d}w\right|\left(\mathsf{\Omega }\right)=\mathrm{TV}\left(w\right)$. When $w\in W^{1,1}(\mathsf{\Omega },{\mathbb{R}}^2)$, the TV semi-norm reduces to the $L_1$-norm of the gradient ${\int }_{\mathsf{\Omega }}\parallel \nabla w\parallel dxdy$ so that

$$G(\nabla I,\nabla w)=\parallel \nabla w\parallel ,$$

where we can have either $\parallel \nabla w\parallel =\sqrt{{\parallel \nabla u\parallel }^2+{\parallel \nabla v\parallel }^2}$ or $\parallel \nabla w\parallel =\parallel \nabla u\parallel +\parallel \nabla v\parallel$. We have chosen the first one because the Euclidean norm is known to be rotationally invariant.

For a full account of regularization techniques for the optical flow problem and the associated taxonomy, we refer to [20,21].

In this paper, we will consider a TV regularization model which is written in the discrete form as follows:

$$R\left(w\right):=\sum _{p\in \mathsf{\Omega }}\sum _{q\in {\mathcal{N}}_p}{w}_{p,q},$$

(7)

where $w_{p,q}=\sqrt{{(u_q-u_p)}^2+{(v_q-v_p)}^2}$ and ${\mathcal{N}}_p$ is a set of neighbors of the pixel p. We will also combine the TV regularization with the nonlinear data term (6) used with a robust function $\psi$ in order to remove outliers:

$$\sum _{p\in \mathsf{\Omega }}\psi \left(I[k+1,p+w]-I[k,p]\right).$$

(8)

The robust function used in this paper is

$$\psi \left(s\right)=\left\{\begin{array}{cc}{s}^2/2\hfill & \mathrm{if}\left|\mathrm{s}\right|\le \gamma \hfill \\ {\gamma }^2/2\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

where $\gamma$ is a given threshold.

3. Smooth TV Regularization

In this section, we focus on approximating the non-smooth TV semi-norm (7) by a smooth function:

$$R_{\epsilon }\left(w\right):=\sum _{p\in \mathsf{\Omega }}\sum _{q\in {\mathcal{N}}_p}{\varphi }_{\epsilon }\left(w_{p,q}\right),$$

(9)

where ${\varphi }_{\epsilon }:\mathbb{R}\to {\mathbb{R}}^{+}$ is a smooth approximation of the absolute value function and $\epsilon >0$ is a small parameter adjusting the accuracy of this approximation. In this paper, we will consider variants choices of ${\varphi }_{\epsilon }$ as illustrated in

Table 1. Smooth approximations of the absolute value function and their derivatives. The graphs of these approximations are plotted versus the absolute value function near zero for $\epsilon =0.01$.

Charbonnier	Huber	Green
${\varphi }_{\epsilon }\left(s\right)=\sqrt{s^2+{\epsilon }^2}$	${\varphi }_{\epsilon }\left(s\right)=\left\{\begin{array}{cc}\frac{s^2}{2\epsilon }\hfill & \mathrm{if}\left|s\right|\le \epsilon \hfill \\ \left|s\right|-\frac{\epsilon }{2}\hfill & \mathrm{if}\left|s\right|>\epsilon \hfill \end{array}\right.$	${\varphi }_{\epsilon }\left(s\right)=\epsilon \log \left(2\cosh \left(\frac{s}{\epsilon }\right)\right)$
${\varphi }_{\epsilon }^{\prime }\left(s\right)={\displaystyle \frac{s}{\sqrt{s^2+{\epsilon }^2}}}$	${\varphi }_{\epsilon }^{\prime }\left(s\right)=\left\{\begin{array}{cc}\frac{s}{\epsilon }\hfill & \mathrm{if}\left|s\right|\le \epsilon \hfill \\ \frac{s}{\left|s\right|}\hfill & \mathrm{if}\left|s\right|>\epsilon \hfill \end{array}\right.$	${\varphi }_{\epsilon }^{\prime }\left(s\right)=\tanh \left(\frac{s}{\epsilon }\right)$

.

Notice that the regularization term in (9) with a function ${\varphi }_{\epsilon }$ from Table 1 is a hybrid between the TV regularization (7) and the standard quadratic regularization [18]. It takes the form of a quadratic or nearly quadratic for small values of the optical flow gradient and becomes linear or sublinear for large values of the optical flow gradient. In this way, this smooth regularization will retain the fast Laplacian diffusion inside homogeneous motion regions and its effect is substantially reduced near motion boundaries helping the preservation of these boundaries’ edges. We should also note that the smaller the parameter $\epsilon$ is, the better the function ${\varphi }_{\epsilon }$ approximates the absolute value function; and henceforth the better the smooth regularization (9) approximates the TV regularization (7). In practice, a very small parameter $\epsilon$ might cause numerical instabilities but such a choice is not really needed as the quadratic regularization is preferred inside homogeneous regions.

According to the discussion above, there are minimum conditions that should satisfy any approximation ${\varphi }_{\epsilon }$ (see [10]):

$$\begin{array}{c}{\varphi }_{\epsilon }^{\prime }\left(0\right)=0,\underset{s\to 0^{+}}{\lim }\frac{{\varphi }_{\epsilon }^{\prime }\left(s\right)}{s}=\underset{s\to 0^{+}}{\lim }{\varphi }_{\epsilon }^{\prime \prime }\left(s\right)={\varphi }_{\epsilon }^{\prime \prime }\left(0\right)>0,\\ \underset{s\to \infty }{\lim }\frac{{\varphi }_{\epsilon }^{\prime }\left(s\right)}{s}=\underset{s\to \infty }{\lim }{\varphi }_{\epsilon }^{\prime \prime }\left(s\right)=0\mathrm{and}\underset{s\to \infty }{\lim }\frac{{\varphi }_{\epsilon }^{\prime \prime }\left(s\right)}{{\varphi }_{\epsilon }^{\prime }\left(s\right)/s}=0.\end{array}$$

(10)

By simple calculus, it can be shown that the approximations we are considering in this paper, which are given in Table 1, satisfy the conditions in (10). Notice also that all these approximations are suitable for 1st order numerical convex optimization algorithms since they are all convex and differentiable. However, for 2nd order numerical optimization algorithms, the Charbonnier and Green functions are twice differentiable but the Huber function is not. In this case, the latter approximation is normally replaced by a twice differentiable function called the pseudo Huber approximation ${\varphi }_{\epsilon }\left(s\right)=\epsilon (\sqrt{1+\frac{s^2}{{\epsilon }^2}}-1)$ [12], which is the same as the Charbonnier function except for a vertical translation by $\epsilon$.

3.1. Charbonnier Approximation

The first approximation

$${\varphi }_{\epsilon }\left(s\right)=\sqrt{s^2+{\epsilon }^2}$$

(11)

is referred to as the Charbonnier penalty function [11]. It was first used for optical flow in [5]. This function is clearly strictly convex and infinitely differentiable. Moreover, we can easily prove that it approximates the absolute value function with an error at most equal to $\epsilon$.

Lemma 1.

For $s\in \mathbb{R}$, $0\le {\varphi }_{\epsilon }\left(s\right)-\left|s\right|\le \epsilon ,$ where ${\varphi }_{\epsilon }$ is the Charbonnier function (11).

The Charbonnier TV regularization is therefore an approximation of the TV regularization of order $\epsilon$. Let $\left|\mathsf{\Omega }\right|$ be the total number of image pixels and $\left|\mathcal{N}\right|$ be the fixed size of each neighborhood ${\mathcal{N}}_p$ which is used for the finite difference approximation of the optical flow gradient.

Proposition 1.

$0\le R_{\epsilon }\left(w\right)-R\left(w\right)\le \epsilon \left|\mathsf{\Omega }\right|\left|\mathcal{N}\right|$, where ${\varphi }_{\epsilon }$ in $R_{\epsilon }$ is the Charbonnier function (11).

Proof. 

Using the previous lemma, we get

$$0\le R_{\epsilon }\left(w\right)-R\left(w\right)=\sum _{p\in \mathsf{\Omega }}\sum _{q\in {\mathcal{N}}_p}\left({\varphi }_{\epsilon }\left(w_{p,q}\right)-\varphi \left(w_{p,q}\right)\right)\le \epsilon \left|\mathsf{\Omega }\right|\left|\mathcal{N}\right|.$$

 ☐

3.2. Huber Approximation

The Huber function

$${\varphi }_{\epsilon }\left(s\right)=\left\{\begin{array}{cc}\frac{s^2}{2\epsilon }\hfill & \mathrm{if}\left|s\right|\le \epsilon \hfill \\ \left|s\right|-\frac{\epsilon }{2}\hfill & \mathrm{if}\left|s\right|>\epsilon \hfill \end{array}\right.$$

(12)

was initially used by Huber (see [13]) as an M-estimator in the field of robust statistics. Its use for optical flow computation was first discussed in [2]. Later, it was used as a smooth approximation of the $l_1$ norm as in [16]. The Huber function is clearly convex and continuously differentiable.

We want to relate the Huber regularization in (9) to the TV and quadratic regularization. First, the following lemma shows that the Huber function approximates the absolute function with an error of order $\frac{\epsilon }{2}$.

Lemma 2.

For $s\in \mathbb{R}$, $\left|s\right|-{\displaystyle \frac{\epsilon }{2}}\le {\varphi }_{\epsilon }\left(s\right)\le \left|s\right|,$ where ${\varphi }_{\epsilon }$ is the Huber function (12).

Proof. 

Let $s\in \mathbb{R}$. Suppose first that $\left|s\right|\le \epsilon$. Hence

$$\left|s\right|-\frac{\epsilon }{2}\le \frac{\left|s\right|}{2}=\frac{s^2}{2\left|s\right|}\le {\varphi }_{\epsilon }\left(s\right)=\frac{s^2}{2\epsilon }\le \frac{s^2}{2\left|s\right|}\le \left|s\right|.$$

Now, if $\left|s\right|>\epsilon$, then ${\varphi }_{\epsilon }\left(s\right)=\left|s\right|-{\displaystyle \frac{\epsilon }{2}}>\left|s\right|.$ ☐

This shows that the Huber TV regularization has a maximum theoretical error twice better than that of the Charbonnier TV regularization.

Proposition 2.

$-{\displaystyle \frac{\epsilon }{2}}\left|\mathsf{\Omega }\right|\left|\mathcal{N}\right|\le R_{\epsilon }\left(w\right)-R\left(w\right)\le 0$, where ${\varphi }_{\epsilon }$ in $R_{\epsilon }$ is the Huber function (12).

3.3. Green Approximation

The Green penalty function

$${\varphi }_{\epsilon }\left(s\right)=\epsilon \log (2\cosh \left(\frac{s}{\epsilon }\right))$$

(13)

was originally used in [14] for the maximum likelihood reconstruction from emission tomography data as a convex extension of the Geman and McClure function [15]. This penalty function was introduced for optical flow computation in [3,4]. Again, this function is strictly convex and infinitely differentiable inheriting these properties from the log cosh function. Notice that we have translated the original Green function by a factor $\epsilon \log 2$, which is the maximum approximation error as shown by the following lemma.

Lemma 3.

For $s\in \mathbb{R}$, $\left|{\varphi }_{\epsilon }\left(s\right)-\left|s\right|\right|\le \epsilon \log 2,$ where ${\varphi }_{\epsilon }$ is the Green function (13).

Proof. 

Let $s\in \mathbb{R}$. First, we have

$${\varphi }_{\epsilon }\left(s\right)=\epsilon \log ({\mathrm{e}}^{s/\epsilon }+{\mathrm{e}}^{-s/\epsilon })=s+\epsilon \log (1+{\mathrm{e}}^{-2s/\epsilon }).$$

Hence, for $s>0$, we have $\left|\left|s\right|-{\varphi }_{\epsilon }\left(s\right)\right|=\epsilon \log (1+{\mathrm{e}}^{-2s/\epsilon }).$ Now, if $s>0$, then

$$\left|\left|s\right|-{\varphi }_{\epsilon }\left(s\right)\right|=\epsilon \left|\frac{2s}{\epsilon }+\log (1+{\mathrm{e}}^{-2s/\epsilon })\right|=\epsilon \log (1+{\mathrm{e}}^{2s/\epsilon }).$$

Therefore, whatever the sign of s, we get

$$\left|\left|s\right|-{\varphi }_{\epsilon }\left(s\right)\right|=\epsilon \log (1+{\mathrm{e}}^{-2\left|s\right|/\epsilon })\le \epsilon \log 2.$$

 ☐

The Green TV regularization approximates the TV regularization with an order $\epsilon$ as well but the maximum error is slightly greater than that of the Huber TV regularization.

Proposition 3.

$|R_{\epsilon }\left(w\right)-R\left(w\right)|\le \epsilon |\mathsf{\Omega }\left|\right|\mathcal{N}|\log 2$, where ${\varphi }_{\epsilon }$ in $R_{\epsilon }$ is the Green function (13).

4. Experimental Results

We want to minimize the energy functional (4) where D is given by (8), R is given by (9) and ${\varphi }_{\epsilon }$ is one of the smooth approximations presented in the previous section. We choose to adopt the discretize-optimize approach by applying a numerical optimization algorithm to this discrete version of the optical flow minimization problem. The problem is of a large-scale type and therefore we solve it using a multiresolution line search truncated Newton method as developed in [22,23]. The method first builds a pyramid of images at different levels of resolution. It starts then at the coarsest level with a zero flow field and applies a number of iterations of the line search truncated Newton (LSTN) algorithm. Afterwards, the obtained coarse estimation is taken to the next fine level by bilinear interpolation. This process is repeated until reaching the finest level where a good initial estimate of the optical flow is obtained and henceforth refined by the LSTN algorithm until convergence is reached.

The parameter $\gamma$, present in the data term and which is shared by the three functionals, was fixed to a value between ten and twenty depending on the nature of the image sequence. However, in order to have a fair comparison, the parameters $\alpha$ and $\epsilon$ involved in the regularization term yielding a different energy functional, are tuned for each functional to have the best results. From the experiments, we have noticed that functionals with the Charbonnier and Huber approximations will share in general the same set of optimal parameters. As expected the set of optimal parameters for the Green function is different, especially for the value of $\epsilon$ since the $\log \cosh$ function has a different transition level between its quadratic and linear parts.

In Figure 1, we show the colored based representation of the ground truth and the best optical flow estimates obtained using Charbonnier, Huber and Green smooth TV regularizations for the Middlebury training benchmark [24] using the best parameters. Notice first how the motion boundaries are preserved for all images in Figure 1. This is indeed a famous property of the TV regularization that has been inherited by its three smooth approximations. In Figure 2, we present other tested image sequences that have different types of movement. The first three images have a translation of different sizes: half, one and ten for the spiral, peppers and band sequences, respectively. The baboon sequence has a rotation movement and the Lena sequence has a homography mapping. These five images are standard test images in image processing that have been used as the first frames and the second frames have been generated by applying the movements described above. The Marble blocks sequence, which has a zoom transformation, was obtained from the Image Sequence Server, Institut für Algorithmen und Kognitive Systeme, (Group Prof. Dr. H.-H. Nagel), University of Karlsruhe, Germany and was first used in [25]. Finally, the rotating sphere sequence was generated by the Computer Vision Research Group at the University of Otago, New Zealandand the book sequence by the Computer Laboratory at Cambridge University.

Table 2. Average Angular Error (AAE) using Charbonnier, Huber and Green TV regularizations for the Middlebury datasets (top) and the image sequences given in Figure 2 in the same order (bottom).

	Dimetrodon	Grove2	Grove3	Hydrangea	Rubberwhale	Urban2	Urban3	Venus
Charbonnier	2.561	2.736	6.857	3.335	6.629	6.657	7.209	8.085
Huber	2.562	2.716	6.819	3.364	6.632	6.579	7.288	8.083
Green	2.539	2.899	7.333	3.179	6.979	6.490	8.433	8.400
	1/2 Pixel	1 Pixel	10 Pixels	Rotation	Homography	Zoom	Sphere	Book
Charbonnier	2.412	0.307	0.009	0.794	2.724	6.981	3.525	0.773
Huber	2.382	0.308	0.006	0.767	2.668	6.901	3.588	0.715
Green	6.519	0.299	0.200	1.779	2.872	7.480	3.830	1.531

and

Table 3. Average Endpoint Error (AEE) using Charbonnier, Huber and Green TV regularizations for the Middlebury datasets (top) and the image sequences given in Figure 2 in the same order (bottom).

	Dimetrodon	Grove2	Grove3	Hydrangea	Rubberwhale	Urban2	Urban3	Venus
Charbonnier	0.139	0.183	0.774	0.371	0.201	0.818	1.015	0.557
Huber	0.140	0.180	0.780	0.378	0.201	0.799	1.009	0.551
Green	0.139	0.200	0.794	0.344	0.209	0.816	1.016	0.673
	1/2 Pixel	1 Pixel	10 Pixels	Rotation	Homography	Zoom	Sphere	Book
Charbonnier	0.051	0.010	0.017	0.025	0.081	0.221	0.107	2.946
Huber	0.050	0.010	0.011	0.025	0.078	0.218	0.111	2.952
Green	0.139	0.009	0.037	0.050	0.081	0.236	0.114	3.145

present the performance comparison of the three approximations in terms of the quality of the optical flow estimation measured by the average angular error (AAE) and the average endpoint error (AEE). In

Table 4. Displaced Frame Difference (DFD) using Charbonnier, Huber and Green TV regularizations for the Middlebury datasets (top) and the image sequences given in Figure 2 in the same order (bottom).

	Dimetrodon	Grove2	Grove3	Hydrangea	Rubberwhale	Urban2	Urban3	Venus
Charbonnier	0.470	1.350	1.686	1.163	0.564	0.398	0.471	0.762
Huber	0.471	1.365	1.834	1.210	0.566	0.383	0.463	0.769
Green	0.454	0.997	1.651	0.942	0.393	0.384	0.421	0.976
	1/2 Pixel	1 Pixel	10 Pixels	Rotation	Homography	Zoom	Sphere	Book
Charbonnier	0.305	0.015	0.000	0.986	0.409	1.000	0.065	1.080
Huber	0.315	0.020	0.000	0.986	0.416	0.995	0.069	1.001
Green	0.065	0.006	0.000	0.804	0.319	0.799	0.023	1.053

, we give also the interpolation error measured by the displaced frame difference (DFD), which corresponds to the data term in the energy functional (4). Then in

Table 5. Number of gradient evaluations (Ng) using Charbonnier, Huber and Green TV regularizations for the Middlebury datasets (top) and the image sequences given in Figure 2 in the same order (bottom).

	Dimetrodon	Grove2	Grove3	Hydrangea	Rubberwhale	Urban2	Urban3	Venus
Charbonnier	359	364	531	417	592	555	1138	638
Huber	328	332	487	396	581	500	1015	613
Green	296	582	610	534	508	526	779	976
	1/2 Pixel	1 Pixel	10 Pixels	Rotation	Homography	Zoom	Sphere	Book
Charbonnier	685	331	525	231	445	188	150	420
Huber	619	502	592	259	378	192	155	503
Green	525	459	532	319	695	283	312	507

, we provide a comparison with respect to the speed of convergence given by the number of gradient evaluations (Ng).

First, we remark that the Charbonnier and Huber approximations lead to similar results with a slight preference for the latter. Globally, these two approximations perform better than the Green TV regularization in terms of both the average angular error and the average endpoint error of the estimated optical flow solution, and the speed of convergence as shown in

Table 6. Overall performance of Charbonnier, Huber and Green TV regularizations for the sixteen tested images.

	AAE	AEE	DFD	Ng
Charbonnier	61.594	7.516	10.724	7569
Huber	61.378	7.493	10.863	7452
Green	70.762	8.002	9.287	8443

. On a total of sixteen image sequences, Huber method has performed better half of the time in terms of AEE with an average of 3.836 per image sequence. It has also 9 times a better AEE with an average of 0.468. The method needs an average of 466 gradient evaluations to reach the estimated solution. This is slightly better than Charbonnier method, which has averages of 3.849, 0.469 and 473 for AAE, AEE and Ng, respectively. Nevertheless, the Green approximation has the best performance with respect to the interpolation error. The method has performed better on thirteen image sequences out of sixteen with an average DFD of 0.580 per sequence; while Charbonnier and Huber approximations have an average DFD of 0.670 and 0.678, respectively. On the other hand, Green method has better AAE and three different sequences, better AEE for two sequences, and better Ng for four sequences.

We have noticed also that the Green method is very sensitive to the parameter $\epsilon$, which is due to the sensitivity of the hyperbolic function cosh to roundoff errors. The Charbonnier and Huber approximations suffer less from this problem. This might explain their wide use as smooth TV approximations in image processing. In Figure 3, we show the dependence of the estimated solution on the parameter $\epsilon$ for these two approximations using the Yosemite sequence, which was created by Lynn Quam at SRI and first used for optical flow in [26]. The dependence is shown in terms of AAE, AEE and Ng. For the Yosemite sequence with clouds, we can see that both the Charbonnier and Huber TV approximations give similar results for values of $\epsilon$ between ${10}^{-6}$ and $0.1$. For $\epsilon <{10}^{-5}$, Charbonnier approximation is slightly better than Huber approximation but the latter is performing better for values of $\epsilon$ around ${10}^{-4}$. Otherwise, the two approximations are performing almost the same except for large values of $\epsilon$ near $0.1$ where Charbonnier gives slightly better AEE but Huber has better AAE. In Figure 4 and

Table 7. Average Angular Error (AAE), Average Endpoint Error (AEE) and number of gradient evaluations (Ng) using best parameters for Charbonnier and Huber TV regularizations on the Yosemite sequence with clouds (Yosemitec) and without clouds (Yosemite).

	Yosemitec		Yosemite
	Charbonnier	Huber	Charbonnier	Huber
AAE	4.773	4.794	2.286	2.338
EPE	0.280	0.288	0.113	0.114
DFD	1.433	1.440	0.936	0.937
Ng	619	489	532	418

, the results are shown using the best parameters.

5. Conclusions

We have investigated the use of three smooth approximations of the TV regularization in the context of the optical flow problem. We have used the same non linear data optical flow term and the same multilevel truncated Newton algorithm for the three approximations. On sixteen tested image sequences, the Huber function has confirmed its best theoretical approximation with an overall better performance in terms of both the quality of the estimated optical flow and the speed of convergence. Although the Charbonnier function has the worst theoretical approximation, it has performed almost the same as the Huber function and better than the Green function. On the other hand, in terms of the interpolation error, the Green function appears to be the best method. It has performed better on thirteen images out of sixteen.