Computational Optimization of the 3D Least-Squares Matching Algorithm by Direct Calculation of Normal Equations

Abstract

3D least-squares matching is an algorithm that allows to measure subvoxel-precise displacements between two data sets of computed tomography voxel data. The determination of precise displacement vector fields is an important tool for deformation analyses in in-situ X-ray micro-tomography time series. The goal of the work presented in this publication is the development and validation of an optimized algorithm for 3D least-squares matching saving computation time and memory. 3D least-squares matching is a gradient-based method to determine geometric (and optionally also radiometric) transformation parameters between consecutive cuboids in voxel data. These parameters are obtained by an iterative Gauss-Markov process. Herein, the most crucial point concerning computation time is the calculation of the normal equations using matrix multiplications. In the paper at hand, a direct normal equation computation approach is proposed, minimizing the number of computation steps. A theoretical comparison shows, that the number of multiplications is reduced by 28% and the number of additions by 17%. In a practical test, the computation time of the 3D least-squares matching algorithm was proven to be reduced by 27%.

1. Introduction

Volume data analysis in a voxel space representation is a logical extension of 2D image data processing. In the field of Photogrammetry and Computer Vision, image correlation techniques are often used for fine measurement of corresponding points in image sequences or stereo image pairs. Early contributions applied cross-correlation to image data [1]. In the early 80’s, gradient based algorithms were developed using an iterative least squares method [2] that also offers subpixel precision. Lucas and Kanade [2] only introduced two translation parameters, whereas Ackermann [3] and Grün [4] extended the model toward an affine and radiometric transformation, which also considers rotation, scaling and shear to obtain a better adaption for perspective distortions. As an example for the least-squares matching (LSM) algorithm, an image section of a textured concrete surface with two views is illustrated in Figure 1. Figure 1a shows a section of the reference image with the patch (subimage) consisting of the pixels included in the computation. The pixels are illustrated as red squares with red circles in the centers. In Figure 1b, an image section of the second view as well as the transformed patch (red squares with red circles) are depicted. In this example, the transformation includes translation, rotation, scale and shear and the corresponding parameters are obtained by the LSM algorithm. The most important parameters are the shifts, which typically show an accuracy of 0.02 to 0.05 pixels in this kind of application.

It is also possible to use more complex geometric models for the perspective transformation [5]. Grün and Baltsavias [6] introduced geometrically constrained multiphoto matching, where the solution can inherently be forced to the epipolar line. Other contributions combined digital image matching and object surface reconstruction. In addition to a geometric 3D model of the surface, they used a radiometric model for the surface and the images. The unknowns of the surface densities, 3D points and image orientation parameters were computed in an iterative least-squares adjustment [7,8].

Maas et al. [9] transferred the idea from the 2D image matching to 3D voxel data (digital volume correlation DVC). They applied the subvoxel-precise cuboid tracking (3D least-squares matching) with 12 parameters of a 3D affine transformation to a sequence of multi-temporal voxel representations of mixing fluids. Bay et al. [10] extended the cross-correlation technique to 3D, combined it with a subvoxel-precise refinement and applied it to X-ray tomography data for strain analysis. Liebold et al. [11] also introduced a radiometric transformation to 3D cuboid matching.

In the last years, also free DVC code was provided. For instance, TomoWarp2 ensures the computation of 3D displacement fields using 3D cross-correlation [12]. They also implemented a further step to calculate subvoxel positions considering the maximum cross-correlation integer position and the 26 neighbor voxel around. The free software package SPAM is based on TomoWarp2 [13] and also provides a tool for the coarse registration that can consider a rotation between the reference and the deformed volume. In ALDVC [14], a Fast Fourier Transform (FFT) is applied for the estimation of initial shift values, followed by an algorithm including a 12-parameter affine transformation similar to 3D least-squares matching for computing subvoxel positions.

The least-squares matching algorithm contains an adjustment procedure where normal equations are computed. To speed up the process, Sutton et al. [15] showed a direct computation of the normal equations for 2D least-squares matching including affine parameters, and Jin et al. [16] also included radiometric parameters.

The publication at hand is based on the work of Maas et al. [9] and Liebold et al. [11]. It presents an optimized algorithm for the 3D LSM, likewise to what has been shown for 2D [15,16]. The central goal of the work is the acceleration of the 3D algorithm in order to save computation time and memory, without restrictions to the universality of the algorithm. The paper is structured as follows: In the next section, an overview of the computation of 3D least-squares matching is given including the optimization. In addition, a comparison to the standard method is shown. The publication closes with a conclusion.

2. 3D Least-Squares Matching

2.1. Mathematical Model

In the following, some formulae of Liebold et al. [11] are reused and extended. 3D least-squares matching is a gradient-based method and is applied to two voxel data sets. In the reference volume, a small cuboid is defined containing the voxel of the cuboid’s center (point to be matched) and its neighborhood. The dimensions $di{m}_x$, $di{m}_y$ and $di{m}_z$ are odd numbers, and a typical cuboid size is for example $13\times 13\times 13=2197$ voxel. The aim is to find the corresponding cuboid in the second volume data set. The coordinates of the center of the cuboid in the reference state $x_{Ref,c},y_{Ref,c},z_{Ref,c}$ are integer values, whereas the coordinates of the voxel in the second state $x_{Def},y_{Def},z_{Def}$ are floating-point values.

Similar to Figure 1 for 2D, Figure 2 illustrates the 3D case with a voxel grid (gray). In the reference state, a subvolume consisting of e.g., $5\times 5\times 5$ voxel (green cuboid with red edges and red center points, Figure 2a). In the second state, depicted in Figure 2b, the affine-transformed cuboid as well as the faded reference subvolume are shown. The shift vector that connects the center points of the subvolumes is colored in magenta. 3D LSM is used to determine the transformation parameters, including the three translations.

The range of the cuboid in the reference volume is:

$$\begin{array}{cccc}& x_{Ref}\in [x_{Ref,c}-h_x,x_{Ref,c}+h_x]& \wedge & x_{Ref}\in \mathbb{N}\\ & y_{Ref}\in [y_{Ref,c}-h_y,y_{Ref,c}+h_y]& \wedge & y_{Ref}\in \mathbb{N}\\ & z_{Ref}\in [z_{Ref,c}-h_z,z_{Ref,c}+h_z]& \wedge & z_{Ref}\in \mathbb{N}\end{array}$$

(1)

where $h_x=\frac{{dim}_x-1}{2}$; $h_y=\frac{{dim}_y-1}{2}$; $h_z=\frac{{dim}_z-1}{2}$

Equation (2) shows the relationship between the gray values of the first (reference) and second (deformed) volume, also taking into account radiometric parameters ($r_0,r_1$). This relationship is valid for each voxel of the cuboid.

$$f(x_{Ref},y_{Ref},z_{Ref})=r_0+r_1·g(x_{Def},y_{Def},z_{Def})$$

(2)

where

• $x_{Ref},y_{Ref},z_{Ref}= coordinates\; (reference\; state)$
• $x_{Def},y_{Def},z_{Def}= \mathrm{coordinates\; (deformed\; state)}$
• $f(x_{Ref},y_{Ref},x_{Ref})= gray\; value\; (reference\; state)$
• $g(x_{Def},y_{Def},z_{Def})= gray\; value\; (deformed\; state)$
• $r_0,r_0= brightness\; and\; contrast\; parameters$

Between the reference and the deformed state, a coordinate transformation is performed. Similar to [3] in 2D and [9] in 3D, an affine transformation is used as mathematical model containing displacements, rotations, scaling and shear, see Equation (3), where $a_0,b_0$ and $c_0$ are the translation parameters.

$$\begin{array}{ccc}{x}_{Def}& =& x_{Ref,c}+a_0+a_1·\tilde{x}+a_2·\tilde{y}+a_3·\tilde{z}\\ y_{Def}& =& y_{Ref,c}+b_0+b_1·\tilde{x}+b_2·\tilde{y}+b_3·\tilde{z}\\ z_{Def}& =& z_{Ref,c}+c_0+c_1·\tilde{x}+c_2·\tilde{y}+c_3·\tilde{z}\end{array}$$

(3)

where

• $a_i,b_i,c_i= parameters\; of\; the\; affine\; transformation$
• $\tilde{x},\tilde{y},\tilde{z}= reduced\; reference\; coordinates$

The reduced reference coordinates are computed as follows:

$$\begin{array}{ccc}\tilde{x}& =& x_{Ref}-x_{Ref,c}\\ \tilde{y}& =& y_{Ref}-y_{Ref,c}\\ \tilde{z}& =& z_{Ref}-z_{Ref,c}\end{array}$$

(4)

Thus, the vector of unknowns is:

$$\mathbf{p}={\left(\begin{array}{cccccccccccccc}{a}_0,& a_1,& a_2,& a_3,& b_0,& b_1,& b_2,& b_3,& c_0,& c_1,& c_2,& c_3,& r_0,& r_1\end{array}\right)}^T$$

(5)

The parameter vector $\mathbf{p}$ consists of 14 unknowns, and a cuboid with an exemplary size of $13\times 13\times 13=2197$ voxel (observations) leads to an over-determination. Thus, the parameters of $\mathbf{p}$ are computed in a least-squares adjustment process. Therefore, residuals are added to Equation (2) that results in the observation equations:

$$f(x_{Ref},y_{Ref},z_{Ref})+v(x_{Ref},y_{Ref},z_{Ref})=r_0+r_1·g(x_{Def},y_{Def},z_{Def})$$

(6)

where $v(x_{Ref},y_{Ref},z_{Ref})$ = residuum

The first terms of the Taylor series expansion are used to linearize the observation equation, see Equation (7) where initial values of the parameters are included. They are marked with an additional zero subscript. Section 2.2 shows a way to obtain initial values for 3D LSM.

$$\begin{array}{c}f(x_{Ref},y_{Ref},z_{Ref})+v(x_{Ref},y_{Ref},z_{Ref})\approx r_{0,0}+\mathrm{d}{r}_0+(r_{1,0}+\mathrm{d}{r}_1)·g(x_{Def,0},y_{Def,0},z_{Def,0})\hfill \\ \hfill +r_1·(g_{x,0}·\mathrm{d}{x}_{Def}+g_{y,0}·\mathrm{d}{y}_{Def}+g_{z,0}·\mathrm{d}{z}_{Def})\end{array}$$

(7)

where

• $r_{\mathrm{0,0}}= \mathrm{initial\; value\; of\; the\; brightness\; parameter} r_0$
• $r_{\mathrm{1,0}}= \mathrm{initial\; value\; of\; the\; contrast\; parameter} r_1$
• $x_{Def,0}= x \mathrm{position\; (deformed\; state)\; computed\; with\; the\; initial\; values} a_{i,0}of{a}_i$
• $y_{Def,0}= y \mathrm{position\; (deformed\; state)\; computed\; with\; the\; initial\; values} b_{i,0}of{b}_i$
• $z_{Def,0}= z \mathrm{position\; (deformed\; state)\; computed\; with\; the\; initial\; values} c_{i,0}of{c}_i$
• $g_{x,0},g_{y,0},g_{z,0}= volume\; gradients\; at\; the\; initial\; positions x_{Def,0},y_{Def,0},z_{Def,0}$

The volume gradients of the gray values are written with their short forms in Equation (7). The full notation is:

$$\begin{array}{cc}{g}_{x,0}=& \frac{\partial g(x_{Def},y_{Def},z_{Def})}{\partial x_{Def}}{|}_{x_{Def},y_{Def},z_{Def}=x_{Def,0},y_{Def,0},z_{Def,0}}\\ g_{y,0}=& \frac{\partial g(x_{Def},y_{Def},z_{Def})}{\partial y_{Def}}{|}_{x_{Def},y_{Def},z_{Def}=x_{Def,0},y_{Def,0},z_{Def,0}}\\ g_{z,0}=& \frac{\partial g(x_{Def},y_{Def},z_{Def})}{\partial z_{Def}}{|}_{x_{Def},y_{Def},z_{Def}=x_{Def,0},y_{Def,0},z_{Def,0}}\end{array}$$

(8)

There are different ways to obtain these gradients: One possibility is the computation of central differences $g_{x,i},g_{y,i},g_{z,i}$ (numerical differentiation, Equation (9)) at integer positions $x_{Def,i},y_{Def,i},z_{Def,i}$ ($x_{Def,i}\in \mathbb{N},y_{Def,i}\in \mathbb{N},z_{Def,i}\in \mathbb{N}$) combined with a tri-linear interpolation in order to calculate the derivatives at the floating point values of $x_{Def},y_{Def}$ and $z_{Def}$.

$$\begin{array}{cc}{g}_{x,i}\approx & 0.5·(g(x_{Def,i}+1,y_{Def,i},z_{Def,i})-g(x_{Def,i}-1,y_{Def,i},z_{Def,i}))\\ g_{y,i}\approx & 0.5·(g(x_{Def,i},y_{Def,i}+1,z_{Def,i})-g(x_{Def,i},y_{Def,i}-1,z_{Def,i}))\\ g_{z,i}\approx & 0.5·(g(x_{Def,i},y_{Def,i},z_{Def,i}+1)-g(x_{Def,i},y_{Def,i},z_{Def,i}-1))\end{array}$$

(9)

where

• $g_{x,i}= derivative\; in x direction\; at\; integer\; positions$
• $g_{y,i}= derivative\; in y direction\; at\; integer\; positions$
• $g_{z,i}= derivative\; in z direction\; at\; integer\; positions$

Another approach is the use of tri-cubic spline interpolation to compute the derivatives [17] (derivatives of cubic polynomials). And a further possibility is the calculation of the derivatives in the reference volume as an approximation. The advantage of the last point is once-only computation in the iterative least-squares process.

Thus, the differentials $\mathrm{d}{x}_{Def}$, $\mathrm{d}{y}_{Def}$ and $\mathrm{d}{z}_{Def}$ are:

$$\begin{array}{cc}\mathrm{d}{x}_{Def}& =\sum _{i=0}^3\frac{\partial x_{Def}}{\partial a_i}·\mathrm{d}{a}_i=\mathrm{d}{a}_0+\tilde{x}·\mathrm{d}{a}_1+\tilde{y}·\mathrm{d}{a}_2+\tilde{z}·\mathrm{d}{a}_3\\ \mathrm{d}{y}_{Def}& =\sum _{i=0}^3\frac{\partial y_{Def}}{\partial b_i}·\mathrm{d}{b}_i=\mathrm{d}{b}_0+\tilde{x}·\mathrm{d}{b}_1+\tilde{y}·\mathrm{d}{b}_2+\tilde{z}·\mathrm{d}{b}_3\\ \mathrm{d}{z}_{Def}& =\sum _{i=0}^3\frac{\partial z_{Def}}{\partial c_i}·\mathrm{d}{c}_i=\mathrm{d}{c}_0+\tilde{x}·\mathrm{d}{c}_1+\tilde{y}·\mathrm{d}{c}_2+\tilde{z}·\mathrm{d}{c}_3\end{array}$$

(10)

where

• ${dx}_{Def}= \mathrm{differential\; of\; the} x \mathrm{coordinate}$
• ${dy}_{Def}= \mathrm{differential\; of\; the} y \mathrm{coordinate}$
• ${dz}_{Def}= \mathrm{differential\; of\; the} z \mathrm{coordinate}$

The vector of corrections to the unknowns is:

$$\mathrm{d}\mathbf{p}={\left(\begin{array}{cccccccccccccc}\mathrm{d}{a}_0,& \mathrm{d}{a}_1,& \mathrm{d}{a}_2,& \mathrm{d}{a}_3,& \mathrm{d}{b}_0,& \mathrm{d}{b}_1,& \mathrm{d}{b}_2,& \mathrm{d}{b}_3,& \mathrm{d}{c}_0,& \mathrm{d}{c}_1,& \mathrm{d}{c}_2,& \mathrm{d}{c}_3,& \mathrm{d}{r}_0,& \mathrm{d}{r}_1\end{array}\right)}^T$$

(11)

The vector of unknowns results from the sum of the vector of the initial values ${\mathbf{p}}_0$ and the vector of the corrections to the unknowns $\mathrm{d}\mathbf{p}$:

$$\mathbf{p}={\mathbf{p}}_0+\mathrm{d}\mathbf{p}$$

(12)

The linearized observation equations can be written in matrix notation:

$$\mathbf{A}·\mathrm{d}\mathbf{p}=\mathbf{l}+\mathbf{v}$$

(13)

where

• A = Jacobian matrix
• l = reduced observation vector
• v = residual vector

The Jacobian matrix is built up as follows:

$$\mathbf{A}=\left(\begin{array}{cccc}{\mathbf{A}}_1& {\mathbf{A}}_2& {\mathbf{A}}_3& {\mathbf{A}}_4\end{array}\right)$$

(14)

where

$${\mathbf{A}}_1=\left(\begin{array}{cccc}\begin{array}{c}\hfill \frac{\partial f}{\partial a_0}{|}_{\mathbf{p}={\mathbf{p}}_0}\end{array}& \frac{\partial f}{\partial a_1}{|}_{\mathbf{p}={\mathbf{p}}_0}& \frac{\partial f}{\partial a_2}{|}_{\mathbf{p}={\mathbf{p}}_0}& \frac{\partial f}{\partial a_3}{|}_{\mathbf{p}={\mathbf{p}}_0}\\ ⋮& ⋮& ⋮& ⋮\end{array}\right)=r_1·\left(\begin{array}{cccc}\hfill g_{x,1}& g_{x,1}·{\tilde{x}}_1& g_{x,1}·{\tilde{y}}_1\hfill & g_{x,1}·{\tilde{z}}_1\\ \hfill g_{x,2}& g_{x,2}·{\tilde{x}}_2& g_{x,2}·{\tilde{y}}_2\hfill & g_{x,2}·{\tilde{z}}_2\\ \hfill ⋮& ⋮& ⋮\hfill & ⋮\\ \hfill g_{x,n}& g_{x,n}·{\tilde{x}}_n& g_{x,n}·{\tilde{y}}_n\hfill & g_{x,n}·{\tilde{z}}_n\end{array}\right)$$

(15)

$${\mathbf{A}}_2=\left(\begin{array}{cccc}\begin{array}{c}\hfill \frac{\partial f}{\partial b_0}{|}_{\mathbf{p}={\mathbf{p}}_0}\end{array}& \frac{\partial f}{\partial b_1}{|}_{\mathbf{p}={\mathbf{p}}_0}& \frac{\partial f}{\partial b_2}{|}_{\mathbf{p}={\mathbf{p}}_0}& \frac{\partial f}{\partial b_3}{|}_{\mathbf{p}={\mathbf{p}}_0}\\ ⋮& ⋮& ⋮& ⋮\end{array}\right)=r_1·\left(\begin{array}{cccc}\hfill g_{y,1}& g_{y,1}·{\tilde{x}}_1& g_{y,1}·{\tilde{y}}_1\hfill & g_{y,1}·{\tilde{z}}_1\\ \hfill g_{y,2}& g_{y,2}·{\tilde{x}}_2& g_{y,2}·{\tilde{y}}_2\hfill & g_{y,2}·{\tilde{z}}_2\\ \hfill ⋮& ⋮& ⋮\hfill & ⋮\\ \hfill g_{y,n}& g_{y,n}·{\tilde{x}}_n& g_{y,n}·{\tilde{y}}_n\hfill & g_{y,n}·{\tilde{z}}_n\end{array}\right)$$

(16)

$${\mathbf{A}}_3=\left(\begin{array}{cccc}\begin{array}{c}\hfill \frac{\partial f}{\partial c_0}{|}_{\mathbf{p}={\mathbf{p}}_0}\end{array}& \frac{\partial f}{\partial c_1}{|}_{\mathbf{p}={\mathbf{p}}_0}& \frac{\partial f}{\partial c_2}{|}_{\mathbf{p}={\mathbf{p}}_0}& \frac{\partial f}{\partial c_3}{|}_{\mathbf{p}={\mathbf{p}}_0}\\ ⋮& ⋮& ⋮& ⋮\end{array}\right)=r_1·\left(\begin{array}{cccc}\hfill g_{z,1}& g_{z,1}·{\tilde{x}}_1& g_{z,1}·{\tilde{y}}_1\hfill & g_{z,1}·{\tilde{z}}_1\\ \hfill g_{z,2}& g_{z,2}·{\tilde{x}}_2& g_{z,2}·{\tilde{y}}_2\hfill & g_{z,2}·{\tilde{z}}_2\\ \hfill ⋮& ⋮& ⋮\hfill & ⋮\\ \hfill g_{z,n}& g_{z,n}·{\tilde{x}}_n& g_{z,n}·{\tilde{y}}_n\hfill & g_{z,n}·{\tilde{z}}_n\end{array}\right)$$

(17)

$${\mathbf{A}}_4=\begin{array}{c}\left(\begin{array}{cc}\frac{\partial f}{\partial r_0}{|}_{\mathbf{p}={\mathbf{p}}_0}& \frac{\partial f}{\partial r_1}{|}_{\mathbf{p}={\mathbf{p}}_0}\\ ⋮& ⋮\end{array}\right)=\left(\begin{array}{cc}\hfill 1& g_1\\ \hfill 1& g_2\\ \hfill ⋮& ⋮\\ \hfill 1& g_n\end{array}\right)\end{array}$$

(18)

The reduced observation vector $\mathbf{l}$ is:

$$\mathbf{l}=\left(\begin{array}{c}{f}_1-r_0-r_1·g_1\\ f_2-r_0-r_1·g_2\\ ⋮\\ f_n-r_0-r_1·g_n\end{array}\right)$$

(19)

In the Gauss-Markov model, the weighted sum of the squared residuals is minimized:

$${\mathbf{v}}^T·\mathbf{W}·\mathbf{v}\to \underset{\mathrm{d}\mathbf{p}}{\min }$$

(20)

where $\mathbf{W}$ = matrix of weights

In the following, only the case of equally weighted observations ($\mathbf{W}=\mathbf{I}$, $\mathbf{I}$: identity matrix) is considered, see Equation (21).

$${\mathbf{v}}^T·\mathbf{v}\to \underset{\mathrm{d}\mathbf{p}}{\min }$$

(21)

The solution of this problem can be obtained using the normal equations:

$${\mathbf{A}}^T·\mathbf{A}·\mathrm{d}\mathbf{p}={\mathbf{A}}^T·\mathbf{l}$$

(22)

As the computation of the normal matrix ${\mathbf{A}}^T·\mathbf{A}$ and the right hand side ${\mathbf{A}}^T·\mathbf{l}$ are done directly, it is not necessary to build up and store the whole Jacobian matrix $\mathbf{A}$ and the reduced observation vector $\mathbf{l}$. The detailed computation process is shown in Section 2.3.

After setting up the matrices ${\mathbf{A}}^T·\mathbf{A}$ and ${\mathbf{A}}^T·\mathbf{l}$, Equation (22) is solved for $\mathrm{d}\mathbf{p}$ and the initial values of the unknowns are updated:

$$\mathbf{p}={\mathbf{p}}_0+\mathrm{d}\mathbf{p}$$

(23)

The steps of the interpolation of the gray values and their derivatives as well as the computation of ${\mathbf{A}}^T·\mathbf{A}$ and ${\mathbf{A}}^T·\mathbf{l}$, the determination of $\mathrm{d}\mathbf{p}$ and the update of the unknowns are repeated until the process converges.

After the iterative adjustment routine, the residuals are determined by rearranging Equation (6). The gray values $g_i$ are obtained by applying Equation (3) with the updated affine parameters and tri-linear (or tri-cubic) interpolation. The individual residual $v_i$ is:

$$v_i=r_0+r_1·g_i(x_{Def},y_{Def},z_{Def})-f_i(x_{Ref},y_{Ref},z_{Ref})$$

(24)

where $v_i$ = individual residual

The standard deviation of the unit weight is:

$$s_0=\sqrt{\frac{{\sum }_{i=1}^n{v}_i^2}{n-u}}$$

(25)

where

• s₀ = standard deviation of the unit weight
• n = number of voxel of the cuboid
• n = number of unknowns

2.2. Initial Values

The model in Equation (2) is not linear and if the movements between the volume data sets may exceed the dimensions of the cuboid, initial values have to be obtained for the 3D LSM algorithm.

Often, mainly translations and only small rotations occur between different epochs of measurements. In these cases, 3D cross-correlation can be used to obtain initial shifts [10]. An overview of the optimized algorithm can be found in Appendix A.

2.3. Direct Computation of the Normal Equations

As mentioned above, the normal matrix (${\mathbf{A}}^T·\mathbf{A}$) and the right hand side vector (${\mathbf{A}}^T·\mathbf{l}$) are computed directly, so that building up the Jacobian matrix $\mathbf{A}$ as well as the reduced observation vector $\mathbf{l}$ and the matrix multiplications can be avoided in order to gain efficiency and save memory. To compute the terms of the normal equations, the dot products of the columns of the Jacobian matrix (Equation (14)) as well as the dot products with the reduced observation vector are calculated (each combination of columns). Equation (26) shows the normal matrix that is organized in submatrices. ${\mathbf{G}}_{11,i}$, ${\mathbf{G}}_{12,i}$, ${\mathbf{G}}_{13,i}$, ${\mathbf{G}}_{22,i}$, ${\mathbf{G}}_{23,i}$, ${\mathbf{G}}_{33,i}$ and ${\mathbf{G}}_{44,i}$ as well as the whole normal matrix are symmetric matrices and only the upper triangular matrices are shown. In Equation (28), the right hand side of the normal equations is depicted.

$${\mathbf{A}}^T·\mathbf{A}=\left(\begin{array}{cccc}{\mathbf{A}}_1^T·{\mathbf{A}}_1& {\mathbf{A}}_1^T·{\mathbf{A}}_2& {\mathbf{A}}_1^T·{\mathbf{A}}_3& {\mathbf{A}}_1^T·{\mathbf{A}}_4\\ & {\mathbf{A}}_2^T·{\mathbf{A}}_2& {\mathbf{A}}_2^T·{\mathbf{A}}_3& {\mathbf{A}}_2^T·{\mathbf{A}}_4\\ & & {\mathbf{A}}_3^T·{\mathbf{A}}_3& {\mathbf{A}}_3^T·{\mathbf{A}}_4\\ & & & {\mathbf{A}}_4^T·{\mathbf{A}}_4\end{array}\right)=\left(\begin{array}{cccc}\hfill r_1^2·\sum _{i=1}^n{\mathbf{G}}_{11,i}& r_1^2·\sum _{i=1}^n{\mathbf{G}}_{12,i}& r_1^2·\sum _{i=1}^n{\mathbf{G}}_{13,i}\hfill & r_1·\sum _{i=1}^n{\mathbf{G}}_{14,i}\\ \hfill & r_1^2·\sum _{i=1}^n{\mathbf{G}}_{22,i}& r_1^2·\sum _{i=1}^n{\mathbf{G}}_{23,i}\hfill & r_1·\sum _{i=1}^n{\mathbf{G}}_{24,i}\\ \hfill & & r_1^2·\sum _{i=1}^n{\mathbf{G}}_{33,i}\hfill & r_1·\sum _{i=1}^n{\mathbf{G}}_{34,i}\\ \hfill & & & \sum _{i=1}^n{\mathbf{G}}_{44,i}\end{array}\right)$$

(26)

where

$$\begin{gathered} {\mathbf{G}}_{11,i}=\left(\begin{array}{cccc}\begin{array}{c}\hfill g_{x,i}^2\end{array}& g_{x,i}^2·{\tilde{x}}_i& g_{x,i}^2·{\tilde{y}}_i& g_{x,i}^2·{\tilde{z}}_i\\ & g_{x,i}^2·{\tilde{x}}_i^2& g_{x,i}^2·{\tilde{x}}_i·{\tilde{y}}_i& g_{x,i}^2·{\tilde{x}}_i·{\tilde{z}}_i\\ & & g_{x,i}^2·{\tilde{y}}_i^2& g_{x,i}^2·{\tilde{y}}_i·{\tilde{z}}_i\\ & & & g_{x,i}^2·{\tilde{z}}_i^2\end{array}\right),{\mathbf{G}}_{12,i}=\left(\begin{array}{cccc}\begin{array}{c}\hfill g_{x,i}·g_{y,i}\end{array}& g_{x,i}·gy{x}_i·{\tilde{x}}_i& g_{x,i}·gy{y}_i·{\tilde{y}}_i& g_{x,i}·gy{z}_i·{\tilde{z}}_i\\ & gx{x}_i·gx{y}_i·{\tilde{x}}_i^2& gx{x}_i·gy{y}_i·{\tilde{x}}_i·{\tilde{y}}_i& g{x}_i·g{y}_i·{\tilde{x}}_i·{\tilde{z}}_i\\ & & g{x}_i·g{y}_i·{\tilde{y}}_i^2& g{x}_i·g{y}_i·{\tilde{y}}_i·{\tilde{z}}_i\\ & & & g{x}_i·g{y}_i·{\tilde{z}}_i^2\end{array}\right), \\ {\mathbf{G}}_{22,i}=\left(\begin{array}{cccc}\begin{array}{c}\hfill g_{y,i}^2\end{array}& g_{y,i}^2·{\tilde{x}}_i& g_{y,i}^2·{\tilde{y}}_i& g_{y,i}^2·{\tilde{z}}_i\\ & g_{y,i}^2·{\tilde{x}}_i^2& g_{y,i}^2·{\tilde{x}}_i·{\tilde{y}}_i& g_{y,i}^2·{\tilde{x}}_i·{\tilde{z}}_i\\ & & g_{y,i}^2·{\tilde{y}}_i& g_{y,i}^2·{\tilde{y}}_i·{\tilde{z}}_i\\ & & & g_{y,i}^2·{\tilde{z}}_i^2\end{array}\right),{\mathbf{G}}_{13,i}=\left(\begin{array}{cccc}\begin{array}{c}\hfill g_{x,i}·g_{z,i}\end{array}& g_{x,i}·gz{x}_i·{\tilde{x}}_i& g_{x,i}·gz{y}_i·{\tilde{y}}_i& g_{x,i}·gz{z}_i·{\tilde{z}}_i\\ & g_{x,i}·g_{z,i}·{\tilde{x}}_i^2& g_{x,i}·g_{z,i}·{\tilde{x}}_i·{\tilde{y}}_i& g_{x,i}·g_{z,i}·{\tilde{x}}_i·{\tilde{z}}_i\\ & & g_{x,i}·g_{z,i}·{\tilde{y}}_i^2& g_{x,i}·g_{z,i}·{\tilde{y}}_i·{\tilde{z}}_i\\ & & & g_{x,i}·g_{z,i}·{\tilde{z}}_i^2\end{array}\right), \\ {\mathbf{G}}_{23,i}=\left(\begin{array}{cccc}\begin{array}{c}\hfill g_{y,i}·g_{z,i}\end{array}& g_{y,i}·g_{z,i}··{\tilde{x}}_i& g_{y,i}·g_{z,i}·{\tilde{y}}_i& g_{y,i}·g_{z,i}·{\tilde{z}}_i\\ & g_{y,i}·g_{z,i}·{\tilde{x}}_i^2& g_{y,i}·g_{z,i}·{\tilde{x}}_i·{\tilde{y}}_i& g_{y,i}·g_{z,i}·{\tilde{x}}_i·{\tilde{z}}_i\\ & & g_{y,i}·g_{z,i}·{\tilde{y}}_i^2& g_{y,i}·g_{z,i}·{\tilde{y}}_i·{\tilde{z}}_i\\ & & & g_{y,i}·g_{z,i}·{\tilde{z}}_i^2\end{array}\right),{\mathbf{G}}_{33,i}=\left(\begin{array}{cccc}\begin{array}{c}\hfill g_{z,i}^2\end{array}& g_{z,i}^2·{\tilde{y}}_i& g_{z,i}^2·{\tilde{y}}_i& g_{z,i}^2·{\tilde{z}}_i\\ & g_{z,i}^2·{\tilde{x}}_i^2& g_{z,i}^2·{\tilde{x}}_i·{\tilde{y}}_i& g_{z,i}^2·{\tilde{x}}_i·{\tilde{z}}_i\\ & & g_{z,i}^2·{\tilde{y}}_i^2& g_{z,i}^2·{\tilde{y}}_i·{\tilde{z}}_i\\ & & & g_{z,i}^2·{\tilde{z}}_i^2\end{array}\right), \\ {\mathbf{G}}_{14,i}=\left(\begin{array}{cc}\begin{array}{c}\hfill g_{x,i}\end{array}& g_i·g_{x,i}\\ g_{x,i}·{\tilde{x}}_i& g_i·g_{x,i}·{\tilde{x}}_i\\ g_{x,i}·{\tilde{x}}_i& g_i·g_{x,i}·{\tilde{x}}_i\\ g_{x,i}·{\tilde{z}}_i& g_i·g_{x,i}·{\tilde{z}}_i\end{array}\right),{\mathbf{G}}_{24,i}=\left(\begin{array}{cc}\begin{array}{c}\hfill g_{y,i}\end{array}& g_i·g_{y,i}\\ g_{y,i}·{\tilde{x}}_i& g_i·g_{y,i}·{\tilde{x}}_i\\ g_{y,i}·{\tilde{y}}_i& g_i·g_{y,i}·{\tilde{y}}_i\\ g_{y,i}·{\tilde{z}}_i& g_i·g_{y,i}·{\tilde{z}}_i\end{array}\right),{\mathbf{G}}_{34,i}=\left(\begin{array}{cc}\begin{array}{c}\hfill g_{z,i}\end{array}& g_i·g_{z,i}\\ g_{z,i}·{\tilde{x}}_i& g_i·g_{z,i}·{\tilde{x}}_i\\ g_{z,i}·{\tilde{y}}_i& g_i·g_{z,i}·{\tilde{y}}_i\\ g_{z,i}·{\tilde{z}}_i& g_i·g_{z,i}·{\tilde{z}}_i\end{array}\right),{\mathbf{G}}_{44,i}=\left(\begin{array}{cc}\hfill 1& g_i\\ & g_i^2\end{array}\right) \end{gathered}$$

(27)

$${\mathbf{A}}^T·\mathbf{l}=\left(\begin{array}{c}{\mathbf{A}}_1^T·\mathbf{l}\\ {\mathbf{A}}_2^T·\mathbf{l}\\ {\mathbf{A}}_3^T·\mathbf{l}\\ {\mathbf{A}}_4^T·\mathbf{l}\end{array}\right)=\left(\begin{array}{c}{r}_1·\sum _{i=1}^n{f}_i·g_{x,i}-r_0·r_1·\sum _{i=1}^n{g}_{x,i}-r_1^2·\sum _{i=1}^n{g}_i·g_{x,i}\\ r_1·\sum _{i=1}^n{f}_i·g_{x,i}·{\tilde{x}}_i-r_0·r_1·\sum _{i=1}^n{g}_{x,i}·{\tilde{x}}_i-r_1^2·\sum _{i=1}^n{g}_i·g_{x,i}·{\tilde{x}}_i\\ r_1·\sum _{i=1}^n{f}_i·g_{x,i}·{\tilde{y}}_i-r_0·r_1·\sum _{i=1}^n{g}_{x,i}·{\tilde{y}}_i-r_1^2·\sum _{i=1}^n{g}_i·g_{x,i}·{\tilde{y}}_i\\ r_1·\sum _{i=1}^n{f}_i·g_{x,i}·{\tilde{z}}_i-r_0·r_1·\sum _{i=1}^n{g}_{x,i}·{\tilde{z}}_i-r_1^2·\sum _{i=1}^n{g}_i·g_{x,i}·{\tilde{z}}_i\\ r_1·\sum _{i=1}^n{f}_i·g_{y,i}-r_0·r_1·\sum _{i=1}^n{g}_{y,i}-r_1^2·\sum _{i=1}^n{g}_i·g_{y,i}\\ r_1·\sum _{i=1}^n{f}_i·g_{y,i}·{\tilde{x}}_i-r_0·r_1·\sum _{i=1}^n{g}_{y,i}·{\tilde{x}}_i-r_1^2·\sum _{i=1}^n{g}_i·g_{y,i}·{\tilde{x}}_i\\ r_1·\sum _{i=1}^n{f}_i·g_{y,i}·{\tilde{y}}_i-r_0·r_1·\sum _{i=1}^n{g}_{y,i}·{\tilde{y}}_i-r_1^2·\sum _{i=1}^n{g}_i·g_{y,i}·{\tilde{y}}_i\\ r_1·\sum _{i=1}^n{f}_i·g_{y,i}·{\tilde{z}}_i-r_0·r_1·\sum _{i=1}^n{g}_{y,i}·{\tilde{z}}_i-r_1^2·\sum _{i=1}^n{g}_i·g_{y,i}·{\tilde{z}}_i\\ r_1·\sum _{i=1}^n{f}_i·g_{z,i}-r_0·r_1·\sum _{i=1}^n{g}_{z,i}-r_1^2·\sum _{i=1}^n{g}_i·g_{z,i}\\ r_1·\sum _{i=1}^n{f}_i·g_{z,i}·{\tilde{x}}_i-r_0·r_1·\sum _{i=1}^n{g}_{z,i}·{\tilde{x}}_i-r_1^2·\sum _{i=1}^n{g}_i·g_{z,i}·{\tilde{x}}_i\\ r_1·\sum _{i=1}^n{f}_i·g_{z,i}·{\tilde{y}}_i-r_0·r_1·\sum _{i=1}^n{g}_{z,i}·{\tilde{y}}_i-r_1^2·\sum _{i=1}^n{g}_i·g_{z,i}·{\tilde{y}}_i\\ r_1·\sum _{i=1}^n{f}_i·g_{z,i}·{\tilde{z}}_i-r_0·r_1·\sum _{i=1}^n{g}_{z,i}·{\tilde{z}}_i-r_1^2·\sum _{i=1}^n{g}_i·g_{z,i}·{\tilde{z}}_i\\ \sum _{i=1}^n{f}_i-r_0·n-r_1·\sum _{i=1}^n{g}_i\\ \sum _{i=1}^n{f}_i·g_i-r_0·\sum _{i=1}^n{g}_i-r_1·\sum _{i=1}^n{g}_i^2\end{array}\right)$$

(28)

In fact, the geometric and radiometric parameters are usually independent on each other and are therefore often solved independently in LSM. If only the affine parameters are unknowns and the radiometric parameters are excluded from the model, the last two columns and last two rows of ${\mathbf{A}}^T·\mathbf{A}$ as well as the last two values of ${\mathbf{A}}^T·\mathbf{l}$ are omitted. Appendix B and Appendix C consider the two cases concerning this point.

For each voxel, the following terms are computed and saved in temporal variables because they are used several times:

$$\begin{array}{c}\hfill gx{x}_i=g_{x,i}·{\tilde{x}}_i\\ \hfill gx{y}_i=g_{x,i}·{\tilde{y}}_i\\ \hfill gx{z}_i=g_{x,i}·{\tilde{z}}_i\\ \hfill gy{x}_i=g_{y,i}·{\tilde{x}}_i\\ \hfill gy{y}_i=g_{y,i}·{\tilde{y}}_i\\ \hfill gy{z}_i=g_{y,i}·{\tilde{z}}_i\\ \hfill gz{x}_i=g_{z,i}·{\tilde{x}}_i\\ \hfill gz{y}_i=g_{z,i}·{\tilde{y}}_i\\ \hfill gz{z}_i=g_{z,i}·{\tilde{z}}_i\end{array}$$

(29)

Substituting the terms of Equation (29), the submatrices of the normal matrix and the right hand sight are shown in Equations (30) and (31). This reduces the number of computation steps, too.

$$\begin{gathered} {\mathbf{G}}_{11,i}=\left(\begin{array}{cccc}\begin{array}{c}\hfill g_{x,i}^2\end{array}& g_{x,i}·gx{x}_i& g_{x,i}·gx{y}_i& g_{x,i}·gx{z}_i\\ & gx{x}_i^2& gx{x}_i·gx{y}_i& gx{x}_i·gx{z}_i\\ & & gx{y}_i^2& gx{y}_i·gx{z}_i\\ & & & gx{z}_i^2\end{array}\right),{\mathbf{G}}_{12,i}=\left(\begin{array}{cccc}\begin{array}{c}\hfill g_{x,i}·g_{y,i}\end{array}& g_{x,i}·gy{x}_i& g_{x,i}·gy{y}_i& g_{x,i}·gy{z}_i\\ & gx{x}_i·gy{x}_i& gx{x}_i·gy{y}_i& gx{x}_i·gy{z}_i\\ & & gx{y}_i·gy{y}_i& gx{y}_i·gy{z}_i\\ & & & gx{z}_i·gy{z}_i\end{array}\right), \\ {\mathbf{G}}_{22,i}=\left(\begin{array}{cccc}\begin{array}{c}\hfill g_{y,i}^2\end{array}& g_{y,i}·gy{x}_i& g_{y,i}·gy{y}_i& g_{y,i}·gy{z}_i\\ & gy{x}_i^2& gy{x}_i·gy{y}_i& gy{x}_i·gy{z}_i\\ & & gy{y}_i^2& gy{y}_i·gy{z}_i\\ & & & gy{z}_i^2\end{array}\right),{\mathbf{G}}_{13,i}=\left(\begin{array}{cccc}\begin{array}{c}\hfill g_{x,i}·g_{z,i}\end{array}& g_{x,i}·gz{x}_i& g_{x,i}·gz{y}_i& g_{x,i}·gz{z}_i\\ & gx{x}_i·gz{x}_i& gx{x}_i·gz{y}_i& gx{x}_i·gz{z}_i\\ & & gx{y}_i·gz{y}_i& gx{y}_i·gz{z}_i\\ & & & gx{z}_i·gz{z}_i\end{array}\right), \\ {\mathbf{G}}_{23,i}=\left(\begin{array}{cccc}\begin{array}{c}\hfill g_{y,i}·g_{z,i}\end{array}& g_{y,i}·gz{x}_i& g_{y,i}·gz{y}_i& g_{y,i}·gz{z}_i\\ & gy{x}_i·gz{x}_i& gy{x}_i·gz{y}_i& gy{x}_i·gz{z}_i\\ & & gy{y}_i·gz{y}_i& gy{y}_i·gz{z}_i\\ & & & gy{z}_i·gz{z}_i\end{array}\right),{\mathbf{G}}_{33,i}=\left(\begin{array}{cccc}\begin{array}{c}\hfill g_{z,i}^2\end{array}& g_{z,i}·gz{x}_i& g_{z,i}·gz{y}_i& g_{z,i}·gz{z}_i\\ & gz{x}_i^2& gz{x}_i·gz{y}_i& gz{x}_i·gz{z}_i\\ & & gz{y}_i^2& gz{y}_i·gz{z}_i\\ & & & gz{z}_i^2\end{array}\right), \\ {\mathbf{G}}_{14,i}=\left(\begin{array}{cc}\begin{array}{c}\hfill g_{x,i}\end{array}& g_i·g_{x,i}\\ gx{x}_i& g_i·gx{x}_i\\ gx{y}_i& g_i·gx{y}_i\\ gx{z}_i& g_i·gx{z}_i\end{array}\right),{\mathbf{G}}_{24,i}=\left(\begin{array}{cc}\begin{array}{c}\hfill g_{y,i}\end{array}& g_i·g_{y,i}\\ gy{x}_i& g_i·gy{x}_i\\ gy{y}_i& g_i·gy{y}_i\\ gy{z}_i& g_i·gy{z}_i\end{array}\right),{\mathbf{G}}_{34,i}=\left(\begin{array}{cc}\begin{array}{c}\hfill g_{z,i}\end{array}& g_i·g_{z,i}\\ gz{x}_i& g_i·gz{x}_i\\ gz{y}_i& g_i·gz{y}_i\\ gz{z}_i& g_i·gz{z}_i\end{array}\right),{\mathbf{G}}_{44,i}=\left(\begin{array}{cc}\hfill 1& g_i\\ & g_i^2\end{array}\right) \end{gathered}$$

(30)

$${\mathbf{A}}^T·\mathbf{l}=\left(\begin{array}{c}\hfill r_1·\sum _{i=1}^n{f}_i·g_{x,i}-r_0·r_1·\sum _{i=1}^n{g}_{x,i}-r_1^2·\sum _{i=1}^n{g}_i·g_{x,i}\\ \hfill r_1·\sum _{i=1}^n{f}_i·gx{x}_i-r_0·r_1·\sum _{i=1}^{n}gx{x}_i-r_1^2·\sum _{i=1}^n{g}_i·gx{x}_i\\ \hfill r_1·\sum _{i=1}^n{f}_i·gx{y}_i-r_0·r_1·\sum _{i=1}^{n}gx{y}_i-r_1^2·\sum _{i=1}^n{g}_i·gx{y}_i\\ \hfill r_1·\sum _{i=1}^n{f}_i·gx{z}_i-r_0·r_1·\sum _{i=1}^{n}gx{z}_i-r_1^2·\sum _{i=1}^n{g}_i·gx{z}_i\\ \hfill r_1·\sum _{i=1}^n{f}_i·g_{y,i}-r_0·r_1·\sum _{i=1}^n{g}_{y,i}-r_1^2·\sum _{i=1}^n{g}_i·g_{y,i}\\ \hfill r_1·\sum _{i=1}^n{f}_i·gy{x}_i-r_0·r_1·\sum _{i=1}^{n}gy{x}_i-r_1^2·\sum _{i=1}^n{g}_i·gy{x}_i\\ \hfill r_1·\sum _{i=1}^n{f}_i·gy{y}_i-r_0·r_1·\sum _{i=1}^{n}gy{y}_i-r_1^2·\sum _{i=1}^n{g}_i·gy{y}_i\\ \hfill r_1·\sum _{i=1}^n{f}_i·gy{z}_i-r_0·r_1·\sum _{i=1}^{n}gy{z}_i-r_1^2·\sum _{i=1}^n{g}_i·gy{z}_i\\ \hfill r_1·\sum _{i=1}^n{f}_i·g_{z,i}-r_0·r_1·\sum _{i=1}^n{g}_{z,i}-r_1^2·\sum _{i=1}^n{g}_i·g_{z,i}\\ \hfill r_1·\sum _{i=1}^n{f}_i·gz{x}_i-r_0·r_1·\sum _{i=1}^{n}gz{x}_i-r_1^2·\sum _{i=1}^n{g}_i·gz{x}_i\\ \hfill r_1·\sum _{i=1}^n{f}_i·gz{y}_i-r_0·r_1·\sum _{i=1}^{n}gz{y}_i-r_1^2·\sum _{i=1}^n{g}_i·gz{y}_i\\ \hfill r_1·\sum _{i=1}^n{f}_i·gz{z}_i-r_0·r_1·\sum _{i=1}^{n}gz{z}_i-r_1^2·\sum _{i=1}^n{g}_i·gz{z}_i\\ \sum _{i=1}^n{f}_i-r_0·n-r_1·\sum _{i=1}^n{g}_i\\ \sum _{i=1}^n{f}_i·g_i-r_0·\sum _{i=1}^n{g}_i-r_1·\sum _{i=1}^n{g}_i^2\end{array}\right)$$

(31)

2.4. Computational Effort

The computational costs are compared between the direct computation and the standard way that includes building up $\mathbf{A}$ and $\mathbf{l}$ as well as the matrix multiplications (${\mathbf{A}}^T·\mathbf{A}$ and ${\mathbf{A}}^T·\mathbf{l}$). The detailed determination of the number of multiplication steps and addition steps is shown in the following.

2.4.1. Computational Costs of the Direct Method

A total of 100 terms have to be computed to form the normal equations directly, see Equation (32). This results from 60 terms of the matrices ${\mathbf{G}}_{11,i}$, ${\mathbf{G}}_{12,i}$, ${\mathbf{G}}_{13,i}$, ${\mathbf{G}}_{22,i}$, ${\mathbf{G}}_{23,i}$, ${\mathbf{G}}_{33,i}$ and 26 terms of the matrices ${\mathbf{G}}_{14,i}$, ${\mathbf{G}}_{24,i}$, ${\mathbf{G}}_{34,i}$, ${\mathbf{G}}_{44,i}$ as well as 14 further terms of ${\mathbf{A}}^T·\mathbf{l}$. 26 terms (sums) of ${\mathbf{A}}^T·\mathbf{l}$ are already computed in ${\mathbf{G}}_{14,i}$, ${\mathbf{G}}_{24,i}$, ${\mathbf{G}}_{34,i}$ and ${\mathbf{G}}_{44,i}$.

$$\begin{array}{c}\begin{array}{c}\hfill {\sum }_{i=1}^n{g}_{x,i}^2,{\sum }_{i=1}^n{g}_{x,i}·gx{x}_i,{\sum }_{i=1}^n{g}_{x,i}·gx{y}_i,{\sum }_{i=1}^n{g}_{x,i}·gx{z}_i,{\sum }_{i=1}^{n}gx{x}_i^2,{\sum }_{i=1}^{n}gx{x}_i·gx{y}_i,{\sum }_{i=1}^{n}gx{x}_i·gx{z}_i,{\sum }_{i=1}^{n}gx{y}_i^2,{\sum }_{i=1}^{n}gx{y}_i·gx{z}_i,{\sum }_{i=1}^{n}gx{z}_i^2,\end{array}\hfill \\ \begin{array}{c}\hfill {\sum }_{i=1}^n{g}_{x,i}·g_{y,i},{\sum }_{i=1}^n{g}_{x,i}·gy{x}_i,{\sum }_{i=1}^n{g}_{x,i}·gy{y}_i,{\sum }_{i=1}^n{g}_{x,i}·gy{z}_i,{\sum }_{i=1}^{n}gx{x}_i·gy{x}_i,{\sum }_{i=1}^{n}gx{x}_i·gy{y}_i,{\sum }_{i=1}^{n}gx{x}_i·gy{z}_i,{\sum }_{i=1}^{n}gx{y}_i·gy{y}_i,{\sum }_{i=1}^{n}gx{y}_i·gy{z}_i,{\sum }_{i=1}^{n}gx{z}_i·gy{z}_i,\end{array}\hfill \\ \begin{array}{c}\hfill {\sum }_{i=1}^n{g}_{x,i}·g_{z,i},{\sum }_{i=1}^n{g}_{x,i}·gz{x}_i,{\sum }_{i=1}^n{g}_{x,i}·gz{y}_i,{\sum }_{i=1}^n{g}_{x,i}·gz{z}_i,{\sum }_{i=1}^{n}gx{x}_i·gz{x}_i,{\sum }_{i=1}^{n}gx{x}_i·gz{y}_i,{\sum }_{i=1}^{n}gx{x}_i·gz{z}_i,{\sum }_{i=1}^{n}gx{y}_i·gz{y}_i,{\sum }_{i=1}^{n}gx{y}_i·gz{z}_i,{\sum }_{i=1}^{n}gx{z}_i·gz{z}_i,\end{array}\hfill \\ \begin{array}{c}\hfill {\sum }_{i=1}^n{g}_{x,i},{\sum }_{i=1}^n{g}_i·g_{x,i},{\sum }_{i=1}^{n}gx{x}_i,{\sum }_{i=1}^n{g}_i·gx{x}_i,{\sum }_{i=1}^{n}gx{y}_i,{\sum }_{i=1}^n{g}_i·gx{y}_i,{\sum }_{i=1}^{n}gx{z}_i,{\sum }_{i=1}^n{g}_i·gx{z}_i,\end{array}\hfill \\ \begin{array}{c}\hfill {\sum }_{i=1}^n{g}_{y,i}^2,{\sum }_{i=1}^n{g}_{y,i}·gy{x}_i,{\sum }_{i=1}^n{g}_{y,i}·gy{y}_i,{\sum }_{i=1}^n{g}_{y,i}·gy{z}_i,{\sum }_{i=1}^{n}gy{x}_i^2,{\sum }_{i=1}^{n}gy{x}_i·gy{y}_i,{\sum }_{i=1}^{n}gy{x}_i·gy{z}_i,{\sum }_{i=1}^{n}gy{y}_i^2,{\sum }_{i=1}^{n}gy{y}_i·gy{z}_i,{\sum }_{i=1}^{n}gy{z}_i^2,\end{array}\hfill \\ \begin{array}{c}\hfill {\sum }_{i=1}^n{g}_{y,i}·g_{z,i},{\sum }_{i=1}^n{g}_{y,i}·gz{x}_i,{\sum }_{i=1}^n{g}_{y,i}·gz{y}_i,{\sum }_{i=1}^n{g}_{y,i}·gz{z}_i,{\sum }_{i=1}^{n}gy{x}_i·gz{x}_i,{\sum }_{i=1}^{n}gy{x}_i·gz{y}_i,{\sum }_{i=1}^{n}gy{x}_i·gz{z}_i,{\sum }_{i=1}^{n}gy{y}_i·gz{y}_i,{\sum }_{i=1}^{n}gy{y}_i·gz{z}_i,{\sum }_{i=1}^{n}gy{z}_i·gz{z}_i,\end{array}\hfill \\ \begin{array}{c}\hfill {\sum }_{i=1}^n{g}_{y,i},{\sum }_{i=1}^n{g}_i·g_{y,i},{\sum }_{i=1}^{n}gy{x}_i,{\sum }_{i=1}^n{g}_i·gy{x}_i,{\sum }_{i=1}^{n}gy{y}_i,{\sum }_{i=1}^n{g}_i·gy{y}_i,{\sum }_{i=1}^{n}gy{z}_i,{\sum }_{i=1}^n{g}_i·gy{z}_i,\end{array}\hfill \\ \begin{array}{c}\hfill {\sum }_{i=1}^n{g}_{z,i}^2,{\sum }_{i=1}^n{g}_{z,i}·gz{x}_i,{\sum }_{i=1}^n{g}_{z,i}·gz{y}_i,{\sum }_{i=1}^n{g}_{z,i}·gz{z}_i,{\sum }_{i=1}^{n}gz{x}_i^2,{\sum }_{i=1}^{n}gz{x}_i·gz{y}_i,{\sum }_{i=1}^{n}gz{x}_i·gz{z}_i,{\sum }_{i=1}^{n}gz{y}_i^2,{\sum }_{i=1}^{n}gz{y}_i·gz{z}_i,{\sum }_{i=1}^{n}gz{z}_i^2,\end{array}\hfill \\ \begin{array}{c}\hfill {\sum }_{i=1}^n{g}_{z,i},{\sum }_{i=1}^n{g}_i·g_{z,i},{\sum }_{i=1}^{n}gz{x}_i,{\sum }_{i=1}^n{g}_i·gz{x}_i,{\sum }_{i=1}^{n}gz{y}_i,{\sum }_{i=1}^n{g}_i·gz{y}_i,{\sum }_{i=1}^{n}gz{z}_i,{\sum }_{i=1}^n{g}_i·gz{z}_i,\end{array}\hfill \\ \begin{array}{c}\hfill {\sum }_{i=1}^n{g}_i,{\sum }_{i=1}^n{g}_i^2,{\sum }_{i=1}^n{f}_i·g_{x,i},{\sum }_{i=1}^n{f}_i·gx{x}_i,{\sum }_{i=1}^n{f}_i·gx{y}_i,{\sum }_{i=1}^n{f}_i·gx{z}_i,{\sum }_{i=1}^n{f}_i·g_{y,i},{\sum }_{i=1}^n{f}_i·gy{x}_i,{\sum }_{i=1}^n{f}_i·gy{y}_i,{\sum }_{i=1}^n{f}_i·gy{z}_i,\end{array}\hfill \\ \begin{array}{c}\hfill {\sum }_{i=1}^n{f}_i·g_{z,i},{\sum }_{i=1}^n{f}_i·gz{x}_i,{\sum }_{i=1}^n{f}_i·gz{y}_i,{\sum }_{i=1}^n{f}_i·gz{z}_i,{\sum }_{i=1}^n{f}_i,{\sum }_{i=1}^n{f}_i·g_i\end{array}\hfill \end{array}$$

(32)

To evaluate the computational effort, the number of additions and multiplications are counted:

• Multiplications per voxel:

  −

  Nine multiplications in terms of Equation (29).

  −

  86 multiplications in the 100 terms (Equation (32), inside the sums).

• Additions per voxel:

  −

  100 additions in the 100 sums (Equation (32)).

• Further multiplications:

  −

  Computation of the product $r_1·r_1$.

  −

  84 multiplications of the prefactors with the sums in Equation (26) for the normal matrix.

  −

  40 multiplications for the right hand side (Equation (31)) outside of the sums.

• Further additions:

  −

  28 additions for the right hand side (Equation (31)).

Altogether, for the direct computation of ${\mathbf{A}}^T·\mathbf{A}$ and ${\mathbf{A}}^T·\mathbf{l}$, 95 multiplications and 100 additions per voxel are required. The additional multiplication and summations for building up ${\mathbf{A}}^T·\mathbf{A}$ and ${\mathbf{A}}^T·\mathbf{l}$ can be neglected due to the large number of voxel.

2.4.2. Computational Costs of the Standard Method

The computational effort of the standard method contains building up the Jacobian matrix $\mathbf{A}$ and the reduced observation vector $\mathbf{l}$ as well as the matrix-matrix multiplication and the matrix-vector multiplication.

The following terms are calculated and saved in temporal variables for each line of the matrix $\mathbf{A}$ (for each voxel):

$$\begin{array}{c}\hfill r1g{x}_i=r_1·g_{x,i}\\ \hfill r1g{y}_i=r_1·g_{y,i}\\ \hfill r1g{z}_i=r_1·g_{z,i}\end{array}$$

(33)

For the each line of matrix $\mathbf{A}$, the following nine terms have to be computed:

$$\begin{array}{c}r1g{x}_i·{\tilde{x}}_i;r1g{x}_i·{\tilde{y}}_i;r1g{x}_i·{\tilde{z}}_i;r1g{y}_i·{\tilde{x}}_i;r1g{y}_i·{\tilde{y}}_i;r1g{y}_i·{\tilde{z}}_i;r1g{z}_i·{\tilde{x}}_i;r1g{z}_i·{\tilde{y}}_i;r1g{z}_i·{\tilde{z}}_i\end{array}$$

(34)

For each element of $\mathbf{l}$, there are one multiplication and two additions (subtractions):

$$f_i-r_0-r_1·g_i$$

(35)

• Multiplications per voxel:

  −

  One multiplication for building up the reduced observation vector $\mathbf{l}$.

  −

  Three multiplications for the terms of Equation (33) that are computed temporally due to multiple use for building up the Jacobian matrix.

  −

  Nine further multiplications for building up the Jacobian matrix of Equation (34).

  −

  105 multiplications (105 elements) due to the matrix multiplication of ${\mathbf{A}}^T·\mathbf{A}$ (only upper triangular matrix).

  −

  14 multiplications (14 elements) due to the matrix-vector multiplication of ${\mathbf{A}}^T·\mathbf{l}$.

• Additions per voxel:

  −

  Two additions for building up the reduced observation vector $\mathbf{l}$.

  −

  105 additions (105 elements) due to the matrix multiplication of ${\mathbf{A}}^T·\mathbf{A}$ (only upper triangular matrix).

  −

  14 additions (14 elements) due to the matrix-vector multiplication of ${\mathbf{A}}^T·\mathbf{l}$.

Altogether, for building up $\mathbf{A}$ and $\mathbf{l}$ and for the matrix multiplications ${\mathbf{A}}^T·\mathbf{A}$ and ${\mathbf{A}}^T·\mathbf{l}$, 132 multiplications and 121 additions per voxel are required.

2.4.3. Theoretical Comparison

Table 1. Comparison of the computational effort between the direct calculation of the normal equations and the standard method.

Method	Multiplications per vx	Additions per vx
Standard	132	121
Direct	95	100

summarizes the consideration and shows the number of multiplications and additions per voxel for the standard method and the direct computation of the normal equations. The direct method reduces the number of multiplications by 28% and reduces the number of additions by 17%. Appendix B and Appendix C also consider the cases that the radiometric parameters are fixed and that these parameters are omitted. In addition to the reduced number of multiplications and summations, the direct method also save memory because the computation of the $n\times 14$ Jacobian matrix ($\mathbf{A}$) and $n\times 1$ observation vector ($\mathbf{l}$) is avoided.

2.4.4. Comparison of Time Measurements in Practise

The comparison was done on a desktop machine with a AMD Ryzen 7 3800X 8-Cores (16 threads) processor at 3.9 GHz. In the test, 3D LSM was applied to 146,933 cuboids defined in the first two data sets of an in-situ tension test of Lorenzoni et al. [18]. The patch dimension was set to $15\times 15\times 15$ vx. The algorithm is implemented in C++ (using the GNU Compiler Collection, https://gcc.gnu.org) and compiled with the optimization option -O3. The loop over the matching points with the computation of the 3D LSM is parallelized with the OpenMP library (16 parallel units), and the time measurement only considers this loop. The time measurement includes the whole 3D LSM process so that not only the calculation of the normal matrix ${\mathbf{A}}^T·\mathbf{A}$ and the right hand side vector ${\mathbf{A}}^T·\mathbf{l}$ is analyzed. However, the computation of the normal equations is one of the most time-consuming steps herein.

Different calculation modes are tested: Method M1 stands for the direct computation. M2 represents the standard computation with building up the Jacobian matrix $\mathbf{A}$ and the reduced observation vector $\mathbf{l}$ as well as the matrix multiplications (only upper triangular matrix of the normal matrix) with an own implementation. M3 and M4 are similar to M2, but the matrix multiplications are done using the Eigen library. Eigen is a C++ template library for linear algebra (release 3.4.0, https://eigen.tuxfamily.org/). For M3, the full matrix of ${\mathbf{A}}^T·\mathbf{A}$ is computed, and for M4, only the upper triangular matrix is computed. For all modes, before the calculation of the normal matrix begins, the values of the interpolated gray values and derivatives are computed and saved in separate arrays. All modes lead to the same numerical results, only the calculation times differ.

Table 2. Comparison of the computation times with different modes for the calculation of the normal equations.

Mode 1	M1	M2	M3	M4
Computation time [s]	12.8	23.2	19.3	17.5

shows the calculation times for the different modes. The direct method is the fastest variant and almost twice as fast as the own standard computation M2. Compared to the best matrix multiplication variant M4 using Eigen (only upper triangular matrix), the direct method reduces the computation time by 27%. This reduction approximately agrees with the theoretical decrease of the multiplications of 28% (Section 2.4.3). The computation time difference between M3 and M4 is very small (only 9% decrease), although significantly more matrix elements (91) have to be computed (M3: 132 multiplications and 121 addition per voxel, M4: 223 multiplications and 212 addition per voxel).

3. Conclusions

The paper presents an optimized algorithm to compute 3D least-squares matching in voxel data sequences including geometric and radiometric parameters. Compared to the standard method of the estimation of the normal equations, the number of multiplications is reduced by 28% and the number of additions is reduced by 17%. In a practical test, the computation time was decreased by 27%.