Image Reconstruction and Evaluation: Applications on Micro-Surfaces and Lenna Image Representation

Abstract

This article develops algorithms for the characterization and the visualization of micro-scale features using a small number of sample points, with the goal of mitigating the measurement shortcomings, which are often destructive or time consuming. The popular measurement techniques that are used in imaging of micro-surfaces include the 3D stylus or interferometric profilometry and Scanning Electron Microscopy (SEM), where both could represent the micro-surface characteristics in terms of 3D dimensional topology and greyscale image, respectively. Such images could be highly dense; therefore, traditional image processing techniques might be computationally expensive. We implement the algorithms in several case studies to rapidly examine the microscopic features of micro-surface of Microelectromechanical System (MEMS), and then we validate the results using a popular greyscale image; i.e., “Lenna” image. The contributions of this research include: First, development of local and global algorithm based on modified Thin Plate Spline (TPS) model to reconstruct high resolution images of the micro-surface’s topography, and its derivatives using low resolution images. Second, development of a bending energy algorithm from our modified TPS model for filtering out image defects. Finally, development of a computationally efficient technique, referred to as Windowing, which combines TPS and Linear Sequential Estimation (LSE) methods, to enhance the visualization of images. The Windowing technique allows rapid image reconstruction based on the reduction of inverse problem.

1. Introduction

The measurement of Micro–Nano features requires specialized imaging tools such as optical microscopes, thermal cameras, Scanning Electron Microscopy (SEM), and contact and non-contact position sensors. SEM can image conductive and non-conductive surfaces at a range of magnification from Millimeter to Nano scale; however, imaging a large micro-surface at high resolution is time-consuming and often destructive or impractical. For example, it is possible to capture a full view of a 3D microelectromechanical (MEMS) structure, such as that in Figure 1a, using SEM at 130X magnification, but the image quality at the interface, including the electrical and mechanical interconnections between the two structures, is poor. Thus, a 597X magnification of a smaller area was captured to reveal the micron features details, as shown in Figure 1b. The problem becomes one of stitching multiple images into a single image. In addition, the one drawback of SEM images is they do not provide depth information [1]. Toward these challenges, one of our objectives is to develop automatic and piece-wise algorithms that can reconstruct and enhance image visualization at minimum cost function. The reconstruction of images usually involves the interpolation of a large number of intermediate pixels from a smaller number of control points, which represents a digital elevation model (DEM) with the elevation being a quantity that represents the attribute interest. Typically, control points are sample locations of a captured or sampled pixel quantity at known (x,y) coordinates. Such quantities might be the pixel intensity matrix of an image or the sampled elevations of a micro-surface. DEMs carry vital information about the feature topography, and can be used to characterize or visualize images using reconstruction, feature extraction, and filtering enhancement techniques [2,3,4]. There is no single technique that is suitable for all types of image enhancement. For example, low pass and median filtering techniques alone are not appropriate for high quality reconstruction of images because they deteriorate its high frequency components [5]. Therefore, researchers developed a wide range of techniques that target the image’s components such as Markov random field (MRF), wavelet decomposition, Gaussian filter, multi-quadratic bi-harmonic method, Shepard’s method, C⁻¹-Function interpolation method, kernel-based modeling method, and Thin Plate Spline (TPS) method [6,7,8]. In the kernel-based method, a penalized likelihood of the projection data is used to reconstruct images from the estimated coefficient model. However, the method is challenging because the inverse problem is ill-posed and the resultant image is usually very noisy [8]. TPS algorithm is used in the representation of 3D object from 2D images. TPS requires no assumption to be made with respect to the distribution or the location of the DEM. This method has become popular in the medical imaging because the reduction of the registration from global into local allows the analysis of local changes of image [9,10]. While many researches showed the applications of TPS in the visualization of macroscopic objects, little information is known about its ability to characterize or visualize micro and nano structures. Towards this goal, we aim to expand upon the application of TPS algorithm for imagining of MEMS structures.

In Section 2, we derive mathematical models based on a modified TPS algorithm for the synthesis of the topological features of micro-surfaces. The interpolation algorithm is used to reconstruct images from few scattered data points. Then, we derive characterization algorithms to describe and extract image features in terms of topological slopes, curvatures, and bending components. We develop a Linear Sequential Estimation (LSE) based on TPS algorithm for a rapid reconstruction of images. Furthermore, we describe enhancement techniques to improve the image visualization, including image stitching, defect removal and resolution improvement. In Section 3, we conduct experiments to benchmark the performance of the proposed techniques. Finally, in Section 4, we demonstrate the algorithms using two illustrative examples: micro-surface of a MEMS based on a cloud of sampled points, and Lenna image.

2. Materials and Methods

2.1. Modified TPS Algorithm

The TPS is a non-stochastic algorithm that was first developed to minimize the bending energy of thin-plate structures. Such characteristic could be advantageous in smoothing the deformations due to large image pixilation or small micro-surface roughness [11,12]. This is because TPS has minimal bending energy properties over the entire object while it maintains the correlation with the DEMs [13]. The TPS algorithm uses radial basis function (RBFs), such as logarithmic RBFs and multi-quadratic function, which emphasizes on the local characteristics of the object, i.e., the reconstruction is largely unaffected at locations far from the RBFs. There is no unique procedure to define the RBFs; therefore, several basis functions have been developed and used in attempt to reconstruct objects from the measured DEMs [6,10,14].

In this research, we want to obtain a continuous, smooth, and differentiable model of an image or micro-surface that passes through its DEM. DEMs can be constructed with different patterns of data points as shown in Figure 2, depending on the target micro-surface, collection procedures and techniques employed. Toward this goal, we utilize the bi-harmonic partial differential equation of a thin plate under point loading [4,14,15]. We write

$$z_k\left(x,y\right)=a_o+a_1{x}_k+a_2{y}_k+{\displaystyle \sum }_{i=1}^N{F}_i\mathsf{\phi }\left(r_{ik}\right)+C_k{F}_k$$

(1)

$${{\displaystyle \sum }}_{i=1}^N{F}_i=0,{{\displaystyle \sum }}_{i=1}^N{x}_i{F}_i=0,\text{ and }{{\displaystyle \sum }}_{i=1}^N{y}_i{F}_i=0$$

(2)

where Equations (1) and (2) correspond to the logarithmic RBFs of $\mathsf{\phi }\left(r_{ik}\right)=r_{ik}^2\ln \left(r_{ik}^2\right)$ and the boundary conditions, respectively. The radial distance $r_{ik}$ is $\sqrt{{\left(x_i-x_k\right)}^2+{\left(y_i-y_k\right)}^2}$. $z_k$ is the height or the pixel intensity level which is measured at ${\left(x,y\right)}_k$ coordinate. as and Fs are the coefficients of a nominal plane and the weighting factors of micro-surface, respectively. $C_k=16\mathsf{\pi }D/K_k$ is the elastic constant at the interpolated point. $K_k$ and D are the spring and rigidity constants, respectively.

The interpolation function $z_k\left(x,y\right)$ is divided into two parts [12]: (1) a linear part $a_o+a_1{x}_k+a_2{y}_k$, and (2) the sum of the independent principal warp ${{\displaystyle \sum }}_{i=1}^N{F}_i{r}_{ik}^2\ln \left(r_{ik}^2\right)+C_k{F}_k$ ${{\displaystyle \sum }}_{i=1}^N{F}_i{r}_{ik}^2\ln \left(r_{ik}^2\right)+C_k.$ In this form, the logarithmic function is not defined when $r_{ij}$ value is zero. Therefore, we add a small dimensionless quantity ε relative to $r_{ik}^2$ to guarantee that Equation (1) is continuously differentiable function. This yields a modified TPS algorithm.

$$z_k\left(x,y\right)=a_o+a_1{x}_k+a_2{y}_k+{\displaystyle \sum }_{i=1}^N{F}_i{r}_{ik}^2\ln \left(r_{ik}^2+\mathsf{\epsilon }\right)+C_k{F}_k.$$

(3)

The above equation suggests that interpolated area could be either sharpened when ε is large or smoothen if it is small. The as’ and Fs’ coefficients in the boundary Equation (2) are unknowns, and they can be estimated by using DEM. We describe DEM by any arbitrary N control points whose Cartesian locations and magnitude values are known. We can write an N system of linear equations by substituting the DEM in Equation (3). The boundary conditions provide an additional three linear equations. The system of linear equations can be written in matrix form

$$\tilde{Z}=HB,$$

(4)

where we define the TPS coefficient vector X by

$$B={\left[F_1,F_2,\dots ,F_N,a_o,a_1,a_2\right]}^T,$$

(5)

and the measurement vector $\tilde{Z}$

$$\tilde{Z}={\left[\widetilde{z_1},\widetilde{z_2},\dots ,\widetilde{z_N},0,0,0\right]}^T.$$

(6)

The measurement model matrix H becomes

$$H=\left[\begin{array}{cccccccc}{C}_1& r_{12}^2\ln (r_{12}^2+\mathsf{\epsilon })& \cdots & \ln (r_{1,N-1}^2+\mathsf{\epsilon })& \ln (r_{1,N}^2+\mathsf{\epsilon })& l& x_1& y_1\\ r_{12}^2\ln (r_{12}^2+\mathsf{\epsilon })& C_2& \cdots & \ln (r_{1,N-1}^2+\mathsf{\epsilon })& \ln (r_{2,N}^2+\mathsf{\epsilon })& l& x_2& y_2\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ \ln (r_{N-1,1}^2+\mathsf{\epsilon })& \ln (r_{N-1,2}^2+\mathsf{\epsilon })& \cdots & C_{N-1}& \ln (r_{N-1,N}^2+\mathsf{\epsilon })& l& x_{N-1}& y_{N-1}\\ \ln (r_{N,1}^2+\mathsf{\epsilon })& \ln (r_{N,2}^2+\mathsf{\epsilon })& \cdots & \ln (r_{N,N-1}^2+\mathsf{\epsilon })& C_N& 1& x_N& y_N\\ 1& 1& \cdots & 1& 1& 0& 0& 0\\ x_1& x_2& \cdots & x_{N-1}& x_N& 0& 0& 0\\ y_1& y_2& \cdots & y_{N-1}& y_N& 0& 0& 0\end{array}\right]$$

(7)

where H is $\left(N+3\right)\times \left(N+3\right)$. The distance between any two points can be measured relative to each other in any order. Therefore, H is symmetric matrix where $r_{ik}=r_{ki}$. The inverse of H always exists except at singularity, which is either caused by dependent measurements, i.e., replication, or if the H matrix is constructed from three collinear samples. Therefore, if H is a positive definite and symmetric matrix, then we can estimate the coefficient vector $B$ from

$$B=H^{-1}\tilde{Z}.$$

(8)

The estimation algorithmic Equation (3) becomes a continuous and differentiable model whose coefficients are found from the information that is measured for the micro-surface.

2.2. Characterization of MEMS Micro-Surfaces by Using Modified TPS Algorithm

There are two popular types of bi-dimensional (2D to 2D ½) representations of microscale surfaces: SEM, and micro-surface profilometery. SEM produces an image of a sample by scanning it with a focused beam of electrons that contains information about the sample’s micro-surface topology. The produced image contains information about the planner dimension of the micro-surface, $\left(x,y\right)$, but it does not directly provide depth information. The micro-surface profilometery approach uses contact or non-contact micron measurement sensing instrumentation, which is coordinated in a multi-axis micro-resolution motion stage. The produced image is typically a 2D ½ micro-surface constructed form a cloud of points of $\left(x,y,z\right)$. For each of them, the same image-based analysis can be performed [16]. MEMS micro-surfaces exhibit topographical components that range from Nano, to Micro, to Meso scale size. The non-stationary and the multi-scale nature of a MEMS micro-surface could limit its description, which is typically provided by majority of the topographical characterization method employed [5,15,17]. Such micro-surface characterization algorithms tend to provide parameterized functions that are strongly scale dependent; however, one scale parameter cannot provide sufficient information about the micro-surface in all scales. This is because a scale parameter could conflict with multi-scale parameters as in the Permengham parameters that are used to describe 2D ½ micro-surfaces. These parameters were designed to characterize particular morphological features such as shape, elongation, shape complexity, and micro-surface directionality [18]. To overcome such shortcoming, we define the micro-surface characteristics of MEMS micro-surfaces by using the topological definitions that are commonly used in the description of geological landscapes; particularly, we obtain models from the derivatives of Equation (3) to characterize MEMS micro-surfaces. For example, the incremental change in micro-surface height as a point moves in lateral or horizontal directions is caused by a topography that contains peaks–hills–and valleys–dips. The angle switching between hills and valleys corresponds to changes in the continuity of micro-surface slopes. This corresponds to finding the slopes of a micro-surface in the x and y directions at the interpolated points; therefore, the slopes, $S_x$ and $S_y$, are the first partial derivative of the reconstructed micro-surface in x and y at sample k, respectively.

$${S_x|}_k=\frac{\mathsf{\delta }{z}_k\left(x,y\right)}{\mathsf{\delta }x}=a_1+2{\displaystyle \sum }_{i=1}^N{F}_i\left(\frac{r_{ik}^2}{r_{ik}^2+\mathsf{\epsilon }}+\ln \left(r_{ik}^2+\mathsf{\epsilon }\right)\right)\left(x_k-x_i\right)$$

(9)

$${S_y|}_k=\frac{\mathsf{\delta }{z}_k\left(x,y\right)}{\mathsf{\delta }y}=a_2+2{{\displaystyle \sum }}_{i=1}^N{F}_i\left(\frac{r_{ik}^2}{r_{ik}^2+\mathsf{\epsilon }}+\ln \left(r_{ik}^2+\mathsf{\epsilon }\right)\right)\left(y_k-y_i\right).$$

(10)

The second partial derivative of the reconstructed micro-surface represents the slope variation of the micro-surface, or in the geological terms: the dip variation of the geological unit [6]. The micro-surface curvatures, $S_{xx}$ and $S_{yy}$, in x, y and xy or yx are

$${S_{xx}|}_k=2{{\displaystyle \sum }}_{i=1}^N{F}_i\left(\frac{r_{ik}^2}{r_{ik}^2+\epsilon }+\ln \left(r_{ik}^2+\mathsf{\epsilon }\right)+2{\left(x_k-x_i\right)}^2\left(\frac{r_{ik}^2+2\mathsf{\epsilon }}{{\left(r_{ik}^2+\mathsf{\epsilon }\right)}^2}\right)\right),$$

(11)

$${S_{yy}|}_k=2{{\displaystyle \sum }}_{i=1}^N{F}_i\left(\frac{r_{ik}^2}{r_{ik}^2+\epsilon }+\ln \left(r_{ik}^2+\mathsf{\epsilon }\right)+2{\left(y_k-y_i\right)}^2\left(\frac{r_{ik}^2+2\mathsf{\epsilon }}{{\left(r_{ik}^2+\mathsf{\epsilon }\right)}^2}\right)\right),\text{ and}$$

(12)

$${S_{yx}|}_k={S_{xy}|}_k=4{{\displaystyle \sum }}_{i=1}^N{F}_i\left(\frac{r_{ik}^2+2\mathsf{\epsilon }}{{\left(r_{ik}^2+\mathsf{\epsilon }\right)}^2}\right)\left(y_k-y_i\right)\left(x_k-x_i\right).$$

(13)

We should point out that the second partial derivatives of the reconstructed micro-surface at the reconstructed points could be used for finite-element microstructural analysis [12].

The strike of a dipping layer is commonly used to describe the geological landscapes, and it is defined by the intersection of that layer with an imaginary horizontal plane; forming a line of equal elevations [7]. The strike is calculated by setting the x and y derivatives of Equation (3) to zero. This gives

$${\frac{dy}{dx}|}_k=\frac{a_1+2{{\displaystyle \sum }}_{i=1}^N{F}_i\left(\frac{r_{ik}^2}{r_{ik}^2+\mathsf{\epsilon }}+\ln \left(r_{ik}^2+\mathsf{\epsilon }\right)\right)\left(x_k-x_i\right)}{a_2+2{{\displaystyle \sum }}_{i=1}^N{F}_i\left(\frac{r_{ik}^2}{r_{ik}^2+\mathsf{\epsilon }}+\ln \left(r_{ik}^2+\mathsf{\epsilon }\right)\right)\left(y_k-y_i\right)}.$$

(14)

This slope is the tangent of the angle between the positive x-axis and the strike line. The strike angle, δ, is the angle measured clockwise from the positive y-axis to the strike line, and it is a measure of the topographical contour slopes $dy/dx$. The strike angle, estimated in degree, becomes

$${\mathsf{\delta }}_k={90}^{°}-\arctan \left({\frac{dy}{dx}|}_k\right).$$

(15)

A vector, which is normal to the micro-surface at the reconstructed point, is defined by the azimuth, α, and the inclination—or dip angle θ in degrees. The direction of the true dip angle is always perpendicular to the strike; therefore, the azimuth measures the normality of a topographical micro-surface, and it is measured clockwise from the positive y-axis—or the north—such that the geological strike is equal to α + 90°. The azimuth is given by

$${\mathsf{\alpha }}_k={270}^{°}-\arctan \left({\frac{Sy}{Sx}|}_k\right).$$

(16)

By definition, dips at the reconstructed points depict positive inclination that is orthogonal to the strike line; therefore, the inclination or dip angle θ must be positive and is given by

$${\mathsf{\theta }}_k=\mathrm{artan}\left(\sqrt{{{S_x|}_k}^2+{{S_y|}_k}^2}\right).$$

(17)

The final characteristic that we obtain in this section describes the energy stored due to the deformations of micro-surface. The deformation index of a thin plate is subjected to slight bending, and it is derived for micro-surface based on the entire distributed bending energies, i.e., $I_z=\mathsf{\gamma }{\displaystyle \iint }\left(S_{xx}^2+S_{yy}^2+S_{xy}^2\right)dxdy$ [19]. Applying this definition to Equations (11)–(13), we can define the following bending energy index at reconstructed point

$${I_z|}_k={\left({S_{xx}|}_k\right)}^2+{\left({S_{yy}|}_k\right)}^2+{\left({S_{xy}|}_k\right)}^2.$$

(18)

The constant of proportionality, γ, has unit of J/m² in the SI unit system. In addition, it is used as a qualitative scaling factor. The TPS function, z, in Equation (3) minimizes the nonnegative quantity of I_z; therefore, a planar micro-surface will have zero bending energy at areas where curvatures have high energy. A reconstructed topography of the bending energy is used as a qualitative measure of the geometry of a micro-surface at a location. For example, it views the negative features on a MEMS micro-surface image such as a micro-machined surface [15].

2.3. Image Processing Based on Modified TPS Algorithms

The classical image reconstruction techniques can be categorized into: intensity-based method, contour-based method, and shape-based method [20]. The intensity-based method requires information about the pixel intensity values and their locations within the image. We found previously that 2D ½ micro-surface can be reconstructed from cloud of points whose locations and height values are known in the Cartesian coordinate. It is possible to utilize TPS algorithm in the characterization and reconstruction of intensity-based images by replacing the z-value with intensity value. In this section, we will further develop techniques based on TPS algorithm for several image-processing enhancement, including denoising and rapid reconstruction.

2.3.1. Modified TPS Based Image-Reconstruction Method

The quality of an image tends to have better visualization when: (1) the image is reconstructed at high resolution; and (2) the original pixel intensity values of the image or the control points are inherited in the reconstructed image. Reconstruction of an image often involves the interpolation of regularly spaced intermediate pixels. Therefore, we select modified TPS with logarithmic RBFs algorithm, to interpolate the value of a new pixel within image region. Because the TPS coefficients in Equation (8) is solved by using the control points, the modified TPS algorithm, which is Equation (3), passes through these control points; therefore, the value of an interpolated pixel depends on the selected neighborhood that contains the control points. There are two types of neighborhoods:

• A spatial neighborhood: this technique reconstructs a region within an image by using the information that is available within it, and without the knowledge from other neighboring regions. The method can be applied either globally for the entire image, or locally for a selected region within the image.
• Spatio-to-temporal neighborhood: this technique reconstructs an image by sequentially updating the information from multiple neighborhoods. For example, an image could be reconstructed and warped from several regions.

2.3.2. Modified TPS Based Denoising Method

Image denoising is the process of filtering out impulsive defects, which typically requires algorithms for detection and removal of defects, and an algorithm that reconstruct the removed defects. Roman et al. [21] introduced a TPS method for denoising of old paintings with the additive noise modeled as random variable with zero-mean Gaussian distribution, and with the goal to achieve an acceptable trade-off between the noise removal and smoothing of edges. With the goal to decrease unwanted smoothing effect of TPS denoising, the proposed method by Roman uses a weighted TPS approximation of image data computed sequentially over small overlapping patches, and centered at each pixel. In this research, however, we describe procedures of a modified TPS image denoising that can: first, detect large defects, second recover lost information, and finally obtain a smooth reconstruction of the image. The method procedures is described as follow:

• Construct bending energy image, which is a greyscale intensity image of the defected image, and it is computed by using the bending energy index—Equation (18)—in the global coordinate. The impulsive defects are located in the constructed bending energy image because they are artifacts that tend to have black or white level with strong edges at their spatial boundaries [22]; therefore, a white pixel in the bending energy image corresponds to a black pixel in the original image, and it refers to a local maximum. A black pixel indicates a local minimum, and it corresponds to a white pixel in the original image.
• Obtain the locations of the defects from step (1), and then remove them to create a defect-free DEM; i.e., the original pixel intensity values are free of defect.
• Calculate the modified TPS coefficients from the updated DEM—from step (2)—by using Equation (8). There are two ways to calculate the modified TPS coefficients:

  • Global: select all the available information from the updated DEM.
  • Local: select a neighborhood for each defect.
• Interpolate the intensity value of the removed data by using Equation (3).
• If required, reconstruct high-resolution image by using the method described in Section 2.3.1.

2.3.3. LSE-TPS Based Warping Algorithm

In this section, we develop a LSE-TPS warping algorithm for the reconstruction of an image from multiple images. The goal is to warp multiple images into a single image, such as a torn image, while maintaining their spatial boundaries at the interface smooth. The spatial boundaries could be reconstructed at the same image resolution by using the method in Section 2.3.1, where a “good” warping would be dependent on a single neighborhood.

Our objective becomes to estimate the TPS coefficients from multiple neighborhoods; particularly we want to add weighting factors and directionality among neighborhoods. This way it becomes possible to emphasize on the reconstruction to selective neighborhoods. In this section, we refer neighborhood by a patch. Toward the end of this goal, we develop a LSE algorithm that updates the modified TPS coefficients between successive patches.

Suppose there is a L sequentially arranged patches. The application of LSE requires equal sizes of radial basis; i.e., each patch has n number of control points. Upon merging two successive patches—(k) and (k + 1) —into a single measurement model, we can write

$$\tilde{Z}=HB+\nu ;\text{ where}$$

(19)

$$\tilde{Z}=\left[\frac{{\tilde{Z}}_k}{{\tilde{Z}}_{k+1}}\right],H=\left[\frac{{\tilde{H}}_k}{{\tilde{H}}_{k+1}}\right],\text{ and }=\left[\frac{{\tilde{\nu }}_k}{{\tilde{\nu }}_{k+1}}\right]$$

(20)

This measurement model relates the states vector $\tilde{Z}$ to the measurement matrix H, and the measurement error, $\nu$, which is an additive Gaussian noise. In a static image, the $C_j\text{'}$s are set to zero; therefore, the H matrix becomes symmetric; where the elements in the main diagonal are equal to zero, and the TPS coefficient in B vector are not cross-correlated. The covariance of the measurement error is stationary in a single image, and it could be represented by a random diagonal variance matrix of zero means and known standard deviation. In snapshot imaging, such as SEM, the measurement errors and the process weight matrix, $W$, are neither correlated forward nor backward in time. In addition, we assume that $W$ and $\nu$ are not cross correlated within an image i.e., $E{\left\{{\upsilon }_k{W}_k\right\}}^T=0$.

Our goal is to minimize the propagation of the measurement errors when the estimated coefficients X are updated from one patch of data to another—window to window. We suggest that there exist an optimal coefficient vector X such that the Jacobin cost function, $J={\nu }^{T}W\nu$, is minimum; i.e.,

$$\underset{X}{\min }{\left(Z-HB\right)}^{T}W\left(Z-HB\right)$$

(21)

where the process weight matrix is diagonal, and it is given by

$$W=\left[\begin{array}{cc}{W}_k& 0\\ 0& W_{k+1}\end{array}\right].$$

(22)

$W_k$ and $W_{k+1}$ corresponds to patches ($k$) and $\left(k+1\right)$, respectively. The size of each process matrix is $\left(N+3\right)\times \left(N+3\right)$. The optimization of the Jacobian cost function with respect to $B$ produces minimum error. This is because the Hessian, which is the gradient of H, is positive definite; therefore, the optimization process can be written into the following covariance recursion process form

$$P_k={\left[H_k^T{W}_k{H}_k\right]}^{-1},$$

(23)

$$K_{k+1}=P_k{H}_{k+1}^T{\left[H_{k+1}{P}_k{H}_{k+1}^T+W_{k+1}^T\right]}^{-1},$$

(24)

$$P_{k+1}=\left[I-K_{k+1}{H}_{k+1}\right]P_k,$$

(25)

$${\widehat{B}}_{k+1}={\widehat{B}}_k+K_{k+1}\left({\tilde{Z}}_{k+1}-H_{k+1}{\widehat{B}}_k\right),$$

(26)

where Equations (23) and (25) are the information matrices, Equation (24) is the Kalman estimator gain matrix, and Equation (26) is the Kalman update Equation. These recursion equations estimate the TPS coefficient of patch $\left(k+1\right)$ by using its observation measurement and the modified TPS coefficient of patch $\left(k\right)$; therefore, a patch could be reconstructed based on its current information and the information available from the successive patch. The LSE-TPS method requires a-priori estimate of the coefficients. We can initialize the recursion process at k = 0 by

$${\tilde{B}}_{k=0}=H_{k=0}^{-1}{Z}_{k=0}.$$

(27)

2.3.4. LSE-TPS Based Rapid Reconstruction Method

The solution of the radial basis functions requires matrix inversion, which could be problematic for large numbers of control points [6,7,19]. Equation (3) is dependent on the arithmetic operations that are required to compute the inverse of H; therefore, a large number of control points increases the number of computations. This mainly causes inaccuracy in the estimation of the modified TPS coefficient due to the accumulation of round-off errors. To improve the computational efficiency of the reconstruction and the visualization, we aim to reduce the size of the matrix H. Towards this goal, we develop the following Windowing technique:

Divide the image into L number of patches; each has equal number of control points. For example, Figure 3a contains 4-patches.

• Reconstruct each patch by using its own control points.
• Warp the patches at their spatial boundaries—for example, regions Ω and $ℳ$ in Figure 3a—by selecting a neighboring set of control points for each boundary. Then, reconstruct the boundaries by using adequate method from Section 2.3.1, or the LSE-TPS algorithm that we described in Section 2.3.3. Figure 3b illustrate warping of multiple patches into a single image.

3. Discussion

In this section, we test the performance of the modified TPS algorithms and compare it with popular interpolation methods used in image processing.

3.1. Performance Studies of the Modified TPS and Windwoing Techniques

We study the perfromance of the Modified TPS for a grayscale image by evalauting its noise-to-signal ratio (SNR) and the accurcy of the interpolation algorithim, which is presented in Section 2.3.1. We define the accuracy by compairing the pixel value of the interpolated location with the true value of the pixel. To conduct such an experiment, we extracted a 100 × 100 Pixel² image from Figure 1a, then reduced the resolution of the extracted image by half. This is done by removing the intermediate pixels from the 100 × 100 Pixel² image. The results are shown in Figure 4.

We use the global modified TPS method to double the resolution of the 50 × 50 Pixel² image in Figure 4b. The pixel intensity is based on a grayscale image with a maximum intensity value of 256. The intermediate points, which bisect the mid distances between every two adjacent pixels in column 37 of Figure 4b, are interpolated using Equation (3).

We define the accumulative error between the true pixel value and the interpolated pixel at a location $\left(x,y\right)$ by

$$e_{MTPS}={\displaystyle \sum }_{k=1}^M\left|{\tilde{z}}_k\left(x,y\right)-z_k\left(x,y\right)\right|$$

(28)

where $M$ is the number of samples, and the true values of $\tilde{z}$ vector are obtained from original image of 100 × 100 Pixel² shown in Figure 4b. The total error based on the modified TPS with ɛ = 10⁻⁶ is calculated at column 37 and it is equal to 363.5187. It should be noted that the intermediate points suffer from small deviation from the true value as shown in Figure 5 and as noted by the non-zero values of accumulative error $e_{MTPS}$. The estimated value of a pixel at the DEM location is equal to the true value, i.e., coincident. This is shown in Figure 6, which plots three superimposed micro-surfaces in three dimension: (1) the triangulation of true pixel of image in Figure 4b; (2) the interpolation of the entire image whose pixels are evaluated at the location of each true pixel by using the Modified TPS method described in Section 2.3.1; and (3) the intermediate pixels which are interpolated along column 37.

The effect of correction factor ɛ in the accuracy of the modified TPS algorithm in grayscale image is studied for a range of values with results shown in

Table 1. Effect of correction coefficient in the accuracy of the modified TPS method.

ɛ	10−6	10−5	10−4	10−3	10−2	10−1	1	10	100
${\mathit{e}}_{\mathit{M}\mathit{T}\mathit{P}\mathit{S}}$	363.5187	363.5192	363.5243	363.5745	364.0648	368.0492	381.5303	392.1309	77852.1

. The method stays accurate for small values of ɛ whose values are comparable to the maximum pixel intensity value in grayscale image, but the method fails when ɛ becomes comparably large.

We utilize MATLAB built-in functions [23] to compare the accuracy of the Modified TPS with a few popular interpolation methods used in image processing toolbox. $Z_t$ = interp2($X,Y,\tilde{Z},X_t,Y_t$, ‘Method’) function returns matrix $Z_t$ containing elements corresponding to the elements of $X_t$ and $Y_t$ and determined by interpolation within the two-dimensional function specified by matrices $X_t$, $Y_t$, and $\tilde{Z}$. The methods that were tested are “nearest” or nearest neighbor interpolation, “linear” or Linear interpolation (default), “pline” or cubic spline interpolation, and “cubic” or cubic interpolation, as long as data is uniformly-spaced. Otherwise, this method is the same as “spline”. We use these methods to interpolate for the target values and then compare it with the modified TPS by using the cost function $e_{MTPS}$. The simulation results are summarized in

Table 2. Effect of correction factor in the accuracy of modified TPS method.

Method	$\mathit{S}\mathit{N}\mathit{R}$ Ϙ	${\mathit{e}}_{\mathit{M}\mathit{T}\mathit{P}\mathit{S}}$ †
DEM: 100 × 100 Pixel2	15.1	-
DEM: 50 × 50 Pixel2	14.9	-
Interpolation, Modified TPS with ɛ = 10−6 **	16.4	363.57
Interpolation, linear method **	15.4	2767.75
Interpolation, cubic method **	15.7	2799.50
Interpolation, nearest method **	14.9	2825.00
Interpolation, spline method **	16.2	2829.80

** Reconstructed to resolution of 100 × 100 Pixel² by using DEM size of 50 × 50 Pixel²; ^† Evaluated along column 37; ^Ϙ Evaluated for the entire reconstructed image with a resolution of 100 × 100 Pixel².

, and show that the modified TPS is on average 770% more accurate than other listed methods.

Another measure used to check on the quality of the interpolation method effectiveness is the relationship between SNR and the interpolation resolution. We designed a case study where a true image with 50 × 50 Pixel² is used to reconstruct 100 × 100 Pixel² image. The SNR values of the entire true images—50 × 50 Pixel² and 100 × 100 Pixel²—are 14.9 and 15.1, respectively. Then, we compare the values of the SNR which is obtained for each method to reconstruct 100 × 100 Pixel² image. Table 2 shows that the nearest method has the lowest SNR among other methods and it is sufficiently equal to the SNR for the true image of 50 × 50 Pixel². This is due to the locality nature of the “nearest” method, which emphasizes the information available around the region of interpolated pixel. However, the modified TPS has the highest SNR value of 16.4 which could be explained by the global nature of the applied modified TPS method where all DEMs in the 50 × 50 Pixel² is weighted in the interpolation.

The last performance measure of the modified TPS method is to understand how the SNR value varies with the number of the interpolated pixels in a given image. We use the source image of 50 × 50 Pixel² in Figure 4b to increase the image resolution by using modified TPS method, and with ɛ = 10⁻⁶. The interpolation is carried out to interpolate the intermediate points between pixels, and with step up resolutions as indicated in

Table 3. SNR relative to interpolation size in a modified TPS method.

Resolution Enhancement (Pixel2) Interpolation by Modified TPS with ε = 10−6 *	SNR ω
50 × 50 Pixel2	14.90
$100\times 100$ Pixel2	16.39
$164\times 164$ Pixel2	16.56
$246\times 246$ Pixel2	16.60
$491\times 491$ Pixel2	16.67

* Interpolation by using DEM size of 50 × 50 Pixel²; ^ω Evaluated for the entire reconstructed image.

. It can be concluded that the value of the calculated SNR increases as the resolution of the reconstructed image increase, however, to understand how such change is related to the size of the reconstructed image relative to the true image, we define SNR error percentage by $\mathsf{\beta }=\left(SN{R}_{Re}-SN{R}_{Tr}\right)/SN{R}_{Tr}\times 100\%$. Where $SN{R}_{Re}$ is the signal-noise ratio of the reconstructed image, and $SN{R}_{Tr}$ is the signal-noise-ratio of the true image used in the reconstruction. The simulation results in Figure 7 shows exponential convergence of β. i.e., the signal-to-noise ratio of a reconstructed image has maximum limit where it reaches saturation at certain pixel resolution. In this case study, the pixel resolution that corresponds to the saturation limit could be obtained from the following suggested fit-model: ${\mathsf{\beta }}^{*}=\left(p_{1}x+p_2\right)/\left(x+q_1\right)$, where $x$ is the pixel intensity expressed by $Log\left(Pixe{l}^2\right)$, and $\left\{p_1,p_2,q_1\right\}$ is a set of undetermined coefficients. The fit analysis is carried out by using MATLAB “fittool” GUI function. The coefficients $\left\{p_1,p_2,q_1\right\}$ are evaluated and found equal to $\left\{12.91,-101,-7.425\right\}$. The “goodness of fit” is evaluated by using the regression coefficient, R², which is equal to 0.9999. The final value of the fit-model, ${\mathsf{\beta }}^{*}$, can be found by using “limit theory”, and it gives ${\mathsf{\beta }}^{*}\left(x\to \infty \right)=12.91\%$, i.e., $SN{R}_{Re}\left(PixelIntensity\to \infty \right)=16.82$. The pixel intensity that corresponds to 1% within the range of the final value of $SN{R}_{Re}$ is ~381 × 381 Pixel². This explains that the SNR value of the reconstructed image would not change significantly at high resolution.

On the other hand, the computational efficiency of the windowing technique, which was discussed in Section 2.3.4, could be evaluated by comparing it with the global modified TPS method in terms of the number of computations required to invert H matrix. For example, the number of the computations required to invert H by using “Householder transformation algorithm” is 3/2n³ + O(n²) [24], where n is the total number of control points within an image. The Householder is a stable and efficient algorithm that is typically used for the inversion of large symmetrical matrices [25]; however, when an image is divided into L patches, the number of the computations becomes 3L/2(n/L)³ + O(n²); therefore, the number of the computation decreases when the number of patches increases. For example, suppose an image contains 1024 pixels. The number of computation required to inverse H matrix with L = 1 and L = 24 are 1.6 × 10⁹ and 2.8 × 10⁶, respectively.

3.2. Previous Studies

A comparison between the bending energy methods is compared with a few popular methods in

Table 4. Comparison with methods used in edge detection.

Method	Description
SIFT	The method uses corner detection algorithm, image matching, affine deformation, viewpoint change, noise, and illumination changes [26]. It shows stability in scale and rotation, robustness to localization error, and reduces the time and computational complexity of images [27].
Harris Corner Detector	The method is based on the local auto-correlation function of a signal particularly to detect gray scale image location that has large gradients in all directions at a predetermined scale. The method has good consistency on natural imagery [28].
SUSAN	The method is based on non-linear filtering, and it associates each pixel of the image a small area of similar brightness from neighboring pixels. It is used as edge finder [29].
SURF	The method is based on multiscale space theory and feature detection by using Hessian matrix, which has good performance and accuracy [26], but needs further improvement when the rotation is large [27].
This Study	Our method uses density information of the extracted features, which is incorporated to regularize the overall energy function to reduce the impact from outliers. It is based on corner detection using local or global properties, which is an effective tool to implement the registration of local image parts, especially when the key points are evenly distributed across the whole image.

. In addition,

Table 5. Comparison with Fundamental methods used in image restoration.

Method	Description
Spatial Filtering Method	Typically, smoothing filters are used for blurring and for noise reduction. Sharpening of the image is done by spatial differentiation which enhances edges and other discontinuities and de-emphasizes areas with slowly varying intensities [9]. Example of spatial methods include the Sobel Method [30], Prewitt Method [31], and Laplacian method [32].
Genetic Algorithm	The method is a stochastic optimization algorithm that is based on four steps: selection, crossover, mutation and fitness function [33]. A genetic algorithm needs less information to solve any problem as compared to conventional optimization methods.
Neural Network Method	Restoration is carried out by parameter evaluation and image construction procedure using learning BP neural network to determine the best image transform function [34]. Neural Network is used to remove noise, motion blur, out-of-focus blur, and distortion caused by low-resolution images [35].
Fuzzy Method	Used to handle uncertainty to enhance contrast of the images and to remove unclearness in edges, boundaries, regions and features of image [36].
This study	We described modified TPS methods, which are based on the original pixel intensity values to evaluate model parameter based on logarithmic RBFs of bending surface of minimum deformation. The model can restore selected areas or entire image, and it is suitable to increase the image resolution, and restore lost information. We combined TPS and LSE in a windowing technique to improve the computation time of reconstruction.

provides a comparison between the developed reconstructions techniques with a few popular fundamental methods used in image restoration.

4. Application Results

In this section, we implement techniques presented in Section 2 into two applications. Software codes were developed in this study by using MATLAB program [23].

4.1. Characterization of Micro-Surfaces Topology

We found in previous sections that the addition of a constant, ɛ, in the TPS model makes it possible to obtain contentious derivatives of the model in the spatial domain. Therefore, in this illustrative example, we characterize the topology of a micro-surface that contains roughness and waviness components. The height variation of a MEMS micro-surface in Figure 8a is measured at different location $\left(x,y\right)$, and then it is registered in DEM. A 3D MEMS profiler, “Veeco” [37] is used to scan the micro-surface at spatial pixel resolution of 0.193 µm, and with a total number of control points of 479 × 639 Pixel². Figure 8b shows a 3D visualization of the pad micro-surface plotted by using all the measured control points. The characterization of such data-rich model is computationally inefficient; therefore, a sufficiently small number of control points would be preferred.

A smaller data set of 10 × 10 Pixel² is uniformly sampled from the DEM at a lateral pixel resolution of 0.965 µm. Figure 9a shows a linear triangulation of 100 control points covering an area of 92.5 µm × 123.3 μm. Equation (8) is applied to extract the TPS coefficients, and then the micro-surface is constructed at a smaller pixel size of 0.5 µm by using Equation (3). Figure 9b shows a continuous and smooth micro-surface that passes through all the 100 control points. The variation of the slopes in the x and y coordinates are obtained from Equations (9) and (10), and then they are plotted at 0.5 µm resolution, as shown in Figure 9c,d, respectively. Similarly, the curvature variation in the x and y coordinates are constructed by using Equations (11) and (12), with the results shown in Figure 9e,f, respectively. These later micro-surfaces measure the topography fluctuation; where a spiky point implies that there is a saddle point close to right angle. The strike angle in Equation (15) is a measure of the topographical contour slopes. Figure 9g shows a reconstruction of the strike micro-surface; i.e., non-uniform fluctuation of strike angles within ±90° range. The dips micro-surface in Figure 9h is constructed from Equation (17), and it shows positive inclination orthogonal to the strike line. The bending energy in Figure 9i is constructed from Equation (18), and it shows the bent variation in the micro-surface due to curvatures. The nodes, which are located on the infinite part of the extrapolated micro-surface, have negligible bending energies; therefore, the energy index can be used to detect the sudden changes within micro-surface such as holes, groves, and standing features in MEMS structures.

4.2. Visual Enhancement of Grayscale Lenna Image

SEM imaging suffers from the following shortcomings: (1) higher magnification leads to smaller field of view; therefore warping multiple images is needed for larger view; (2) high magnification often causes some damages to the specimen due to high electron field; therefore, there might be a need to filter out defects; and (3) high magnification could be non-feasible; therefore, there might be a need to reconstruct a higher resolution image from the lower magnification image. Because our algorithms can be applied to any greyscale SEM image, we select a standard picture from the image processing literature that is analogues to SEM images, and the goal is to guide the readers with the comparison. In this section, a noise-free Lena image [38], which is shown in Figure 10a, has a resolution of 512 × 512 Pixel², and it is used to study the image processing techniques discussed in Section 4. A sub-image of 51 × 51 Pixel² is extracted from the Lena image as shown in Figure 10b. This later image will be used throughout this section.

The resolution of Figure 10b is reduced into a lower resolution image of 26 × 26 Pixel² as shown in Figure 11a. The down sampling scheme followed here is conducted by removing the intermediate pixels located between every two adjacent columns and rows of the image in Figure 10b. The control points, which are contained in this low resolution image, is used to extract the TPS coefficients by using Equation (8), then a higher resolution image of 260 × 260 Pixel² is reconstructed in the global coordinate by using Equation (3). The result is shown in Figure 11b.

An impulsive defect in the form of a black box is added to Figure 10b as shown in Figure 12a. The bending energy Equation (18) is applied to detect the edges of the defect. Figure 12b is the reconstructed bending energy image, and it shows the border of the defect cornered by white pixels; while the pixels away from defected area are registered black. We further implement the TPS methods to enhance the visualization of a defected image. A defect is added to a poor resolution image of 26 × 26 Pixel² as shown in Figure 13a. The defect is detected by using the bending energy algorithm. The control points, which are corresponding to the defect, are removed from the image, leading to a DEM defect-free. This new DEM is used in Equation (3) to reconstruct a higher resolution image of 51 × 51 Pixel² by using the global TPS algorithm, and the result is shown in Figure 13b.

We investigate the windowing techniques for a 51 × 51 Pixel² image. Figure 14a is divided into 5 × 5 square windows. The modified TPS algorithm is then implemented at the local coordinates of each window with the goal to increase the overall image resolution. The overall image resolution has increased from 100 × 100 Pixel² to 250 × 250 Pixel² as shown in Figure 14b,c, respectively.

A quick comparison between global and local modified TPS methods showed that the number of computations required to inverse H matrix of a 26 × 26 Pixel² are 198, 977 and 37,500, respectively; however, the boundaries between the square windows are not smooth, and they become visually sharp when the resolution increases. This is observed in Figure 10b,c. To mitigate such shortcoming, the rest of this section studies the application of warping by using the procedures that we suggested in Section 2.3.3 and Section 2.3.4.

Figure 15a shows an image torn into four pieces, where the white area refers to the missing control points whose spatial locations (x,y) are known. The goal is to warp the pieces into a single image by using global modified TPS, local modified TPS and LSE-TPS methods. First, in the global modified TPS method, the missing pixels are evaluated by using the control points from all the four pieces together. The missing pixels are constructed at same image resolution, and the result is shown in Figure 15b. Second, the local TPS is used to warp the torn image in Figure 16a. The control points, which are located in the left or right part of the torn image, are used separately in the reconstruction of the missing pixels that are located in column 25. Figure 16b,c are reconstructed by using the information in the left part, and right parts, respectively.

In the last warping method, we use the LSE-TPS to construct the missing column 25 by using the information from both column 24 and 26. In this method, the local TPS coefficients of each column 24 and 26 are computed separately. New TPS coefficients for column 25 are estimated from Equations (23)–(27), then column 25 is reconstructed by using Equation (3). The warping quality depends on the weighting matrix and the direction of propagation. For example, column 25 in Figure 17 is reconstructed by propagating the coefficients from left-right and right-left, respectively.

5. Conclusions

In this research, we developed modified TPS and LSE techniques for the reconstruction and characterization of micro-surfaces and greyscale images. The techniques included: (1) reconstruction of an unknown micro-surface from a set of small data sample points; (2) restoration of bad samples; (3) filtering and warping of greyscale images; and (4) enhancement of micro-surface visualization and characterization of MEMS structures. Because the techniques are obtained from the modified ordinary TPS micro-surface, the source data, which is used to reconstruct micro-surfaces and images, are preserved. The reconstruction of high resolution images were conducted by using local or global logarithmic-based on modified TPS method. Local modified TPS restoration is dependent on a subset of data close to the point of evaluation; whereas a global method is dependent on all data. The benefit of global reconstruction of micro-surfaces is capitalized in its simplicity; however, the method is slow and inaccurate for large size of source data. On the other hand, we found that the windowing and warping techniques, when combined, could overcome the pitfall of the global method. Finally, we showed that the bending energy algorithm detects faults in a greyscale image, and locates bent micro-surface on MEMS structures.