All-in-Focus Three-Dimensional Reconstruction Based on Edge Matching for Artificial Compound Eye

Abstract

An artificial compound eye consists of multiple apertures that allow for a large field of view (FOV) while maintaining a small size. Each aperture captures a sub-image, and multiple sub-images are needed to reconstruct the full FOV. The reconstruction process is depth-related due to the parallax between adjacent apertures. This paper presents an all-in-focus 3D reconstruction method for a specific type of artificial compound eye called the electronic cluster eye (eCley). The proposed method uses edge matching to address the edge blur and large textureless areas existing in the sub-images. First, edges are extracted from each sub-image, and then a matching operator is applied to match the edges based on their shape context and intensity. This produces a sparse matching result that is then propagated to the whole image. Next, a depth consistency check and refinement method is performed to refine the depth of all sub-images. Finally, the sub-images and depth maps are merged to produce the final all-in-focus image and depth map. The experimental results and comparative analysis demonstrate the effectiveness of the proposed method.

1. Introduction

Artificial compound eye (ACE) is a type of camera with a miniature volume and large FOV. Different kinds of artificial compound eye systems have been proposed [1,2,3]. The ACE-captured image is an array of sub-images with slightly different viewpoints. All the sub-images can be reconstructed into one large FOV image, but depth information is required for the all-in-focus image reconstruction. As one of the most famous and typical ACE systems, the eCley (Electronic Cluster Eye [1]) (shown in Figure 1) uses a lens array of $17\times 13$ channels, and adjacent channels have an inherent offset. The height of its optical module, about two times shorter than a comparable single-aperture camera with the same resolution, is only $1.4$ mm. Oberdörster et al. present a stitching method that merges all the sub-images and reconstructs an image with a resolution of $700\times 550$ pixels [4]. This method does not consider the effect of parallax, and the images are merged at a specified distance. Consequently, the object located at another distance will be blurred, as shown in the left image of Figure 2. The reconstructed image with all $17\times 13$ channels without depth information is shown in Figure 2a, and our reconstructed image with central $13\times 13$ channels is shown in Figure 2b. To address this problem, the object distance (depth) information is essential.

Stereo matching, one of the fundamental tasks in computer vision, can estimate the depth of objects from at least two cameras with overlapped FOV [5]. Many stereo matching methods have been proposed in recent years (BP [6], SGBM [7], cost–volume [8], Lac-GwcNet [9], PCW-Net [10], IGEV-Stereo [11], etc.) and have achieved state-of-the-art results. Although the adjacent images of ACE have overlapped FOV, traditional stereo matching methods cannot obtain satisfied depth results directly. As shown in Figure 3e,f, the depth map of the traditional method is noisy and inconsistent. This is mainly because of the following:

• The low-resolution and small FOV. In this paper, the rectified sub-image has only $101\times 101$ pixels, and the FOV of the sub-image is about ${9.18}^{\circ }$, which will lead to a large textureless area in the sub-image.
• The large FOV offset. For the eCley, the inherent offset between adjacent sub-images is about 34 pixels, and the overlap area between adjacent images is less than $70\%$. As shown in Figure 4, Figure 4a gives the target sub-image located in the center and its four neighboring sub-images placed at the corresponding location. Figure 4b–d show the overlap between the target sub-image and each adjacent sub-image, where the green area indicates the non-overlapped area. This will result in much less contextual information being used.
• The photo-consistency may not hold for all pixels. Due to the low resolution and noise, the boundary of objects will be blurred, and the point-wise depth map will be very noisy.

To obtained accurate depth for the ACE, an edge matching and propagation (EdMP) method is proposed. The method takes inspiration from local feature matching and propagation methods [12]. Since there is less texture in the sub-image, the edge feature is used to indicate the salient information. To accurately match the edge features between adjacent images, the edge matching cost is computed by combining a modified census transform [13] and shape context method [14]. Additionally, to eliminate the no-match problem caused by the inherent offset, the edge matching results between a sub-image and its four adjacent sub-images are fused to obtain the complete and robust sparse edge depth. The edge depth is then propagated to the whole sub-image to obtain the dense depth map. Finally, the depth of the sub-image is refined by consistency checking with its four neighboring sub-images. Using the depth maps of all the sub-images, it is easy to obtain an all-in-focus reconstructed image. The proposed method is validated by the real captured images, and the experimental results show superior performance compared to other methods. The sub-image depth is shown in Figure 3. For the sub-image in Figure 3a, the edge depth map, dense depth map, and the refined depth map are given in Figure 3b–d, respectively. And our method achieves much better depth results compared with Figure 3e,f.

The rest of this paper is organized as follows: Section 2 introduces the related work. The overview of the proposed method is presented in Section 3. The details of EdMP are given in Section 4. Section 5 shows the experimental results, and the conclusions are provided in Section 6.

2. Related Work

In 2001, Tanida et al., inspired by the dragonfly’s apposition compound eye, proposed a compact imaging system called TOMBO (Thin Observation Module by Bound Optics) [15]. In 2004, Dupparré et al. proposed another artificial apposition compound eye called APCO (Apposition Compound Eye Objective) [16]. Although APCO has multiple imaging channels, each channel only contains one pixel, which leads to an overall resolution of only $130\times 130$. Dupparré et al. proposed a superposition compound eye system called Cley (Cluster Eye) [17]. In 2007, Brückner et al. proposed an artificial neural superposition eye to mimic its bio-prototype [18]. Druart et al. was inspired by a parasite of wasps called Xenos Peckii and proposed the MULTICAM [19]. Each channel of MULTICAM can be seen as a small camera, and the full FOV image of MULTICAM can be reconstructed by the image processing method. In 2010, Brückner et al. followed this idea and proposed the eCley system [1]. eCley can achieve a VGA resolution while keeping the overall thickness at $1.4$ mm. In 2011, Meyer et al. proposed an oCley system [20]. Apart from ACE with plane structure, many curved ACE systems were also proposed to achieve a large FOV. Jeong et al. proposed a hemispherical compound eye [21]. Song et al. proposed an arthropod-inspired camera [22]. Floreano et al. proposed the CurvACE (Curved Artificial Compound Eye) [23]. Many other ACE systems can be found in [3,24,25,26].

Although a lot of ACE systems were proposed and researched, the research is mainly focused on the optical imaging, and the related image processing method is not well studied. Kitamura et al. [27] rearranged pixels of ACE aperture images to reconstruct the high-resolution image for TOMBO. Nitta et al. proposed an iterative back-projection method to improve the result [28]. However, these methods reconstruct the image only on a specified distance. Horisaki et al. proposed a three-dimensional estimation method [29] and then combined the depth map into image reconstruction [30]. Dobrzynski et al. [31] proposed a flexible compound-eye-like sensor to detect motion and estimate the proximity. Luke et al. proposed compound-eye sensor with motion hyperacuity in the same year [32], and then compared it with a CCD Camera [33]. Gao et al. used stereo matching and the SIFT feature to estimate the depth of a TOMBO image [34]. Park et al. proposed a time-stamp-based optic flow ACE sensor for motion detection [35]. Agrawal et al. proposed an edge detection method for ACE to detect horizontal, vertical, forward diagonal, and backward diagonal edges [36]. Lee et al. proposed a COMPU-EYE system and designed a depth estimation method to reconstruct a high-resolution and sharp image [37]. For eCley, Oberdörster proposed a braiding method to reconstruct the image with a final resolution of $640\times 480$ [4,38], but the braiding method can only focus on a specified distance. Ziegler et al. introduced a depth-related image reconstruction method [39], where the depth of each channel image was obtained by fusing the matching results with four neighboring images. Jiang et al. considered the intensity transitional area and proposed a distance measurement method at the object edge and achieved subpixel precision [40]. Wu et al. considered the oblique incidence of the apertures and proposed a geometry-based 3D reconstruction method named G3D [41]. Here, the geometry relationship of optical channels in eCley is derived and then the mathematical relation between the parallax and depth among unparallel neighboring optical channels is obtained to reconstruct the final image. In addition, Wu et al. also proposed an optical flow method for eCley to estimate the object movement [42], and multi-aperture images of eCley provided good constraints on the consistency of optical flow estimates.

Although the ACE system has a similar structure with other imaging systems, like the integral imaging system [43] and light field system [44], and both can use the stereo principle to obtain the depth of objects, for eCley, the inherent offset between adjacent channels leads to an object that can only span at most three channels in one direction. For light field reconstruction, this is quite sparse light field information; therefore, the light field based image reconstruction method is not considered in this paper.

In summary, some progress has been made in recent work on ACE image reconstruction, but further enhancement and research in areas such as accurate depth estimation and high-resolution image reconstruction are still needed.

3. Overview of Edge Matching and Propagation

The workflow of the proposed method is given in Figure 5, and our method mainly consists of the following five processing modules: (1) image correction; (2) edge depth estimation; (3) edge depth propagation; (4) inconsistency check and depth refinement; and (5) all-in-focus reconstruction.

The five modules are briefly explained below, and the following section will give each module in detail:

• Image correction. Because the ACE image suffers distortion, the first step is to rectify sub-images. This will greatly reduce the complexity of the following depth estimation.
• Edge depth estimation. The inputs to the edge depth estimation module consist of a reference image and four adjacent images, which have been rectified already. The main purpose of this module is to obtain a consistent sparse edge depth map of the reference image.
• Edge depth propagation. The purpose of this module is mainly to propagate the edge depth to the whole image. Associated with over-segmentation results and four neighbor images, this matching cost of the segmented area is much more robust to obtain the correct depth.
• Inconsistency checking and refinement. The main purpose of this module is to identify segmented areas which have unreliable depth. Instead of the left–right check used in many stereo matching algorithms, matching consistency with four neighboring images is used.
• All-in-focus reconstruction. Finally, based on the refined depth map of each sub-image, the sub-image arrays are rendered to reconstruct the all-in-focus image, and the depth maps are also merged into a whole depth map.

4. Edge Matching and Propagation Method

4.1. Image Correction

The incidence angles of the eCley sub-apertures are not parallel but have an inherent offset angle. This results in an oblique incidence of the fringe aperture, which further leads to distortion of the image. As shown in Figure 6a, the chessboard is captured by eCley, and the central sub-image (cyan box) is less distorted, but the fringe sub-image suffers from oblique distortion (red box). The oblique distortion requires a complex process for accurate image reconstruction [41]. Therefore, this paper performs the image correction first, and the correction experimental schematic is shown in Figure 6b. The chessboard is large enough to span all the sub-images, and the image correction method proposed in [42] is used in this paper.

4.2. Edge Depth Estimation

4.2.1. Sub-Image Matching Feature Selection

The resolution of the rectified sub-image is $101\times 101$ pixels, as directly matching the adjacent sub-image may cause a mismatch. To obtain accurate depth, the local feature matching-based method inspired by [12] is proposed. However, the resolution of the sub-image is low and the FOV is small, which leads to few feature points by using many robust feature extraction methods. Figure 7 shows six different features (SURF [45], FAST [46], Harris [47], BRISK [48], MSER [49], and Canny [50]) extracted for the rectified sub-image, and

Table 1. Feature count in one sub-image.

Feature	SURF	FAST	Harris	BRISK	MSER	Canny
Count	6	1	18	3	32	554

gives their counts. The sub-image has fewer SURF, FAST, Harris, BRISK, and MSER feature points, whereas Canny feature points reach 554, much more than the other features. Considering the total pixels of the sub-image is $101\times 101$, the Canny feature is also much less than the pixels, which can largely speed up the computation. Therefore, the edge feature is adopted to obtain the sparse depth.

In addition to the large number of edge features, another reason for choosing edge features is that all-in-focus image reconstruction is mainly meant for obtaining the object depth information, and the depth estimation algorithms are essential to finding the matching points of the reference image points from the corresponding image. In the original information of the image, the edge points are the most informative features and can be used to describe the magnitude and direction of the image gradient. The multiple consecutive edge points can also be concatenated to carve the pattern and describe the shape of the pattern, etc.

Then the sparse edge depth of one sub-image can be estimated by matching it with its four neighboring sub-images. The depth estimation diagram is shown in Figure 8.

4.2.2. Edge Feature Matching

In the eCley, the focal length is $f=0.778$ mm, the pixel size is $3.2$ $\mathsf{\mu }$m, the rectified image is upsampled by a factor of 2 from the original image, and there are 111 pixels between centers of the rectified adjacent sub-images, so $\Delta x$ of the rectified sub-image is $1.6$ $\mathsf{\mu }$m. According to the relationship between disparity and distance of the stereo image in Equation (1),

$$Z=\frac{B·f}{d·\Delta x}$$

(1)

where Z is the distance of the object, and d is the estimated disparity (or object depth). Therefore, when $d=1$, then $Z=172$ mm; when $d=10$, then $Z=17.2$ mm; and when $d=15$, then $Z=11.47$ mm. The distance of the object is only $1.1$ cm when the disparity reaches 15, so, in the edge matching, the disparity is assumed to be below 15.

Although eCley images can extract a large number of Canny edge features, the edge features only respond to the location where the image color changes suddenly, and edge matching is prone to mismatching if only the color information is used. To obtain robust edge depths, two different types of operators (improved Census operator [13] and SC (Shape Context) operator [14]) are fused for edge feature matching.

Supposed for the sub-image I, $I\left(p\right)$ indicates the intensity of pixel p. The improved Census operator of p is given in Equation (2),

$$\begin{array}{c}\hfill T\left(p\right)=\underset{q\in W_p^{\prime }}{⨂}\xi (I\left(p\right),I\left(q\right)),\\ \hfill \xi (I\left(p\right),I\left(q\right))=\left\{\begin{array}{cc}\hfill 1,& \hfill b+\alpha <c\\ \hfill 0,& \hfill b-\alpha \le c\le b+\alpha \\ \hfill -1,& \hfill b-\alpha >c\end{array}\right.\end{array}$$

(2)

where ⨂ denotes concatenation, $W_p^{\prime }$ is the local window centered in pixel p, and $\alpha$ denotes noise tolerance. Due to the low resolution of the sub-image, the system noise may lead to fluctuations in the edge transition region. Hence, the noise tolerance is introduced to improve the matching stability.

The SC operator is widely used in object matching with similar shapes such as digit recognition. Therefore, the SC operator introduces the geometry similarity for edge matching. For pixel p, the SC operator obtains the edge content information from statistical histogram $h_p\left(k\right)$ of window $W_p$ centered in p using Equation (3),

$$h_p\left(k\right)=\#\{q\ne p:(q-p)\in bin\left(k\right)\}$$

(3)

where $bin(·)$ uses the log-polar coordinate system and uniformly partitions the coordinate system into K regions. $bin\left(k\right)$ is the $k_{th}$ part. q is the edge point in $W_p$, and $h_p\left(k\right)$ indicates the number of q in $bin\left(k\right)$.

The Census and SC operators are used to match the edge point. Suppose I is the target sub-image, $I_i\in N_I$ is the adjacent sub-image, and $N_I$ denotes the four neighboring sub-images of I. For sub-image pair I and $I_i$, the matching cost of edge point $I\left(p\right)$ to have a disparity of d is

$$C^i(p,d)=(1-\beta )·C_s(p,p^{\prime }\left(d\right))+\beta ·C_c(p,p^{\prime }\left(d\right))$$

(4)

where $C_s(p,p^{\prime }\left(d\right))$ and $C_c(p,p^{\prime }\left(d\right))$ are the cost of SC and Census operators (given in Equation (5)), respectively. $p^{\prime }\left(d\right)$ indicates the edge point with disparity d in the matching sub-image $I_i$.

$$\begin{array}{ccc}\hfill C_c(p,p^{\prime }\left(d\right))& =\hfill & \hfill Hamm(T\left(p\right),T_i\left(p^{\prime }\left(d\right)\right))\\ \hfill C_s(p,p^{\prime }\left(d\right))& =\hfill & \hfill \frac{1}{2}\sum _{k=1}^K\frac{{[h_p\left(k\right)-h_{p^{\prime }\left(d\right)}\left(k\right)]}^2}{h_p\left(k\right)+h_{p^{\prime }\left(d\right)}\left(k\right)}\end{array}$$

(5)

where $Hamm(·)$ denotes Hamming distance. If no matching point exists in the matching sub-image with disparity d for pixel p, $C^i(p,d)=\infty$, then the winner-takes-all (WTA) method is used to select the edge depth map $D^i\left(p\right)$.

4.2.3. Edge Depth Fusion

$D^i\left(p\right)$ obtains the depth map of I with an adjacent sub-image $I_i$, but the inherent offset between adjacent sub-images leads to an incomplete depth map, as shown in Figure 9. To estimate the complete depth map of I, depth maps of all four neighboring sub-images are fused.

In the depth fusion step, the depth reliability of $D^i\left(p\right)$ is first detected using Equation (6). Then the edge direction of pixel p is utilized, and if the offset between edge direction and depth search direction is smaller than $5^{\circ }$, the depth of pixel p is also considered unreliable, and set to 0.

$$D^i\left(p\right)=\left\{\begin{array}{cc}\hfill d,& \hfill C^i(p,d)\le \delta \\ \hfill 0,& \hfill C^i(p,d)>\delta \end{array}\right.$$

(6)

After that, all four edge depths are fused to obtain the final edge map $D\left(p\right)$, which is fused in the following three cases:

• The pixel p in the four $D^i\left(p\right)$ has the same depth. The depth of $D\left(p\right)$ remains the same.
• The pixel p in the four $D^i\left(p\right)$ has different depths, and the difference is smaller than 2 pixels. The smaller depth is chosen.
• The pixel p in the four $D^i\left(p\right)$ has different depths, and the difference is not smaller than 2 pixels. The depth of the pixel is considered unreliable, and $D\left(p\right)$ is set to 0.

4.3. Edge Depth Propagation

Since the edge depth map is only a sparse depth map, it is necessary to propagate the sparse depth map to obtain a dense map. Based on the assumption that the depth will not abruptly change in a similar color region, the propagation process mainly utilizes the sub-eye image brightness information to segment the image into regions and then propagates the edge depth to the regions. The pseudo-code is shown in Algorithm 1. The sub-images I, $I_i$, and the edge depth $D\left(p\right)$ are the inputs to Algorithm 1; the output is the dense depth map $D_d\left(p\right)$. The target image I is first over-segmented by the meanshift method [51] in this paper (step 1), then each segmented area $Am$ and segmented information S are obtained. The depth value and the depth number for each area $S.area$ are saved (step 2–step 5). The depth of the neighboring area is used (step 9–step 11). The matching cost of area $S.area$ is then computed according to the candidate depth (step 12–step 14) in Equation (7). The area depth is then selected by WTA.

Algorithm 1 Edge Depth Propagation

Input: Edge depth map ($D\left(p\right)$) and color images ($I,I_i$) Output: Dense depth map $D_d\left(p\right)$ 1: Image over-segmentation, $[Am,S]=meanshift\_segmentation\left(I\right)$ 2: $\mathbf{for}$ each segmented area $S.area$ $\mathbf{do}$ 3:  $d\_can\leftarrow$ find edge depth values in this area; 4:  $d\_num\leftarrow$ the number of depth in this area; 5: $\mathbf{end}\mathbf{for}$ 6: $S.area\leftarrow$ sort $S.area$ based on $d_{n}um$; 7: $d_s=0$ 8: $\mathbf{for}$ each segmented area $S.area$ $\mathbf{do}$ 9:  $\mathbf{for}$ each neighboring area $Am$ $\mathbf{do}$ 10:  $d\_can\leftarrow$ add the estimated depth $d_s$; 11: $\mathbf{end}\mathbf{for}$ 12: $\mathbf{for}$ each candidate depth $d\_can$ $\mathbf{do}$ 13:  $c\left(d\right)\leftarrow$ compute the matching cost; 14: $\mathbf{end}\mathbf{for}$ 15: $d_s=argmin\left(c\left(d\right)\right)$ 16: $D_d(S.area)=d_s$

$$\begin{array}{cc}\hfill c\left(d\right)=& \frac{1}{num}\sum _{num}\frac{\sum _i{\delta }_i\left(p\right)c^i(p,d)}{\sum _i{\delta }_i\left(p\right)+ϵ}\hfill \\ \hfill c^i(p,d)=& (1-\lambda )·{Mt}^i(p,d)+\lambda ·{Mg}^i(p,d)\hfill \\ \hfill {Mt}^i(p,d)=& min\left\{\right|I\left(p\right)-I_i\left(p^{\prime }\left(d\right)\right)|,{\tau }_t\}\hfill \\ \hfill {Mg}^i(p,d)=& min\left\{\right|G\left(p\right)-G_i\left(p^{\prime }\left(d\right)\right)|,{\tau }_g\}\hfill \end{array}$$

(7)

where $num$ is the pixel number of the corresponding area, ${\delta }_i\left(p\right)=1$ if pixel p in I has a corresponding edge pixel with depth d in sub-image $I_i$, $ϵ$ is a small constant to avoid dividing by zero. ${Mt}^i(p,d)$ and ${Mg}^i(p,d)$ are intensity cost and gradient cost, respectively. $\lambda$ is weight. $G\left(p\right)$ computes the gradient of p; $p^{\prime }\left(d\right)$ denotes the corresponding pixel in sub-image $I_i$ to have the depth of d, ${\tau }_t$; and ${\tau }_g$ are the truncate threshold.

4.4. Inconsistency Check and Depth Refinement

The propagated depth map of sub-image I only uses the color and gradient information of itself. Therefore, the depth of the image array can be obtained independently. However, the estimated depth may be wrong due to the low context in the sub-image. Considering the adjacent sub-images have a common FOV, the depth information of the neighboring sub-image is also utilized to refine the estimated depth map $D_d\left(p\right)$. The consistency check and depth refinement module is mainly divided into two steps. The first step is to check the consistency and fill the depth in each segmented area, and the second step is to check the depth consistency for each pixel and then the depth of the inconsistent pixel is interpolated by using the matting Laplacian method [52].

4.5. All-in-Focus Reconstruction

After the inconsistency check and depth refinement, the depth map of each sub-image is obtained. Due to the small FOV of the sub-image in the eCley, the sub-image can only capture a small part of the entire FOV. The full FOV image is obtained by fusing all sub-images.

The rectified sub-image can be considered spatially uniformly sampled, and the spatial coordinates $(x,y,z)$ of each pixel can be obtained by projecting the sub-image array into space. To reconstruct the image with the full FOV, the image range is obtained based on the $x,y$ upper and lower limits. Then the position of the pixel points in the reconstructed image and the sampling interval are determined based on the reconstructed image resolution. There are multiple pixels in the resampling interval. When the z values of all pixels are close, the intensity of the resampled pixel is the average intensity of all pixels, and the depth of the resampled pixel is the average depth of all pixels. When the z values of all pixels are not close enough, occlusion may occur, the intensity of the resampled pixel is assigned with the average intensity of the pixels with the smaller z, and the depth of the resampled pixel is assigned with the average depth of the pixels with the smaller z. After iterating all the resample pixels, the reconstructed image $I^{\prime }$ and fused depth map $D^{\prime }$ are obtained.

5. Experiments

In this section, the proposed method is validated and compared quantitatively and qualitatively with three methods [6,8,41]. All algorithms are implemented in Matlab on the same laptop. The effectiveness of the EdMP method is mainly verified by the depth estimation results of the sub-image, the definition of the depth-based 2D reconstructed images, and the 3D point cloud images.

5.1. Experimental Settings

eCley has $17\times 13$ sub-images, but only the central $13\times 13$ sub-images are used for all-in-focus reconstruction. Each sub-image has a resolution of $101\times 101$ pixels after rectification. For the parameters, the noise tolerance $\alpha =0.05$, the weight $\beta =0.07$, $\gamma =0.5$, $\delta =0.08$ empirically. By comparing the MSE of a test image, the different values of $\alpha$, $\beta$, $\gamma$, and $\delta$ have similar MSE. The size of local window influence the computation time and depth estimation result, as illustrated in [41]. To balance the efficiency and effectiveness, $W\left(p\right)$ and $W^{\prime }\left(p\right)$ are both $15\times 15$ in this paper. K is set as the same as in [13] and consists of 3 radius regions and 12 angle regions for a total of 36 regions.

The qualitative evaluation metrics include the reconstruction error and the reconstructed image definition. The reconstruction error is measured by MSE (mean squared error) in [41]. The image definition is mainly measured by comparing the gradient of the image; therefore, four commonly used gradient functions (Brenner gradient $D_b$, Tenengrad gradient $D_t$, SMD method $D_s$, and Product of intensity variance $D_p$) for evaluating image definition are used [53].

$$D_b=\sum _x\sum _y{|I(x+2,y)-I(x,y)|}^2$$

(8)

$$\begin{array}{c}\hfill D_t=\sum _x\sum _y{\left|G(x,y)\right|}^2,G(x,y)>T\\ \hfill G(x,y)=\sqrt{G_x^2(x,y)+G_y^2(x,y)}\end{array}$$

(9)

$$D_s=\sum _x\sum _y{\left(\right|I(x+1,y)-I(x,y)|}^2{+|I(x,y+1)-I(x,y)|}^2)$$

(10)

$$D_p=\sum _x\sum _y\left(\right|I(x+1,y)\ast I(x,y)|\ast |I(x,y+1)-I(x,y)\left|\right)$$

(11)

where $G_x(x,y)$ and $G_y(x,y)$ are the Sobel gradient. The higher the value of these four functions, the more drastic the changes in the image edge and the higher the image definition.

5.2. Evaluation

5.2.1. Image Correction

To achieve accurate depth estimation, image correction plays a crucial role. To verify the effect of image correction, in Figure 10, the calibration board and another test pattern are placed parallel to eCley, and the sub-images are fused and reconstructed at a certain distance. If the sub-images are corrected and reconstructed at the right distance, the reconstructed image will be all-in-focus. As shown in Figure 10a,b, the reconstructed image without correction is blurry at the border of the image, while the image is all-in-focus after correction. Figure 10c,d show the MSE of each pixel in reconstructed image of (a) and (b), where the white area has larger MSE than the black area. As one can see, the reconstructed image after correction has larger MSE in the transition area. The MSE of the whole image of (a) and (b) are $0.0427$ and $0.0205$, respectively. The sub-images of the calibration board before and after correction are shown in (e) and (f), respectively. The sub-images of different channels may have different distortions, which are corrected with our method. Moreover, with the test pattern, the sub-images are fused and reconstructed in two resolutions—$480\times 480$ and $640\times 640$—and the reconstructed images in both resolutions are blurry before correction, while all-in-focus after correction.

5.2.2. Edge Depth Estimation

In Figure 11, a comparison of the edge-matching operators is presented. The results show that both the Census and SC operators are capable of producing accurate edge depth maps. However, using a single operator can result in depth instability for some edge points and be discarded. By combining the two operators, more edge points on the map can obtain depth values and have greater support in the subsequent depth propagation process.

5.2.3. Sub-Image Depth

Figure 12 shows the depth map of the eCley sub-image array after edge depth propagation and optimization. As seen in the figure, the depth map becomes denser after edge depth propagation. However, it suffers from a large number of inconsistencies and depth abruptness due to the process not using the depth information from neighboring images. After the consistency check and depth optimization, the image array’s depth becomes highly consistent.

5.2.4. Image Reconstruction

The reconstructed image is evaluated in this section. The test sub-image arrays after correction are shown in Figure 13, and the objects in the experiment are located at different distances. Figure 14 compares the all-in-focus reconstructed image of our method with images that are focused on a specific distance. As shown in the figure, if the reconstruction is focused only on a certain distance, objects located at other distances will appear out of focus and become blurry. However, by combining the distances during reconstruction, the reconstructed image will be all-in-focus.

The reconstructed images are compared with three methods (BP [6], Cost Filter [8], and G3D [41]). The images and corresponding quantitative evaluation results are shown in Figure 15 and

Table 2. Reconstructed image evaluation.

eCley Image	Metrics	Sub-Image 1	Sub-Image 2	Sub-Image 3
BP [6] 2003	MSE	0.0823	0.0593	0.0766
	$D_b$	2732	1910	1783
	$D_t$	11682	7427	11386
	$D_s$	665	367	489
	$D_p$	3982	2417	3142
Cost Filter [8] 2012	MSE	0.0843	0.0654	0.0829
	$D_b$	2662	1821	1704
	$D_t$	10821	7309	10923
	$D_s$	587	321.4	501
	$D_p$	3811	2398.2	3028.7
G3D [41] 2018	MSE	0.048	0.0536	0.0535
	$D_b$	2883 1	1987.2	1847
	$D_t$	11861	7654.5	11556
	$D_s$	684.4	385	519.8
	$D_p$	4099	2501.8	3179.8
EdMP [this work] 2024	MSE	0.0412	0.0507	0.0476
	$D_b$	2520	2288.1	2045
	$D_t$	16,446	11,983	17,116
	$D_s$	463.6	622.7	648.9
	$D_p$	3105	3273.8	3522.7

¹ The bold means the best.

, respectively. To better visualize the result of Table 2, Figure 16 shows the result in a histogram. It is evident from the results that EdMP outperforms the other methods in terms of all the metrics, except for the metrics $D_b$, $D_s$, and $D_p$ of sub-image 1. This indicates that EdMP is capable of reconstructing images with the best definition. The reconstructed image in Figure 15, especially the details shown in the bottom row, also demonstrates that EdMP yields the sharpest images. To evaluate the computational complexity of the proposed method, all the methods are tested on a HP laptop with Intel Core i5-3210M 2.5 GHz CPU (Intel, Santa Clara, CA, USA) and Matlab2018b (version 9.5.0.944444), and the average execution times of these four methods are listed in

Table 3. The average execution time.

Method	EdMP	Cost Filter	BP	G3D
Time (s)	180.24	197.92	312.77	169.53

. G3D projects the sub-image directly into space to obtain pixel depth information, so the G3D method is the most efficient. EdMP is slightly slower than G3D due to the need for neighboring sub-image matching, but the method is significantly more efficient compared to Cost Filter and BP.

After the image is reconstructed, it is re-projected using the depth map to create a 3D point cloud map. The point cloud maps are shown in Figure 17. Based on the red box of images, the 3D point cloud created by EdMP is the most complete among the four methods for different objects. This indicates that the reconstructed images of EdMP have the most accurate depth information.

5.2.5. Failure Cases

It must be noted that the estimated depth map is propagated based on the image over-segmentation result, and the segmented area is set to an identical depth. When the scenarios contain a sloping plane, the proposed method will fail and the reconstructed image along the sloping plane will be blurry and aliased. As shown in Figure 18, the sloping plane is segmented with different sizes. Although finer segmentation granularity improves depth map results and reconstructed images to some extent, slope plane depth discontinuities persist, and the reconstructed images are still aliased and blurred.

By comparison with another three methods, EdMP achieved the best result for eCley-captured ACE images. This is mainly because the BP and Cost Filter methods are directly modified from the stereo matching method, but the ACE image has a lower resolution, smaller FOV, and also a large offset angle. Without incorporating multiple adjacent sub-images, the depth estimation will be influenced by the noise and image quality. G3D utilized all adjacent sub-images to estimate the depth. However, G3D calculated the incidence angle of each pixel based on the theoretical design values. The depth estimation performance is influenced by the image distortion. Our method, EdMP, corrected the image distortion parameters and utilized four adjacent sub-images for depth estimation. From the failure case, we can see that, since the EdMP uses depth propagation to estimate the depth of the non-edge area, this mechanism will fail in the sloping plane, so a sub-pixel depth estimation method is needed. In addition, the depth obtained by depth propagation can also be influenced by large textureless areas, and the depth estimation on textureless areas is also a fundamental problem in stereo depth estimation.

6. Conclusions

In this paper, we propose an all-in-focus reconstruction method EdMP for eCley. Based on the rectified eCley image, EdMP fully considers the eCley camera and imaging characteristics, and maintains the parallax and depth consistency of multiple neighboring channels by compensating for the inherent offset between sub-images. We propose to use edge information for sparse depth acquisition, obtain reliable depth information by matching four neighboring channels simultaneously, and then propagate the depth to the full image using the depth plane consistency assumption, and finally further optimize the depth by performing a uniform consistency detection for all sub-image depths. In the experiments, EdMP has the smallest MSE among four methods and achieves an average of 0.0465, compared to 0.0517, 0.0727, and 0.0775 for the G3D, BP, and Cost Filter methods, respectively. The effectiveness of EdMP is validated in the experiments and the disadvantages of EdMP is also analyzed. In the future, the depth estimation and reconstruction of the slope plane and the robust depth estimation in the textureless area will be further researched.