Ghost Removal from Forward-Scan Sonar Views near the Sea Surface for Image Enhancement and 3-D Object Modeling

Abstract

Underwater sonar is the primary remote sensing and imaging modality within turbid environments with poor visibility. The two-dimensional (2-D) images of a target near the air–sea interface (or resting on a hard seabed), acquired by forward-scan sonar (FSS), are generally corrupted by the ghost and sometimes mirror components, formed by the multipath propagation of transmitted acoustic beams. In the processing of the 2-D FSS views to generate an accurate three-dimensional (3-D) object model, the corrupted regions have to be discarded. The sonar tilt angle and distance from the sea surface are two important parameters for the accurate localization of the ghost and mirror components. We propose a unified optimization technique for improving both the measurements of these two parameters from inexpensive sensors and the accuracy of a 3-D object model using 2-D FSS images at known poses. The solution is obtained by the recursive updating of sonar parameters and 3-D object model. Utilizing the 3-D object model, we can enhance the original images and generate synthetic views for arbitrary sonar poses. We demonstrate the performance of our method in experiments with the synthetic and real images of three targets: two dominantly convex coral rocks and a highly concave toy wood table.

1. Introduction

In addition to enabling the generation of environmental maps, building 3-D object models from 2-D images provides information for automatic target recognition, establishing the types, sizes, and relative spatial arrangement of various scene objects, collision-free navigation, landmark-based positioning, reallocating objects within mapped sites, and change detection.

For underwater applications, 2-D optical imaging has restricted range in relatively clear waters and quickly becomes ineffective under poor visibility and turbidity—conditions that are commonly encountered in polluted ports, shallow turbulent surf zones, rivers, marine sanctuaries, lakes, ponds, and other still bodies of water. Here, sonar is the preferred and primary visual sensing modality, by virtue of acoustic signal penetrability through silt, mud, and other similar sources of water turbidity.

The imaging sonars have been commonly utilized in subsea remote sensing applications to generate 2-D visual and 3-D topographical maps of the ocean floor [1,2]. In one deployment scenario, a side-scan sonar (SSS) operating at frequencies of 100–500 KHz can map narrow strips of the terrain that extend up to several hundred meters on either side of a ship [3,4]. The typical (across-track) range resolutions of several centimeters can be improved to 1–2 [cm] at operating frequencies of about 1 MHz. This is achieved at the cost of decreased maximum ranges of only 10’s of meters because the acoustic waves at these high frequencies attenuate faster than those at lower frequencies that are commonly employed for long-range applications.

The Synthetic Aperture Sonar (SAS) technology [2,5,6] overcomes the decreasing SSS resolution (when operating at the lower frequencies required for long-distance applications) by integrating data collected over multiple sonar transmissions with a synthetic array. Here, constant resolution with range can be achieved by scaling of the array; increasing the array length for longer ranges.

The 2-D multi-beam FSS systems, originally developed in the early 2000s at operating frequencies of 100’s KHz, have been commonly deployed near the front end of the hull for obstacle-free navigation in shallow, poorly-chartered waters [7,8,9]. The advanced units operating at 2–3 MHz produce high-resolution video images of small objects with sub-centimeter resolution at 10–30 frames/s but a limited maximum range of about a dozen meters; e.g., [10,11,12]. The mapping of relatively large areas of the sea bottom can be achieved by image mosaicing [13,14,15,16,17]. The increasing use of these high-resolution FSS systems has sparked interest in developing algorithms for automated 2-D sonar image processing, interpretation, and 3-D object reconstruction.

In 2-D ranging systems, it is common to employ the spherical coordinate ${(\Re ,\theta ,\varphi )}^T$ for the 3-D position of an object/scene surface points. Here, the elevation angle $\varphi$ is lost in the projection of these 3-D points onto the 2-D sonar image. The 3-D reconstruction from FSS data involves the estimation of the unknown $\varphi$ by various “shape from X” techniques [18]. For example, several 2-D images at a multitude of known poses can be used to generate high-resolution 3-D object models [19,20,21,22,23,24,25], with key applications in target recognition, environmental mapping, and marine scientific studies that rely on the distribution of object types and sizes. However, satisfactory performance often requires certain implicit/explicit restrictive assumptions and conditions about the environment, sonar trajectory, as well as target shape. For example, nearby reflecting surfaces in cluttered scenes and (or) surface patches within object concavities generate multipath that can corrupt and degrade the target image. Thus, the key conditions to minimize data corruption for accurate 3-D model reconstruction include the following: (1) dominantly convex targets (with only mild concavities producing no more than negligible multipath reverberation); and (2) no nearby surfaces with strong acoustic reflectance, e.g., air–sea interface, hard bottoms, and clutter.

The sonar images of targets resting on hard bottoms or in shallow water (floating) near the air–water interface can be corrupted by the contributions from the multipath propagation of acoustic beams [19,20,26,27]. These generate ghost regions that often overlap with the target image (the multipath phenomenon also arises in deploying a high-resolution radar on an autonomous road vehicle for environmental perception [25,28,29,30,31], where the key objective is to locate the “ghost” vehicles in the scene images). Referring to each of the four views of a coral rock in Figure 1, the indistinguishable overlapping ghost object corrupts both the shape and intensity values of (some parts within) the coral image region. Additionally, a mirror region is formed by the “virtual mirror object” above the water surface (or below the seabed). As in (a), the mirror component (nearly) coincides with the ghost region when the image plane is (roughly) parallel to the sea surface (or bottom). It separates into a distinct blob upon rotating the sonar about the viewing axis; here, from 0° to 67.5° in steps of 22.5°.

In this work, we investigate the generation of a 3-D model for an object floating near the sea surface, imaged by an FSS from a multitude of known poses. To this end, improved 3-D modeling performance directly depends on the accurate knowledge of sonar pose relative to the air–water interface, allowing us to circumvent the shape distortion in the 3-D object modeling process. To be precise, the air–water interface distance and the sonar tilt angle are two key pose parameters in localizing the ghost and mirror components. Both parameters can be determined using relatively inexpensive sensors [32,33], but the measurement inaccuracies are generally unacceptable. We propose to estimate these with reduced uncertainty and (or) improved precision by devising an integrated recursive optimization technique that builds on an earlier 3-D modeling framework [19,20]. Moreover, we exploit the mirror contours, where distinct in certain views, as a regularizer to improve the reconstruction accuracy.

Additionally, our method enables the generation of ghost-free object views from the model-based synthetic images [26,27,34,35,36], using a look-up table that is constructed from the intensity distributions within the uncorrupted object regions in the data and synthetic images. Experimental results confirm that our method performs consistently with real and computer-generated data under similar imaging conditions. The real data includes the images at known poses of two dominantly convex coral rocks and a concave toy wood table, recorded by a lens-based Dual-Frequency IDentification SONar (DIDSON) [10].

The remaining sections are organized as follows: Section 2 covers (1) some notations and relevant technical background, including 3-D coordinate transformation, 3-D-to-2-D sonar projection, representation of sonar data as a beam-bin array, and transformation to generate a 2-D FSS image; (2) a unified optimization framework for 3-D target modeling and sonar pose estimation, error metrics applied for the quantitative assessment of the results, and the generation of ghost-free object images. In Section 3, we first present the results of experiments with synthetic data, aimed at assessing the accuracy and convergence rate of our optimization scheme for 3-D modeling and sonar pose estimation. Here, we utilize the 3-D models of our three targets generated by a Kinect camera under the same imaging conditions as the real data. We then give the results of experiments with real data. We summarize our contributions in Section 4.

2. Materials and Methods

2.1. Notation, Coordinate Transformation and FSS Image Formation

The Cartesian $\mathbf{P}=(X,Y,Z)$ and spherical $\mathbf{\Pi }=(\Re ,\theta ,\varphi )$ coordinates of a 3-D point transform according to the following equations:

$$\mathbf{P}=\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]=\Re \left[\begin{array}{c}cos\varphi sin\theta \\ cos\varphi cos\theta \\ sin\varphi \end{array}\right]⟶\mathbf{\Pi }=\left[\begin{array}{c}\Re \\ \theta \\ \varphi \end{array}\right]=\left[\begin{array}{c}\sqrt{X^2+Y^2+Z^2}\\ ta{n}^{-1}(X/Y)\\ ta{n}^{-1}(Z/\sqrt{X^2+Y^2})\end{array}\right]$$

(1)

Figure 2a depicts a single sonar beam in the azimuth direction $\theta$. In a multi-beam FSS, the horizontal field of view (FoV) $\left|\theta \right|\le W_{\theta }$ is covered by $N_b$ beams, each with a narrow horizontal width $\delta \theta$. The vertical FoV is fixed by the half-angle beam width $W_{\varphi }$. For most existing FSS systems, we have $\delta \theta =0.25°-1°$, 6° ≤ W_ϕ ≤ 10° and 15° ≤ W_θ ≤ 65.

At various sample times, the round-trip times of flight $T_{tof}$ of target echos in a direction $\theta$ yield the range measurements $\Re ={\displaystyle \frac{1}{2}}{T}_{tof}\upsilon$, where $\upsilon$ is the sound speed within the medium. Ideally, we calculate $(\Re ,\theta )$ from the so-called “beam-bin” $(b,B)$ coordinates by the following linear equations:

$$\begin{array}{cc}\hfill \Re & ={\Re }_{\min }+(B-1)\delta \Re (B=1,2,\dots ,N_B)\hfill \\ \hfill \theta & =-W_{\theta }+(b-1)\delta \theta (b=1,2,\dots ,N_b)\hfill \end{array}$$

(2)

where $N_B$ range bins cover the minimum range ${\Re }_{\min }$ to maximum range ${\Re }_{\max }$ with resolution $\delta \Re =({\Re }_{\max }-{\Re }_{\min })/(N_B-1)$. In practice, deviations from these models are determined and rectified by calibration; e.g., a “mild” cubic equation models the b-to-$\theta$ transformation in a lens-based DIDSON with lens distortion [37].

The beam-bin intensity array $I(b,B)$ encodes the strength of (collective) echoes from potentially several scene surface patches within the elevation arc $W_{\varphi }$, leading to the inherent many-to-one projection ambiguity of FSS imaging. We construct the 2-D FSS image $I(x,y)$ from the beam-bin data using the transformation

$$\left[\begin{array}{c}x\\ y\end{array}\right]=\Re \left[\begin{array}{c}sin\theta \\ cos\theta \end{array}\right]$$

(3)

The elevation angles $\varphi$ of points on an object or scene surfaces (over some region of interest) are lost in the 3-D-to-2-D projection process. The application of a 3-D reconstruction technique involves the recovery of unknown $\varphi$ from the radiometric and (or) geometric cues in one or more FSS images. In this work, we utilize the data captured with a DIDSON unit, for which an image formation model has been derived [26]; this enables synthesizing images of a given object at known poses relative to the sonar.

2.2. 3-D Object Modeling

Applying the space carving (SC) paradigm [38], a 3-D reconstruction method has been proposed [39] to generate an object surface model $S^o$ from M images $I_m$ at distinct sonar poses ${\mathbf{m}}_m$ ($m=0,1,\dots ,M-1)$. The $M=M_p{M}_r$ poses are represented by 3-D translation ${\mathbf{t}}_i$ ($i=0,1,\dots ,M_p-1$) and rotation ${\mathbf{r}}_j$ ($j=0,1,\dots ,M_r-1$) vectors relative to reference pose ${\mathbf{m}}_0={[{\mathbf{t}}_0^T,{\mathbf{r}}_0^T]}^T$. The carving efficiency is improved by acquiring $M_r$ rotated images at each of $M_p$ positions around the object; through incremental roll rotations $d{\omega }_r=\pi /M_r$ [rad] (about the viewing axis) from $-\pi$ to $-\pi +(M_r-1)d{\omega }_r$ (The shape complexity, namely, convexity or concavity are two key factors in setting $M_p$ and $M_r$).

The SC method exploits geometric cues solely. First, the 3-D space visible in all sonar poses is discretized into volume elements (voxels). Each is examined to label as an object voxel if it projects onto the feasible region in every view; otherwise it is labeled as a non-object voxel and discarded. The feasible region of each view is defined by the object highlight and dark/shadow region behind it, onto which the object surface patches occluded by the visible surfaces may project. The surface model $S^o$ is the outer boundary of the volume formed by all object voxels, as shown in Figure 2b for the coral-two rock in our experiments. Referring to Figure 2b, the SC model $S^o$ consists of a closed mesh of $N_T$ triangles formed by $N_p$ vertices ${\mathbf{P}}_l^o$ ($l=1,\dots ,N_p$); the triangulation connectivity list $T_s$ fixes the $N_T$ triangles in $S^o$, formed by the triplet sets $\{{\mathbf{P}}_i^o,{\mathbf{P}}_j^o,{\mathbf{P}}_k^o\}$.

The 3-D model refinement scheme in [19,20] is an efficient implementation of the optimization framework in [40] that utilizes the intensity data within object regions in various views by minimizing the discrepancy between the data and synthetic target images, generated by the sonar image formation model in [26]. Starting with the SC mesh model $S^o$, the sought-after optimal model $\tilde{S}$ is the discrete approximation of the true continuous surface $S^c$, defined by the $N_p$ displaced vertices ${\tilde{\mathbf{P}}}_l={\mathbf{P}}_l^o+{\tilde{\mathbf{V}}}_l$ ($l=1,\dots ,N_p$) of the $N_T$ triangles in the triangular connectivity list $T_s$ of $S^o$. The optimal displacements $\tilde{\mathbf{V}}=[{\tilde{\mathbf{V}}}_1,{\tilde{\mathbf{V}}}_2,\dots ,{\tilde{\mathbf{V}}}_{N_p}]$ are calculated iteratively by minimizing the sum-squared difference between the data $I_m$ and synthetic images ${\tilde{I}}_m$ over all M poses:

$$\tilde{\mathbf{V}}=arg\underset{\tilde{S}}{min}E\left(S\right)=\sum _{m=1}^M{\left|I_m-{\tilde{I}}_m(x,y;S)\right|}^2$$

(4)

At iteration t, the incremental 3-D vertex displacement field ${\tilde{\mathbf{V}}}^t$ (to revise the 3-D model ${\tilde{S}}^t$) is estimated from the 2-D motion fields ${\mathbf{v}}_m^t$ that align the data and synthetic images at “relevant” sonar poses ${\mathbf{m}}_m$ ($m=1,2,\dots M^{-}\le M$). Here, $(M-M^{-})$ views may be excluded if the vertices/patches are invisible or the 2-D motion vectors have large errors.

For targets imaged near the sea surface, we need to account for the impact of multipath propagation, namely, the object region corrupted by the ghost component. To this end, multipath modeling serves two purposes: (1) to localize and discard the ghost-corrupted object region to avoid the distortion of the 3-D model; and (2) to exploit the geometric constraints imposed by the mirror image boundaries in relevant views to improve the 3-D object modeling.It is noted that the multipath arising from reverberation within object concavities and ground reflection for objects on the seabed have been incorporated in sonar simulators [26,27,34,35,36,41,42,43]. However, these investigations offer no model validation with real data and calibrated objects. In our work, we rely solely on geometric cues, i.e., the ghost and mirror locations and the mirror contour, for 3-D model refinement.

2.3. Unified Optimization Famework

Inexpensive depth sensors with an accuracy of 0.1% have negligible error at depths of a few meters [32]. However, a low-cost angle sensor with an uncertainty of $\pm 0.6°$ [33] can introduce large biases in localizing the ghost, mirror, and thus the uncorrupted object regions. We next describe a unified optimization scheme by incorporating the estimation of the two pose parameters within the 3-D target modeling framework.

Figure 3a depicts the block diagram of the process: recursively updating the pose parameters and the 3-D shape. Here, we employ the 3-D object model estimate to calculate the new values of sonar depth d and tilt angle $\beta$. These are adopted only if they yield a smaller error, subsequently employed in the next iteration of 3-D shape refinement. The iterative updating of the 3-D target model, detailed in [19,20] and depicted in Figure 3b, consists of three key steps: (1) computation of 2-D motion vectors in various views for the alignment of object regions in the data and synthetic images (constructed from the 3-D model); (2) estimation of the 3-D motions of corresponding triangular-patch centers from the 2-D image motion fields for all views; (3) transformation of 3-D motions from the patch centers to the 3-D model vertices. Finally, we update the 3-D shape by displacing the model vertices. The execution of the two-mode optimization continues as long as the objective error function diminishes.

2.4. Sonar Pose Optimization

Referring to Figure 2d, the ghost image, generally overlapping with the more distant region of the object image, is formed by the sound propagation along the longer multipath routes. The paths for all visible surface patches are readily calculated based on the sonar pose relative to the target and to the air–water interface [19,20]. However, inaccuracies in the initial 3-D SC model (note that the 3-D model incorporates the relative sonar-to-target pose) and the depth and tilt sensor measurements can lead to the imprecise location of the ghost region, thus diminishing the effectiveness of utilizing the image intensity and contour information for various views. We minimize both types of error through the optimization method described next.

We employ superscript ${(·)}^O$ and ${(·)}^M$ for entities associated with the object and mirror components, respectively; e.g., 3-D models $S^O$ and $S^M$ explicitly differentiate between the true model and the virtual 3-D mirror object, respectively. For a sonar pose ${\mathbf{m}}_m$ ($m=1,2,\dots ,M)$, a frontal patch refers to the visible surface patch at shortest range along each sonar beam, projecting onto a frontal contour point in the corresponding view.

Let ${\mathcal{C}}_m^O=\left\{c_{mk}^O\right|k=1,2,\dots ,N_m^O\}$ and ${\mathcal{C}}_m^M=\left\{c_{ml}^M\right|l=1,2,\dots ,N_m^M\}$ denote the sets of frontal contour points of the object and mirror regions in image $I_m$. Likewise, ${\tilde{\mathcal{C}}}_m^O=\left\{{\tilde{c}}_{mi}^O\right|i=1,2,\dots ,{\tilde{N}}_m^O\}$ and ${\tilde{\mathcal{C}}}_m^M=\left\{{\tilde{c}}_{mj}^M\right|j=1,2,\dots ,{\tilde{N}}_m^M\}$ denote the same in the synthetic image ${\tilde{I}}_m$. Note that $N_m^O\ne {\tilde{N}}_m^O$ and $N_m^M\ne {\tilde{N}}_m^M$ generally, for real quantized data. We perform contour alignment by the Iterative Closest Point (ICP) algorithm [44,45], namely, the iteratively reweighted least squares (IRLS) variant [46,47]. A key advantage of the ICP methods is the applicability in registering two unequal-size unmatched point sets.

Figure 4 depicts these contours superimposed on a sample real image. The registration of contours ${\tilde{\mathcal{C}}}_m^O$ and ${\mathcal{C}}_m^O$ with ${\tilde{N}}_m^{\prime O}\le {\tilde{N}}_m^O$ inlier matching points $\{{\widetilde{c^{\prime }}}_{mi}^O,c_{mk}^O\}$ (with indeces $\{i,k(i\left)\right\}$) yields the planar transformation ${\widetilde{{\mathcal{C}}^{\prime }}}_m^O=T^O\left({\tilde{\mathcal{C}}}_m^O\right)$ (${\tilde{\mathcal{C}}}_m^{\prime O}=\left\{{\tilde{c}}_{mi}^{\prime O}\right|i=1,2,\dots {\tilde{N}}_m^{\prime O}\}$) ($T^O(.)$ involves 2-D translation and in-plane rotation).

The 2-D motion of contour ${\tilde{\mathcal{C}}}_m^O$ is given by ${\tilde{\mathcal{V}}}_m^O=\left\{{\tilde{\mathbf{v}}}_{mi}^O\right|i=1,2,\dots ,{\tilde{N}}_m^{\prime O}\}$, where ${\tilde{\mathbf{v}}}_{mi}^O={\tilde{c}}_{mi}^{\prime O}-{\tilde{c}}_{mi}^O$.

For each synthetic image ${\tilde{I}}_m$, the contour error is computed based on the discrepancy between the contour pixels $c_{mk\left(i\right)}^O$ in the data $I_m$ and the transformed contour pixels ${\tilde{c}}_{mi}^{\prime O}$ in the synthetic view ${\tilde{I}}_m^t$:

$$\mathrm{CE}\left({\tilde{I}}_m\right)={\displaystyle \frac{1}{{\tilde{N}}_m^{\prime O}}}\sum _{i=1}^{{\tilde{N}}_m^{\prime O}}\left|c_{mk\left(i\right)}^O-{\tilde{c}}_{mi}^{\prime O}\right|$$

(5)

Following the same steps for the mirror image, ${\tilde{N}}_m^{\prime M}$ pairs $\{{\widetilde{c^{\prime }}}_{mj}^M,c_{ml}^M\}$ of inlier mirror-contour correspondences with indeces $\{j,l(j\left)\right\}$ are established by the transformation ${\widetilde{{\mathcal{C}}^{\prime }}}_m^M=T^M\left({\tilde{\mathcal{C}}}_m^M\right)$. This yields the 2-D mirror-contour motions ${\tilde{\mathcal{V}}}_m^M=\left\{{\tilde{\mathbf{v}}}_{mj}^M\right|j=1,2,...,{\tilde{N}}_m^{\prime M}\le {\tilde{N}}_m^M\}$, where ${\tilde{\mathbf{v}}}_{mj}^M={\tilde{c}}_{mj}^{\prime M}-{\tilde{c}}_{mj}^M$.

With accurate sonar pose parameters D and $\beta$, the 2-D motions ${\tilde{\mathcal{V}}}_m^O$ and ${\tilde{\mathcal{V}}}_m^M$ computed from images $I_m$ and ${\tilde{I}}_m$ (for pose ${\mathbf{m}}_m$) incorporate the inaccuracies of estimated 3-D models ${\tilde{S}}^O$ and ${\tilde{S}}^M$ in a consistent manner.

To explain, we utilize the simplified notation $\left\{z\right|z=1,2,\dots ,Z_m\}$ for the set of $Z_m$ frontal patch indeces in view ${\mathbf{m}}_m$ for both the true and estimated object-mirror pairs $\{S^O,S^M\}$ and $\{{\tilde{S}}^O,{\tilde{S}}^M\}$, respectively. Moreover, the 2-D motion vectors ${\mathbf{v}}_{mz}^O$ and ${\mathbf{v}}_{mz}^M$ align the frontal contours of objects and mirror regions in the real and synthetic images, respectively. Finally, the erroneous 2-D motion vectors based on imperfect knowledge of sonar depth and tilt angle are denoted ${\tilde{\mathbf{v}}}_{mz}^O$ and ${\tilde{\mathbf{v}}}_{mz}^M$.

Next, let the pairs $\{{{\mathbf{P}}_z^c}^O,{\widetilde{{\mathbf{P}}_z^c}}^O\}$ represent the true and estimated 3-D frontal patch centers on $\{S^O,{\tilde{S}}^O\}$, projecting onto matching 2-D object-contour points $\{c_{mz}^O,{\tilde{c}}_{mz}^O\}$ in the data $I_m$ and synthetic view ${\tilde{I}}_m$, respectively. In [19,20], we show how to calculate the 3-D motion of the patch center ${\tilde{\mathbf{V}}}_z^O={\widetilde{{\mathbf{P}}_z^c}}^O-{{\mathbf{P}}_z^c}^O$ from the redundant 2-D object-contour motions ${\tilde{\mathbf{v}}}_{mz}^O$ in visible views $m=1,2,\dots ,M^{-}\le M$.

In the same manner, (1) the patch-center pair $\{{{\mathbf{P}}_z^c}^M,{\widetilde{{\mathbf{P}}_z^c}}^M\}$ of true and estimated 3-D mirror model pair $\{S^M,{\tilde{S}}^M\}$ project onto matching 2-D mirror-contour pairs $\{c_{mz}^M,{\tilde{c}}_{mz}^M\}$ in the data $I_m$ and synthetic view ${\tilde{I}}_m$, respectively; and (2) the 3-D mirror patch-center motion ${\tilde{\mathbf{V}}}_z^M={\widetilde{{\mathbf{P}}_z^c}}^M-{{\mathbf{P}}_z^c}^M$ is calculated from the redundant 2-D mirror-contour motions ${\tilde{\mathbf{v}}}_{mz}^M$ over the visible mirror patches.

The consistent manner means that mapping of the pair $\{{{\mathbf{P}}_z^c}^O,{\widetilde{{\mathbf{P}}_z^c}}^O\}$ to two corresponding 3-D mirror pairs on $\{S^M,{\tilde{S}}^M\}$ yields the same terminal points $\{{{\mathbf{P}}_z^c}^M,{\widetilde{{\mathbf{P}}_z^c}}^M\}$ that are derived from the 2-D mirror-contour motions ${\tilde{\mathbf{v}}}_{mz}^M$ (calculated from all views where the mirror surface patch centered at ${\widetilde{{\mathbf{P}}_z^c}}^M$ is visible).

This so-called consistency does not hold when sonar pose parameters are imprecise: deviations arise in ${\tilde{\mathbf{V}}}_z^O$ and ${\tilde{\mathbf{V}}}_z^M$ due to the inaccurate 2-D motions calculated from the dislocated object and mirror contours. First, we expect the 3-D motion vector ${\tilde{\mathbf{V}}}_z^O$ to be larger than the estimate based on precise sonar depth and tilt measurements. Accordingly, we would seek sonar pose parameters that minimize $\parallel {\tilde{\mathbf{V}}}_z^O\parallel$. Next, inconsistency exists between ${\tilde{\mathbf{V}}}_z^M$, calculated from the 2-D mirror contour motions, and ${\mathbf{V}}_z^M$ based on the mapping of 3-D patch-center motion from the object ${\tilde{S}}^O$ to the virtual mirror ${\tilde{S}}^M$. We can directly calculate the discrepancy $\parallel {\mathbf{V}}_z^M-{\tilde{\mathbf{V}}}_z^M\parallel$, which is also to be minimized by the optimal sonar pose parameters. Collectively, we define and minimize the error metric $E_{\mu }(D,\beta )$ that integrates the errors ${\mu }^m$ for all sonar poses:

$${\mu }^m={\displaystyle \frac{1}{Z_m}}\sum _{z=1}^{Z_m}\parallel {\mathbf{V}}_z^O\parallel +\parallel {\mathbf{V}}_z^M-{\tilde{\mathbf{V}}}_z^M\parallel ;E_{\mu }(D,\beta )={\displaystyle \frac{1}{M}}\sum _{m=1}^M{\mu }^m$$

(6)

The error metric $E_{\mu }(D,\beta )$ is a highly nonlinear function of parameters D and $\beta$, requiring a computationally expensive non-linear optimization scheme (e.g., some variation of iterative gradient descent [48]). Moreover, the convergence to the global minimum cannot be guaranteed. Instead, we determine the optimal values $D^{\ast }$ and ${\beta }^{\ast }$ more efficiently by multi-resolution grid search in a 2-D parameter space. That is, representing the uncertainty bounds of sensor measurements $D_0$ (of depth) and ${\beta }_0$ (of tilt angle) by $\pm e_D$ and $\pm e_{\beta }$, we calculate $E_{\mu }(D,\beta )$ over the 2-D grid ${\mathcal{U}}_g=[D_o-e_D,D_o+e_D;{\beta }_o-e_{\beta },{\beta }_o+e_{\beta }]$ with (low to high) grid resolution $\delta D$ and $\delta \beta$:

$$(D^{\ast },{\beta }^{\ast })=arg\underset{(D,\beta )\in {\mathcal{U}}_g}{min}{E}_{\mu }(D,\beta )$$

(7)

We update D and $\beta$ in a “greedy” manner: $(D^{\ast },{\beta }^{\ast })$ is accepted if it gives a smaller error $E_{\mu }$ compared with the latest value; see the “Sonar Orientation Optimization” block in Figure 3a. Next, we carry out the computational steps in the “3-D Model Optimization” block as described in [20].

2.5. Error Metric

We define two error measures to assess the performance of our method [19,20]: to quantify the 3-D object model error and the discrepancy between the data and synthetic images generated from the 3-D model. Denoting ${\tilde{\mathcal{I}}}^t=\left\{{\tilde{I}}_m^t\right|m=1,2,\dots ,M\}$ the set of synthetic beam-bin images at iteration t, we employ the average intensity error (IE) comprising of pixel set ${\mathrm{\Omega }}_m^t$ within the uncorrupted object region of ${\tilde{I}}_m^t$, in calculating the average intensity error (AIE) over the whole synthetic image set:

$$\mathrm{IE}\left({\tilde{I}}_m^t\right)={\displaystyle \frac{{\displaystyle 1}}{{\displaystyle |{\mathrm{\Omega }}_m^t|}}}\sum _{(b,B)\in {\mathrm{\Omega }}_m^t}\left|{\tilde{I}}_m^t(b,B)-I^t(b,B)\right|\mathrm{AIE}\left({\tilde{\mathcal{I}}}^t\right)={\displaystyle \frac{{\displaystyle 1}}{{\displaystyle M}}}\sum _{m=1}^M\mathrm{IE}\left({\tilde{I}}_m\right)$$

(8)

where $|{\mathrm{\Omega }}_m^t|$ denotes the cardinality of set ${\mathrm{\Omega }}_m^t$. We also compute the average contour error (ACE) from (5):

$$\mathrm{ACE}\left({\tilde{\mathcal{I}}}^t\right)={\displaystyle \frac{{\displaystyle 1}}{{\displaystyle M}}}\sum _{m=1}^M\mathrm{CE}\left({\tilde{I}}_m^t\right)$$

(9)

Finally, these two average errors (over the entire data set) are normalized with respect to their initial values (using the synthetic images of the 3-D SC solution):

$$NAIE\left({\tilde{I}}^t\right)=AIE\left({\tilde{I}}^t\right)/AIE\left({\tilde{I}}^0\right)NACE\left({\tilde{I}}^t\right)=ACE\left({\tilde{I}}^t\right)/ACE\left({\tilde{I}}^0\right)$$

(10)

and combined to establish the image error at iteration t:

$$E_I\left(t\right)={\displaystyle \frac{{\displaystyle 1}}{{\displaystyle 2}}}(NAIE\left({\tilde{I}}^t\right)+NACE\left({\tilde{I}}^t\right)$$

(11)

The 3-D modeling error can be quantified in terms of the volumes $V^c=V\left(S^c\right)$ and $\tilde{V}=V\left(\tilde{S}\right)$ of the true and estimated 3-D object models. With no knowledge of $S^c$, we instead utilize the model generated by a Kinect camera [49], albeit not perfect. Accordingly, we define the normalized volumetric error at iteration t [39]:

$$E_V^t={\displaystyle \frac{{\tilde{V}}^t+V^c-2V_{{\tilde{S}}^t\cap S^c}}{{\tilde{V}}^t+V^c-V_{{\tilde{S}}^t\cap S^c}}};V_{{\tilde{S}}^t\cap S^c}=V({\tilde{S}}^t\cap S^c)$$

(12)

The metric $0\le E_V\le 1$ is the ratio of non-common volumes of the two 3-D models to their union. Thus, zero error indicates two identical volumes, and the maximum unity error corresponds to no common volume. We utilize $E_V$ primarily to determine the correlation with $E_I$ and whether $E_I$ (which is computable) can serve as an indicator of $E_V$ (which we cannot determine in practice).

2.6. Enhanced Target Images

As discussed, almost all object views are generally corrupted by the ghost image, and only some by the mirror image component. For any arbitrary sonar pose, we can synthesize a noise-free, uncorrupted image from the 3-D target model. In addition, it is highly desired to generate enhanced real object views by removing the ghost contributions.

Let $I_o=\cup (I_o^N,I_o^O)$ and $I_g=\cup (I_g^N,I_g^O)$ denote the intensity values of object and ghost components, respectively, formed by the union of their respective overlapping and non-overlapping regions (steps 2a and 2b in Figure 5b,c). Ignoring the impact of noise, $I_o^N$ and $I_g^N$ are measured directly from the image. The object image enhancement involves the generation of $I_o$ by the estimation of $I_o^O$. In the process given below to generate the object image, we assume the transformation to map the synthetic image to the real data within the non-overlap object region can be reliably applied to estimate the intensity values of the object within the overlap region. Referring to Figure 5, the complete process involves the following steps:

• Generation of synthetic logarithmic object image $L_o$ based on the model in [26], applied to the 3-D object model $\tilde{S}$ (step 1);
• Localization of ghost region by employing the estimated sonar pose parameters [19,20] (step 1);
• Segmentation of non-overlapping and overlapping object regions $L_o^N$ from $L_o^O$, respectively (step 2a), and the generation of normalized intensity values ${\widehat{L}}_o^N$ with zero mean and unit variance;
• Calculation of mean-variance pair $\{\overline{I},{\sigma }_I\}$ of intensity values $I_o^N$, and applying the transformation $L_o^N={\sigma }_I{\widehat{L}}_o^N+\overline{I}$;
• Construction of look-up table (LUT) to transform $L_o^N$ to $I_o^N$ by matching their histograms (step 2a);
• Computation of scaled intensity values $L_o^O={\sigma }_I{\widehat{L}}_o^O+\overline{I}$ within overlapping object region, and applying the LUT to map $L_o^O$ values to $I_o^O$ values (step 2a).

In some applications, including the development and verification of a ghost image formation model, it may be desired to isolate the ghost component from the real data. Figure 5c–e depicts the process, once the object image has been computed as described above.

3. Results

Experiments with both synthetic and real images of three targets are presented to demonstrate different aspects of our method. These include two “dominantly-convex” coral rocks with mild local concavities and a concave “toy” wood table; see Figure 6.

The simulations with synthetic data allow us to assess the performance and convergence of the unified optimization scheme, namely, errors in converged sonar depth and tilt angle values, the discrepancy between the optimized 3-D model (initialized by the SC solution) and the Kinect 3-D model, as well as the consistency of results from experiments with synthetic and real data.

For ground truth, we employ the 3-D surface mesh models of these objects generated by a Kinect camera, utilizing Kinect video while slowly rotating the target in air at a distance of about 1 [m]. Although not perfect, various spot measurements confirm an accuracy of better than an order of magnitude in comparison to the average error of 3-D models generated from the 2-D sonar data. The real data for the two coral rocks comprises 16 total images at two opposite, north (N) and south (S) sides of each object. At each position, eight images are captured while the sonar rolls on a motorized rotator about the viewing axis from −90° to 67.5°, in increments of 22.5°; the unrotated (zero-degree roll) at the S position is the reference view. For the wood table, we have a total of 32 images from four sides: east (E) and west (W), in addition to N and S.

We employ these sonar poses and the parameters of each experiment listed in Figure 6 to generate the synthetic images of the 3-D Kinect model by applying the DIDSON image formation model [26]. In assessing convergence, all experiments are initialized with the 3-D SC model [39] and imprecise depth and tilt-angle values.

Finally, the mirror component overlaps with the object region in three views: at the reference pose and two rotated views at roll angles of ±22.5°. Discarding these, the optimization is applied to 10 images of each coral rock and 20 images of the wood table (recall that we require a distinct mirror region boundary in the optimization process). Moreover, we treat the discarded views as the new ones to compare with the computer-generated images.

3.1. Experiments with Synthetic Data

Figure 7a–d depicts the results of the experiment with the coral-one data. The sonar depth and tilt parameters in (a,b) approach the assumed values of 43 [cm] and 10.3° (shown in red dashed lines) in seven iterations. In (d), we compare the initialized SC solution (top: blue mesh) and optimized 3-D model (bottom: blue mesh), superimposed on the Kinect model (black mesh). Decreasing in tandem, the image $E_I$ and volumetric $E_V$ errors in (c) confirm that the reduced image error leads to an improved 3-D model. Thus, the measurable $E_I$ serves as an indicator of incalculable 3-D modeling accuracy. Initialized with the SC solution, the improved target model after seven iterations reduces the volumetric error by about 16% and the image error by better than 20%.

Similar results are obtained for the second coral rock. Referring to Figure 8a–d, the sonar depth in (a) and tilt angle in (b) approach the assumed values of 48 [cm] and 11.5° (red dashed lines), and the image and volumetric errors in (c) improve by better than 20% in four iterations.

A noteworthy observation is that the reconstructed 3-D models of both rocks preserve (some of) the mild local concavities of their surfaces. For example, these are pronounced in the $XY$ and $XZ$ views of coral-one; see columns 2–3 in Figure 7d and Figure 8d.

The experimental results for the concave wood table are presented in Figure 9a–d. Here, the reverberation among the inner surfaces of the four legs leads to their inflated appearances in different views. Minimizing the discrepancy between the data and synthetic images yields a 3-D model with thickened legs. Consequently, the volumetric improvement (with respect to the initial SC solution) in (c) is relatively insignificant, despite an image error improvement of better than 20%. Still, the reconstructed 3-D model captures the structural and topological properties of the shape, including the top surface, four legs, and the deep concavity within these legs.

Moreover, the convergence of the sonar depth in (a) and tilt angle in (b) takes 10 iterations. A notable observation in (c) is the non-monotonic convergence of the image (red circles) and volumetric (blue squares) errors. These errors increase (before resuming the improvement in accuracy) at iterations where the sonar pose parameters diverge from their true values (red dashed lines).

3.2. Experiments with Real Data

Figure 7d’, Figure 8d’, and Figure 9d’ are the 3-D Kinect (red. mesh) and the optimized model based on initialization with the SC solution (blue). The reconstruction accuracy of the optimized model is consistent with those from the computer simulations, albeit at a slower convergence rate; compare (c) and second row in (d) with (c’,d’) in each figure. The same conclusion is drawn from the wood table reconstruction, despite inferior precision relative to the two dominantly convex coral rocks. Interestingly, the convergence is achieved monotonically and faster, directly tied to the same behavior for the sonar pose parameters.

Here again, some of the mild concavities of the two coral rocks are recovered, as in the experiments with the computer-generated data. Moreover, the reconstructed model captures the structural and topological properties of the wood table. The largest discrepancy arises in the coral-two reconstruction over the SW–W and E–NE regions (e.g., see $XY$ projection). Here, the thickness variations are not replicated in the estimated model, mainly due to the unavailability of data from the E and W directions (this has been confirmed in an experiment with real-synthetic mixed data, using additional computer-generated views from these two positions [19,20]).

Next, we present sample results of target image enhancement. For the first coral rock, Figure 10 depicts three samples, out of five rotated views used in the optimization at S (pos 1) and N (pos 2) sonar positions. Columns (a,b) include the original data and the processed ghost-free image, respectively. Columns (c,d) are the synthetic images constructed from the initial SC and optimized 3-D models, respectively. We can readily confirm that the consistency within the object region in the data and synthetic images improves in almost all views after optimization. The same conclusions are drawn from the results for the second coral rock in Figure 11 and the wood table images in Figure 12 (despite the lower 3-D reconstruction accuracy of the optimized solution).

Sample sonar images at zero and ±22.5° roll angles (which are not used in the optimization) are found in Figure 13 and Figure 14. Here, results in the last column can be treated as synthetic views at new arbitrary sonar poses and can be compared with the ghost-free data in column (b) to assess accuracy in the presence of unmodeled environmental factors (of the original data).

4. Conclusions

In this work, we address the reconstruction of a 3-D object model from 2-D FSS images at known poses, with multipath propagation due to sea surface reflectance. To this end, we propose a unified optimization framework to improve the accuracy of the two key sonar pose parameters (measured by low-cost sensors) and the 3-D model. Embedded in the approach to localize and remove the ghost images formed by multipath is the generation of synthetic views from the estimated 3-D object model based on the environmental conditions. This has a number of applications, including the generation of a large volume of training data under a variety of environmental conditions for object recognition. Moreover, a LUT is generated from the distributions of object intensities within the non-overlap region in the real and synthetic images and utilized in generating the object within the ghost-corrupted overlap region.

The targets in three experiments with synthetic and real data are two dominantly convex coral rocks and a concave toy wood table. Assuming the same setup as for the real data, computer simulations with synthetic data highlight the accuracy and convergence in 10 or fewer iterations of the estimated pose parameters and the 3-D target model. Comparable performances are achieved in experiments with real data.

The results of the wood table experiment demonstrate the applicability of our method in constructing shapes with deep concavities, albeit at lower accuracy. Here, the primary complexity is the inability to account for the shape-dependent multipath without a priori knowledge of the target shape. The multipath reverberations from object surfaces within concavities lead to the thickened appearances of object parts. To decouple the direct and multipath components, we require knowledge of the unknown 3-D object model. This poses a chicken-and-egg problem, which is faced by most other methods for 3-D object reconstruction from FSS imagery. Aside from collecting a much larger volume of data from different poses looking into concavities at close ranges, further investigation is needed for devising a suitable mechanism to accurately account for the multipath generated by the surface patches within concavities.