Seeing through Wavy Water–Air Interface: A Restoration Model for Instantaneous Images Distorted by Surface Waves

: Imaging through a wavy water–air interface is challenging since light rays are bent by unknown amounts, leading to complex geometric distortions. Considering the restoration of instantaneous distorted images, this paper proposes an image recovery model via structured light projection. The algorithm is composed of two separate parts. In the first part, an algorithm for the determination of the instantaneous shape of the water surface via structured light projection is developed. Then, we synchronously recover the distorted airborne scene image through reverse ray tracing in the second part. The experimental results show that, compared with the state-of-the-art meth-ods, the proposed method not only can overcome the influence of changes in natural illumination conditions for WAI reconstruction, but also can significantly reduce the distortion and achieve better performance.


Introduction
Viewing an airborne scene through a wavy water-air interface using a submerged camera creates a virtual periscope, which is of great significance in both the military application field and marine biology research [1][2][3][4].Unlike other underwater imaging systems, the difficulties in such imaging scenarios mainly come from the water-air interface (WAI).Water surface fluctuations are complex and random movements, which will lead to irregular geometric distortion and motion blur in the image, causing the distortion of airborne scenes.Removing such distortions from an instantaneous distorted image is challenging since the shape of the interface is not known a priori and must be estimated simultaneously with the real scene image.
In previous works, recovering images distorted by a wavy water surface mainly relied on high-resolution video streams [5][6][7][8][9][10][11][12].These methods always require high complexity and large data sets, so they are difficult to apply to real-time observation scenarios.However, in application scenarios such as path planning and obstacle avoidance for underwater vehicles, aerial target detection, recognition and tracking, etc., recovery methods for instant images are particularly important.
Previous studies show that the distorted scene image can be effectively recovered by reconstructing the shape of the water-air interface [13][14][15][16][17][18][19][20][21][22][23][24][25].Milder et al. [15,16] emphasized first the estimation of the shape of the distorted water surface and then reconstructing the scenes.They proposed a method to estimate the water surface by analyzing the sky brightness; in their image system, an upward-looking submerged camera was used to capture the panoramic above-surface scene.The incident light from above will gradually disappear as the normal surface deviates from the viewing line of sight.Assuming that it was completely dark underwater and the sky was uniform, the brightness of the sky thus determined the surface radial slope, and they used a harmonic wave model to estimate the water surface and inverse ray tracing to reconstruct a distortion-free image.Turalev et al. [17] studied the recovery method of water surface distortion, and then designed an experimental setup.They first utilized multiple illumination sources (red to illuminate the water surface and blue for the underwater object) to capture an image of the object and a glitter pattern of the water surface simultaneously.Then, the slope of the water surface was estimated using the glitter pattern [18].Finally, the geometrical distortion effects of the underwater object image were eliminated by multiple accumulations of short-exposure images [19][20][21][22].Alterman et al. [23] added an additional imaging sensor to measure the wavy water surface in real time.It works as an adaptive optics system but utilizing the sun as a guide star.The wavefront sensor consists of a pinhole array imaging the sun onto a diffuser plane directly behind it and a camera capturing the distribution of images of the sun on the diffuser plane.These pinholes have an extremely narrow field of view, so the water surface on their own line of sight is small enough to be assumed isoplanatic.Then, sampled normal vectors of the water surface were deduced from the position of each sun image through pinholes and used to estimate the water surface.Yoav Y. Schechner et al. [24] considered the problem of multi-view stereo through a dynamic refractive interface.They used multiple cameras along a wide baseline to observe a scene under uncorrelated distortions and recover sparse point clouds.R. H. Gardashov et al. [25] developed a method for recovering the single instantaneous images of underwater objects distorted by surface waves.They first determined an instantaneous shape of the wavy sea surface using the characteristics of the sun glints, and then corrected an image distorted by the wavy surface through reverse projection.However, it was only applicable to scenarios of monitoring from the sky, since there were no characteristics of sun glints when viewing the airborne scenes from underwater [15,23].The above methods all attempted to first estimate the slope distribution of the water surface and then recovered the distorted images.However, several current methods of recovery of the water surface are not suitable for harsh application conditions, requiring special illumination settings or relying on special ambient illumination conditions (requiring the sun or uniform brightness in the sky).Furthermore, the accuracy of the estimation is also unable to meet the requirements.
In order to overcome the dependence of previous methods on natural illumination conditions and realize the distortion correction of instantaneously distorted images, an image restoration model based on structured light projection is proposed in this paper.Compared to previous approaches, our method does not require special natural illumination [15][16][17]23,25], multiple viewpoints [24] or a complex experimental setup [17,23].It only requires a simple projection setup and an image of the distorted scene.The main contributions of this paper are as follows: (1) we propose a new image restoration model for instant images via structured light projection, (2) we introduce a WAI reconstruction algorithm based on structured light, (3) we analyze some of its limitations.

Snell's Window
By imaging through the water-air interface using a submerged camera, one can observe the whole sky.However, it does not stretch 180  from horizon to horizon, as it does above water.Instead, it is compressed into a circular area spanning approximately 97.2  , regardless of the observer's depth.According to Snell's Law [26,27], this occurs because light rays are bent when entering or exiting water.The shrunken sky (celestial hemisphere) seen by submerged observers is called Snell's window (SW).SW is surrounded by a dark field that represents light that is entirely internally reflected from the sea and back to the observer from the underside of the water's surface.
As shown in Figure 1, a submerged camera observes from underwater at 3D location C .CM and CN are the total reflection boundaries, MCN = 97.2∠  .The conical area formed by the rotation of CM and CN around the optical axis is the Snell cone, and the circular area on the water surface between them is Snell's window, the boundary of which is the extinction boundary, and the outside is the dark field.Assuming that the brightness value of the hemisphere space above the water surface is 1, the brightness value of the underwater scene is 0, and the energy loss of absorption and scattering by the water body is ignored.Figure 2 shows the normalized illuminance as a function of the height angle of incident light rays (blue) and the normalized illuminance as a function of the elevation angle of the field of view of the camera (red) when the water surface is still.According to Figure 2, it does not receive the irradiance from the sky at all when the elevation angle of the camera water (0 , 41.4 ) (138.6 ,180 ) . This occurs because, within this field of view, light rays are totally internally reflected from the sea and back to the observer from the underside of the water's surface.In deep water, there is very little light coming from below and so this part is dark and shows no apparent color or structure.Therefore, the field of view of the underwater camera is constricted within the Snell window within water (41.4 ,138.6 ) θ ∈   .

Optical Properties of Sea Water
The energy attenuation of light in water is mainly caused by the absorption attenuation of the water body and the scattering effect of suspended particles in the water.Studies have shown [28][29][30] that the transmission of light in water is caused by two independent physical processes, namely absorption and scattering, and the energy decays exponentially.According to the Beer-Lambert law, the radiance attenuation of monochromatic light can be expressed as C. Smith et al. [31] performed a detailed measurement of the attenuation coefficient of light in clear seawater and gave the measurement data of the absorption attenuation coefficient and scattering coefficient of seawater as a function of wavelength.The absorption and scattering attenuation coefficients of seawater as a function of wavelength are shown in Figure 3.The results show that seawater displays the selective absorption of light of different wavelengths, in addition to the existence of scattering properties.As shown in Figure 3, it is not difficult to find that the transmittance of seawater in the bluegreen band of the spectrum is relatively large, and the light energy attenuation is the smallest, which is called the "blue-green window".Therefore, in structured light projection, a light source in this band can be selected for wavefront sampling to reduce the influence of light absorption and scattering on the WAI reconstruction.

Model Descriptions
The system model is shown in Figure 4, in which the projector first projects an adaptive and adjustable structured light pattern onto the water surface, and then the camera s acquires the distorted structured light image from the diffuser plane, while the camera v captures the airborne scene through the same WAI.The algorithm is composed of two separate parts.In the first part, an algorithm for the determination of the instantaneous shape of the water surface from structured light is developed.Then, we synchronously recover the distorted airborne scene image through reverse ray tracing in the second part.

WAI Reconstruction Algorithm Based on Finite Difference
In this section, we propose an algorithm for the determination of the instantaneous shape of the water surface from structured light.According to the law of reflection, we first calculate the WAI normals of sampled points using the location information of the feature points between the reference structured light image and the distorted structured light image.Then, the WAI shape is estimated utilizing the finite difference method.

Sampling of WAI Normals
The algorithm first takes advantage of the position information of the feature points of the structured light pattern to perform the quasi-periodic sampling of the wave surface to be measured.Figure 5 shows a simulation example of WAI sampling via structured light projection.Figure 5a is the preset structured light pattern, and Figure 5b is the projection on the water-air interface when the water surface is flat (note: in the daytime or moonlit night, no projection is formed on the water-air interface; a virtual image is introduced for the convenience of analysis).Given the system parameters, the location distribution of the WAI sampling points can be obtained using perspective projection transformation [32][33][34], as shown in Figure 5c.The structured light pattern of blue-green stripes projected by the projector is reflected by the water surface, forming a distorted structured light image on the diffuser plane.This section solves the sampled normals of the WAI according to the relationship between the incident ray, the reflected ray and the normal vector.
As shown in Figure 6, the global coordinate system is established with the projected  is periodic when the WAI is still.Meanwhile, { } k q is quasi-periodic, having a perturbation to periodicity, when the WAI is wavy.Considering that the variations in the height of the water surface are small compared to the work depth of the system ( ), we have where 0 h is the system height, which is the average underwater depth of the projector.
The value can be determined in the field, using a pressure-based depth gauge.
p k c is the propagation length of p ˆk v at the z-axis.Moreover, the 3D location of the spot k s can be extracted from the distorted structured light image.We adopt the corner detection algorithm of Reference [35] for feature extraction and matching.Therefore, the vector of re- s q s q .Using the vector form of the law of reflection at the water interface, Here, × is the cross product.Using the axial components of the vectors p ˆk v , ˆk v and ˆk N , Equation ( 4) can be converted into the dot product form, x N , y N , z N are the axial components of the vector ˆk N .Equation ( 5) also can be phrased in matrix form where The WAI normal ˆk N is estimated by solving Equation ( 6): it is the null subspace of A .This process is repeated for each sampled point located at k q .It yields a set of sam- 3.2.2.Reconstruction of the WAI Assume that ( , ) h x y is the height of an arbitrary sampled point , ) x y ( of the water surface, and Z is a 2D height field.The gradient of Z can be given by x y h x y h x y x y x y x y where ( , ) x x y Z , ( , ) y x y Z are the x and y components of the two-dimensional numeri- cal gradient of the arbitrary sampled point ( , )  x y , respectively.î and ˆj are the unit vectors in the x , y axis directions, respectively.Therefore, we can obtain the first-order partial differential equation as follows: ( , ) = ( , ) ( , ) = ( , ) The sampled normals { } ˆk N corresponding to the sampled points { } k q , estimated in Section 3.2.1, are known.Moreover, the normal vector of an arbitrary sampled point on the wave surface can be expressed as , 1 ( , ) ( , ) For the gradient operator of the 2D discrete function ( , ) h x y , according to the finite difference theory [36], we use the central difference formula to approximate the first derivative.Therefore, Equation ( 9) becomes

h x y h x y h x y j m y y y y y y y y y
where m , n represent the dimensions of the grid of sampled points { } ˆk N .Noting that H is the vector of length mn representing the height field ( , ) h x y sampled on a m n × grid, the two vectors of length mn representing each component of the gradient field can be written as where x G and y G are two sparse matrices, of size mn mn × , defining the linear combi- nations of the elements of H to produce each gradient.Equation ( 11) can be merged into a single linear system,

T [ , ]
x y where T

[ , ]
x y = G G G is a rectangular sparse matrix of size 2mn mn × , and Ξ is a vector of length 2mn .This system thus gives 2mn equations with mn unknowns.It is over-de- termined, so a direct inversion is not possible.However, an estimate of H may be obtained, by minimizing the residual [37], where  represents the Euclidean norm.The WAI shape estimated by the numerical integration of the inverse gradient operator is sparse and discontinuous, whereas the WAI is typically smooth and integrable.Thus, we further perform the bicubic interpolation algorithm [38] for H to estimate the WAI shape.

Image Restoration Algorithm through Ray Tracing
Component V of the imaging sensor views the airborne scene through the wavy WAI, as shown in Figure 4.As the shape of the WAI is known, according to the principle of 3D camera imaging, this section proposes an image restoration algorithm based on inverse ray tracing.The principle of the algorithm is shown in Figure 7. length of camera v is f .In this coordinate frame, the 3D location of the pixel u is where u is the 2D location of the pixel on the image plane, and T represents transposi- tion.Relative to the global coordinate system, the pose of camera v is defined by a rotation matrix R and a translation vector t .In the global coordinate system, the 3D location of the pixel u can be expressed as Let cam = u 0 in Equation (15), and the origin of camera v in the global coordinate system is In the global system, according to inverse ray tracing [31], the back-projected ray where the ray direction vector is where l denotes the propagation length along the ray.In a perspective system, the in- verse projected ray intersects the water-air interface at w ( ) WAI ( ) where the WAI estimated in Section 3.2 is known.The WAI normal vector is N .According to the vector form of Snell's law, the direction vector of the airborne observing ray is given by where n is the refractive index of water， 4 / 3 n = .ˆa v is the undeflected direction vector pointing to the object.Supposing that the airborne rays { } a v of all of the pixels in the distorted image intersect with a plane in the air object ∏ , we can easily recover the distorted image through perspective projection.

Limitations
The limitations of system component V are described in detail in Ref. [23].Hence, we focus here on the limitations, sensitivities and resolution trade-off concerning only the structured light projection system S .Primarily, S is not limited by natural illumination conditions.It is suitable for daytime as well as moonlit nights.Other sensitivities and limitations of S are geometric, as analyzed next.

Sensitivity to Variations in N for Structured Light
The WAI normal is perturbed around the z axis.The direction vector of the reflected ray v is obtained by the law of reflection, where p v is the direction vector of the projected ray from the projected center through an arbitrary feature point.Substituting ˆ= − N z into Equation ( 21) yields the direction vector of the reflected ray, when the WAI is flat, ˆ′ v .When the water is wavy, the reflected angle changes by ˆârccos ( ) The perturbation has two principal components.The first principal component is the meridional plane component, which is in the xoz plane.In this component, the WAI nor- mal N rotates around the y axis.Here, we analyze the sensitivity of structured light to changes in N by taking the projected ray along the optical axis of the system as an example.Assuming that pro θ is the elevation angle of the projector, and the unit vector where θ is the inclination angle of the WAI.Another perturbation component is the sag- ittal plane, which resides in the yoz plane.In this component, N rotates around the x axis; then, Substituting Equations ( 23) or (24) into Equations ( 21) and ( 22

Resolution Analysis
In this paper, we perform quasi-periodic sampling for the WAI, utilizing the position information of the feature points of the structured light pattern, while the neighboring sampling interval is approximately equal to D (Figure 9).Reducing D enables the recovery of shorter WAI wavelengths.However, as we describe below, a shorter D decreases the angular resolution of N .The angular resolution of N increases with z h .
Here, we further analyze the relationship between the angular resolution of N with the interval D and z h in the meridional plane, based on geometric optics.
Assume that the x coordinate of the sampled point k q is k h , while . The projected ray ˆk v reflects by the WAI, and then irradiates a spot on the diffuser plane, at location k s .The reflected angle can be given by w f l a t ˆˆˆârccos ( ) arccos ( where k ψ is the angular deviation of the reflected ray (relative to the angle in flat water, where 0 h is the average underwater depth of the projector.As the projector follows a central projection, the reflected angles of the adjacent sampling points k h and Suppose that the projected ray corresponding to 1 k h + is parallel to the pro- jected ray ˆk v corresponding to k h , and the corresponding location of the spot Actually, the location of the spot 1 k s + on the diffuser plane corresponding to according to the central projection, is given by From Equations ( 29) and ( 30), we obtain Given p Δ , z h , 0 h and θ Δ , when the WAI is flat, it is easy to establish the corre- spondence of k h with k s , since . To maintain correct correspondence, this relation has to be satisfied also when the WAI is wavy, k ∀ : Equation ( 33) includes the small margin / 2 p Δ , which guarantees that the adjacent projected spots k s , 1 k s + do not merge into a single spot.From Equations ( 25), ( 26), (32) and (33), Therefore, when 0 h h z − is too large or too small, D limits the dynamic range of the WAI slope changes [ ] ; beyond this range, it will cause a spot detection error.
Assume that the WAI has spatial period λ , in which the WAI inclination varies as , and 0 Φ > .In the worst case, at which is minimal, the WAI angle changes maximally between samples: To avoid correspondence errors, the sampled interval D must satisfy As described in Section 3.2, the WAI sampling is quasi-periodic when the WAI is wavy, while the sampling tends to be periodic whose period is D around when the perturbation amplitude of the water surface becomes smaller.According to the Nyquist sampling criterion ( / 2) D λ > , combining it with Equation ( 35) yields where θ Δ Φ represents the relative angular resolution in which the WAI can be recovered: the ratio of uncertainty to dynamic range.Since 2 8 θ Δ Φ  , we obtain an uncertainty principle As Equation (37) shows, the relative WAI slope angular resolution can be traded off for the spatial resolution ( ) λ of the WAI, before errors stemming from aliasing and corre- spondence take effect.

Results
In the experiment, the proposed method was implemented in the MATLAB environment (MathWorks Co., Natick, MA, USA).This section first demonstrates the process of the WAI reconstruction and the image restoration.To verify the performance of the proposed method, we make a comparison with the state-of-the-art method, namely Alterman's method [23], whose application scenarios and scopes are similar to those of our method.We tested the two methods with the same data set.The source codes and test data are available online in [39].

System Parameters
In this paper, we coded our image recovery scheme described above, which can simulate the process of WAI reconstruction and image restoration according to the system parameters.The system parameters including the projector, camera v for observing airborne scenes and camera s for capturing the structured light image are shown in Table 1.
Moreover, the following parameters are included: Table 1.System parameters of the projector, camera v and camera s.

System Parameters
Projector Camera v Camera s CCD/LCD size

WAI Simulation
The motion of ocean waves is a complex random process.Using a spectrum to describe ocean waves is one of the most effective means to study ocean waves, since the modeling process of the spectrum is based on a large number of actual observation data [40][41][42][43].According to the Longuet-Higgins model [40], the height distribution of ocean waves can be expressed as where ( , , )  x y t η represents the height distribution of a point ( , )  x y on the WAI at time t , ij a is the amplitude of each harmonic, i ω is the harmonic frequency, i k is the wave number of the harmonic, j θ is the azimuth of the harmonic, ij ε is the initial phase of the harmonic, and M , N represent the sampling number of the frequency range of the wave spectrum and the sampling number of the azimuth angle, respectively.
The amplitude of the harmonics ij a can be expressed in the wave spectrum as where ( , )  S ω θ is the direction spectrum.( ) S ω and ( ) ϕ θ denote the spectrum and directional distribution functions of ocean waves, respectively.ω Δ and θ Δ are the sam- pling interval for frequency and direction angle, respectively.According to the linear wave theory [44], i k and i ω meet the dispersion equation where g is the acceleration of gravity.
There are various versions of the directional spectrum.We use the P-M spectrum and the directional distribution function suggested by ITTC (International Towing Tank Conference), whose expression is as follows [45]: where U is the average wind speed at a height of 19.5 m above the sea surface.In Figure 10, we show the simulated ocean wave when 1.0 / U m s = , 10 t s = .

Reconstruction of the WAI
An example of a full-system computer simulation is shown in Figure 11.A distorted structured light image was captured by camera s from the diffuser plane.The structured pattern projected by the projector is reflected by the WAI (Figure 10) and then forms the distorted structured light on the diffuser plane.Using the locations of feature points of the distorted structured light image and reference image, the WAI shape is estimated based on the finite difference method.Intuitively, the reconstructed WAI is similar to the ground truth, but it exhibits a bias, as shown in Figure 11f.We use the root mean square error (RMSE) as the objective evaluation index.RMSE is given by From Equation ( 44), the value of RMSE is 1.7845 mm.The absolute error distribution between the reconstructed WAI and the ground truth is shown in Figure 12.The unrecoverable bias is explained in Section 3.2.1.The research results show that the algorithm can reconstruct the wave surface when the error is allowed.

Comparative Analysis with Alterman's Method
Alterman's method [23] has strict requirements for natural illumination, is not suitable for cloudy weather and requires sunlight.Moreover, the accuracy of WAI reconstruction varies with the position of the sun.Here, we adopt Alterman's method and our algorithm to reconstruct the same WAI (Figure 10), respectively.The relationship between the absolute error distribution of the reconstructed WAI and the ground truth with the incident angle of the sunray sun θ using Alterman's method is shown in Figure 13. Figure 11 shows the RMSE and the maximum absolute error ( max Ae ) as a function of the zenith angle of the sunray using Alterman's method.Combining Figures 13 and 14, it is obvious that the restoration accuracy is limited by the illumination conditions.As the zenith angle of the sunray increases, the maximum absolute error and the stability of the system decrease.Furthermore, as shown in Figure 14, the minimum mean square error of Alterman's method is 3.4603 mm, while the RMSE of our algorithm is 1.7845 mm.The research results show that, compared with the model included in Alterman's method, the proposed algorithm can overcome the influence of changes in natural illumination conditions for WAI reconstruction and improve the accuracy of WAI restoration.

Image Restoration
The submerged camera images a checkerboard located at height

Image Quality Metrics
We use three standard image quality/similarity metrics for quantitative evaluation: (1) mean square error (MSE) [2], (2) peak signal-to-noise ratio (PSNR) [46], (3) structural similarity index (SSIM) [47], in which the expressions of MSE, PSNR and SSIM are, respectively, [ ] where I is the ground-truth image.max( )  denotes the maximum possible value.F u and I u represent the means of the image F and the image I , respectively.

Results of Quantitative Analysis
Figure 16 shows the results of the two methods to recover the distorted image, and the comparison results are presented in Table 2.The results show that compared with Alterman's method, the proposed method has a significant improvement in performance indicators such as PSNR, MSE and SSIM, which proves the effectiveness of the algorithm.

Table 2.
Comparison of image quality metrics among the proposed method, the distorted image and Alterman's method [23].

Conclusions
The image restoration model via structured light projection is a novel approach for virtual periscopes.Different from previous methods, we do not require special natural illumination [15,16,23,25], multiple viewpoints [24] or image accumulation processes [17][18][19][20][21][22]; we only require a simple projection setup and an image of the distorted scene.This means that our method can be applicable to more scenarios-for instance, monitoring the habits of seabirds, path planning and obstacle avoidance for underwater vehicles, airborne target detection, recognition and tracking and seafloor mapping, etc.
In Section 5, we coded our image recovering scheme described above, which can simulate the process of WAI reconstruction and image restoration according to the system parameters.Compared with the state-of-the-art method, our method can overcome the influence of changes in natural illumination conditions for WAI reconstruction and improve the accuracy of WAI restoration.Furthermore, the results show that our method can significantly reduce the distortion and performs better in the recovery of distortions.
In the future, we plan to conduct further tests in the laboratory and at sea to verify the effectiveness of our approach.
Similar to other instantaneous distorted image restoration algorithms [11,15,17,23,25], the proposed method still cannot eliminate the problem of loss of image details caused by random refraction of the WAI (marked in red in Figure 16).However, this problem can later be handled by image fusion algorithms [48][49][50][51] after images are corrected by our process.Rather than addressing the full-blown distortions in raw images, such video post-processing methods may handle more easily images whose distortions are residual.
In Section 4, the limitation of the algorithm is analyzed, and it is found that errors stemming from aliasing and correspondence will occur in structured light when the spatial frequency of the WAI is large.In the future, it is necessary to further explore a solution to this problem, such as structured light encoding [52], optical flow method [53], etc.

Figure 2 .
Figure 2. Normalized illuminance distribution of Snell's window for flat water.
intensity of a radiation of wavelength i λ with the starting intensity 0 I after traveling the distance d through a material with the attenuation coefficient i of the energy losses due to the absorption coef- ficient ( ) i a λ and the scattering coefficient ( ) i b λ :

Figure 3 .
Figure 3. Absorption and scattering coefficients of light in pure ocean.

Figure 4 .
Figure 4. Geometry of the image restoration model via structured light projection, comprising a structured light projection system S and an observing system V , where component S includes a projector, diffuser plane and camera.

Figure 5 .
Figure 5. Examples of WAI sampling via structured light projection: (a) structured light pattern on the display element of the projector; (b) the virtual image formed on the water surface via structured light projection when the WAI is flat; (c) sampling point distribution of the WAI for flat water surface.
origin, and the z axis denotes the height above the projector.k p is the 3D location of the arbitrary feature point on the structured light pattern to be projected, and the corresponding projected ray is reflected by the WAI, at 3D location k q , and then the reflected ray irradiates a spot on the diffuser difusser ∏ , at 3D location.ˆk N is the corresponding WAI normal, and k ′ s is the corresponding spot when the WAI is flat.

Figure 6 .
Figure 6.The principle of sampling of WAI normals.

c
are the x , y and z components of the vector p ˆk v , respectively.ka , k b , k c are the axial components of the vector ˆk v .

Figure 7 .
Figure 7.The principle of reverse ray tracing.An internal coordinate system of V consists of its optical axis and the lateral pixel coordinates of the image plane.The origin of the coordinate system of camera v is at the optical center lab o .The optical axis intersects the image plane at location c .The focal

Figure 8 .
Figure 8.(a) Deflection of the reflection angle ψ as a function of the inclination of the WAI when pro 30 θ =  .(b) Deflection of the reflection angle ψ as a function of the inclination of the WAI when pro 45 θ =  .

Figure 9 .
Figure 9. Geometry of the structured light projection system S .Let the sensor S determine k s with spatial uncertainty p Δ .This uncertainty may be due to the camera pixel size, diffuser characteristics and so on.This spatial uncertainty converts to uncertainties in the measured angular deviation ψ Δ and the measured angular deviation of the inclination angle of the WAI normal θ Δ ,

Figure 11 .
Figure 11.Simulation for WAI shape reconstruction.(a) Reference structured light image.(b) Distorted structured light image.(c) Results of feature point extraction for the reference structure light image using the method of Ref. [34].(d) Results of feature point extraction for the distorted structured light image using the method of Ref. [34].(e) Recovered WAI shape.(f) Ground-truth WAI [red] and reconstructed WAI [blue].

Figure 12 .
Figure 12.Absolute error distribution between the reconstructed WAI and the ground-truth WAI.

Figure 13 .Figure 14 .
Figure 13.Absolute error distribution between reconstructed WAI and the ground truth for the different sun θ using Alterman's method.(a) Absolute error distribution when sun 0 θ =  ; (b) absolute er- the wavy WAI.The results are shown in Figure15; the size of the image is 300 250 pixel × .The scatter plot shows the coordinates of the corner points of the colorcoded square checkerboards in the following three images, namely the ground truth (red), the distorted image (purple) and the recovery image (blue).In addition, the standard deviation (STD) of the corner positions in the distorted image is 26.2783 pixels, and the STD of the image restored by the estimated WAI is reduced to 1.2247 pixels.The results show that the image restoration method via structured light projection can significantly reduce the distortion.

Figure 15 .
Figure 15.(a) Ground-truth image; (b) an image distorted by a wavy WAI; (c) recovered image; (d) a scatter plot showing coordinates of the corner points of the color-coded square checkerboards in above three images.

2 F σ and 2 I
σ represent the variance of F and I , respectively.FI σ is the covariance of F and I . 1 (0.01 ) c L = × and 2 (0.03 ) c L = × are constants, in which L represents the dynamic range of pixel values.

Figure 16 .
Figure 16.Results of images recovered by the estimated wavy WAI with the two methods.