Super-Resolution Remote Imaging Using Time Encoded Remote Apertures

Abstract

Imaging of scenes using light or other wave phenomena is subject to the diffraction limit. The spatial profile of a wave propagating between a scene and the imaging system is distorted by diffraction resulting in a loss of resolution that is proportional with traveled distance. We show here that it is possible to reconstruct sparse scenes from the temporal profile of the wave-front using only one spatial pixel or a spatial average. The temporal profile of the wave is not affected by diffraction yielding an imaging method that can in theory achieve wavelength scale resolution independent of distance from the scene.

1. Introduction

1.1. Previous Attempts to Beat the Diffraction Limit of Resolution

The diffraction limit naturally limits the spatial resolution of an image acquired by a conventional imaging system with a finite aperture. A wave traveling from the scene to the imaging system is subject to diffraction. The wave reaching the aperture of the imaging system has been degraded by diffraction. It can no longer be used to reconstruct an image of the object at the optimal resolution. Any spatial wavefront is subject to this diffraction, which causes a well-known linear drop in resolution with distance. The resulting linear dependence of resolution on imaging distance is known as the Rayleigh criterion.

Over the last several years, many works in different fields have been done to beat the diffraction limit. Astronomical imaging is an area where super-resolution methods are extensively studied. Most of the super-resolution methods used in astronomy combine image capturing techniques with image post-processing. In early years, co-addition methods, capturing successively multiple pictures within a short exposure time and applying shift-and-add algorithm [1] or linear reconstruction scheme [2] to combine the images, were actively used. Different regularization-based optimization methods were developed: Orieux et al. [3] successfully applied quadratic-regularized optimization to the “Spectral and Photometric Imaging Receiver” data. Rust et al. [4] demonstrated the high-resolution fluorescence microscopy method (STORM) that utilizes high-accuracy localization of photoswitchable fluorophores. The method operates past the diffraction limit. Jarret et al. [5] proposed a method using a deconvolution technique, maximum correlation method, combined with a re-sampling kernel to super-resolve “Wide-field Infrared Survey Explorer” data.

Several image post-processing methods exist in digital imaging applications in order to construct high-resolution images from low-resolution images. Nonuniform interpolation approach [6], frequency domain approach [7], regularized optimizations (deterministic [8,9,10,11] and stochastic [12,13,14,15]), iterative back-projection [16], adaptive filtering approach [17] are widely used super-resolution imaging methods. These super-resolution methods provide high-resolution image outputs, mitigating the effects of diffraction. However, all of the methods still suffer from the fundamental limit: the resolution of super-resolved images still depends on the distance and aperture of the imaging system. The further the region of interest, the lower the image resolutions.

It is well known that a scattering medium around an object can be used to couple evanescent waves into the far-field. This means that images of the embedded object with resolutions similar to the wavelength can be captured from large distances [18]. In existing approaches, this requires prior characterization of the scattering medium from the location of the object, which makes the method unsuitable for many practical applications. Here, we show that scattering can be leveraged in sparse scenes to determine the geometry of the entire scattering scene (object and medium) without prior information based on just a single pixel time response of the scene.

1.2. Our Contribution

In this work, we propose a novel super-resolution imaging approach that reconstructs scenes from internally scattered light while using their time response. This temporal signal is not subject to diffraction. We show that the resolution of our method is independent of distance and aperture size, and we can reconstruct the scene up to Euclidean congruence. We demonstrate the capabilities of the novel approach while using simulated and experimental data.

2. Time Encoded Remote Apertures (TERA)

2.1. Fundamentals of Imaging

In this section, we overview the fundamental process of obtaining an image from an object and define terms to avoid confusion. Consider two objects emitting or reflecting light $p_1,p_1$ and an imaging system C with an aperture size D placed at a distance d from for the object (Figure 1a). To infer positions of object points $p_i$, $i=1,2$ within the scene, the imaging system redirects all rays from points $p_i$, $i=1,2$, such that they constructively interfere on exactly one point in the focal plane. In other words, the imaging system evaluates the length of a light ray, as encoded in its phase. In this case, the resolution r of the imaging system is determined by the Rayleigh’s diffraction limit:

$$r=\frac{1.22\lambda d}{D}$$

(1)

where $\lambda$ is the wavelength of the emitted light and determines how accurately the imaging system can determine the length of each ray, d is a distance between focal plane of the imaging system and the object, and D is the diameter of the aperture of the imaging system. Figure 1a shows an example of a diffraction pattern (Airy disk) with two targets ($p_i$, $i=1,2$) resulting from a circular aperture. Another way of imaging is to measure the time of flight of intensity fluctuation. We call these intensity fluctuations as second-order coherence. One can use a short-pulsed laser to illuminate the object and lens-less time-of-flight detector to observe back-scattered light and reconstruct an image. The resolution of such an image is described by the Rayleigh criterion, except that the wavelength $\lambda$ is replaced by the time resolution $\tau$ of the time-of-flight detector. We call it a transient Rayleigh criterion.

2.2. Tera Imaging Overview

Consider a situation when one can detect not only direct (first bounce) back-scattered lights, but also the lights that bounced between the objects(second bounce). Such a signal can be obtained by illuminating the scene with a short-pulsed laser and detecting the returning light with a single time-of-flight detector. The temporal signal of these multi-bounce lights directly encodes the information about the distance between two object points in the scene. Figure 1b shows the simplified example of how the information about the scene is encoded in the temporal signal. Suppose the scene contains two small point objects $p_1$ and $p_2$ with a negligibly small diameter $\delta$. The distances from $p_1$ and $p_2$ to the imaging system C are $d_1$ and $d_2$ correspondingly. The distance between objects $p_1$ and $p_2$ is $d_3$. The scene is being illuminated by a pulsed laser, and a time-of-flight detector measures the returning light from the scene. The time-resolved measurement or time response of the scene is illustrated on the left side of Figure 1b. The first impulse is due to the light directly reflected from object $p_1$, and it appears at $t_1=\frac{2d_1}{c}$, where c is the speed of light. Similarly, the second impulse appearing at $t_2=\frac{2d_2}{c}$, corresponds to the light reflected from the object $p_2$. The last impulse at $t_3$ is due to the light traveled following two paths: ($C\to p_1\to p_2\to C$) and ($C\to p_2\to p_1\to C$). Clearly, these two paths have the same travel distances and, thus, in the time response, they both appear at $t_3=\frac{d1+d2+d3}{c}$. By using these three values, $t_1,t_2,t_3$, one can find $d1,d2$, and $d3$. In other words, one can completely reconstruct relative positions of objects $p_1$ and $p_2$ up to Euclidean congruence.

Now, we fix the distance $d_3$ and keep increasing the distances $d_1$ and $d_2$. Naturally, at some point, the distance $d_3$ will be below the Rayleigh diffraction limit for the conventional imaging system, i.e., one can not visually separate two points ($p_1$ and $p_2$). However, for the proposed imaging system, the increase of the distances $d_1$ and $d_2$ only results in the time shift of the entire signal. The difference between three time tags, $t_1,t_2,t_3$, is conserved. Therefore, one can still recover the scene($d_1,d_2$ and $d_3$) up to Euclidean congruence. We note that we can find exact relative distance ($d3$) between the objects and distances ($d1,d2$) from the imaging system. However, the orientation (rotation) of the point-cloud remains ambiguous. Because we are using single-pixel detector, one can rotate the entire point-cloud along the depth direction or translate the entire point-cloud along the perpendicular direction to the depth and record the same time response. We call it reconstructing point-cloud up to Euclidean congruence.

Naturally, the arising question is whether it is possible to reconstruct the scene when the number of objects is larger. Is the reconstruction unique, i.e., is it possible that two or more configurations generate the same time response? Below, we show that the reconstruction of a point-cloud with n objects is possible under some assumptions.

In this section, we use the results of Gkioulekas et al. [19], which shows that, when the measured time-response of the scene contains a sufficiently rich set of first and second bounces, we can reconstruct the point-cloud up to Euclidean congruence. Here, we follow the same notation as [19].

Now, consider a scenario where the scene consist of multiple point objects (Figure 2a). Let $p_1,...,p_n\in {\mathbb{R}}^3$ be n point objects in the scene and $p_0\in {\mathbb{R}}^3$ be a position of our imaging system (a laser and a detector are co-located). See Figure 2b. A graph $\mathbf{S}=(p_1,...,p_n)$ is a scene configuration and $\mathbf{K}=(p_0,p_1,...,p_n)$ is a total configuration. The scene is illuminated with a pulsed laser and the detector observes returning signal from the scene. We define light path ${\alpha }_k=[p_0,p_1,...,p_z,p_0](z\in \mathbb{N})$ as a finite sequence of points where light has traveled. z is some integer. Note that any sequence $\alpha$ starts from $p_0$ and ends at $p_0$, since our light source and detectors are co-located. First bounce, ping, is a light path ${\alpha }_k=[p_0,p_i,p_0]$ for $i=1,...,n$. Second bounce, loop, is a light path ${\alpha }_k=[p_0,p_i,p_j,p_0]$ for $i\ne j$. Let $v_k$ be a length of ${\alpha }_k$. Subsequently, our measurement contains ensemble $\beta =[v_1,v_2,...,v_k]$, where $v_k$ are returned first or second bounce.

Next, let ${\mathbf{K}}_5$ be a sub-graph of $\mathbf{K}$ that has 5 vertices, including $p_0$. If the measurement ensemble $\beta$ contains all pings and triangles of ${\mathbf{K}}_5$ that starts and ends at $p_0$, and then we say that ${\mathbf{K}}_5$ is contained within $\beta$. See Figure 2c. In this example, the measurement ensemble $\beta$ contains four pings ($\alpha =[p_0,p_i,p_0]$ for $i=1,...,4$) and six loops ($\alpha =[p_0,p_i,p_j,p_0]$ for $i,j\in (1,2,3,4)$ and $i\ne j$). Next, if the measurement ensemble $\beta$ contains a ping ($\alpha =[p_0,p_4,p_0]$) and three loops ($\alpha =[p_0,p_i,p_4,p_0]$ for $i=1,...,3$), then we say that $\beta$ allows for trilateration. See Figure 2d.

Finally, we can apply the theorem from Gkioulekas et al. [19], which says that if $\mathbf{K}=(p_0,p_1,...,p_n)$ is unknown configuration and $\beta$ is the measurement ensemble of $\mathbf{K}$ that allows for trilateration, then one can find trilateration based process to reconstruct $\mathbf{K}$ up to Euclidean congruence. Readers can find detailed mathematical proof of the statement in [19]. Note that there exists a generic (or trivial) point configuration $\mathbf{K}$ for which unique reconstruction is not possible. However, these special configurations occur very rarely [19].

3. Reconstruction Algorithm

In this section, we design a trilateration based reconstruction algorithm for our imaging system. There exist algorithms (TRIBOND [20] and LIGA [21]) that reconstruct a point-cloud given list of unassigned edge measurements. TRIBOND is a deterministic algorithm that addresses the unassigned distance geometry problem (uDGP). It has been successfully applied to reconstruct the structure of molecules and nanoparticles while using edge distance lists extracted from X-ray or neutron diffraction data. However, we can not apply the TRIBOND algorithm directly to our data, because it contains not only the first bounce, but also the second bounce. Moreover, these bounces are unlabeled. It requires modification. Gkioulekas et al. [19] showed that it is possible to reconstruct point-cloud using unlabeled edge measurements. Here, we modify the TRIBOND algorithm [20] with method described in [19] to process data that were acquired by our imaging system. The modified TRIBOND algorithm (see Algorithm 1) consists of two parts: core finding and adding a vertex.

3.1. Core Finding

The first step of the buildup algorithm is to find the core. The core of the embedding point-cloud is an over-constrained set of five points, including the source, see Figure 2c. The core can be broken down into three pieces: the base triangle and two tetrahedra. The base triangle (Figure 2c $(p_0,p_1,p_3)$) is constructed using two first (Figure 2c dashed green line) and one second (Figure 2a solid red line) bounce. Each tetrahedra (Figure 2a $(p_0,p_1,p_2,p_3)$ and $(p_0,p_1,p_4,p_3)$) uses one first and two second bounces. Because the bounces are not labeled, one has to exhaustively search over all possible first and second bounce pairs to build the base triangle and tetrahedra. Finally, we loop through all of the remaining bounces to find one second bounce in order to test and check whether it fits into bridge bond $(p_2,p_4)$ between two vertices of the tetrahedra. If a correct second bounce is found and the bridge bond is satisfied. We found a core structure and can move to “adding a vertex”. If the bridge bond is not satisfied, then we restart “core finding” and choose another base triangle and tetrahedra by exhaustive search. This process is repeated until the core structure is found.

3.2. Adding a Vertex

After core is found, next step is to iteratively add a vertex to the core. First, we choose a random tetrahedron from the current structure. The next step is to search over all possible combinations of four distances from the remaining distance pool. Three distances will form one first and two second bounce distances. The remaining distance is used to test the bridge bond for the chosen rigid substructure. For instance, in Figure 2d $(p_0,p_1,p_2,p_3)$ is chosen as a rigid substructure and $p_4$ is added point while using one first bounce $[p_0,p_4,p_0]$ and two second bounces ($[p_0,p_1,p_4,p_0]$ and $[p_0,p_2,p_4,p_0]$). The second bounce $[p_0,p_3,p_4]$ is used for the bridge bond check.

Algorithm 1: Modified TRIBOND

4. Simulations

Here, we test the modified TRIBOND algorithm using simulated data. We generate simulated data using a simplified version of the transient light transport renderer [22,23]. The renderer is successfully used in many times of flight applications [24,25]. Similar to the previous chapter, we consider the following scenario. Let $p_1,...,p_n\in {\mathbb{R}}^3$ be n point objects in the scene and $p_0\in {\mathbb{R}}^3$ be a position of the TERA imaging system. Let $d_i$ be the distance from $p_i$ to $p_0$ for $i=1,...,n$ and $d_{ij}$ be the distance from $p_i$ to $p_j$ for all $i\ne j$. See Figure 3. In this simulation, we assume the exact distance list and do not consider the noise. The simulated scene contains $n=10$ point objects that are located far away from the imaging system. The Figure 3 shows the simulated time response, which contains a mixture of first and second bounce. The first and second bounces signals are also shown separately. Locations of the peaks in the signal are marked with blue triangles. We transform these locations of the peaks into a distance list and use it as an input to the modified TRIBOND algorithm. On the right side of the Figure 3, the result of reconstruction is shown. The reconstruction matches the original point-cloud configuration up to Euclidean congruence; in other words, all of the pair-wise distances were recovered; however, the rotation and transpose of the point-cloud are remaining unknown.

We conduct the following simulation in order to address more general scenarios and numerically evaluate the algorithm. Our sensor’s dark count rate is 10 photons per second. The temporal impulse response of the system is 80 ps. See Section 5 for more details on the experimental hardware. Additionally, Poisson noise was added to the simulated data. We assume that, after pulsing, noise can be avoided by using a fast gated SPAD detector, and we, therefore, do not add after pulsing noise in the simulation. Laser power is assumed to be 10 W. We generate sets of simulated scenes with five to twenty diffuse targets with varying sizes from 1 to 8 cm. Reconstructing all these scenes would not be computationally feasible. In order to evaluate whether we have sufficient data to reconstruct the scene, we note that, in order to reconstruct a vertex of a point cloud, it is sufficient to have one first bounce and two second bounce signals from that vertex (see Section 3.2 and Gkioulekas et al. [19] for details). By counting number peaks and comparing to ground truth pairwise distances, we evaluate which vertexes of the scene remain recoverable given the correctly identified peaks. Figure 4 shows the results of statistical analysis. A large number and large diameter sizes of objects may occlude some possible light paths and can create overlapped peaks, which reduces the recoverability rate. However, a high recoverability rate is achieved for a small number of points with small diameter sizes, since most of the peaks are available. Note that an additional complication to the reconstruction is provided by false peaks in the data. Section 5 discusses this issue.

5. Proof of Concept Experiments

In this section, we demonstrate the performance of the algorithm using experimental data. The experimental setup is shown in Figure 5a. The imaging system contains an ultra-fast laser (One Five Katana HP amplified diode laser) that emits laser pulses with 50 ps width and 10 MHz repetition rate. An example of the laser illumination area is shown in Figure 5d. The laser has 1 W emission power at the wavelength of 532 nm. The scene is located around 4 m away from the imaging system. The returning scattered light from the scene is collected by a SPAD (PMD series), which has 20 micron diameter active-area. The SPAD has a photon detection efficiency of 49% at wavelength 555 nm. The dark count rate(DCR) is 10 photons per second. After pulsing probability(AP) is 0.1–3%. Dead time is 77 ns. The SPAD is coupled with a time-correlated single-photon counter (PicoQuant HydraHarp). The total effective temporal impulse response of our imaging system is 80 ps. Exposure time for each scene capture was set to 2 min. Note that we do not use any lenses in front of the SPAD.

Figure 5b,c,e,f show examples of experimental scenes. The scene is located approximately 4 m away from the imaging system. Circular patches and stars have a size of 4 × 4 cm, and the plane is 8 × 8 cm. First, we test the method using two circular patches. We place to patches 15 cm apart from each other and move them apart up to 35 cm with 1 cm spacing (21 different positions). Note that one patch is stationary, while another is moving. See Figure 6. The first column shows the actual scene picture. The second column is a reference simulated picture of the scene using a camera with an aperture size of 20 microns. The ground truth scene has been captured while using a regular camera, and the image was convolved with a blur kernel that corresponds to 20 micron sensor at the corresponding depth. The third column is a reconstruction of the scene using acquired data. Finally, the fourth column shows the actual data. In the reconstruction, one can see that the two patches clearly are separated. Figure 7 shows a plot of ground truth (green) and reconstructed (blue) distances. The RMS error between ground truth and reconstructed distances is 0.03 cm. Next, we test the method using different targets (star, plane, circular patch). Note that the circular patch and plane are white and they have diffuse surfaces, whereas stars are brown and made of wood. See Figure 8a,b. One can notice that the shape of the star and plane’s peak are slightly different from the circular patches, but still, three peaks exist in the data. Stars have sharper and more pronounced peaks, whereas planes have wide peaks. Lastly, in Figure 8c, the reconstruction of three circular patches is shown. In the data, there are three first bounces and three second bounces peaks.

As was mentioned in the simulation section, in some cases, it is possible for objects to block some light paths, which results in missing peaks in the collected time response. Peaks may also be too weak to detect and be missing for that reason. We can reconstruct vertexes that have at least one first and two second bounce distance peaks preserved in the data (Section 3.2). The objects can also have low reflectance or be not well-orientated, which may lead to lower peak intensity. However, as long as the peak is detected, the actual intensity is not relevant. If needed, one can increase exposure time or laser power to get a better signal. Additionally, peaks can also overlap with each other. One possible way to address overlapped peaks is to add a post processing step, where all of the discovered nodes are randomly tested for a potential double peak and added to the distance list for further reconstruction. It is an interesting approach and it is subject for further study.

In addition to missing peaks, the collected data may contain incorrect peaks due to background noise from ambient light, dark count rate, after pulsing. Incorrect peaks may also be what we call orphaned peaks that are true peaks from a patch for which not enough peaks are available to reconstruct the patch. A wrong peak can create an invalid vertex in the point-cloud if it satisfies the vertex adding condition (Section 3.2). The algorithm would find an incorrect core and most likely only be able to account for a very small fraction of the detected peaks in a small network. Hence, it can lead to an incorrect reconstruction. In this case, the re-initialization of the algorithm that happens routinely as part of the described algorithm can help. Because all distances used for core build-up and vertex adding parts are randomized, there is a probability that the wrong peak will not be used in the core and would be left over after all of the correct nodes and edges have been identified. A small number of false peaks thus would not prevent a correct reconstruction. A better way to deal with incorrectly detected peaks is an interesting subject for further research.

Our SPAD’s dark count rate is 10 photons per second. Coupled with 1 W laser, this dark count rate is negligible in our experiments. After pulsing probability is around 0.1–3% and can also be ignored in our experiments. One way to reduce after pulsing noise is to use fast time-gated SPADs. Gated mode allows a SPAD to be sensitive only during a short gate window of hundreds of picoseconds. The gating window can be moved in time. By progressively moving the active window through the scene in time, one can largely eliminate after pulsing.

6. Conclusions

In this paper, we present a novel imaging method that does not depend on the fundamental diffraction limit. The method makes use of hardware and algorithm to break the resolution limit under certain conditions of the scene (point-cloud assumption of the scene). In conventional imaging systems, the fundamental diffraction limit is governed by the size of the aperture, wave-length, and distance to the target. With fixed wave-length and distance, our proposed method does not depend on the size of the aperture, but instead, depends on the time resolution of the imaging system. The method is robust to change in surface materials. Currently, the main limitation is that the method is only applicable to the point-cloud scenes. Such sparse scenes commonly occur in aerial or space imaging scenarios. The paper gives base theory and introduces a method to use multiply scattered light (second bounce). As a next step, we are exploring possibilities of using multiply scattered light for continuous surfaces. As well as using machine learning to do the classification of objects below resolution limit while using multiply scattered light.