Sinusoidal Single-Pixel Imaging Based on Fourier Positive–Negative Intensity Correlation

Single-pixel imaging techniques extend the time dimension to reconstruct a target scene in the spatial domain based on single-pixel detectors. Structured light illumination modulates the target scene by utilizing multi-pattern projection, and the reflected or transmitted light is measured by a single-pixel detector as total intensity. To reduce the imaging time and capture high-quality images with a single-pixel imaging technique, orthogonal patterns have been used instead of random patterns in recent years. The most representative among them are Hadamard patterns and Fourier sinusoidal patterns. Here, we present an alternative Fourier single-pixel imaging technique that can reconstruct high-quality images with an intensity correlation algorithm using acquired Fourier positive–negative images. We use the Fourier matrix to generate sinusoidal and phase-shifting sinusoid-modulated structural illumination patterns, which correspond to Fourier negative imaging and positive imaging, respectively. The proposed technique can obtain two centrosymmetric images in the intermediate imaging course. A high-quality image is reconstructed by applying intensity correlation to the negative and positive images for phase compensation. We performed simulations and experiments, which obtained high-quality images, demonstrating the feasibility of the methods. The proposed technique has the potential to image under sub-sampling conditions.


Introduction
Optical cameras are some of the earliest and most commonly used remote sensing equipment. An optical camera generally comprises a lens unit and a photosensitive unit. In recent years, the photosensitive unit has changed from traditional films to electronic devices, such as the charge-coupled device (CCD) and the complementary metal oxide semiconductor (CMOS). The performance of a digital camera is determined by the performance of each pixel on the detector array. From mobile phones to professional, digital single-lens reflex (SLR) cameras, the number of pixels that make up a sensor chip is both a performance indicator and a marketing necessity. When multi-pixel detectors cannot be applied due to the weak signals caused by scattering or absorption losses, a single-pixel detector might be used as an alternative method to perform imaging. Researchers have shown that an image can be captured

Experiments and Methods
Although Hadamard single-pixel imaging and Fourier single-pixel imaging are the mainstream of SPI in recent years, we want to show a new application of orthogonal sinusoidal patterns in SPI. We use the Fourier matrix to generate sinusoidal and cosine structural illumination patterns, both of which can reconstruct images in the imaging course with symmetrical overlapping shadows due to missing phase information. A clear image is finally reconstructed by applying the intensity correlation to the negative and positive images for phase compensation [13,14,33,34], In the past, positive and negative ghost imaging was a technology derived from the selection of the detector's recording intensity data combined with a specific algorithm, which focused on data selection Sensors 2020, 20, 1674 3 of 9 and algorithm programming. The positive and negative images obtained here are attributed to the inherent characteristic of the Fourier matrix. We named the phenomenon "Fourier positive-negative intensity correlation". The proposed technique may bring new methods to SPI.
To generate orthogonal sinusoidal and cosine illumination patterns, we used the Fourier matrix. We verified the effectiveness of the proposed technique by numerical simulations and experimentation. We calculated the signal to noise ratio (SNR) of reconstructed images to evaluate the image quality. We used 16,384 two-dimensional (2D) basis images and 16,384 projective illumination patterns to obtain 128 × 128 spatial resolution images in simulations and experiments. Finally, we obtained clear images without an image-processing algorithm.
To generate orthogonal sinusoidal and cosine patterns, we needed to first generate the complex Fourier matrix. We defined a complex matrix F as the Fourier matrix, and it was no longer a member of the real number field R n , but a member of the n-dimensional complex domain C n , and every element in F was a complex number. The N × N square matrix F with entries is given by where j, k = 0, 1, 2, · · · , N − 1 and i is the imaginary number and is defined with the formula i = √ −1, and normalized by 1/ √ N to make it a unitary matrix. The discrete N × N square Fourier matrix is defined below: Euler's formula is defined as follows: where w is a member of the N th roots of unity ( 1, w, w 2 , · · · , w n−1 ) and is defined with the formula According to the properties of the unitary matrix, the number of measuring times required by applying the discrete Fourier matrix as the modulation matrix of the light source in SPI to reconstruct the complete image is equivalent to the spatial resolution (the number of pixels included in the image) of the image. In reality, we could not reproduce complex numbers in a single projection by structured light illumination patterns. As a result, we proposed a sinusoidal single-pixel imaging technique based on Fourier positive-negative intensity correlation. The imaging course is divided into two steps. First, two intensity images are obtained by modulating the structured light with sine and cosine functions. Secondly, a clear image is obtained by phase compensation of the two images. As mentioned above, the complex Fourier matrix is divided into two parts. The real part of the discrete N × N square Fourier matrix is made up of a cosine function and a row of the real number one. The imaginary part of the discrete N × N square Fourier matrix is made up of a sine function and a row of the real number zero. To transform the sine-and cosine-based modulation of structured light into a form that can be played on projectors, we extended the trigonometric functions' values to the positive range. The family of trigonometric functions 1, cos kπx l , sin kπx l satisfies the condition of orthogonal completeness. In the interval [−l, l], the following relationship is satisfied, where k is a natural number.
To modulate the three-dimensional (3D) scene with sinusoidal illumination patterns and finally reconstruct 2D images, we needed to reshape the one-dimensional (1D) sine and cosine signals into a 2D format. We used the real part of a 128 × 128 square Fourier matrix to illustrate the operation methods.
As shown in Figure 1, to generate a grayscale 2D illumination pattern, a reshape method is applied to a 1D discrete cosine signal that is a row of the real part of the Fourier matrix. For example, there are 16,384 discrete points in a signal period. For ease of viewing, we intercept a one-eighth period from the 129th entire signal, and it is easy to concatenate these 2048 discrete points into a cosine signal through curve fitting. To reconstruct a 2D square image with the 1D cosine signal, we reshape the total period, generating a 128 × 128 pattern of pixels. We set the first 128 points on the first row, the 129th to 256th points on the second row and so on, until a square pattern is generated.
Sensors 2020, 20, x FOR PEER REVIEW 4 of 9 To modulate the three-dimensional (3D) scene with sinusoidal illumination patterns and finally reconstruct 2D images, we needed to reshape the one-dimensional (1D) sine and cosine signals into a 2D format. We used the real part of a 128 128 × square Fourier matrix to illustrate the operation methods.
As shown in Figure 1, to generate a grayscale 2D illumination pattern, a reshape method is applied to a 1D discrete cosine signal that is a row of the real part of the Fourier matrix. For example, there are 16,384 discrete points in a signal period. For ease of viewing, we intercept a one-eighth period from the 129th entire signal, and it is easy to concatenate these 2048 discrete points into a cosine signal through curve fitting. To reconstruct a 2D square image with the 1D cosine signal, we reshape the total period, generating a 128 128 × pattern of pixels. We set the first 128 points on the first row, the 129th to 256th points on the second row and so on, until a square pattern is generated. Defining (x) f as a period function in the interval [ ] , l l − , we calculate the limit of the Fourier series of the defined function: After estimating, it can be simplified as: where ( ) A w and ( ) B w are Fourier transform coefficients. They can be calculated by using integrals as below: The Fourier integrals in complex form can be derived from the above Fourier integrals in the real number field. We simulate the proposed sinusoidal SPI technique in mathematics based on the Fourier integrals above. To reconstruct an image of the target scene employing Fourier positivenegative intensity correlation, the sinusoidal structured light field modulated by the object surface can be expressed as: where (x, y) F is the reconstructed image of the target, and P F (x,y) and N F (x,y) are the Fourier positive image and the Fourier negative image, respectively. r-n (x, y) I and i-n (x, y) I are the spatial intensity distributions of the Nth reshaped 2D Fourier real part pattern and the Nth reshaped 2D Fourier imaginary part pattern, respectively.  is the average algorithm for the Defining f (x) as a period function in the interval [−l, l], we calculate the limit of the Fourier series of the defined function: After estimating, it can be simplified as: where A(w) and B(w) are Fourier transform coefficients. They can be calculated by using integrals as below: The Fourier integrals in complex form can be derived from the above Fourier integrals in the real number field. We simulate the proposed sinusoidal SPI technique in mathematics based on the Fourier integrals above. To reconstruct an image of the target scene employing Fourier positive-negative intensity correlation, the sinusoidal structured light field modulated by the object surface can be expressed as: where F(x, y) is the reconstructed image of the target, and F P (x, y) and F N (x, y) are the Fourier positive image and the Fourier negative image, respectively. I r−n (x, y) and I i−n (x, y) are the spatial intensity distributions of the Nth reshaped 2D Fourier real part pattern and the Nth reshaped 2D Fourier imaginary part pattern, respectively. · · · is the average algorithm for the items in parentheses. I r−n Sensors 2020, 20, 1674 5 of 9 and I i−n are the intensities measured by the single-pixel detector after the Fourier positive-negative illumination patterns are modulated by the target surface, which can be expressed by the integrals: where O(x, y) is the target function. The proposed imaging technique is devoted to obtaining the exact target function by Equations (9)-(10). I r−n (x, y) and I i−n (x, y) can be substituted by the 2D sinusoidal pattern's matrix according to the propagation theory of light field in free space. The other variable in Equation (10) is measured and recorded after each frame of illumination. Thus, F(x, y) can be reconstructed. We introduced root mean square error (RMSE) to compare the difference between the real target function O(x, y) and the reconstructed target function F(x, y) in simulations, and signal to noise ratio (SNR) in experiments. RMSE and SNR are calculated by the equations below [32]: where M and N are the numbers of spatial image pixels, I S is the average intensity of the signal (in our experiment, we used the simple USAF 1951 resolution board as the target, and it was calculated from the bright white region we chose), I B is the average intensity of the background (it was calculated from the dark black region we chose), and σ S and σ B are the standard deviations of the above regions, respectively.

Results
To evaluate the performance of the proposed technique, we prove it through simulations and experiments. We firstly generate a complex discrete 16, 384 × 16, 384 square Fourier matrix and separate it into two parts, the real cosine matrix and the imaginary sine matrix. Half of the cosine and sine matrix data are selected according to symmetry for subsequent calculation. We reshape the selected discrete sampling points into 2D square patterns and obtain 8192 grayscale cosine modulation patterns and 8192 grayscale sine modulation patterns. Their element values are normalized to the range of 0 to 255 for projecting, and each pattern corresponds to a coefficient for reconstruction. To keep dimension the same for calculation, we use 128 × 128-pixel images as the target in numerical simulations. We use a printed USAF 1951 resolution board as the target in experiments. We obtain Fourier positive images and Fourier negative images simultaneously in numerical simulations, which benefit from the software's parallel computing. It is also convenient to obtain clear images with Fourier positive and negative images by phase compensation.
The imaging steps are shown in Figure 2, and we choose two targets as examples. The proposed technique obtains three images, and the final image is reconstructed by positive-negative images intensity correlation (we also call it phase compensation). The positive-negative images are the result of two overlapped images. The two overlapped images are made up of an image and its 180-degree rotated duplicate with different intensities. We have not found a method to directly extract the original image in a positive or negative image.
Any continuous signals can be expressed as infinite superpositions of sine wave signals with different frequencies. We apply this principle to a 2D space and propose the imaging technique. Different frequency information plays different roles in image structure. The main component of general images is low-frequency information, which forms the basic gray level of the image. High-frequency information forms the edges and details of the image. Therefore, it is possible to reconstruct images with low-frequency patterns and a low sampling ratio. The simulation results are shown in Figure 3. The resolution of the resulting image is 128 × 128 pixels, and the full sampling ratio means 16,384 measurements. We use RMSE to compare the quality of images resulting from the different sampling ratios. One of the curves is undulating obviously, and both of them show a downward trend, roughly. There are obvious ringing effects in the edges of the image when the sampling ratio is low. The direct cause of this effect is the loss of information in the process of imaging, especially the loss of high-frequency information. This seriously reduces the quality of the reconstructed image and makes it difficult to perform a subsequent image-processing algorithm on the image. We also noticed that the Fourier matrix has inherent flaws. In the high-frequency region, the phenomenon of insufficient sampling capacity is obvious. We choose the 8060th cosine signal, for example, and take the same approach as described above. We intercept 1/128 period from the 8060th entire signal, and we use MATLAB to connect the 128 discrete points (red points in Figure 4). It no longer seems like a cosine signal, although its generation is based on the cosine function. The reason for this is that in the high-frequency region, there are not enough sampling points to describe the cosine signal completely. To obtain clear result images when the sampling ratio is low, reordering the patterns by frequency may be effective. The reordering needs to pay attention to two points. The first is proposing a method to evaluate the high-low degree of the frequencies belonging to the patterns by complexity levels, and the second is choosing high-frequency patterns as accurately as possible. Any continuous signals can be expressed as infinite superpositions of sine wave signals with different frequencies. We apply this principle to a 2D space and propose the imaging technique. Different frequency information plays different roles in image structure. The main component of general images is low-frequency information, which forms the basic gray level of the image. Highfrequency information forms the edges and details of the image. Therefore, it is possible to reconstruct images with low-frequency patterns and a low sampling ratio. The simulation results are shown in Figure 3. The resolution of the resulting image is 128 128 × pixels, and the full sampling ratio means 16,384 measurements. We use RMSE to compare the quality of images resulting from the different sampling ratios. One of the curves is undulating obviously, and both of them show a downward trend, roughly. There are obvious ringing effects in the edges of the image when the sampling ratio is low. The direct cause of this effect is the loss of information in the process of imaging, especially the loss of high-frequency information. This seriously reduces the quality of the reconstructed image and makes it difficult to perform a subsequent image-processing algorithm on the image. We also noticed that the Fourier matrix has inherent flaws. In the high-frequency region, the phenomenon of insufficient sampling capacity is obvious. We choose the 8060th cosine signal, for example, and take the same approach as described above. We intercept 1 / 128 period from the 8060th entire signal, and we use MATLAB to connect the 128 discrete points (red points in Figure 4). It no longer seems like a cosine signal, although its generation is based on the cosine function. The reason for this is that in the high-frequency region, there are not enough sampling points to describe the cosine signal completely. To obtain clear result images when the sampling ratio is low, reordering the patterns by frequency may be effective. The reordering needs to pay attention to two points. The first is proposing a method to evaluate the high-low degree of the frequencies belonging to the patterns by complexity levels, and the second is choosing high-frequency patterns as accurately as possible.  Any continuous signals can be expressed as infinite superpositions of sine wave signals with different frequencies. We apply this principle to a 2D space and propose the imaging technique. Different frequency information plays different roles in image structure. The main component of general images is low-frequency information, which forms the basic gray level of the image. Highfrequency information forms the edges and details of the image. Therefore, it is possible to reconstruct images with low-frequency patterns and a low sampling ratio. The simulation results are shown in Figure 3. The resolution of the resulting image is 128 128 × pixels, and the full sampling ratio means 16,384 measurements. We use RMSE to compare the quality of images resulting from the different sampling ratios. One of the curves is undulating obviously, and both of them show a downward trend, roughly. There are obvious ringing effects in the edges of the image when the sampling ratio is low. The direct cause of this effect is the loss of information in the process of imaging, especially the loss of high-frequency information. This seriously reduces the quality of the reconstructed image and makes it difficult to perform a subsequent image-processing algorithm on the image. We also noticed that the Fourier matrix has inherent flaws. In the high-frequency region, the phenomenon of insufficient sampling capacity is obvious. We choose the 8060th cosine signal, for example, and take the same approach as described above. We intercept 1 / 128 period from the 8060th entire signal, and we use MATLAB to connect the 128 discrete points (red points in Figure 4). It no longer seems like a cosine signal, although its generation is based on the cosine function. The reason for this is that in the high-frequency region, there are not enough sampling points to describe the cosine signal completely. To obtain clear result images when the sampling ratio is low, reordering the patterns by frequency may be effective. The reordering needs to pay attention to two points. The first is proposing a method to evaluate the high-low degree of the frequencies belonging to the patterns by complexity levels, and the second is choosing high-frequency patterns as accurately as possible.   We used the natural order Fourier matrix to reconstruct a 128 128 × -pixel image in the experiment. As shown in Figure 5, there are no light sources except the commercial projector in the experimental set-up. The projector keeps the projection frequency at 6 frames per second. When the frame frequency is increased, frames are sometimes lost. We used a USAF 1951 resolution board as the target, printed on an A4 paper. To reconstruct a clear image, 16,384 illumination patterns are needed. We used 8192 cosine illumination patterns and the same number of sine illumination patterns to measure the reflected intensity by a photodiode, and the results were recorded on the laptop computer. We obtained positive and negative images, and a clear image was finally reconstructed by Fourier positive-negative intensity correlation. It took over 0.75 h to measure and record the intensities to obtain a perfect image under the above conditions. The change of lighting in the We used the natural order Fourier matrix to reconstruct a 128 × 128-pixel image in the experiment. As shown in Figure 5, there are no light sources except the commercial projector in the experimental set-up. The projector keeps the projection frequency at 6 frames per second. When the frame frequency is increased, frames are sometimes lost. We used a USAF 1951 resolution board as the target, printed on an A4 paper. To reconstruct a clear image, 16,384 illumination patterns are needed. We used Sensors 2020, 20, 1674 7 of 9 8192 cosine illumination patterns and the same number of sine illumination patterns to measure the reflected intensity by a photodiode, and the results were recorded on the laptop computer. We obtained positive and negative images, and a clear image was finally reconstructed by Fourier positive-negative intensity correlation. It took over 0.75 h to measure and record the intensities to obtain a perfect image under the above conditions. The change of lighting in the environment over such a long experiment time would have a certain influence on the experimental results. The black background of the printed target minimizes the interference of reflected stray light in the experiment. We also remained in the darkroom during the experiment to avoid interference from outside light sources.
We used the natural order Fourier matrix to reconstruct a 128 128 × -pixel image in the experiment. As shown in Figure 5, there are no light sources except the commercial projector in the experimental set-up. The projector keeps the projection frequency at 6 frames per second. When the frame frequency is increased, frames are sometimes lost. We used a USAF 1951 resolution board as the target, printed on an A4 paper. To reconstruct a clear image, 16,384 illumination patterns are needed. We used 8192 cosine illumination patterns and the same number of sine illumination patterns to measure the reflected intensity by a photodiode, and the results were recorded on the laptop computer. We obtained positive and negative images, and a clear image was finally reconstructed by Fourier positive-negative intensity correlation. It took over 0.75 h to measure and record the intensities to obtain a perfect image under the above conditions. The change of lighting in the environment over such a long experiment time would have a certain influence on the experimental results. The black background of the printed target minimizes the interference of reflected stray light in the experiment. We also remained in the darkroom during the experiment to avoid interference from outside light sources. The images reconstructed by the proposed technique are shown as Figure 6. The SNR was calculated with Equation (11); the region of the signal is boxed with a red rectangle, and the region of the background is boxed with a yellow rectangle. There are some white stripes on the final image, with the main reason being the density inhomogeneity of the printed black background, which is especially obvious under the strong light of the projector. Excluding the imperfections of the experimental results, the imaging process and experimental phenomena are consistent with the numerical simulations.  The images reconstructed by the proposed technique are shown as Figure 6. The SNR was calculated with Equation (11); the region of the signal is boxed with a red rectangle, and the region of the background is boxed with a yellow rectangle. There are some white stripes on the final image, with the main reason being the density inhomogeneity of the printed black background, which is especially obvious under the strong light of the projector. Excluding the imperfections of the experimental results, the imaging process and experimental phenomena are consistent with the numerical simulations. experiment. As shown in Figure 5, there are no light sources except the commercial projector in the experimental set-up. The projector keeps the projection frequency at 6 frames per second. When the frame frequency is increased, frames are sometimes lost. We used a USAF 1951 resolution board as the target, printed on an A4 paper. To reconstruct a clear image, 16,384 illumination patterns are needed. We used 8192 cosine illumination patterns and the same number of sine illumination patterns to measure the reflected intensity by a photodiode, and the results were recorded on the laptop computer. We obtained positive and negative images, and a clear image was finally reconstructed by Fourier positive-negative intensity correlation. It took over 0.75 h to measure and record the intensities to obtain a perfect image under the above conditions. The change of lighting in the environment over such a long experiment time would have a certain influence on the experimental results. The black background of the printed target minimizes the interference of reflected stray light in the experiment. We also remained in the darkroom during the experiment to avoid interference from outside light sources. The images reconstructed by the proposed technique are shown as Figure 6. The SNR was calculated with Equation (11); the region of the signal is boxed with a red rectangle, and the region of the background is boxed with a yellow rectangle. There are some white stripes on the final image, with the main reason being the density inhomogeneity of the printed black background, which is especially obvious under the strong light of the projector. Excluding the imperfections of the experimental results, the imaging process and experimental phenomena are consistent with the numerical simulations.

Discussion
In this paper, we use the cosine and sinusoidal functions to modulate the illumination patterns in Fourier single-pixel imaging. The orthogonality of the illumination patterns is achieved by Fourier positive-negative intensity correlation, which we also call phase compensation. Using numerical simulations and experimental results, we provide the imaging process and phenomena, and the experiment results agree with the numerical simulations. We acquire three images based on the proposed technique, and a clear image is reconstructed by Fourier positive-negative intensity correlation of the other two images. The SNR of the results confirms the effectiveness of the proposed imaging technique.
Compared with the existing mature single-pixel imaging techniques, such as Hadamard single-pixel imaging (HSI) and Fourier single-pixel imaging (FSI) [23], the proposed technique can also reconstruct perfect images with full sampling ratios based on such grayscale illumination patterns as FSI and HSI. When the sampling ratio is less than 25%, the ringing effect of the reconstructed image is particularly obvious and the image quality is worse than the results shown in the present article [23]. The main reason for this lies in the pattern frequency used in the low sampling ratio.
We choose the original sequence of the discrete Fourier matrix for measurement in the present article, rather than choosing low-frequency patterns.
To reduce the imaging time and capture high-quality images based on the proposed technique under low sampling ratios as shown in [23], the following aspects are worthy of further research. The ups and downs of the RMSE curve provide the possibility for the reordering of the illumination patterns based on the frequencies. The obvious ringing effect under the low sampling ratio requires one to obtain high-frequency information. The illumination mode based on the projector also needs to be replaced to reduce the imaging time. A binary device such as a digital micro-mirror device (DMD) is worthy of consideration, as it has been proved feasible to perform Fast Fourier single-pixel imaging via binary illumination [28,29]. In summary, we can introduce frequency-domain filtering in the proposed technique by pattern selection. The low frequencies and accurate high frequencies ensure good images, and we will concentrate on the reordering method and selection of illumination patterns in the next work.