Time-division color holographic projection in large size using a digital micromirror device

: Holographic projection is a simple projection because it enlarges or reduces reconstructed images without using a zoom lens. However, one major problem associated with this projection is the deterioration of image quality as the reconstructed image enlarges. In this paper, we propose a time-division holographic projection, in which the original image is divided into blocks and the holograms of each block are calculated. Using a digital micromirror device (DMD), the holograms were projected at high speed to obtain the entire reconstructed image. However, the holograms on the DMD need to be binarized, thereby causing uneven brightness between the divided blocks. We correct this by controlling the displaying time of each hologram. Additionally, combining both the proposed and noise reduction methods, the image quality of the reconstructed image was improved. Results from the simulation and optical reconstructions show we obtained a full-color reconstruction image with reduced noise and uneven brightness.


Introduction
Computer-generated holograms (CGHs) [1] are widely used in optical fields such as holographic displays [2] and wavefront control [3] because they can control light waves. Holographic projections [4,5], a type of holographic display, employ holographic features to enable special projections (e.g., multi-projection [5,6]) that are difficult to achieve using conventional projections. Additionally, holographic projections do not require zoom lenses because the enlargement and reduction of the reconstructed image can be done computationally [7]. Therefore, in principle, it is a simple projector consisting only of a light source and a spatial light modulator (SLM) [5]. One critical challenge for the practical realization of holographic projection is the degradation of image quality due to noise superimposed on the reconstructed image. Multiple factors can cause noise, including the random phase added to the original image and the expansion of the pixel pitch when enlarging the reconstructed image. Several studies have proposed methods for reducing noise. The multi-random phase method [8] averages out the noise by reconstructing CGHs with different random phases in a time-division manner at high speed. Each reconstructed point of the CGH image has side lobes due to diffraction effects, which cause noise due to random interference between these side lobes. In the pixel separation method [9], several images with separated pixels are prepared from an original image, and the CGHs of these pixel-separated images are calculated. Using time-division reconstruction of CGHs at high speed opens the possibility of a reconstructed image with reduced noise. The down-sampling method [10] reduces noise in the reconstructed image by obtaining a down-sampled CGH from the original image. The random phase-free method [11] adds a virtual convergence light to the original image, resulting in a good reconstructed image without a random phase. However, this method works well for amplitude holograms but needs improvements for phase-only and binary holograms [12,13]. Optimization of CGH using the Gerchberg-Saxton algorithm [14,15] and the gradient descent method [16] obtains a reconstructed image with good image quality, however, it is computationally expensive due to iterative calculations. This paper proposes a time-division holographic projection projected on a large screen. The proposed method divides an original image into blocks and the CGHs of each block generated is calculated. Then, a digital micromirror device (DMD) is used to project the CGHs at high speed to obtain the entire reconstructed image. The CGH to be displayed on the DMD needs to be binarized. However, the binarized CGHs lose the brightness information of the divided original image, causing uneven brightness in the reconstructed image. To solve this problem, the proposed method changed the display time of the CGHs proportional to the average brightness value of the divided original image. Additionally, combining our method with the multi-random phase [8] and pixel separation methods [9], we improved the image quality of the reconstructed image. From the simulation and optical reconstructions, we obtain a large-color reconstructed image with reduced noise and color unevenness. The rest of the paper is organized as follows: Section 2 describes each aforementioned corrective method. Section 3 presents the simulation and optical results. Section 4 holds the conclusion.

Methods
In a holographic projection, a random phase is added to the original image to diffuse the object light widely. Then, the CGH is generated using propagation calculation. Given the complex amplitude of the original image ( , ), the random phase ( , ), the complex amplitude of the CGH ( , ), the propagation distance , and the light propagation operator Prop [・], the CGH calculation is expressed by Since a DMD is used in this study, only the real part of Eq. (1) is extracted and binarized to generate binary CGH. In the binarization, we use one if the real part is positive, and zero otherwise. Figure 1 shows the simulated results of the reconstructed image when the size of the original image is twice larger than the CGH. Figure 1(a) shows the original image. Without the random phase, only the part with the same size as the CGH is reconstructed, as shown in Fig. 1(b). However, by adding a random phase, the entire original image is reconstructed as shown in Fig. 1(c). This causes speckle noise in the reconstructed image. There are methods to reduce these speckle noises, such as the multi-random phase and pixel separation methods.  In the following subsections, we explain the multi-random phase and pixel separation methods. Further, the proposed methods, a combination of multi-random and pixel separation methods, and the correction method for color unevenness are explained.

Multi-random phase method
The multi-random phase method [8] averages out the noise in the reconstructed image using rapidly switching multiple CGHs generated by adding different random phases to the original image. When the number of CGHs is , the signal-to-noise ratio of the reconstructed image decreases by √N.

Pixel separation method
The pixel separation method [9] is a method for preparing multiple images such that the pixels of the original image are separated at certain intervals, and superimposing the reconstructed images by rapidly switching the CGHs of each separated image. Separating the pixels reduces undesired interference between side robes of adjacent reconstructed pixels. Figure 2 shows how the pixels of an original image are separated. We group × pixels and then generate the × separated images. Figure 2(a) shows a case where a single group consists of 6 pixels (2 × 3). Figure 2 (b) shows the separated image of pixels assigned "1" in Figure 2 (a). We generate six pixel-separated images as shown in Figures 2(b) to 2(g), and calculate the CGHs for each image.

Proposed method: Time-division reconstruction method
In a holographic projection [7], the reconstructed image is enlarged by setting the pixel pitch of the original image larger than that of the CGH. Thus, we obtain a large reconstructed image without a zoom lens. However, with increased pixel pitch and a fixed number of pixels, the image quality deteriorates.
A time-division reconstruction method that divides the original image into blocks after up-sampling without changing the pixel pitch is proposed and projects the CGHs of the divided blocks by switching them rapidly. Since the pixel pitch is not manipulated, projecting a large-screen reconstruction image with higher quality than the method for changing the pixel pitch is possible [7]. Figure 3 shows an example of calculating CGH when the original image is divided into four after adding a random phase. Since the centers of each divided block have an offset from the center of the CGH, then using off-axis diffraction to calculate the propagation from divided blocks of CGH is necessary. Several off-axis calculations have been proposed [17,18], but this study used that proposed in [18]. Figure 4 shows the reconstruction process. High-speed DMD was used and by switching the CGHs at high speed, we obtained the entire reconstructed image. Figure 5 shows our optical system. For color reconstruction, a time-division color reconstruction method was used [19]. RGB semiconductor lasers were used as light sources and switched the RGB lasers using the synchronizing signals outputted from the DMD. A 4f optical system and a spatial filter were placed between the DMD and the screen to eliminate direct light.

Uneven color correction
CGHs generated using the method outlined in Figure 3 should be binarized before displayed on DMD. However, this study binarized CGHs after the original image was divided, so the relative brightness information between each CGH is lost. Thus, uneven colors occurred in the reconstructed image. Figure 6 displays the reconstructed image using the optical system in Figure 5. Figures 6 (a) and (b) show the original color image and the optical reconstruction image with color unevenness, respectively.
To correct the color unevenness of each divided block closer to its original brightness, we increase the projection time when the divided block is darkened. Conversely, if the divided brock is brighter, the projection time is shortened to eliminate color unevenness. Therefore, the projection time of each divided block is adjusted to perform time-division reconstruction. The projection time used is determined from the average brightness value of the original blocked images. The method is shown in Figure 7 and consists of three steps.
In Figure 7(a), the average brightness values of each divided block of the original image are calculated. In Figure 7(b), an arbitrarily divided block is defined as a reference (the blue rectangular indicates the reference block) , and the ratio of the average value per divided block to the reference block is calculated. In Figure 7(c), multiplying the ratio by the hologram display time of the reference block, the projection times of each divided block can be determined and set in the DMD.

Results
This chapter presents the results of simulations and optical experiments using each method described in section 2. Table 1 shows the parameters for calculating CGHs. The resolution and pixel pitch of the CGHs was determined using DMD. We used DLP LightCrafter 6500 made by TEXAS INSTRUMENTS, which has a refresh rate of 9,523 Hz and displays binary CGHs. Table 2 summarizes the proposed and comparative methods.

Methods Summary
Comparison methods (a) Pixel pitch expansion Used the method of in [7]. The pixel pitch of the original image is four times (30.24 µm) to enlarge the reconstructed image.
(b) Pixel pitch expansion + Multi-random phase This combines (a) above with the multi-random phase method [8].
Six CGHs with different random phases are projected.

(c) Pixel pitch expansion + Pixel separation
This combines method (a) with the pixel separation method [9] and projects six CGHs with different separation images.
Proposed methods

(d) Time-division reconstruction + Uneven color correction
A method that combines time-division reconstruction method with uneven color correction. Up-samples the original image to four times its resolution (7,680 × 4,320 pixels). Divide the original image horizontally and vertically into 4 × 4 blocks. (e)

Time-division reconstruction + Uneven color correction + Multi-random phase
This combines method (d) with the multi-random phase method [8]. Six CGHs with different random phases were projected onto each block. (f)

Time-division reconstruction + Uneven color correction + Pixel separation
This combines method (d) with the pixel separation method [9] and projects six CGHs with different separation images. Figure 9 shows the reconstruction results from the simulation. The size of all reconstructed images is approximately 11.3 cm × 6.5 cm. The methods used in Figures 9 (a)-(f) correspond to (a)-(f) in Table 2. Table 3 shows the image quality evaluation using mean squared error (MSE). The MSE evaluates the average error per pixel between the original and reconstructed images. The smaller the MSE, the closer the reconstructed image is to the original image. The MSE is expressed by

Simulation results
where we denote the original image as ( , ), the reconstruction intensity of the reconstructed image as ( , ), and the size of the original and reconstructed images is × . Comparing Figs. 9 (a) and (d), Fig.9(d) has higher contrast. However, comparing Tables 3(a) and 3(d), there is no significant difference in MSE. This is because the time-division reconstruction method creates boundaries between the divided blocks, which hinders the decrease in the MSE.
Compared to Fig. 9(a), applying the multi-random phase method or pixel separation methods reduces the speckle noise, improving the image quality in each reconstructed image. Subjectively, no significant difference in the quality of the reconstructed image between the multi-random phase and pixel separation methods exists. Table 3 also shows that the MSEs are reduced by applying either the multi-random phase or pixel separation methods. Quantitatively, no significant difference between the MSEs of the multi-random phase and pixel separation method exists. Figures 9(e) and (f) of the proposed methods are subjectively superior to other methods. In particular, Fig. 9(f) has the best MSE.
(e) (f) Figure 9. Reconstruction images by simulation: the methods used in (a) to (f) correspond to Table 2.   Table 2. Fig. 10(d) has a higher contrast than Fig. 10(a). Compared to Fig. 10(a), applying the multi-random phase or pixel separation methods reduces speckle noise and improves the image quality in each recon-structed image. Subjectively, no significant difference in the quality of the reconstructed image using the multi-random phase method and the pixel separation method.

Optical reconstruction results
In Fig. 10(d) obtained using the proposed time-division method by averaging six reconstructed images, the projection time per one reconstructed image was about 5.3 ms. Thus, the projection time per average reconstruction was 31.8 ms in Figs. 10 (e) and (f) since six images were projected in each division block. The image quality of the reconstructed image can be improved by increasing the number of projections for each divided block. However, due to the upper limit used in the frame rate of DMD, it is necessary to determine the number of projections for each divided block so that flicker does not occur in the reconstructed image.
(e) (f) Figure 10. Optical reconstruction results: the methods used in (a) to (f) correspond to Table 2. The brightness and contrast were adjusted to make the reconstructed image clearer.

Full-color reconstruction results
The optical system shown in Figure 5 was used for the full-color reconstruction. We used a method of time-division switching for RGB light sources [19]. Only the proposed method in Table 2(d) (Time-division reconstruction + Uneven color correction) was validated because it requires three times binary CGHs than the proposed method. Figure 11 shows the results of the full-color reconstruction. The brightness and contrast were adjusted to make the reconstructed image clearer. Since the number of divided blocks is 4 × 4, 48 binary CGHs are required to obtain the full-color reconstruction image. The overall size of the reconstructed image was 6.5 cm and 11.6 cm in height and width, respectively. The multi-random phase and pixel separation methods could not be applied because the number of projections increased. To confirm the effectiveness of the proposed method, Fig. 12 shows the reconstructed image using another original image. As in Fig. 11, we obtained a full-color reconstruction image with both reduced noise and color unevenness. Figure 13 shows a reconstructed video "Vegetables" using the proposed method in the Supplementary Section (Video S1) to confirm the effectiveness of the proposed method.

Conclusions
To improve the image quality of the reconstructed image in large-screen holographic projection using binary CGHs, we proposed a time-division method and a method to reduce color unevenness. The average values of the division blocks were used to adjust the display times of the binary CGHs and reduce the color unevenness. Furthermore, the multi-random phase and pixel separation methods combined with the proposed were used to reduce the speckle noise. The effectiveness of the proposed method was confirmed by simulation and optical reconstructions. Additionally, synchronizing the DMD with the RGB light sources showed that the proposed method was capable of full-color reconstruction.