Compression of Phase-Only Holograms with JPEG Standard and Deep Learning

Abstract

It is a critical issue to reduce the enormous amount of data in the processing, storage and transmission of a hologram in digital format. In photograph compression, the JPEG standard is commonly supported by almost every system and device. It will be favorable if JPEG standard is applicable to hologram compression, with advantages of universal compatibility. However, the reconstructed image from a JPEG compressed hologram suffers from severe quality degradation since some high frequency features in the hologram will be lost during the compression process. In this work, we employ a deep convolutional neural network to reduce the artifacts in a JPEG compressed hologram. Simulation and experimental results reveal that our proposed “JPEG + deep learning” hologram compression scheme can achieve satisfactory reconstruction results for a computer-generated phase-only hologram after compression.

1. Introduction

Computer Generated Holography (CGH) allows the recording and reconstruction of a desired complex light wavefront including both amplitude and phase information. With the development of computer and optical technologies, CGH has been extensively applied in many fields such as three-dimensional dynamic holographic display [1,2,3], holographic projection [4,5,6,7], virtual and augmented reality [8,9], optical tweezers [10] and optical information security [11,12]. The research works on CGH generation, conversion and compression algorithms receive much attention in the past decade. For example, the enormous amount of calculation in the generation of a complex hologram from a 3D object model consisting of many points is a huge challenge and the fast CGH calculation problem has been investigated from different perspectives [13,14,15,16,17]. Another major concern is that holographic display devices such as spatial light modulator (SLM) usually cannot display both the amplitude and phase part of a complex hologram simultaneously. Fast and high-quality phase-only hologram calculation from an object image is favorable for commonly used phase-only type SLMs. A phase-only hologram (Figure 1) can be calculated from an object image with various methods such as Gerchberg–Saxton iterative algorithm [18,19,20], error diffusion algorithm [21,22,23,24], random phase mask method [25,26,27] and down-sampling mask method [28,29]. Compared with other methods, error diffusion method [21,22,23,24] has several advantages including no forward and backward field propagation, reduction of high frequency noise in the result and direct converting a complex hologram into a phase only one [8]. In this paper, the error diffusion method is employed for the computer generation of phase-only holograms from object images.

In addition to the numerical generation of complex or phase-only holograms, the compression of hologram data is a critical issue. Since the unit pixel size of a hologram is typically rather small (a few micrometers), the entire hologram may contain a huge number of pixels and demands considerable storage cost and transmission bandwidth. It is necessary to develop efficient hologram compression algorithms to reduce the size of data representing a hologram. In the past decade, much research effort has been made on hologram compression [30,31,32,33,34] and a literature review can be found in [30]. In essence, a hologram can be regarded as a special kind of two-dimensional (2D) image. For natural 2D images (e.g., photographs), standardized image compression algorithms such as JPEG [35] have been widely employed. However, some image properties of holograms significantly differ from photographs. For example, a photograph is usually locally smooth while a hologram consists of many high-frequency fringe patterns. Consequently, JPEG compression often cannot give optimal compression performance for holograms [30]. Instead of directly adopting JPEG standard, many customized compression algorithms are proposed for holograms such as Fresnelets [31], Wavelet-Bandelets Transform [32], vector quantization [33], enhanced wavelet transform [34] and rational covariance extension approach [36,37]. These customized hologram image compression methods can yield very satisfactory performance for certain types of holograms. However, on the downside, these customized algorithms are not compatible with commonly used JPEG standard, which is supported very extensively by almost every computer system and digital device while these customized compression methods are not very generally adopted. A hologram compression method is very favorable if it has compatibility with JPEG standard and can achieve good compression efficiency at the same time. In this work, we propose a phase-only hologram compression and decompression scheme incorporating both JPEG standard and deep learning post-processing. In this scheme, the compression steps are identical to JPEG compression standard and the decompression steps are implemented with JPEG decompression plus enhancement by a deep convolutional network.

This article is organized as follows. In Section 2, the error diffusion method for phase-only hologram generation is briefly described. In Section 3, the working principles of JPEG image compression standard are discussed and our proposed artifact reduction scheme for JPEG compressed holograms using deep convolutional network is presented. In Section 4, simulation and experimental results are demonstrated to verify the effectiveness of our proposed scheme. In Section 5, a brief conclusion is provided.

2. Computer Generated Phase-Only Hologram with Error Diffusion Method

Error diffusion algorithm [21,22,23,24] is a non-iterative and high-quality method for calculating a phase-only hologram from an object image on computer. The working principles of phase-only hologram calculation with error diffusion method are described below. To start with, $O(x,y)$ denotes the object image and a complex Fresnel hologram $H(x,y)$ can be calculated based on the Fresnel diffraction formulas illustrated by Equations (1) and (2).

$$f(x,y;z)=\frac{exp\left(i\frac{2\pi z}{\lambda }\right)}{i\lambda z}exp\left[i\frac{\pi \left(x^2+y^2\right)}{\lambda z}\right]$$

(1)

$$H\left(x,y\right)=O\left(x,y\right)\ast f(x,y;z)$$

(2)

where $f(x,y;z)$ denotes the impulse function for Fresnel transform, $\lambda$ denotes the wavelength of illumination light and $\mathit{z}$ denotes the distance between the object image plane and hologram plane. As shown in Figure 2, a Fresnel zone plate can be generated at the hologram plane for each object point with a Fresnel impulse function. The final calculated hologram is a superposition of the Fresnel zone plates from all the points (or pixels) in the object image. More detailed explanation about the Fresnel hologram calculation can be found in [13,14,15,16,17].

The complex hologram $H(x,y)$ can be converted to a phase-only hologram $P(x,y)$ with error diffusion algorithm [21,22,23,24] to achieve high-quality holographic display on a phase-only spatial light modulator. In the error diffusion method, the complex value of each hologram pixel is forced to be unity amplitude and the resulting complex error is diffused to neighboring pixel values. The following operations (Equations (3)–(8)) are performed on each individual holographic pixel sequentially in row-by-row and column-by-column scanning manner, as illustrated in Figure 3.

$$E\left(x_j,y_j\right)=H\left(x_j,y_j\right)-P\left(x_j,y_j\right)$$

(3)

$$H\left(x_j,y_j+1\right)\leftarrow H\left(x_j,y_j+1\right)+w_{1}E\left(x_j,y_j\right)$$

(4)

$$H\left(x_j+1,y_j-1\right)\leftarrow H\left(x_j+1,y_j-1\right)+w_{2}E\left(x_j,y_j\right)$$

(5)

$$H\left(x_j+1,y_j\right)\leftarrow H\left(x_j+1,y_j\right)+w_{3}E\left(x_j,y_j\right)$$

(6)

$$H\left(x_j+1,y_j+1\right)\leftarrow H\left(x_j+1,y_j+1\right)+w_{4}E\left(x_j,y_j\right)$$

(7)

$$P\left(x_j,y_j\right)=\mathit{Phase}\left[H\left(x_j,y_j\right)\right]$$

(8)

where $P(x_j,y_j)$ denotes the phase-only pixel value at position $(x_j,y_j)$ after the complex pixel value $H(x_j,y_j)$ is phase truncated (the magnitude is forced to be unity), $E(x_j,y_j)$ denotes the complex error and four different weighting coefficients, $w_1=7/16$, $w_2=3/16$, $w_3=5/16$ and $w_4=1/16$, are imposed on four different directions, as shown in Figure 3.

After each pixel in a complex hologram is processed, the entire hologram will become a phase-only type hologram. More details about the error diffusion methods for phase-only hologram generation can be found in [21,22,23,24]. In practice, bi-directional error diffusion, a slightly modified version of the above unidirectional error diffusion, can yield slightly better performance [21].

3. JPEG Image Compression and Proposed Artifact Reduction Scheme by Deep Convolutional Network

Each pixel in a phase-only hologram has an intensity value ranging from 0 to 2 $\pi$ and a phase-only hologram can be regarded as a gray-scale intensity image. Various image compression algorithms can be attempted for the compression of phase-only holograms.

JPEG is a very commonly used lossy compression scheme for digital images, which was firstly proposed in 1992 by Joint Photographic Experts Group [35]. In JPEG scheme, the original image (e.g., a gray-level photograph or a phase-only hologram) is first divided into 8 × 8 pixel blocks and each individual block undergoes discrete cosine transform (DCT). Then, the coefficients in the transformed domain of each block are quantized with more quantization levels for low frequency components and less quantization levels for high frequency components. Subsequently, zigzag coding, entropy coding and Huffman coding are performed on the quantized coefficients. Finally, the original image becomes a compressed bit stream with much smaller data size than the original uncompressed image. A reconstructed image with certain image degradation can be obtained when the JPEG binary bit stream is decompressed with the inverse procedures as compression.

JPEG compression can significantly reduce the data size of an original natural image and only introduce minor quality degradation in the decompressed image. The reason is that a natural image (e.g., a photograph) is usually locally smooth and most of its information is concentrated in the low frequency coefficients of DCT spectrum. The discarded high frequency information in the quantization step has negligible effect on the image quality. However, phase-only holograms have substantially different image characteristics compared with photographs [38]. One feature of a hologram is that it contains many high frequency fringe patterns. If JPEG compression is directly applied to a phase-only hologram, the decompressed result after compression will suffer from heavy damage and the holographically reconstructed image from the JPEG compressed hologram will be severely degraded as well.

In this work, we employed deep convolutional neural networks to reduce the artifacts of JPEG compressed phase-only holograms. In the past few years, deep learning methods receive much attention and exert considerable impact in many fields such as image quality enhancement [39,40,41,42]. A previous work [39] proposed a 20-layer network DnCNN to reduce the Gaussian noise on the noisy images. Although DnCNN is designed for denoising task, it shows promising performance on image compression artifacts reduction and super-resolution as well. Assisted by the prior knowledge, a previous work [40,42] proposed denoiser-based and total variation (TV)-based solutions, respectively for image restoration task. Very recently, deep learning has been introduced to the holographic research area and gained success in different applications [43,44,45,46,47,48,49].

The network structure employed in our task is illustrated in Figure 4a, based on the previous works of restoring JPEG compressed photographs [50]. The overall flowchart of our proposed “JPEG + deep learning” hologram compression scheme is illustrated in Figure 4b. The proposed CNN consists of four convolutional layers, and each layer has a specific function. The first layer is used for feature extraction, and with $9\times 9$ filter size it generates 64 feature maps. By adding a bias term to the outputs and employing ReLU as the activation function, the convolution operation in this layer can be formulated as:

$$F_1\left(\mathbf{I}\right)=ReLU(W_1\ast \mathbf{I}+B_1)$$

(9)

where W and B represent the weights and biases, respectively; “*” denotes the convolution operation; and F represents the nonlinear mapping process. Different from the previous approach [51] where zero-padding is not adopted, in the proposed network, all the filters whose size is larger than 1 adopt the zero-padding strategy to maintain the size of the reconstructed images unchanged.

Since the features extracted by the first layer are in low quality, the second convolutional layer is used for feature enhancement, and, with $7\times 7$ filter size, it generates 32 feature maps. Because more nonlinear mappings are involved, more potential features can be extracted, and the third layer is used for nonlinear mapping. With $1\times 1$ filter size, it generates 16 feature maps. The last layer is used for image reconstruction and the corresponding filter size is $5\times 5$. After the first three convolutional layers, features in the inputs are successfully extracted and enhanced by optimizing the output of the network to generate the final recovered image. The operation of these layers can be formulated as:

$$F_i\left(\mathbf{I}\right)=ReLU(W_i\ast F_{i-1}\left(\mathbf{I}\right)+B_i)$$

(10)

where i indicates the ith convolutional layer; $F_i\left(\mathbf{I}\right)$ and $F_{i-1}\left(\mathbf{I}\right)$ are the outputs of the ith and $(i-1)$th convolutional layer, respectively; and $W_i$ and $B_i$ represent the weights and biases for the ith layer, respectively. Since there is no activation function after the last conventional layer, the final restored images can be expressed as

$$F_4\left(\mathbf{I}\right)=W_4\ast F_3\left(\mathbf{I}\right)+B_4$$

(11)

In addition, aiming to learn the optimal values for the weight and bias of each layer, the L2 distance between the reconstructed hologram images $F(\mathbf{I};\Theta )$ and corresponding uncompressed ground truth hologram images ${\mathbf{I}}_{GT}$ needs to be minimized. Given a set of m uncompressed images ${\mathbf{I}}_{GT}^i$, the corresponding compressed images ${\mathbf{I}}^i$, the Mean Squared Error (MSE) loss function is:

$$Loss(\Theta )=\frac{1}{m}\sum _{i=1}^m{\parallel F({\mathbf{I}}^i;\Theta )-{\mathbf{I}}_{GT}^i\parallel }_2^2$$

(12)

where $\Theta$ contains the parameters of the network, including both weights and biases. The network weighting parameters are trained separately for phase-only holograms generated at different distances. Hence, the artifacts caused by JPEG compression in phase-only holograms such as high frequency noise and blocking effect can be suppressed by the proposed convolutional network.

4. Results and Discussion

In our work, ten pairs of compressed and uncompressed phase-only holograms ($1024\times 1024$ pixel size) calculated from ten different object images (512 × 512 pixel size) with error diffusion method were employed as the training holograms (Figure 5). Two compressed phase-only holograms generated from “Pepper” and “Cameraman” images, respectively, were employed as the test holograms (Figure 6). Data augmentation was performed: firstly, we rotated the training images by ${90}^{\circ }$, ${180}^{\circ }$ and ${270}^{\circ }$, respectively; and, secondly, we flipped them horizontally. Taking into account the training complexity, the small patch training strategy was adopted, so that training images were split into $31\times 31$ patches with the stride of 10 and there were $960,384$ training sub-images in total. Since different holograms usually have many similar image blocks and fringe patterns, the replication properties in hologram images allowed a small number of examples to contain most universal hologram features, which has been revealed in previous works [52]. Hence, the total training sub-images was sufficient for our proposed four-layer network. In this work, stochastic gradient descent backpropagation solver was used with the batch size 128. The initial learning rate was $0.0001$ and decayed every five epochs by a factor of 10. There were 25 epochs in total. We used the deep learning platform caffe [53] on an NVIDIA GTX TITAN X GPU with 3072 CUDA cores and 12 GB of RAM to implement all the operations in our network. The training time of a four-layer network was 5 h and the test time for an image with $512\times 768$ was around 0.32 s.

In the computer calculation of phase-only holograms, the optical wavelength of illumination light was set to be 532 nm, the pixel size was 8 $\mathsf{\mu }$m and the distance between object image and hologram plane was 0.3 m or 0.5 m. The ten training holograms and two testing holograms were then compressed by JPEG when the quality factor is set to 1. The JPEG compressed hologram was restored by our proposed deep convolutional network. One example of the original hologram, JPEG compressed hologram and restored hologram with our proposed scheme is shown in Figure 7. It can be observed that some high frequency patterns are deteriorated in the hologram after JPEG compression and restored to certain extent after the quality enhancement using our proposed network.

The reconstructed images from original uncompressed holograms, JPEG compressed holograms and restored holograms were compared in numerical simulations. The reconstructed images are shown in Figure 8. The PSNR, SSIM [54], multi-scale SSIM (MS-SSIM) [55], visual information fidelity (VIF) [56] and information fidelity criterion (IFC) [57] values of the reconstructed results from compressed holograms and restored holograms, in comparison with the reconstructed results from uncompressed holograms, are presented in

Table 1. PSNR (dB), SSIM, MS-SSIM, VIF and IFC values of reconstructed results from JPEG compressed holograms and restored holograms with our proposed scheme.

		Cameraman		Pepper
		0.5 m	0.3 m	0.5 m	0.3 m
Compression ratio		7.2113	7.1111	7.2113	7.1608
Reconstructed image from compressed hologram	PSNR	19.10	17.83	18.92	17.64
	SSIM	0.1651	0.0967	0.2007	0.1036
	MS-SSIM	0.6091	0.4628	0.6907	0.5252
	VIF	0.3946	0.2183	0.6396	0.3438
	IFC	0.4522	0.2519	0.5543	0.2835
Reconstructed image from restored hologram	PSNR	28.86	26.86	29.88	27.16
	SSIM	0.6036	0.4465	0.6767	0.4852
	MS-SSIM	0.8798	0.8022	0.9138	0.8343
	VIF	0.5378	0.4316	0.6841	0.6306
	IFC	0.9027	0.5098	1.2064	0.6379

. The compression ratio is approximately 7 and data size of original hologram, 1024 KB (1 MB), is reduced to around 142 KB to 144 KB after JPEG compression. All these assessment values reveal that our proposed scheme can significantly enhance the reconstructed image quality from JPEG compressed holograms.

Our proposed scheme was further verified in optical experiments and the calculated holograms were optically reconstructed using the optical setup [6] shown in Figure 9. The hologram was reconstructed with a phase-only Holoeye PLUTO spatial light modulator (SLM). The system parameters were the same as the numerical simulation (wavelength: 532 nm; pixel size: 8 $\mathsf{\mu }$m; hologram size: 1024 × 1024 pixels; object-hologram distance: 0.3 m/0.5 m). The phase-only holograms loaded into the SLM were assigned with an additional carrier phase to transversally shift the desired reconstructions away from the un-diffracted (zero-order) beam component. The projected beam from the SLM is transmitted through a 4-f optical filtering system containing a low-pass iris with 3.3 mm diameter to block the zero-order noise. The optically reconstructed results are shown in Figure 10. It can be observed that the quality of reconstructed images after hologram enhancement with deep learning are improved, compared with the ones from compressed holograms without restoration.

5. Conclusions

It is essential to reduce the enormous amount of data representing a computer-generated phase-only hologram in the processing, transmission and storage process. JPEG photograph compression standard can be attempted for hologram compression with an advantage of universal compatibility, compared with customized hologram compression algorithms. Deep convolutional networks can be employed to reduce the artifacts for a JPEG compressed hologram. Simulation and experimental results reveal that our proposed “JPEG + deep learning” hologram compression scheme can maintain the reconstructed image quality when the data size of original hologram is significantly reduced.