Recent Advances in the Processing and Rendering Algorithms for Computer-Generated Holography

Abstract

Digital holography represents a novel media which promises to revolutionize the way the users interacts with content. This paper presents an in-depth review of the state-of-the-art algorithms for advanced processing and rendering of computer-generated holography. Open-access holographic data are selected and characterized as references for the experimental analysis. The design of a tool for digital hologram rendering and quality evaluation is presented and implemented as an open-source reference software, with the aim to encourage the approach to the holography research area, and simplify the rendering and quality evaluation tasks. Exploration studies focused on the reproducibility of the results are reported, showing a practical application of the proposed architecture for standardization activities. A final discussion on the results obtained is reported, also highlighting the future developments of the reconstruction software that is made publicly available with this work.

1. Introduction

Computer-generated holography is a novel digital media promising to revolutionize the interaction between users and digital content [1]. A holographic signal is a complex wavefield which brings all the characteristics needed to the observer for having a three-dimensional perception of the scene exactly as in reality. The interest in holographic imaging is growing both in academic and industrial areas. The ISO/IEC SC29 WG1 JPEG committee is currently developing a standard for coding different light-field modalities (JPEG Pleno) which comprise the definition of a framework to facilitate the representation of holographicdata [2]. High resolution and wide viewing angles, necessary for realizing high-quality and immersive holographic applications, make holographic imaging very data intensive. This aspect raises a set of open issues related to the areas of data coding, transmission, storage, rendering, and quality evaluation [3].

Digital holography applies not only in display applications, but also in a broad range of scientific fields such as microscopy and interferometry. Microscopy applications are heterogeneous, ranging from refractive index measurements and surface topography [4] to 3D mapping of biological cells for stimulation purposes and acquisition of the fluorescence distribution in a single recording step [5]. In the interferometric field, holography is applied as an extension of the classic techniques, enabling a non-contact, non-invasive analysis of test samples. Also in this case several application can be found, such as cortical bone compression and load tests [6], in which the digital hologram interferometric acquisitions of different bone groups have been exploited for comparison purpose, or the study of wood deformation and strain distribution caused by humidity variations [7].

Typically, holography employs coherent light sources, such as lasers, for the acquisition and reproduction processes. However, to reduce the inherent problem of speckle noise, incoherent techniques have also been developed. There are two main techniques to obtain incoherent holograms [8]. The first is the Optical Scanning Holography (OSH) that employs a laser light source during the acquisition process. Despite the light source being coherent, the acquired hologram can be incoherent: this feature depends on the acquisition geometry [8,9]. The other technique is called Fresnel Incoherent Correlation Holography (FICH), and in this case the light source can be incoherent (such as the sunlight) [8,10].

The acquisition of holograms with optical setups requires particular equipment (such as lasers, beam splitters, waveplates, and acquisition plates) and properly prepared environments. In contrast, digital holography, and in particular the computer-generated holography, allows the simulation of the entire acquisition process with numerical calculations [11]. Also, the reconstruction process of digital holograms can be simulated with numerical algorithms. Recent works have greatly improved digital hologram generation techniques, especially in the context of the tridimensionality, allowing the reproduction of the complete scene depth at one time [12].

However, holographic data digitization raises the challenge of data compression: although there exist advanced coding methods for image and video contents, they cannot reach the same performance on holographic contents, because of the different nature of the data. So far, different coding solutions have been proposed. Data quantization followed by different lossless compression techniques have been analyzed in [13,14], while the performance of different quantization techniques have been compared in [15]. The transform coding with well-known transforms has also been investigated, using the DWT (Discrete Wavelet Transform) [16] and the DCT (Discrete Cosine Transform) [17,18] transforms. To increase the holographic content-awareness during the transform stage, other types of transforms have also been proposed, such as Fresnelets and Gabor wavelets [19,20]. Furthermore, variation of standard still-image and video codecs have been proposed and investigated to improve compression performance on holographic data.Some examples include the JPEG 2000 variation proposed in [21] and the HEVC extension proposed in [22]. Despite the various proposals, no effective solution has been established on digital holographic data coding at the time of writing, and thus research in this field is still very active.

Image-quality evaluation, with objective and subjective experiments, is another crucial aspect in holographic field, often closely linked with data compression. Holographic displays have been improved, focusing on the image quality [23] and on the compactness and portability [24], nevertheless they are not actually ready for consumer production. Due to the lack of high-performance holographic displays, other types of displays are used in these experiments, such as autostereoscopic [25], and light field [26]. Moreover, conventional 2D displays are often adopted for experimenting image reproduction schemes for analyzing the scene from different perspectives and reconstruction distances [27,28,29]. In these cases, the rendering and the image-quality evaluation processes are often carried out with computerized simulations. However, the absence of universally recognized standard methodologies and tools represent an obstacle that hampers the progress in this field and fragments the efforts of the research community.

The main contribution of this work to the state of the art is to provide a reference reconstruction software for digital holograms that can be employed in different experimentation and/or standardization activities on holography. These activities include, but are not limited to: data compression; design and/or evaluation of objective and subjective metrics for holographic image-quality assessment; and design and/or evaluation of image enhancement techniques applicable to holographic images. This tool supports the reconstruction of most holograms belonging to the JPEG Pleno Database [30], i.e., to the best of the authors’ knowledge, one of the biggest and most heterogeneous datasets actually publicly available. Moreover, the proposed software has been embedded in a compression architecture to test several standard codecs (JPEG XT, JPEG XS, JPEG 2000, JPEG LS, PNG, and HEVC) for digital hologram data compression.

The paper is structured as follows. In Section 2 a brief description of the different computer-generated hologram techniques is provided, reporting also some hardware and software tools recently proposed for digital hologram generation and rendering. Section 3 provides a description of the different hologram datasets included in the publicly available JPEG Pleno Database. In Section 4 the reconstruction software proposed in this work is presented and its features are described, while in Section 5 a compression testbed with standard codecs, whose architecture includes the rendering software, is employed to benchmark different standard codecs on holographic data. Finally, in Section 6 the conclusions are drawn, and future work is described.

2. Algorithms for Advanced Processing and Rendering of Computer-Generated Holography

In this section, the main techniques of computerized hologram generation and rendering tools will be described and analyzed, including also examples of publicly available software implementations that are currently developed and maintained. The generation and rendering processes, especially for high-quality holograms, can be very complex and burdensome also for modern hardware architectures. Research in this area is very active not only for improving the final image quality, but also for reducing the computational complexity of the existing techniques.

2.1. Computer-Generated Hologram Generation

The increasing processing power of Central Process Units (CPUs) and Graphics Processing Units (GPUs) has led to the development of several different computer-generated hologram (CGH) techniques. In the following, some of the most common generation methods will be illustrated.

2.1.1. Point-Cloud Approach

In point-cloud approaches, the scene of interest is discretized through a set of points, properly distributed in the tridimensional space. On first approximation, to acquire the hologram of the scene, each point is considered a light-emitting primitive: the hologram is obtained by summing the contribution of all points in the scene on the acquisition plane (the hologram plane). The main advantage of this method is the relatively simple calculations involved during the acquisition process. Another advantage is the possibility of recording not only synthetic scenes, but also real scenes, previously acquired as point clouds. However, when the scene consists of a huge amount of points and occlusion or reflection effects need to be taken into account, the computational cost of the algorithm is typically unacceptable for modern hardware. To speed up the generation process, several techniques have been proposed in the literature [31], such as the Look-Up-Tables (LUTs), originally proposed in [32] that consist of pre-calculated, store-and-reuse parts of the computations performed during the recording process. Another acceleration method is the wavefront-recording plane (WRP) [33], which consists of the interposition of an intermediate plane (the WRP) between the point cloud and the hologram plane. The contributions of the points are first evaluated in the WRP, and then the content of the WRP is propagated through the hologram plane. The original WRP method has also been extended using multiple WRPs [34]. Other recent speed-up proposals rely on wavelet transforms [35] that allow representation of the emitted light with a reduced number of coefficients. This method, called WASABI, has also been combined with the WRP concept to further increase computational speeds [36].

2.1.2. Layer-Based Approach

The layer-based method consists of slicing the scene of interest in different layers. These layers are typically parallel to the hologram plane, crossing the scene along his depth. Each layer is considered a light-emitting primitive and is propagated towards the hologram plane. The hologram of the scene is obtained summing all contributions of all layers involved [37,38]. If compared to other methods, such as the point-cloud approach, the layer-based synthesis turns out to be faster, because the number of layers could be lower than the number of points that describe the scene. Despite this fact, to ensure that the depth of the scene is acquired without visual-perceptible discontinuities, the number of layers required may be high [37], then different acceleration techniques have been proposed to improve the generation process, such as those proposed in [39] and in [40]. Recently, an acceleration method based on LUTs has been proposed [41], enabling the generation of CGH video sequences, minimizing the memory required and the overall hologram generation process complexity.

2.1.3. Polygon-Based Approach

In the polygon-based method, the scene to be acquired is composed of a set of 2D surfaces. Similar to the previous approaches, each surface is considered a light-emitter, and its contribution is propagated on the hologram plane. To generate the hologram of the whole scene, the contributions of all the surfaces are summed on the hologram plane. This method is typically less computationally expensive than the point-cloud synthesis, especially if the complexity of the scene is not high, because the number of polygons needed to approximate the scene is lower than the number of points that would occur in a point-cloud representation [42]. When the complexity of the scene is high (and therefore a large number of polygons are present) more advanced techniques are needed to speed up the calculations [43]; moreover, the hidden-surface removal process can be very computationally intensive, so the search in this field is still very active [44].

2.1.4. Ray-Based Approach

The ray-based approach exploits an intermediate plane between the scene and the hologram plane: the so-called light-ray sampling plane (RS). The light field forming the scene is sampled in the RS plane. This light field often is captured as a set of images showing the scene from different perspectives. These images are Fourier-transformed, and the wavefront obtained with this transform is propagated to the hologram plane, simulating the object wave. Along the path towards the hologram plane, the object wave interferes with a reference wave, and this interference is recorded in the hologram plane to acquire the hologram of the scene. An advantage of this method is that it can be applied to both synthetic and real scenes, acquired through camera arrays that record different views of the scene simultaneously. Furthermore, the reproduction of reflections by surfaces or occlusion effects can be easily taken into account [45]. The computational cost, similar to the other existing methods, can be high for the generation of high-resolution holograms. Examples of solutions proposed to speed up the generation processes involve the use of multiple GPUs [46] or the tile segmentation of the hologram [47].

2.2. CGHs Rendering

The phenomenon of light propagation is crucial in the process of generation and reconstruction of holograms. It can be expressed numerically by means of scalar diffraction theory. Let $U_0(x_0,y_0)$ be the optical field at the hologram plane, placed at $z=0$; the reconstructed wavefield $U_r(x,y)$ in the reconstruction plane, placed at distance z is given by [48]:

$$U_r(x,y)=\frac{z}{j\lambda }\int \int U_0(x_0,y_0)\frac{\exp \left(j\frac{2\pi }{\lambda }r\right)}{r^2}d{x}_{0}d{y}_0,$$

(1)

where j is the imaginary unit, $\lambda$ is the wavelength and $r=\sqrt[]{{(x-x_0)}^2+{(y-y_0)}^2+z^2}$ is the distance between a point in the hologram plane and a point in the reconstruction plane. From Equation (1) different approximations can be derived that are often implemented via software employing one or more bidimensional Discrete Fourier Transforms (2D DFT). Among these approximations, the Angular Spectrum Method (ASM) and the Fresnel transformation [49] are two widely used techniques for digital holograms reconstruction.

Starting from Equation (1), if z is much bigger than the other terms, the approximation $r\approx z$ can be performed, and Equation (1) can be rewritten as:

$$U_r(x,y)=-\frac{jk}{2\pi z}\int \int U_0(x_0,y_0)\exp \left(jkr\right)d{x}_{0}d{y}_0,$$

(2)

where $k=\frac{2\pi }{\lambda }$. The approximation $r\approx z$ has not been performed in the exponent because in this case small variations in r can result in strong variations in $\exp \left(jkr\right)$. The integral in Equation (2) is a convolution:

$$U_r(x,y)=U_0\odot S_h,$$

(3)

where $S_h$ is the kernel:

$$S_h(x,y)=-\frac{jk}{2\pi z}\exp \left(jk\sqrt[]{x^2+y^2+z^2}\right).$$

(4)

To simplify the calculations, the well-known Fresnel approximation can be applied:

$$r=\sqrt[]{{(x-x_0)}^2+{(y-y_0)}^2+z^2}\approx z+\frac{{(x-x_0)}^2+{(y-y_0)}^2}{2z}$$

(5)

that holds if $z^3>>\frac{k}{8}{[{(x-x_0)}^2+{(y-y_0)}^2]}_{max}^2$. In this case, the new kernel will be:

$$S_f(x,y)=-\frac{jk}{2\pi z}\exp (jkz)\exp \left[\frac{jk}{2z}(x^2+y^2)\right].$$

(6)

Using this approximation, the reconstructed wavefield at the reconstruction plane can be calculated with one 2D DFT as [50]:

$$U_r(x,y)=2\pi \exp \left[\frac{jk}{2z}(x^2+y^2)\right]F\left\{U_0(x_0,y_0)S_f(x,y)\right\},$$

(7)

where F indicates the Fourier Transform that can be implemented via software with a 2D DFT. The Equation (7) represents the Frensnel transformation (or Fresnel Method).

An alternative reconstruction method is the ASM. If $U_0(x_0,y_0)$ is Fourier-transformed, its Angular Spectrum can be obtained:

$$A_0(k_x,k_y)=\frac{1}{2\pi }\int \int U_0(x_0,y_0)\exp [-j(k_x{x}_0+k_y{y}_0)]d{x}_{0}d{y}_0,$$

(8)

and obviously with the Inverse Fourier Transform $U_0(x_0,y_0)$ can be retrieved:

$$U_0(x_0,y_0)=\frac{1}{2\pi }\int \int A_0(k_x,k_y)\exp \left[j(k_x{x}_0+k_y{y}_0)\right]d{k}_{x}d{k}_y,$$

(9)

that can also be seen as a set of plane waves propagating in different directions $\mathbf{k}=(k_x,k_y,k_z)$, where $k_z=\sqrt[]{k^2-k_x^2-k_y^2}$, in which every component has amplitude $A_0(k_x,k_y)$. During the propagation at distance z, another phase factor $e^{j{k}_{z}z}$ needs to be taken into account. The reconstructed wavefield at the reconstruction plane will be:

$$U_r(x,y)=\frac{1}{2\pi }\int \int A_0(k_x,k_y)\exp \left[j(k_{x}x+k_{y}y+k_{z}z)\right]d{k}_{x}d{k}_y,$$

(10)

that is the Inverse Fourier Transform of $A_0(k_x,k_y)e^{j{k}_{z}z}$. In summary the reconstructed wavefield is [50]:

$$\begin{array}{cc}\hfill U_r(x,y)& =F^{-1}\left\{A_0(k_x,k_y)\exp \left[j\sqrt{k^2-k_x^2-k_y^{2}z}\right]\right\}\hfill \\ & =F^{-1}\left\{F\left\{U_0\right\}\exp \left[j\sqrt{k^2-k_x^2-k_y^{2}z}\right]\right\},\hfill \end{array}$$

(11)

where $F^{-1}$ indicates the Inverse Fourier Transform and it is assumed that $k^2\ge k_x^2+k_y^2$. As it can be noted from Equation (11), the ASM needs two Fourier Transforms in order to be implemented (thus is more computational-intensive compared to the Fresnel Method), but it does not involve any approximation as in the Frensel Method.

The reconstruction process (but also the generation process) can be an operation that requires not-negligible computational resources, especially if the hologram has high resolution and is in color. Thus, some software implementations have been proposed to perform the reconstruction process (and in some cases also the generation process) of digital holograms in a computation-efficient way. An example of reconstruction software is proposed in [51]. This program is specialized in the reconstructions of holograms belonging to microscopy, Optical Coherent Tomography (OCT), vibrometry, angiography, and plethysmography. The author shows that $1024\times 1024$ and $2048\times 2048$ interferograms can be rendered in real time, exploiting the parallel computing power of modern GPUs. Another proposal is the Computational Wave Optics library for C++ (CWO++) [52], a C++ class library that implements different light-diffraction models for digital hologram reconstruction. The authors also show a CGH hologram generation from a point cloud. Other than exploiting the CPU power, this library allows the use of the GPU to hasten the calculations. Both the cited software implementations are currently maintained and publicly available.

Other than the software development, the research has also been headed towards special-purpose hardware modules. With typical computer hardware configurations, operations such as real-time generation or rendering are hard to achieve, if not impossible in some cases, depending on hardware specifications and CGH features. In [53], a GPU cluster composed of up to 12 GPUs has been shown to render holographic images at 30 frames-per-second in real time. Such solutions, based on general-purpose hardware, provide a high number of cores to speed up calculations through parallelism; moreover, the availability of efficient software libraries released directly from the hardware manufacturer simplifies the development operations. However, the necessary infrastructure to establish these systems and the total energy consumption might be limiting factors in determined use scenarios. Therefore, special-purpose solutions have also been proposed, which are typically smaller in size and require less energy compared to general-purpose hardware [44]. One of the main drawbacks is longer design and development times [44]. An example of a recently proposed solution of this type is the HORN-8 system [54,55], composed of 8 Field Programmable Gate Arrays (FPGAs) mounted on a custom Peripheral Component Interconnect Express (PCI-E) board. In this system, the hologram generation method implemented is the point-cloud approach. The results obtained suggest that using dedicated hardware is a viable alternative to high-parallelized general-purpose hardware, and for hologram rendering in real time. In particular, in [56] the HORN-8 system has been used to calculate and display, at video rate, a CGH generated from a 3D model composed of 390,000 points, highlighting the advantage in terms of performance, compared to general-purpose CPU and GPU hardware. Another recent hardware prototype is showed in [57], which focuses on lightness and portability. The core processor is based on a single FPGA chip. This system exploits the layer-based approach for CGH generation and include a user eye-tracking mechanism.

3. Digital Hologram Reference Datasets

To advance the progress within the context of holography, it is essential to have as many samples as possible on which to carry out the experiments. Moreover, holography is a technique with several and different application fields, ranging from display purposes to microscopy or interferometry. As can be seen also from Section 2.1, there is no single method for CGH generation, and a similar situation can be seen in optical acquisition, in which different setups and methods exist [11]. These aspects make it necessary to also have a good degree of variety in samples types.

Despite the number of different techniques and proposed algorithms present in the literature for digital hologram generation, there are very few publicly available hologram datasets. In the last few years, some research groups have publicly released their own samples, to attempt to mitigate this problem. In [58], the dataset Interfere I has been proposed. The samples are both two-dimensional and three-dimensional and they have been generated with an algorithm created by the authors that supports the occlusion effect. The dataset consists of 5 CGHs. All the holograms are monochromatic and do not support parallax effects; they have pixel pitch equal to $8\mathsf{\mu }$m and resolution equal to $1920\times 1080$, except for the sample 3D Cat, whose pixel pitch is equal to $2\mathsf{\mu }$m and it has resolution of $8192\times 8192$. The wavelength used during the generation process is equal to 632.8 nm. The sources of 2D holograms are two-dimensional images, while the sources of three-dimensional samples are point clouds. The generation algorithm has been improved in [59], including, in addition to occlusion, the simulation of diffuse light reflection from surfaces: the light is spread over a wide angle, providing an extended view of the scene with full parallax support. This dataset is called Interfere II and includes 6 diffuse and 6 specular monochromatic samples, generated from point clouds. The resolution is $8192\times 8192$, the pixel pitch is $1\mathsf{\mu }$m and the generation wavelength 633 nm for all samples. Recently, also Interfere III has been released [60] consisting of 6 color CGH generated from point clouds with an algorithm which extends the Phong illumination model that allows ambient, diffuse, and specular reflection support. The samples have resolutions of $1920\times 1080$ and $2048\times 2048$ with pixel pitch equal to $8\mathsf{\mu }$m, with the exception of Biplane 16k which has resolution of 16,384 × 16,384, pixel pitch of $1\mathsf{\mu }$m and has also full parallax support. The wavelengths used during the generation are equal to 633 nm, 532 nm and 460 nm for the red, green, and blue channels, respectively.

The Institute of Research and Technology B-com has developed a color CGH dataset, generated with two different algorithms, described in [40,61]: the former is a hybrid technique that combines the point-cloud and the layer-based approach and includes occlusion effect support, while the latter is based on multiple 2D-with-depth images of the scene acquired from different viewpoints from which a point cloud is derived. The algorithm takes also into account specular reflections. The dataset comprises 21 holograms, some of them with full parallax support, generated from synthetic scenes. The resolutions range from $1920\times 1080$ to 16,384 × 16,384 with different pixel pitches, from $6.4\mathsf{\mu }$m to $0.4\mathsf{\mu }$m. Two video sequences at different resolutions of real scenes are also proposed.

Other than the CGHs dataset, there are also few examples of optically acquired hologram datasets. The EmergImg Holographic Database [62] is one of them, and it includes holograms of real objects acquired with the phase-shifting technique. Each hologram derives from the acquisition and combination of four interferograms. The database is composed of two sample sets. The first set (v1) includes three $972\times 972$ samples with $4.4\mathsf{\mu }$m pixel pitch and a video sequence of 54 holograms at $600\times 600$ resolution. The second set (v2) comprises six $2588\times 2588$ samples with pixel pitch $2.2\mathsf{\mu }$m. The samples are all monochromatic and do not support relevant parallax effects, also because there are actually strong limits in resolution and pixel pitch of currently available optical acquisition hardware, if compared to CGH acquisition techniques.

Microscopy and tomography are two other important fields in which holography has been demonstrated to provide important improvements over standard techniques [63,64], and currently different commercial hardware apparatus and related rendering software are available. Despite this, there is not a big availability of these types of hologram. One example of a publicly available dataset is the Tomocube [65], which comprises four monochrome samples with resolutions from $512\times 512$ to 24,478 × 16,641 and pixel pitch from $0.08\mathsf{\mu }$m to $0.366\mathsf{\mu }$m.

Although some holographic datasets are actually publicly available as reported above, it is necessary that their number and variety in type continue to grow: perhaps even more than other imaging techniques, research areas within holography such as data compression, image rendering, enhancement, and objective and subjective quality assessment require a high number of samples to provide results with an adequate generality.

4. Digital Hologram Rendering and Quality Evaluation

4.1. Rendering Algorithm

As described in the Section 2.2, currently there are some freely accessible software applications that allow the rendering of digital holograms. From the research point of view, having a wide variety of available data is crucial, as stated in Section 3, but, especially for those who approach the study of this imaging technique for the first time, it is also important to establish a clear correspondence between the operations that the software is able to do, and the samples on which these operations can be performed. Holography is an area in which standardization activities are in their initial stages [2], and a standard data representation format is not available yet. Moreover, holography has different application fields such as display, tomography, microscopy, and interferometry and often every application requires specific processing steps that are not necessary or useful for another application. It is, therefore, important to clearly specify on which type of data a reconstruction tool has been tested and therefore can be used.

This work proposes a tool for the reconstruction of digital holograms dedicated to display applications, designed to simplify, even for non-expert users, the reproduction of most of the currently accessible and downloadable holographic datasets. The architecture of the proposed tool is shown in Figure 1 and has been developed in MATLAB language.

The first block, Load Hologram, allows the user to load the hologram in memory through a window dialog. The currently supported datasets are:

• B-com, 8-bit and 32-bit format, (the hologram sequences reconstruction is allowed frame-by-frame only);
• Interfere, I, II, and III;
• EmergImg, v1 and v2, (the hologram sequences reconstruction is allowed frame-by-frame only).

Once the complex matrix representing the hologram is loaded, the next step is to set the view on the scene to be reconstructed: this operation is performed by Set Viewpoint. A specific view on the scene is obtained selecting only a portion of the entire hologram, thus generating the so-called “synthetic aperture” (e.g., if the aperture is positioned on the top-right side of the hologram, the user obtains the top-right view of the scene). To facilitate the aperture positioning, this tool employs two angles, as shown in Figure 2a, in which $d_r$ is the reconstruction distance.

The horizontal position of the aperture is declared through the angle $\theta$, defined as $\theta =\arctan (x_c/d_r)$, while the vertical position is set through the angle $\varphi$, defined as $\varphi =\arctan (y_c/d_r)$. A third angle, $\psi$, is used to set the aperture’s dimension as shown in Figure 2b and proposed in [66]. The reconstruction distance and the three angles for aperture position and size definition are declared by the user before launching the tool. The synthetic aperture can be apodized with a bidimensional Hanning window as proposed in [26] to reduce strong diffraction effects at the aperture’s borders that may occurs in the reconstruction process. The synthetic aperture setting and its apodization are optional steps that the user can decide whether or not to perform. The hologram reconstruction is performed in the Numerical Reconstruction (Num. Recons.) block: the additional reconstruction parameters such as wavelength(s), pixel pitch and reconstruction method can be set by the user by means of external configuration files, in order to avoid re-declaration of these parameters during subsequent reconstructions that share equal parameters. Currently, the tool implements two reconstructions methods, namely the ASM and the Fresnel Method. Subsequently, the Absolute (ABS) block extracts the complex magnitude of the reconstructed wavefield. At this stage, the reconstructed image can be enhanced with a linear histogram stretching after a percentile clip to remove extreme values, as suggested in [26]. This optional process is performed on the Clipping & Stretch. block, and the user can select the percentile value at which the clipping is executed. Finally, the reconstructed image is generated as standard PNG image.

In Figure 3, two reconstructed images are reported. Figure 3a shows Dices 8K of the B-com dataset, reconstructed with the ASM at a distance $z=0.004$ m with a synthetic aperture of size $\psi =6^{\circ }$ positioned at $\theta =-7^{\circ }$, $\psi =3^{\circ }$. Figure 3b shows Skull of the EmergImg v2 dataset, reconstructed with the Fresnel Method at a distance $z=0.169$ m with synthetic aperture size equal to the hologram size.

4.2. Quality Evaluation

Digital holograms, whether they are optically acquired or computer-generated, typically include much more data if compared with standard image or video contents. Data compression is therefore an important milestone for the evolution of this imaging technology, as much as the development of high-quality holographic displays and the improvement of generation techniques. Without an efficient coding technique, it is impossible to process and transmit holographic data in a similar way as other contents, such as images and videos. As already pointed out in Section 1, standard image or video codecs are not optimal solutions, since they are designed for completely different data types. Nevertheless, during the design, development and test of new coding solutions, dedicated for holographic data, it is also important to compare the obtained results with current state-of-the-art codecs, to have a common reference. In this work, a comparative evaluation of different standard coding solution has been carried out with the architecture reported in Figure 4 that will be described in the following, and that includes the rendering algorithm described in Section 4.1.

The compression architecture (Figure 4) requires that the input raw hologram is represented in the complex-valued form, since the majority of dataset currently publicly available are expressed in this form. However, it is easily adaptable to other representation forms as phase-only or double-phase. Moreover, in this work only the algebraic form is considered, because it is known from previous state-of-the-art works that it shows better compression performance than polar form [22,62,67].

From the raw complex hologram, represented by the Complex raw data block, the real and imaginary components are extracted in the Split in components block. The two components are two distinct real-valued matrices. In the Range mapping & Quantization block these two matrices are quantized with 16 bits per sample (bps), since most of current standard codecs support integer data only. Subsequently, the two quantized components are coded (Coding block) and then decoded (Decoding block) with a standard codec. At this stage, the first quality evaluation is carried out, in the so-called hologram domain. The hologram’s real and imaginary components are compared before and after the compression with the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) objective metrics, taking the component before the coding stage as reference. After the decoding process, the Complex regeneration block takes the two components as inputs and merge them again in a single complex matrix. The reconstruction process is then performed from the decoded data (F’), on the right, but also from the original raw data (F), on the left. The Reconstruction & Rendering block is essentially the algorithm described in Section 4.1, with a custom Load Hologram block, in order to load the input data automatically, without user intervention. The two reconstructed images are finally compared with the PSNR and SSIM metrics in the so-called reconstructed domain, taking the image originating from the raw data as reference.

5. Exploration Studies

In this work, the compression architecture described in Section 4.2 has been applied to two different holograms, Dices 8K of the B-com dataset and Skull of the EmergImg v2 dataset. The former is a full-parallax color hologram, numerically computed from a synthetic scene, while the latter is grayscale and it has been physically acquired through an optical setup. This choice allows observation and comparison of the performance of the codecs on two relatively different types of data. More details about the samples are available in Section 3. The experimentation involves the use of 5 still-image codecs, namely JPEG XT (Libjpeg implementation [68]), JPEG XS [69], JPEG 2000 (Kakadu Software implementation [70]), tested in lossy mode, in addition to JPEG LS (Libjpeg implementation [71]) and PNG (FFmpeg implementation [72]), which are lossless codecs. These are among the most recent and used codecs for standard images. Moreover, the HEVC (HM 16.19 [73]) video codec is employed for lossy coding, in Intra mode. This is useful for verifying whether the performance of HEVC that in previous works was demonstrated to be higher than some standard codecs on grayscale CGH [22,67] is also confirmed with color CGH samples and with respect to the JPEG XT and JPEG XS codecs. The exploration studies have been performed on a Windows 10 Pro machine, equipped with a dual Intel Xeon E5-2640-v4 CPU, 64 GB of RAM and a Nvidia Quadro P5000 GPU.

Table 1. Lossy compression results.

Hologram	HEVC		JPEG 2000		JPEG XS		JPEG XT
	Total Rate [bps]	Compressed Size [MB]	Total Rate [bps]	Compressed Size [MB]	Total Rate [bps]	Compressed Size [MB]	Total Rate [bps]	Compressed Size [MB]
Dices 8K (Source Size: 805.31 MB)	2.16	18.12	0.5	4.19	0.5	4.19	6.15	51.59
	5.42	45.47	1.99	16.69	2	16.78	8.7	72.98
	8.26	69.29	3.95	33.13	4	33.55	12.5	104.86
	11.83	99.24	7.95	66.69	8	67.11	17.22	144.45
	15.67	131.45	11.98	100.49	12	100.66	20.97	175.91
	19.84	166.43	15.73	131.95	16	134.22	26.95	226.07
Skull (Source Size: 26.79 MB)	0.45	0.38	0.49	0.41	0.5	0.42	2.59	2.17
	1.16	0.97	1.99	1.67	2	1.67	3.51	2.94
	1.92	1.61	3.97	3.32	4	3.35	4.69	3.93
	3.13	2.62	7.90	6.61	8	6.70	6.31	5.28
	4.65	3.89	11.99	10.04	12	10.05	7.40	6.19
	6.03	5.05	13.90	11.64	16	13.39	9.04	7.57

shows the numerical results obtained after the coding process. For each codec, the compressed total bitrate (the sum of real and imaginary bitrates) and the corresponding compressed file size are reported.

Figure 5a,b show the rate-distortion curves in terms of mean PSNR, calculated in the hologram domain, for Dices 8K and Skull respectively.

Since for both samples the results obtained for the real and imaginary parts are almost identical, the means of these results are shown for each sample. Analyzing the graph in Figure 5a, related to Dices 8K, it can be noted that the HEVC has better performance than the other codecs. Among the still-image codecs, the best is JPEG 2000, followed by the most recent JPEG XS, and by the JPEG XT. The results for Skull in terms of mean PSNR are shown in Figure 5e: it can be noted that with this sample there is not a significant difference between HEVC and JPEG 2000. Moreover, the JPEG XS is the worst codec, perhaps even penalized by the fact that there is not a specific profile for grayscale inputs (dev profile has been used). The mean SSIM results obtained in the hologram domain are shown in Figure 5b (Dices 8K) and in Figure 5f (Skull): the SSIM metric confirms the results obtained in terms of PSNR, even if after nearly 8 bps for Dices 8K and 2 bps for Skull there is no significant distinction between codec performance with this metric. The analysis performed in the hologram domain provides information on how much the data have been altered by the encoding process. However, another fundamental aspect is to establish how much this data alteration, i.e., the coding process, affects the holographic image quality. Therefore, analysis in the reconstructed domain has been performed.

The results obtained in the reconstructed domain, in terms of PSNR, are shown in Figure 5c,g, for Dices 8K and Skull respectively. It can be noted that for both samples the results are very similar to those obtained in the hologram domain. Based on these results, the use of PSNR metric in the hologram domain can be effective in predicting the results that will be obtained on the holographic image, after the hologram reconstruction. The SSIM results (Figure 5d,h) are in accordance with PSNR results also in this domain. It can be also observed that at high bitrates (hence when the holographic image quality is also high), the SSIM discriminates the performance of the various codecs less than the PSNR.

From the inspection of the obtained results it can be observed that most of the time HEVC performs better than the other codecs. This is probably because HEVC employs very complex prediction tools that compare several different possible coding solutions based not only on block transform and quantization but also on a so-called intra-prediction aimed at finding redundancies between adjacent blocks. Moreover, the results show that the performance of JPEG2000 are similar to those of HEVC. These results suggest that the wavelet transform adopted by JPEG2000 can provide a compact representation of the information of the holographic signal. The lower performance of JPEG XS with respect to HEVC and JPEG2000 are mainly related to the low complexity design of the JPEG XS encoder. While for JPEG XT the main reason for the lower performance is that for providing backward compatibility with the legacy JPEG standard, it makes use of JPEG coding tools.

The results obtained using the lossless codecs are shown in

Table 2. PNG compression results.

Hologram	Prediction Type	Real Part [bps]	Imaginary Part [bps]	Total Rate [bps]	CR
	None	46.1	45.95	92.04	1.04
	Sub	45.36	45.2	90.56	1.06
	Up	45.53	45.37	90.89	1.06
Dices 8K	Avg	44.76	44.61	89.37	1.07
	Paeth	44.72	44.57	89.29	1.08
	Mixed	44.72	44.57	89.29	1.08
	None	11.71	11.68	23.4	1.37
	Sub	11.4	11.36	22.76	1.41
	Up	11.37	11.33	22.7	1.41
Skull	Avg	11.17	11.12	22.29	1.43
	Paeth	11.13	11.09	22.22	1.44
	Mixed	11.13	11.09	22.22	1.44

(PNG codec) and in

Table 3. JPEG LS compression results.

Sample Name	Real Part [bps]	Imaginary Part [bps]	Total Rate [bps]	CR
Dices 8K	40.53	40.28	80.82	1.19
Skull	10.06	9.98	20.04	1.60

(JPEG LS codec). In these tables, also the Compression Ratio (CR) is reported, defined as

$$CR=\frac{quantizedhologramsize}{compressedhologramsize}.$$

(12)

The quantization process (carried out in the Range mapping & Quantization block, Figure 4) maps the raw real-value representation of the hologram to a positive fixed-point representation, allocating in this case 16 bits to store each real (or imaginary) sample of the output Re (or Im) matrix.

The data quantization process is not part of the codec itself, and it contributes in a non-negligible manner to reduce the data size. The PNG codec (Table 2) shows negligible differences in performance when the prediction mode changes. The CR varies between 1.04 and 1.08 for Dices 8K and between 1.37 and 1.44 for Skull. Especially with the first sample (color full-parallax CGH hologram), the PNG coding is thus ineffective, because the CR is very close to 1 (no compression). The JPEG LS shows slightly higher performance, with CR of 1.19 for Dices 8K and 1.60 for Skull. The real and imaginary components show, with both codecs, very similar results in terms of bps, as already seen during the lossy experiments. However, the overall lossless coding results are lower than the typical performance expected from these codecs (during standard image compression). Moreover, it can be noted that the color CGH is more difficult to compress than the optically acquired one.

6. Conclusions

In this work, after an initial introduction to digital holography and the challenges that it currently poses, different CGH generation techniques have been illustrated, together with some dedicated hardware solutions and software implementations that are currently publicly available. Subsequently, a reconstruction software implementation has been illustrated and made publicly available (http://digitalmedia.diee.unica.it/nrsh) along with a detailed user guide. This software, unlike other open-access implementations, aims to be a reference reconstruction software that can be used not only by those approaching digital holography for the first time, but also during the various exploration studies that will be conducted in the future, such as data compression and/or evaluation and enhancement, of rendered image quality. For these reasons, the proposed software focuses on the reconstruction of holograms belonging to the JPEG Pleno Database, which contains different types of publicly accessible holograms.

A practical application of the proposed software has been shown in the context of data compression, in which a benchmark of different standard codecs for images and videos has been performed on two holograms of different types. Although the performance on the grayscale optical-acquired sample are nearly identical to JPEG 2000, HEVC has been confirmed to be the standard lossy codec with the best performance, as already pointed out by the state of the art. Thus, HEVC could be used in subsequent exploration studies as codec anchor, to evaluate new compression techniques. The most recent lossless codecs have instead shown great difficulty in compressing digital hologram data, especially on the computer-generated sample under test, obtaining CRs often close to 1 and in any case lower than 2. Finally, it has emerged that real and imaginary components behave very similarly during compression, both on CGH and on optically acquired holograms.

Future developments of this work will include: updates to the set of the software-supported samples, as soon as new holograms are added to the JPEG Pleno Database; support of GPU hardware during the rendering process, to improve performance compared to the CPU-only rendering, as currently happens; and inclusion of additional options for image-quality enhancement, in particular aimed to reduce the speckle noise, of which holographic images are often affected.