KULF-TT53: A Display-Specific Turntable-Based Light Field Dataset for Subjective Quality Assessment

Abstract

Light field datasets enable researchers to conduct both objective and subjective quality assessments, which are particularly useful when acquisition equipment or resources are not available. Such datasets may vary in terms of capture technology and methodology, content, quality characteristics (e.g., resolution), and the availability of subjective ratings. When contents of a light field dataset are visualized on a light field display, the display system matches the received input to its output capabilities through various processes, such as interpolation. Therefore, one of the most straightforward methods to create light field contents for a specific display is to consider its visualization parameters during acquisition. In this paper, we introduce a novel display-specific light field dataset, captured using a DSLR camera and a turntable rig. The visual data of the seven static scenes were recorded twice by using two settings of angular resolution. While both were acquired uniformly within a 53-degree angle, which matches the viewing cone of the display they were captured for, one dataset consists of 70 views per content, while the other of 140. Capturing the contents twice was a more straightforward solution than downsampling, as the latter approach could either degrade the quality or make the FOV size inaccurate. The paper provides a detailed characterization of the captured contents, as well as compressed variations of the contents with various codecs, together with the calculated values of commonly-used quality metrics for the compressed light field contents. We expect that this dataset will be useful for the research community working on light field compression, processing, and quality assessment, for instance to perform subjective quality assessment tests on a display with a 53-degree display cone and to test new interpolation methods and objective quality metrics. In future work, we will also focus on subjective tests and provide relevant results. This dataset is made free to access for the research community.

1. Introduction

Research on light field visualization modalities is continuously progressing, including studies on perceptual aspects and quality assessment, with the first relevant international recommendation published recently [1], following a lack of guidelines and standard procedures. While access to real light field displays—which often appear in the scientific literature as super multiview (SMV) displays [2,3,4,5,6,7]—is still rather limited at the time of writing this paper. There is, in fact, a steady stream of research efforts to pave the way for future use cases of light field visualization. Light field displays and the contents they visualize can be characterized by various key performance indicators (KPIs) [8]. Among the KPIs that are both applicable to display and content, resolution (spatial and angular) and field of view (FOV) need to be particularly highlighted due to the universal importance. Practically, spatial resolution determines visualization fidelity at the plane of the screen, angular resolution is responsible for the smoothness of the parallax effect trough the density of distinct light rays, and FOV is basically the angle of visualization. Of course, these KPIs are highly intertwined (e.g., spatial and angular resolution [9,10,11]), yet they can be—and they should be—addressed separately as well.

If we approach light field content as an array of 2D views—which is a 1D array for horizontal-only parallax (HOP)—and a 2D array for full-parallax (FP) visualization, then angular resolution is technically the ratio of the total number of views and the angle in which they are spread evenly (i.e., the FOV). Hence, the number of views of the content as an input parameter on its own does not determine the corresponding quality characteristics. The FOV and the angular resolution of the display describe the valid viewing area (VVA) of the visualization system; while the FOV is the angle that encloses the area, the size of the area is set by angular density (i.e., greater angular resolution values can support greater viewing distances).

When a given light field content is visualized on a specific display, the characteristics of the content must be matched to the capabilities of the display. In an ideal scenario, the parameters align perfectly. In any other case, the content must be matched via techniques locally enabled by the visualization system (e.g., interpolation). For example, if the density of the rasterized 2D views are lower than the angular resolution of the display, then the display attempts to map the input to the output through a process that is basically an estimation—and hence, the perceived quality becomes degraded.

When creating content datasets for light field displays, it is possible to capture or render content with the display KPIs in mind. For instance, the dataset published by Tamboli et al. [12] offers a single view per degree, and since it was designed for a display with a 50-degree FOV, each content is composed of 50 views. In this paper, we present a novel light field dataset, specifically designed for a commercially-available light field display. The HOP content was captured as a series of 2D images, by using a digital single-lens reflex camera (DSLR). The seven static scenes (i.e., models) were rotated on a turntable, in order to accurately achieve the different perspectives. In essence, the camera was placed at a given position (fixed distance from the model) and orientation (looking at the center of the model) while the model itself rotated at a constant speed. Each model was captured with two values of angular density (i.e., two different capture frequencies). As the number of acquired views were 70 and 140, and the dataset FOV was 53 degrees—matching the FOV of the light field display—the corresponding values of angular resolution were 1.32 and 2.64 views per degree. In the alternate, degree-based format—where a smaller number indicates higher angular density—these values were 0.76 and 0.38 degrees. Again, the novelty of our work lies in the creation of a light field dataset that takes into consideration the light field display on which it is aimed to be visualized on. Although this somewhat narrows and limits the universality of the dataset, at the same time, it may avoid issues related to device-based interpolation, mismatch between the viewing angle of input and output light fields, and other potential changes in visualization quality.

The contents of the dataset are characterized and objectively measured by using the following techniques and processes:

• Measurement of spatial information ($S{I}_V$), pooled among the different views;
• Measurement of similarity among the views, measured via a modification of the temporal information ($TI$) metric, that we denote ($T{I}_V$);
• Measurement of standard deviation of the Y channel for each view;
• Measurement of colorfulness ($C{F}_V$), pooled among the different views;
• Measurement of color distribution in $Lab\ast$—also known as the $CIELab$ space—based on the technical report of the International Commission on Illumination ($CIE$);
• Measurement of color distribution in the red, green, and blue ($RGB$) space;
• Measurement of color distribution in the hue, saturation, and value ($HSV$) space;
• Measurement of color distribution in the hue, saturation, and lightness—also known as luminance—($HSL$) space;
• Application of degradation to views by advanced video coding ($AVC$), high-efficiency video coding ($HEVC$), video codec 9 ($VP9$), and AOMedia video 1 ($AV1$) encoders;
• Objective quality assessment that includes mean square error ($MS{E}_V$), peak signal-to-noise ratio ($PSN{R}_V$), and structural similarity index ($SSI{M}_V$).

Additionally, it should be noted that various light-field-specific objective quality metrics already exist at the time of writing this paper. However, the existing algorithms and methods were evaluated on conventional 2D displays and other display types that are not actual light field displays. For example, the Win5-LID dataset [13] was based on a stereoscopic display (3D television with shutter glasses), which was used to evaluate the performance of the tensor-oriented no-reference light field image quality assessment [14]; the MPI-LFA dataset [15] had subjective tests on a liquid-crystal display (LCD) desktop monitor that was viewed via glasses, which was used by the no-reference light field image quality assessment based on spatial-angular measurement [16]; and the VALID dataset [17] used an LCD monitor and a light-emitting diode (LED) display, which was used by the no-reference light field image quality assessment based on micro-lens image [18] metric. Therefore, as the dataset is specifically created to be visualized not only on light field displays in general, but for a given display model, our work does not extend to the utilization of such an objective quality assessment.

The remainder of this paper is structured as follows. The review of the related literature (i.e., similar datasets) is presented in Section 2. The experimental setup for creating the light field dataset is detailed in Section 3. Section 4 focuses on the methods of content characterization. Objective quality assessments are presented in Section 5. The results obtained from the various analyses are introduced in Section 6. Finally, the paper is concluded in Section 7.

2. Related Work

Most publicly available light field datasets consist of static contents (i.e., static scenes and models). For instance, the dataset created by Paudyal et al. [19]—using the Lytro Illum light field camera—comprises fifteen indoor and outdoor scenes, accompanied by content characterization measurements, such as CF and SI. Numerous static-scene datasets [17,20,21,22,23] were captured by using the Lytro Illum plenoptic camera, not including motion or video content among them. Other static-content light field datasets [15,24,25,26,27] were captured by using DSLR cameras without incorporating video elements. A high-angular-resolution dataset consisting of seven objects is introduced by the work of Tamboli et al. [24]. The dataset was generated using three different cameras placed at separate positions, each capturing the object at half-degree intervals as it rotated. A total of 720 images were recorded for each camera. A number of light field datasets are based on synthetically generated content. For instance, Wu et al. [28] utilized red, green, blue plus depth (RGB-D) images as input data and synthesized them, achieved through their proposed all-software algorithm to enhance spatial and angular resolutions.

In contrast, only a very limited number of datasets contain light field videos. Among the few public datasets that fit this criterion is the work of Guillo et al. [29], which consists of objects on a rotating turntable, captured via the raytrix R8 plenoptic camera (manufactured by raytrix GmbH, Kiel, Germany). Each video clip lasts for ten seconds, has a frame rate of 30 frames per second (fps), and contains a total of 300 individual frames for each content. Each frame shows 25 different views. Notably, as described in the work of Guillo et al. [29], all the motion in the dataset was due to the turntable.

Another category of light field datasets involves the use of multi-array cameras, capturing distinct dimensions of a scene based on individual camera positions within an array. In particular, a dataset created by Vaish and Adams [30] employs an array of 100 video graphics array (VGA) video cameras to assemble the contents of both static and real video scenes.

There are multiple comprehensive summaries on the available light field datasets [31,32], offering detailed information on the majority of these contents. Additionally, the recently published international recommendation [1] lists the relevant datasets as well.

There is a limited amount of research conducted in terms of color measurements and content characterizations on light field datasets. CIELab and HSV are the usual color spaces that are chosen. The research conducted by Bora et al. [33] explains color image segmentation. It highlights that the HSV color space outperforms CIELab. The instructions for image segmentation involve the initial application of both color spaces, followed by the application of subsequent processing units. Additionally, quality assessments—using MSE and PSNR—are conducted after the segmentation process. The research by Basak et al. [34] used RGB, HSV, and HSL color spaces as input variables for datasets to develop multiple linear regression (MLR) and support vector machine regression (SVM-R) models. In the case of evaluation of fruit ripeness, color plays a crucial role; Pardede et al. [35] studied fruit ripeness based on different color features of RGB, HSV, HSL, and CIELab. Images were resized to 100 × 100 pixels, and color features were extracted from these color spaces. A scheme proposed by Ravishankar et al. [36] learns multiplicative layers optimized through a convolutional neural network (CNN). HEVC is used to encode the dataset, but only the RGB color space is utilized to characterize the contents. The work specifies the aim of exploring HSV/HSL or CIELab for enhanced image perception in the future. Indrabayu et al. [37] converted RGB to HSV color space. The three components were used for classification in the use case of strawberry ripeness based on skin tone colors.

Paudyal et al. [19] used CF and SI to test the characteristics of light field images before carrying out subjective evaluation, and Faria et al. [38] created a light field image dataset for skin lesions and used SI and CF for the assessment of the texture characteristics. Shi et al. [13] also evaluated the light field dataset of Faria et al. [38] for the characterization parameters and utilized both SI and CF for this reason. Regarding spatial and temporal content characterization, Barman and Martini [39] evaluated video characterization based on SI and TI for different values of bit depth of pixels of the video content, but the contents were not light field. As mentioned above, Paudyal et al. [19] calculates SI, on a single image per content, according to what can be inferred from the paper.

Objective quality metrics—including PSNR and SSIM—were applied for some existing datasets (e.g., by Amirpour et al. [40]), in addition to conducting subjective tests on the dataset. Phicong et al. [41] proposed objective assessment metrics for light field-image quality assessment (LF-IQA) to enhance light field content analysis. Perra et al. [42] conducted a study on the light field JPEG Pleno encoder developed by the JPEG committee (ISO/IEC JTC1/SC29/WG1). They evaluated the encoded light field contents using the PSNR objective metric. Wang et al. [43] introduced a convolutional network designed to enhance the quality of light field images. They achieved this by first extracting both angular and spatial features from the initial light field image, and then by combining these features, using their special interactive mechanism. This approach yielded superior results compared to its previous methods. Jin et al. [44] conducted a study on an enhanced disparity learning mechanism for super-resolution light field content. Their study involved incorporating both convolution-based and transformer-based approaches to evaluate the PSNR, specifically focusing on the disparity between training and inference stages. Xiao et al. [45] introduced a real-world light field dataset, captured with the Illium Lytro plenoptic camera. They observed spatial degradation in the contents and subsequently evaluated the dataset in terms of the PSNR quality metric. Yu et al. [46] proposed light field network models, designed to handle the light field contents of varying quality. The authors trained these models and evaluated their performance using synthetic and real-world datasets, assessing the results in terms of PSNR and SSIM quality metrics. The objective metrics used in existing open source datasets are reported in Table 4 of the work of Shafiee and Martini [32].

The recommended method to perform quality assessment for light field content is via subjective tests, since rendering and visualization play an important role, not yet captured appropriately by objective metrics. Subjective tests are conducted using both conventional 2D displays and light field displays. To standardize light field subjective tests, recommendations are provided [1,47] for assessing the quality of experience (QoE) of light field content visualization. Darukumalli et al. [48] conducted a study on the HoloVizio C80 light field cinema system to assess how the zoom level impacts the subjective quality of light field contents. Guindy et al. [49] created light field contents for different types of camera animations and visualized them on the same light field display to assess feasibility. Tamboli et al. [12] introduced a light field dataset—assisted by a turntable—of real-world objects, capturing one view per degree for a complete 360 degrees. The authors carried out subjective tests on a Holovizio 721RC light field display with a 50-degree FOV. Simon et al. [50] investigated the impact of resolution values and viewing distance on light field content for test participants with reduced visual capabilities using the HoloVizio 640RC display system. A comprehensive review of the subjective tests carried out on real light field displays is provided by the work of Kara et al. [47]. There are numerous other works that report subjective tests that used either 2D displays or conventional 3D displays to visualize light field contents.

3. Capture Configuration

The contents in this work were captured using a Canon 77D DSLR camera (manufactured by Canon Inc., Tokyo, Japan) [51]. To acquire the contents (in the form of a series of 2D images), a turntable was used. The selected objects were placed on a turntable, which was rotated clockwise at a consistent speed, completing one full turn in a period of 19 s. The camera was placed at a fixed distance of 20 cm from the objects, with a fixed 18 mm focal length, ensuring consistent spatial relationships. The 2D spatial resolution of each view in the contents is 1920 × 1080 pixels. Each content was captured in front of a dark blue background. The color of the background was chosen in order to provide contrast with the involved models, most of which are red, green, and yellow. The level of illuminance in the capture environment—measured with a lux meter—was registered at 320 lux for every item.

The dataset contains seven distinct sets of objects, some of which consist of multiple objects rather than a single entity. Additionally, two of the contents (‘Marbles’ and ‘Sharpener’) feature semi-translucent objects. Generally, the seven contents were selected with the aim of creating a dataset that is diverse along multiple dimensions (e.g., structural complexity, color variety). Sample views of the captured contents are illustrated in Figure 1.

Content ‘Camera’ is an actual, functioning, small-scale neuromorphic camera [52]. Content ‘Cars’ consists of two toy cars, one of which is green and black, and the other one is red and light blue. Content ‘Duck’ is a black plastic duck. Content ‘Eiffel’ consists of a small ivory-colored model of the Eiffel tower, as well as a yellow and orange plane and the same green and black toy car from content ‘Cars’. Content ‘Marbles’ is a set of four glass marbles with different patterns and colors. Content ‘Sharpener’ is a conventional pencil sharpener with a transparent container. Content ‘Toys’ consists of four plastic toys, with smooth surfaces and with different variations of the colors red, orange, and purple.

Each content within the dataset was captured in a 53-degree FOV and consists of 70 and 140 views, which—as stated earlier in the paper—correspond to 1.32 and 2.64 views per degree. The dataset was created for the Looking Glass Factory [53] 32-inch, commercially-available light field display, which provides visualization in a 53-degree FOV—which is evidently the reason why the models were captured in a 53-degree FOV.

An alternative solution to capturing the content at both 140 views and 70 views would be to capture 140 views and then use downsampling to achieve the 70 viewss. One way to reduce the angular resolution in such a manner is simply to skip every second view. However, in such a case, either the leftmost or the rightmost view is skipped, resulting in a lower FOV value. The other solution is to utilize an actual downsampling method, yet such may compromise the quality of the content due to their nature—after all, methods such as light field reconstruction, interpolation, and view synthesis are estimations, approximations, and therefore, not perfectly accurate by definition.

The selected contents were modeled in a circular format to mathematically determine their coverage circumference, and the corresponding camera FOV was measured on the basis of the calculated radius of the objects. This is illustrated in Figure 2.

4. Content Characterization Methodology

Every evaluation in this work is performed on the model variant with higher angular resolution; each input of the analysis consists of 140 views. Additionally, the suffix ‘v’ is appended to every quality metric, denoting that they are sub-metrics based on the available views. In this section, several color measurements are provided, as well as SI${}_V$, TI${}_V$, and CF${}_V$ characterizations of light field contents. Since a conventional DSLR camera was used to capture the contents, depth information was not measured initially; it can only be approximated. For example, Ranftl et al. [54] introduced a dense prediction transformer based on neural networks that can be used for such a purpose. Figure 3 shows the depth maps of the view samples of the light field dataset, which were created by using this aforementioned method.

4.1. Color Measurements

The dataset—captured using electronic sensors—employs the RGB color space as its primary representation. The amplitude intensity of each color channel (red, green, and blue) is represented with 8 bits. This leads to values ranging from 0 to 255, providing each channel with the capacity to convey 256 distinct intensity levels.

When considering all three color channels together, they can produce a wide range of colors by combining their respective intensity levels. The total number of unique colors that can be represented in this RGB color space is ${256}^3$, resulting in a palette of 16,777,216 distinct colors.

The HSL color space shares a conceptual foundation similar to that of the HSV color space, utilizing a 3D cylindrical coordinate system. However, it distinguishes itself by incorporating the characteristics of two cones within its model. Because of this unique combination of two cones, the HSL color space is often referred to as the “bi-hexcone model” [35]. The equation for transforming from RGB to HSL [34,35] is presented as Equation (1):

$$\begin{array}{cc}\hfill L& =\frac{\mathrm{Max}+\mathrm{Min}}{2},\hfill \\ \hfill H& =\left\{\begin{array}{cc}\mathrm{undefined},\hfill & \mathrm{if}\mathrm{Max}=\mathrm{Min}\hfill \\ \left(\frac{G-B}{\mathrm{Max}-\mathrm{Min}}\right)\times {60}^{\circ },\hfill & \mathrm{if}\mathrm{Max}=R\hfill \\ \left(\frac{B-R}{\mathrm{Max}-\mathrm{Min}}+2\right)\times {60}^{\circ },\hfill & \mathrm{if}\mathrm{Max}=G\hfill \\ \left(\frac{R-G}{\mathrm{Max}-\mathrm{Min}}+4\right)\times {60}^{\circ },\hfill & \mathrm{if}\mathrm{Max}=B,\hfill \end{array}\right.\hfill \\ \hfill S& =\left\{\begin{array}{cc}0,\hfill & \mathrm{if}\mathrm{Max}=\mathrm{Min}\hfill \\ \frac{\mathrm{Max}-\mathrm{Min}}{2L/255},\hfill & \mathrm{if}L\ge 127\hfill \\ \frac{\mathrm{Max}-\mathrm{Min}}{2-(2L/255)},\hfill & \mathrm{if}L<127,\hfill \end{array}\right.\hfill \end{array}$$

(1)

where “Max” refers to the maximum value among the red (R), green (G), and blue (B) color channel values in the RGB color model, and “Min” refers to the minimum value.

The conversion from RGB to HSV is described in Equation (2):

$$\begin{array}{cc}\hfill V& =\mathrm{Max}(R,G,B),\hfill \\ \hfill H& =\left\{\begin{array}{cc}\mathrm{undefined},\hfill & \mathrm{if}\mathrm{Max}=\mathrm{Min}\hfill \\ \left(\frac{G-B}{\mathrm{Max}-\mathrm{Min}}\right)\times A,\hfill & \mathrm{if}\mathrm{Max}=R\hfill \\ \left(\frac{B-R}{\mathrm{Max}-\mathrm{Min}}+2\right)\times A,\hfill & \mathrm{if}\mathrm{Max}=G\hfill \\ \left(\frac{R-G}{\mathrm{Max}-\mathrm{Min}}+4\right)\times A,\hfill & \mathrm{if}\mathrm{Max}=B,\hfill \end{array}\right.\hfill \\ \hfill S& =\left\{\begin{array}{cc}0,\hfill & \mathrm{if}\mathrm{Max}=\mathrm{Min}\hfill \\ \mathrm{Max}-\mathrm{Min},\hfill & \mathrm{others},\hfill \end{array}\right.\hfill \end{array}$$

(2)

where Max = max(R,G,B), Min = min(R,G,B), and the value of A is equal to $\mathsf{\Pi }/3$ if H is in radians, and 60 degrees if H is in degrees.

The CIELab color space is designed to closely align with the perceptive capabilities of the human eye. To derive the CIELab color space, a transformation from the RGB to XYZ color space is first performed, followed by a conversion from XYZ to CIELab. Based on the documentation of CIE [55], as well as the works of Schanda [56] and Pardede et al. [35], this conversion process is facilitated using Equations (3)–(5):

$$\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]=\left[\begin{array}{ccc}0.412453& 0.357580& 0.180423\\ 0.212671& 0.715160& 0.072169\\ 0.019334& 0.119193& 0.950277\end{array}\right]\left[\begin{array}{c}R\\ G\\ B\end{array}\right],$$

(3)

$$\begin{array}{cc}\hfill \delta & =\frac{6}{29},\hfill \\ \hfill f\left(t\right)& =\left\{\begin{array}{cc}\sqrt[3]{t}\hfill & \mathrm{if}t>{\delta }^3\hfill \\ {\displaystyle \frac{t}{3{\delta }^2}}+\frac{4}{29}\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill L& =116\times f\left(\frac{Y}{Yn}\right)-16,\hfill \\ \hfill a& =500\times \left(f\left(\frac{X}{Xn}\right)-f\left(\frac{Y}{Yn}\right)\right),\hfill \\ \hfill b& =200\times \left(f\left(\frac{Y}{Yn}\right)-f\left(\frac{Z}{Zn}\right)\right).\hfill \end{array}$$

(5)

The CIE standard Illuminant D65 (6504-degree Kelvin) is provided in Equation (6):

$$\begin{array}{cc}\hfill X_{\mathrm{n}}& =95.0489,\hfill \\ \hfill Y_{\mathrm{n}}& =100,\hfill \\ \hfill Z_{\mathrm{n}}& =108.8840.\hfill \end{array}$$

(6)

The CIE standard Illuminant D50 (5000-degree Kelvin) is provided in Equation (7):

$$\begin{array}{cc}\hfill X_{\mathrm{n}}& =96.4212,\hfill \\ \hfill Y_{\mathrm{n}}& =100,\hfill \\ \hfill Z_{\mathrm{n}}& =82.5188.\hfill \end{array}$$

(7)

Equations (1) and (2) are based on the works of Basak et al. [34] and Pardede et al. [35], Equation (3) is also based on the work of Pardede et al. [35], Equation (4) is based on the work of work of Schanda [56], Equation (5) is based on the works of Pardede et al. [35] and Schanda [56], and Equations (6) and (7) are based on the CIE technical report [55].

4.2. $S{I}_V$, $T{I}_V$, and $C{F}_V$ Content Characterizations

We propose here three content characterization metrics for light field content, obtained by slightly modifying/adapting three commonly used measures of content complexity (i.e., SI, TI, and colourfulness) to characterize our contents in terms of spatial complexity, inter-view complexity, and colorfulness: $S{I}_V$, $T{I}_V$, and $C{F}_V$, where V refers to “views”. $S{I}_V$ pertains to the distribution and arrangement of visual elements within a static light field scene. $TI$ normally refers to temporal complexity, in terms of changes in visual content over time, such as motion, movement of objects, or variations in lighting. Here we consider $T{I}_V$, and we define it as a measure of the changes over the adjacent perspectives. $C{F}_V$ evaluates the intensity and vividness of the views representing the scene.

$S{I}_V$ and $T{I}_V$ are used to calculate the complexity of the light field content, helping to find the correct data rate for real-time light field communication. Higher spatial and temporal values typically require higher data rates to achieve satisfactory quality [57]. We also expect that for light field data, higher $S{I}_V$ and $T{I}_V$ are associated with higher data rates required for compression.

The combination of $S{I}_V$ and $T{I}_V$ allows for a more comprehensive characterization of light field content. For the computation of $S{I}_V$, the first step is to filter each view out of all available views in the light field content (luminance component) using the Sobel filter. Next, the standard deviation is calculated over the pixels in each Sobel-filtered view. This operation is then repeated for all available views in the light field content, and the maximum value is selected.

To detect horizontal and vertical edges of the luminance component of each view, the kernels of Equations (8) and (9) are applied, respectively:

$$G_x=\left[\begin{array}{ccc}-1& 0& 1\\ -2& 0& 2\\ -1& 0& 1\end{array}\right]\ast I,$$

(8)

$$G_y=\left[\begin{array}{ccc}-1& -2& -1\\ 0& 0& 0\\ 1& 2& 1\end{array}\right]\ast I.$$

(9)

The magnitude of the gradient of Equations (8) and (9) are calculated in Equation (10):

$$S{I}_p=\sqrt{G{x}^2+G{y}^2}.$$

(10)

The mean value is measured in Equation (11):

$$S{I}_{mean}=\frac{1}{P}\sum S{I}_p,$$

(11)

where P is the number of pixels available in each view. Having the mean value of each Sobel-filtered view in Equation (11), the standard deviation is then calculated in Equation (12) [58]:

$$S{I}_{std}=\sqrt{\frac{1}{P}\sum {(S{I}_p-S{I}_{mean})}^2}.$$

(12)

The maximum standard deviation value over the views yields $S{I}_V$, as calculated in Equation (13):

$$S{I}_V=ma{x}_v\left\{S{I}_{std}\right\}.$$

(13)

Similar to $S{I}_V$, $T{I}_V$ plays a critical role in video compression and transmission, as it directly affects the data rate required to represent and transmit a video sequence. Instead of a video containing a series of frames, for the light field contents, we measured the $T{I}_V$ based on the available views within each light field content. To compute $T{I}_V$, the differences between the corresponding pixels in the luminance component of two successive views are first computed. Then, the standard deviation of the differences is calculated and the maximum value of all views in each light field content is selected as $T{I}_V$. The initial step in computing the TI value is represented in Equation (14):

$$M_p^n=V_p^n-V_p^{n-1},$$

(14)

where $M_p^n$ means the pixel intensity difference between the current view $V_p^n$, denoted view n, and the previous view $V_p^{n-1}$, which is view $n-1$ [58].

The standard deviation is computed in Equation (14). The maximum value among all available views of the light field content is selected as the final TI result, as shown in Equation (15):

$$T{I}_V=ma{x}_v\left\{std\left[M_p^n\right]\right\},$$

(15)

which is performed for each content within the dataset.

Another method of calculating the complexity of a content or an image is the calculation of the variance of the luminance [57]. We adapt it to our context of light field contents, calculating it for multiple views.

Finally, the $C{F}_V$ value shows the diversity and intensity of the available colors within the images [38], the calculation of which is shown in Equation (16):

$$C{F}_V=\sqrt{\sigma {\left(rg\right)}^2+\sigma {\left(yb\right)}^2}+0.3\sqrt{\mu {\left(rg\right)}^2+\mu {\left(yb\right)}^2},$$

(16)

where

$$\begin{array}{cc}\hfill rg& =R-G,\hfill \\ \hfill yb& =0.5(R+G)-B,\hfill \end{array}$$

where $\sigma (·)$ represents the standard deviation, and $\mu (·)$ shows the mean value.

5. Data Compression and Objective Quality Assessment Methodology

In order to exploit inter-view redundancy (i.e., the similarities between adjacent perspectives) we compressed the acquired contents using video encoders considering the views as video frames. In particular, the created light field contents—each consisting of 140 views, as stated earlier—are encoded using four different video encoders: AVC [59], HEVC [60], VP9 [61], and AV1 [62], for bit rates ranging from 1 Mbps to 30 Mbps. The fast forward MPEG (ffmpeg) application was used for encoding, but note that the actual bit rate results may not exactly match the specified bit rates due to imperfect rate control in the codecs. More details on this are provided in Section 6.

Following compression, the quality of the impaired contents is assessed by using three quality assessment metrics: $MS{E}_V$, $PSN{R}_V$, and $SSI{M}_V$, where V—similarly to $S{I}_V$, $T{I}_V$, and $C{F}_V$—refers to “views”. As the objective is to assess the quality of the degraded contents, these metrics calculate the mean of $MS{E}_V$, $PSN{R}_V$, and $SSI{M}_V$, respectively, over the different views.

Striking a balance between video quality and bandwidth requirements is critical for any compression method, and the added quality metrics will also support further relevant studies, as shown as an example by a recent work [63].

5.1. $PSN{R}_V$

One of the methods chosen to assess the quality of the impaired versions of the images/views is PSNR [64], described by Equation (17):

$$PSN{R}_v=10·{log}_{10}\left(\frac{MA{X}_I^2}{MS{E}_v}\right),$$

(17)

where ${MAX}_I$ denotes the highest achievable pixel value within the view, and $MS{E}_v$ quantifies the difference between the original and distorted views of the content. $MS{E}_v$ is calculated as shown in Equation (18):

$$MS{E}_v=\frac{1}{M\times N}\sum _{i=0}^{M-1}\sum _{j=0}^{N-1}{(I(i,j)-K(i,j))}^2,$$

(18)

where I and K are the original and impaired versions of the view, respectively, and M and N are the dimensions of the view.

The value of ${\mathrm{MAX}}_I$ varies based on the number of available bits in each single pixel. For instance, if there are 8 bits in each pixel, then the value is equal to $2^8=256$, but it starts from zero and ends at 255. Similarly, for 10-bit pixels, the value is equal to $2^{10}=1024$, which similarly starts from zero and ends at 1023. This means that there are 1024 different possible values for 10-bit pixels. Therefore, for the first case, the value of ${\mathrm{MAX}}_I$ is 255, and for 10-bit pixels, the value of ${\mathrm{MAX}}_I$ is equal to 1023. We consider here:

$$PSN{R}_V=\frac{1}{V}\sum _{v=1}^{V}PSN{R}_v,$$

(19)

where V is the number of views.

5.2. $SSI{M}_V$

The SSIM [64]—the formula of which is provided in Equation (20)—is a method to measure the similarity between two images. It compares the structural information of an image, such as edges and textures, rather than just comparing the pixel values. This method has been applied in various research studies related to quality assessment [65,66,67,68,69,70,71,72,73].

Unlike $PSN{R}_v$—which only measures the difference between two views in terms of their pixel values—$SSI{M}_v$ is a more comprehensive metric that takes into account the structural information of the views. $SSI{M}_v$ assesses the perceived quality of views by comparing their luminance, contrast, and structure with that of the reference image. Therefore, $SSI{M}_v$ is considered a more accurate measure of view quality than $PSN{R}_v$, which does not account for the characteristics of the human visual system (HVS).

For example, in the work of Salem et al. [74], both PSNR and SSIM were used to reconstruct the quality assessment metrics.

The main $SSI{M}_v$ formula is mentioned in Equation (20):

$$SSI{M}_v(I,K)=\frac{(2{\mu }_I{\mu }_K+c_1)(2{\sigma }_{IK}+c_2)}{({\mu }_I^2+{\mu }_K^2+c_1)({\sigma }_I^2+{\sigma }_K^2+c_2)},$$

(20)

where I and K represent the original and distorted views, while ${\mu }_I$ and ${\mu }_K$ denote their respective means. Similarly, ${\sigma }_I$ and ${\sigma }_K$ refer to the standard deviations of I and K, and ${\sigma }_{IK}$ represents their covariance. Constants $c_1$ and $c_2$ are included to prevent division by zero. The resulting $SSI{M}_v$ output varies from 0 to 1, with 1 signifying a perfect similarity between the two images. Similar to the above, we consider here:

$$SSI{M}_V=\frac{1}{V}\sum _{v=1}^{V}SSI{M}_v,$$

(21)

where V is the number of views.

6. Content Characterization and Objective Quality Assessment Results

In this section, the aforementioned metrics are utilized to evaluate our novel light field dataset in terms of content complexity and objective quality of the compressed versions of the contents.

6.1. Content Characterization

In this section, we present the analyzed results for color measurements, standard deviation over the available views in each content, and the characterization of spatial, temporal, and colorfulness information.

6.1.1. Color Measurements

Fifteen different result values are provided for each content in the dataset, as shown in

Table 1. The average color measurement values for the contents of the dataset.

Content	H(HSV)	S(HSV)	V(HSV)	H(HSL)	S(HSL)	L(HSL)	R	G	B	L(D65)	a(D65)	b(D65)	L(D50)	a(D50)	b(D50)
Camera	111.57	60.58	37.71	111.62	53.15	27.06	63.62	43.86	84.04	68.12	27.88	$-31.02$	68.12	26.02	$-48.06$
Cars	106.85	60.38	39.89	107.16	53.09	29.11	64.16	51.59	92.50	72.63	23.75	$-33.45$	72.63	21.82	$-51.49$
Duck	90.30	54.38	38.31	90.31	46.80	28.57	66.90	52.30	86.79	73.81	21.61	$-26.78$	73.81	19.66	$-44.37$
Eiffel	100.33	61.48	35.29	100.45	52.60	26.18	62.94	50.47	76.31	68.33	22.17	$-28.21$	68.33	20.33	$-45.02$
Marbles	102.24	59.69	31.53	102.24	48.83	23.66	55.45	47.79	71.10	67.15	19.02	$-27.33$	67.15	17.22	$-43.85$
Sharpener	101.57	57.52	37.33	101.57	50.33	27.81	60.95	49.42	89.37	70.58	23.46	$-33.21$	70.58	21.57	$-50.88$
Toys	101.95	71.86	43.40	102.51	64.29	28.45	86.85	46.43	67.29	70.91	32.61	$-17.23$	70.91	30.66	$-33.41$

. These results represent the mean values of all pixels in each light field view within the content, followed by the average of all views. This cumulative process generates the single result provided in the table.

The color space range for each component is categorized as below.

HSV:

• Hue describes the color tone, spanning from 0 to 360 degrees;
• Saturation signifies the color’s intensity, with a range of 0% (dull) to 100% (vivid);
• Vivid represents the brightness level, ranging from 0% (completely dark or black) to 100% (full brightness).

HSL:

• Hue defines the color tone and operates within the 0 to 360-degree spectrum;
• Saturation quantifies the color’s intensity, varying from 0% (pale) to 100% (highly saturated);
• This attribute gauges the darkness or lightness of the color and spans from 0% (dark) to 100% (light).

RGB:

• Red Signifies the red color’s intensity, which spans from 0 to 255;
• Green represents the green color’s intensity, which varies from 0 to 255;
• Blue, this parameter quantifies the blue color’s intensity and operates in the 0 to 255 range.

In the CIELab color space, three distinct axes are utilized:

• L measures the transition from darkness to lightness, with values ranging from 0% to 100%,

  –

  Lower values indicate darker colors or shades;

  –

  Higher values correspond to lighter colors or shades.

• a* represents the color axis between green (−128) and red (+127),

  –

  Negative values (−): Suggest a shift towards the green side;

  –

  Positive values (+): Indicate a shift towards the red side.

• b* represents the color axis from blueness (−128) to yellowness (+127),

  –

  Negative values (−): Suggest a shift towards the blue side;

  –

  Positive values (+): Represent a shift towards the yellow side.

6.1.2. Analysis of Standard Deviation across Available Views of the Contents

As highlighted in [57], the variance and standard deviation of the luminance of an image are associated with contrast and hence to its (spatial) complexity for compression; hence, we calculated for the contents in the dataset, the standard deviation of the luminance in each single image/view. The results are shown in Figure 4 in the form of histogram plots, where the x-axis displays the standard deviation values, and the y-axis shows the number of occurrences in all available views for each light field content. The higher the standard deviation values, the higher the (spatial) complexity of the content. We can observe that when the histogram presents spikes, most of the views have a very similar standard deviation/contrast, while “flatter” histograms are associated with a more uniform distribution of standard deviation/contrast values across the views.

6.1.3. Characterization of Spatial, Temporal, and Colorfulness Information

Based on the analysis shown in Figure 5, it is evident that the contents labeled ‘Cars’ and ‘Eiffel’ exhibit higher spatial information across the 140 views. Additionally, in terms of $T{I}_V$, two of the contents, namely ‘Eiffel’ and ‘Toys’, reached the maximum values among all the contents of the dataset. For the transparent contents, ‘Sharpener’ and ‘Marbles’ exhibit the lowest values in terms of both $S{I}_V$ and $T{I}_V$ across their views. Regarding $C{F}_V$, content ‘Toys’ demonstrates the highest values. Table 1 presents the results for the different color channels in various color spaces across the 140 views of each content.

6.2. Data Compression

As highlighted before, the actual datarates are often different from the values set for encoding, due to inaccuracies in the rate-control algorithms adopted in the codecs. The results of the actual bitrate values obtained are provided in

Table 2. The actual practical results (in Mbps) of the encoders for contents Camera, Cars, Duck, and Eiffel.

Bit Rate	Camera				Cars				Duck				Eiffel
Values	AVC	HEVC	VP9	AV1	AVC	HEVC	VP9	AV1	AVC	HEVC	VP9	AV1	AVC	HEVC	VP9	AV1
1	0.60	0.80	0.91	1.03	0.57	0.74	0.92	1.11	0.60	0.80	0.94	1.00	0.54	0.66	0.88	1.07
2	1.99	1.71	1.77	2.09	1.69	1.63	1.81	2.16	1.92	1.72	1.79	2.23	1.54	1.52	1.74	2.20
3	4.62	2.65	2.63	3.67	3.39	2.54	2.72	3.44	4.01	2.64	2.67	3.44	2.91	2.39	2.63	3.40
4	7.12	3.60	3.53	5.21	5.02	3.48	3.60	5.18	6.10	3.62	3.56	5.02	4.27	3.31	3.53	4.98
5	8.18	4.57	4.47	7.05	6.22	4.45	4.50	6.95	7.28	4.56	4.43	6.95	5.37	4.26	4.39	6.53
6	8.59	5.56	5.28	8.98	6.97	5.36	5.37	9.04	7.70	5.54	5.30	8.51	6.33	5.16	5.24	8.16
7	9.00	6.50	6.15	10.70	7.83	6.36	6.26	11.12	8.26	6.49	6.16	10.22	7.31	6.12	6.15	9.83
8	9.10	7.46	6.92	11.67	8.41	7.27	7.04	12.55	8.63	7.48	7.06	12.11	8.12	7.04	6.99	11.89
9	9.33	8.48	7.73	13.61	9.09	8.29	7.86	13.59	9.22	8.37	7.83	13.53	8.94	8.06	7.64	14.03
10	9.70	9.37	8.38	15.70	9.82	9.18	8.61	15.47	9.82	9.38	8.65	15.25	9.65	8.90	8.43	15.54
20	20.24	18.93	15.02	37.61	19.92	18.80	15.42	38.63	20.33	18.94	15.26	36.53	20.08	18.46	15.31	35.69
25	26.03	23.66	18.25	47.50	25.68	23.52	18.47	47.53	26.00	23.82	18.33	46.12	25.37	23.15	18.39	45.80
30	31.89	28.52	21.05	57.12	31.65	28.18	21.44	57.41	31.98	28.42	21.30	55.12	31.02	27.72	21.14	55.78

and

Table 3. The actual practical results (in Mbps) of the encoders for contents Marbles, Sharpener, and Toys.

Bit Rate	Marbles				Sharpener				Toys
Values	AVC	HEVC	VP9	AV1	AVC	HEVC	VP9	AV1	AVC	HEVC	VP9	AV1
1	0.59	0.76	0.90	1.04	0.59	0.78	0.91	0.92	0.68	0.73	0.89	1.20
2	1.71	1.66	1.80	2.06	1.96	1.68	1.79	1.94	1.47	1.58	1.74	2.14
3	3.19	2.56	2.70	3.28	4.04	2.61	2.64	3.39	2.61	2.45	2.66	3.35
4	4.81	3.51	3.57	5.26	5.68	3.57	3.51	4.83	3.81	3.35	3.49	4.86
5	5.81	4.47	4.44	7.08	7.10	4.55	4.39	6.73	4.94	4.30	4.35	6.38
6	6.76	5.43	5.29	8.70	7.68	5.48	5.27	9.20	6.03	5.25	5.17	8.13
7	7.68	6.34	6.10	10.36	8.26	6.49	6.10	10.71	7.10	6.19	5.99	9.98
8	8.47	7.34	6.88	12.48	8.71	7.39	6.87	11.82	8.13	7.14	6.85	11.15
9	9.24	8.27	7.70	13.75	9.28	8.38	7.75	13.88	9.09	8.08	7.63	14.18
10	9.92	9.23	8.52	15.60	9.85	9.38	8.50	15.87	10.00	9.07	8.41	15.34
20	20.04	18.88	15.00	36.81	20.39	18.82	15.70	38.00	20.07	18.56	15.18	36.37
25	25.47	23.77	18.22	47.32	26.06	23.66	18.33	47.88	25.37	23.30	18.51	46.29
30	31.43	28.55	21.33	56.74	31.95	28.47	21.24	57.10	30.82	28.03	21.47	57.06

. The first column of the table reports the bitrate value sets in the codecs (in Mbps). We refer to bitrates rather than file size for the scene since we consider the views of a static scene as the frames of a video for the encoding process. Since the encoding was performed assuming 30 fps, 1 Mbps means that, on average, each view is represented with 33 kbits hence 140 views with 4.67 Mbits or 583 kB, while 30 Mbps correspond to a file size of 17.5 MB. The other columns report the actual values obtained with different codecs for the different contents.

The quality results—as depicted in Figure 6 and Figure 7—clearly demonstrate how the inconsistency between the applied bit rates and the actual values—all listed in Table 2 and Table 3—affects the plot lengths, causing them to be unequal.

6.3. Objective Quality Assessment Results

Following the characterization analysis of the contents mentioned above and compression, the quality of the compressed versions is objectively assessed.

To accomplish this, all contents are initially encoded with varying bit rate values and by different encoders, including AVC, HEVC, VP9, and AV1. Subsequently, the quality of the impaired encoded contents is evaluated in relation to the original contents (“full-reference” quality assessment).

Figure 6 and Figure 7 report the objective quality results in terms of $MS{E}_V$, $PSN{R}_V$, and $SSI{M}_V$ versus compression ratio, for the different contents (‘Camera’, ‘Cars’, ‘Duck’, ‘Eiffel’ in Figure 6 and ‘Marbles’, ‘Sharpener’, and ‘Toys’ in Figure 7). We can observe that AV1 performs the best in terms of quality assessment for both $MS{E}_V$/$PSN{R}_V$ and $SSI{M}_V$. Following AV1, VP9 is the second best encoder, followed by HEVC and then AVC, which shows the lowest performance for our dataset. Armipour et al. [75] and Hajihashemi et al. [76] used the same evaluation methods for their light field contents, with some differences in the performance of certain codecs. Quality values follow the order of highest to lowest for AV1, VP9, HEVC, and AVC, respectively. It is notable that all the mentioned encoders are applied to all 140 views within each content of the light field dataset, followed by the application of the attributed quality metrics. We can also observe that quality values span a wider range for some of the contents (e.g., ‘Cars’ and ‘Toys’) than for others (e.g., ‘Duck’). Since the texture of the objects is relatively simple and the background is quite uniform, high compression ratios are achieved in general, while a very high quality is kept.

7. Conclusions

In conclusion, this paper presented a novel light field dataset captured using the Canon 77D DSLR camera, with the aim to support future research on displays with a 53 degrees viewing cone, such as the 32-inch Looking Glass Factory light field display, particularly in the field of subjective quality assessment. Various measures for content characterization were calculated for the contents in this dataset. Moreover, different encoders were applied to the contents in the proposed dataset, covering all 140 available views of each content, with varying compression ratios. Quality scores of the compressed versions are also provided, calculated via commonly used objective quality metrics.

In future work, the introduced light field contents shall undergo subjective evaluation on the aforementioned light field display, with a particular focus on assessing the transparency effects during display. We also plan to expand the dataset, for instance, with further codecs, including versatile video coding (VVC) [77].