Learning to Measure Stereoscopic S3D Image Perceptual Quality on the Basis of Binocular Rivalry Response

Abstract

Blind perceptual quality measurement of stereoscopic 3D (S3D) images has become an important and challenging issue in the research field of S3D imaging. In this paper, a blind S3D image quality measurement (IQM) method that does not depend on examples of distorted S3D images and corresponding subjective scores is proposed. As the main contribution of this work, we replace human subjective scores with a quality codebook of binocular rivalry responses (BRRs); this allows blind S3D-IQM methods to be learned without evaluation performance loss. Our results, using the publicly accessible LIVE S3D dataset, confirm that our method is highly robust and efficient.

1. Introduction

In recent years, stereoscopic 3D (S3D) technologies have attracted considerable research interest. S3D measurement can be classified into S3D content measurement (e.g., visual discomfort, quality of experience, or unnatural experience) and S3D image quality measurement (IQM) (e.g., the measurement of compression distortions or transmission impairments for S3D images). However, research on S3D content measurement is still limited. In recent decades, blind S3D-IQM has been attracting attention in the research field of S3D imaging [1,2,3]. State-of-the-art blind S3D-IQM methods learn to measure S3D quality via regression from the subjective scores of training samples. For instance, in [4], Akhter et al. introduced a blind S3D-IQM method that extracts both 2D and S3D quality-aware features from an S3D image and a depth/disparity map. Next, a regression model is adapted to obtain a final quality value from the above-mentioned features. Similar studies were also conducted in [5]; Chen et al. proposed a blind S3D-IQM method based on a “cyclopean” view and disparity map. Zhou et al. introduced blind S3D-IQM based on the self-similarity of binocular features [6]. Other relevant existing studies can be found in [7,8,9,10,11,12,13,14]. These methods require human-scored S3D images for training; however, obtaining subjective scores through subjective evaluation is often expensive, cumbersome, and time-consuming, which limits their usability in practice. Thus, a significant problem is triggered: can we learn to measure S3D image quality blindly without relying on human-scored S3D images?

Quality measurement of 2D images has been extensively studied, and several excellent methods have been proposed [15,16,17,18]. For example, in [16], Xue et al. presented a quality-aware clustering metric, which can learn centroids to act as a quality-aware codebook for measuring distorted image quality. Inspired by [16], we propose a blind S3D-IQM method that learns a quality codebook of binocular rivalry responses (BRRs). The main contributions of this work are three-fold.

First, in the offline training phase, we learn a quality codebook from the BRRs of the original and distorted S3D images; this codebook will be used to replace human subjective scores to learn blind S3D-IQM methods.

Second, a BRR-weighted local binary pattern (LBP) histogram scheme is used to extract quality-predictive features; this scheme is effective for describing degradation patterns.

Third, in the online testing phase, a distance-based weighting scheme is used for perceptual quality pooling. Experimental results obtained using the publicly accessible LIVE S3D dataset indicate that the proposed method performs competitively and provides quality prediction performance and generalization ability.

2. Proposed Method

The block diagram for our blind S3D-IQM method is depicted in Figure 1. During the offline training phase, similarity scores and a weighted LBP histogram are generated from training S3D images; subsequently, a quality codebook is constructed. During the online testing phase, by implementing feature extraction for test S3D images, blind perceptual quality pooling can be easily achieved by a distance-based weighting scheme. Our method is described in detail as follows.

(1) Training dataset preparation: We select 12 reference S3D images collected from Mobile S3DTV and MPEG (please refer to Figure 2) [19]. From each reference S3D image, we then generate distorted S3D images using four types of distortions, including JPEG2000 (JP2K), Gaussian blur (Gblur), white noise (WN), and JPEG. For each distortion, five quality levels are generated by controlling different parameters.

(2) Quality measurement of the training S3D images: Neuroscience has made rapid progress in comprehending the human visual mechanism and how S3D visual signals are transmitted to the human brain. BRR is a well-researched visual mechanism of competition in the visual cortex [20]. BRR is a human visual response whereby the binocular visual mechanism alternates between contradictory monocular images viewed by both eyes. Some findings in [20] showed that BRR can be strongly regulated by low-level sensory features, and a classic linear mathematical model is provided. Motivated by the above, 2D-Gabor is used for both views to simulate the rivalries. Consequently, BRR, which can be denoted by R_bin(x,y,σ), is defined as

$$R_{bin}(x,y,\sigma )=W_l(x,y)\cdot M_l(x,y,\sigma )+W_r(x+d,y)\cdot M_r(x+d,y,\sigma )$$

(1)

where d denotes a disparity value estimated using an S3D matching algorithm [21] (Reference [22] reports that similar performance is attained when using either the ground truth or the estimated disparity). Subscripts “l” and “r” are the left and right views, respectively. M_l(x,y,σ) and M_r(x,y,σ) are the monocular responses of both views, which can be obtained using a difference of Gaussian (DoG) filter

$$h_{\sigma }(r,\sigma )=\frac{1}{2\pi {\sigma }^2}\exp (-\frac{r^2}{2{\sigma }^2})-\frac{1}{2\pi k^2{\sigma }^2}\exp (-\frac{r^2}{2k^2{\sigma }^2})$$

(2)

where σ denotes the scales parameter and k is used to control the scales of the DoG. r denotes the l₂ norm. In this study, scales σ of the DoG are set to 0, 1, 1.6, 2.56, and 4.096 (the same values as the filter used in [2]), with k = 1.6. W_l and W_r denote the weights for imitating the BRR. They are represented as W_l(x,y) = G_l(x,y)/(G_l(x,y) + G_r(x + d,y)) and W_r(x + d,y) = G_r(x + d,y)/(G_l(x,y) + G_r(x + d,y)), respectively, where G_l(x,y) and G_r(x,y) denote the respective 2D-Gabor responses of both views. The 2D Gabor filter can be defined as:

$$G(x,y,{\sigma }_x,{\sigma }_y,{\zeta }_x,{\zeta }_y^{},\theta )=\frac{1}{2\pi {\sigma }_x{\sigma }_y}{e}^{-(1/2)[{((x\cos \theta +y\sin \theta )/{\sigma }_x)}^2+{((y\cos \theta -\sin \theta )/{\sigma }_y)}^2]}{e}^{i(x{\zeta }_x+y{\zeta }_y)}$$

(3)

where σ_x and σ_y denote the standard deviations along the x and y axes, ζ_x and ζ_y, and θ denotes spatial frequencies and orients the filter. The design of the 2D-Gabor filter was based on the research conducted by Su et al. [23].

The similarity score between the BRRs of the original S3D image and distorted S3D images is defined as:

$$S_n=\frac{1}{\Omega }{\displaystyle \sum _{x,y}\left({\displaystyle \prod _{\sigma }\left(\frac{2\langle R_{o,n}(x,y,\sigma ),R_{d,n}(x,y,\sigma )\rangle +T}{{\Vert R_{o,n}(x,y,\sigma )\Vert }_2\cdot {\Vert R_{d,n}(x,y,\sigma )\Vert }_2+T}\right)}\right)}$$

(4)

where T is a small positive constant. In this study, we set T = 0.085 in our experiments. Ω indicates the whole spatial domain, ${\Vert \cdot \Vert }_2$ denotes the l₂ norm, and $\langle \cdot \rangle$ represents the inner product. Subscript “n” is the index of the n-th training distorted S3D images. Subscripts “o” and “d” are the original and distorted S3D images, respectively.

(3) Quality-predictive features extraction: Based on Marr’s theory [24], local structural features based on the visual cortex (area V1) are related to image quality. LBPs are basic feature extractors that are commonly used to represent an image’s structural and textural information [25]. First, we compute the rotation invariant uniform LBP map, $LB{P}_{P,R,BRR,n}^{riu2}$, for BRR by comparing BRR intensity against its eight neighbors (in consideration of computational complexity, R and P were set to 1 and 8, respectively). To effectively capture textural and structural features, the BRR-weighted LBP histogram is applied to extract quality-predictive features of BRR, which are defined as

$$h_n^{}(k,\sigma )=\frac{1}{\Omega }{\displaystyle \sum _{x,y}{R}_{d,n}(x,y,\sigma )\times f(LB{P}_{P,R,BRR,n}^{riu2}(x,y,\sigma ),k)}$$

(5)

where $f(i,j)=\{\begin{array}{cc}1,& \mathrm{if}i=j\\ 0,& \mathrm{otherwise}\end{array},i\&j\in [0,P+1]$.

(4) BRR quality codebook: Thus far, we have gained a set of human-scored S3D images for training. All the quality scores (denoted by {S_n}) and all the feature vectors (denoted by $\left\{h_n^{}(k,\sigma )\right\}$) of the distorted S3D images for training constitute a BRR quality codebook to reflect different quality levels of visual signals and associated quality-predictive features.

(5) Blind perceptual quality pooling: After obtaining a BRR quality codebook, we use the codes to predict the quality score of a test S3D image, relying on the hypothesis that S3D images with similar quality-aware features should share similar visual qualities. First, let $h_m^d(k,\sigma )$ represent the quality-aware features of the n-th test S3D image. We define the distance H_m,n between $h_m^d(k,\sigma )$ and $h_n^{}(k,\sigma )$ as the product of the chi-square distance between each pair of quality-predictive features.

$$H_{m,n}[h_m^d(k,\sigma ),h_n^{}(k,\sigma )]={\displaystyle \sum _{n=1}^N\frac{{(h_m^d(k,\sigma )-h_n^{}(k,\sigma ))}^2}{h_m^d(k,\sigma )+h_n^{}(k,\sigma )}}$$

(6)

where N is the total number of training S3D images. Let $\{{\widehat{H}}_{m,1},{\widehat{H}}_{m,2},\cdots ,{\widehat{H}}_{m,9}{\widehat{H}}_{m,10}\}$ represent the chi-square distances of 10 of the nearest quality-predictive features, which are calculated using the n-th test set and the selected training set. A distance-based weighting method is then utilized to measure the perceptual visual quality of the test set. The normalized weight for the 10 similarity scores of the selected training set, which is denoted by $w_{m,n^{\prime }}$, can be represented as

$$w_{m,n^{\prime }}={\left({\widehat{H}}_{m,n^{\prime }}\right)}^{-1}/{\displaystyle \sum _{n^{\prime }=1}^{10}{\left({\widehat{H}}_{m,n^{\prime }}\right)}^{-1}}.$$

(7)

By distributing larger weights to the similarity values of the selected neighboring training S3D images, the final quality value of the n-th test S3D image is calculated as

$$Q_m={\displaystyle \sum _{n^{\prime }=1}^{10}{w}_{m,n^{\prime }}}\cdot S_{n^{\prime }}.$$

(8)

3. Results

In the experiment, the effectiveness of our method was verified using the LIVE S3D [5] dataset, which includes 365 symmetrically distorted S3D images (in the future, we plan to understand how asymmetrical distortion affects perceptual quality (for the case of blur artifacts, the perceptual quality of asymmetric blurred S3D images is mainly dominated by the view that contains more information. For the case of JPEG artifacts, the binocular perceptual quality has a tendency towards the lower quality view) and extension of the proposed method to facilitate asymmetrical distortion S3D-IQM). In order to facilitate the experiment (refer to [16]), we utilized only four distortion types: JP2K, WN, JPEG, and Gblur. The images in the dataset were assigned difference mean opinion scores (DMOS), which represent subjective judgments.

In order to compare the proposed method against existing S3D-IQM methods, two widely used performance criteria were calculated by applying the DMOS and the predicted quality scores. The first was a Pearson’s linear correlation coefficient (PLCC), which was computed to evaluate the prediction accuracy. The second was a Spearman’s rank order correlation coefficient (SROCC), which measured the prediction monotonicity. For an ideal IQM method, values close to 1 were needed for SROCC and PLCC.

To demonstrate the robustness and effectiveness of the proposed method, we compared it with several existing representative IQM methods, including three 2D-IQM methods—SSIM (full reference, FR) [26], IL_NIQE (blind, not depending on DMOSs) [15], and Xue’s method (blind, not depending on DMOSs) [16]—and seven S3D-IQM methods—Lin’s method (FR) [2], Akhter’s method (blind, depending on DMOSs) [4], Chen’s method (blind, depending on DMOSs) [5], Yue’s method (blind, depending on DMOSs) [9], Zhou’s method (blind, depending on DMOSs) [12], Shao’s method (blind, not depending on DMOSs) [13], and Zhou’s method (blind, not depending on DMOSs) [14]. All IQM methods are based on the grayscales of S3D images. The SROCC and PLCC results are listed in

Table 1. Overall performance evaluation of nine methods.

Criteria	FR		Blind
			Depending on Difference Mean Opinion Scores (DMOS)				Not Depending on DMOS
	SSIM	Lin [2]	Akhter [4]	Chen [5]	Yue [9]	Zhou [12]	IL_NIQE	Xue [16]	Shao [13]	Zhou [14]	Proposed
PLCC	0.899	0.873	0.640	0.901	0.937	0.973	0.896	0.871	0.877	0.887	0.925
SROCC	0.882	0.830	0.395	0.899	0.914	0.965	0.876	0.873	0.866	0.892	0.887

; the criterion that provides the best evaluation performance is emphasized in boldface (all the blind methods that do not depend on DMOSs). As shown in Table 1, compared with other methods, it is evident that our method correlates well with the DMOSs, owing to its sufficient application of human visual perception. It thereby delivers a performance that is superior to most blind and FR methods, including SSIM and IL_NIQE, and the methods proposed by Xue, Lin, Shao, and Zhou. Thus, our method is an accurate, effective, and consistent S3D-IQM method, even though it does not use human-scored S3D images.

Another experiment was conducted to further investigate the evaluation ability of the IQM methods for specific distortion types. In this experiment, we tested the performance of the state-of-the-art methods on individual distortion types. The SROCC and PLCC results are listed in

Table 2. Performance of nine methods for each individual distortion type.

Distortion	Criteria	FR		Blind
				Depending on DMOS				Not Depending on DMOS
		SSIM	Lin [2]	Akhter [4]	Chen [5]	Yue [9]	Zhou [12]	IL_NIQE	Xue [16]	Shao [13]	Zhou [14]	Proposed
JP2K	PLCC	0.865	0.838	0.905	0.907	0.934	0.988	0.854	0.919	0.901	0.848	0.939
	SROCC	0.857	0.839	0.866	0.863	0.832	0.961	0.861	0.886	0.870	0.837	0.887
JPEG	PLCC	0.485	0.214	0.729	0.695	0.744	0.916	0.533	0.722	0.456	0.626	0.673
	SROCC	0.435	0.199	0.675	0.617	0.595	0.912	0.544	0.682	0.429	0.638	0.612
WN	PLCC	0.937	0.928	0.904	0.917	0.962	0.988	0.927	0.858	0.950	0.925	0.943
	SROCC	0.940	0.928	0.914	0.919	0.932	0.965	0.920	0.938	0.914	0.931	0.909
Gblur	PLCC	0.920	0.948	0.617	0.917	0.971	0.974	0.904	0.923	0.919	0.899	0.976
	SROCC	0.882	0.935	0.555	0.878	0.857	0.855	0.873	0.871	0.932	0.833	0.903

. For each individual distortion type, the top two results are emphasized in boldface (all blind methods that do not depend on DMOS). As shown in Table 2, our method placed among the top methods most often. Although some methods may provide good performance for some distortion types, the proposed method performs better than, or on par with, the best method for each individual distortion type.

Next, we discuss the influence of disparity estimation performance on the proposed method. The performance of the proposed method using ground truth, SAD [27], and Klaus [21] are listed in

Table 3. The performance of the proposed method using different stereo methods.

Stereo Method	Pearson’s Linear Correlation Coefficient (PLCC)	Spearman’s Rank Order Correlation Coefficient (SROCC)
Ground Truth	0.927	0.891
SAD	0.921	0.883
Klaus	0.925	0.887

. Table 3 shows that a similar performance is attained when using either the ground truth or the estimated disparity.

4. Conclusions

In this paper, we proposed a new blind S3D-IQM method that does not depend on DMOS during learning. The main contribution of this study is that we constructed a quality-aware codebook to obtain blind IQM using a perceptual pooling strategy. Experimental results show that the proposed method can produce assessments that are statistically much more consistent with DMOS.

Further work will mainly focus on: (1) other binocular visual perception (e.g., binocular fusion, binocular suppression) can be considered for improving the performance of the proposed method, (2) the effect of disparity information needs to be further addressed and, besides disparity, many other factors, such as monocular depth cues, visual comfort, visual direction, etc., should be considered in modeling S3D-IQM without relying on human-scored S3D images.