5.1. Findings and Their Implications
The two previous sections have presented the development and experimental results of the proposed no-reference evaluation model in assessing the quality of depth maps used in view rendering via DIBR. This model has exploited the edges existing in the available depth maps and compared those to the edges in the corresponding colour views. Through this comparison, edge pixels in-depth maps have been classified into correct and error edge pixels. The results have been correlated with the subjective results. The result analysis of the model has provided clear indications of depth map performance and its dependency on the associated colour sequence selection.
The novelty of this work is realised by the adoption of a proposed depth map quality measure, namely the DEC measure, as an indication of the quality of the view rendered using the particular depth map whose quality has been scrutinised. DEC measure has proved to be a very powerful tool in quantifying the quality of a depth map in rendering the DIBR based view. No-reference approach adopted whilst developing the proposed quality evaluation model is different from the conventional approaches of no-reference quality assessment methods. While conventional methods are blindly assessing the quality of colour views directly, our proposed model provided the quality measure indirectly by assessing the quality of the depth map used in the rendering process.
The developed no-reference quality evaluation model can be used to indicate the quality of the depth map used in the rendering process, as it has provided strong correlation with the subjective assessment results of the views rendered using these depth maps, as seen in Figure 10
. The developed model produced good results in evaluating the quality of the rendered views, particularly when compared to the results of the traditional 2D full-reference objective quality assessment methods or the conventional no-reference methods reported in the literature. To be specific, the DEC measurement-based model has offered a 27% improvement in correlation with the subjective results when compared to the VQM correlation results when compared to the results shown in Table 3
and Table 4
. A better performance can also be observed when the results presented in Figure 8
and Figure 10
are compared. This improvement has also been verified by considering the correlation for the range of video sequences tested. VQM has been selected for benchmarking, as it was overall the best performing objective quality assessment method among the state-of-the-art methods employed in this research.
It is also clear from the results presented in Table 2
that the conventional approaches to no-reference quality evaluation do not offer good results when the views assessed are the views resulting from the DIBR process. This can be explained by the fact that these methods are designed to assess conventional artefacts and distortions in a natural scene. The artefacts specific to the synthesised DIBR rendered views are of a different nature. Thus, measuring the quality of the depth map used in the rendering process, as advocated by the study presented in this paper, is more representative of the overall quality of the rendered view.
Another observation that can be made from the obtained experimental results is related to the effect of using different colour sequences in the rendering process on the resulting quality of the rendered view. It seems it is more beneficial to use colour sequences with high spatio-temporal information to more accurately assess the performance of a set of depth maps in the rendering process. This observation is more important when the depth dataset has relatively equivalent quality performances. As a final observation, the developed model can be considered as an effective measurement of the quality of the depth map used in the rendering process, particularly when there is a lack of real depth reference in immersive video applications for the depth comparison purposes.
In summary, subjective evaluation results have been used for training a model, and through a series of regression and curve-fitting operations, we have arrived at devising our resulting equation (Section 3.1.2
) which plays the centre role in predicting the subjective evaluation results as close as possible to the real subjective assessment scores. The results then demonstrate that the accuracy obtained is better than those of 2D objective metrics and/or other no-reference techniques presented earlier. By nature, the proposed no-reference quality evaluation model based on DEC measurement is limited in computational complexity, since the calculation of edges requires a two-dimensional filter. Furthermore, the training process is performed offline to compute the model coefficients, which with today’s fast CPU and GPU capabilities and availability is not a demanding operation as opposed to those other quality evaluation techniques that are based heavily on machine learning techniques.
We have presented the results compared with those obtained from the literature, based on conventional implementations (as introduced in Section 2.2
), where possible and available. We also compared our results against the well-known objective quality metrics and showed that our model performed significantly better than those.
5.2. Future Research Directions
Depth map quality is a key area in developing immersive video applications, as it provides the potential for generating more views available at the receiver end without excessively utilising valuable communication bandwidth resources. As such, further research will be beneficial in the following directions. Firstly, developing a complementary measure that consists of inter-view depth confidence, coupling with the depth map quality model proposed here, is envisaged to offer a higher degree of objective quality estimation when richer MVD datasets are available for testing. The inter-view depth confidence can be explained as a per-view and per-pixel spatial confidence map that outlines the pixel positions, for which the depth value is consistent across different views. A spatial confidence map can be generated in the forward projection of the depth values of all available views into the coordinate system of a common viewpoint (i.e., the synthesised view position). This is then followed by the calculation of depth differences at every pixel location. This will provide added depth confidence measure when two reference colour views and their associated depth maps are used in the rendering process. The overall model can combine the DEC measurement presented in this paper together with inter-view confidence measurement with the prospect for predicting the quality of the resulting rendered views in much higher accuracies.
Secondly, limitations on accurate depth data generation for wide baseline camera setups call for further research. Depth map production for wide baseline camera setups is very challenging that has been widely researched. Depth map quality has an impact on the quality of the rendered views in those camera setups. Particularly, depth quality is expected to have a more prominent impact when the target rendered view location is set at a further distance from the reference colour view than the stereoscopic distance. As such, errors in mapping colour information from the reference to the correct locations in the target view will significantly increase with the growing distance between the views. The DEC measurement-based model, together with the aforementioned inter-view depth confidence measurement, would provide a remedy to overcome inaccuracies owing to the larger view distances in-depth map quality prediction in these setups.
Lastly, future research may involve studying the effect of utilising the developed depth map quality model as direct feedback into the depth production process, where it provides an input for the depth estimation and post-processing techniques. This can particularly be beneficial when real-time depth production process is considered for enhancing QoE of viewers in live 3DTV/FTV immersive video applications. The advantage of such implementation would come from exploiting the DEC measurement-based model’s performance in instantly assessing depth map quality. This is essential in enabling faster and more convenient depth map production without resorting to the time and effort consuming subjective assessment methodology each time new developments into depth map production are introduced.