You are currently viewing a new version of our website. To view the old version click .
Sensors
  • Article
  • Open Access

Published: 19 April 2021

Impact of Scene Content on High Resolution Video Quality

,
,
and
Department of Multimedia and Information-Communication Technology, University of Zilina, Univerzitna 1, 010 26 Zilina, Slovakia
*
Author to whom correspondence should be addressed.
This article belongs to the Special Issue Smart Sensor Technologies for IoT

Abstract

This paper deals with the impact of content on the perceived video quality evaluated using the subjective Absolute Category Rating (ACR) method. The assessment was conducted on eight types of video sequences with diverse content obtained from the SJTU dataset. The sequences were encoded at 5 different constant bitrates in two widely video compression standards H.264/AVC and H.265/HEVC at Full HD and Ultra HD resolutions, which means 160 annotated video sequences were created. The length of Group of Pictures (GOP) was set to half the framerate value, as is typical for video intended for transmission over a noisy communication channel. The evaluation was performed in two laboratories: one situated at the University of Zilina, and the second at the VSB—Technical University in Ostrava. The results acquired in both laboratories reached/showed a high correlation. Notwithstanding the fact that the sequences with low Spatial Information (SI) and Temporal Information (TI) values reached better Mean Opinion Score (MOS) score than the sequences with higher SI and TI values, these two parameters are not sufficient for scene description, and this domain should be the subject of further research. The evaluation results led us to the conclusion that it is unnecessary to use the H.265/HEVC codec for compression of Full HD sequences and the compression efficiency of the H.265 codec by the Ultra HD resolution reaches the compression efficiency of both codecs by the Full HD resolution. This paper also includes the recommendations for minimum bitrate thresholds at which the video sequences at both resolutions retain good and fair subjectively perceived quality.

1. Introduction

In recent years, the number of various types of surveillance and data collection cameras located both indoors and outdoors have been constantly increasing. Popularity of home security cameras is also growing as even high-quality models become more affordable. Typical surveillance cameras applications include public safety, protection of facilities against theft or vandalism, remote video monitoring, traffic surveillance, weather monitoring, or more special cases, such as animal monitoring or data collection, for statistical or marketing purposes. Today, due to the pandemic situation, face recognition with and without a protective mask is also becoming a point of interest for researchers in cooperation with technology companies [1,2,3]. It is important to realize that each such employed sensor produces a tremendous amount of data to be subsequently transmitted over the network or further processed, which calls for effective video compression. Furthermore, whether the image or video is presented to a live person or a machine learning algorithm (most often for its classification or segmentation), the best results can be achieved when the image is of the highest achievable quality. This implies one common goal for the distributors, communication service providers, or even broadcasting companies, to optimally set the compression parameters so that perceived video quality is maximal, while the bandwidth requirements are minimal. This challenge leads to increased interest in the analysis of video content followed by the individual setting of the compression parameters of video sequences with different types of scene content. Even though many studies deal with the video quality assessment using subjective methods, the demand exceeds the supply; there is still a lack of video quality datasets, as well as recorded subjective tests, conducted on these datasets. Very popular and extensively used datasets, such as References [4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27], come from the University of Texas and were developed by the Laboratory for Image and Video Engineering. Another very popular option is the VQEG-HDTV database [28], which is a result of international project of VQEG (Video Quality Experts Group) consortium. Other well-known datasets are BVI-HD [29], BVI textures [30] and BVI-HFR [31] developed at the University of Bristol, AVT-VQDB-UHD-1 database [32] made by the Ilmenau University of Technology, Ultra Video Group (UVG) dataset [33] composed at the Tampere University, SJTU 4K video quality database [34] from the Shanghai Jiao Tong University, Image and Video Processing Subjective Quality Video Database [35] developed at the Chinese University of Hong Kong, collection of IRCCyN/IVP databases from the Institut de Recherche en Communications et Cybernétique de Nantes [36], Konstanz Natural Video Database (KoNViD-1k) made by the Universitaet Konstanz [37,38], MCL-V [39], and [40] databases from the MSC University of Southern California, Scalable Video Database [41,42] composed at the EPFL, ReTRiEVED Video Quality Database [43] made by the Universita Degli Studi or TUM databases [44,45] developed by the Technical University of Munich. Taking into account the demand and importance of measuring the performance of video quality assessment techniques, a number of studies on perceptual evaluation was presented. Most of them merely compare the quality of video sequences with various characteristics evaluated by different subjective methods and do not examine the content aspect. Rerabek et al. [46] examined a rate-distortion performance analysis and mutual comparison of one of the latest video coding standards H.265/HEVC with VP9 codec. Ramzan et al. [47] presented a performance evaluation of three coding standards—Advanced Video Coding (H.264/MPEG-AVC), High-Efficiency Video Coding (H.265/MPEG-HEVC), and VP9, based on subjective and objective quality evaluations. Two different sequences at both resolutions (Full HD and Ultra HD) were tested using the DSIS method. Bienik et al. [48] measured the impact of the compression formats, namely H.264, H.265, and VP9 on perceived video quality. The evaluation was performed on four Full HD sequences using the Absolute Category Rating (ACR) and DSCQS methods. Xu et al. [49] presented a subjective video quality assessment on 4K Ultra-High Definition (UHD) videos using the DSCQS method. Six different test sequences were used for the evaluation.Herrou et al. [50] focused on a performance comparison between HEVC and VP9 in the HDR context through both objective and subjective evaluations. Dumic et al. [51] offered findings on subjective assessment of H.265 versus H.264 Video Coding for High-Definition Video Systems. For the evaluation, a database consisting of 120 degraded HD video sequences with 4 contents encoded at various compression rates to H.265/HEVC and H.264/AVC formats was compiled. Milovanovic et al. [52] subjectively compared the coding efficiency of three video coding standards (MPEG-H HEVC, H.264/MPEG-4 AVC, and H.262/MPEG-2). Sotelo et al. [53] presented a subjective quality assessment of HEVC/H.265 compressed 4K Ultra-High-Definition (UHD) videos in a laboratory viewing environment. Kufa et al. [54] explored coding efficiency performance of High Efficiency Video Coding (HEVC) and VP9 compression formats on video content in Full HD and UHD resolutions. Deep et al. [55] focused on the comparison of HEVC and VP9 based on both subjective and objective evaluation on various (720p, 1080p, and 2160p) test videos. Akyazi et al. [56] examined the compression efficiency of HEVC/H.265, VP9, and AV1 codecs based on subjective quality assessment. Our survey of research papers shows that there is still a lack of databases of video sequences annotated according to subjective evaluation. Therefore, this paper brings new subjective results and also explores the impact of the video content on the subjective assessment. We decided to compare today’s most used compression standards—H.264/AVC and H.265/HEVC—on video sequences at Full HD and Ultra HD resolutions. Our publication follows Reference [57], where a new 4K video dataset was compiled with full subjective scores (Mean Opinion Score (MOS)) of videos at different bitrates compressed by HEVC/H.265 codec evaluated by the Double Stimulus Impairment Scale (DSIS) method, variant II. For our measurements, we decided to use the Absolute Category Rating (ACR) method.

2. Dataset Description and Preparation

2.1. Dataset Description

For our measurements, we used the dataset from the Media Lab of the Shanghai Jiao Tang University [34]. We selected eight sequences with various scene content, illustrated in Figure 1, classified according to the Temporal Information (TI) and Spatial Information (SI) from this database. SI defines the amount of spatial detail in an image and is higher for more spatially complex scenes, while TI represents the number of temporal changes in a video sequence and is higher for high motion sequences [58]. The spatial perceptual information is based on the Sobel filter and is represented by the formula:
S I = m a x t i m e [ s t d s p a c e [ S o b e l ( F n ) ] ] ,
where F n stands for video frame, and s t d s p a c e for the standard deviation over the pixels in each Sobel-filtered frame. The temporal information is computed as:
T I = m a x t i m e [ s t d s p a c e [ M n ( i , j ) ] ] ,
where M n ( i , j ) is the difference between pixels at the same position in the frame belonging to two consecutive frames, i.e.,
M n ( i , j ) = F n ( i , j ) F n 1 ( i , j ) ,
where F n ( i , j ) is the pixel at the i-th row, and j-th column of n-th frame in time [58]. Both of these parameters were calculated for each sequence using the Mitsu tool [59] and plotted in Figure 2. The general specification of the dataset is given in Table 1, and the content of individual sequences is briefly described in Table 2.
Figure 1. Printscreens of used test sequences. Reprinted with permission from [60], Copyright 2021, Uhrina.
Figure 2. Spatial Information (SI) and Temporal Information (TI) diagram of used test sequences. Reprinted with permission from [60], Copyright 2021, Uhrina.
Table 1. Parameters of test sequences.
Table 2. Characteristics of the test sequences.

2.2. Dataset Preparation

In our research, we decided to explore the quality of 8-bit video sequences at two commonly used resolutions, i.e., Full HD (FHD) and Ultra HD (UHD) with a typical chroma subsampled YUV 4:2:0 format. Because original sequences were uncompressed and YUV 4:4:4 color format at Ultra HD resolution with 10-bit depth was used, we had to convert them to the appropriate formats. Therefore, all test sequences were first chroma subsampled from YUV 4:4:4 to YUV 4:2:0 format and also the bit depth was changed from 10 to 8 bits per channel. Subsequently, all these conversion steps were repeated for Full HD resolution utilizing the FFmpeg tool [61]. As we wanted to assess also Full HD in addition to Ultra HD, the resolution also had to be altered. For all these conversion steps we used once again the FFmpeg tool [61]. Correspondingly, two uncompressed test sequences were generated (Figure 3) for each type of content, which adds up to 16 videos. We call them the source video sequences (SRCs) for the rest of this paper.
Figure 3. Process of preparing the test sequences: chroma subsampling, bit depth, and resolution changing.

2.3. Coding Process

All these test video sequences (SRCs) were afterwards encoded to both compression standards to be evaluated, i.e., H.264/AVC and H.265/HEVC. As a quality restriction parameter, we decided to use the constant bitrate. We selected 5 various target bitrates: 1, 3, 5, 10 and 15 Mbps based on our previous research [62] which have shown the efficiency of codecs growing nonlinearly with increasing bitrate. We have limited the number of bitrates to 5 as a compromise between the complexity and time requirements of subjective testing and precision of the measurements. For the purposes of our research, we decided to use the Group of Pictures (GOP) length typical for video intended for transfer over a noisy communication channel. The GOP length is based on the framerate of used video sequences and is commonly set to half of the framerate value. Accordingly, given that test video sequences had a framerate of 30 fps (frames per second), we chose the GOP length of 15 frames, i.e., M = 3, N = 15. The first number, labeled with M letter, expresses the distance between two anchor frames (I or P) and the second number, denoted with N letter, stands for the distance between two key frames (I). For this coding process, we used once again the FFmpeg tool, which contains libraries x264 and x265 for H.264/AVC and H.265/HEVC codec, respectively [61], creating the total of 160 video sequences for the subjective quality assessment. We refer to them as PVSs (Processed Video Sequences) for the rest of this paper. The FFmpeg command example for encoding the Wood test sequence to the H.264 format at 1 Mbps bitrate is: ffmpeg -i Wood_1920x1080_30fps_420_8bit_YUV.yuv -vcodec libx264 -command example-params keyint=15:min-keyint=15:bframes=3:b-adapt=1:bitrate=1000:vbv-maxrate=1000:vbv-bufsize=1000 Wood_1920x1080_30fps_420_8bit_H264_01M.mp4.

3. Subjective Quality Assessment

During the subjective testing, all created PVSs were shown to people of different ages and genders to evaluate their quality. We decided to use the Absolute Category Rating (ACR) method [58,63] which belongs to the category of Single Stimulus (SS) subjective video quality assessment techniques. The principle of this method is that the degraded sequences are presented to the observers one at a time, and they are asked to rate its quality on a five-level grading scale, where 1 indicates the bad quality, and 5 stands for the excellent quality. The measurement was conducted in two laboratories separately: one situated at the University of Zilina (UNIZA), and the second at the VŠB – Technical University in Ostrava. The video sequences were presented on three types of displays (Table 3) depending on the resolution of the test sequences in the laboratories under normal indoor illumination conditions.
Table 3. Types of used displays.
Thirty participants, mostly students, were involved in the testing in each laboratory. All of them were naive observers which means they had no expertise in the image artefacts that may be introduced by the system under test. Naturally, they were thoroughly acquainted with the method of assessment, types of impairment, grading scale, sequence, and timing as required by Reference [58]. The statistical distribution of the number of men and women who took part in the tests, as well as the average age of all observers, is shown in Table 4. The course of the entire subjective assessment process is represented by Figure 4.
Table 4. Statistical characteristic of the observers.
Figure 4. Complete process of coding and assessing the video quality.

4. Statistical Analysis and Presentation of the Results

After performing the subjective tests, we processed all collected results statistically; for each test sequence, codec, and resolution, the Mean Opinion Score (MOS) and 95 percent Confidence Interval (CI) in accordance with Reference [64] were calculated and plotted in graphs, a shown below. The presentation of the results could be divided into five parts. In the first part, the cross-comparison of the results obtained from different laboratories, i.e., from UNIZA and VŠB, is performed using the Pearson correlation coefficient (PCC) and the Root Mean Square Error (RMSE). In the second part, the bitrate impact on the perceived video quality depending on the scene content is plotted. The third part deals with the Analysis of Variance (ANOVA) which was applied on the acquired data. In the fourth part, the impact of the bitrate on the perceived video quality in terms of the used codec and resolution is presented. Finally, in the fifth part, the minimum bitrate thresholds at which the video sequence should be encoded to reach certain quality are determined.

4.1. Correlation between the Results from Individual Laboratories

To compare the MOS values obtained from both laboratories, i.e., from UNIZA and VŠB, and, to find out the correlation, the Pearson correlation coefficient (PCC), as well the Root Mean Square Error (RMSE) were calculated. All computations were done for both codecs and resolutions, as well as for all test sequences. The results are plotted in Figure 5 and Figure 6 and are shown in Table 5.
Figure 5. Comparison of Mean Opinion Score (MOS) values obtained from different laboratories. Each spot represents MOS values for corresponding codec, resolution, and test sequence
Figure 6. Comparison of MOS values obtained from different laboratories. Each spot represents averaged MOS values from particular test sequences for corresponding codec and resolution.
Table 5. Correlation of MOS score between the laboratories.
As we can see from Figure 5 and Figure 6, as well as from Table 5, there is a high correlation between the results from both laboratories. The lowest correlation was reached by the combination of Full HD resolution and H.264 codec. This is most likely due to the different displays used in the assessments; at the UNIZA laboratory, the Full HD display was used, while, at the VŠB laboratory, the Ultra HD display was used. Vice versa, the highest correlation rate was achieved by video sequences encoded to H.264 at UHD resolution.

4.2. Impact of Bitrate on Video Quality Depending on Scene Content

Figure 7 shows the impact of the bitrate on the perceived video quality (defined by the MOS with associated CI). In this figure, eight graphs are inserted considering used codec, resolution, and laboratory where the evaluation was conducted. Sequences with different scene contents are color-coded in the graphs; each curve represents MOS values for a given test sequence. Figure 8 shows the average MOS values obtained from UNIZA and VSB laboratories.
Figure 7. Bitrate impact on the perceived video quality (defined by the MOS score with associated Confidence Interval (CI)) depending on codec and resolution for both laboratories independently. Each curve represents MOS values for each type of used test sequence.
Figure 8. Bitrate impact on the perceived video quality (defined by the MOS score with associated CI) depending on the codec and resolution for both laboratories jointly. Each curve represents averaged MOS values from both laboratories for each type of used test sequence.
It is apparent from the graphs that the sequences with the lowest SI and TI values, such as the “Bund Nightscape” and the “Construction Field”, reached the best MOS value. Vice versa, the observers rated the sequences situated in the middle of the SI-TI diagram, such as the “Marathon” or “Runners”, as of worst quality. Interesting cases are the “Campfire Party” and the “Fountains” sequences. The “Campfire Party” contains a lot of movement (high TI values) but not many details (low SI values) and reached low MOS value, while the “Fountains” sequence lies near to the “Bund Nightscape” and the “Construction Field” sequences, meaning it has low both TI and SI values and also scored low on the MOS scale. A special case is the “Wood” sequence which is situated at the upper right corner of the SI-TI diagram. Nevertheless, its quality was perceived as similar to the sequences “Fountains” and “Runners”. All these differences are more pronounced:
  • at low bitrates—with increasing bitrate, the perceived quality rises, too, and approaches the perceived quality of sequences with low SI-TI values,
  • at Ultra HD resolution rather than at Full HD resolution, and
  • at H.265 codec rather than at H.264 codec.
Based on these results, we can state that the compression efficiency and related video quality depends on the content of the sequences. However, the sequence representation and description only by the spatial and temporal information is not sufficient and should be the subject of further research. We suggest other parameters should be used to describe the scene, such as, for instance, the luminance and contrast or the colors occurring in the scene. In addition, the psychological factors should be considered. Based on the results, we can also state that the temporal information has greater impact on the perceived quality than the number of the objects defined by the spatial information.

4.3. Analysis of Variance

To verify what stemmed from the graphical representation of the subjective evaluation results, the ANOVA was applied on the data [65]. The three-way ANOVA was used to compare the significance and influence of individual sequence parameters on the resulting perceived video quality. The interaction between three independent variables, bitrate (X1), content (scene type) (X2), and resolution (X3) in Table 6 or compression standard (X3) in Table 7 was examined, with video quality being considered a dependent variable. Table 6 and Table 7 depict the three-way ANOVA matrices. The F-value, also called the F-ratio is calculated as the variance of the group means divided by the mean of the within group variances (Mean Squared Error). Greater F-value indicates more significant variation. In ANOVA, the p-value, i.e., the probability of getting the observed result at random, is also determined. For the source of variation to be regarded as insignificant, the p-value must be higher than a given alpha level, commonly set to 0.05. When performing ANOVA, the p-value is also determined to investigate the probability of rejecting the hypothesis.
Table 6. Three-way Analysis of Variance (ANOVA) using video codec as a criterion.
Table 7. Three-way ANOVA using video resolution as a criterion.
Based on the analysis of the tables, the following conclusions can be drawn. Table 6 indicates that for H.265 encoded sequences, the effect of resolution can be ignored, since this variable was deemed statistically insignificant. In contrast, in the case of the H.264 codec, this negative phenomenon does not occur and resolution is the second most important parameter that determines the subjectively perceived quality. For both codecs, an alteration in bitrate results in a maximum change in the subjective MOS. According to Table 7, the impact of compression format on the perceived quality is considered statistically insignificant for Full HD video sequences. However, that is not the case for Ultra HD resolution, where deployed codec is the second most influential variable. Equivalently to Table 6, the bitrate has the greatest effect on the subjective video quality assessment results. All remaining ANOVA test results in both tables can be regarded statistically significant based on their p-values.

4.4. Impact of Bitrate on Video Quality Depending on Codec and Resolution

Figure 9 shows the impact of the bitrate on the perceived video quality (defined by the MOS with associated CI) plotted separately for each type of video sequence. In this figure, eight graphs are inset, considering examined test sequence, which show the impact of used codec and resolution on the perceived quality of a given sequence; curve represents averaged MOS values from both laboratories for a given codec and resolution.
Figure 9. Bitrate impact on the perceived video quality (defined by the MOS score with associated CI) depending on used test sequence. Each curve represents averaged MOS values from both laboratories for corresponding codec and resolution.
In Figure 10, the averaged MOS value from both laboratories from all used test sequences for each codec and resolution is plotted.
Figure 10. Bitrate impact on the perceived video quality (defined by the MOS score with associated CI). Each curve represents averaged MOS values from both laboratories for corresponding codec and resolution—average MOS score.
We can draw several conclusions from Figure 9 and Figure 10. Firstly, it is apparent that the H.265 compression standard yields better quality than the H.264 codec. This is a generally known fact and we expected it. But what is interesting and important is that the efficiency difference between these two codecs is negligible for the Full HD video sequences. Therefore, it is inessential to use H.265 compression standard at this resolution, as the observers will not see any notable differences. The use of H.265 codec is relevant only for the videos at the Ultra HD resolution, particularly at low bitrates. This is due to the fact that the quality of H.264 encoded video sequences increases with the rising bitrate up to the point where it reaches or even surpasses the perceived quality of H.265 sequences. Secondly, the compression efficiency of the H.265 compression standard at the Ultra HD resolution reaches the compression efficiency of both codecs at the Full HD resolution.
Indisputably, the conclusions drawn from the Analysis of Variance (ANOVA) and the graphical representation of the subjective quality evaluation results coincide. These findings could be beneficial for visual media content providers and broadcasting companies, as they indicate how to adjust video compression parameters to improve its quality. The fastest growth of perceived video quality is apparently due to an increase in bitrate. Specifically, the quality increases most rapidly until the bitrate reaches a value of approximately 5 Mbps. The analyses also revealed which combination of resolution and compression format is best used so that the resulting quality of visual content is perceived by viewers as good as possible.

4.5. Minimum Bitrate Thresholds Suggestions

Finally, Figure 10 shows the minimum bitrate thresholds at which the video sequences should be encoded to achieve good (4) or fair (3) quality. These quality thresholds are based on MOS values of used ACR method and are important for the bitrate setting of each codec to maintain a certain quality. Table 8 shows the mentioned minimum bitrates.
Table 8. Minimum bitrate thresholds to achieve good (4) and fair (3) video quality.
From Table 8, it follows that to achieve a good quality (value 4 on MOS scale), the video sequence must be coded to minimum 7.50 Mbps by both codecs for Full HD resolution and to 11.55 Mbps by H.264 codec and 9.00 Mbps by H.265 codec for Ultra HD resolution. To reach fair quality (value 3 on MOS scale), the minimum thresholds for the bitrates are 2.80 Mbps by H.264 codec and 2.60 Mbps by H.265 codec for Full HD resolution and 4.50 Mbps by H.264 codec and 2.80 by H.265 codec for Ultra HD resolution.

5. Conclusions

This paper dealt with the content impact on the perceived video quality evaluated using the subjective Absolute Category Rating (ACR) method. Eight types of video sequences with various scene content were evaluated. Two widely used video compression standards H.264/AVC and H.265/HEVC in combination with Full HD and Ultra HD resolutions, were tested. In the coding process, we selected 5 various bitrates based on our previous research, which showed that the efficiency of codecs grows nonlinearly with increasing bitrate. The number of bitrates was a compromise between the complexity and time requirements of subjective testing. In total, we created an annotated database which contains 160 different video sequences coded at constant bitrates with GOP set to half of the framerate value which is typical for video intended for transfer over a noisy communication channel. The perceived quality of the sequences was evaluated employing the subjective ACR method. The assessment was conducted in two laboratories: one situated at the University of Zilina, and the second at the VSB—Technical University in Ostrava. First, we calculated the correlation of the MOS values between both laboratories using the Pearson correlation coefficient (PCC) and the Root Mean Square Error (RMSE). The correlation proved to be considerably high. After that, we described the impact of the bitrate on video quality depending on scene content defined by Spatial (SI) and Temporal information (TI). The results showed that even if the sequences with low SI and TI values reach better MOS than the sequences with higher SI and TI values, these two parameters are not sufficient for scene description, and this domain should be the subject of further research. Subsequently, we described the impact of bitrate on video quality depending on codec and resolution. Based on the results, we concluded that the employment of the H.265 codec for compression of Full HD sequences is inessential, as the people did not observe any significant differences. Furthermore, we stated that the compression efficiency of the H.265 codec by the Ultra HD resolution reaches the compression efficiency of both codecs by the Full HD resolution. We also applied the ANOVA to verify what stemmed from the graphical representation of the subjective evaluation results. Finally, we determined the minimum bitrate thresholds at which the video sequences at both resolutions retain good and fair subjectively perceived quality.

Author Contributions

Conceptualization, A.H. and M.U.; methodology, J.B. and L.S.; validation, J.B., L.S. and M.U.; formal analysis, A.H. and J.B.; investigation, J.B. and L.S.; resources, M.U.; data curation, J.B. and L.S.; writing—original draft preparation, A.H. and M.U.; writing—review and editing, A.H., L.S. and M.U.; visualization, L.S. and M.U; project administration, M.U. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the Slovak Research and Development Agency under the project PP-COVID-20-0100: DOLORES.AI: The pandemic guard system.

Institutional Review Board Statement

Not applicable.

Data Availability Statement

The data presented in this study are available on request from the corresponding author.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses or interpretation of data; in the writing of the manuscript, or in the decision to publish the result.

References

  1. Damer, N.; Grebe, J.H.; Chen, C.; Boutros, F.; Kirchbuchner, F.; Kuijper, A. The Effect of Wearing a Mask on Face Recognition Performance: An Exploratory Study. In Proceedings of the 2020 International Conference of the Biometrics Special Interest Group (BIOSIG), Darmstadt, Germany, 16–18 September 2020; pp. 1–6. [Google Scholar]
  2. Joshi, A.S.; Joshi, S.S.; Kanahasabai, G.; Kapil, R.; Gupta, S. Deep Learning Framework to Detect Face Masks from Video Footage. In Proceedings of the 2020 12th International Conference on Computational Intelligence and Communication Networks (CICN), Bhimtal, India, 25–26 September 2020. [Google Scholar]
  3. Tan, W.; Liu, J. Application of Face Recognition in Tracing COVID-19 Fever Patients and Close Contacts. In Proceedings of the 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), Miami, FL, USA, 14–17 December 2020. [Google Scholar]
  4. Seshadrinathan, K.; Soundararajan, R.; Bovik, A.C.; Cormack, L.K. Study of Subjective and Objective Quality Assessment of Video. IEEE Trans. Image Process. 2010, 19, 1427–1441. [Google Scholar] [CrossRef] [PubMed]
  5. Seshadrinathan, K.; Soundararajan, R.; Bovik, A.C.; Cormack, L.K. A subjective study to evaluate video quality assessment algorithms. In Human Vision and Electronic Imaging XV; Rogowitz, B.E., Pappas, T.N., Eds.; SPIE: Bellingham, WA, USA, 2010. [Google Scholar]
  6. Moorthy, A.K.; Choi, L.K.; Bovik, A.C.; de Veciana, G. Video Quality Assessment on Mobile Devices: Subjective, Behavioral and Objective Studies. IEEE J. Sel. Top. Signal Process. 2012, 6, 652–671. [Google Scholar] [CrossRef]
  7. Moorthy, A.K.; Choi, L.K.; Veciana, G.; Bovik, A. Subjective Analysis of Video Quality on Mobile Devices. In Proceedings of the 6th International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM), Scottsdale, AZ, USA, 19–20 January 2012; Volume 720, pp. 1–6. [Google Scholar]
  8. Chen, C.; Choi, L.K.; de Veciana, G.; Caramanis, C.; Heath, R.W.; Bovik, A.C. A dynamic system model of time-varying subjective quality of video streams over HTTP. In Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada, 26–31 May 2013. [Google Scholar]
  9. Chen, C.; Zhu, X.; de Veciana, G.; Bovik, A.C.; Heath, R.W. Adaptive video transmission with subjective quality constraints. In Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, France, 27–30 October 2014. [Google Scholar]
  10. Chen, C.; Choi, L.K.; de Veciana, G.; Caramanis, C.; Heath, R.W.; Bovik, A.C. Modeling the Time—Varying Subjective Quality of HTTP Video Streams With Rate Adaptations. IEEE Trans. Image Process. 2014, 23, 2206–2221. [Google Scholar] [CrossRef] [PubMed]
  11. Chen, C.; Zhu, X.; de Veciana, G.; Bovik, A.C.; Heath, R.W. Rate Adaptation and Admission Control for Video Transmission With Subjective Quality Constraints. IEEE J. Sel. Top. Signal Process. 2015, 9, 22–36. [Google Scholar] [CrossRef]
  12. Choi, L.K.; Cormack, L.K.; Bovik, A.C. On the visibility of flicker distortions in naturalistic videos. In Proceedings of the 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX), Klagenfurt am Wörthersee, Austria, 3–5 July 2013. [Google Scholar]
  13. Choi, L.K.; Cormack, L.K.; Bovik, A.C. Motion silencing of flicker distortions on naturalistic videos. Signal Process. Image Commun. 2015, 39, 328–341. [Google Scholar] [CrossRef]
  14. Choi, L.K.; Cormack, L.K.; Bovik, A.C. Eccentricity effect of motion silencing on naturalistic videos. In Proceedings of the 2015 IEEE Global Conference on Signal and Information Processing (GlobalSIP), Orlando, FL, USA, 14–16 December 2015. [Google Scholar]
  15. Choi, L.K.; Cormack, L.K.; Bovik, A.C. LIVE Flicker Video Database. 2015. Available online: http://live.ece.utexas.edu/research/quality/live_flicker_video.html (accessed on 18 April 2021).
  16. Ghadiyaram, D.; Bovik, A.C.; Yeganeh, H.; Kordasiewicz, R.; Gallant, M. Study of the effects of stalling events on the quality of experience of mobile streaming videos. In Proceedings of the 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP), Atlanta, GA, USA, 3–5 December 2014. [Google Scholar]
  17. Ghadiyaram, D.; Bovik, A.; Yeganeh, H.; Kordasiewicz, R.; Gallant, M. LIVE Mobile Stall Video Database, 2016. Available online: http://live.ece.utexas.edu/research/LIVEStallStudy/index.html (accessed on 18 April 2021).
  18. Ghadiyaram, D.; Pan, J.; Bovik, A.C.; Moorthy, A.K.; Panda, P.; Yang, K.C. In-Capture Mobile Video Distortions: A Study of Subjective Behavior and Objective Algorithms. IEEE Trans. Circuits Syst. Video Technol. 2018, 28, 2061–2077. [Google Scholar] [CrossRef]
  19. Ghadiyaram, D.; Pan, J.; Bovik, A.C.; Moorthy, A.K.; Panda, P.; Yang, K.C. LIVE-Qualcomm Mobile In-Capture Video Quality Database, 2017. Available online: http://live.ece.utexas.edu/research/incaptureDatabase/index.html (accessed on 18 April 2021).
  20. Bampis, C.G.; Li, Z.; Moorthy, A.K.; Katsavounidis, I.; Aaron, A.; Bovik, A.C. Study of Temporal Effects on Subjective Video Quality of Experience. IEEE Trans. Image Process. 2017, 26, 5217–5231. [Google Scholar] [CrossRef] [PubMed]
  21. Bampis, C.G.; Li, Z.; Moorthy, A.K.; Katsavounidis, I.; Aaron, A.; Bovik, A.C. LIVE Netflix Video Quality of Experience Database, 2016. Available online: http://live.ece.utexas.edu/research/LIVE_NFLXStudy/nflx_index.html (accessed on 18 April 2021).
  22. Ghadiyaram, D.; Pan, J.; Bovik, A.C. A Subjective and Objective Study of Stalling Events in Mobile Streaming Videos. IEEE Trans. Circuits Syst. Video Technol. 2019, 29, 183–197. [Google Scholar] [CrossRef]
  23. Ghadiyaram, D.; Pan, J.; Bovik, A. LIVE Mobile Stall Video Database-II, 2017. Available online: http://live.ece.utexas.edu/research/LIVEStallStudy/index.html (accessed on 18 April 2021).
  24. Bampis, C.G.; Li, Z.; Katsavounidis, I.; Huang, T.Y.; Ekanadham, C.; Bovik, A. Towards Perceptually Optimized End-to-end Adaptive Video Streaming. arXiv 2018, arXiv:1808.03898. [Google Scholar]
  25. Sinno, Z.; Bovik, A.C. Large Scale Subjective Video Quality Study. In Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece, 7–10 October 2018. [Google Scholar]
  26. Sinno, Z.; Bovik, A.C. Large-Scale Study of Perceptual Video Quality. IEEE Trans. Image Process. 2019, 28, 612–627. [Google Scholar] [CrossRef] [PubMed]
  27. Sinno, Z.; Bovik, A. LIVE Video Quality Challenge Database, 2018. Available online: http://live.ece.utexas.edu/research/LIVEVQC/index.html (accessed on 18 April 2021).
  28. VQEG Database. 2010. Available online: https://www.its.bldrdoc.gov/vqeg/projects/frtv-phase-i/frtv-phase-i.aspx (accessed on 18 April 2021).
  29. Zhang, F.; Moss, F.M.; Baddeley, R.; Bull, D.R. BVI-HD: A Video Quality Database for HEVC Compressed and Texture Synthesized Content. IEEE Trans. Multimed. 2018, 20, 2620–2630. [Google Scholar] [CrossRef]
  30. Papadopoulos, M.A.; Zhang, F.; Agrafiotis, D.; Bull, D. A video texture database for perceptual compression and quality assessment. In Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada, 27–30 September 2015. [Google Scholar]
  31. Mackin, A.; Zhang, F.; Bull, D.R. A study of subjective video quality at various frame rates. In Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada, 27–30 September 2015. [Google Scholar]
  32. Rao, R.R.R.; Goring, S.; Robitza, W.; Feiten, B.; Raake, A. AVT-VQDB-UHD-1: A Large Scale Video Quality Database for UHD-1. In Proceedings of the 2019 IEEE International Symposium on Multimedia (ISM), San Diego, CA, USA, 9–11 December 2019. [Google Scholar]
  33. Mercat, A.; Viitanen, M.; Vanne, J. UVG dataset. In Proceedings of the 11th ACM Multimedia Systems Conference, Istanbul, Turkey, 28 September–1 October 2020. [Google Scholar]
  34. Song, L.; Tang, X.; Zhang, W.; Yang, X.; Xia, P. The SJTU 4K video sequence dataset. In Proceedings of the 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX), Klagenfurt am Wörthersee, Austria, 3–5 July 2013. [Google Scholar]
  35. IVP Database. 2011. Available online: http://ivp.ee.cuhk.edu.hk/index.shtml (accessed on 18 April 2021).
  36. Péchard, S.; Pépion, R.; Le Callet, P. Suitable Methodology in Subjective Video Quality Assessment: A Resolution Dependent Paradigm. In Proceedings of the International Workshop on Image Media Quality and Its Applications, IMQA2008, Kyoto, Japan, 5–6 September 2008; Volume 6. [Google Scholar]
  37. The Konstanz Natural Video Database. Available online: http://database.mmsp-kn.de/konvid-1k-database.html (accessed on 18 April 2021).
  38. Hosu, V.; Hahn, F.; Jenadeleh, M.; Lin, H.; Men, H.; Sziranyi, T.; Li, S.; Saupe, D. The Konstanz Natural Video Database (KoNViD-1k). In Proceedings of the 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX), Erfurt, Germany, 31 May– 2 June 2017; pp. 1–6. [Google Scholar]
  39. Lin, J.Y.; Song, R.; Wu, C.H.; Liu, T.; Wang, H.; Kuo, C.C.J. MCL-V: A streaming video quality assessment database. J. Vis. Commun. Image Represent. 2015, 30, 1–9. [Google Scholar] [CrossRef]
  40. Wang, H.; Gan, W.; Hu, S.; Lin, J.Y.; Jin, L.; Song, L.; Wang, P.; Katsavounidis, I.; Aaron, A.; Kuo, C.C.J. MCL-JCV: A JND-based H 264/AVC video quality assessment dataset. In Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016. [Google Scholar]
  41. Lee, J.S.; Simone, F.D.; Ramzan, N.; Zhao, Z.; Kurutepe, E.; Sikora, T.; Ostermann, J.; Izquierdo, E.; Ebrahimi, T. Subjective evaluation of scalable video coding for content distribution. In Proceedings of the International Conference on Multimedia—MM ’10, Firenze, Italy, 25–29 October 2010; pp. 65–72. [Google Scholar]
  42. Lee, J.S.; Simone, F.D.; Ebrahimi, T. Subjective Quality Evaluation via Paired Comparison: Application to Scalable Video Coding. IEEE Trans. Multimed. 2011, 13, 882–893. [Google Scholar] [CrossRef]
  43. Paudyal, P.; Battisti, F.; Carli, M. A Study On The Effects of Quality of Service Parameters on Perceived Video Quality. In Proceedings of the 5th European Workshop on Visual Information Processing, EUVIP 2014, Paris, France, 10–12 December 2014. [Google Scholar]
  44. Keimel, C.; Habigt, J.; Habigt, T.; Rothbucher, M.; Diepold, K. Visual quality of current coding technologies at high definition IPTV bitrates. In Proceedings of the 2010 IEEE International Workshop on Multimedia Signal Processing, Saint-Malo, France, 4–6 October 2010. [Google Scholar]
  45. Keimel, C.; Redl, A.; Diepold, K. The TUM high definition video datasets. In Proceedings of the 2012 Fourth International Workshop on Quality of Multimedia Experience, Melbourne, VIC, Australia, 5–7 July 2012. [Google Scholar]
  46. Řeřábek, M.; Ebrahimi, T. Comparison of compression efficiency between HEVC/H.265 and VP9 based on subjective assessments. In Applications of Digital Image Processing XXXVII; Tescher, A.G., Ed.; SPIE: Bellingham, WA, USA, 2014. [Google Scholar]
  47. Ramzan, N.; Pervez, Z.; Amira, A. Quality of experience evaluation of H 265/MPEG-HEVC and VP9 comparison efficiency. In Proceedings of the 2014 26th International Conference on Microelectronics (ICM), Doha, Qatar, 14–17 December 2014. [Google Scholar]
  48. Bienik, J.; Uhrina, M.; Vaculik, M.; Mizdos, T. Perceived Quality of Full HD Video—Subjective Quality Assessment. Adv. Electr. Electron. Eng. 2016, 14. [Google Scholar] [CrossRef]
  49. Xu, J.; Jiang, X. Assessment of subjective video quality on 4K Ultra High Definition videos. In Proceedings of the 2015 8th International Congress on Image and Signal Processing (CISP), Shenyang, China, 14–16 October 2015. [Google Scholar]
  50. Herrou, G.; Hamidouche, W.; Ducloux, X. HDR video quality evaluation of HEVC and VP9 codecs. In Proceedings of the 2016 Picture Coding Symposium (PCS), Nuremberg, Germany, 4–7 December 2016. [Google Scholar]
  51. Dumic, E.; Grgic, S.; Sakic, K.; Frank, D. Subjective quality assessment of H 265 versus H.264 Video Coding for High-Definition Video Systems. In Proceedings of the 2015 13th International Conference on Telecommunications (ConTEL), Graz, Austria, 13–15 July 2015. [Google Scholar]
  52. Milovanovic, D.; Milicevic, Z.; Bojkovic, Z. MPEG video deployment in digital television: HEVC vs AVC codec performance study. In Proceedings of the 2013 11th International Conference on Telecommunications in Modern Satellite, Cable and Broadcasting Services (TELSIKS), Nis, Serbia, 16–19 October 2013. [Google Scholar]
  53. Sotelo, R.; Joskowicz, J.; Anedda, M.; Murroni, M.; Giusto, D.D. Subjective video quality assessments for 4K UHDTV. In Proceedings of the 2017 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), Cagliari, Italy, 7–9 June 2017. [Google Scholar]
  54. Kufa, J.; Polak, L.; Kratochvil, T. HEVC/H 265 vs. VP9 for Full HD and UHD video: Is there any difference in QoE? In Proceedings of the 2016 International Symposium ELMAR, Zadar, Croatia, 12–14 September 2016. [Google Scholar]
  55. Deep, V.; Elarabi, T. HEVC/H 265 vs. VP9 state-of-the-art video coding comparison for HD and UHD applications. In Proceedings of the 2017 IEEE 30th Canadian Conference on Electrical and Computer Engineering (CCECE), Windsor, ON, Canada, 30 April–3 May 2017. [Google Scholar]
  56. Akyazi, P.; Ebrahimi, T. Comparison of Compression Efficiency between HEVC/H 265, VP9 and AV1 based on Subjective Quality Assessments. In Proceedings of the 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX), Cagliari, Italy, 29 May–1 June 2018. [Google Scholar]
  57. Zhu, Y.; Song, L.; Xie, R.; Zhang, W. SJTU 4K video subjective quality dataset for content adaptive bit rate estimation without encoding. In Proceedings of the 2016 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), Nara, Japan, 1–3 June 2016. [Google Scholar]
  58. ITU-R. Recommendation ITU-R BT.500-14—Subjective Video Quality Assessment Methods for Multimedia Applications. 2019. Available online: https://www.itu.int/rec/R-REC-BT.500 (accessed on 18 April 2021).
  59. Mitsu. Mitsu Tool. Available online: http://vq.kt.agh.edu.pl/metrics.html (accessed on 18 April 2021).
  60. Uhrina, M.; Bienik, J.; Mizdos, T. QoE on H 264 and H.265: Crowdsourcing versus Laboratory Testing. In Proceedings of the 2020 30th International Conference Radioelektronika (RADIOELEKTRONIKA), Bratislava, Slovakia, 15–16 April 2020; Volume 30, pp. 1–6. [Google Scholar]
  61. FFmpeg. A Complete, Cross-Platform Solution to Record, Convert and Stream Audio and Video ffmpeg.org. Available online: https://www.ffmpeg.org (accessed on 18 April 2021).
  62. Uhrina, M.; Bienik, J.; Vaculik, M. Coding efficiency of HEVC/H 265 and VP9 compression standards for high resolutions. In Proceedings of the 2016 26th International Conference Radioelektronika (RADIOELEKTRONIKA), Kosice, Slovakia, 19–20 April 2016. [Google Scholar]
  63. ITU-T. Recommendation ITU-T P.910—Subjective Video Quality Assessment Methods for Multimedia Applications. 2008. Available online: https://www.itu.int/rec/T-REC-P.910-200804-I/en (accessed on 18 April 2021).
  64. ITU-T. Recommendation ITU-T P.1401—Methods, Metrics and Procedures for Statistical Evaluation, Qualification and Comparison of Objective Quality Prediction Models. 2020. Available online: https://www.itu.int/rec/T-REC-P.1401-202001-I/en (accessed on 18 April 2021).
  65. Judd, C.M.; McClelland, G.H.; Ryan, C.S. Data Analysis: A Model Comparison Approach to Regression, ANOVA, and Beyond; Routledge: Oxfordshire, UK, 2017. [Google Scholar]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.