Improved Combined Metric for Automatic Quality Assessment of Stitched Images
Abstract
:1. Introduction
2. The Quality Evaluation of Stitched Images
2.1. The Verification Methodology of the Developed Metrics
2.2. The Overview of Quality Metrics
3. The Proposed Approach
- —differential entropy;
- —average local entropy for the stitched image;
- —differential variance of the local entropy;
- —variance of the edge-based structural index (var());
- —variance of the luminance and contrast index (var(K));
- —absolute difference of standard deviations.
4. The Experimental Results
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
AMD | Advanced Micro Devices |
AVC | Advanced Video Coding |
BRIEF | Binary Robust Independent Elementary Features |
BRISK | Binary Robust Invariant Scalable Keypoints |
CCSID | Color Correction-based Stitched Image Database |
CNN | Convolutional Neural Network |
CVIQD | Compressed VR Image Quality Database |
DoG | Difference of Gaussians |
FAST | Features from Accelerated Segment Test |
FREAK | Fast Retina Keypoint |
FR IQA | Full-Reference Image Quality Assessment |
FSIM | Feature Similarity |
GGD | Generalized Gaussian Distribution |
GMM | Gaussian Mixture Model |
HEVC | High-Efficiency Video Coding |
HMD | Head-Mounted Displays |
iCID | improved Color Image Difference |
ISIQA | Indian Institute of Science Stitched Image Quality Assessment (dataset) |
JPEG | Joint Photographic Experts Group |
KROCC | Kendall Rank Order Correlation Coefficient |
MIQM | Multi-view Image Quality Measure |
MOS | Mean Opinion Scores |
MVAQD | Multi-Distortion Visual Attention Quality Dataset |
NR IQA | No-Reference Image Quality Assessment |
NRQQA | No-Reference Quantitative Quality Assessment |
OIQA | Omnidirectional Image Quality Assessment |
OR | Outlier Ratio |
ORB | Oriented FAST and Rotated BRIEF |
PLCC | Pearson’s Linear Correlation Coefficient |
RAM | Random Access Memory |
RANSAC | RANdom SAmpling Consensus |
RMSE | Root Mean Squared Error |
ROI | region of interest |
SIFT | Scale Invariant Feature Transform |
SIQE | Stitched Image Quality Evaluation |
SROCC | Spearman Rank Order Correlation Coefficient |
SSIM | Structural Similarity |
SURF | Speeded-Up Robust Features |
SVM | Support Vector Machine |
SVR | Support Vector Regression |
UAV | unmanned aerial vehicle |
VQEG | Video Quality Experts Group |
VSLAM | Visual Simultaneous Localization and Mapping |
VR | Virtual Reality |
References
- Liu, L.; Guo, L.; Dong, N.; Tian, W.; Li, C.; Zhang, F. The Research and Application of Image Stitching in the Robot Target Recognition. In Lecture Notes in Electrical Engineering; Springer: Berlin/Heidelberg, Germany, 2012; pp. 105–111. [Google Scholar] [CrossRef]
- Zhou, L.; Tian, Y.; Lu, G.; Wu, X.; Zhang, Q. Linear Protection Grid Optimized Image Stitching for Mobile Robots. In Proceedings of the 2019 IEEE International Conference on Real-time Computing and Robotics (RCAR), Irkutsk, Russia, 4–9 August 2019. [Google Scholar] [CrossRef]
- Ulrich, M.; Forstner, A.; Reinhart, G. High-accuracy 3D image stitching for robot-based inspection systems. In Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada, 27–30 September 2015. [Google Scholar] [CrossRef]
- Schlagenhauf, T.; Brander, T.; Fleischer, J. A stitching algorithm for automated surface inspection of rotationally symmetric components. CIRP J. Manuf. Sci. Technol. 2021, 35, 169–177. [Google Scholar] [CrossRef]
- Xie, R.; Yao, J.; Liu, K.; Lu, X.; Liu, Y.; Xia, M.; Zeng, Q. Automatic multi-image stitching for concrete bridge inspection by combining point and line features. Autom. Constr. 2018, 90, 265–280. [Google Scholar] [CrossRef]
- Samsudin, S.; Adwan, S.; Arof, H.; Mokhtar, N.; Ibrahim, F. Development of Automated Image Stitching System for Radiographic Images. J. Digit. Imaging 2012, 26, 361–370. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Lowe, D. Object recognition from local scale-invariant features. In Proceedings of the Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece, 20–27 September 1999. [Google Scholar] [CrossRef]
- Bay, H.; Tuytelaars, T.; Gool, L.V. SURF: Speeded Up Robust Features. In Computer Vision—ECCV 2006; Springer: Berlin/Heidelberg, Germany, 2006; pp. 404–417. [Google Scholar] [CrossRef]
- Rosten, E.; Drummond, T. Fusing points and lines for high performance tracking. In Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV’05), Beijing, China, 17–21 October 2005; Volume 1. [Google Scholar] [CrossRef]
- Calonder, M.; Lepetit, V.; Strecha, C.; Fua, P. BRIEF: Binary Robust Independent Elementary Features. In Computer Vision—ECCV 2010; Springer: Berlin/Heidelberg, Germany, 2010; pp. 778–792. [Google Scholar] [CrossRef]
- Rublee, E.; Rabaud, V.; Konolige, K.; Bradski, G. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011. [Google Scholar] [CrossRef]
- Leutenegger, S.; Chli, M.; Siegwart, R.Y. BRISK: Binary Robust invariant scalable keypoints. In Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011. [Google Scholar] [CrossRef] [Green Version]
- Alahi, A.; Ortiz, R.; Vandergheynst, P. FREAK: Fast Retina Keypoint. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, 16–21 June 2012. [Google Scholar] [CrossRef] [Green Version]
- Huang, W.; Han, X. An Improved RANSAC Algorithm of Color Image Stitching. In Lecture Notes in Electrical Engineering; Springer: Berlin/Heidelberg, Germany, 2013; pp. 21–28. [Google Scholar] [CrossRef]
- Guo, L.S.; Dong, N.Q.; Tian, W.; Li, C.X.; Zhang, F.Z. The Application of Image Stitching in the Robot Target Recognition. Adv. Mater. Res. 2011, 327, 149–152. [Google Scholar] [CrossRef]
- Dong, Y.; Pei, M.; Zhang, L.; Xu, B.; Wu, Y.; Jia, Y. Stitching Videos from a Fisheye Lens Camera and a Wide-Angle Lens Camera for Telepresence Robots. Int. J. Soc. Robot. 2021, 14, 733–745. [Google Scholar] [CrossRef]
- Lee, W.T.; Chen, H.I.; Chen, M.S.; Shen, I.C.; Chen, B.Y. High-resolution 360 Video Foveated Stitching for Real-time VR. Comput. Graph. Forum 2017, 36, 115–123. [Google Scholar] [CrossRef]
- Limonov, A.; Yu, X.; Juan, L.; Lei, C.; Jian, Y. Stereoscopic realtime 360-degree video stitching. In Proceedings of the 2018 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA, 12–14 January 2018. [Google Scholar] [CrossRef]
- Meng, X.; Wang, W.; Leong, B. SkyStitch: A Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching. In Proceedings of the 23rd ACM International Conference on Multimedia, Brisbane, Australia, 26–30 October 2015. [Google Scholar] [CrossRef]
- Madhusudana, P.C.; Soundararajan, R. Subjective and Objective Quality Assessment of Stitched Images for Virtual Reality. IEEE Trans. Image Process. 2019, 28, 5620–5635. [Google Scholar] [CrossRef]
- Hou, J.; Lin, W.; Zhao, B. Content-Dependency Reduction With Multi-Task Learning In Blind Stitched Panoramic Image Quality Assessment. In Proceedings of the 2020 IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates, 25–28 October 2020. [Google Scholar] [CrossRef]
- Duan, H.; Zhai, G.; Min, X.; Zhu, Y.; Fang, Y.; Yang, X. Perceptual Quality Assessment of Omnidirectional Images. In Proceedings of the 2018 IEEE International Symposium on Circuits and Systems (ISCAS), Florence, Italy, 27–30 May 2018. [Google Scholar] [CrossRef]
- Wang, X.; Chai, X.; Shao, F. Quality assessment for color correction-based stitched images via bi-directional matching. J. Vis. Commun. Image Represent. 2021, 75, 103051. [Google Scholar] [CrossRef]
- Cheung, G.; Yang, L.; Tan, Z.; Huang, Z. A Content-Aware Metric for Stitched Panoramic Image Quality Assessment. In Proceedings of the 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy, 22–29 October 2017. [Google Scholar] [CrossRef]
- Sun, W.; Gu, K.; Ma, S.; Zhu, W.; Liu, N.; Zhai, G. A Large-Scale Compressed 360-Degree Spherical Image Database: From Subjective Quality Evaluation to Objective Model Comparison. In Proceedings of the 2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP), Vancouver, BC, Canada, 29–31 August 2018. [Google Scholar] [CrossRef]
- Zheng, X.; Jiang, G.; Yu, M.; Jiang, H. Segmented Spherical Projection-Based Blind Omnidirectional Image Quality Assessment. IEEE Access 2020, 8, 31647–31659. [Google Scholar] [CrossRef]
- VQEG. Final Report from the Video Quality Experts Group on the Validation of Objective Models of Video Quality Assessment Phase II; Technical Report; Video Quality Experts Group, 2003; Available online: https://vqeg.org/VQEGSharedFiles/Publications/Validation_Tests/FRTV_Phase2/FRTV_Phase2_Final_Report.pdf (accessed on 10 July 2022).
- Sheikh, H.; Sabir, M.; Bovik, A. A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms. IEEE Trans. Image Process. 2006, 15, 3440–3451. [Google Scholar] [CrossRef]
- Solh, M.; AlRegib, G. MIQM: A novel Multi-view Images Quality Measure. In Proceedings of the 2009 International Workshop on Quality of Multimedia Experience, San Diego, CA, USA, 29–31 July 2009. [Google Scholar] [CrossRef]
- Solh, M.; AlRegib, G. MIQM: A Multicamera Image Quality Measure. IEEE Trans. Image Process. 2012, 21, 3902–3914. [Google Scholar] [CrossRef] [PubMed]
- Okarma, K.; Chlewicki, W.; Kopytek, M.; Marciniak, B.; Lukin, V. Entropy-Based Combined Metric for Automatic Objective Quality Assessment of Stitched Panoramic Images. Entropy 2021, 23, 1525. [Google Scholar] [CrossRef] [PubMed]
- Qureshi, H.; Khan, M.; Hafiz, R.; Cho, Y.; Cha, J. Quantitative quality assessment of stitched panoramic images. IET Image Process. 2012, 6, 1348–1358. [Google Scholar] [CrossRef]
- Wang, Z.; Bovik, A.; Sheikh, H.; Simoncelli, E. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Bellavia, F.; Colombo, C. Dissecting and Reassembling Color Correction Algorithms for Image Stitching. IEEE Trans. Image Process. 2018, 27, 735–748. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhang, L.; Zhang, L.; Mou, X.; Zhang, D. FSIM: A Feature Similarity Index for Image Quality Assessment. IEEE Trans. Image Process. 2011, 20, 2378–2386. [Google Scholar] [CrossRef] [Green Version]
- Preiss, J.; Fernandes, F.; Urban, P. Color-Image Quality Assessment: From Prediction to Optimization. IEEE Trans. Image Process. 2014, 23, 1366–1378. [Google Scholar] [CrossRef]
- Xu, W.; Mulligan, J. Performance evaluation of color correction approaches for automatic multi-view image and video stitching. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, CA, USA, 13–18 June 2010. [Google Scholar] [CrossRef]
- Yu, S.; Li, T.; Xu, X.; Tao, H.; Yu, L.; Wang, Y. NRQQA: A No-Reference Quantitative Quality Assessment Method for Stitched Images. In Proceedings of the ACM Multimedia Asia, Beijing, China, 15–18 December 2019; pp. 1–6, Article No. 14. [Google Scholar] [CrossRef]
- Okarma, K. Combined image similarity index. Optical Rev. 2012, 19, 349–354. [Google Scholar] [CrossRef]
- Okarma, K.; Lech, P.; Lukin, V.V. Combined Full-Reference Image Quality Metrics for Objective Assessment of Multiply Distorted Images. Electronics 2021, 10, 2256. [Google Scholar] [CrossRef]
- Oszust, M. Decision Fusion for Image Quality Assessment using an Optimization Approach. IEEE Signal Process. Lett. 2016, 23, 65–69. [Google Scholar] [CrossRef]
- Ullah, H.; Afzal, S.; Khan, I.U. Perceptual Quality Assessment of Panoramic Stitched Contents for Immersive Applications: A Prospective Survey. Virtual Real. Intell. Hardw. 2022, 4, 223–246. [Google Scholar] [CrossRef]
- Okarma, K.; Kopytek, M. Application of Image Entropy Analysis for the Quality Assessment of Stitched Images. In Progress in Image Processing, Pattern Recognition and Communication Systems; Choraś, M., Choraś, R.S., Kurzyński, M., Trajdos, P., Pejaś, J., Hyla, T., Eds.; Springer International Publishing: Cham, Switzerland, 2022; pp. 125–132. [Google Scholar] [CrossRef]
- Liu, Z.; Mo, Z. Combining Local and Global Features for Quality Assessment of Stitched Images in Virtual Reality. In Proceedings of the 2021 9th International Conference on Information Technology: IoT and Smart City, Guangzhou, China, 22–25 December 2021. [Google Scholar] [CrossRef]
- Mittal, A.; Moorthy, A.K.; Bovik, A.C. No-Reference Image Quality Assessment in the Spatial Domain. IEEE Trans. Image Process. 2012, 21, 4695–4708. [Google Scholar] [CrossRef] [PubMed]
Metric | Correlation with MOS | ||
---|---|---|---|
PLCC | SROCC | KROCC | |
Results provided by other researchers for 80% training and 20% testing sets | |||
SIQE reported in [20] | 0.8395 | 0.8318 | - |
SIQE (median) reported in [44] | 0.8030 | 0.7820 | - |
Liu and Mo [44] | 0.8432 | 0.8013 | - |
Results obtained for the whole ISIQA database | |||
SIQE [20] | 0.7488 | 0.7057 | 0.5308 |
EntSIQE [43] | 0.8012 | 0.7920 | 0.5971 |
EntSIQE [43] | 0.8101 | 0.7945 | 0.5990 |
EntSIQE [31] | 0.8338 | 0.8338 | 0.6418 |
EntSIQE [31] | 0.8337 | 0.8341 | 0.6432 |
CombSIQE (proposed) | 0.8684 | 0.8665 | 0.6810 |
Metric | Correlation with MOS | ||
---|---|---|---|
PLCC | SROCC | KROCC | |
CombSIQE (proposed) | 0.8684 | 0.8665 | 0.6810 |
only weighted sum | 0.8387 | 0.8363 | 0.6439 |
only weighted product | 0.8362 | 0.8338 | 0.6431 |
without (SIQE) | 0.3963 | 0.4177 | 0.2862 |
without | 0.8671 | 0.8672 | 0.6828 |
without | 0.8390 | 0.8350 | 0.6455 |
without | 0.8668 | 0.8623 | 0.6773 |
without | 0.8611 | 0.8591 | 0.6730 |
without | 0.8655 | 0.8632 | 0.6773 |
without | 0.8611 | 0.8590 | 0.6721 |
without | 0.8523 | 0.8519 | 0.6610 |
without | 0.8676 | 0.8681 | 0.6837 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Okarma, K.; Kopytek, M. Improved Combined Metric for Automatic Quality Assessment of Stitched Images. Appl. Sci. 2022, 12, 10284. https://doi.org/10.3390/app122010284
Okarma K, Kopytek M. Improved Combined Metric for Automatic Quality Assessment of Stitched Images. Applied Sciences. 2022; 12(20):10284. https://doi.org/10.3390/app122010284
Chicago/Turabian StyleOkarma, Krzysztof, and Mateusz Kopytek. 2022. "Improved Combined Metric for Automatic Quality Assessment of Stitched Images" Applied Sciences 12, no. 20: 10284. https://doi.org/10.3390/app122010284
APA StyleOkarma, K., & Kopytek, M. (2022). Improved Combined Metric for Automatic Quality Assessment of Stitched Images. Applied Sciences, 12(20), 10284. https://doi.org/10.3390/app122010284