Interactive Trimap Generation for Digital Matting Based on Single-Sample Learning
Abstract
:1. Introduction
2. Related Work
3. Problem Setup
4. Proposed Method
4.1. Guidance Branch
4.2. Segmentation Branch
4.3. Generate Branch
5. Dataset and Experiments
5.1. Dataset
5.2. Experiments
6. Conclusions
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
References
- Chuang, Y.Y.; Curless, B.; Salesin, D.H.; Szeliski, R. A bayesian approach to digital matting. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001), Kauai, HI, USA, 8–14 December 2001; Volume 2. [Google Scholar]
- Wang, J.; Cohen, M.F. An iterative optimization approach for unified image segmentation and matting. In Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, Beijing, China, 17–21 October 2005; Volume 2, pp. 936–943. [Google Scholar]
- Levin, A.; Rav-Acha, A.; Lischinski, D. Spectral matting. IEEE Trans. Pattern Anal. Mach. Intell. 2008, 30, 1699–1712. [Google Scholar] [CrossRef]
- Gupta, V.; Raman, S. Automatic trimap generation for image matting. In Proceedings of the 2016 International Conference on Signal and Information Processing (IConSIP), Nanded, India, 6–8 October 2016; pp. 1–5. [Google Scholar]
- Hsieh, C.L.; Lee, M.S. Automatic trimap generation for digital image matting. In Proceedings of the 2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, Kaohsiung, Taiwan, 29 October–1 November 2013; pp. 1–5. [Google Scholar]
- Zheng, Y.; Kambhamettu, C. Learning based digital matting. In Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan, 27 September–4 October 2009; pp. 889–896. [Google Scholar]
- Levin, A.; Lischinski, D.; Weiss, Y. A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 2007, 30, 228–242. [Google Scholar] [CrossRef] [Green Version]
- Rother, C.; Kolmogorov, V.; Blake, A. “GrabCut” interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. TOG 2004, 23, 309–314. [Google Scholar] [CrossRef]
- Li, Y.; Sun, J.; Tang, C.K.; Shum, H.Y. Lazy snapping. ACM Trans. Graph. ToG 2004, 23, 303–308. [Google Scholar] [CrossRef]
- Wang, J.; Agrawala, M.; Cohen, M.F. Soft scissors: An interactive tool for realtime high quality matting. ACM Trans. Graph. 2007, 26, 9-es. [Google Scholar] [CrossRef]
- Cho, D.; Tai, Y.W.; Kweon, I. Natural image matting using deep convolutional neural networks. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11–14 October 2016; pp. 626–643. [Google Scholar]
- Chen, Q.; Li, D.; Tang, C.K. KNN matting. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 2175–2188. [Google Scholar] [CrossRef] [PubMed]
- Xu, N.; Price, B.; Cohen, S.; Huang, T. Deep image matting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 2970–2979. [Google Scholar]
- Rhemann, C.; Rother, C.; Rav-Acha, A.; Sharp, T. High resolution matting via interactive trimap segmentation. In Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA, 23–28 June 2008; pp. 1–8. [Google Scholar]
- Shahrian, E.; Rajan, D.; Price, B.; Cohen, S. Improving image matting using comprehensive sampling sets. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 636–643. [Google Scholar]
- Cho, D.; Kim, S.; Tai, Y.W.; Kweon, I.S. Automatic Trimap Generation and Consistent Matting for Light-Field Images. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 1504–1517. [Google Scholar] [CrossRef]
- Juan, O.; Keriven, R. Trimap segmentation for fast and user-friendly alpha matting. In Proceedings of the International Workshop on Variational, Geometric, and Level Set Methods in Computer Vision, Beijing, China, 16 October 2005; pp. 186–197. [Google Scholar]
- Shen, X.; Tao, X.; Gao, H.; Zhou, C.; Jia, J. Deep automatic portrait matting. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11–14 October 2016; pp. 92–107. [Google Scholar]
- Gastal, E.S.; Oliveira, M.M. Shared sampling for real-time alpha matting. In Computer Graphics Forum; Wiley Online Library: Hoboken, NJ, USA, 2010; Volume 29, pp. 575–584. [Google Scholar]
- Feng, X.; Liang, X.; Zhang, Z. A cluster sampling method for image matting via sparse coding. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11–14 October 2016; pp. 204–219. [Google Scholar]
- Karacan, L.; Erdem, A.; Erdem, E. Image matting with KL-divergence based sparse sampling. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 424–432. [Google Scholar]
- Aksoy, Y.; Ozan Aydin, T.; Pollefeys, M. Designing effective inter-pixel information flow for natural image matting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 29–37. [Google Scholar]
- Zhu, L.; Huang, Z.; Li, Z.; Xie, L.; Shen, H.T. Exploring auxiliary context: Discrete semantic transfer hashing for scalable image retrieval. IEEE Trans. Neural Netw. Learn. Syst. 2018, 29, 5264–5276. [Google Scholar] [CrossRef] [PubMed]
- Fang, Y.; Zhang, H.; Ren, Y. Graph regularised sparse NMF factorisation for imagery de-noising. IET Comput. Vis. 2018, 12, 466–475. [Google Scholar] [CrossRef]
- Zong, J.; Meng, L.; Zhang, H.; Wan, W. JND-based Multiple Description Image Coding. KSII Trans. Internet Inf. Syst. 2017, 11, 3935–3949. [Google Scholar]
- Jiang, Y.; Zheng, Y.; Hou, S.; Chang, Y.; Gee, J. Multimodal image alignment via linear mapping between feature modalities. J. Healthc. Eng. 2017, 2017. [Google Scholar] [CrossRef] [Green Version]
- Zhao, M.; Zhang, H.; Meng, L. An angle structure descriptor for image retrieval. China Commun. 2016, 13, 222–230. [Google Scholar] [CrossRef]
- Zhao, M.; Zhang, H.; Sun, J. A novel image retrieval method based on multi-trend structure descriptor. J. Vis. Commun. Image Represent. 2016, 38, 73–81. [Google Scholar] [CrossRef]
- Liu, H. Generative 3d images in a visual evolutionary computing system. Comput. Sci. Inf. Syst. 2010, 7, 111–126. [Google Scholar] [CrossRef]
- Wang, C.X.; Xu, M.; Wan, W.; Wang, J.; Meng, L.; Li, J.; Sun, J. Robust Image Watermarking via Perceptual Structural Regularity-based JND Model. TIIS 2019, 13, 1080–1099. [Google Scholar]
- Deng, X.; Zheng, Y.; Xu, Y.; Xi, X.; Li, N.; Yin, Y. Graph cut based automatic aorta segmentation with an adaptive smoothness constraint in 3D abdominal CT images. Neurocomputing 2018, 310, 46–58. [Google Scholar] [CrossRef]
- Li, J.B.; Chu, S.C.; Pan, J.S. Kernel Learning Algorithms for Face Recognition; Springer: Berlin, Germany, 2014. [Google Scholar]
- Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 5–9 October 2015; pp. 234–241. [Google Scholar]
- Long, J.; Shelhamer, E.; Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 3431–3440. [Google Scholar]
- He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2961–2969. [Google Scholar]
- Chen, L.C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A.L. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 40, 834–848. [Google Scholar] [CrossRef]
- Khoreva, A.; Benenson, R.; Hosang, J.; Hein, M.; Schiele, B. Simple does it: Weakly supervised instance and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 876–885. [Google Scholar]
- Bearman, A.; Russakovsky, O.; Ferrari, V.; Fei-Fei, L. What’s the point: Semantic segmentation with point supervision. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11–14 October 2016; pp. 549–565. [Google Scholar]
- Shaban, A.; Bansal, S.; Liu, Z.; Essa, I.; Boots, B. One-shot learning for semantic segmentation. arXiv 2017, arXiv:1709.03410. [Google Scholar]
- Rakelly, K.; Shelhamer, E.; Darrell, T.; Efros, A.A.; Levine, S. Few-shot segmentation propagation with guided networks. arXiv 2018, arXiv:1806.07373. [Google Scholar]
- Xu, N.; Price, B.; Cohen, S.; Yang, J.; Huang, T.S. Deep interactive object selection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 373–381. [Google Scholar]
- Everingham, M.; Van Gool, L.; Williams, C.; Winn, J.; Zisserman, A. The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. 2010, 88, 303–338. [Google Scholar] [CrossRef] [Green Version]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
- Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; Li, F.-F. Imagenet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 248–255. [Google Scholar]
- Rhemann, C.; Rother, C.; Wang, J.; Gelautz, M.; Kohli, P.; Rott, P. A perceptually motivated online benchmark for image matting. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 1826–1833. [Google Scholar]
l = 0 | l = 1 | l = 2 | l = 3 |
---|---|---|---|
aeroplance, bicycle | diningtable, car, cat | bus, dog, horse | plant, sheep, sofa |
bird, person | chair, cow | motorbike, bottle | train, tv/monitor |
Image | Proposed | Trimap | Scribble |
---|---|---|---|
GT01 | 28.715 | 8.271 | 156.781 |
GT02 | 25.474 | 7.792 | 118.215 |
GT03 | 24.662 | 35.854 | 139.766 |
GT04 | 67.918 | 51.443 | 334.628 |
GT05 | 22.414 | 5.555 | 127.115 |
GT06 | 29.505 | 10.736 | 171.267 |
GT07 | 26.378 | 12.405 | 136.459 |
GT08 | 43.088 | 34.998 | 220.026 |
GT09 | 33.668 | 19.962 | 181.406 |
GT10 | 26.914 | 10.842 | 150.306 |
GT11 | 30.473 | 14.962 | 158.58 |
GT12 | 20.902 | 9.351 | 99.622 |
GT13 | 54.067 | 25.136 | 311.447 |
GT14 | 33.329 | 10.390 | 159.199 |
GT15 | 27.823 | 11.347 | 155.263 |
GT16 | 37.342 | 22.030 | 204.322 |
GT17 | 24.297 | 11.356 | 125.960 |
GT18 | 32.171 | 10.183 | 162.286 |
GT19 | 22.662 | 4.938 | 121.062 |
GT20 | 25.018 | 13.519 | 132.551 |
GT21 | 30.688 | 16.915 | 170.912 |
GT22 | 26.687 | 12.586 | 124.645 |
GT23 | 25.184 | 12.544 | 114.109 |
GT24 | 28.701 | 12.341 | 136.212 |
GT25 | 33.181 | 11.302 | 191.416 |
GT26 | 49.373 | 25.675 | 230.560 |
GT27 | 77.891 | 56.878 | 450.836 |
Image | Trimap-1 | Trimap-2 |
---|---|---|
GT01 | 0.368 | 0.345 |
GT02 | 0.25 | 0.271 |
GT03 | 0.466 | 0.408 |
GT04 | 0.418 | 0.411 |
GT05 | 0.363 | 0.467 |
GT06 | 0.386 | 0.458 |
GT07 | 0.396 | 0.476 |
GT08 | 0.557 | 0.519 |
GT09 | 0.56 | 0.569 |
GT10 | 0.441 | 0.508 |
GT11 | 0.402 | 0.49 |
GT12 | 0.51 | 0.574 |
GT13 | 0.325 | 0.343 |
GT14 | 0.359 | 0.437 |
GT15 | 0.373 | 0.448 |
GT16 | 0.569 | 0.577 |
GT17 | 0.501 | 0.518 |
GT18 | 0.385 | 0.504 |
GT19 | 0.326 | 0.449 |
GT20 | 0.476 | 0.527 |
GT21 | 0.259 | 0.27 |
GT22 | 0.504 | 0.581 |
GT23 | 0.516 | 0.577 |
GT24 | 0.307 | 0.35 |
GT25 | 0.229 | 0.288 |
GT26 | 0.461 | 0.49 |
GT27 | 0.407 | 0.454 |
Image | Proposed | Trimap | Scribble |
---|---|---|---|
image0 | 44.199 | 8.271 | 156.781 |
image1 | 47.724 | 7.792 | 118.215 |
image2 | 51.2 | 35.854 | 139.766 |
image3 | 46.745 | 51.443 | 334.628 |
image4 | 33.635 | 5.555 | 127.115 |
image5 | 39.439 | 10.736 | 171.267 |
image6 | 40.58 | 12.405 | 136.459 |
image7 | 40.58 | 34.998 | 220.026 |
image8 | 41.584 | 19.962 | 181.406 |
image9 | 27.275 | 10.842 | 150.306 |
image10 | 68.825 | 14.962 | 158.58 |
image11 | 49.825 | 9.351 | 99.622 |
image12 | 60.375 | 25.136 | 311.447 |
image13 | 74.821 | 10.390 | 159.199 |
image14 | 117.501 | 11.347 | 155.263 |
image15 | 22.508 | 22.030 | 204.322 |
image16 | 51.626 | 11.356 | 125.960 |
image17 | 34.524 | 10.183 | 162.286 |
image18 | 23.09 | 4.938 | 121.062 |
image19 | 54.023 | 13.519 | 132.551 |
Image | Trimap-1 | Trimap-2 |
---|---|---|
image0 | 0.3 | 0.22 |
image1 | 0.629 | 0.67 |
image2 | 0.32 | 0.296 |
image3 | 0.42 | 0.318 |
image4 | 0.362 | 0.269 |
image5 | 0.227 | 0.272 |
image6 | 0.529 | 0.527 |
image7 | 0.531 | 0.521 |
image8 | 0.368 | 0.635 |
image9 | 0.441 | 0.538 |
image10 | 0.653 | 0.662 |
image11 | 0.594 | 0.406 |
image12 | 0.626 | 0.533 |
image13 | 0.355 | 0.352 |
image14 | 0.435 | 0.556 |
image15 | 0.393 | 0.414 |
image16 | 0.676 | 0.442 |
image17 | 0.45 | 0.322 |
image18 | 0.43 | 0.28 |
image19 | 0.42 | 0.396 |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chen, Z.; Zheng, Y.; Li, X.; Luo, R.; Jia, W.; Lian, J.; Li, C. Interactive Trimap Generation for Digital Matting Based on Single-Sample Learning. Electronics 2020, 9, 659. https://doi.org/10.3390/electronics9040659
Chen Z, Zheng Y, Li X, Luo R, Jia W, Lian J, Li C. Interactive Trimap Generation for Digital Matting Based on Single-Sample Learning. Electronics. 2020; 9(4):659. https://doi.org/10.3390/electronics9040659
Chicago/Turabian StyleChen, Zhenpeng, Yuanjie Zheng, Xiaojie Li, Rong Luo, Weikuan Jia, Jian Lian, and Chengjiang Li. 2020. "Interactive Trimap Generation for Digital Matting Based on Single-Sample Learning" Electronics 9, no. 4: 659. https://doi.org/10.3390/electronics9040659
APA StyleChen, Z., Zheng, Y., Li, X., Luo, R., Jia, W., Lian, J., & Li, C. (2020). Interactive Trimap Generation for Digital Matting Based on Single-Sample Learning. Electronics, 9(4), 659. https://doi.org/10.3390/electronics9040659