Multi-Label Classification Based on Low Rank Representation for Image Annotation
Abstract
:1. Introduction
- We apply graph-based multi-label classification to annotate remote sensing images associated with multiple concepts (labels).
- We exploit LRR for graph construction in the feature space and label space of images, respectively.
- The semantic graph constructed in the label space can effectively capture global label correlation and improve the accuracy of image annotation.
- The proposed MLC-LRR can take advantage of limited labeled images and abundant unlabeled images and shows improved performance compared to other related methods on annotating images.
2. Related Work
3. Methodology
3.1. Low Rank Representation for Feature-Based Graph Construction
3.2. Low Rank Representation for Semantic Graph Construction
3.3. Graph-Based Multi-Label Classification
4. Results and Discussion
4.1. Evaluation Metrics
4.2. Experimental Results on Annotating Remote Sensing Images
4.3. The Benefit of the Semantic Graph
4.4. Experimental Results on Other Multi-Label Image Datasets
4.5. Discussion
4.5.1. Results Analysis on Land Cover Images
4.5.2. Results Analysis on Other Multi-Label Images
4.6. Toy Examples
5. Conclusions
Acknowledgments
Author Contributions
Conflicts of Interest
References
- Chen, K.; Jian, P.; Zhou, Z.; Zhang, D. Semantic annotation of high-resolution remote sensing images via Gaussian process multi-instance multilabel learning. IEEE Geosci. Remote Sens. Lett. 2013, 10, 1285–1289. [Google Scholar] [CrossRef]
- Lu, D.; Weng, Q. A survey of image classification methods and techniques for improving classification performance. Int. J. Remote Sens. 2007, 28, 823–870. [Google Scholar] [CrossRef]
- Xia, G.S.; Wang, Z.; Xiong, C.; Zhang, L. Accurate annotation of remote sensing images via active spectral clustering with little expert knowledge. Remote Sens. 2015, 7, 15014–15045. [Google Scholar] [CrossRef]
- Huang, L.; Chen, C.; Li, W.; Du, Q. Remote sensing image scene classification using multi-scale completed local binary patterns and Fisher vectors. Remote Sens. 2016, 8, 483. [Google Scholar] [CrossRef]
- Ko, C.; Sohn, G.; Remmel, T.K.; Miller, J.R. Maximizing the diversity of ensemble random forests for tree genera classification using high density LiDAR Data. Remote Sens. 2016, 8, 646. [Google Scholar] [CrossRef]
- Keshava, N. A survey of spectral unmixing algorithms. Linc. Lab. J. 2003, 14, 55–78. [Google Scholar]
- Tsoumakas, G.; Katakis, I.; Vlahavas, I. Mining multi-label data. In Data Mining and Knowledge Discovery Handbook; Springer: New York, NY, USA, 2009; pp. 667–685. [Google Scholar]
- Zhang, M.L.; Zhou, Z.H. A review on multi-label learning algorithms. IEEE Trans. Knowl. Data Eng. 2014, 26, 1819–1837. [Google Scholar] [CrossRef]
- Zhou, T.H.; Wang, L.; Ryu, K.H. Supporting keyword search for image retrieval with integration of probabilistic annotation. Sustainability 2015, 7, 6303–6320. [Google Scholar] [CrossRef]
- Karalas, K.; Tsagkatakis, G.; Zervakis, M.; Tsakalides, P. Deep learning for multi-label land cover classification. Proc. SPIE 2015, 96430. [Google Scholar] [CrossRef]
- Karalas, K.; Tsagkatakis, G.; Zervakis, M.; Tsakalides, P. Land classification using remotely sensed data: Going multilabel. IEEE Trans. Geosci. Remote Sens. 2016, 54, 3548–3563. [Google Scholar] [CrossRef]
- Santos, A.; Canuto, A.; Neto, A.F. A comparative analysis of classification methods to multi-label tasks in different application domains. Int. J. Comput. Inf. Syst. Ind. Manag. Appl. 2011, 3, 218–227. [Google Scholar]
- Boutell, M.R.; Luo, J.; Shen, X.; Brown, C.M. Learning multi-label scene classification. Pattern Recognit. 2004, 37, 1757–1771. [Google Scholar] [CrossRef]
- Ghamrawi, N.; McCallum, A. Collective multi-label classification. In Proceedings of the 14th ACM International Conference on Information and Knowledge Management, ACM, Bremen, Germany, 31 October–5 November 2005; pp. 195–200.
- Read, J.; Pfahringer, B.; Holmes, G.; Frank, E. Classifier chains for multi-label classification. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Bled, Slovenia, 7–11 September 2009; pp. 254–269.
- Camps-Valls, G.; Muñoz-Marí, J.; Gómez-Chova, L.; Richter, K.; Calpe-Maravilla, J. Biophysical parameter estimation with a semisupervised support vector machine. IEEE Geosci. Remote Sens. Lett. 2009, 6, 248–252. [Google Scholar] [CrossRef]
- Tuia, D.; Verrelst, J.; Alonso, L.; Pérez-Cruz, F.; Camps-Valls, G. Multioutput support vector regression for remote sensing biophysical parameter estimation. IEEE Geosci. Remote Sens. Lett. 2011, 8, 804–808. [Google Scholar] [CrossRef]
- Chen, G.; Song, Y.; Wang, F.; Zhang, C. Semi-supervised Multi-label Learning by Solving a Sylvester Equation. In Proceedings of the 8th SIAM International Conference on Data Mining, Atlanta, GA, USA, 24–26 April 2008; pp. 410–419.
- Kong, X.; Ng, M.K.; Zhou, Z.H. Transductive multilabel learning via label set propagation. IEEE Trans. Knowl. Data Eng. 2013, 99, 704–719. [Google Scholar] [CrossRef]
- Wang, C.; Yan, S.; Zhang, L.; Zhang, H.J. Multi-label sparse coding for automatic image annotation. In Proceedings of the 20th IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 1643–1650.
- Erkan, A.N.; Camps-Valls, G.; Altun, Y. Semi-supervised remote sensing image classification via maximum entropy. In Proceedings of the IEEE International Workshop on Machine Learning for Signal Processing, Kittila, Finland, 29 August–1 September 2010; pp. 313–318.
- Li, J.; Bioucas-Dias, J.M.; Plaza, A. Semisupervised hyperspectral image classification using soft sparse multinomial logistic regression. IEEE Geosci. Remote Sens. Lett. 2013, 10, 318–322. [Google Scholar]
- Uhlmann, S.; Kiranyaz, S.; Gabbouj, M. Semi-supervised learning for ill-posed polarimetric SAR classification. Remote Sens. 2014, 6, 4801–4830. [Google Scholar] [CrossRef]
- Tang, J.; Hong, R.; Yan, S.; Chua, T.S.; Qi, G.J.; Jain, R. Image annotation by k nn-sparse graph-based label propagation over noisily tagged web images. ACM Trans. Intell. Syst. Technol. 2011, 2, 135–136. [Google Scholar] [CrossRef]
- Jing, L.; Yang, L.; Yu, J.; Ng, M.K. Semi-supervised low-rank mapping learning for multi-label classification. In Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 8–10 June 2015; pp. 1483–1491.
- Hu, D.Y.; Reichel, L. Krylov-subspace methods for the Sylvester equation. J. Linear Algebra Its Appl. 1992, 172, 283–313. [Google Scholar] [CrossRef]
- Yu, G.; Rangwala, H.; Domeniconi, C.; Zhang, G.; Yu, Z. Protein function prediction using multilabel ensemble classification. IEEE/ACM Trans. Comput. Biol. Bioinf. 2013, 10, 1045–1057. [Google Scholar] [CrossRef] [PubMed]
- Tong, H.; Faloutsos, C.; Pan, J.Y. Fast random walk with restart and its applications. In Proceedings of the 6th IEEE International Conference on Data Mining, Hong Kong, China, 18–22 December 2006; pp. 613–622.
- Wang, H.; Huang, H.; Ding, C. Function-function correlated multi-label protein function prediction over interaction networks. J. Comput. Biol. 2013, 20, 322–343. [Google Scholar] [CrossRef] [PubMed]
- Ding, C.; Simon, H.D.; Jin, R.; Li, T. A learning framework using Green’s function and kernel regularization with application to recommender system. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Jose, CA, USA, 12–15 August 2007; pp. 260–269.
- Wang, B.; Tu, Z.; Tsotsos, J.K. Dynamic label propagation for semi-supervised multi-class multi-label classification. In Proceedings of the 15th IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December 2013; pp. 425–432.
- Lu, X.; Wang, Y.; Yuan, Y. Graph-regularized low-rank representation for destriping of hyperspectral images. IEEE Trans. Geosci. Remote Sens. 2013, 51, 4009–4018. [Google Scholar] [CrossRef]
- Shi, Q.; Du, B.; Zhang, L. Domain adaptation for remote sensing image classification: A low-rank reconstruction and instance weighting label propagation inspired algorithm. IEEE Trans. Geosci. Remote Sens. 2015, 53, 1–13. [Google Scholar]
- Liu, G.; Lin, Z.; Yan, S.; Sun, J.; Yu, Y.; Ma, Y. Robust recovery of subspace structures by low-rank representation. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 171–184. [Google Scholar] [CrossRef] [PubMed]
- Belkin, M.; Niyogi, P.; Sindhwani, V. Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples. J. Mach. Learn. Res. 2006, 7, 2399–2434. [Google Scholar]
- Zhu, X. Semi-Supervised Learning Literature Survey; Technical Report 1530, Computer Sciences; University of Wisconsin-Madison: Madison, WI, USA, 2008. [Google Scholar]
- Yang, S.; Wang, X.; Wang, M.; Han, Y.; Jiao, L. Semi-supervised low-rank representation graph for pattern recognition. IET Image Process. 2013, 7, 131–136. [Google Scholar] [CrossRef]
- Liang, H.; Li, Q. Hyperspectral imagery classification using sparse representations of convolutional neural network features. Remote Sens. 2016, 8, 99. [Google Scholar] [CrossRef]
- He, R.; Zheng, W.S.; Hu, B.G.; Kong, X.W. Nonnegative Sparse Coding for Discriminative Semi-supervised Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, USA, 20–25 June 2011; pp. 2849–2856.
- Zhuang, L.; Gao, H.; Lin, Z.; Ma, Y.; Zhang, X.; Yu, N. Non-negative low rank and sparse graph for semi-supervised learning. In Proceedings of the 23th IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, 16–21 June 2012; pp. 2328–2335.
- Fazel, M. Matrix Rank Minimization with Applications. Ph.D. Thesis, Stanford University, Stanford, CA, USA, 2002. [Google Scholar]
- Vandenberghe, L.; Boyd, S. Semidefinite programming. SIAM Rev. 1996, 38, 49–95. [Google Scholar] [CrossRef]
- Cai, J.F.; Candès, E.J.; Shen, Z. A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 2008, 20, 1956–1982. [Google Scholar] [CrossRef]
- Zhang, H.; Lin, Z.; Zhang, C. A counterexample for the validity of using nuclear norm as a convex surrogate of rank. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Prague, Czech Republic, 23–27 September 2013; pp. 226–241.
- Boyd, S.; Parikh, N.; Chu, E.; Peleato, B.; Eckstein, J. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 2011, 3, 1–122. [Google Scholar] [CrossRef]
- Lin, Z.; Liu, R.; Su, Z. Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation. In Proceedings of the Advances in Neural Information Processing Systems Conference, Granada, Spain, 12–14 December 2011; pp. 612–620.
- Larsen, R.M. Lanczos Bidiagonalization With Partial Reorthogonalization; Technical report, DAIMI PB-357; Department of Computer Science, Aarhus University: Aarhus, Denmark, 1998. [Google Scholar]
- Zheng, Y.; Zhang, X.; Yang, S.; Jiao, L. Low-rank representation with local constraint for graph construction. Neurocomputing 2013, 122, 398–405. [Google Scholar] [CrossRef]
- Read, J.; Pfahringer, B.; Holmes, G.; Frank, E. Classifier chains for multi-label classification. Mach. Learn. 2011, 85, 333–359. [Google Scholar] [CrossRef]
- Tsoumakas, G.; Vlahavas, I. Random k-labelsets: An ensemble method for multilabel classification. In Proceedings of the European Conference on Machine Learning, Warsaw, Poland, 17–21 September 2007; Springer: Berlin, Germany, 2007; pp. 406–417. [Google Scholar]
- Chung, F.R. Spectral Graph Theory; Published for the Conference Board of the Mathematical Sciences by the American Mathematical Society; AMS: Providence, RI, USA, 1997. [Google Scholar]
- Wang, F.; Zhang, C. Label propagation through linear neighborhoods. IEEE Trans. Knowl. Data Eng. 2007, 20, 55–67. [Google Scholar] [CrossRef]
- Goncalves, E.C.; Plastino, A.; Freitas, A.A. A genetic algorithm for optimizing the label ordering in multi-label classifier chains. In Proceedings of the IEEE 25th International Conference on Tools with Artificial Intelligence, Washington, DC, USA, 4–6 November 2013; pp. 469–476.
- Duygulu, P.; Barnard, K.; de Freitas, J.F.; Forsyth, D.A. Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In Proceedings of the 7th European Conference on Computer Vision, Copenhagen, Denmark, 28–31 May 2002; pp. 97–112.
- Huiskes, M.J.; Lew, M.S. The MIR flickr retrieval evaluation. In Proceedings of the 1th ACM International Conference on Multimedia Information Retrieval, Vancouver, BC, Canada, 30–31 October 2008; pp. 39–43.
- Von Ahn, L.; Dabbish, L. Labeling images with a computer game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vienna, Austria, 24–29 April 2004; pp. 319–326.
- Bucak, S.S.; Jin, R.; Jain, A.K. Multi-label learning with incomplete class assignments. In Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, USA, 20–25 June 2011; pp. 2801–2808.
- Wang, J.G.; Li, J.; Yau, W.Y.; Sung, E. Boosting dense SIFT descriptors and shape contexts of face images for gender recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA, 13–18 June 2010; pp. 96–102.
- Grubinger, M.; Clough, P.; Müller, H.; Deselaers, T. The IAPR tc-12 Benchmark: A New Evaluation Resource for Visual Information Systems. In Proceedings of the International Workshop OntoImage’2006 Language Resources for Content-Based Image Retrieval, Genoa, Italy, 22 May 2006; pp. 10–20.
- De Meo, P.; Quattrone, G.; Ursino, D. Exploitation of semantic relationships and hierarchical data structures to support a user in his annotation and browsing activities in folksonomies. Inf. Syst. 2009, 34, 511–535. [Google Scholar] [CrossRef]
- Cai, Y.; Li, Q.; Xie, H.; Min, H. Exploring personalized searches using tag-based user profiles and resource profiles in folksonomy. Neural Netw. 2014, 58, 98–110. [Google Scholar] [CrossRef] [PubMed]
- De Meo, P.; Quattrone, G.; Ursino, D. A query expansion and user profile enrichment approach to improve the performance of recommender systems operating on a folksonomy. User Model. User Adapt. Interact. 2010, 20, 41–86. [Google Scholar] [CrossRef]
- Kim, H.N.; Rawashdeh, M.; Alghamdi, A.; El Saddik, A. Folksonomy-based personalized search and ranking in social media services. Inf. Syst. 2012, 37, 61–76. [Google Scholar] [CrossRef]
- Kim, H.N.; Roczniak, A.; Lévy, P.; El Saddik, A. Social media filtering based on collaborative tagging in semantic space. Multimed. Tools Appl. 2012, 56, 63–89. [Google Scholar] [CrossRef]
No. | CLC Code | Description |
---|---|---|
1 | 111 | Continuous urban fabric |
2 | 121 | Industrial or commercial units |
3 | 122 | Road and rail networks and assoc.land |
4 | 124 | Airports |
5 | 131 | Mineral extraction sites |
6 | 132 | Dump sites |
7 | 133 | Construction sites |
8 | 141 | Green urban areas |
9 | 142 | Sport and leisure facilities |
10 | 212 | Permanently irrigated land |
11 | 213 | Rice fields |
12 | 223 | Olive groves |
13 | 241 | Annual crops assoc. with perm.crops |
14 | 322 | Moors and heathland |
15 | 331 | Beaches, dunes, sands |
16 | 332 | Bare rocks |
17 | 411 | Inland marshes |
18 | 412 | Peat bogs |
19 | 421 | Salt marshes |
20 | 521 | Coastal lagoons |
Datasets | N | D | C | Avg |
---|---|---|---|---|
Land Cover | 12,291 | 57 | 20 | 2.037 |
Flags | 194 | 19 | 7 | 3.392 |
Scene | 2407 | 294 | 6 | 2.158 |
Core15k | 4395 | 1000 | 260 | 3.61 |
MIRFlickr | 4951 | 1000 | 457 | 7.31 |
ESPGame | 10,457 | 1000 | 268 | 6.41 |
MicroAvg | 1-RankLoss | AvgPrec | AUC | Coverage ↓ | |
---|---|---|---|---|---|
10% | |||||
MSC | |||||
DLP | |||||
Tram | |||||
TMC | |||||
FCML | |||||
MLC-LRR | |||||
15% | |||||
MSC | |||||
DLP | |||||
Tram | |||||
TMC | |||||
FCML | |||||
MLC-LRR |
MicroAvg | 1-RankLoss | AvgPrec | AUC | Coverage ↓ | |
---|---|---|---|---|---|
Flags | |||||
MSC | |||||
DLP | |||||
Tram | |||||
TMC | |||||
FCML | |||||
MLC-LRR | |||||
Scene | |||||
MSC | |||||
DLP | |||||
Tram | |||||
TMC | |||||
FCML | |||||
MLC-LRR | |||||
Core15k | |||||
MSC | |||||
DLP | |||||
Tram | |||||
TMC | |||||
FCML | |||||
MLC-LRR | |||||
MIR Flickr | |||||
MSC | |||||
DLP | |||||
Tram | |||||
TMC | |||||
FCML | |||||
MLC-LRR | |||||
ESPGame | |||||
MSC | |||||
DLP | |||||
Tram | |||||
TMC | |||||
FCML | |||||
MLC-LRR |
True Labels | bike, front, house, mountain | building, car, group | front, house, lamp |
people, sky, street, tree, wall | people, sky, street, tree | roof, sky, street, tree | |
MSC | building, front, house | building, cloud, front | building, front, house |
lamp, man, mountain | house, mountain, people | lamp, man, mountain | |
people, sky, square, tree | sky, square, tower, tree | people, sky, tree, wall | |
Tram | building, front, house | desert, front, house | bush, flower, front |
man, mountain, people | mountain, people, rock | house, lamp, lawn | |
roof, sky, street, woman | sky, square, street, tree | people, square, tree, street | |
TMC | building, front, house | building, front, house | building, front, house |
lamp, man, mountain | lamp, man, mountain | lamp, man, mountain | |
people, sky, tree, wall | people, sky, tree, wall | people, sky, tree, wall | |
FCML | boy, bridge, cliff | boy, bridge, cliff | boy, bridge, cliff |
cycling, cyclist, hair | cycling, cyclist, hair | cycling, cyclist, hair | |
jersey, people, short, sweater | jersey, people, short, sweater | jersey, people, short, sweater | |
DLP | building, front, house | building, front, house | building, front, house |
lamp, mountain, people | lamp, mountain, people | lamp, mountain, people | |
sky, street, tree, wall | sky, street, tree, wall | sky, street, tree, wall | |
MLC-LRR | building, front, house | building, group, house | building, front, house |
lamp, mountain, people | lamp, mountain, people | lamp, mountain, people | |
sky, street, tree, wall | sky, street, tree, wall | roof, sky, street, tree |
© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license ( http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Tan, Q.; Liu, Y.; Chen, X.; Yu, G. Multi-Label Classification Based on Low Rank Representation for Image Annotation. Remote Sens. 2017, 9, 109. https://doi.org/10.3390/rs9020109
Tan Q, Liu Y, Chen X, Yu G. Multi-Label Classification Based on Low Rank Representation for Image Annotation. Remote Sensing. 2017; 9(2):109. https://doi.org/10.3390/rs9020109
Chicago/Turabian StyleTan, Qiaoyu, Yezi Liu, Xia Chen, and Guoxian Yu. 2017. "Multi-Label Classification Based on Low Rank Representation for Image Annotation" Remote Sensing 9, no. 2: 109. https://doi.org/10.3390/rs9020109