Incremental Canonical Correlation Analysis
Abstract
:Featured Application
Abstract
1. Introduction
2. Related Work
3. Methodology
3.1. Review of Canonical Correlation Analysis
3.2. Incremental Canonical Correlation Analysis (ICCA)
4. Experiments
4.1. Settings
Algorithm 1. Incremental Subspace Algorithm |
Input: samples and , new samples and . |
Output: the updated subspace , the updated mean and covariance ,, ; |
1. Update the mean , and covariance and of samples, according to Equation (5); |
2. Update the mean , of current samples according to Equation (6); |
3. according to Equations (4) and (5); |
4. Calculate , , , and according to Equation (7); |
5. Calculate according to Equation (8); |
6. Calculate and to yield new subspace , according to Equation (9). |
4.2. Results
4.2.1. Efficiency Comparison
4.2.2. Qualitative Comparison
4.2.3. Quantitative Comparison
5. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Harold, H. Relations between two Sets of Variables. Biometrika 1936, 28, 3–4. [Google Scholar]
- Hardoon, D.; Szedmak, S.; Shawe-Taylor, J. Canonical Correlation Analysis: An Overview with Application to Learning Methods. Neural Comput. 2004, 16, 2639–2664. [Google Scholar] [CrossRef] [Green Version]
- Sun, L.; Ceran, B.; Ye, J.P. A scalable two-stage approach for a class of dimensionality reduction techniques. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, 25–28 July 2010. [Google Scholar]
- Yuan, Y.H.; Sun, Q.S.; Ge, H.W. Fractional-order embedding canonical correlation analysis and its applications to multi-view dimensionality reduction and recognition. Pattern Recognit. 2014, 47, 1411–1424. [Google Scholar] [CrossRef]
- Zhang, Y.; Zhang, J.; Pan, Z.; Zhang, D. Multi-view dimensionality reduction via canonical random correlation analysis. Front. Comput. Sci. 2016, 10, 856–869. [Google Scholar] [CrossRef]
- Blaschko, M.B.; Lampert, C.H. Correlational spectral clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA, 23–28 July 2008; pp. 1–8. [Google Scholar]
- Chaudhuri, K.; Kakade, S.M.; Livescu, K.; Sridharan, K. Multi-view clustering via canonical correlation analysis. In Proceedings of the 26th Annual International Conference on Machine Learning, Montreal, QC, Canada, 14–18 June 2009. [Google Scholar]
- Kakade, S.M.; Foster, D.P. Multi-view Regression Via Canonical Correlation Analysis. In International Conference on Computational Learning Theory; Bshouty, N.H., Gentile, C., Eds.; Springer: Berlin/Heidelberg, Germany, 2007. [Google Scholar]
- Lambert, Z.V.; Wildt, A.R.; Durand, R.M. Redundancy analysis: An alternative to canonical correlation and multivariate multiple regression in exploring interset associations. Psychol. Bull. 1988, 104, 282–289. [Google Scholar] [CrossRef]
- Dhillon, P.; Rodu, J.; Foster, D.; Ungar, L. Two Step CCA: A new spectral method for estimating vector models of words. Comput. Sci. 2012, 2, 67–74. [Google Scholar]
- Gong, Y.; Ke, Q.; Isard, M.; Lazebnik, S. A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their Semantics. Int. J. Comput. Vis. 2014, 106, 210–233. [Google Scholar] [CrossRef] [Green Version]
- Nam, K.M.; Song, H.J.; Kim, J.D. Find Alternative Biomarker via Word Embedding. In Proceedings of the Green and Smart Technology, Lisbon, Portugal, 19–22 December 2015; pp. 789–792. [Google Scholar]
- Kim, T.K.; Kittler, J.; Cipolla, R. Discriminative Learning and Recognition of Image Set Classes Using Canonical Correlations. IEEE Trans. Pattern Anal. Mach. Intell. 2007, 29, 1005–1018. [Google Scholar] [CrossRef] [PubMed]
- Su, Y.; Fu, Y.; Gao, X.; Tian, Q. Discriminant Learning Through Multiple Principal Angles for Visual Recognition. IEEE Trans. Image Process. 2012, 21, 1381–1390. [Google Scholar] [CrossRef]
- Yi, Z.; Schneider, J.G. Multi-Label Output Codes using Canonical Correlation Analysis. Mach. Learn. Res. 2011, 15, 873–882. [Google Scholar]
- Wang, M.; Shao, W.; Hao, X.; Shen, L.; Zhang, D. Identify Consistent Cross-Modality Imaging Genetic Patterns via Discriminant Sparse Canonical Correlation Analysis. IEEE/ACM Trans. Comput. Biol. Bioinform. 2019, 99, 1. [Google Scholar] [CrossRef] [PubMed]
- Hu, R.; Zhu, X.; Cheng, D.; He, W.; Yan, Y.; Song, J.; Zhang, S. Graph self-representation method for unsupervised feature selection. Neurocomputing 2017, 220, 130–137. [Google Scholar] [CrossRef]
- Li, S.; Zeng, C.; Fu, Y.; Liu, S. Optimizing multi-graph learning based salient object detection. Signal Process. Image Commun. 2017, 55, 93–105. [Google Scholar] [CrossRef] [Green Version]
- Peng, Y.; Wang, S.; Long, X.; Lu, B.L. Discriminative graph regularized extreme learning machine and its application to face recognition. Neurocomputing 2015, 149, 340–353. [Google Scholar] [CrossRef]
- Li, S.; Tang, C.; Liu, X.; Liu, Y.; Chen, J. Dual graph regularized compact feature representation for unsupervised feature selection. Neurocomputing 2019, 331, 77–96. [Google Scholar] [CrossRef]
- Tang, C.; Zhu, X.; Liu, X.; Li, M.; Wang, P.; Zhang, C.; Wang, L. Learning a Joint Affinity Graph for Multiview Subspace Clustering. IEEE Trans. Multimed. 2019, 21, 1724–1736. [Google Scholar] [CrossRef]
- Andrew, G.; Arora, R.; Bilmes, J.; Livescu, K. Deep Canonical Correlation Analysis. In Proceedings of the International Conference on International Conference on Machine Learning, Atlanta, GE, USA, 17–19 June 2013. [Google Scholar]
- Arthur, T.; Cathy, P.; Vincent, G.; Le, K.A.; Jacques, G.; Vincent, F. Variable Selection for Generalized Canonical Correlation Analysis. Biostatistics 2014, 15, 569–583. [Google Scholar]
- Benton, A.; Khayrallah, H.; Gujral, B.; Reisinger, D.; Arora, R. Deep Generalized Canonical Correlation Analysis. In Proceedings of the 4th Workshop on Representation Learning for NLP, Florence, Italy, 2 August 2019; pp. 1–6. [Google Scholar]
- Allen-Zhu, Z.; Li, Y. Doubly Accelerated Methods for Faster CCA and Generalized Eigendecomposition. In Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia, 6–11 August 2017; pp. 98–106. [Google Scholar]
- Arora, R.; Marinov, T.V.; Mianjy, P.; Srebro, N. Stochastic Approximation for Canonical Correlation Analysis. In Proceedings of the Annual Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 4775–4784. [Google Scholar]
- Bhatia, K.; Pacchiano, A.; Flammarion, N.; Bartlett, P.L.; Jordan, M.I. Gen-Oja: A Simple and Efficient Algorithm for Streaming Generalized Eigenvector Computation. In Proceedings of the Annual Conference on Neural Information Processing Systems, Montréal, QC, Canada, 3–8 December 2018. [Google Scholar]
- Yger, F.; Berar, M.; Gasso, G.; Rakotomamonjy, A. Adaptive canonical correlation analysis based on matrix manifolds. In Proceedings of the 29th International Conference on Machine Learning, Edinburgh, UK, 26 June–1 July 2012. [Google Scholar]
- Gao, C.; Garber, D.; Srebro, N.; Wang, J.; Wang, W. Stochastic Canonical Correlation Analysis. J. Mach. Learn. Res. 2019, 20, 1–46. [Google Scholar]
- Kanatsoulis, C.I.; Fu, X.; Sidiropoulos, N.D.; Hong, M. Structured SUMCOR Multiview Canonical Correlation Analysis for Large-Scale Data. IEEE Trans. Signal Process. 2019, 67, 306–319. [Google Scholar] [CrossRef]
- Lu, Y.; Foster, D.P. Large scale canonical correlation analysis with iterative least squares. In Proceedings of the Annual Conference on Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; pp. 91–99. [Google Scholar]
- Chen, Z.H.; Li, X.G.; Yang, L.; Haupt, J.; Zhao, T. On constrained nonconvex stochastic optimization: A case study for generalized eigenvalue decomposition. In Proceedings of the 22nd International Conference on Artifificial Intelligence and Statistics, Naha, Okinawa, Japan, 16–18 April 2019; pp. 916–925. [Google Scholar]
- Ma, Z.; Lu, Y.C.; Foster, D.P. Finding linear structure in large datasets with scalable canonical correlation analysis. In Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 6–11 July 2015; pp. 169–178. [Google Scholar]
- Kettenring, J.R. Canonical analysis of several sets of variables. Biometrika 1971, 58, 433–451. [Google Scholar] [CrossRef]
- Golub, G.H.; Zha, H. The Canonical Correlations of Matrix Pairs and their Numerical Computation. In Linear Algebra for Signal Processing; Bojanczyk, A., Cybenko, G., Eds.; Springer: New York, NY, USA, 1995. [Google Scholar]
- Avron, H.; Boutsidis, C.; Toledo, S.; Zouzias, A. Efficient Dimensionality Reduction for Canonical Correlation Analysis. Sci. Comput. 2014, 36, 347–355. [Google Scholar] [CrossRef] [Green Version]
- Tropp, J.A. Improved Analysis of the Subsampled Randomized Hadamard Transform. Adv. Data Sci. Adapt. Anal. 2011, 3, 115–126. [Google Scholar] [CrossRef] [Green Version]
- Wang, W.R.; Wang, J.L.; Srebro, N. Globally convergent stochastic optimization for canonical correlation analysis. Adv. Neural Inf. Proc. Syst. 2016, 1, 766–774. [Google Scholar]
- Ge, R.; Jin, C.; Kakade, S.M.; Netrapalli, P.; Sidford, A. Efficient algorithms for large-scale generalized eigenvector computation and canonical correlation analysis. In Proceedings of the 33rd International Conference on International Conference on Machine Learning, New York, NY, USA, 20–22 June 2016; pp. 2741–2750. [Google Scholar]
- Xu, Z.Q.; Li, P. Towards Practical Alternating Least-Squares for CCA. In Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 8–14 December 2019. [Google Scholar]
- Kim, M. Correlation-based incremental visual tracking. Pattern Recognit. 2012, 45, 1050–1060. [Google Scholar] [CrossRef]
- Ross, D.A.; Lim, J.; Lin, R.S.; Yang, M.H. Incremental Learning for Robust Visual Tracking. Int. J. Comput. Vis. 2008, 77, 125–141. [Google Scholar] [CrossRef]
- Bhatia, K.; Pacchiano, A.; Flammarion, N.; Bartlett, P.L.; Jordan, M.I. Gen-Oja: A Two-time-scale approach for Streaming CCA. arXiv 2018, arXiv:1811.08393. Available online: https://arxiv.org/abs/1811.08393 (accessed on 4 November 2020).
- Levey, A.; Lindenbaum, M. Sequential Karhunen-Loeve basis extraction and its application to images. IEEE Trans. Image Process. 2000, 9, 1371–1374. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Pan, J.; Lim, J.; Su, Z.X.; Yang, M.H. L0-Regularized Object Representation for Visual Tracking. In Proceedings of the British Machine Vision Conference, Nottingham, UK, 1–5 September 2014. [Google Scholar]
- Adam, A.; Rivlin, E.; Shimshoni, I. Robust Fragments-based Tracking using the Integral Histogram. In Proceedings of the Computer Society Conference on Computer Vision and Pattern Recognition, New York, NY, USA, 17–22 June 2006. [Google Scholar]
- Kwon, J.; Lee, K.M. Visual tracking decomposition. Computer Vision & Pattern Recognition. In Proceedings of the 23rd IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 13–18 June 2010; pp. 1269–1276. [Google Scholar]
- Bao, C.L.; Wu, Y.; Li, H.B.; Ji, H. Real time robust L1 tracker using accelerated proximal gradient approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, 16–21 June 2012. [Google Scholar]
- Babenko, B.; Yang, M.H.; Belongie, S.J. Visual tracking with online Multiple Instance Learning. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 983–990. [Google Scholar]
- Wu, Y.; Lim, J.; Yang, M.H. Online Object Tracking: A Benchmark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 2411–2418. [Google Scholar]
ICCA | IVT | L1APG | TLD | VTD | MIL | Frag | |
---|---|---|---|---|---|---|---|
Basketball | 31.01 | 22.09 | 2.36 | 19.81 | 13.36 | 30.6 | 2.74 |
Car4 | 30.8 | 22.1 | 1.6747 | 14.97 | 8.26 | 27.73 | 7.07 |
Cardark | 33.12 | 32.41 | 3.27 | 25.22 | 18.95 | 27.16 | 6.68 |
Carscale | 25.01 | 22.23 | 2.59 | 23.79 | 12.96 | 24.66 | 3.47 |
David3 | 32.71 | 31.65 | 3.01 | 19.65 | 14.10 | 25.45 | 3.19 |
Deer | 23.7 | 21.9 | 2.1516 | 16.62 | 12.15 | 17.38 | 2.68 |
Faceocc1 | 28.87 | 27.31 | 4.56 | 17.85 | 13.80 | 27.78 | 4.27 |
Faceocc2 | 30.95 | 22.49 | 3 | 21.93 | 13.80 | 29.01 | 5.32 |
Fish | 37.05 | 29.01 | 2.22 | 36.11 | 16.48 | 25.02 | 5.39 |
Football | 31.53 | 22.06 | 3.44 | 20.99 | 13.67 | 31.56 | 2.87 |
Crossing | 34.12 | 27.84 | 1.48 | 30.14 | 12.96 | 11.6 | 4.89 |
Dog1 | 23.03 | 22.86 | 1.66 | 10.03 | 8.44 | 18.67 | 6.71 |
Jogging-2 | 26.81 | 5.3 | 0.18 | 26.71 | 19.82 | 30.06 | 5.36 |
Jumping | 29.76 | 8.53 | 1.02 | 10.78 | 14.87 | 29.8 | 7.41 |
Mhyang | 31.34 | 7.31 | 3.17 | 12.09 | 7.76 | 46.3 | 3.15 |
Mountain Bike | 32.36 | 31.96 | 3.02 | 17.98 | 14.27 | 27.93 | 3.67 |
Singer1 | 35.79 | 22.49 | 2.29 | 23.19 | 14.49 | 34.56 | 3.44 |
Walking | 33.08 | 32.22 | 4.34 | 12.45 | 13.52 | 28.34 | 2.19 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhao, H.; Sun, D.; Luo, Z. Incremental Canonical Correlation Analysis. Appl. Sci. 2020, 10, 7827. https://doi.org/10.3390/app10217827
Zhao H, Sun D, Luo Z. Incremental Canonical Correlation Analysis. Applied Sciences. 2020; 10(21):7827. https://doi.org/10.3390/app10217827
Chicago/Turabian StyleZhao, Hongmin, Dongting Sun, and Zhigang Luo. 2020. "Incremental Canonical Correlation Analysis" Applied Sciences 10, no. 21: 7827. https://doi.org/10.3390/app10217827
APA StyleZhao, H., Sun, D., & Luo, Z. (2020). Incremental Canonical Correlation Analysis. Applied Sciences, 10(21), 7827. https://doi.org/10.3390/app10217827