# Representation Learning for Fine-Grained Change Detection

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Applications of Change Detection

#### 2.1. Remote Sensing

#### 2.2. Video Surveillance

#### 2.3. Healthcare

#### 2.4. Monitoring Man-Made Systems

## 3. History of Change Detection

#### 3.1. Statistical Methods

#### 3.2. Deep Learning

## 4. Representation Learning for Fine-Grained Change Detection

#### 4.1. Change Representation

#### 4.1.1. One-Shot Learning

#### 4.1.2. Graph Embedding

#### 4.1.3. Unsupervised Learning

#### 4.2. Types of Representation Learning Architectures

#### 4.2.1. Meta-Learning

#### 4.2.2. Metric Learning

#### 4.2.3. Deep Generative Models

#### 4.2.4. Geometric Deep Learning

#### 4.3. Understanding the Latent Space of Representations

#### 4.3.1. Latent Space Visualisation

#### 4.3.2. Multi-Task/Multi-Metric Correlation

#### 4.3.3. Alternate Space Representation

#### 4.3.4. Structured Representations

## 5. Challenges, Comparisons, and Future Directions for Change Representation Techniques

#### 5.1. Real-Time and Online Change Detection

#### 5.2. Change Detection on Heterogeneous Data

#### 5.3. Interpreting Change from Representations

#### 5.3.1. Trialling Different Visualisations

#### 5.3.2. Explainable Change Detection

#### 5.3.3. Theoretically Grounded Change Detection

#### 5.3.4. Latent Space Alignment

## 6. Overview

## Funding

## Acknowledgments

## Conflicts of Interest

## Abbreviations

ADAS | Advanced Driver Assistance System |

CA | Change Analysis |

CD | Change Detection |

CNN | Convolutional Neural Network |

CR | Change Representation |

CUSUM | Cumulative Sum |

DL | Deep Learning |

DML | Deep Metric Learning |

DNN | Deep Neural Network |

DWT | Discrete Wavelet Transform |

FG | Fine-Grained |

GAN | Generative Adversarial Networks |

GDL | Geometric Deep Learning |

GNN | Graph Neural Network |

IID | Independent and Identically Distributed |

QCD | Quickest Change Detection |

t-SNE | t-Distributed Stochastic Neighbour Embedding |

LTSA | Local Tangent Space Alignment |

MAML | Memory Augmented Meta Learning |

MTL | Multi-Task Learning |

MTML | Multi-Task Metric Learning |

PCA | Principal Component Analysis |

PELT | Pruned Exact Linear Time |

PH | Persistent Homology |

PSD | Positive Semi Definite |

RL | Representation Learning |

SPC | Statistical Process Control |

TDA | Topological Data Analysis |

UMAP | Uniform Manifold Approximation and Projection |

VAE | Variational AutoEncoder |

XAI | eXplainable Artificial Intelligence |

## References

- Li, M.D.; Chang, K.; Bearce, B.; Chang, C.Y.; Huang, A.J.; Campbell, J.P.; Brown, J.M.; Singh, P.; Hoebel, K.V.; Erdo, D.; et al. Siamese neural networks for continuous disease severity evaluation and change detection in medical imaging. NPJ Digit. Med.
**2020**, 3, 1–9. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Wei, X.S.; Wu, J.; Cui, Q. Deep learning for fine-grained image analysis: A survey. arXiv
**2019**, arXiv:1907.03069. [Google Scholar] - Mallet, C.; Le Bris, A. Current challenges in operational very high resolution land-cover mapping. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2020**, 43, 703–710. [Google Scholar] [CrossRef] - Lahoz, W.A.; Schneider, P. Data assimilation: Making sense of Earth Observation. Front. Environ. Sci.
**2014**, 2, 16. [Google Scholar] [CrossRef] [Green Version] - Paolanti, M.; Pietrini, R.; Mancini, A.; Frontoni, E.; Zingaretti, P. Deep understanding of shopper behaviours and interactions using RGB-D vision. Mach. Vis. Appl.
**2020**. [Google Scholar] [CrossRef] - Guerrero-Ibáñez, J.; Zeadally, S.; Contreras-Castillo, J. Sensor technologies for intelligent transportation systems. Sensors
**2018**, 18, 1212. [Google Scholar] [CrossRef] [Green Version] - Ziemann, A.K.; Ren, C.X.; Theiler, J. Multi-sensor anomalous change detection at scale. In Algorithms, Technologies, and Applications for Multispectral and Hyperspectral Imagery XXV; Messinger, D.W., Velez-Reyes, M., Eds.; SPIE: Baltimore, MD, USA, 2019; Volume 10986, p. 37. [Google Scholar] [CrossRef]
- Awty-Carroll, K.; Bunting, P.; Hardy, A.; Bell, G. An Evaluation and Comparison of Four Dense Time Series Change Detection Methods Using Simulated Data. Remote Sens.
**2019**, 11, 2779. [Google Scholar] [CrossRef] [Green Version] - Qin, D.; Zhou, X.; Zhou, W.; Huang, G.; Ren, Y.; Horan, B.; He, J.; Kito, N. MSIM: A change detection framework for damage assessment in natural disasters. Expert Syst. Appl.
**2018**, 97, 372–383. [Google Scholar] [CrossRef] - Shi, W.; Zhang, M.; Zhang, R.; Chen, S.; Zhan, Z. Change detection based on artificial intelligence: State-of-the-art and challenges. Remote Sens.
**2020**, 12, 1688. [Google Scholar] [CrossRef] - Senanayake, R.; Ott, L.; O’Callaghan, S.; Ramos, F. Spatio-temporal hilbert maps for continuous occupancy representation in dynamic environments. In Proceedings of the 30th International Conference on Neural Information Processing Systems, Kyoto, Japan, 16–21 October 2016; pp. 3925–3933. [Google Scholar]
- Foody, G.M.; Atkinson, P.M. Uncertainty in Remote Sensing and GIS; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 2002; pp. 1–307. [Google Scholar] [CrossRef]
- Qin, R.; Tian, J.; Reinartz, P. 3D change detection—Approaches and applications. ISPRS J. Photogramm. Remote. Sens.
**2016**, 122, 41–56. [Google Scholar] [CrossRef] [Green Version] - Lopez-Molina, C.; De Baets, B.; Bustince, H. Quantitative error measures for edge detection. Pattern Recognit.
**2013**, 46, 1125–1139. [Google Scholar] [CrossRef] - Xie, X.; Ge, S.; Xie, M.; Hu, F.; Jiang, N. An improved industrial sub-pixel edge detection algorithm based on coarse and precise location. J. Ambient Intell. Humaniz. Comput.
**2020**, 11, 2061–2070. [Google Scholar] [CrossRef] - Tao, J.; Turjo, M.; Tan, Y.P. Quickest change detection for health-care video surveillance. In Proceedings of the 2006 IEEE International Symposium on Circuits and Systems, Kos, Greece, 21–24 May 2006; pp. 505–508. [Google Scholar] [CrossRef]
- Gao, Y.; Liu, H.; Sun, X.; Wang, C.; Liu, Y. Violence detection using Oriented VIolent Flows. Image Vis. Comput.
**2016**, 48, 37–41. [Google Scholar] [CrossRef] - Kataoka, H.; Satoh, Y.; Aoki, Y.; Oikawa, S.; Matsui, Y. Temporal and fine-grained pedestrian action recognition on driving recorder database. Sensors
**2018**, 18, 627. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Aslam Khan, F.; Member, S.; Hasan Haldar, N.A.; Ali, A.; Iftikhar, M.; Zia, T.A.; Zomaya, A.Y. A Continuous Change Detection Mechanism to Identify Anomalies in ECG Signals for WBAN-Based Healthcare Environments. IEEE Access
**2017**, 5, 13531–13544. [Google Scholar] [CrossRef] - Riboni, D.; Bettini, C.; Civitarese, G.; Janjua, Z.H.; Helaoui, R. SmartFABER: Recognizing Fine-grained Abnormal Behaviors for Early Detection of Mild Cognitive Impairment. Artif. Intell. Med.
**2016**, 67, 57–74. [Google Scholar] [CrossRef] [Green Version] - Sprint, G.; Cook, D.J.; Schmitter-Edgecombe, M. Unsupervised detection and analysis of changes in everyday physical activity data. J. Biomed. Inform.
**2016**, 63, 54–65. [Google Scholar] [CrossRef] [PubMed] - Satija, U.; Ramkumar, B.; Manikandan, M.S. Robust cardiac event change detection method for long-term healthcare monitoring applications. Healthc. Technol. Lett.
**2016**, 3, 116–123. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Colt, R.G.; Várady, C.H.; Volpi, R.; Malagò, L. Automatic Feature Extraction for Heartbeat Anomaly Detection. arXiv
**2021**, arXiv:2102.12289. [Google Scholar] - Klein, M.; Fensel, D.; Kiryakov, A.; Ognyanov, D. Ontology versioning and change detection on the web. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Berlin/Heidelberg, Germany, 2002; Volume 2473, pp. 197–212. [Google Scholar] [CrossRef] [Green Version]
- Uribe, D.O.; Schoukens, J.; Stroop, R. Improved Tactile Resonance Sensor for Robotic Assisted Surgery David. Mech. Syst. Signal Process.
**2018**, 99, 600–610. [Google Scholar] [CrossRef] [Green Version] - Liu, S.; Wright, A.; Hauskrecht, M. Change-point detection method for clinical decision support system rule monitoring. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Berlin/Heidelberg, Germany, 2017; Volume 10259, pp. 126–135. [Google Scholar] [CrossRef] [Green Version]
- Borg, M.; De La Vara, J.L.; Wnuk, K. Practitioners’ Perspectives on Change Impact Analysis for Safety-Critical Software—A Preliminary Analysis. In International Conference on Computer Safety, Reliability, and Security; Springer: Cham, Switzerland, 2016; pp. 346–358. [Google Scholar]
- Patel, H.R.; Shah, V.A. Passive Fault Tolerant Control System Using Feed-forward Neural Network for Two-Tank Interacting Conical Level Control System against Partial Actuator Failures and Disturbances. In IFAC-PapersOnLine; Elsevier B.V.: Amsterdam, The Netherlands, 2019; Volume 52, pp. 141–146. [Google Scholar] [CrossRef]
- Kelly, A. Mobile Robotics; Cambridge University Press: New York, NY, USA, 2013; Volume 9781107031, pp. 1–701. [Google Scholar] [CrossRef]
- Zhou, T.; Dickson, J.L.; Geoffrey Chase, J. Autoregressive Modeling of Drift and Random Error to Characterize a Continuous Intravascular Glucose Monitoring Sensor. J. Diabetes Sci. Technol.
**2018**, 12, 90–104. [Google Scholar] [CrossRef] - Cleveland, R.B.; Cleveland, W.S.; McRae, J.E.; Terpenning, I. STL: A seasonal-trend decomposition. J. Off. Stat.
**1990**, 6, 3–73. [Google Scholar] - Killick, R.; Fearnhead, P.; Eckley, I.A. Optimal detection of changepoints with a linear computational cost. J. Am. Stat. Assoc.
**2012**, 107, 1590–1598. [Google Scholar] [CrossRef] - Aoga, J. Global Constraints for Mining Sets and Sequences. arXiv
**2018**, arXiv:1511.0835. [Google Scholar] - Siegmund, D.; Venkatraman, E.S. Using the Generalized Likelihood Ratio Statistic for Sequential Detection of a Change-Point. Ann. Stat.
**1995**, 23, 255–271. [Google Scholar] [CrossRef] - Oskiper, T.; Poor, H.V. Online activity detection in a multiuser environment using the matrix CUSUM algorithm. IEEE Trans. Inf. Theory
**2002**, 48, 477–493. [Google Scholar] [CrossRef] - Montes De Oca, V.; Jeske, D.R.; Zhang, Q.; Rendon, C.; Marvasti, M. A cusum change-point detection algorithm for non-stationary sequences with application to data network surveillance. J. Syst. Softw.
**2010**, 83, 1288–1297. [Google Scholar] [CrossRef] - Zhang, X.; Woodall, W.H. Dynamic probability control limits for risk-adjusted Bernoulli CUSUM charts. Stat. Med.
**2015**, 34, 3336–3348. [Google Scholar] [CrossRef] [PubMed] - O’ Mahony, N.; Campbell, S.; Carvalho, A.; Harapanahalli, S.; Velasco Hernandez, G.; Krpalkova, L.; Riordan, D.; Walsh, J. Deep Learning vs. Traditional Computer Vision. In Advances in Computer Vision; Chapter Deep Learn; Springer: Cham, Switzerland, 2019; pp. 128–144. [Google Scholar]
- Górska, U.; Rupp, A.; Boubenec, Y.; Celikel, T.; Englitz, B. Evidence integration in natural acoustic textures during active and passive listening. eNeuro
**2018**, 5. [Google Scholar] [CrossRef] [Green Version] - O’ Mahony, N.; Campbell, S.; Krpalkova, L.; Riordan, D.; Walsh, J. Computer Vision for 3D Perception A Review. In Intelligent Systems and Applications, 869th ed.; Bardsiri, A.K., Hashemi, S.M., Eds.; Springer: Cham, Switzerland, 2019; Chapter 59; pp. 788–804. [Google Scholar] [CrossRef] [Green Version]
- Cai, H.; Zheng, V.W.; Chang, K.C.C. A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applications. IEEE Trans. Knowl. Data Eng.
**2018**, 30, 1616–1637. [Google Scholar] [CrossRef] [Green Version] - Dai, Y.; Wang, S.; Xiong, N.N.; Guo, W. A Survey on Knowledge Graph Embedding: Approaches, Applications and Benchmarks. Electronics
**2020**, 9, 750. [Google Scholar] [CrossRef] - Dong, H.; Ma, W.; Wu, Y.; Zhang, J.; Jiao, L. Self-Supervised Representation Learning for Remote Sensing Image Change Detection Based on Temporal Prediction. Remote Sens.
**2020**, 12, 1868. [Google Scholar] [CrossRef] - Lim, S.K.; Loo, Y.; Tran, N.T.; Cheung, N.M.; Roig, G.; Elovici, Y. DOPING: Generative Data Augmentation for Unsupervised Anomaly Detection with GAN. In Proceedings of the 2018 IEEE International Conference on Data Mining (ICDM), Singapore, 17–20 November 2018; pp. 1122–1127. [Google Scholar]
- Santoro, A.; Bartunov, S.; Botvinick, M.; Wierstra, D.; Lillicrap, T. One-shot Learning with Memory-Augmented Neural Networks. arXiv
**2016**, arXiv:1605.06065. [Google Scholar] - Parisi, G.I.; Kemker, R.; Part, J.L.; Kanan, C.; Wermter, S. Continual Lifelong Learning with Neural Networks: A Review. Neural Netw.
**2019**, 113, 54–71. [Google Scholar] [CrossRef] [PubMed] - Musgrave, K.; Belongie, S.; Lim, S.N. A Metric Learning Reality Check. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer Science and Business Media Deutschland GmbH: Berlin, Germany, 2020; Volume 12370, pp. 681–699. [Google Scholar] [CrossRef]
- O’ Mahony, N.; Campbell, S.; Carvalho, A.; Krpalkova, L.; Hernandez, G.V.; Harapanahalli, S.; Riordan, D.; Walsh, J. One-shot learning for custom identification tasks; A review. In Procedia Manufacturing; Elsevier B.V.: Amsterdam, The Netherlands, 2019; Volume 38, pp. 186–193. [Google Scholar] [CrossRef]
- Manmatha, R.; Wu, C.Y.; Smola, A.J.; Krahenbuhl, P. Sampling Matters in Deep Embedding Learning. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2859–2867. [Google Scholar] [CrossRef] [Green Version]
- Lu, C.y.; Jeric, D.; Rustia, A.; Lin, T.T. Generative Adversarial Network Based Image Augmentation for Insect Pest Classification Enhancement. IFAC-PapersOnLine
**2019**, 52, 1–5. [Google Scholar] [CrossRef] - Zhang, K.; Luo, W.; Zhong, Y.; Ma, L.; Liu, W.; Li, H. Adversarial Spatio-Temporal Learning for Video Deblurring. arXiv
**2018**, arXiv:1804.00533v1. [Google Scholar] [CrossRef] [Green Version] - Li, Y.; Zhao, W.; Wang, H.; Sung, M.; Guibas, L. GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud. arXiv
**2018**, arXiv:1812.03320. [Google Scholar] - Mehrotra, A.; Dukkipati, A. Generative Adversarial Residual Pairwise Networks for One Shot Learning. arXiv
**2017**, arXiv:1703.08033. [Google Scholar] - Wang, C.; Chen, Z.; Shang, K. Label-Removed Generative Adversarial Networks Incorporating with K-Means. Neurocomputing
**2019**, 361, 126–136. [Google Scholar] [CrossRef] [Green Version] - Salimans, T.; Goodfellow, I.; Zaremba, W.; Cheung, V.; Radford, A.; Chen, X. Improved Techniques for Training GANs. In Advances in Neural Information Processing Systems; Curran Associates Inc.: Red Hook, NY, USA, 2016; pp. 2234–2242. [Google Scholar]
- Kingma, D.P.; Welling, M. An introduction to variational autoencoders. arXiv
**2019**, arXiv:1906.02691. [Google Scholar] [CrossRef] - Fortuin, V.; Hüser, M.; Locatello, F.; Strathmann, H.; Rätsch, G. SOM-VAE: Interpretable Discrete Representation Learning on Time Series. arXiv
**2018**, arXiv:1806.02199. [Google Scholar] - Hudson, D.A.; Zitnick, C.L. Generative Adversarial Transformers. arXiv
**2021**, arXiv:1412.0767. [Google Scholar] - Girdhar, R.; Carreira, J.; Doersch, C.; Zisserman, A. Video Action Transformer Network. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2019, Long Beach, CA, USA, 16–20 June 2019; pp. 244–253. [Google Scholar] [CrossRef] [Green Version]
- Barla, N. Understanding Representation Learning with Autoencoder: Everything You Need to Know about Representation and Feature Learning—neptune.ai. 2021. Available online: https://neptune.ai/blog/understanding-representation-learning-with-autoencoder-everything-you-need-to-know-about-representation-and-feature-learning (accessed on 5 March 2021).
- Creswell, A.; White, T.; Dumoulin, V.; Arulkumaran, K.; Sengupta, B.; Bharath, A.A. Generative Adversarial Networks: An Overview. IEEE Signal Process. Mag.
**2018**, 35, 53–65. [Google Scholar] [CrossRef] [Green Version] - Grigorescu, S. Generative One-Shot Learning (GOL): A Semi-Parametric Approach to One-Shot Learning in Autonomous Vision. In Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia, 21–25 May 2018. [Google Scholar] [CrossRef] [Green Version]
- Minervini, P.; Demeester, T.; Rocktäschel, T.; Riedel, S. Adversarial Sets for Regularising Neural Link Predictors. arXiv
**2017**, arXiv:1707.07596. [Google Scholar] - Bronstein, M.M.; Bruna, J.; Lecun, Y.; Szlam, A.; Vandergheynst, P. Geometric deep learning: Going beyond Euclidean data. IEEE Sig. Proc. Mag.
**2017**, 34, 18–42. [Google Scholar] [CrossRef] [Green Version] - Hamilton, W.L.; Ying, R.; Leskovec, J. Representation Learning on Graphs: Methods and Applications. arXiv
**2017**, arXiv:1709.05584. [Google Scholar] - Bronstein, A.M. Spectral descriptors for deformable shapes. arXiv
**2011**, arXiv:1110.5015. [Google Scholar] - Garcia, V.; Bruna, J. Few-Shot Learning with Graph Neural Networks. arXiv
**2017**, arXiv:1711.04043. [Google Scholar] - Zhu, W.; Razavian, N. Graph Neural Network on Electronic Health Records for Predicting Alzheimer’s Disease. arXiv
**2019**, arXiv:1912.03761. [Google Scholar] - Zhang, R.; Hao, Y.; Yu, D.; Chang, W.C.; Lai, G.; Yang, Y. Correlation-aware Unsupervised Change-point Detection via Graph Neural Networks. arXiv
**2020**, arXiv:2004.11934. [Google Scholar] - Kalinicheva, E.; Ienco, D.; Sublime, J.; Trocan, M. Unsupervised Change Detection Analysis in Satellite Image Time Series Using Deep Learning Combined with Graph-Based Approaches. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.
**2020**, 13, 1450–1466. [Google Scholar] [CrossRef] - Murphy, C.; Laurence, E.; Allard, A. Deep learning of stochastic contagion dynamics on complex networks. arXiv
**2020**, arXiv:2006.05410. [Google Scholar] - Garofalo, M.; Pellegrino, M.A.; Altabba, A.; Cochez, M. Leveraging Knowledge Graph Embedding Techniques for Industry 4.0 Use Cases. arXiv
**2018**, arXiv:1808.00434. [Google Scholar] - Wattenberg, M.; Viégas, F.; Johnson, I. How to Use t-SNE Effectively. Distill
**2017**, 1, e2. [Google Scholar] [CrossRef] - Liu, Y.; Jun, E.; Li, Q.; Heer, J. Latent Space Cartography: Visual Analysis of Vector Space Embeddings. Comput. Graph. Forum
**2019**, 38, 67–78. [Google Scholar] [CrossRef] - Frenzel, M.F.; Teleaga, B.; Ushio, A. Latent Space Cartography: Generalised Metric-Inspired Measures and Measure-Based Transformations for Generative Models. arXiv
**2019**, arXiv:1902.02113. [Google Scholar] - Recanatesi, S.; Farrell, M.; Lajoie, G.; Deneve, S.; Rigotti, M.; Shea-Brown, E. Predictive learning extracts latent space representations from sensory observations. bioRxiv
**2018**. [Google Scholar] [CrossRef] [Green Version] - Moon, K.R.; van Dijk, D.; Wang, Z.; Gigante, S.; Burkhardt, D.B.; Chen, W.S.; Yim, K.; van den Elzen, A.; Hirn, M.J.; Coifman, R.R.; et al. Visualizing structure and transitions in high-dimensional biological data. Nat. Biotechnol.
**2019**, 37, 1482–1492. [Google Scholar] [CrossRef] [PubMed] - Han, S.W. Efficient Change Detection Methods for Bio and Healthcare Surveillance; Georgia Institute of Technology: Atlanta, GA, USA, 2010. [Google Scholar]
- Fan, H.; Zhang, X.; Mei, S.; Chen, K.; Chen, X. M2gsnet: Multi-modal multi-task graph spatiotemporal network for ultra-short-term wind farm cluster power prediction. Appl. Sci.
**2020**, 10, 7915. [Google Scholar] [CrossRef] - Zhang, Y.; Yang, Q. A Survey on Multi-Task Learning. arXiv
**2017**, arXiv:1707.08114. [Google Scholar] - Caye Daudt, R.; Le Saux, B.; Boulch, A.; Gousseau, Y. Multitask learning for large-scale semantic change detection. Comput. Vis. Image Underst.
**2019**, 187, 102783. [Google Scholar] [CrossRef] [Green Version] - Liu, T.; Tao, D.; Song, M.; Maybank, S.J. Algorithm-Dependent Generalization Bounds for Multi-Task Learning. IEEE Trans. Pattern Anal. Mach. Intell.
**2017**, 39, 227–241. [Google Scholar] [CrossRef] - Garg, S.; Liang, Y. Functional Regularization for Representation Learning: A Unified Theoretical Perspective. arXiv
**2020**, arXiv:2008.02447. [Google Scholar] - Huang, R.; Sun, S. Kernel regression with sparse metric learning. J. Intell. Fuzzy Syst.
**2013**, 24, 775–787. [Google Scholar] [CrossRef] [Green Version] - Capela, F.; Nouchi, V.; Van Deursen, R.; Tetko, I.V.; Godin, G. Multitask Learning on Graph Neural Networks Applied to Molecular Property Predictions. arXiv
**2019**, arXiv:1910.13124. [Google Scholar] - Liang, B.; Wu, P.; Tong, X.; Qiu, Y. Regression and subgroup detection for heterogeneous samples. Comput. Stat.
**2020**, 35, 1853–1878. [Google Scholar] [CrossRef] - Sun, Y.; Zhang, X.; Huang, J.; Wang, H.; Xin, Q. Fine-Grained Building Change Detection From Very High-Spatial-Resolution Remote Sensing Images Based on Deep Multitask Learning. IEEE Geosci. Remote Sens. Lett.
**2020**, 1–5. [Google Scholar] [CrossRef] - Wang, L.; Zhu, D. Tackling multiple ordinal regression problems: Sparse and deep multi-task learning approaches. arXiv
**2019**, arXiv:1907.12508v2. [Google Scholar] - Yang, P.; Huang, K.; Liu, C.L. Geometry preserving multi-task metric learning. Mach. Learn.
**2013**, 92, 133–175. [Google Scholar] [CrossRef] - Mathieu, E.; Le Lan, C.; Maddison, C.J.; Tomioka, R.; Whye Teh, Y. Continuous Hierarchical Representations with Poincaré Variational AutoEncoders. arXiv
**2019**, arXiv:1901.06033. [Google Scholar] - Smith, A.L.; Asta, D.M.; Calder, C.A. The Geometry of Continuous Latent Space Models for Network Data. Stat. Sci. Rev. J. Inst. Math. Stat.
**2019**, 34, 428. [Google Scholar] [CrossRef] - Huang, L.; Lu, J.; Tan, Y.P. Multi-manifold metric learning for face recognition based on image sets. J. Vis. Commun. Image Represent.
**2014**, 25, 1774–1783. [Google Scholar] [CrossRef] - Jain, P. Unsupervised Metric Learning Using Low Dimensional Embedding. Preprints
**2018**. [Google Scholar] [CrossRef] - Perrault-Joncas, D.; Melia, M. Metric Learning and Manifolds: Preserving the Intrinsic Geometry; University of Washington: Seattle, WA, USA, 2012; pp. 1–37. [Google Scholar]
- Li, Y. Curvature-aware Manifold Learning. arXiv
**2017**, arXiv:1706.07167. [Google Scholar] [CrossRef] [Green Version] - Dutta, U.K.; Harandi, M.; Sekhar, C.C. Affinity Guided Geometric Semi-Supervised Metric Learning. arXiv
**2020**, arXiv:2002.12394. [Google Scholar] - Sapienza, F.; Groisman, P.; Jonckheere, M. Weighted Geodesic Distance Following Fermat’s Principle. 2018. Available online: https://openreview.net/forum?id=BJfaMIJwG (accessed on 12 June 2021).
- Feng, G.; Liu, W.; Tao, D.; Zhou, Y. Hessian Regularized Distance Metric Learning for People Re-Identification. Neural Process. Lett.
**2019**, 50, 2087–2100. [Google Scholar] [CrossRef] - Bengio, Y.; Courville, A.; Vincent, P. Representation Learning: A Review and New Perspectives. IEEE Trans. Pattern Anal. Mach. Intell.
**2012**, 35, 1798–1828. [Google Scholar] [CrossRef] - Tang, J.; Shao, Z.; Ma, L. Fine-Grained Expression Manipulation Via Structured Latent Space. In Proceedings of the 2020 IEEE International Conference on Multimedia and Expo (ICME), London, UK, 6–10 July 2020. [Google Scholar]
- Tschannen, M.; Djolonga, J.; Rubenstein, P.K.; Gelly, S.; Lucic, M. On Mutual Information Maximization for Rep-Resentation Learning. arXiv
**2019**, arXiv:1907.13625. [Google Scholar] - Chen, P.; Jia, T.; Wu, P.; Wu, J.; Chen, D. Learning Deep Representations by Mutual Information for Person Re-identification. arXiv
**2019**, arXiv:1908.05860. [Google Scholar] - Hjelm, D.; Bachman, P. Representation Learning with Video Deep InfoMax. arXiv
**2020**, arXiv:2007.13278. [Google Scholar] - Liu, L.; Li, X.; Cheung, W.K.; Xu, C. A Structural Representation Learning for Multi-relational Networks. arXiv
**2018**, arXiv:1805.06197. [Google Scholar] - Veličković, P.; Fedus, W.; Hamilton, W.L.; Liò, P.; Bengioy, Y.; Hjelm, R.D. Deep graph infomax. arXiv
**2018**, arXiv:1809.10341. [Google Scholar] - Wang, Z.; Zhang, Y.; Zhang, Y.; Jiang, J.; Yang, R.; Zhao, J.; Xia, G. Pianotree Vae: Structured Representation Learning for Polyphonic Music. arXiv
**2020**, arXiv:2008.07118. [Google Scholar] - Nakka, K.K.; Salzmann, M. Deep Attentional Structured Representation Learning for Visual Recognition. arXiv
**2018**, arXiv:1805.05389. [Google Scholar] - Finn, C.; Abbeel, P.; Levine, S. Lifelong Few-Shot Learning. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 6–11 August 2017. [Google Scholar]
- Finman, R.; Whelan, T.; Kaess, M.; Leonard, J.J. Toward lifelong object segmentation from change detection in dense RGB-D maps. In Proceedings of the 2013 European Conference on Mobile Robots (ECMR 2013), Barcelona, Spain, 25–27 September 2013; pp. 178–185. [Google Scholar] [CrossRef] [Green Version]
- Paavilainen, P.; Korhonen, I.; Lötjönen, J.; Cluitmans, L.; Jylhä, M.; Särelä, A.; Partinen, M. Circadian activity rhythm in demented and non-demented nursing-home residents measured by telemetric actigraphy. J. Sleep Res.
**2005**, 14, 61–68. [Google Scholar] [CrossRef] [PubMed] - Wang, S.; Skubic, M.; Zhu, Y. Activity density map visualization and dissimilarity comparison for eldercare monitoring. IEEE Trans. Inf. Technol. Biomed.
**2012**, 16, 607–614. [Google Scholar] [CrossRef] [PubMed] - Weinberger, K.Q.; Tesauro, G. Metric Learning for Kernel Regression. In Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, AISTATS 2007, San Juan, Puerto Rico, 21–24 March 2007. [Google Scholar]
- Taha, A.; Chen, Y.T.; Misu, T.; Shrivastava, A.; Davis, L. Unsupervised data uncertainty learning in visual retrieval systems. arXiv
**2019**, arXiv:1902.02586. [Google Scholar] - Shi, Y.; Bellet, A.; Sha, F. Sparse Compositional Metric Learning. arXiv
**2014**, arXiv:1404.4105. [Google Scholar] - Ying, Y.; Huang, K.; Campbell, C. Sparse Metric Learning via Smooth Optimization. In Proceedings of the 22nd International Conference on Neural Information Processing Systems (NIPS’09), Vancouver, BC, Canada, 7–10 December 2009. [Google Scholar]
- Aiordachioaie, D.; Popescu, T.D. Change Detection by Feature Extraction and Processing from Time-Frequency Images. In Proceedings of the 10th International Conference on Electronics, Computers and Artificial Intelligence (ECAI 2018), Iasi, Romania, 28–30 June 2018. [Google Scholar] [CrossRef]
- Hajij, M.; Zamzmi, G.; Cai, X. Persistent Homology and Graphs Representation Learning. arXiv
**2021**, arXiv:2102.12926. [Google Scholar] - Munch, E. A User’s Guide to Topological Data Analysis. J. Learn. Anal.
**2017**, 4, 47–61. [Google Scholar] [CrossRef] - Bouchaffra, D.; Ykhlef, F. Persistent Homology for Land Cover Change Detection. In Oxford Research Encyclopedia of Natural Hazard Science; Oxford University Press: Oxford, UK, 2021. [Google Scholar] [CrossRef]
- Hajij, M.; Wang, B.; Scheidegger, C.; Rosen, P. Visual Detection of Structural Changes in Time-Varying Graphs Using Persistent Homology. In Proceedings of the IEEE Pacific Visualization Symposium, Kobe, Japan, 10–13 April 2018; pp. 125–134. [Google Scholar] [CrossRef] [Green Version]
- Chung, M.K.; Hanson, J.L.; Ye, J.; Davidson, R.J.; Pollak, S.D. Persistent Homology in Sparse Regression and its Application to Brain Morphometry. IEEE Trans. Med. Imaging
**2014**, 34, 1928–1939. [Google Scholar] [CrossRef] [Green Version] - Hido, S.; Idé, T.; Kashima, H.; Kubo, H.; Matsuzawa, H. Unsupervised change analysis using supervised learning. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Berlin/Heidelberg, Germany, 2008; Volume 5012, pp. 148–159. [Google Scholar] [CrossRef] [Green Version]
- Aminikhanghahi, S.; Cook, D.J. A survey of methods for time series change point detection. Knowl. Inf. Syst.
**2017**, 51, 339–367. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Fallati, L.; Savini, A.; Sterlacchini, S.; Galli, P. Land use and land cover (LULC) of the Republic of the Maldives: First national map and LULC change analysis using remote-sensing data. Environ. Monit. Assess.
**2017**, 189, 1–15. [Google Scholar] [CrossRef] [PubMed] - Selvaraju, R.R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. Int. J. Comput. Vis.
**2016**, 128, 336–359. [Google Scholar] [CrossRef] [Green Version] - Shi, S.; Zhang, X.; Fan, W. A Modified Perturbed Sampling Method for Local Interpretable Model-agnostic Explanation. arXiv
**2020**, arXiv:2002.07434. [Google Scholar] - O’Mahony, N.; Campbell, S.; Carvalho, A.; Krpalkova, L.; Velasco-Hernandez, G.; Riordan, D.; Walsh, J. Understanding and Exploiting Dependent Variables with Deep Metric Learning. In Advances in Intelligent Systems and Computing; Springer: Berlin/Heidelberg, Germany, 2021; Volume 1250, pp. 97–113. [Google Scholar] [CrossRef]
- Zhu, S.; Yang, T.; Chen, C. Visual Explanation for Deep Metric Learning. arXiv
**2019**, arXiv:1909.12977. [Google Scholar] - Verma, S.; Dickerson, J.; Hines, K. Counterfactual Explanations for Machine Learning: A Review. arXiv
**2020**, arXiv:2010.10596. [Google Scholar] - Schölkopf, B.; Locatello, F.; Bauer, S.; Ke, N.R.; Kalchbrenner, N.; Goyal, A.; Bengio, Y. Towards Causal Representation Learning. Proc. IEEE
**2021**, 109, 612–634. [Google Scholar] [CrossRef] - Yao, L.; Li, S.; Li, Y.; Huai, M.; Gao, J.; Zhang, A. Representation Learning for Treatment Effect Estimation from Observational Data. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018); Curran Associates Inc.: Red Hook, NY, USA, 2018. [Google Scholar]
- Borghesi, A.; Baldo, F.; Milano, M. Improving Deep Learning Models via Constraint-Based Domain Knowledge: A Brief Survey. arXiv
**2020**, arXiv:2005.10691. [Google Scholar] - Seo, S.; Liu, Y. Differentiable Physics-informed Graph Networks. arXiv
**2019**, arXiv:1902.02950. [Google Scholar] - Cranmer, M.; Greydanus, S.; Hoyer, S.; Research, G.; Battaglia, P.; Spergel, D.; Ho, S. Lagrangian Neural Networks. arXiv
**2020**, arXiv:2003.04630. [Google Scholar] - Greydanus, S.; Dzamba, M.; Yosinski, J. Hamiltonian Neural Networks. arXiv
**2019**, arXiv:1906.01563. [Google Scholar] - Rocktäschel, T.; Singh, S.; Riedel, S. Injecting Logical Background Knowledge into Embeddings for Relation Extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; Association for Computational Linguistics: Stroudsburg, PA, USA, 2015; pp. 1119–1129. [Google Scholar] [CrossRef] [Green Version]
- Gsponer, S.; Costabello, L.; Van, C.L.; Pai, S.; Gueret, C.; Ifrim, G.; Lecue, F. Background Knowledge Injection for Interpretable Sequence Classification. arXiv
**2020**, arXiv:2006.14248. [Google Scholar]

**Figure 1.**A triplet-based metric learning architecture. Each of the three samples is passed through the same embedding network, and the loss function determines how to space them apart in latent space.

**Figure 2.**The autoencoder architecture can be considered a form of representation learning where the mid-level encoded data are interpreted as output. Reproduced with permission [60].

**Figure 3.**Graph neural network for representation learning. Note: dotted lines indicate learnt edge features and node colour changes indicate the aggregation of information by convolutional layers. Reproduced with permission [67].

**Figure 4.**Latent space visualisation tools: (

**a**) latent space cartography. Reproduced with permission [74]. (

**b**) Generalised metric-inspired measures and measure-based transformations for generative models. Reproduced with permission [75]. (

**c**) PHATE. Reproduced with permission [77]. (

**d**) Manifold analysis for navigation tasks, where a navigating agent learns to predict the upcoming sensory observation, and the dynamical and geometrical properties are captured in a neural representation manifold. Reproduced with permission [76].

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

O’Mahony, N.; Campbell, S.; Krpalkova, L.; Carvalho, A.; Walsh, J.; Riordan, D.
Representation Learning for Fine-Grained Change Detection. *Sensors* **2021**, *21*, 4486.
https://doi.org/10.3390/s21134486

**AMA Style**

O’Mahony N, Campbell S, Krpalkova L, Carvalho A, Walsh J, Riordan D.
Representation Learning for Fine-Grained Change Detection. *Sensors*. 2021; 21(13):4486.
https://doi.org/10.3390/s21134486

**Chicago/Turabian Style**

O’Mahony, Niall, Sean Campbell, Lenka Krpalkova, Anderson Carvalho, Joseph Walsh, and Daniel Riordan.
2021. "Representation Learning for Fine-Grained Change Detection" *Sensors* 21, no. 13: 4486.
https://doi.org/10.3390/s21134486