Remote Sensing and Machine Learning for Safer Railways: A Review

: Regular railway inspections are crucial for maintaining their safety and efficiency. However, traditional inspection methods are complex and expensive. Consequently, there has been a significant shift toward combining remote sensing (RS) and machine learning (ML) techniques to enhance the efficiency and accuracy of railway defect monitoring while reducing costs. The advantages of RS-ML techniques include their ability to automate and refine inspection processes and address challenges such as image quality and methodological limitations. However, the integration of RS and ML in railway monitoring is an emerging field, with diverse methodologies and outcomes that the research has not yet synthesized. To fill this gap, this study conducted a systematic literature review (SLR) to consolidate the existing research on RS-ML applications in railway inspection. The SLR meticulously compiled and analyzed relevant studies, evaluating the evolution of research trends, methodological approaches, and the geographic distribution of contributions. The findings showed a notable increase in relevant research activity over the last five years, highlighting the growing interest in this realm. The key methodological patterns emphasize the predominance of approaches based on convolutional neural networks, a variant of artificial neural networks, in achieving high levels of precision. These findings serve as a foundational resource for academics, researchers, and practitioners in the fields of computer science, engineering, and transportation to help guide future research directions and foster the development of more efficient, accurate, and cost-effective railway inspection methods.


Introduction
The condition assessment of vast linear transportation structures like roads and railways pose challenges due to the associated costs and time requirements.However, regularly monitoring these structures is necessary for ensuring the safety, efficiency, and reliability of the supply chains that depend on them.Conventional methods of condition monitoring focus on human observation and historical incident data analysis.However, in recent years, remote sensing (RS) has gained prominence, especially in the realm of rail defect detection.RS involves capturing data about an object without direct contact, offering an effective approach for enhancing inspection and maintenance without disrupting operations [1].
The recent literature in RS of transportation infrastructure has explored various techniques, including the spectral image differencing procedure (SIDP) [2], linear chargecoupled devices (CCDs) [3], 3D laser cameras [4], LiDAR [5], multi-temporal interferometric synthetic aperture radar (MT-InSAR) [6], and drone-based monitoring [7].With advancements in machine learning (ML), organizations can now process and analyze captured data more accurately and promptly [8].Hence, the combination of RS and ML techniques promise to revolutionize railway condition monitoring.ML is a branch of artificial intelligence (AI) that enable computers to learn from data without explicit programming [9].Greater volumes of training data enhance ML efficacy.Moreover, the data analyzed for ML applications can manifest in diverse forms, including texts and captured images.
While the literature has addressed numerous methods for rail inspection, there is a notable gap in comprehensive literature reviews specifically examining the convergence of RS and ML methods in this realm.This study aimed to fill that void by conducting a systematic literature review (SLR) on rail infrastructure monitoring that integrates RS and ML techniques.The SLR analyzed the employed methods, results, and challenges addressed and provided a descriptive analysis of pertinent publications.The review scrutinized technical aspects such as the model architecture, accuracy, datasets, and learning rates of the ML algorithms.The study consolidated and unveiled relationships between these varied studies, comparing different approaches, providing topic classifications, and presenting distribution maps by year, journal, and country, thereby paving the way for future research directions.
The research questions associated with the SLR were as follows: • What is the current state of the art that combines RS and ML methods to monitor the condition of the railways?• What are the advantages and disadvantages of these methods, and how will they benefit future researchers?• What are the different applications of these methods in railway monitoring?
This SLR serves as an effective tool to map, evaluate, and summarize the disparate literature, thus enhancing conceptual development within the field.The structure of the remaining publication is as follows: Section 2 presents the SLR method, which followed the PRISMA guidelines [10].Section 3 presents a descriptive analysis of the review.Section 4 discusses the results.Section 5 concludes the review and offers insights for future research directions.

Research Method
The choice of the SLR method for this study stemmed from its rigorous and structured approach [10], which is essential for synthesizing the existing literature in a field as complex and varied as railway infrastructure monitoring using RS and ML technologies.SLRs are particularly suited for areas with diverse methodologies and findings, as they provide a comprehensive overview of the state of the research and can identify gaps and suggest areas for future investigation.This method ensures a transparent, reproducible research process, enhancing the credibility and reliability of the findings by minimizing bias through predefined inclusion and exclusion criteria.Central to the SLR methodology, the authors implemented several steps to thoroughly examine the relevant literature by formulating the review questions, searching for publications, evaluating and refining the publications selected, analyzing and synthesizing the findings, and reporting the outcomes.
Following the establishment of the main objective, the authors formulated the final research questions as outlined in the introduction section.The publication search phase used specific inclusion and exclusion criteria.Initially, the search examined general publications related to RS, transportation infrastructure, and ML, in addition to SLR publications on other topics.After reviewing the initial set of publications, the authors narrowed the search to focus solely on publications relating to both RS and ML in the context of rail infrastructure.The inclusion criteria were peer-reviewed journal publications encompassing all three aspects-RS, ML, and railway infrastructure inspection.That is, the authors excluded any publication that lacked all three elements.
The initial search utilized Google Scholar for its strength in locating academic literature from multiple databases, including Scopus, IEEE, and Web of Science.The search encompassed entering the specific phrases shown in Table 1.The number of records initially identified was 2151, but further screening of the titles and abstracts yielded 42 publications, as summarized in Table 1.Further scrutiny of each publication yielded 10 relevant publications after excluding 32, of which, 29 did not fit the topic scope and 3 were not from journals.
Subsequently, the authors applied the snowball technique to explore the cited works of the 10 relevant publications, which uncovered an additional 113 relevant publications.
Careful examination of these included adding short summaries or review points gleaned from each.The authors then established a table to catalog all the methods used, including the main features of their methods, as discussed later in the results section.This scrutiny removed 68 publications from the 113, resulting in 45 that were relevant.Of the excluded publications, 28 were out of scope, 2 were duplicates, 5 lacked ML content, 12 did not focus on railway infrastructure, and 21 were not journal publications.The 10 initial publications selected plus the 45 screened from the snowball procedure resulted in a final tally of 55 relevant publications for further analysis.

Descriptive Analysis
The subsections that follow discuss the citation relationship network of the relevant publications and provide their distribution by year, journal, author country, and method.

Literature Network
This literature review aimed to acquire insights into the existing scholarly works.The authors discovered a cluster of interconnected research that formed a cohesive body of related works within the realm of RS-ML railway inspection.Figure 1 illustrates this relationship by using the 2024 version of an online tool called Litmaps [11].
The connecting lines trace the citations in the selected body of literature.The horizontal axis represents the publication date.Hence, lines leading from the left of an article connect to its references.The vertical axis encodes the number of citations, with larger circles representing a higher number of citations.The clustering of most publications suggests a close interrelation, indicating shared concepts and common methodologies within the field.However, certain publications, such as [12][13][14], stand out as outliers because of their more distant or divergent research focus compared with the closely connected majority.
The network diagram helps to visualize the interconnections and thematic concentrations within the existing research.This type of mapping facilitates the identification of key research clusters and outlier studies, illustrating how different methodologies and findings relate to one another.It also helps to pinpoint influential studies and emerging trends to provide a clearer picture of the research landscape.Furthermore, this visualization helps to enhance our understanding of how the field evolved by illustrating how newer methodologies built upon or diverged from established practices.These insights can help to inform future research directions and potential areas for innovation.The connecting lines trace the citations in the selected body of literature.The horizontal axis represents the publication date.Hence, lines leading from the left of an article connect to its references.The vertical axis encodes the number of citations, with larger circles representing a higher number of citations.The clustering of most publications suggests a close interrelation, indicating shared concepts and common methodologies within the field.However, certain publications, such as [12][13][14], stand out as outliers because of their more distant or divergent research focus compared with the closely connected majority.
The network diagram helps to visualize the interconnections and thematic concentrations within the existing research.This type of mapping facilitates the identification of key research clusters and outlier studies, illustrating how different methodologies and findings relate to one another.It also helps to pinpoint influential studies and emerging trends to provide a clearer picture of the research landscape.Furthermore, this visualization helps to enhance our understanding of how the field evolved by illustrating how newer methodologies built upon or diverged from established practices.These insights can help to inform future research directions and potential areas for innovation.

Distribution by Year
Figure 2 illustrates the chronological order of the reviewed publications.Most publications appeared recently, from 2017 onward.Notably, 2021 and 2022 accounted for 27% and 25% of the publications, respectively.In 2019 and 2018, there were eight and four publications, respectively.There was only one relevant publication per year from 2010 to 2016.This pattern suggests that the combined use of RS and ML in railway condition monitoring is at the forefront of current research.Traditional ML algorithms such as support vector machines (SVMs), logistic regression, naïve Bayes, random forest, and gradient boosting gave way to deep learning architectures that utilize large multi-layer artificial neural networks (ANNs), including variants like convolution neural networks (CNNs) that researchers optimized for machine vision [15].

Distribution by Year
Figure 2 illustrates the chronological order of the reviewed publications.Most publications appeared recently, from 2017 onward.Notably, 2021 and 2022 accounted for 27% and 25% of the publications, respectively.In 2019 and 2018, there were eight and four publications, respectively.There was only one relevant publication per year from 2010 to 2016.This pattern suggests that the combined use of RS and ML in railway condition monitoring is at the forefront of current research.Traditional ML algorithms such as support vector machines (SVMs), logistic regression, naïve Bayes, random forest, and gradient boosting gave way to deep learning architectures that utilize large multi-layer artificial neural networks (ANNs), including variants like convolution neural networks (CNNs) that researchers optimized for machine vision [15].

Distribution by Author Country
Figure 4 shows the country of origin for the authors of the reviewed publications.To create this figure, the review identified the country of every author, based on their affiliations, and tallied their representation in the reviewed corpus of literature.
China dominated with 167 authors, followed by the United States with 26 and Sweden with 19.The Netherlands and Brazil were close behind, each with authors contributing to 11 and 8 publications, respectively.Authors from the other 13 countries represented only 18% of the total authorship.China and the United States alone accounted for 68% of the authorship with China dominating at 60%.

Methods Reported
When broadly categorized, ML models are either supervised or unsupervised, with the former requiring labeled data to train the models.The types of supervised ML models include naïve Bayes (NB), decision tree (DT), logistic regression (LR), artificial neural network (ANN), and others [16].There are also several types of unsupervised ML models that seek to identify data clusters that have similar characteristics [17].The authors meticulously counted the type of methods used in each of the selected publications to gain insights into their approaches.Table 2 summarizes the dominant methods reported in the final batch of relevant articles reviewed, considering that some studies employed multiple methods.Figure 5 shows how the methods were distributed based on their frequency of appearance and the year of the publication.Figure 6 shows how the publications were distributed by year and split by the methods used.
Figure 4 shows the country of origin for the authors of the reviewed publications.To create this figure, the review identified the country of every author, based on their affiliations, and tallied their representation in the reviewed corpus of literature.China dominated with 167 authors, followed by the United States with 26 and Sweden with 19.The Netherlands and Brazil were close behind, each with authors contributing to 11 and 8 publications, respectively.Authors from the other 13 countries represented only 18% of the total authorship.China and the United States alone accounted for 68% of the authorship with China dominating at 60%.

Methods Reported
When broadly categorized, ML models are either supervised or unsupervised, with the former requiring labeled data to train the models.The types of supervised ML models include naïve Bayes (NB), decision tree (DT), logistic regression (LR), artificial neural network (ANN), and others [16].There are also several types of unsupervised ML models that seek to identify data clusters that have similar characteristics [17].The authors meticulously counted the type of methods used in each of the selected publications to gain insights into their approaches.Table 2 summarizes the dominant methods reported in the final batch of relevant articles reviewed, considering that some studies employed multiple methods.Figure 5 shows how the methods were distributed based on their frequency of appearance and the year of the publication.Figure 6 shows how the publications were distributed by year and split by the methods used.Look Once (YOLO).This method differs from the traditional two-step object detection techniques that first propose regions of an image and then classify their contents.Instead, YOLO frames object detection as a single regression problem straight from image pixels to bounding box coordinates for evaluating the probabilities of object presence.Four publications utilized surveys and reviews.The other publications used a variety of other statistical methods.Table 3 defines the acronyms for the parameters reported in Table 4, which provides some details about the architectures of the ANN model variants, the images analyzed, and the metrics of their effectiveness.Wavelet Scattering Networks (WSNs) N Jin (2021) [63] Case Studies N BNSF (2019) [64] Dropper Fault Recognition Network (DFRN) N Liu et al. ( 2019) [65] Based on the studies that disclosed details of the datasets used, they varied widely in size, source, and quality, reflecting the diversity of RS and ML applications in railway inspection.The majority of studies employed publicly available datasets, which included high-resolution images and sensor data from real-world railway environments, while others generated synthetic datasets to train and validate their models.A further assessment of the details of these datasets can influence the generalizability and reliability of the research outcomes.This diversity in data sources highlights the adaptability of RS and ML techniques across different contexts and conditions, enhancing the understanding of their potential limitations and strengths.
Most of the models used are a variant of ANNs as indicated Table 2.The entry "Y" and "N" in the "ANN" column indicates yes and no, respectively, if the model is a variant of ANN.A notable 20% of the methods used are outside the ANN domain.CNN methods emerged as the most frequently employed, featured in 11 different studies.Eleven articles utilized non-ANN methods, encompassing memory-based anomaly detection, big data analytics (BDA), and others.Six publications applied an ANN variant called You Only Look Once (YOLO).This method differs from the traditional two-step object detection techniques that first propose regions of an image and then classify their contents.Instead, YOLO frames object detection as a single regression problem straight from image pixels to bounding box coordinates for evaluating the probabilities of object presence.Four publications utilized surveys and reviews.The other publications used a variety of other statistical methods.Table 3 defines the acronyms for the parameters reported in Table 4, which provides some details about the architectures of the ANN model variants, the images analyzed, and the metrics of their effectiveness.

Discussion
Table 4 includes an outline of the parameters for each of the ANN methods used, showcasing the variation in the number and type of architectures across different approaches.CNN emerged as the dominant method utilized in the body of publications analyzed.CNNs distinguish themselves through a multi-layer, multi-resolution approach that loosely simulates the visual neural system of the human brain.Research has demonstrated that CNNs perform better than traditional ML techniques in machine vision applications [66].However, the use of CNN and traditional ML approaches vary based on the specific task [67].The following subsections expand on the methodological patterns observed, propose a classification of the railway condition monitoring applications reviewed, provide some case studies along with challenges encountered, explore future directions, and discuss the limitations of the study.

Methodological Patterns
This section examines the various models and algorithms documented in the reviewed literature, with a focus on their practical implications and limitations in real-world railway inspection scenarios.The effectiveness of each model, using CNNs, YOLO, or GANs, depended on its applicability in different railway monitoring conditions.Hence, they differed in specific strengths, such as high accuracy or real-time processing capabilities, and limitations, such as high computational demands or sensitivity to data quality.The authors noted several commonalities among the methods.Noteworthy machine vision models included versions of YOLO, Faster R-CNN, Mask R-CNN, ResNet, VGGNet, and others.Most of the reviewed studies utilized image datasets comprising hundreds or thousands of images for training and testing the deep learning models.Some studies incorporated video footage converted into images, with image resolutions varying from 256 × 256 pixels to 4096 × 2160 pixels.
Some studies discussed the effectiveness of transfer learning in enhancing accuracy by utilizing pre-trained models like ImageNet [52].Moreover, models underwent finetuning to set parameters such as learning rate, number of epochs, batch size, dropout rate, momentum, decay rate, and regularization techniques.Mean Average Precision (mAP) and accuracy emerged as the most common evaluation metrics, with some studies incorporating Intersection over Union (IoU) and F1-scores to assess performance.It is noteworthy that most models achieved high accuracy levels, surpassing 90%.Another notable trend in the reviewed methods was that both non-ANN and ANN-based methods required preprocessing steps, such as noise removal, contrast enhancement, and converting RGB to greyscale.These studies used statistical methods like edge detection, Haar transform, SVM, kNN, random forest, and frequency-resolved optical gating (FROG) to identify defects based on real-world image datasets.Some of the studies compared multiple methods.

Application Classification
Table 5 categorizes the publications based on their application objective.The category of track defect detection included subcategories such as assessing rail surface deterioration and detecting rail cracks.It is evident that a majority of the studies focused on defect detection on the rail tracks or their fasteners.

Challenges Reported
Some publications discussed difficulties with their chosen methods, while others grappled with issues related to the quality of the captured images.For instance, Hashmi et al. (2022) used three deep learning models: convolutional 1D (Conv1D), convolutional 2D (Conv2D), and long short-term memory (LSTM) [20].Notably, the training time for Conv2D exceeded the duration required by the other models.Sener et al. (2022) disclosed that one disadvantage of their approach was that it requires preprocessing to normalize the images before training the model [22].Wang et al. (2021) found that blurred images can severely decrease detection accuracy [74].Kangwei et al. (2019) reported that a general disadvantage of ML methods is oscillations in the training convergence, which require an optimized learning rate that the model cannot establish itself [53].Chandran et al. (2021) described the need for continuous calibration of the data collection system to compensate for distance variations between the sensor and the object [35].Zhang et al. (2022) emphasized the difficulty of pixel-level labeling of faults to train the network [68].2021)'s study was inspired by three backbone network structures [40].Therefore, the new models may lead to the development of more robust and adaptable systems capable of handling complex, real-world scenarios in railway inspection and beyond.
Regarding the efficiency and discriminative capabilities of different ML models, GANs can be more robust through the generation of high-quality synthetic data, enhancing model training where real-world datasets are incomplete or biased.While GANs are invaluable for data augmentation and simulation, their performance in direct defect detection tasks is not necessarily superior to task-specific models like CNNs or YOLO.These models, particularly CNNs and fully convolutional networks, leverage the spatial hierarchy in images, making them more efficient for the high-precision localization required in railway inspection.In contrast, YOLO's integrated approach to bounding box prediction and class probability makes it exceptionally fast and suitable for real-time applications, though sometimes at the cost of precision when compared to CNNs.
The objective of most of the reviewed publications was to ensure railway safety and reliability.Using RS-ML methods in railway inspection can enhance public safety by ensuring more reliable and frequent inspections, potentially reducing the risk of accidents.These methods also offer the potential for cost reductions in maintenance and repair, passing on the savings to both service providers and users [36].Moreover, the ability of these technologies to pinpoint defects more accurately and quickly than traditional methods could lead to longer-lasting infrastructure, reducing the environmental impact through the less frequent need for raw materials and construction activities [43].

Limitations
The authors acknowledge the limitations of this review, including the possibility that the chosen keywords may not have captured all the relevant publications.The dynamic nature of research means that new related studies might emerge during the writing and analysis phases of the present study.For instance, this work began in 2023, with 2022 being the previous full year for comparative analysis.Therefore, the authors encourage researchers in related fields to conduct further searches to ensure a comprehensive understanding.

Conclusions
This publication offers a thorough overview of previous research on RS-ML-based railway inspection, providing insights into the various methodologies used and their pros and cons for academics and practitioners.The study aids future researchers in fields such as computer science, engineering, and transportation in evaluating methods and finding relevant literature.The study analyzes research from different angles, including a descriptive analysis of publication trends by year, journal, country, and methods while examining the interconnections between studies through a visual map.The study also reviewed the applied methods in RS-ML-based railway inspection, categorizing them, detailing significant ANN studies, and focusing on their methodologies, datasets, and performance metrics.The review identified patterns in methodologies and discussed the benefits and drawbacks of RS-ML-based railway inspection.Most of the reviewed studies aimed to enhance safety, reduce costs, or improve railway inspection quality.Noting the experimental nature of most of the research, the authors propose future research avenues, such as investigating diverse geographical locations and the unique defects associated with them, like objects, vegetation, and rust.

18 Figure 2 .
Figure 2. The number of publications by year.3.3.Distribution by Journal Figure 3 provides a distribution view of the relevant publications across various journals.The set of 55 reviewed publications were distributed across 40 different journals.

Figure 2 .
Figure 2. The number of publications by year.

3. 3 .
Figure 3 provides a distribution view of the relevant publications across various journals.The set of 55 reviewed publications were distributed across 40 different journals.

Figure 2 .
Figure 2. The number of publications by year.

Figure 3
Figure 3 provides a distribution view of the relevant publications across various journals.The set of 55 reviewed publications were distributed across 40 different journals.

Figure 3 .
Figure 3. Proportion of publications by journal.IEEE Access and IEEE Sensors Journal contained most of the reviewed publications, with five appearing in each.Following closely were the journals Measurement and Sensors, each containing three of the reviewed publications.Additionally, Applied Sciences, IEEE Transactions on Intelligent Transportation Systems, and Neurocomputing each contained two of the reviewed publications.The remaining journals, collectively labeled "Others", contained one of the reviewed publications each.In summary, the diverse array of journals approached the topic from various perspectives, including computer science, engineering, and transportation.

Figure 3 .
Figure 3. Proportion of publications by journal.IEEE Access and IEEE Sensors Journal contained most of the reviewed publications, with five appearing in each.Following closely were the journals Measurement and Sensors, each containing three of the reviewed publications.Additionally, Applied Sciences, IEEE Transactions on Intelligent Transportation Systems, and Neurocomputing each contained two of the reviewed publications.The remaining journals, collectively labeled "Others", contained one of the reviewed publications each.In summary, the diverse array of journals approached the topic from various perspectives, including computer science, engineering, and transportation.

Figure 4 .
Figure 4.The number of publications by country.

Figure 4 .
Figure 4.The number of publications by country.

Figure 5 .
Figure 5. Distribution of methods by publication year.

Figure 5 .
Figure 5. Distribution of methods by publication year.

18 Figure 6 .
Figure 6.Distribution of publication year split by methods.

Figure 6 .
Figure 6.Distribution of publication year split by methods.

4. 4 .
Future Directions RS-ML-based railway inspection research pushes the boundaries of the current applied methods.It inspires researchers to develop new models and algorithms.For instance, Yang et al. (2022) modified a CNN model to work with fewer training images and to provide a more precise segmentation [75].Chen et al. (2022) improved the Mask R-CNN model [50].Chen et al. (

Table 1 .
Initial search phrases used to search Google Scholar.

Table 2 .
Methods used by each publication.

Table 3 .
Acronyms used in model development and image processing.

Table 4 .
Overview of parameters for the ANN variant methods.

Table 3 .
Acronyms used in model development and image processing.

Table 5 .
Application method categories.