A Multimodal Learning Approach for Protecting the Metro System of Medellin Colombia Against Corrupted User Traffic Data

Highlights

What are the main findings?

• The novel concepts of Self-Supervised Tabular Learning and Large Multimodal Models are integrated to create a multimodal learning solution for auditing the metro system of Medellin, Colombia.
• On publicly available data, in an offline process, corrupted user traffic is detected, explained, and corroborated using SHAP values and the image understanding process of a Large Multimodal Model.

What are the implication of the main findings?

• A visibility layer is added for smart proper policy making, also shedding light on the pros and opportunities of the current publicly available data.
• Each abnormal passenger behavior is not only flagged, but a thorough justification is also provided to enhance the robustness of the detections.

Abstract

A critical task in infrastructure security is to model user traffic in transportation systems to alert whenever anomalous behavior is observed. Discerning those abnormal samples is possible by auditing the available data, which then enables proper policy making to guarantee fair tariffs and the design of strategies to tackle problems such as passenger congestion. In this paper, we present an offline cybersecurity approach for the multimodal modeling of user traffic for the Colombian metro. To identify the anomalies, we design custom Deep Autoencoders based on the embeddings produced by the Self-Supervised Learning TabNet architecture. Additionally, we provide explainability through a SHAP-based component and the analysis of external image data using LLaVA as the selected Large Multimodal Model. The results indicate that most problems that occur on one metro line also affect the other, demonstrating the interconnectivity of the metro system, a crucial aspect that motivates the coordinated emergency response to improve the passenger travel experience. Although the detected problems might already have been identified and reported on social media, the transparency provided helps create confidence when an abnormality is observed, and in case there is no backup information on our official external data sources, it represents an alert to examine it more deeply, becoming an intelligent assessment tool for the metro. This article also sheds light on the potential of the publicly available dataset used and the importance of expanding its existing variables and information.

1. Introduction

Infrastructure security is a relevant research area for protecting the goods and services of modern societies, ranging from Smart Grid systems [1] to transportation technology. Transportation encompasses not only the protection of the in-vehicle networks for self-driving cars [2] but also the security of bus and metro systems, including their several routes, the movement of people, and the monitoring of user traffic behavior [3].

As an example, the Metropolitan Transportation Authority of New York, in the United States, provides information on the use of the metro system in the city, including updated data on the ridership between stations, allowing policy making for the expansion of the metro with new wagons, the discussion of stopping fare evaders, as well as the impact of congestion pricing [4,5,6].

In this paper, as part of an effort to replicate in Latin America successful initiatives undertaken in cities such as New York to modernize transportation systems, we employ recently released public data from the Colombian metro [7]. Although passenger fares are fixed, regardless of the number of traveled stations [8], by using the information on the hourly number of users in each metro line, we can provide intelligence on policy making to reduce commute times by allocating wagons when necessary, while also laying foundational work toward a dynamic tariff policy with fairer prices suitable for heterogeneous demands across stations. Transportation-related policies are vast in scope, so we are targeting those where the primary objective is traffic management and transportation safety [9], based on the results and observations obtained from this public dataset.

To achieve this visibility level for the Colombian metro, we design a smart cybersecurity solution to flag and automatically explain any abnormal behavior detected. These anomalies can be seen as corrupted data continuously fed into the currently available control dashboards [10], for example, due to extreme weather conditions, repairs, social events, or early closures that affect the metro lines.

Given that the available information is provided in tabular format, and motivated by the recent advances in tabular learning and Self-Supervised Learning (S-SL) [11], we propose two Deep Autoencoders based on the embeddings produced by the TabNet architecture [12]. S-SL is a novel training strategy that enables the construction of robust encoders from unlabeled data, extracting patterns that capture the general context of the problem, followed by a fine-tuning stage that only requires a small amount of labeled instances [13]. This paradigm is particularly valuable in the cybersecurity domain, where challenges evolve rapidly, and the scarcity of labeled data is a well-documented limitation [14].

Moreover, building on the original SHAP methodology for the interpretability of supervised learning models [15], we provide SHAP-based explainability for our S-SL setting [16]. Lastly, by using the LLaVA framework [17] for image understanding, as a specialized assistant based on Large Multimodal Models (LMMs) [18], we enhance our results by providing additional information regarding the potential causes of the anomalies. These results are achieved through the analysis of posts from the official X account of the Colombian metro (@metrodemedellin. https://x.com/metrodemedellin, accessed on 28 July 2025) and the local government weather forecast account (@siatamedellin. https://x.com/siatamedellin, accessed on 28 July 2025).

Although our Artificial Intelligence (AI) solution works offline and the detected problems might already have been identified and reported on social media, the transparency provided in our detection process, reinforced with external data, increases confidence when an anomaly is identified. Furthermore, in case there is no information on our official external data sources, our system provides an alert to carry out further investigation, serving as an intelligent assessment tool for the metro.

The public data available in [10] also includes buses, trams, and cable cars that belong to the transportation system. However, the analysis presented here is focused on the metro because it carries the largest share of the daily commute, with more than 650,000 passengers on average, across lines A and B in the city of Medellin, Antioquia. Regarding the operational scale in peak hours, such as 5 pm, it can reach over 100 thousand people traveling in both lines. Each user record is registered using the Automatic Fare Collection (AFC) system of the Cívica card [19]. Nonetheless, an important limitation in the dataset is that no information regarding passenger destination is provided, nor can it be inferred, since data about fare price and traversed stations are not recorded.

Therefore, the main contributions of this work are summarized as follows:

• A multimodal AI solution that functions as an offline data inspector for the control dashboards of the Colombian metro. In late information fusion [20], the system integrates posts from X representing external influencing factors, with numerical user traffic data. This offline inspection serves as a visibility tool to flag and to explain potential anomalies that may have been overlooked and that require further analysis.
• The design of Deep Autoencoders using the embeddings produced by the S-SL TabNet architecture, in tandem with the calculation of SHAP values for unsupervised learning, while also exploring the use of LMMs such as LLaVA to provide additional information for finding the reason behind each of the detected anomalies.
• We are closing the gap between the integration of novel AI approaches such as multimodal learning, S-SL, and LMMs for intelligent metro systems.

The rest of this document is organized as follows: Section 2 shows the related literature on the integration of AI for modeling passenger flow patterns in modern metro systems; Section 3 introduces our proposed data inspection solution, explaining the user traffic data from the Colombian metro, the proposed Deep Autoencoders based on S-SL, the integration of SHAP values for interpretability, and how the LLaVA framework works. Section 4 shows the results obtained and a discussion of them. Lastly, Section 5 provides our final observations and presents future work ideas.

2. Related Work

This section reviews the state of the art on AI integration in modern metro systems to model passenger flow patterns. To the best of our knowledge, this study represents the first effort in the literature to use passenger flow data from the Colombian metro and conduct AI experiments on it. See

Table 1. Comparison between this work and the related state of the art about integrating AI in modern metro systems for passenger flow modeling.

	Self-Supervised Learning	Explainable AI	Large Multimodal Models	External Data Sources
Wang et al., 2019 [21]	✗	✗	✗	✔
Wang et al., 2021 [22]	✗	✗	✗	✗
Zhang et al., 2021 [23]	✗	✔	✗	✔
Wei et al., 2023 [24]	✗	✗	✗	✔
Bapaume et al., 2023 [25]	✔	✔	✗	✔
Wu et al., 2025 [3]	✗	✔	✗	✔
Zhang et al., 2025 [26]	✗	✗	✗	✔
This work	✔	✔	✔	✔

for a summary of the listed documents and the key differences compared to this work.

Wang et al. [21] proposed a time series approach with a dynamic time window technique to predict abnormal passenger inflow in the Shanghai metro system. The dataset comprised over 200 million activity records from 1 million users during April 2015, covering 288 stations. Each record consisted of information about the paid fare, which helped infer the user’s destination. The authors concluded that the importance of adding this latter feature is useful for smart traffic congestion control. Regarding explainability, they crawled three different websites, gathering event information such as sports and concerts, which later helped justify the abnormalities detected. Although a comprehensive regression analysis for time series forecasting was conducted, there is room for improvement by exploring anomaly detection from a classification perspective, specifically using the novel S-SL training strategy.

Also implementing windowing for real-time detection in time series, Wei et al. [24], working with data from the Beijing metro, built an unsupervised learning model based on Principal Component Analysis and 2D matrices to detect anomalies in passenger flow patterns. These matrices indicate the flow (inflow or outflow) observed per station during a specific time interval (days or weeks). The proposed algorithm decomposes each original observation matrix into a low-rank normal matrix and a sparse abnormal matrix. This solution relied on external social media data (such as festivals) to explain the detected anomalies. However, this strategy, i.e., only distributing traffic into matrices, does not scale to incorporate equally important features such as the holiday and weekend variables. This limitation hinders the implementation of an explainability method for quantitatively analyzing how additional variables weigh in the final predictions.

Wu et al. [3] worked with data from the Hangzhou metro system to predict passenger inflow and outflow across 80 different stations, with 70 million records collected during January 2019. The proposed model was a hybrid, including K-means clustering and XGBoost. The purpose of using K-means was to group metro stations with similar passenger flow patterns, thus facilitating the training of the n XGBoost models corresponding to the n station clusters. Interpretability was achieved using SHAP values and Accumulated Local Effects [27], revealing, for example, that the passenger volumes are higher on weekdays than on weekends, and that the presence of business facilities near the metro stations has a significant impact. In addition, open-source information, such as points of interest, temperature, and weather, was also considered a key factor for irregular travel patterns. Lastly, the holiday variable was not evaluated in this study, overlooking the relevance of this feature for traffic prediction.

In [22], the concept of hypergraph learning is introduced to model the Origin-Destination (OD) traffic flow in metro systems. Hypergraphs are an extension of conventional graphs, where the edges between vertices represent not only the topological connection between stations but also higher-order information such as the number of vehicles, operating time, and the time travel interval. This way, spatio-temporal relationships are modeled to get a stronger flow prediction. The authors used the same Hangzhou dataset as the work in [3], and additional information from the Beijing metro system. The latter was collected in 2015, covering a network of 327 stations and 22 lines. The hour, day, and week granularities are considered with a different hypergraph for each one, fusing the output for node-level prediction. Nevertheless, this architecture could benefit from external crawled information indicating real-time social events (e.g., concerts and festivals) in each city.

Zhang et al. [26], based on passenger flow patterns, analyzed the impact of user traffic and other factors, such as the wagons’ carrying capacity and travel time between stations, to create a Deep Reinforcement Learning (DRL) solution to reduce delays. Working with data from the Yizhuang line of the Beijing metro, which covers 13 stations, a DRL agent is built to handle the complexity of regulating 20 different trains during the morning peak (i.e., from 6:30 am to 9 am). When comparing the regulations provided by this DRL solution, the departure deviations are diminished, solving the domino effect that causes the delay in one of the trains, and therefore, the resulting congestion of passengers and poor quality of service. Although more than 25 factors are evaluated, from the physical characteristics of the train to passenger flow patterns and time information, no interpretability method was used to distinguish the main variables, harming the auditing of the decisions behind each control action.

The works discussed thus far focus on traffic modeling applications in China, but in the state of the art, there are other relevant advances in cities such as London and Paris. In the United Kingdom, Zhang et al. [23] proposed a fuzzy logic system with data clustering, to predict passenger flow in different underground trains in the London metro, using as training data the one-year information from the Victoria line, including the variables of weather conditions, external events (e.g., a football match or a concert), day of the week (i.e., weekday or weekend), etc. Regarding France, Bapaume et al. [25] focused on the Paris metro line 9, using three years of information for the task of passenger flow forecasting, training Vision Transformers [28] with synthetic images that indicate external information such as strikes and sports events occurring near stations.

Based on this literature review, passenger flow modeling in major cities such as Shanghai and Beijing highlights the importance of incorporating outflow data when analyzing abnormal behaviors. Furthermore, the chosen data source is critical, as external factors such as social events and weather conditions play an essential role in conducting a comprehensive study and justifying the detected anomalies. The reviewed studies employ a variety of AI techniques, ranging from standard XGBoost models to hypergraph theory, DRL, and transformers, to extract the intelligence obtained from these data. Nonetheless, to the best of our knowledge, we are the first effort to integrate the novel S-SL training paradigm for building Deep Autoencoders based on numerical user traffic data, while incorporating weekend and holiday variables, along with the calculation of SHAP-based values for interpretability, and the use of LMMs for image understanding of X posts from the Colombian metro and local weather forecasting accounts.

In the next section, we present details about the methodology followed to build the proposed multimodal learning cybersecurity solution.

3. Methodology

In this section, we present our methodology. It includes the analysis of the Colombian metro dataset to define the relevant features, the design of the Deep Autoencoders architectures based on TabNet, the calculation of the most salient features based on SHAP values, and finally, the integration of LMMs to strengthen the justification of each abnormal sample. Please refer to Figure 1 for an overview of these steps during the offline inference stage.

In Figure 1, two independent models work together, namely a custom unimodal Deep Autoencoder and LLaVA, where the outputs of each are then concatenated for the final detection. The output of the Deep Autoencoder is the datetime with abnormal inflow traffic behavior, and the output of the LMM is the potential X post for that case, exemplifying this as a late multimodal fusion technique [20] that enhances the detection results.

3.1. User Traffic Data from the Colombian Metro

The only metro system in Colombia is located in the city of Medellin, Antioquia. Public data are provided, indicating the movement of people at the hour granularity level, between 4 am and 11 pm, in each of the two existing lines: A and B [7]. This information spans from 1 January 2019 until 31 March 2025 and is gathered from the entry registration of a passenger using the Cívica Card. In total, there are 2278 records per metro line.

According to the last 100 records in these data, on average, every day more than 650,000 people use the metro, making it the transportation system with the highest volume of users when compared to buses, trams, and cable cars [10]. See Figure 2 for the distribution of users, where line A moves the majority of people. Line A comprises 21 stations, from south to north, becoming the backbone of the metro system. On the other hand, line B includes seven stations, expanding perpendicularly to line A. See Figure 3 for the location of these stations in the city of Medellin.

From the more than 2000 records available per line, we selected those dated after the official lifting of the COVID-19 restrictions in Colombia (i.e., 30 June 2022). According to [29], the pandemic drastically disrupted passenger mobility in the metro system and led to serious economic losses. Therefore, in order to model this user traffic behavior with Deep Autoencoders, the final dataset for each line included 1005 records distributed across:

• 731 samples for training: 30 June 2022–30 June 2024.
• 184 for validation: 1 July 2024–31 December 2024.
• 90 for testing: 1 January 2025–31 March 2025.

Based on the original features indicating the date, hour, and total number of passengers, we created two additional variables to determine whether the corresponding day is a holiday and/or a weekend. Since passenger mobility varies between weekdays and important celebration days in the city, the goal of these two additional columns is to handle these cases. In particular, there can exist 4 scenarios:

• It is a weekday, but it is a holiday.
• It is a weekday, but it is not a holiday.
• It is a weekend, but it is a holiday.
• It is a weekend, but it is not a holiday.

Ten different holidays were considered, including New Year, Independence Day, Labor Day, and religious celebrations such as Christmas and Holy Week. To handle missing values, cells were filled with 0, this was considered the best practice according to the Total number of passengers column. This column and the Day variable were removed after preprocessing for our final feature set. See

Table 2. Description of the 22 features extracted from the public Colombian metro dataset.

Feature	Description
Date information
Is weekend	Boolean numerical value to indicate if the day is a weekday (0) or weekend (1).
Is holiday	Boolean numerical value to indicate if the day is a holiday (1) in the country or not (0).
Time information
04:00:00, 05:00:00,…, 23:00:00	Granularity at the hour level. The normal operation time of the metro spans from 4 am to 11 pm.

for a breakdown of the final group of variables used in this work, which covers date and time information.

Building on prior work with Autoencoders for reconstruction-based anomaly detection [30] and the use of S-SL in classification problems in cybersecurity [31,32], we categorized the numerical user traffic in the metro dataset into four different classes. The categories per hour are low, usual, high, and outlier inflow traffic, and are calculated according to the Equations (1)–(3), where the outlier category occurs when a number is too small or too high to fit into one of the other three classes. The variables $q1$ and $q3$ indicate the first and third quartiles, respectively, while $whisker\text{\_}high$ and $whisker\text{\_}lower$ are the upper and lower extreme values, excluding outliers.

$$low = (whisker\text{\_}lower, q1 - 1)$$

(1)

$$usual = (q1, q3 - 1)$$

(2)

$$high = (q3, whisker\text{\_}high)$$

(3)

For training, these four categories were encoded as: {low: 0, usual: 1, high: 2, outlier: −1}. This way, the $high$ class is assigned a higher order than the $low$ class to reflect the intended ordinal encoding. See Figure 4 for the number of occurrences of each of these passenger inflow categories per metro line. In line A, $outlier$ traffic accounts for 6.1% of the total cells, while in line B it accounts for 5.69%. Therefore, instead of removing this class to model a cleaner behavior per line, we consider it valuable information to model genuine deviations from normal traffic patterns. As expected, most of the traffic (i.e., over 90% per line) belongs to the other three categories, thus validating our strategy for categorizing the original numerical data.

3.2. Deep Autoencoders Based on Self-Supervised Learning

TabNet, short for Tabular Network [12], is an S-SL model in which the architecture follows an autoencoder. The encoder consists of multiple sequential steps (blocks) incorporating both feature and attentive transformers, which, together with the sparsemax activation function [33], enable the model to focus on the most relevant features for decision-making. Sparsemax works by zeroing out some of these variables, filtering large output spaces.

Each feature transformer includes fully connected layers, batch normalization, and the Gated Linear Unit activation function [34]. Conversely, each attentive transformer includes a fully connected layer, batch normalization, and the sparsemax masking function. We propose to extract the embeddings (latent space z) produced by the TabNet encoder and pass this encoded representation into a set of fully connected layers, increasing in dimension, to reconstruct the input data. See Figure 5 for the proposed Deep Autoencoder architecture used in each metro line.

Relying on the assumption that the self-attention mechanism in TabNet is sufficiently robust to produce high-quality feature representations, we decouple the original multi-step decoder with feature transformers and integrate our proposed block of fully connected layers. Since our initial goal was to train a robust reconstructor capable of discerning abnormalities in the user traffic behavior, the encoder remained frozen during the training stage, only optimizing the linear decoder. This decoder included ReLU activation functions and started from an embedding dimension of eight, followed by linear layers of 16, 32, 64, and 128 neurons. See

Table 3. Hyperparameter values selected for the extracted encoders from the original TabNet architecture.

Variable	Selected Value
Batch size	32
Virtual batch size	32
Maximum learning rate	5 × ${10}^{-5}$
Minimum learning rate	1 × ${10}^{-5}$
Number of blocks (steps)	3
Number of shared layers	2
Number of independent layers	2
Momentum	0.3
Embedding dimension	8
Feature re-usage ($\gamma$)	2
Dimension of the attention layer	8

for the hyperparameter values manually selected through experimentation for the encoders of our Deep Autoencoders. The architecture remains identical across metro lines, with variations only from the input data.

Early stopping was conducted by evaluating the validation loss. Furthermore, cosine annealing was selected as the learning rate scheduler, and Adam was used as the optimizer. The source code provided in the pytorch-tabnet GitHub repository (pytorch-tabnet 4.1.0. https://dreamquark-ai.github.io/tabnet, accessed on 20 August 2025) was analyzed to only extract the encoder part and produce the required embeddings. This whole implementation, also considering the linear decoder part, is coded using PyTorch v2.6.0 [35].

Reconstruction Threshold ($ϵ$)

To calculate the reconstruction threshold ($ϵ$) that works as a delimiter between normal and abnormal days, the Equation (4) was defined, using the mean squared error as the criterion to measure the quality of the reconstructions in the validation set. A value of k equal to one was established, allowing for more sensitivity in the anomaly detection task. Although the 99th percentile of the distribution of the mean squared errors can also be used as the delimiter, we did not test this approach because, in our methodology, it is not relevant if a large number of abnormalities are flagged in the inflow traffic, since each of these detected instances can then be filtered with the help of the explanations provided by our LMM.

$$ϵ=mean\left(mse\right) + k\ast std\left(mse\right)$$

(4)

No other metrics were used to measure the detection rate performance, such as precision or F1 score. The reason is that these custom Deep Autoencoders enable an unsupervised exploration phase within the large amount of posts in the two provided X accounts. Therefore, there is no ground-truth set of abnormal (corrupted) samples from which to benchmark the detection results. Besides the $ϵ$ value and explainability, overfitting was properly evaluated using early stopping and the validation set to find the optimal number of training iterations.

3.3. SHAP Values for Deep Autoencoders

SHAP is a game theory approach that provides feature importance values in supervised black box models [15]. More recently, an adaptation of this algorithm was proposed for Autoencoders [16], which we transfer to our case. Specifically, our Deep Autoencoders, during training and inference, are fed with the 22 variables defined in Table 2, where each record represents a different day. Therefore, once an anomaly is detected, the advantage of having the original granularity at the hour level is lost because anomalies are just days with abnormal traffic behavior. To handle this issue, we need to provide interpretability to the model, finding those features (e.g., hours) that contribute more to the high reconstruction error of the detected anomaly.

The approach mentioned in [16] is based on two important matrices: $X\text{\_}train$ and $X\text{\_}explain$. $X\text{\_}train$ is used to build the local explanation baseline model to calculate the SHAP values, and its size (i.e., the number of background samples) is defined by a variable called $background\text{\_}set$; $X\text{\_}explain$ includes the anomalies that need explanation, where the top abnormal samples are selected by mean squared error, limited by the variable $num\text{\_}anomalies\text{\_}to\text{\_}explain$.

For the Colombian metro, $X\text{\_}explain$ is the set of anomalies detected in each line (A or B), with no limit on the variable $num\text{\_}anomalies\text{\_}to\text{\_}explain$, since interpretability is required for each of the abnormal samples. Then, for each row in $X\text{\_}explain$, the top features are extracted according to how much percentage of the total reconstruction error of that specific record we want to explain, with a value of 100% for this scenario. Lastly, the top variables by SHAP value, explaining these most important features, are listed. Regarding $X\text{\_}train$, it was built with a shuffle of samples, specifically a $background\text{\_}set$ of 365 instances (one year).

3.4. Large Multimodal Models: LLaVA Framework

LMMs are augmented Large Language Models (LLMs) that handle not only text but also other input data modalities such as images, videos, and audio [36]. Some LMMs are intended for image understanding, following the Equation (5), where a vision encoder works in tandem with an LLM backbone [37].

$$Image+Text\Rightarrow Text$$

(5)

LLaVA (Large Language and Vision Assistant) is a state-of-the-art framework for image understanding [38]. Specifically, we use the checkpoint of 13 billion parameters (13B) [17], and provide as input a text-image pair, where the image passes through a CLIP [39] vision encoder, then through a Multi-layer Perceptron model acting as the vision-language connector, and finally through the Vicuna [40] backbone as the language model, which also handles the input user prompt after tokenization. The open-source version of LLaVA 13B was obtained from Ollama (Ollama platform: https://ollama.com/, accessed on 25 August 2025).

In the proposed solution, LLaVA is used at inference time to analyze posts from the official X account of the Colombian metro and the weather account of the local government, to identify potential causes behind each detected anomaly. Each input text-image pair consists of a screenshot of a post and a user prompt, as in Figure 1. LLaVA provides an additional layer of explainability, complementing the SHAP-based interpretability approach applied to the Deep Autoencoders.

4. Experimental Results and Discussion

In this section, we present the classification and explainability results obtained from the proposed multimodal learning approach.

4.1. Anomalies Detected by the Deep Autoencoders

Given the limited number of 731 training records, an important challenge is the potential risk of overfitting. Therefore, to mitigate this issue, we performed early stopping with a patience value of 15 to measure the improvement in the validation loss. Moreover, to help navigate the complex solution space that this passenger inflow problem may represent, we modified the learning rate by using cosine annealing. See Figure 6 for the corresponding plots calculated during this training process, for lines A and B.

As depicted, convergence was achieved in a similar number of epochs, 29 for line A and 34 for line B, and benefited from a small learning rate, close to 4 × ${10}^{-5}$. This indicates that the Deep Autoencoders, without too many iterations, achieved confident predictions, potentially due to well-defined patterns learned from the feature set defined in Table 2 and the four scenarios listed in Section 3. These four cases were obtained due to the Is weekend and Is holiday variables created.

Using the best model from the best epoch in each line, the reconstruction threshold $ϵ$ was calculated based on the validation set. See Figure 7 for the corresponding histograms, where the reconstruction errors from the testing samples are plotted, with the red dashed line separating the normal instances from the detected anomalies.

As expected, in both histograms, the high frequency and high number of bins are on the left side of the reconstruction threshold, since the anomalies are not abundant. From each test set of 90 different samples per metro line, each Deep Autoencoder detected six anomalies. Remarkably, as indicated in

Table 4. Days detected as anomalies by the different Deep Autoencoders. 67% of the occurrences in one metro line also exist in the other, proving their interconnectivity. These days in common are in green color.

Line A	Line B
2 January 2025	1 January 2025
3 January 2025	2 January 2025
7 January 2025	3 January 2025
9 January 2025	7 January 2025
5 March 2025	10 January 2025
24 March 2025	24 March 2025

, 67% of the anomalies detected in each line also occur in the other, demonstrating the interconnectivity of the Colombian metro, where problems in one line also affect the other. This is a crucial aspect that motivates the coordinated emergency response to improve the passenger travel experience [41].

4.2. SHAP-Based Explainability and Image Understanding Using LLaVA

To investigate each abnormal day further, SHAP values are calculated following the algorithm explained in Section 3. To assess the magnitude, rather than the direction, of each feature’s contribution to the high reconstruction errors associated with anomalies, we use absolute SHAP values. Two cases are selected to evaluate the proposed multimodal learning approach: first, the analysis of an abnormal day not detected by both Deep Autoencoders and then the analysis of an abnormal day detected by both.

From the isolated events in Table 4, the most recent anomaly occurred on 5 March 2025, in line A. The top five absolute SHAP values for this event are in Figure 8, indicating that a variation in passenger inflow traffic was notoriously observed, for example, between 5 and 6 pm. Taking into account these five main characteristics (all hours), we filtered the X posts of the official metro account and the weather account of the local government for that day, to then use LLaVA to perform the image understanding task on those posts. See Figure 9 to observe the results of this final proposed step.

From Figure 9, we can read that a traffic accident could affect line A of the metro system at 1 pm. This hour is exactly in the ranking provided in Figure 8, as one of the most important variables for explaining the high reconstruction error of that day. Congestion could have occurred at the station on line A, where the pink bus route should start, considerably increasing user traffic during that time, probably because passengers decided to move more stations down line A when realizing the pink line was interrupted.

For anomalies detected in both lines, the first abnormal case was selected, corresponding to 2 January 2025. See Figure 10 for the absolute SHAP values that explain the hours when more abnormal behavior was observed during that day. From both rankings, 4 am and 12 pm are the times in common. Therefore, the intuition is that something relevant occurred at those hours. See Figure 11 for the identified X post that could explain the anomaly.

In Figure 11, the X post states moderate rainfall in the city of Medellin at 4 am. The formal public transport system in Medellin includes buses that serve the metro lines [42], so the weather conditions that morning might have caused those bus routes to get stuck in traffic, reducing the number of people in the metro stations. Regarding the other common time in both lines (i.e., 12 pm), no post was found in the two X accounts for that abnormal date, thus, an additional external data source would be beneficial, or we have just detected an overlooked problem like the malfunction in the AFC readers of one or some stations at that particular hour.

These two selected cases show the potential of the created approach for offline data auditing processes in both lines of the Colombian metro system. However, there are some important issues to discuss, and these are critical to attend to if we pursue a robust deployment of this solution.

4.3. Limitations of the Proposed Solution

Although the proposed multimodal learning approach, using numerical data from the Colombian metro, and images from two specific X accounts, shows its usefulness in identifying and explaining days with abnormal user traffic, the publicly available dataset used [7], is not as comprehensive as others, such as the commented cases of major cities like New York, Beijing, Shanghai, and Hangzhou, explained in Section 2. In particular, there is a lack of information on which stations the users go to and what fare they pay for the routes they follow. This OD data is crucial and has been leveraged before to measure efficiency in the travel times between metro stations [43]. Adding this OD pair will require expanding the available date information with another category to handle more patterns beyond weekdays and holidays.

Another issue is how we collect the posts from the X accounts of the Colombian metro and the local weather forecast. Currently, we conduct a manual research process using the X advanced search feature [44], filtering by the date and hours of the detected anomaly. But we are limited by the information shared on that social network. Therefore, if we partner with the company responsible for the management of the Colombian metro system, we could have an external data source directly connected to our AI approach, allowing better and faster retrieval of potential events affecting user traffic.

Furthermore, LLaVA is susceptible to hallucination. Hallucination is a recent concept used for foundation models, and can be defined as the generation of answers that may sound coherent, but the model’s reasoning is incorrect [45]. The answers provided in Figure 9 and Figure 11 are good enough. However, LLaVA is not yet a specialist in the transportation system of Medellin. Therefore, we propose to conduct in the future a context augmentation process by providing LLaVA with a corpus of different bus, tram, and cable car lines connected to the metro of Medellin, indicating how many lines exist, how they operate, in what directions, at what times, etc.

5. Conclusions and Future Work

Modeling user traffic behavior in metro systems is crucial as a visibility strategy for smart policy making and the protection against corrupted (abnormal) passenger patterns. In this paper, a solution to this area of infrastructure security was presented with a multimodal learning cybersecurity approach for the offline auditing of user traffic in the Colombian metro. We strongly believe that this solution represents a step toward defining a dynamic and fair pricing strategy for different stations according to their demand, and we also shed light on the importance of having destination data to better predict passenger congestion problems.

To the best of our knowledge, we are the first in the literature to use the publicly available data of the Colombian metro system. Furthermore, this solution is outstanding in the state of the art because we explore the S-SL TabNet architecture for producing high-quality embeddings, as well as two different methods of explainability based on absolute SHAP values for custom Deep Autoencoders, and the image understanding process of external data sources using LLaVA.

The proposed data categorization algorithm and the proper training of our custom Deep Autoencoder architectures produced results that demonstrate the interconnectivity of this metro, where the majority of the anomalies detected on one line also occur on the other, motivating a coordinated response for any emergency. Moreover, our classification task not only flagged those days with abnormal passenger inflow behavior, but also carefully explained the hours and detected the external events that could be responsible for those high reconstruction errors.

This study opens academic questions around S-SL and multimodal learning, like: what is the most suitable early fusion technique to model numerical, text, and image data in the pre-training phase of S-SL algorithms? By achieving this, the quality of the embeddings produced by the TabNet architecture will be highly improved, as they will not solely rely on numerical traffic data.

As future work, we plan to evaluate the proposed multimodal learning solution using data from other metro systems, providing a robust benchmark that considers not only passenger inflow patterns but also destination data and information at the station granularity level, as shared by some cities in China and the United States. Another direction is the online deployment of this system for real-time anomaly detection, which will involve modifying the hourly rate of user traffic to collect this information every couple of minutes instead.

We hope that this article serves as a valuable resource, for example, to the company responsible for the management of the Colombian metro, since we demonstrated the potential of their publicly available dataset and the importance of expanding its existing variables and information.