Leveraging Machine Learning for Optimal Pilgrim Crowd Management

Abstract

The Hajj pilgrimage involves high crowd density within limited time and space, making traditional crowd control methods insufficient for real-time alerts or predictive safety measures. This research proposes a machine learning-based system to enhance crowd management by detecting abnormal behavior and forecasting future conditions. The study utilizes the Hajjv2 dataset, which consists of annotated video frames capturing various crowd behaviors across multiple Hajj locations. After data preprocessing and feature extraction, including crowd density, speed, direction, and object area, two models are employed: the Isolation Forest algorithm for anomaly detection and a Long Short-Term Memory (LSTM) neural network for forecasting crowd behavior. The system integrates the results of both models to issue real-time alerts based on predefined thresholds. Evaluation results indicate that the Isolation Forest model achieved an average accuracy of 91% across all test sets, effectively identifying abnormal movement patterns. The LSTM model produced reliable predictions of average crowd speed with a low Mean Squared Error (MSE) of 0.000439. Together, these models form a robust alert mechanism that enables early identification of risks. In summary, this study presents an intelligent, scalable solution for enhancing crowd safety during the Hajj. It illustrates the practical value of machine learning in enabling proactive and informed crowd management strategies.

1. Introduction

The Hajj pilgrimage, one of the five pillars of Islam, is among the world’s largest recurring religious events, attracting millions of Muslims annually to Mecca, Saudi Arabia. Pilgrims gather within constrained areas and timeframes to perform a series of rituals, leading to extreme crowd densities. For example, on the Day of Arafat alone, over one million individuals congregate in a single location [1]. This mass movement creates unique challenges for public safety, emergency response, and logistical coordination.

Traditional crowd management strategies, such as deploying personnel, expanding infrastructure, and using surveillance systems, have served to mitigate risks during Hajj. However, these methods often fall short in providing real-time alerts or proactive measures to prevent critical incidents [2,3]. This shortcoming underscores the need for intelligent systems that can adapt to dynamic crowd behavior and help prevent disasters before they occur.

In recent years, advancements in artificial intelligence, particularly machine learning (ML) and deep learning (DL), have opened new avenues for crowd analysis and management [4,5,6]. Techniques such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) are increasingly applied to video surveillance data to estimate crowd density, identify anomalous behavior, and forecast movement patterns. These technologies have shown promising results in mass events worldwide, including the Hajj [7,8,9,10].

Despite this progress, several critical gaps remain. Many existing solutions lack predictive capabilities or real-time alerting systems. Furthermore, high-density and occluded scenes continue to challenge detection models, and the diversity of crowd behavior is often underrepresented in public datasets. These issues limit the effectiveness and generalizability of current approaches.

Many existing systems either detect current anomalies or estimate crowd density but do not offer both real-time anomaly detection and forecasting in one framework. Our integrated approach uniquely combines these capabilities, enabling both reactive and predictive alerts tailored for high-density, time-sensitive scenarios such as the Hajj.

This study aims to address these gaps by developing a machine learning-based system tailored for optimal crowd management during the Hajj. Specifically, it introduces an integrated framework that uses the Isolation Forest algorithm for anomaly detection and a Long Short-Term Memory (LSTM) neural network for forecasting crowd behavior. The system is evaluated using the Hajjv2 dataset, which includes annotated video data capturing real crowd dynamics during Hajj rituals.

The principal findings indicate that the proposed approach can effectively detect abnormal crowd patterns and anticipate future crowd conditions with high accuracy. This system provides a scalable and intelligent foundation for proactive crowd management during large-scale religious gatherings.

2. Materials and Methods

This study proposes a machine learning-based system for proactive crowd management during the Hajj pilgrimage. The methodology involves dataset acquisition, preprocessing, feature engineering, anomaly detection, forecasting, and alert generation. All scripts and code used in the analysis will be made available upon request. No restrictions apply to data or code access. This study did not involve human or animal subjects and therefore did not require ethical approval. No generative AI tools were used for content creation, analysis, or design; only standard development libraries and platforms were utilized.

An overview of the proposed system pipeline is illustrated in Figure 1, detailing the sequence from data acquisition to real-time alert generation.

2.1. Dataset Description

The research utilizes the Hajjv2 dataset, a structured dataset specifically designed for crowd behavior analysis during the Hajj pilgrimage. It contains annotated video frames captured at key ritual sites and includes labels for anomalous behavior such as standing, running, sitting, and moving in the opposite direction [5]. The dataset comprises 18 videos, evenly split into 9 training and 9 testing videos, each identified by a unique video ID. In total, it includes over 300,000 object instances.

Each frame includes features such as coordinates (X, Y), object size (Width, Height), movement class, and binary anomaly labels. The diversity and density captured across various Hajj locations make this dataset suitable for developing both anomaly detection and forecasting models.

2.2. Data Preprocessing

To ensure data quality and model readiness, the dataset underwent several preprocessing steps:

• Data Cleaning: Frames with missing or inconsistent entries (e.g., null values in object coordinates or labels) were removed. Only relevant classes, such as “moving in opposite” or “standing,” were retained to reduce noise.
• Normalization: Min-Max scaling was applied to positional and dimensional features (X, Y, Width, Height), bringing all values into a 0–1 range to ensure uniform input for the models [11].
• Feature Engineering: The following features were derived:

  ○

  Crowd Density: Estimated by dividing the number of detected objects by the area of the frame [11].

  ○

  Object Speed: Computed using the Euclidean distance between consecutive frame positions [12].

  ○

  Movement Angle: Calculated using the arctangent of the change in Y and X positions to determine direction [13].

  ○

  Object Area: Derived by multiplying bounding box width and height to indicate object scale, aiding in distinguishing between individuals and non-pedestrian elements [14].

2.3. Anomaly Detection

The anomaly detection model is based on the Isolation Forest algorithm, a tree-based ensemble method well-suited for high-dimensional, unlabeled datasets. It works by isolating observations through recursive random splits and is particularly effective in identifying sparse or abnormal data points [15].

The model was trained using normalized features—Area, Crowd Density, Speed, and Angle—and evaluated on a 70:30 train–test split. The contamination parameter was set to “auto” to estimate the proportion of anomalies from the dataset distribution. The contamination parameter was set to “auto” to allow the model to infer the proportion of anomalies directly from the data. Preliminary trials with fixed values (0.01–0.1) showed similar patterns, but the “auto” setting offered consistent and balanced results across all test videos. We experimented with values ranging from 0.01 to 0.1 and found that the “auto” setting provided a good balance between recall and precision. This model was chosen for its high accuracy and computational efficiency in surveillance applications, as supported by previous work in anomaly detection [15].

2.4. Forecasting with LSTM

To forecast crowd behavior trends over time, a Long Short-Term Memory (LSTM) neural network was implemented. LSTM networks are well-suited for sequential data with temporal dependencies and have shown strong performance in modeling time-series patterns in human mobility and urban crowd flows [16].

• The input time series consisted of per-frame averages of crowd density, speed, and movement angle.
• The dataset was split into 80% training and 20% testing data.
• The network architecture included one LSTM layer with 50 units and a Dense output layer.
• It was trained for 30 epochs using Mean Squared Error (MSE) as the loss function.

The forecast error threshold was empirically set at the 95th percentile to trigger alerts only when unusual future behaviors were predicted. The threshold was set at the 95th percentile of forecast errors to capture the most extreme deviations from expected crowd behavior while minimizing spurious alerts. This empirical choice reflects a balance between sensitivity and operational practicality.

For the LSTM model, we set the number of units to 50 and used a batch size of 64 based on validation performance. The number of epochs was fixed at 30 after observing convergence patterns, and the learning rate was set to 0.001 using the Adam optimizer. These hyperparameters were determined empirically to minimize overfitting while ensuring stable training. The model was trained with a batch size of 64 using the Adam optimizer with a learning rate of 0.001. A single LSTM layer was used to reduce computational overhead and latency, which is essential for real-time applications.

2.5. Alert System

A hybrid alert system was developed to merge the results of both models:

• Anomaly Alerts were triggered whenever the Isolation Forest model labeled an instance as anomalous.
• Forecast Alerts were activated when the LSTM forecast error exceeded the defined threshold of 0.2, corresponding to the top 5% of historical prediction deviations.

These alerts were merged into a single field to create a comprehensive flag per frame, identifying frames with either reactive (current anomaly) or proactive (forecasted risk) warnings. The unified alert system uses a logical OR condition: an alert is raised for a frame if it is flagged by either the anomaly detection model (Isolation Forest) or the forecasting model (LSTM). This ensures that both reactive and predictive risks are captured, allowing the system to respond effectively to present anomalies as well as anticipate potential crowd hazards.

2.6. Tools and Environment

• Programming Language: Python 3.9.
• Libraries: pandas, numpy, scikit-learn, matplotlib, seaborn, tensorflow/keras.
• Hardware: A high-performance workstation with an NVIDIA GPU (Nvidia, Santa Clara, CA, USA) for deep learning model training.
• Version Control: Git.

3. Results

This section presents the experimental results of the proposed system, which integrates anomaly detection and forecasting for crowd behavior analysis during the Hajj pilgrimage. The results are organized into four main parts: dataset analysis, anomaly detection, forecasting performance, and the final alert system. Each subsection includes quantitative metrics and visualizations to support interpretation.

3.1. Dataset Overview and Structure

The experiments were conducted using the Hajjv2 dataset, which includes annotated video data from various Hajj ritual locations. Each of the 18 videos was segmented into individual frames with object-level annotations. Nine videos were used for training and nine for testing.

Table 1. First five rows of the Hajjv2 dataset.

Video_No	Frame_No	X	Y	Width	Height	Classes	Obj_Name	Label
12	1	265	320	18	25	moving_in_opposite	movingo1	0
12	1	64	440	18	29	moving_in_opposite	Movingo3	0
12	1	321	383	18	27	moving_in_opposite	Movingo14	1
12	1	333	400	16	20	moving_in_opposite	movingo16	0
12	1	316	350	15	25	moving_in_opposite	movingo119	0

shows the first five rows of the dataset, presenting attributes such as Video_No, Frame_No, X, Y, Width, Height, Classes, Obj_Name, and Label. These attributes support both spatial and behavioral modeling.

Figure 2 summarizes the dataset distribution across videos, including the number of objects and anomalies per video, highlighting the variability in density and activity levels.

3.2. Data Cleaning and Validation

Before model training, missing values in essential fields such as coordinates and dimensions were removed, and irrelevant classes were filtered. Figure 3 illustrates the percentage of cleaned versus raw data, ensuring the dataset used in modeling was valid and consistent.

3.3. Isolation Forest-Based Anomaly Detection

The Isolation Forest model was trained on normalized features: object area, crowd density, speed, and angle of movement. Anomalies were identified based on deviations from learned patterns of normal behavior.

Figure 4 shows anomaly detection results for a sample video, where orange points indicate detected anomalies over time.

Figure 5 generalizes this analysis across all nine test videos. It highlights the model’s ability to detect behavioral anomalies consistently, even under varying crowd densities.

3.3.1. Evaluation of Isolation Forest Model Performance

The model’s performance was quantitatively assessed using accuracy, precision, recall, and F1-score. These metrics were computed using the labeled ground truth from the Hajjv2 dataset.

Table 2. Performance metrics (precision, recall, F1-score) for Isolation Forest.

Video Number	-	Precision	Recall	F1-Score	Support	Accuracy
2	0	88%	97%	92%	909	88%
	1	87%	65%	75%	339
3	0	92%	96%	94%	3454	90%
	1	85%	75%	75%	1152
5	0	95%	98%	96%	2702	94%
	1	93%	85%	88%	948
7	0	91%	98%	95%	3323	92%
	1	93%	75%	83%	1209
8	0	93%	98%	95%	1739	93%
	1	93%	80%	86%	697
9	0	92%	95%	93%	11022	90%
	1	83%	77%	80%	3921
10	0	92%	94%	93%	3948	90%
	1	88%	84%	86%	2091
11	0	87%	96%	91%	6138	86%
	1	81%	51%	62%	1813
12	0	93%	97%	95%	4319	93%
	1	90%	80%	85%	1512
Average	0	91%	97%	95%	-	91%
	1	88%	75%	80%	-

summarizes performance metrics for all nine test videos. The Isolation Forest achieved an average accuracy of 91%, with relatively high precision in identifying abnormal behavior.

3.3.2. Confusion Matrix Analysis

Figure 6 presents the confusion matrix for a representative dataset, showing the distribution of true positives, true negatives, false positives, and false negatives.

Figure 7 expands this to all datasets, confirming the model’s robustness despite a class imbalance (normal labels outweigh anomalies). This evaluation demonstrates the model’s reliability in detecting outliers that may indicate safety risks.

3.4. Forecasting Crowd Behavior Using LSTM

The forecasting module leverages a Long Short-Term Memory (LSTM) network to predict future trends in average speed, crowd density, and movement angle.

3.4.1. Temporal Analysis of Crowd Behavior

Figure 8, Figure 9 and Figure 10 show the temporal patterns of average speed, crowd density, and movement direction per frame for a sample video. These patterns served as the input time-series for the forecasting model.

3.4.2. LSTM-Based Forecasting Performance

The model was trained on 80% of the frames and tested on the remaining 20%. It achieved a Mean Squared Error (MSE) as low as 0.000439 for average speed prediction.

Table 3. Forecasting MSE across all test datasets.

Video Number	MSE Value
2	0.00123
3	0.0000156
5	0.00000481
7	0.00000337
8	0.0009747
9	0.000002584
10	0.000268
11	0.0000190
12	0.00143
Average MSE Value	0.000439

lists the Mean Squared Error (MSE) values for all videos, with most staying below 0.002, confirming the model’s strong predictive ability. These MSE values pertain exclusively to the forecasting of average crowd speed.

Figure 11 plots the predicted vs. actual average speed for single videos, showing high alignment and generalizability.

3.4.3. LSTM Training Evaluation

Figure 12 and Figure 13 depict the model’s training loss curves for one video and across all videos, respectively. The smooth convergence confirms the stability of the model training process.

3.5. Anomaly and Forecasting Alert System

The final component combines the outputs of the anomaly detection and forecasting modules to generate real-time alerts for abnormal crowd behavior and risk-prone future conditions.

3.5.1. Alert Triggering Mechanism

Alerts were flagged in two cases:

• When the Isolation Forest model detected abnormal behavior (AlertType = “Anomaly”).
• When the forecasted error exceeded the 95th percentile (AlertType = “Forecast_LSTM”).

3.5.2. Alert Evaluation and Visualization

Figure 14 presents the distribution of alerts by type, showing a balanced triggering across anomaly-based and forecast-based detections.

Figure 15 overlays triggered alerts on crowd density and speed metrics, helping to visually associate alerts with risky crowd dynamics.

4. Discussion

This study developed a machine learning-based system for enhancing crowd management during the Hajj pilgrimage, focusing on real-time anomaly detection and predictive forecasting of crowd behavior. The experimental results demonstrate that the proposed framework—using Isolation Forest for anomaly detection and LSTM for forecasting—effectively identifies high-risk crowd behaviors and provides early warnings to support proactive intervention. These findings offer meaningful contributions to the broader field of intelligent crowd monitoring systems.

The strong performance of the Isolation Forest model, with an average accuracy of 91% across diverse video datasets, validates its suitability for detecting anomalous crowd movements in high-density environments. This aligns with findings from Tran et al. [1], who also reported high anomaly detection rates in video surveillance using Isolation Forest. In contrast to traditional threshold-based detection systems, the Isolation Forest model is better equipped to handle complex and noisy data, especially when ground-truth labels are limited or imbalanced.

The LSTM model achieved a Mean Squared Error (MSE) as low as 0.000439 for speed forecasting, confirming its ability to capture temporal dependencies in crowd behavior. This supports the observations of Zhang et al. [2], who highlighted the effectiveness of LSTM in modeling spatio-temporal dynamics in citywide crowd flows. The forecast alerting mechanism developed in this study—triggered when prediction errors exceed the 95th percentile—offers an interpretable and statistically grounded approach to generating early warnings.

Previous studies on Hajj crowd management have reported high accuracy using CNN-based models for density estimation and classification. For example, Bhuiyan et al. (2022) [8] achieved up to 100% accuracy on their custom Hajj-Crowd dataset, and Shah (2024) [6] reported 87% accuracy using a Gradient Boosting Classifier. However, these models were limited to image-based classification and lacked predictive forecasting or integrated alert systems. In contrast, this study uses the Hajjv2 dataset—unlike any used in prior research—which includes detailed temporal data for movement and behavior. Therefore, the results of this study (91% anomaly detection accuracy and 0.000439 MSE in forecasting) cannot be directly compared but offer a more behavior-aware and proactive solution for real-time crowd monitoring.

By integrating outputs from both anomaly detection and forecasting models into a unified alert system, this research bridges a critical gap in the existing literature. Previous studies, such as those by Bhuiyan et al. [3] and Shah [4], have largely focused on either real-time classification or density estimation. However, few have combined real-time anomaly detection with predictive capabilities into a single operational system. This integration is particularly significant for the Hajj context, where real-time crowd conditions can shift rapidly due to spatial constraints and ritual timeframes.

The implications of these findings are considerable. For authorities and safety planners, the system provides both reactive alerts (based on anomalies) and proactive insights (via forecasting), which can inform real-time decision-making, route redirection, and resource deployment. This dual functionality enhances operational preparedness and minimizes the likelihood of crowd-related incidents.

Despite its strengths, the study has some limitations. First, the system was tested on a simulated dataset (Hajjv2), and not in a live deployment. Second, while the forecasting model uses average crowd speed and density, future models could benefit from incorporating more contextual variables such as location metadata, environmental conditions, and multi-camera perspectives. Third, the alert system currently relies on empirical thresholds (e.g., 95th percentile), which may require dynamic adjustment in a real-world setting. This system was evaluated using only the Hajjv2 dataset, which specifically reflects the unique conditions of the Hajj pilgrimage. Future work will involve testing the model on synthetic and real-world datasets from other domains to assess its generalizability.

Future Directions

Future research can build on this work in several ways. First, incorporating real-time data streams—such as live CCTV feeds or sensor data—would enable live deployment and continuous system evaluation. Second, expanding the system to handle multi-modal data (e.g., audio, text reports, mobile app feedback) could improve situational awareness. Third, adopting adaptive thresholding techniques based on contextual risk levels may refine the alert sensitivity. Finally, validating the system through field trials during smaller-scale crowd events would provide critical insights for eventual Hajj deployment.

Additionally, this study did not include an ablation or feature importance analysis. Exploring how individual features—such as speed, density, and movement angle—contribute to the model’s predictions could enhance interpretability and guide future model refinements. Understanding feature relevance would also support more transparent decision-making and help tailor the system to varying crowd conditions across different environments. Deeper or bidirectional LSTM variants will also be explored in future studies to assess potential improvements in predictive performance.

In conclusion, this study presents a scalable and intelligent framework that combines machine learning and time-series forecasting to address the complex challenge of crowd management during Hajj. By offering both anomaly detection and prediction-based alerts, it supports safer, more informed crowd control strategies and sets the foundation for future smart pilgrimage management systems.

5. Conclusions

This research introduced an integrated machine learning framework for crowd management during the Hajj pilgrimage, combining anomaly detection and time-series forecasting to support proactive safety interventions. Using the Hajjv2 dataset, the Isolation Forest model effectively identified abnormal crowd behaviors, while the LSTM model accurately forecasted future changes in speed and density.

The fusion of these models into a unified alert system demonstrated the potential of AI-driven solutions to enhance decision-making and prevent crowd-related incidents in high-density environments. The study contributes to the advancement of intelligent surveillance systems, particularly in religious or large-scale gatherings where timely response is critical.

While the results are promising, future work is needed to validate the system in live settings, integrate real-time data streams, and improve model adaptability to different crowd scenarios. Nonetheless, the proposed system represents a scalable and practical step toward modernizing crowd management at one of the world’s most complex events.