5.1. Evaluation Indicators of the Model
The confusion matrix reflects the classification performance of the LightGBM model on the test set, consisting mainly of four parts: True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). TP represents the number of samples that are actually positive and are predicted as positive by the model. TN represents the number of samples that are actually negative and are predicted as negative by the model. FP represents the number of samples that are actually negative but are predicted as positive by the model. FN represents the number of samples that are actually positive but are predicted as negative by the model. Various performance indicators can be calculated through the confusion matrix, such as Accuracy, Precision, Recall, and F1-Score, which help provide a more comprehensive understanding of LightGBM’s performance.
Accuracy reflects the classification performance of the model, generally indicating that the higher the accuracy, the better the model’s classification performance, with its calculation formula shown in Equation (7):
Precision reflects the proportion of samples predicted as positive by the model that are TP, with its calculation formula shown in Equation (8):
Recall reflects the proportion of TP samples that are correctly predicted as positive by the model, with its calculation formula shown in Equation (9):
The F1-score reflects the balance between precision and recall; it is the harmonic mean of precision and recall, with its calculation formula shown in Equation (10):
The Receiver Operating Characteristic (ROC) curve is a graphical tool used to evaluate the performance of classification models, particularly in classification problems. The ROC curve demonstrates the classification capability of a model by plotting the True Positive Rate (TPR) against the False Positive Rate (FPR) at various classification thresholds. The area under the ROC curve (AUC) is a critical metric for assessing the performance of binary classification models. AUC values range between 0 and 1, with higher values indicating better classification performance of the model. The key advantage of AUC lies in its robustness to imbalanced datasets and its provision of a unified measure for evaluating model performance across different threshold settings.
The model’s prediction results are shown in
Figure 2. As can be seen in
Figure 2a, under condition A (dry road conditions), the AUC is 0.89, which is consistent Wang et al. [
18], who found that speed differentials signicantly impact similar to other machine learning models. The ROC curve indicates that the classifier performs quite well, although there is still room for improvement. The model shows good performance in distinguishing between different categories (accident severity). Dry road conditions provide better traction, which likely explains the relatively lower AUC score compared to other conditions.
In
Figure 2b, under condition B (wet or water road surfaces), the AUC is 0.92, which is higher, slightly inferior to GBDT. This suggests that the model performs better in distinguishing accident severity under wet or waterlogged conditions. These road conditions generally lead to a higher accident rate, as vehicles may skid or lose traction, making it easier for the model to distinguish between different levels of accident severity.
Figure 2c shows that, under condition C (snow/slush/ice road conditions), the AUC is 0.93, indicating that the model can very accurately differentiate between levels of accident severity on icy or snowy roads, albeit slightly inferior to XGBoost. The increased difficulty in controlling vehicles on such surfaces leads to more severe accidents, which might explain the model’s strong performance in these extreme conditions.
In
Figure 2d, under condition D (sand, mud, dirt, oil, or gravel road conditions), the AUC is 0.85, indicating that the model’s classification performance is poorer compared to other road conditions, outperformed by all of the other machine learning models. This may be due to the fact that vehicles exhibit various unstable behaviors on these complex surfaces, leading to more complex accident severity patterns, making it harder for the model to accurately predict and differentiate between accident types.
5.2. Importance of Risk Factors Not Grouped
First, the processed dataset was utilized to predict the severity of highway_rail grade crossing accidents, identifying the impact of various features across all road conditions. This served as a foundation for comparing the effects under different road conditions in subsequent analyses. In this study, SHAP was employed to interpret the LightGBM model, providing insights into the differences in the ranking of factors influencing accident severity.
Using the feature importance rankings derived from the SHAP model, the contributions and interactions of key factors affecting the severity of highway_rail grade crossing accidents were analyzed. Li et al. [
19] demonstrated that driver behaviors, such as gap acceptance, play a significant role in influencing accident severity. This study further supports the importance of behavioral factors in determining outcomes, especially in complex driving enviroments. as illustrated in
Figure 3. The results reveal that the five most influential features are “TRNSPD” (train speed), “AGE” (driver age), “VEHSPD” (vehicle speed), “INVEH” (whether the driver was inside the vehicle), and “TYPACC” (type of accident). However, feature importance alone does not fully account for the multicollinearity between these features, nor does it uncover the underlying mechanisms through which these factors exert their influence.
To assess the contribution of various features to the severity of highway_rail crossing accidents, a SHAP summary plot was generated, as shown in
Figure 4. This study focuses on analyzing the top five factors influencing accident severity:
TRNSPD (train speed): As TRNSPD increases, the corresponding SHAP value also rises. This suggests that higher train speeds tend to result in more severe injuries for the driver, while lower train speeds lead to less severe accidents. The increased severity may be attributed to the reduced reaction time of the train driver at higher speeds, limiting the ability to take timely emergency actions. Additionally, higher train speeds increase braking distances, contributing to more serious injuries.
AGE (driver age): As the driver’s age increases, the SHAP value also increases, indicating that older drivers are more likely to suffer severe injuries. Younger drivers, on the other hand, tend to experience less severe accidents. This could be due to delayed brain responses in older drivers when confronted with sudden danger, preventing them from avoiding the hazard. In contrast, younger drivers generally react more quickly, allowing them to take emergency actions in time, thus reducing the severity of their injuries.
VEHSPD (vehicle speed): As VEHSPD increases, the SHAP value also rises, signifying that higher vehicle speeds are associated with more severe injuries. Lower vehicle speeds tend to result in less severe accidents. This may be due to the fact that higher vehicle speeds reduce the driver’s ability to make accurate judgments quickly, which can lead to more serious accidents, similar to the effect of high train speeds.
INVEH (driver inside the vehicle): It can be observed that when the driver is not inside the vehicle, the severity of injuries is lower compared to when the driver is inside. This is likely because the driver may take timely action to leave the vehicle upon recognizing imminent danger, reducing the extent of injury in a collision between the vehicle and the train.
TYPACC (type of accident): When “TYPACC” equals 1 (train collides with motor vehicle), it has a stronger positive impact on accident severity compared to when “TYPACC” equals 0 (motor vehicle collides with train). This is likely due to the greater kinetic energy of the train, which exerts a stronger force on the vehicle, leading to more severe accidents in train-to-vehicle collisions.
5.3. Importance of Risk Factors Grouped
To evaluate the impact of different features on the severity of highway_rail grade crossing accidents under various road surface conditions, feature importance charts were created for the four road conditions, as shown in
Figure 5.
Road Condition A (dry) (
Figure 5a): The five most influential features affecting accident severity are “TRNSPD” (train speed), “AGE” (driver age), “VEHSPD” (vehicle speed), “INVEH” (whether the driver was in the vehicle), and “POSITION” (vehicle position).
Road Condition B (wet or with water) (
Figure 5b): The top five influential features are “TRNSPD” (train speed), “AGE” (driver age), “VEHSPD” (vehicle speed), “AADTT” (average annual daily traffic), and “TOTOCC” (total occupants in the vehicle).
Road Condition C (snow, slush, or ice) (
Figure 5c): The most influential features are “TRNSPD” (train speed), “INVEH” (whether the driver was in the vehicle), “TIMEMIN” (time in minutes), “Sgnleqp” (signal equipment), and “AGE” (driver age).
Road Condition D (sand, mud, dirt, oil, or gravel) (
Figure 5d): The top five features are “WEATHER” (weather conditions), “VEHSPD” (vehicle speed), “TIMEMIN” (time in minutes), “HwySys” (highway rescue system), and “TRNSPD” (train speed).
To investigate the contributions of various factors to the severity of driver injuries in accidents at highway_rail grade crossings, SHAP force plots were generated for accident datasets under four road conditions, as shown in
Figure 6. The results provide the following insights:
Pavement A (dry) (
Figure 6a): Factors such as DRIVGEN = 1 (female drivers), INVEH = 1 (driver inside the vehicle), TYPACC = 1 (train colliding with a motor vehicle), and VEHSPD = 0 (vehicle speed) increase the severity of driver injuries. In contrast, TRNSPD = 14 (train speed) reduces the severity of injuries.
Pavement B (wet or with water) (
Figure 6b): Under this condition, VISIBILITY = 1 (nighttime) and AADT = 2338 (average annual daily traffic) contribute to increased injury severity. However, factors such as TRNSPD = 40 (train speed), VEHSPD = 10 (vehicle speed), TOTOCC = 1 (single occupant), and INVEH = 0 (driver outside the vehicle) mitigate the severity of injuries.
Pavement C (snow, slush, or ice) (
Figure 6c): Here, INVEH = 1 (driver inside the vehicle) increases the severity of driver injuries, while TRNSPD = 11 (train speed) has a mitigating effect.
Pavement D (sand, mud, dirt, oil, or gravel) (
Figure 6d): Factors such as TRNSPD = 48 (train speed) and WEATHER = 1 (adverse weather conditions) contribute to increased injury severity. On the other hand, factors such as AGE = 20 (driver age), AADT = 20 (average annual daily traffic), and VEHSPD = 5 (vehicle speed) reduce the severity of injuries.
These findings highlight the influence of various contextual factors on the severity of driver injuries under different road conditions, providing valuable insights for targeted safety interventions.
The SHAP summary plots for each road condition, as shown in
Figure 6, provide insights into the factors influencing the severity of highway–rail crossing accidents:
Pavement A (dry) (
Figure 7a): The importance of the “POSITION” feature significantly increases, suggesting that when the vehicle is positioned as “moving over crossing”, “trapped on crossing by traffic”, or “blocked on crossing by gates”, accident severity tends to be higher. This is likely because, when a train is approaching, the vehicle being in these positions may prevent the driver from moving off the crossing in time, leading to more severe accidents. Additionally, it can be observed that as TRNSPD, AGE, and VEHSPD increase, the SHAP values also increase. This indicates that higher values of TRNSPD, AGE, and VEHSPD are associated with more severe injuries for the driver, while lower values of these factors are less likely to result in severe accidents. This may be due to higher speeds and older age reducing the driver’s reaction time and increasing braking distances when facing sudden events, thus amplifying the severity of the accident.
Pavement B (wet/standing water) (
Figure 7b): The importance of “AADTT” (annual average daily traffic) and “TOTOCC” (total occupants) increases. This suggests that when the traffic volume is low and there are fewer vehicle occupants, drivers may feel safer and engage in more aggressive driving behaviors, disregarding the wet road conditions. Similarly, Wang et al. [
20] observed that low traffic volumes often lead to increased risk-taking behavior, which correlates with higher accident severity. Combined with high train speeds, this reduces reaction times for both the driver and the train, potentially leading to severe accidents. Conversely, when the traffic volume is higher and there are more vehicle occupants, drivers are likely to be more cautious, leading to lower accident severity due to reduced vehicle and train speeds. Additionally, similar to Road Condition A, as TRNSPD, AGE, and VEHSPD increase, the SHAP values also increase. This indicates that higher values of TRNSPD, AGE, and VEHSPD contribute to more severe injuries for the driver, while lower values of these factors are less likely to result in severe accidents.
Pavement C (snow/ice) (
Figure 7c): The importance of the “Sgnleqp” (signal equipment) feature increases, indicating that the presence of train signals reduces accident severity compared to crossings without signals. This may be because signals alert drivers, encouraging them to slow down or stop at the crossing, particularly on icy or snow-covered roads, thereby reducing the likelihood or severity of accidents. Additionally, it can be observed that higher TRNSPD tends to result in more severe injuries for the driver. Furthermore, when the driver is inside the vehicle, the severity of the accident increases. This may be due to the fact that drivers inside the vehicle are less likely to take timely self-rescue actions in the event of danger, which can lead to more severe outcomes.
Pavement D (sand, mud, oil, etc.) (
Figure 7d): “WEATHER” emerges as the most influential factor, with clear or cloudy weather being linked to lower accident severity compared to rain, snow, or fog. This is likely because, despite the poor road surface, visibility is generally better in clear weather, allowing drivers more time to react and adjust their speed or route. In contrast, adverse weather conditions further degrade the road’s friction, increasing the risk of skidding or losing control. Li et al. [
21] analyzed mandatory lane-changing behaviors under adverse road conditions and found that such behaviors significantly contribute to accident severity on unstable surfaces. Additionally, “HwySys” (highway rescue system) appears among the top five features for the first time. The presence of a rescue system can reduce response times and provide timely assistance, reducing the severity of the accident. In the absence of a rescue system, injured individuals may not receive prompt help, leading to more severe outcomes.
5.4. Influence of Risk Factors
To further understand the mechanisms by which various features influence the severity of highway_rail grade crossing accidents, SHAP dependency plots were generated for important features. These plots not only capture the main effects of individual factors but also reveal the interactions between two factors and how they influence accident severity, as illustrated in the figures.
Under pavement A, as shown in
Figure 8a,b, the impact of “TRNSPD” (train speed) on accident severity is significant, consistent with the findings of Pan Lu et al. A linear relationship can be observed between “TRNSPD” and accident severity. As TRNSPD increases, the SHAP value rises, indicating higher accident severity. However, once TRNSPD reaches 80, the SHAP value plateaus, suggesting that further increases in train speed do not significantly escalate accident severity. Additionally, “AGE” (driver age) and “VEHSPD” (vehicle speed) also significantly influence accident severity. Younger drivers and lower vehicle speeds are associated with more severe accidents. This could be because, on dry roads, vehicles have better traction, allowing drivers to maintain control at higher speeds. However, at very high speeds, the impact force increases dramatically, leading to more severe accidents. Younger drivers may also engage in more aggressive driving behaviors, contributing to greater accident severity.
As shown in
Figure 8c, unlike other studies where the age threshold was set differently, this study diverges by setting the age boundary at 65 years. When “AGE” is less than 40, SHAP values show a decreasing trend with increasing age, indicating lower accident severity. However, when AGE exceeds 40, the relationship becomes linear, with the SHAP values increasing as AGE rises, suggesting higher accident severity. “VEHSPD” follows a similar pattern to AGE. This could be because younger drivers tend to react more quickly and handle sudden threats better at lower speeds, thus reducing accident severity. In contrast, older drivers traveling at higher speeds may experience longer braking distances, slower reaction times, and an increased likelihood of severe accidents due to these factors.
Under pavement B, as shown in
Figure 9a,b, the impact of train speed on accident severity remains positively correlated within a certain range, similar to dry road conditions. However, the distribution of SHAP values exhibits greater volatility, particularly when the train speed is below 40. In this speed range, there is an almost linear relationship: the higher the train speed, the greater the accident severity. When the train speed exceeds 40, the distribution of SHAP values begins to flatten, indicating that further increases in speed do not lead to higher SHAP values.
Additionally, AGE and VEHSPD significantly affect accident severity. Older drivers and higher VEHSPD are more likely to be involved in serious accidents, possibly due to the slippery road conditions. Train speeds between 20 and 40 represent a critical range. Although the train speed is not high, factors such as slippery surfaces and increased braking distances reduce the driver’s control over the vehicle. Furthermore, older drivers may react more slowly, and higher speeds increase braking distances and reaction times, significantly impacting accident severity.
When train speed exceeds 40, the variability in SHAP values increases significantly, particularly in the 40–60 speed range, where the SHAP values fluctuate greatly. At this point, the impact of train speed on accident severity becomes nonlinear, possibly due to higher uncertainty at high speeds on wet surfaces, influenced by additional variables.
As shown in
Figure 9c, when AGE is below 30, SHAP values exhibit greater volatility, especially around age 20, where the SHAP values are significantly higher than for other age groups. VEHSPD also has a substantial effect on accident severity, with higher speeds leading to more severe accidents. This may be due to younger drivers’ aggressive driving behaviors, their tendency to overlook road conditions due to insufficient driving experience, and their inability to respond effectively to sudden situations. In contrast, older drivers, who have more driving experience, can maintain better control of their vehicles even at higher speeds, thus reducing the severity of accidents.
Under pavement C, as shown in
Figure 10, when the train speed (TRNSPD) is below 40, the SHAP values are relatively low, indicating a lower severity of accidents, regardless of whether the driver is inside the vehicle. However, when the train speed exceeds 40, particularly in the 40–70 range, the severity of accidents increases significantly when the driver is inside the vehicle. This increase in severity is likely due to the substantial difference in mass between trains and motor vehicles, which leaves the driver unable to react effectively in the event of a collision, resulting in more severe accidents.
Under pavement D, as shown in
Figure 11, the SHAP values are lower when the weather is clear, indicating lower accident severity. In contrast, under other weather conditions, the SHAP values are higher, suggesting a greater likelihood of severe accidents. This may be because, on sunny days, drivers benefit from better visibility and are more able to observe road conditions. When there is snow or ice, drivers tend to drive more cautiously, which reduces accident severity. However, in poor weather conditions, visibility decreases, making it difficult for drivers to accurately assess road conditions, potentially leading to more severe accidents.
When VEHSPD is below 20, SHAP values are low, indicating lower accident severity, regardless of when the accident occurs. However, when VEHSPD exceeds 20, the SHAP values increase, making severe accidents more likely. This may be due to the shorter braking distance of vehicles at lower speeds, which gives drivers sufficient reaction time, thus resulting in lower accident severity.
As shown in
Figure 12a, under pavement A, f(x) is generally stable, although some peaks and valleys are still observed. The peaks correspond to higher predicted accident severity, while the valleys indicate lower predicted severity. Considering the SHAP values, in regions with higher f(x), the SHAP value for TRNSPD is shown in red, indicating that train speed has a significant positive contribution to predicting accident severity. Similarly, the SHAP values for AGE and VEHSPD also exhibit significant red areas in the high-accident-severity predictions, suggesting that higher speeds and older drivers increase accident severity.
In
Figure 12b, under pavement B, f(x) shows larger fluctuations, with certain areas indicating a significant increase in predicted accident severity. TRNSPD continues to provide a strong positive contribution when predicting higher accident severity. Additionally, the SHAP values for AGE, AADT, and VEHSPD demonstrate a noticeable positive effect on predicting high accident severity, especially when AADT is high. Increased traffic volume on slippery roads significantly increases accident severity.
Figure 12c illustrates that, under pavement C, f(x) shows some prominent peaks, indicating that the model predicts higher accident severity in certain areas. In regions of predicted high accident severity, the SHAP values for TRNSPD and INVEH are positive, indicating a positive impact on accident severity. Furthermore, the influence of Sgnleqp becomes more significant, suggesting that the interaction between train speed and signaling equipment plays a major role in predicting high accident severity.
Figure 12d shows that, under pavement D, f(x) displays prominent peaks in certain areas. In regions of predicted high accident severity, the SHAP values for WEATHER and VEHSPD are positive, indicating that adverse weather conditions (rain, snow, fog) and higher VEHSPD have a positive impact on the prediction of accident severity.