Simulation-Based Evaluation of Incident Commander (IC) Competencies: A Multivariate Analysis of Certification Outcomes in South Korea

Abstract

This study investigates the certification outcomes of intermediate-level ICs in The National Fire Service Academy in South Korea through a comprehensive quantitative analysis of their evaluated competencies. Using assessment data from 141 candidates collected from 2022 to 2024, we examine how scores on six higher-order competencies—comprising 35 sub-competencies—influence pass or fail results. Descriptive statistics, correlation analysis, logistic regression (a statistical model for binary outcomes), random forest modeling (an ensemble decision-tree machine-learning method), and principal component analysis (PCA; a dimensionality reduction technique) were applied to identify significant predictors of certification success. Visualization techniques, including heatmaps, box plots, and importance bar charts, were used to illustrate performance gaps between successful and unsuccessful candidates. Results indicate that competencies related to decision-making under pressure and crisis leadership most strongly correlate with positive outcomes. Furthermore, unsupervised clustering analysis (a data-driven grouping method) revealed distinctive performance patterns among candidates. These findings suggest that current evaluation frameworks effectively differentiate command readiness but also highlight specific skill domains that may require enhanced instructional focus. The study offers practical implications for fire training academies, policymakers, and certification bodies, particularly in refining curriculum design, competency benchmarks, and evaluation criteria to improve fireground leadership training and assessment standards.

1. Introduction

Incident commanders in the fire service are responsible for directing emergency operations and ensuring firefighter and civilian safety. The decisions made by ICs in the first few minutes of an incident can have life-or-death consequences. Developing effective command skills requires not only experience but also systematic training and evaluation. Effective incident command is critical for emergency response, as the decisions and leadership of incident commanders (ICs) can significantly influence outcomes in firefighting and disaster scenarios [1].

Prior research has identified that inadequate or incorrect decision-making by ICs has contributed to firefighter injuries and death [2]. Modern emergency response demands that ICs possess not only technical firefighting knowledge but also keen decision-making, communication, and leadership skills to manage high-stakes, dynamic situations [3]. However, opportunities to develop and evaluate these critical command competencies on the fireground have diminished in recent years as the overall number of serious incidents has declined. This reduction in real-world exposure can lead to skill fade among ICs, underscoring the need for structured training and assessment interventions to maintain incident command readiness [4].

To address this gap, fire services worldwide have increasingly turned to simulation-based training and certification programs for incident command. Virtual reality (VR) and interactive simulation exercises can safely recreate complex, hazardous scenarios from high-rise fires to multi-casualty incidents, allowing ICs to practice strategic decision-making and tactics under controlled conditions [5]. Studies have demonstrated that computer-based simulation training can improve ICs’ decision-making efficiency and accuracy, increase their confidence in command, and bolster overall preparedness [1]. For example, a survey of firefighters reported significant gains in decision-making competence and job satisfaction from routine simulation drills, although it also revealed that some departments underutilize such training [1]. By enabling repeated practice of critical skills (e.g., size-up, strategy formulation, communication) in realistic yet risk-free environments, simulation-based experiential learning helps translate classroom knowledge into better performance during real emergencies [1].

At the same time, a systematic review of immersive VR education research found that many studies focus on technology development over practical integration, highlighting the need for more theory-driven, evidence-based implementations of simulation training in fields like fire service education [6]. Notably, during the COVID-19 pandemic, remote virtual simulation was trialed for incident command training in Sweden, allowing students to practice incident recognition, comprehension, and decision-making without physical travel or health risks. This approach enhanced cognitive engagement and led to recommendations to embed remote VR assessment exercises into fire-service curricula [7].

The decision environment for ICs is often characterized by uncertainty, time pressure, and rapidly evolving conditions. ICs typically rely on naturalistic decision-making (NDM)—making swift judgments based on pattern recognition and experience rather than formal analytic processes [8]. This recognition-primed approach can be effective for seasoned ICs, but it may lead to errors if a scenario deviates from the decision-maker’s past experience or if stress and fatigue impair cognitive function [9]. Training interventions serve to mitigate these risks by exposing ICs to a wider variety of simulated scenarios (expanding their mental repertoire of cues and responses) and by encouraging the use of structured decision aids when appropriate [9,10]. One such training approach is teaching a structured “decision control” technique: Cohen-Hatton and Honey (2015) found that ICs who momentarily paused to apply a mental checklist (stating their goals, predicting consequences, and weighing risks) engaged in more reflective decision-making and achieved better outcomes in fast-moving fire simulations [10]. This finding suggests that even in high-pressure NDM settings, a brief, deliberate pause to reassess the situation can prevent fixation on a single tactic and promote adaptability as conditions change, ultimately improving safety and effectiveness [10].

Another critical component of effective command decision-making is situational awareness, which is the ability to perceive elements of the environment, understand their meaning, and project future status. High situational awareness allows a IC to anticipate events (like fire spread or structural collapse) and adapt strategies proactively. Studies in emergency management and human factors have long established that training can improve situational awareness and that experienced ICs maintain better situational awareness than novices [11]. Simulation scenarios that incorporate information overload or chaotic, rapidly changing conditions force trainees to practice filtering critical information (such as crew locations, victim statuses, and fire behavior) from background noise. Over time, this kind of training improves ICs’ ability to maintain a “big picture” understanding of the incident amidst chaos [11]. In addition, qualitative research in other emergency domains underscores the breadth of skills required for effective incident management. For example, interviews with Finnish emergency medical service supervisors identified competencies like preparedness, risk assessment, triage, and communication as vital for managing complex hazardous incidents, recommending that chemical, biological, radiological, nuclear, and explosive (CBRNE) scenarios receive dedicated, specialized training beyond generic all-hazards approaches [12].

Recognizing the multifaceted skills needed for incident command, many fire organizations have developed competency frameworks and tiered training programs to guide officer development. In the United Kingdom, for instance, the fire and rescue service identified six core non-technical skill areas—assertive safe leadership, decision making, communication, situational awareness, personal resilience, and teamwork—and incorporated these into the THINCS behavioral marker system to assess incident command performance [3]. Similarly, the UK’s “Effective Command” model provides a structured framework for training and evaluating ICs, focusing on key behaviors in both technical and non-technical domains [13]. These frameworks underscore that effective incident command requires not just procedural knowledge but also behavioral competencies, and they offer a consistent basis for training and evaluation [3,13]. However, such models and markers (e.g., THINCS and Effective Command), while useful in theory and training, can be challenging to implement in routine assessments and often lack empirical benchmarking across large candidate groups. Infrequent exposure to large-scale emergencies can still result in skill degradation, as noted by Lamb et al. (2014), who reported that ICs may experience “skill fade” over time; as a countermeasure, one fire service (Oxfordshire Fire & Rescue) introduced the “Introspect” simulation-based development program to provide regular practice and assessment opportunities for its ICs [13]. This context highlights the need for empirical data to complement these theoretical frameworks, ensuring that training models are validated against observed performance trends.

Empirical studies across different contexts further reinforce the value of simulation-based training for command decision-making. In the United States, Hall’s (2010) research showed that ICs who underwent targeted computer-simulated incident scenarios were better able to size up situations and make strategic decisions than those with only traditional training [2]. In Germany, a survey of 288 command personnel led to the development and validation of multidimensional evaluation instruments (FIRE-CU and FIRE-CPX questionnaires) to rigorously assess tactical and strategic command-unit training effectiveness [14]. Field studies have also observed that participants in simulation training report improved decision-making speed and accuracy, greater confidence in commanding incidents, and better information processing under stress [1,15]. At the same time, certain limitations of virtual training have been noted. For instance, one study found minimal knowledge gain from a single VR hazardous-materials scenario, likely because participants were already experienced; this suggests that a greater variety of scenarios and repeated exposure may be necessary to significantly improve expertise through VR training [16]. Moreover, reviews of advanced firefighter training technology indicate that while VR/AR/MR simulations enhance safety and scalability, current systems still face realism gaps and technical constraints, prompting calls for improvements such as AI-driven scenario generation and biometric feedback to heighten training fidelity [17].

Notably, the benefits of realistic scenario-based training are evident in other high-risk fields as well. Law enforcement research has shown that repeated immersion in critical incident simulations improves officers’ threat assessment and use-of-force decision-making, especially for less experienced personnel [18]. The military has long utilized war-gaming and virtual exercises to develop decision-making skills under stress [19], and recent studies indicate that immersive VR drills can produce decision-making performance in soldiers comparable to live exercises [20]. These cross-disciplinary findings lend support to the fire service’s use of simulations and scenario-based assessments to prepare ICs for the pressures of real emergencies.

In line with international trends, the National Fire Service Academy (NFSA) of Korea implemented a “IC Certification” system in 2021 to systematically train and evaluate fire officers in incident command roles [21]. The program established a standardized, multi-tier framework of education and certification—spanning initial, intermediate, advanced, and (recently) strategic levels—corresponding to the scope of incidents an officer is qualified to manage [21]. The certification emphasizes that prospective ICs must possess not only practical experience but also strong theoretical knowledge and a sense of responsibility in their decision-making roles. However, since the program’s implementation, little empirical data has been collected to evaluate its effectiveness. To date, no data-driven analysis of candidates’ certification performance has been published, leaving a critical gap in understanding the program’s impact on command competency and in informing further training or policy adjustments.

This study seeks to address these gaps by conducting an in-depth, data-driven analysis of performance results from the Intermediate-level IC Certification practical evaluations over the period 2022–2024. Examining multiple cohorts across this timeframe allows us to identify patterns in candidate performance and to determine whether certification outcomes have improved, declined, or remained stable as the program has matured. By analyzing scores across the key competency categories assessed in the practical exams, we can pinpoint specific areas where candidates excel or struggle, thereby highlighting which command skills may require additional training focus. In doing so, this evidence-based evaluation of the NFSA’s certification program provides an empirical complement to existing competency frameworks, offering evidence-backed insights that can validate current practices or indicate areas needing enhancement. By merging quantitative performance data with qualitative lessons from prior research, this study offers a unique contribution—it not only evaluates the current state of incident command preparedness under the NFSA’s certification system, but also provides actionable guidance to enhance the development of fire service leaders moving forward.

The objectives of this study are threefold. First, we examine overall pass rates and temporal trends across the 2022–2024 intermediate certification cohorts to determine if candidate performance has changed over time. Second, we perform a detailed analysis of scores in each evaluated competency category (such as situational assessment, strategy formulation, communication, safety management, etc.) to identify specific strengths and weaknesses among incident command candidates relative to the expected standards. Third, informed by these empirical insights and supplemented by the reviewed literature, we propose targeted recommendations aimed at refining IC training curricula, improving assessment methodologies, and strengthening policy support mechanisms to better prepare future ICs. By merging quantitative performance data with qualitative lessons from prior research, this study offers a unique contribution—it not only evaluates the current state of incident command preparedness under the NFSA’s certification system, but also provides actionable guidance to enhance the development of fire service leaders moving forward.

2. Materials and Methods

2.1. Participants and Exam Procedure

We analyzed de-identified assessment records from an official excel database of the NFA’s Intermediate IC Certification Evaluations. The dataset comprised 141 candidates’ results from five sequential certification (2022~2024 years). The IC certification program under study was developed to evaluate whether mid-career fire officers possess the necessary incident command competencies to lead operations. Candidates for this certification were typically fire captains with several years of field experience, aspiring to qualify as ICs at the battalion or district level. The assessment was structured as a practical examination using scenario simulations.

Each candidate was required to act as IC in a simulated emergency scenario while evaluators assessed their performance against a standardized rubric. Five distinct virtual scenarios—Goshiwon Fire, Karaoke Room Fire, Residential Villa Fire, Apartment Fire, and Construction Site Fire—were carefully designed to progressively increase in complexity, reflecting common high-risk incidents faced by fire officers in the region. Each scenario involved sequential tasks beginning with initial fire suppression and victim rescue, and escalating to managing secondary fire spread and environmental hazards.

Specifically, each scenario presented unique structural and operational challenges:

• Goshiwon Fire: A small dormitory characterized by narrow, crowded spaces, complicating both rescue operations and hose deployment. Originally, Goshiwon facilities provided tiny, private rooms intended for students studying intensively for national exams. However, recently, these dormitories have evolved into affordable housing options primarily for low-income individuals. Dense smoke quickly accumulated due to poor ventilation, significantly increasing firefighting difficulty.
• Karaoke Room Fire: A layout of multiple compartmentalized rooms presented challenges in navigation, search, and rescue. Flammable interior materials produced thick toxic smoke, intensifying suppression difficulties.
• Residential Villa Fire: A multi-story building scenario with fire initially on a lower floor and quickly spreading upward via internal stairways, cutting off escape routes and complicating victim extraction and suppression tactics.
• Apartment Fire: A high-density residential building where a mid-level fire rapidly spread to adjacent apartments, creating multi-room fires that complicated evacuation, rescue operations, and firefighting strategy. Stairwell congestion and high fuel loads posed additional operational challenges.
• Construction Site Fire: A partially constructed building introducing hazards such as structural instability, scaffolding obstructions, limited access, and absence of fixed water supply points. The presence of flammable construction materials and the dynamic nature of the scenario required rapid tactical adaptation.

Detailed parameters and escalation patterns for each scenario are summarized comprehensively in Appendix D

Table A4. Scenario Structure and Difficulty Parameters.

Scenario	Dispatch Trigger	Initial Conditions	Victim Distribution	Fire Locations	Fire Spread	Challenges	Hazard Notes
1. Goshiwon Fire	Fire reported on 2nd and 3rd floors	Upon arrival, one person is hanging from a 2F window requesting rescue. Deployment of an air rescue cushion is possible; use of an aerial ladder truck is not feasible.	6 people total; 2F–1 hanging from window, 1 in hallway, 2 inside rooms 218 and 227; 3F–1 trapped on floor; Roof–1 person taking refuge.	2 (two ignition points on 2F and 3F)	Fire extends upward from 2F to 3F	Delayed arrival due to heavy traffic congestion	City gas supply not shut off (explosion risk); one victim fell from 2F window during escape; Room 227 door locked (entry delayed); thick smoke causing near-zero visibility.
2. Karaoke Room Fire	Fire outbreak on 2nd floor	Upon arrival, heavy smoke is billowing from the 2F windows. One person is visible at a 3F window calling for help.	5 people total; 3F–1 hanging from window, 1 in corridor; 2F–1 collapsed in hallway, 2 trapped inside karaoke rooms.	1 (ignition on 2F)	Fire spreads upward from 2F to 3F	Delayed response due to illegally parked cars	Maze-like interior layout hinders search; flammable soundproofing materials produce dense toxic smoke; insufficient emergency exits make egress difficult.
3. Residential Villa Fire	Fire outbreak on 3rd floor	Upon arrival, a resident is found on a 4F balcony awaiting rescue. Fire and smoke are spreading toward the 4th floor.	5 people total; 4F–1 on balcony; 3F–1 trapped in the burning unit, 1 in stairwell; 2F–1 semi-conscious from smoke; Roof–1 person who fled upward.	1 (ignition in 3F unit)	Fire spreads from 3F up to 4F	None	Rapid smoke spread through open stairwell; no sprinkler system in building; small LPG gas cylinder in use (potential explosion hazard).
4. Apartment Building Fire	Fire reported on 8th floor	Upon arrival, flames are venting out of an 8F apartment. A resident is spotted on a 9F balcony shouting for help.	5 people total; 9F–1 on balcony, 1 in apartment above fire; 8F–1 in burning apartment, 1 collapsed in hallway; 7F–1 overcome by smoke on the floor below.	1 (ignition in 8F apartment)	Fire spreads from 8F up to 9F	None	One resident jumped from 8F before rescue (fatal injuries); broken windows create backdraft risk; high-rise height complicates evacuation and firefighting operations.
5.Construction Site Fire	Fire outbreak in building under construction	Upon arrival, the unfinished structure is engulfed in flames on one side. Debris and construction materials litter the scene.	4 people total; 3 at site–1 worker on upper scaffolding, 1 trapped under debris, 1 incapacitated at ground level; 1 missing (unaccounted for amid the chaos).	1 (ignition in scaffold/structure)	Fire spreads through scaffolding and materials on site	None	Multiple fuel and gas cylinders on site (explosion hazard); structural integrity compromised (collapse risk); lack of on-site water source slows firefighting.

.

Evaluators assessed candidate performance against a standardized rubric structured around critical tasks of incident command, with competencies scored according to clearly defined criteria (see Appendix D) These scenarios were chosen to cover a range of command challenges (from life-saving prioritization to inter-agency coordination). On the day of the certification, each candidate was randomly assigned one scenario to manage, to ensure a fair distribution of scenario types among the group. The simulations were conducted in a dedicated incident command training facility, utilizing computer-based simulation software projected onto large screens as well as role-playing elements. For example, candidates received radio messages and updates from simulated crews (voiced by instructors or through software prompts), and they had to give commands as if directing real units.

The practical evaluation criteria are broken down into six major competency areas, reflecting the critical tasks of incident command

2.2. Performance Measures and Scoring

Exam performance was quantified by scores in six higher-order competencies, each composed of multiple sub-competencies. The competencies were: Situation Evaluation (Competency 1, max 28 points), Command Decision-Making (Competency 2, max 28), Response Activities (Competency 3, max 43), Progress Management (Competency 4, max 35), Communication (Competency 5, max 26), and Crisis Management & Leadership (Competency 6, max 40). In total, 35 sub-competencies were defined (e.g., “3—8: Documenting the tactical situation board,” “6—3: operational leadership tasks,” etc.). Each sub-competency had a maximum point value; a candidate’s sub-competency scores were summed to yield each competency score and an overall score (total maximum 200 points). Scores were normalized to percentages for analysis (raw score ÷ max points for that competency). Inter-rater agreement procedures were in place during scoring to ensure consistency. The detailed scoring rubric for the evaluation (encompassing all six competency categories and 35 behavioral indicators) is provided in Appendix A. This rubric was developed in-house through a structured process involving expert committees. Specifically, a Competency Development Expert Committee and a Competency Evaluation Expert Committee (comprised of senior fire officers and external subject-matter experts) collaboratively defined the critical incident command behaviors and their point values (see Appendix C for the committee structure and functions). The rubric’s content validity was established via expert consensus and iterative refinement, and a formal standard-setting procedure was used to determine the pass/fail threshold for certification. Under this standard, candidates needed to score at least 160 out of 200 points overall (80%) and not receive a “Low” rating on any of seven key behavioral indicators in order to pass (Appendix A outlines these pass criteria). To ensure the evaluation’s fairness and objectivity, the development process and scoring guidelines were overseen by an Operation Committee (Appendix C), which set consistency and transparency standards for the certification. Furthermore, inter-rater reliability was reinforced through extensive evaluator training and calibration sessions prior to the exam, alongside the use of multiple assessors scoring each candidate.

To ensure scoring consistency and fairness, each candidate was evaluated by a panel of three assessors. Two were internal evaluators—senior or peer-level firefighters with equivalent or higher rank and field experience—while the third was an external evaluator: a university professor with expertise in psychology, responsible for assessing candidates’ behavioral and psychological responses under stress. Inter-rater agreement procedures and calibration sessions were conducted prior to evaluation to promote consistency and minimize subjectivity in scoring.

2.3. Statistical Analysis

2.3.1. Group Comparisons (Pass vs. Fail)

We compared mean scores between passing and failing candidates to identify which competencies and sub-competencies differed by outcome. Independent two-sample t-tests (or Welch’s t-test when variances were unequal) were used to test whether the mean competency scores differed significantly between the Pass and the Fail. Boxplots and violin plots were generated to visualize distributions of competency scores by outcome. Effect sizes (Cohen’s d) were computed for key comparisons. Sub-competency score differences were similarly examined with t-tests, with Bonferroni correction applied to control for multiple comparisons. In addition, a chi-squared test of independence was used to compare pass/fail frequencies across binary categories of interest (e.g., pass rates by scenario type).

2.3.2. Competency–Outcome Correlation Analysis

We assessed bivariate associations between each competency score and the binary pass/fail outcome. Pearson’s correlation coefficient (point-biserial form) was calculated to estimate the linear correlation between each continuous competency score and the dichotomous outcome. The correlation analysis identified which competencies were most strongly positively associated with passing. A correlation matrix of all six competencies and the pass indicator was constructed and visualized via heatmap. All significance tests (e.g., testing whether r ≠ 0) were two-tailed with α = 0.05, and p-values were reported for completeness.

2.3.3. Logistic Regression Model

A multivariate logistic regression model was fit to predict the probability of passing (coded 1 = Pass, 0 = Fail) from the six competency scores. This generalized linear model with logit link (binary logistic regression) estimates the log-odds of passing as a linear combination of predictors. All competency scores were entered simultaneously as continuous predictors (percentage scores 0–100). Model coefficients (log-odds) and their standard errors were estimated by maximum likelihood. Statistical significance of each coefficient was assessed via Wald chi-squared tests. Odds ratios (exp [β]) were computed to interpret the effect sizes: an odds ratio > 1 indicates higher scores increase the odds of passing. Collinearity diagnostics were examined to ensure predictors were not excessively correlated. Model fit was evaluated by pseudo-R² (e.g., Nagelkerke R²) and Hosmer–Lemeshow goodness-of-fit. Regression analysis followed standard practice for binary outcomes.

2.3.4. Random Forest Classification and Feature Importance

We implemented a Random Forest classifier to predict Pass/Fail using all 35 sub-competency scores as input features. A random forest is an ensemble of decision trees that improves predictive accuracy by averaging over many decorrelated trees. We used 1000 trees (estimators) with default hyperparameters (scikit-learn implementation) and bootstrap sampling of the training set for each tree. The model was trained on the full dataset using five-fold cross-validation to estimate performance; classification accuracy was noted but not the focus here. After training, we extracted impurity-based feature importance scores for each sub-competency. Gini importance was computed as the average decrease in node impurity attributable to splits on each feature. These importances were aggregated by competency (summing importances of sub-features within each competency) to rank the overall influence of each domain. The top-ranked sub-competencies were identified to highlight specific skills associated with success. This random-forest analysis was based on Breiman’s algorithms and is a standard method for capturing nonlinear interactions and variable importance.

2.3.5. Principal Component Analysis (PCA)

To explore the underlying structure of candidate performance, we applied Principal Component Analysis to the six competency scores (each candidate represented as a six-dimensional point). PCA is a linear dimensionality-reduction technique that transforms the original correlated variables into orthogonal principal components that capture maximal variance. The competency scores were first standardized (zero mean, unit variance) before PCA. The first two principal components (PC1 and PC2) were computed and used to create a scatterplot of candidates (PC1 on horizontal, PC2 on vertical), colored by Pass/Fail. We examined whether PCA produced a visual separation between groups. The proportion of variance explained by PC1 and PC2 was reported. This unsupervised analysis helped to assess if an overall “performance” axis (PC1) distinguished passing from failing profiles.

2.4. Scenario Impact Analysis

To determine whether the type of disaster scenario affected outcomes, we compared pass rates and mean scores across the five scenario groups. Pass rates by scenario were compared using the chi-square test of independence, testing the null hypothesis that pass/fail is independent of scenario type. No a priori weighting of scenarios was applied because scenario assignment was randomized. We also compared mean total scores and competency scores across scenarios using one-way ANOVA (when assumptions of normality and equal variance held), or the Kruskal-Wallis test otherwise, to detect any systematic score differences by scenario. Any significant ANOVA findings were followed by Tukey’s post hoc test. These analyses checked for potential scenario effects on performance.

3. Results

3.1. Sample & Overall Pass Rates

Over five exam cohorts from 2022 to 2024, a total of 141 candidates participated. Among them, 80 passed and 61 failed, yielding an overall pass rate of about 56.4% (95% CI [48%, 64%]). The pass rate varied by cohort—for example, the first half of 2023 saw about 66.7% of candidates pass(n = 24; 95% CI [48%, 83%]), whereas the second half of 2024 had a lower pass rate around 40% (8 passes out of 20; 95% CI [19%, 64%])—but across all sessions combined the success rate was just over half. This indicates that the Intermediate IC Certification is quite competitive, with nearly half of the candidates not meeting the passing criteria.

3.2. Competency Score Differences (Pass vs.Fail)

3.2.1. Higher-Order Competencies

Each candidate was evaluated on six higher-order competencies, comprising a total of 35 sub-competencies. For each competency, candidates received a score (as a fraction of the maximum points for that area). We compared the performance of passing vs failing candidates on these competencies:

According to Figure 1, Box and violin plots comparing competency scores of passing vs failing candidates. Successful candidatesSuccessful candidates show substantially higher scores in all six competency areas, with the largest gaps in competencies 4 and 6. From the chart, we can see that the largest performance gaps are in:

• Crisis Management & Leadership (Competency 6): Successful candidatesSuccessful candidates averaged about 85.4% of points in this area vs 71.2% for unsuccessful candidates—a gap of ~14.2 percentage points (95% CI [9%, 19%]). In terms of raw score, successful candidates scored ~5.7 points higher (out of 40) in this category on average than unsuccessful ones. This was the biggest gap observed among all competencies. This corresponds to a large effect size (Cohen’s d ≈ 1.3), underlining the practical significance of this performance gap.
• Progress Management (Competency 4): Successful candidatesSuccessful candidates averaged ~86.8% vs. Unsuccessful candidates ~72.7% (a ~14.0 point percentage difference, ~4.9 raw points out of 35).

By contrast, the differences, while still significant, were slightly smaller for areas like Situation Evaluation (Competency 1) (~10.6% gap, ~3.0 points out of 28) and Response Activities (Competency 3) (~10.9% gap, ~4.7 points out of 43). Competencies 2 (Command Decision-Making) and 5 (Communication) showed intermediate gaps (~11–12% difference, equivalent to ~3.1 points out of 28 for Command and ~3.1 points out of 26 for Communication). In summary, successful candidates performed better across the board, but Leadership and Progress Management skills appear to distinguish pass vs fail the most, with Communication and other areas slightly less pronounced.

3.2.2. Sub-Competency Differences

Drilling down further, we examined all 35 sub-competencies to identify which specific skills had the biggest score gaps between successful candidates and unsuccessful candidates. Consistently, we found that the largest gaps tended to occur in sub-competencies under the broader areas of Crisis Management/Leadership (Competency 6) and Progress Management (Competency 4):

• Several Crisis Management & Leadership sub-competencies showed very large differences. For instance, the sub-competency “6—3” (a crisis leadership task, e.g., related to operational execution as a leader) had successful candidates scoring about 19.7 percentage points higher on average than unsuccessful candidates. Similarly, “6—1” (responding to sudden crisis situations) showed ~19.3% gap. These represent some of the highest disparities in the entire exam—indicating that strong leadership under crisis conditions is a key hallmark of those who passed.
• Multiple Progress Management sub-competencies also exhibited large gaps. For example, “4—3” (related to prioritization and reporting of actions) showed a ~17.3% difference, “4—5” (managing tactical priorities) about ~16.6%, and “4—2” (implementing measures to improve/worsen situations) ~16.4% gap in favor of passing candidates. These suggest that the ability to manage the progress of the incident (e.g., adjusting tactics and priorities appropriately) clearly separated those who passed from those who failed.
• Other notable gaps were observed in a Communication sub-competency—“5—5”, which appears to correspond to efficient communication—where successful candidates scored ~14.5% higher on average. Also, a Response Activities sub-competency “3—8” (possibly documenting the tactical situation board) had about a 15.5% gap.

Importantly, all of these differences are statistically significant (most with p < 0.001).

Notably, after applying a multiple-comparisons correction (Bonferroni adjusted α ≈ 0.0014 for 35 tests) and controlling the false discovery rate at 5%, these performance gaps remained significant, indicating that our results are robust against Type I error inflation. In fact, virtually every sub-competency saw successful candidates outperform unsuccessful candidates by a meaningful margin. The largest gaps (15–20% range) reinforce that leadership under crisis, resource deployment and adjustment, and maintaining strategic priorities were the areas where failing candidates struggled the most relative to those who passed. Conversely, smaller (though still significant) gaps were seen in some foundational areas like initial situation assessment (e.g., sub-competency 1—4 “Initial Situation Evaluation” had ~8% gap)—meaning even those who failed did not completely flounder in basic situational evaluation, but fell behind more in higher-order leadership and management tasks.

3.3. Key Predictors of Success

Given the differences noted above, we next investigated which competencies (and sub-competencies) are the most critical determinants of a passing outcome. We used several analytical approaches—correlation analysis, logistic regression, and a random forest model—to identify the key predictors that distinguish successful candidates.

3.3.1. Correlation Analysis

First, we computed the correlation between each competency score and the final outcome (Pass = 1, Fail = 0). All six competencies individually have a positive correlation with passing (since higher scores make passing more likely), but the strength of the correlation varies:

As illustrated in Figure 2, correlation heatmap for the six competency scores and the pass/fail outcome. Darker red indicates a stronger positive correlation with passing.

As shown above, “Crisis Management & Leadership” has the strongest correlation with passing (r ≈ 0.77) among the six competencies. In other words, candidates who scored well in the Leadership competency were much more likely to pass. The next strongest predictor by correlation is “Progress Management” (r ≈ 0.75). Both of these align with the earlier observation that these areas had the largest score gaps between successful candidates and unsuccessful candidates.

By contrast, the competency with the weakest (though still substantial) correlation with outcome was “Response Activities” (r ≈ 0.62). The Communication competency was correlated at about r ≈ 0.65, slightly higher than Response. Meanwhile, Situation Evaluation and Command Decision-Making showed intermediate correlation (in the ~0.62–0.68 range). All correlations are statistically significant (p < 0.001).

In practical terms, this suggests that performance in Crisis Management/Leadership and Progress Management is most closely associated with whether a candidate passes, whereas performance in Response Operations and basic Situation Evaluation, while still important, has a slightly weaker association with the final result. (Notably, all competencies are inter-correlated to some degree—strong candidates tend to do well across the board—but the above shows which areas track most strongly with success.)

For example, consider the following question: Is performance in “Response Operations” more influential than “Communications”? Based on correlation alone, Communication appears slightly more influential (r ~0.65 vs. r ~0.62 for Response). This implies that communication skills had a marginally stronger relationship with passing the exam than did response operations skills. However, both are less influential than competencies like Leadership or Progress Management.

3.3.2. Logistic Regression Analysis

We next fit a logistic regression model using all six competency scores to predict the probability of passing. This allows us to see the unique contribution of each competency controlling for the others. The regression results reinforced a similar rank-order of importance:

Logistic regression coefficients for each competency. Competencies 6 and 4 show the highest positive coefficients (most influence on passing), whereas competencies 5 and 1 had the smallest (Figure 3).

Error information: Ninety-five-percent Wald confidence intervals for every coefficient (e.g., Comp6: 0.80 ± 0.12; Comp4: 0.70 ± 0.11) are provided in Appendix C

Table A3. Composition of the Operation Committee for IC Qualification.

Chairman: Director of 119 Operations -Members: Head of Disaster Response Division, Key Section Chiefs
Competency Development Expert Committee	Competency Evaluation Expert Committee	Working-Level Council
- Chair: Head of HR Development, Central Fire School - Members: 10 internal/10 external	- Chair: Commander, Seoul Fire HQ - Members: 10 internal/10 external	- Staff from Disaster Re sponse Division - Operational institution staff

so readers can judge estimate precision and sampling variability.

As the logistic model indicated, Crisis Management & Leadership (Competency 6) emerged as a highly significant predictor—candidates’ scores in this area contributed the most to increasing their odds of passing (all else being equal). Progress Management (Competency 4) was the next most impactful predictor, also statistically significant. Command Decision-Making (Competency 2) and Response Activities (Competency 3) had positive regression coefficients as well, indicating that they contribute to success, but their significance was more marginal when Leadership and Progress were in the model (likely due to overlap with those areas). Communication (Competency 5) and Situation Evaluation (Competency 1) showed the smallest unique effects in the regression (in fact, the communication competency’s coefficient was the lowest). This suggests that once you account for a candidate’s leadership, management, and decision-making abilities, their communication score did not add as much additional predictive power in distinguishing pass vs fail. In other words, communication skills alone were not enough to pass without the other skills, and many candidates who failed likely also did poorly in other areas. (It’s worth noting that communication still had a strong bivariate correlation with outcome; it just correlates strongly with the other competencies too, so its unique contribution is diminished in a multivariate context).

Overall, the logistic model demonstrated good performance, achieving ~85% classification accuracy (with precision and recall ≈ 0.85) and an AUC of ~0.90 in 5-fold cross-validation, indicating a high level of explanatory power. To further examine potential multicollinearity issues, we assessed the variance inflation factors (VIFs) for the six competency predictors. VIF values ranged approximately from 3 to 5, indicating moderate multicollinearity. To mitigate these multicollinearity effects, we refitted the logistic regression model using Lasso regularization. Even under this penalized regression, Competencies 6 (Crisis Management & Leadership) and 4 (Progress Management) remained the most influential predictors, while coefficients for less influential competencies moved closer to zero. This analysis confirms that our conclusions regarding the importance of these competencies are robust and not unduly affected by multicollinearity

In summary, the logistic model confirms that Competencies 6 (Leadership) and 4 (Progress Management) are the most critical success factors, while competencies 3 (Response Operations) and 5 (Communication), despite being important, are somewhat less critical once we account for the top factors. This finding makes intuitive sense—a candidate cannot pass without a minimum proficiency in every area (indeed, failing any one section can cause an overall unsuccessful candidate, per the exam rules), but those who excel in leadership and incident management are especially likely to succeed.

3.3.3. Random Forest Analysis

To further explore predictor importance (and to incorporate the granular sub-competencies), we trained a random forest classifier to predict pass/fail based on all 35 sub-competency scores. We then assessed the feature importance values from this model. The results echoed the earlier findings, both at the competency level and in highlighting specific sub-competencies:

Feature importance by competency from the random forest model. Competency 6 (Crisis/Leadership) and 4 (Progress Management) account for roughly half of the model’s predictive importance.

As the bar chart shows (Figure 4), Crisis Management & Leadership (Competency 6) was the most important set of features in the random forest, accounting for about 28% of the total importance. Progress Management (Competency 4) was second at ~22%. Communication (Competency 5) and Response Activities (Competency 3) were next (around 15% and 13%, respectively), followed by Command Decision-Making (Competency 2 ~12%) and Situation Evaluation (Competency 1 ~10%). This ordering is very consistent with both the correlation and regression analyses—reinforcing that leadership and incident management competencies have the greatest influence on the outcome.

At the sub-competency level, the random forest’s top 5 individual predictors included: a leadership sub-skill related to life-saving operations (code 6—4), the efficient communication skill (5—5), another leadership skill (6—3), a decision-making sub-skill (2—4), and a progress-management skill (4—1). This mix indicates that while leadership and management dominate, communication can also surface as a key individual predictor (when it’s the specific high-value communication tasks). Indeed, sub-competency 5—5 (efficient communication, worth 10 points) was among the highest-ranked features, suggesting that mastering that particular aspect of communication differentiates candidates notably in the model. Overall, however, most of the top 10 features came from competencies 6 and 4, which aligns with our earlier finding that those areas had multiple sub-components with large pass/fail gaps. The random forest classifier also exhibited strong predictive ability, with approximately 90% overall accuracy in 5-fold cross-validation (precision and recall both ~0.90), and an AUC of ~0.94, reflecting excellent discrimination between pass and fail cases.

In conclusion, across all methods, the competencies that distinguish successful candidates most are: Crisis Management & Leadership and Progress Management, followed by (at some distance) Command Decision-Making, Communication, and Response Operations, with Situational Evaluation last. In practical terms, candidates who pass tend to demonstrate strong leadership under crisis, effective management of an incident’s progress and resources, solid tactical decision-making, and good communication—whereas those who fail often particularly lack in the leadership and incident management dimensions. All competencies are important (and indeed failing any one can be disqualifying), but excelling in the top areas seems most critical to ensuring an overall pass.

3.3.4. Pattern Discovery via PCA

One additional analysis was performed to see if the overall pattern of performance could visually distinguish passing vs failing candidates. We used Principal Component Analysis (PCA) on the set of competency scores to reduce the six-dimensional performance profile of each candidate into two principal components for visualization.

Below is a PCA scatterplot of candidates’ competency profiles. Each point represents a candidate in the space of the first two principal components, colored by outcome (Pass = teal circles, Fail = red X’s). PC1 (horizontal axis) captures overall performance (~42% of variance), and PC2 (~14% variance) captures secondary variation.

In the scatterplot above (Figure 5), we observe that passing candidates (teal circles) tend to cluster toward the right side (higher PC1 values), whereas failing candidates (red X’s) cluster more toward the left (lower PC1). This indicates that PC1 largely corresponds to an “overall score” factor—essentially a weighted combination of all competencies—which separates most successful candidatessuccessful candidates from unsuccessful candidates. There is still some overlap (as expected, since a few failing candidates had some strong areas and a few passing candidates had some weaker areas), but the general separation along PC1 is evident. The second principal component (PC2, vertical) did not show a clear distinction between pass/fail; rather, it may reflect differences in specific skill mix (for example, candidates who were relatively stronger in one competency vs another). No distinct clusters emerged beyond the pass/fail separation—suggesting that aside from overall score level, there weren’t obvious sub-groupings of candidates with very different skill profile patterns. In summary, the PCA confirms that successful candidatessuccessful candidates as a group have uniformly stronger competency profiles (hence separated along the primary performance axis), while unsuccessful candidates group toward the lower end of that spectrum.

3.3.5. Impact of the Disaster Scenario Environment

Lastly, we examined whether the type of disaster scenario (the virtual environment assigned: the high-rise apartment fire, the gosiwon fire, the karaoke room fire, the residential villa fire, or construction site fire had any systematic impact on candidate performance or pass rates. The scenarios were randomly assigned, so we would not expect large differences unless certain environments are inherently more challenging. Here’s a summary of pass rates by scenario:

The Figure 6 indicates that apparent differences in pass rate (50–62%) are within overlapping confidence bounds, supporting the inference that scenario type did not significantly influence certification outcomes.

Pass rates by scenario environment. Differences of a few percentage points are observed, but none are statistically significant.

As shown above, the pass rate in each environment is roughly in the same ballpark. The Construction site fire scenario had the highest pass rate (~61.5%) while the high-rise apartment fire scenario had the lowest (~50.0%), with the others (Karaoke room fire, Residential villa fire, Gosiwon fire) falling in the mid-50s to 60% range. However, these differences are based on relatively small sample sizes (24–31 candidates in each scenario) and were not statistically significant (χ² test p ≈ 0.91). In other words, there is no evidence that any particular environment conferred an advantage or disadvantage in terms of the likelihood of passing the exam.

We also looked at average scores and competency performance per environment and likewise found no dramatic differences. For instance, candidates in the construction site fire scenario had a slightly higher average total score (~163 out of 200) than those in the residential villa fire scenario (~159 out of 200), but this is a minor gap. Across environments, the competency score profiles were quite similar. One small observation: candidates in the karaoke room fire scenario actually achieved slightly higher average Communication scores (around 84% of communication points on average) compared to those in other environments (~80%). This could suggest that those in a noisy, the karaoke room fire scenario placed extra emphasis on communication. Conversely, candidates in the residential villa fire scenario had marginally lower average scores in the Response Activities competency (~75.6% vs. ~78–79% in other environments), hinting that in the residential villa fire scenario might have posed a slight challenge in operational response. Nonetheless, these differences are subtle. There was no scenario that consistently produced significantly higher or lower performance across the board.

Bottom line: The disaster scenario type did not substantially affect performance or outcomes. All five scenario types saw both passes and fails, and the distribution of scores suggests that the exam’s difficulty remained equivalent regardless of environment. Each scenario may have its unique context (e.g., a construction site might emphasize different hazards than a high-rise apartment), but overall no scenario was disproportionately harder or easier—the candidates’ competencies and skills were the determining factors in passing, rather than the luck of the scenario draw.

4. Discussion

4.1. Implications of Pass Rates Across Cohorts and Scenarios

The analysis of five exam cohorts from 2022 to 2024 reveals that the Intermediate IC Certification is highly competitive. With 141 candidates, only 80 passing, the overall pass rate was about 56%, meaning nearly half of the examinees did not meet the passing criteria. This highlights that simply taking the exam does not guarantee success and that a substantial portion of candidates struggle to achieve the proficiency required. The pass rates also fluctuated across cohorts: for example, about two-thirds (67%) passed in early 2023, whereas only around 40% passed in late 2024. Such variation suggests that factors like cohort composition or external conditions (e.g., preparation levels) may influence outcomes, although we did not find evidence that the exam’s format or settings changed substantially over time. Importantly, while pass rates differ by session, there is a consistent theme: about half of every group typically falls short. In practical terms, this implies that candidates should be well-prepared in all aspects of the exam, and exam administrators might consider this high failure rate when advising trainees or planning support programs.

It is also worth noting that the random assignment of disaster scenarios (Goshiwon Fire, Karaoke Room, Residential Villa Fire, Apartment building fire, Construction Site Fire.) did not lead to any significant differences in pass rates. All scenario types produced similar overall pass/fail proportions (around 50–60% passing) However, small, non-significant trends were observed: for example, candidates in the construction site fire scenario had marginally higher average total scores (~163/200) compared to those in the residential villa scenario (~159/200), and those in the karaoke room fire scenario achieved slightly higher average Communication scores than peers in other scenarios. While these patterns may reflect subtle scenario-specific task demands (e.g., noisy environments emphasizing communication), they do not appear strong enough to influence overall pass/fail outcomes. This uniformity indicates that the competency framework reliably measures the intended skills across different contexts. For candidates and trainers, the practical takeaway is that focusing on scenario-specific tactics may be less important than mastering the underlying leadership and management competencies that span all disaster types.

4.2. Competency Performance Gaps Between Successful and Unsuccessful Candidates

A key finding from the results is that successful candidates outperformed unsuccessful ones across every measured competency, but the extent of the difference varied by skill area. Crucially, our results highlight that candidates who succeed in the certification consistently excel in specific skill domains, notably those related to NTS. In particular, two competency areas showed notably larger gaps between successful candidatessuccessful candidates and unsuccessful candidates: Crisis Management & Leadership (Competency 6) and Progress Management (Competency 4). On average, successful candidatessuccessful candidates scored around 85% of the maximum points in Leadership versus about 71% for unsuccessful candidates, a raw gap of roughly 5.7 points out of 40 (a 14 percentage-point difference). Similarly, in Progress Management, successful candidatessuccessful candidates averaged ~87% compared to ~73% for unsuccessful candidates (roughly 4.9 points out of 35; also ~14 percentage points). These are the largest gaps observed among all competencies.

These findings align with established frameworks in incident command. For example, the UK’s THINCS (The Incident Command Skills) model emphasizes leadership, decision-making, and resource management as key attributes of effective incident commanders, which is consistent with our observation that Crisis Management & Leadership and Progress Management had the greatest performance gaps between successful and unsuccessful candidates. Similarly, the Naturalistic Decision Making (NDM) framework posits that under uncertain, high-stakes conditions, experienced commanders rely on intuitive decision-making and adaptive leadership; this perspective aligns with our finding that candidates excelling in those same two competencies were far more likely to pass. In essence, both frameworks underscore the critical role of strong crisis leadership and dynamic incident management, providing a theoretical explanation for why Competencies 6 and 4 emerged as such strong predictors of exam success in our study.

By contrast, other competencies showed somewhat smaller (though still meaningful) differences. For example, Situation Evaluation (Competency 1) and Response Activities (Competency 3) exhibited gaps in the 10–11 percentage-point range (equating to a few points’ difference on their scales). Command Decision-Making (Competency 2) and Communication (Competency 5) each showed intermediate gaps (~11–12 percentage points). In all cases, successful candidatessuccessful candidates’ scores were significantly higher. Importantly, even the areas with smaller gaps were still areas where the passing group did better on average. Importantly, even in competencies with smaller gaps, the successful group still outperformed the unsuccessful group on average. This uniform advantage suggests that strong candidates tend to perform better across the board, rather than excelling in only one area. A closer look at the 35 sub-competencies further illuminates these patterns. The largest score discrepancies between successful candidatessuccessful candidates and failures occurred in sub-skills under Crisis Management/Leadership and Progress Management. For instance, in the leadership category, sub-competencies like “responding to sudden crisis situations” and “operational execution as a leader” showed roughly 19% gaps in favor of successful candidatessuccessful candidates. In the progress-management category, sub-skills related to prioritization and adapting tactics (e.g., identifying tactical priorities, reporting progress, adjusting measures) had gaps around 16–17%. These large differences indicate that specific high-level skills—such as taking charge during an evolving crisis and keeping track of incident progress—were particularly decisive. Other notable differences included a communication sub-skill (efficient communication) and a response activity (maintaining the tactical situation board), each with gaps on the order of 14–15%.

All of these sub-competency differences were statistically significant, reinforcing that the observed patterns are unlikely due to chance. The fact that the biggest gaps are in complex leadership and management tasks suggests that failure often reflects a weakness in those higher-order skills. Conversely, the relatively smaller gaps in basic situation assessment imply that even some failing candidates have a reasonable grasp of fundamental situational awareness, but they fall behind more when it comes to commanding the response. In summary, passing candidates demonstrated strength not only in knowing what to do (e.g., situation evaluation and basic response) but, more importantly, in leading others and managing an evolving incident. These insights suggest that focusing training efforts on strengthening command presence, decision-making under stress, and adaptive management may yield the greatest improvements for candidates on the borderline of passing.

4.3. Key Factors in Certification Success

To identify which factors most strongly predict passing the exam, we applied correlation, regression, and machine-learning analyses to the competency scores. Across these methods, a clear pattern emerged: performance in Crisis Management/Leadership (Comp 6) and Progress Management (Competency 4) are the dominant predictors of success.

The simple correlation analysis showed that leadership had the highest correlation with passing (approximately r = 0.77), meaning candidates with higher leadership scores were much more likely to pass. Progress management was nearly as predictive (r ≈ 0.75). Other competencies were positively correlated with passing too, but at lower levels (e.g., Communication r ≈ 0.65, Response Activities r ≈ 0.62, and Situation Evaluation around r ≈ 0.63–0.68). Communication actually showed a slightly higher bivariate correlation with passing than response operations did, suggesting that good communication is somewhat more associated with success than response tasks in a one-on-one sense. However, all six competencies were significantly interrelated and with passing, indicating that strong candidates tended to do well in multiple areas.

We then used logistic regression to control for overlap among competencies. Here again, Leadership (Competency 6) and Progress Management (Competency 4) emerged as the most significant unique contributors to the likelihood of passing. In the presence of all other competencies, higher scores in those two areas most strongly increased the predicted probability of success. In contrast, Communication and Situation Evaluation had the smallest unique effects in this multivariate context. This does not mean communication is unimportant—indeed, many candidates who failed had lower communication scores—but rather that once a candidate’s leadership and management abilities are accounted for, communication adds less incremental predictive power. In practical terms, this suggests that excelling in leadership and incident management may be necessary to pass, whereas good communication alone cannot compensate if leadership is weak. Moreover, because the exam format requires meeting a minimum standard in each competency (such that failing any one section results in an overall failure), a broad base of competence is necessary; however, even with this baseline, candidates who excelled in leadership and progress management had a substantially higher probability of passing.

The random forest analysis, which included all 35 sub-competencies, reinforced these conclusions and highlighted specific skills. It showed that the pool of leadership sub-skills was the single largest contributor to predictive power (around 28% of the model’s feature importance), followed by progress-management skills (about 22%). Communication and response sub-areas contributed moderately (roughly 15% and 13% respectively), with decision-making and situation assessment trailing a bit behind. Among individual sub-skills, top predictors included a life-saving leadership task, efficient communication, another leadership skill, a decision-making task, and a tactical progress-management skill. Notably, an efficient communication sub-skill (5—5) was among the highest-ranked individual predictors, indicating that certain aspects of communication can have a large impact on success when isolated.

Taken together, these analyses paint a consistent picture: success on the Intermediate IC Certification is most strongly determined by advanced leadership and incident management abilities. Candidates who scored high in these areas had a considerably higher chance of passing, even when controlling for other factors. Competencies like response tactics and situational awareness, while still positively related to success, appear to be secondary factors. Our empirical findings also reinforce established theory. In particular, the strong predictive power of the leadership and crisis-management competencies supports the NDM theory assertion that intuitive recognition and adaptive decision-making under uncertainty are critical to effective incident command. Likewise, identifying these competencies as high-impact provides empirical validation for models like THINCS, which posit that leadership and resource-management skills are central to successful incident command. This alignment with theory echoes real-world expectations for incident commanders: while assessing the situation is fundamental, the ability to make strategic decisions, motivate teams, and adapt plans on the fly is what truly distinguishes effective leaders in the field.

Training programs might do well to place extra emphasis on crisis leadership exercises, decision-making under pressure, and progress-tracking skills, since these are the areas that most differentiate successful candidates from unsuccessful candidates. Additionally, candidates could self-assess on these high-impact competencies and seek targeted practice if needed. Examiners, in turn, can be confident that no scenario type biases the outcome, as the exam appears to measure the intended skills consistently across different scenario contexts. Overall, while all measured competencies are important, they do not all carry equal weight in determining success: leadership and comprehensive incident management skills emerged as the most critical for passing the Intermediate IC Certification. Finally, given our study’s observational design and single-country context, we emphasize that these relationships are correlational, and our policy recommendations remain preliminary, requiring further validation and testing in diverse organizational contexts.

5. Conclusions

5.1. Theoretical Contribution

Empirical evidence has firmly established that simulation-based training can enhance ICs’ decision-making performance [1]. Our theoretical contribution is to link naturalistic decision-making (NDM) theory with structured simulation training, explaining that scenario-based exercises act as accelerated experience to expand IC’s repertoire of recognized patterns for intuitive decision-making [8,9]. This perspective extends Klein’s Recognition-Primed Decision model by demonstrating how deliberate practice in varied, realistic scenarios can cultivate expert intuition, even as real incident opportunities decline. Furthermore, by aligning these cognitive insights with established incident command competencies (e.g., leadership, communication, situational awareness), we offer a unified framework that bridges decision-making theory with practical competency development in a safe yet realistic environment.

5.2. Practical Implications

Our findings indicate that leadership and the ability to manage incident progress were the strongest predictors of successful IC certification [22]. This approach builds the “forecasting” ability of ICs to anticipate and control situational changes [23]. Training programs should therefore prioritize developing these NTS (non-technical skills)—beyond technical firefighting expertise—by incorporating modules on decision-making under stress [14,24], team leadership, and dynamic incident planning. Likewise, certification policies should be updated so that assessments (for example, realistic scenario-based evaluations) directly examine these critical competencies in action. This focus on leadership and progress management aligns with international best practices (for instance, the UK’s THINCS fire command framework emphasizes effective leadership and planning [25]), suggesting that strengthening these skills will improve incident command performance across different contexts.

5.3. Limitations and Future Research Directions

Methodologically, this study has several limitations. First, the sample size was relatively small, which undermines statistical power and limits the generalizability of the findings [26]. Second, the random assignment of different simulation scenarios to candidates—while intended to ensure fairness—introduced variability in scenario difficulty. Uncontrolled differences between scenarios may have influenced performance outcomes, complicating the comparison of candidates under uniform conditions. Third, the research was confined to the Korean fire service context, meaning the competency model and results may not directly generalize to other countries or organizational settings [27]. Fourth, the blind assessment recorded only field command performance scores and did not capture participant-level variables (e.g., age, experience, prior training, education). Future studies should include these characteristics to analyse their influence on performance.

Analytically, the exclusive reliance on quantitative models is a concern. Without any qualitative input (e.g., interviews or debriefs), important contextual insights may have been missed in interpreting performance [26]. For instance, PCA relies on the assumption that complex performance data can be represented by a reduced set of orthogonal components. However, this method condenses many variables into a few composite factors, which may not correspond clearly to distinct competencies and thus limit the direct interpretability of results. Likewise, while the random forest algorithm is effective for prediction, it operates as a “black box” model with little transparency in its decision-making or the contributions of individual predictors [28]. This lack of transparency makes it difficult to translate model outputs into practical guidance for competency development. Finally, given the observational and correlational nature of this study, we cannot definitively establish that the identified competencies (e.g., decision-making under pressure) directly cause success or failure in certification outcomes. Accordingly, these findings should be viewed as associations rather than as evidence of causation.

Future research should address these limitations to enhance IC competency assessment. Key directions include:

AI-enhanced simulation tools: Adopt advanced VR/AI simulations to provide realistic, adaptive, and safe competency evaluations [29].

Longitudinal tracking: Monitor candidates post certification to evaluate long-term skill retention and development with field experience.

Qualitative insights: Conduct interviews or debriefings to capture contextual factors behind performance, complementing quantitative metrics.

International comparisons: Validate the competency assessment framework across diverse countries to identify best practices internationally.

Curriculum refinement: Use assessment findings to update training curricula, targeting identified sub-competency weaknesses [30].

These directions, along with larger and more diverse samples, will help build a robust and generalizable framework for assessing IC competencies.