Domain Knowledge-Enhanced Large Language Model Framework for Automated Multiple Choice Question Option Generation in Construction Safety Assessment

Abstract

Construction sites implement various safety management activities, including toolbox meetings, risk assessments, and safety knowledge assessments, to reduce accidents. Multiple-choice question (MCQ)-based assessments are widely used to evaluate worker safety competencies. However, the effectiveness of MCQ assessments depends critically on distractor quality; incorrect options must be plausible enough to challenge uninformed respondents while remaining clearly distinguishable from knowledgeable ones. Manual distractor creation requires substantial expertise and is prone to inconsistency, whereas large language models (LLMs) often generate options that lack domain relevance. This paper proposes context-aware multipath adaptive safety scoring (CoMPASS), an algorithm that integrates construction safety domain knowledge with LLM capabilities for MCQ distractor generation. CoMPASS operates through two pathways: CoMPASS-H leverages a hierarchical hazard factor ontology for hazard identification questions, whereas CoMPASS-R uses hybrid retrieval-augmented generation (RAG) for risk control questions. An evaluation using 50 real construction accident cases with a robotic assessment test (RAT) using frontier LLMs as virtual examinees demonstrated that CoMPASS-R achieved a 90% quality pass rate, whereas all baseline methods failed to meet the composite quality criteria. The proposed framework provides a scalable approach to generating assessment content that supports effective safety management at construction sites.

1. Introduction

The construction industry is among the most hazardous industries worldwide. According to the International Labor Organization, construction workers make up approximately 7% of the global workforce but account for a disproportionately large share of occupational fatalities. Fatality risks are estimated to be three to four times higher than in other industries in developed countries and three to six times higher in developing nations [1,2,3]. International comparisons reveal persistently elevated fatality rates across countries: South Korea recorded an occupational fatality rate of 16.1 per 100,000 construction workers—3.7 times the national average—while the United States reported 9.4 per 100,000 full-time equivalent workers, 2.6 times the all-industry average [3,4]. In South Korea, the construction industry has consistently had the highest fatality rate among all industries, with a rate of 1.57 per 10,000 workers by 2024, approximately four times the national average of 0.39 [5]. Notably, small-scale construction sites with fewer than 50 workers represent 91.5% of all construction sites but account for 72.3% of all fatalities, with mortality rates 4.43 times higher than those of larger sites [2]. Prior research has identified inadequate hazard recognition and improper risk control selection as the primary contributors to construction accidents [2,3]. To address these issues, construction sites have implemented various safety management activities, including toolbox meetings, risk assessments, safety inspections, and safety knowledge assessments. Among these approaches, multiple-choice question (MCQ)-based assessments are widely used to evaluate and reinforce workers’ ability to identify hazards and select appropriate control measures, offering objectivity, scalability, and ease of automated scoring [6]. However, the effectiveness of MCQ-based assessments depends heavily on the quality of the distractors—incorrect options that must be plausible yet clearly distinguishable from the correct answer.

Currently, construction safety MCQs are predominantly developed manually by domain experts, a process that is time- and resource-intensive [7]. Among the components of MCQ development, creating high-quality distractors is the greatest challenge [6]. Haladyna et al. [6] established that effective distractors must be plausible for examinees who lack the requisite knowledge, yet unambiguously incorrect for those who possess it, a principle that requires deep domain expertise and careful consideration of common misconceptions. Pedro et al. [8] further noted that conventional methods for assessing safety knowledge fail to determine whether learners possess the understanding required for practical applications, often encouraging superficial retention rather than genuine comprehension. In practice, manually developed distractors often show inconsistent quality, with some being too obviously incorrect and thus easily eliminated, whereas others inadvertently overlap with the correct answer, creating ambiguity [7]. This inconsistency, combined with the high cost of expert involvement, underscores the need for automated approaches to distractor generation.

The MCQ-based assessment discussed in this study evaluates worker competencies in two critical domains: hazard identification (recognizing direct causal factors in accident scenarios) and risk control method selection (choosing appropriate preventive measures for specific hazards). These domains directly correspond to key failure points identified in accident analyses: workers who cannot accurately identify hazards or select proper control measures are more likely to engage in unsafe behaviors. MCQs targeting these domains require distractors that test whether examinees can distinguish between correct answers and superficially similar but incorrect concepts, a task that demands deep-domain expertise when performed manually.

Recent advances in large language models (LLMs) and retrieval-augmented generation (RAG) have created new opportunities for automated content generation in specialized domains. In the construction sector, Uhm et al. [9] demonstrated that RAG-enhanced GPT models can generate accurate and relevant safety information, whereas Hu et al. [10] successfully applied RAG-LLM pipelines to automated feedback generation in construction management. Comprehensive surveys have documented the evolution of distractor generation from conventional rule-based and ontology-based methods to neural networks and pretrained language model approaches [11], with systematic reviews identifying fine-tuning, prompt engineering, and RAG as the three primary strategies for transformer-based MCQ generation [12]. However, LLM-generated distractors have shown significant quality gaps compared with human-authored distractors. Feng et al. [13] reported that LLM-generated distractors received a mean validity score of 3.28 out of 5.0, significantly lower than the 3.99 achieved by human experts (p < 0.05). This gap is particularly concerning in specialized domains, such as construction safety, where domain-specific knowledge is essential for creating contextually appropriate distractors. Without domain guidance, LLM-generated distractors tend to fall into two failure modes: being semantically too similar to the correct answer, thereby creating ambiguity; or being contextually irrelevant, making them easily eliminated by any reasonably knowledgeable examinee.

Overcoming this challenge requires integrating domain-specific knowledge structures into the LLM-based generation process. In construction safety, ontological representations of hazard factors and their interrelationships can provide a principled basis for identifying distractor candidates that are sufficiently similar to be plausible but sufficiently different to be incorrect. Park and Shin [14] noted that comparatively little attention has been paid to the systematic formation, management, and utilization of safety-related knowledge, identifying a significant gap in the effective dissemination and practical application of safety knowledge on construction sites. Similarly, Xu et al. [15] highlighted that construction is a knowledge-intensive industry in which domain knowledge structures must be continuously expanded and improved; however, conventional expert-dependent approaches suffer from individual knowledge limitations and escalating time costs. Although construction safety ontologies have been developed for accident classification [14] and domain knowledge entity extraction [15] and retrieval-augmented approaches have been explored for general distractor generation with knowledge graph integration [16], their combined application to MCQ distractor generation in the construction safety domain remains unexplored. While knowledge-enhanced approaches combining retrieval mechanisms with structured knowledge have improved distractor quality in general domains [16], these frameworks rely on generic knowledge structures that do not account for domain-specific hierarchical taxonomies, sector-specific accident co-occurrence patterns, or authoritative regulatory knowledge bases. To date, no systematic methodology has been proposed for leveraging domain ontologies and RAG to control the quality of LLM-generated distractors for construction safety assessments.

In this study, context-aware multipath adaptive safety scoring (CoMPASS) was proposed, an algorithm that integrates construction safety domain knowledge with LLM capabilities to generate high-quality MCQ distractors. CoMPASS operates through two specialized pathways. CoMPASS-H leverages ontological similarity and co-occurrence patterns to select plausible yet incorrect hazard factors for hazard identification questions, whereas CoMPASS-R uses a hybrid RAG combined with sentence similarity and ontological distance to identify contextually related but inappropriate risk control measures for risk control questions. To rigorously evaluate the effectiveness of the algorithm, CoMPASS-generated distractors were compared against a baseline using the same LLM (GPT-4.1) without domain rules, thereby isolating the contribution of domain knowledge integration. To address the technical challenges of computing the embedding-based similarity for short Korean construction terms and lengthy safety guideline texts, definition expansion and text summarization preprocessing techniques were introduced. A comprehensive evaluation framework is proposed comprising the incorrectness score (IS), plausibility score (PS), discriminability score (DS), and quality pass rate (QPR) that requires simultaneous satisfaction of all three metric thresholds, along with a robotic assessment test (RAT) that uses multiple frontier LLMs as virtual examinees to assess distractor effectiveness at scale. Figure 1 presents the conceptual framework of this study, illustrating the progression from problem identification and research gaps to the proposed solution and its evaluation.

Figure 1. Conceptual framework of this study.

2. Literature Review

2.1. LLM and RAG Applications in Construction Safety

The construction industry generates substantial volumes of unstructured textual data, including accident reports, safety regulations, inspection records, and training materials that contain valuable knowledge for safety management. Early applications of natural language processing (NLP) in this domain focused on supervised classification and information extraction. Shin et al. [3] developed a KLUE-BERT-based model for classifying 16 site information parameters from accident reports and retrieving similar historical accidents, demonstrating the feasibility of deep language models for construction safety text analysis. Choi et al. [1] employed machine learning methods on national accident data, achieving a 91.98% classification accuracy for predicting fatal construction accidents using features such as seasonality and employment scale.

The emergence of LLMs has substantially expanded the scope of NLP applications in construction safety by enabling not only classification but also the generation and reasoning over safety knowledge. Retrieval-augmented generation (RAG), which integrates document retrieval with LLM generation to reduce hallucinations and ground responses in verified domain knowledge, has proven effective in this context. Uhm et al. [9] showed that RAG-enhanced GPT models could generate accurate and relevant construction safety information by evaluating output quality across seven criteria (correctness, accuracy, completeness, relevance, efficiency, usefulness, and accessibility) with three evaluator groups: researchers, professionals, and field workers. Hu et al. [10] extended this work by applying RAG-LLM pipelines to automated feedback generation in construction management and conducted a mixed-methods evaluation with 81 AEC professionals, measuring fairness, usefulness, engagement, and behavioral intention. Lee et al. [17] directly compared RAG with fine-tuned LLMs for construction safety management knowledge retrieval using Korean safety guidelines and found that RAG approaches outperformed fine-tuning for domain-specific information provision, providing theoretical support for RAG-based methods in safety-critical applications.

Beyond text-based applications, Wang et al. [18] proposed an integrated approach for automatic safety inspection by combining multimodal LLMs with domain knowledge, demonstrating that structured domain knowledge enhances LLM performance for visual safety assessments. Sammour et al. [19] systematically evaluated LLM capabilities in construction safety using various prompt engineering strategies, offering a comprehensive analysis of both the potential and limitations of current LLMs in this context.

Despite these advances, current LLM and RAG applications in construction safety have focused mainly on information retrieval, content generation, and feedback provision. The use of these technologies for the quality control of assessment items, specifically to ensure that LLM-generated MCQ distractors are both plausible within the construction safety domain and unambiguously incorrect, remains unexplored. This gap is significant because distractor quality in safety assessments directly affects the validity of competency evaluations.

2.2. Automated MCQ and Distractor Generation

The quality of multiple-choice questions depends critically on the design of the distractors. Haladyna et al. [6] synthesized 31 item-writing guidelines, establishing the foundational principle that effective distractors must be plausible to examinees who lack the requisite knowledge yet unambiguously incorrect for those who possess it. Building on this principle, the present study identifies three essential properties of high-quality distractors: incorrectness (the distractor must be factually wrong), plausibility (it must appear reasonable to insufficiently knowledgeable examinees), and discriminability (it must differentiate between knowledgeable and unknowledgeable examinees). These three properties form the theoretical basis of the evaluation framework presented in Section 4.

Conventional approaches to automated distractor generation rely on structured knowledge. Patra and Saha [20] used WordNet synset relations and statistical methods to generate named entity distractors using a hybrid rule-based and statistical approach; however, this method is limited by the ambiguity inherent in general-purpose lexical databases. Ren and Zhu [21] proposed a knowledge-driven distractor generation framework using WordNet as the primary knowledge base with feature-based learning-to-rank models, representing one of the most cited studies in this area. However, both approaches are constrained by the coverage and domain specificity of general knowledge resources, such as WordNet, which lack the specialized terminology and conceptual relationships required for technical domains, such as construction safety. Bitew et al. [22] adopted an ontology-agnostic approach, proposing the reuse of distractors from existing MCQ databases based on contextual similarity, which demonstrated scalability but sacrificed domain specificity.

The introduction of pretrained language models and LLMs has transformed the generation of distractors. McNichols et al. [23] explored in-context learning for simultaneous distractor and feedback generation in mathematics education by introducing distribution-based evaluation metrics that compare generated distractor selection patterns with actual student response distributions. Feng et al. [13] conducted a comprehensive study on LLM-based distractor generation for mathematics MCQs, proposing both alignment-based metrics (exact, partial, and proportional matches) and human evaluation criteria, including validity and plausibility on a 5-point scale. Their findings revealed a significant quality gap: LLM-generated distractors received a mean validity score of 3.28 compared to 3.99 for human-authored distractors (p < 0.05), demonstrating that current LLMs alone cannot match expert-level distractor quality.

Comprehensive surveys in this field have confirmed both the rapid progress and the remaining challenges. Alhazmi et al. [11] provided a thorough survey of distractor generation methods, datasets, and evaluation approaches, tracing the evolution from traditional rule-based methods through neural networks to pretrained language models while noting the absence of standardized evaluation frameworks. Awalurahman et al. [12] conducted a systematic literature review of transformer- and LLM-based MCQ generation, analyzed 28 primary studies, and identified three critical gaps: the scarcity of RAG-based approaches, limited use of domain-specific knowledge structures, and the unresolved challenge of automatic quality evaluation.

Most recently, Yu et al. [16] proposed enhancing distractor generation through retrieval-augmented pre-training combined with knowledge graph integration, representing the closest prior work to the present study. Their approach demonstrated that combining retrieval mechanisms with structured knowledge improved distractor quality. However, their work was applied only to general educational domains and did not address the unique requirements of technical safety assessments, where domain ontologies must capture not only semantic similarities but also the safety-critical distinction between correct and incorrect hazard classifications or risk control measures.

Collective evidence from this literature establishes that (1) LLMs alone produce distractors of insufficient quality for high-stakes assessments, (2) domain knowledge integration improves the generation quality, and (3) no existing study has applied knowledge-enhanced distractor generation in the construction safety domain.

2.3. Ontology-Based Knowledge Representation in Construction Safety

Construction safety knowledge is inherently structured: hazard factors exist in hierarchical classifications, risk control measures are organized by applicable scenarios, and causal relationships connect workplace conditions with accident outcomes. Formally representing this structure using ontologies and knowledge graphs enables computational reasoning about safety knowledge.

Several studies have developed ontological frameworks to improve construction safety. Park and Shin [14] proposed a safety domain ontology for fall accidents organized into three superclasses: construction class (BIM-based building elements), conduct class (construction activities and processes), and safety control class (OSHA regulation-based safety information). This ontology was developed using the Protege editor with OWL reasoners and SWRL rules. Their work highlighted that limited attention has been given to the systematic formation, management, and use of safety-related information and knowledge, identifying a significant gap in the practical application of safety knowledge on construction sites. Xu et al. [15] introduced the concept of domain knowledge entities (DKEs), defined as the most basic units of domain knowledge structures representing thinking units with complete logic, and developed NLP-based methods for automatically extracting hierarchical relations (affiliation and parallel) among DKEs from 68,817 Chinese construction documents for metro construction safety risk management.

At the classification level, Hwang et al. [2] applied statistical analysis, LDA topic modeling, and network analysis to systematically identify critical factors and trends in fatal accidents at small-scale construction sites in Korea, revealing patterns in accident causation that inform hazard classification. Choi et al. [1] developed machine learning predictive models using national accident data relying on structured categorizations of accident types, seasons, and employment scales to predict fatality risk.

Despite these advances in ontology construction and knowledge structuring, existing research has primarily applied construction safety ontologies to information retrieval, accident classification, and risk prediction. The potential of ontological relationships—specifically, the similarity and co-occurrence patterns among hazard factors within a classification hierarchy—to systematically identify candidates that are “similar enough to be plausible, yet different enough to be incorrect” for MCQ distractors has not been investigated.

2.4. Research Gaps and Objectives

Based on a comprehensive literature review, three critical research gaps that motivated this study were identified.

(1) Gaps in domain-specific distractor generation algorithms. While LLM-based approaches have been successfully applied to various construction safety tasks—including safety information generation [9], visual safety assessment [18,24], and regulatory compliance checking [25]—no study has addressed the unique challenge of generating educationally valid MCQ distractors for construction safety assessment. Although knowledge-enhanced distractor generation combining retrieval mechanisms with knowledge graph integration has been explored in general domains [16], these approaches employ generic knowledge structures and do not account for the specialized two-level hazard taxonomy, accident-context-specific co-occurrence patterns, or regulatory knowledge bases unique to construction safety. Distractor generation fundamentally differs from other LLM applications; whereas safety monitoring aims to identify correct information or detect violations, distractor generation requires systematically identifying concepts that are plausibly related to the correct answer but are factually incorrect in the given context. This reversal of the typical LLM objective—from “find correct” to “find plausibly incorrect”—necessitates a specialized methodological approach that the current research has not provided.

(2) Gaps in the automated quality evaluation framework. Existing evaluation frameworks for LLM-generated content in construction safety predominantly rely on expert human judgment. Uhm et al. [9] employed 42 human evaluators across three stakeholder groups, whereas Kim et al. [24] combined GPT-4V-based metrics with human expert assessment. However, such manual evaluation approaches face significant scalability limitations, particularly for iterative algorithm development, which requires rapid feedback cycles. The distractor generation literature has proposed alignment- and distribution-based metrics [13,23]; however, these have not been adapted to the construction safety domain, where domain-specific validity criteria are applied. The absence of automated domain-appropriate evaluation methods specifically designed for MCQ distractor quality assessments hinders the development and validation of systematic algorithms.

(3) Gaps in ontology applications for content-generation assessment construction safety ontologies have been developed and applied for accident classification [2], real-time safety monitoring [26], regulatory compliance checking [25], and hazard identification [14]. However, their potential for educational assessment applications, specifically for the systematic selection of distractor candidates based on ontological relationships, remains uninvestigated. The hierarchical structure of hazard factor classification and co-occurrence patterns among factors in similar construction contexts provide a principal basis for identifying “plausible but incorrect” options. This capability has not been leveraged in existing research on distractor generation.

To address these research gaps, the objective of this study was to achieve the following milestones:

(1) Develop a context-aware multipath adaptive safety scoring (CoMPASS) algorithm that integrates a hierarchically structured hazard factor ontology and a RAG-based risk control knowledge base with LLM capabilities for domain-guided distractor generation across two assessment types: hazard identification and risk control method selection.

(2) Establish a four-metric evaluation framework comprising the IS, PS, DS, and QPR, along with an automated RAT methodology that uses frontier LLMs as virtual examinees to measure distractor discriminability without requiring human evaluators.

(3) Validate the effectiveness of the proposed approach through a comparative evaluation against an LLM-only baseline using identical generation models, isolating the contribution of domain knowledge integration for 50 Korean construction accident cases.

3. Proposed Method: CoMPASS Algorithm

3.1. Overall Framework

This paper presents CoMPASS, an algorithm designed to generate high-quality MCQ distractors for construction safety assessment by integrating domain knowledge with LLM capabilities. The algorithm is based on three design principles: (1) using the same LLM (GPT-4.1) for both the proposed method and the baseline, with domain knowledge rules as the only differentiating factor, thereby isolating the contribution of domain knowledge integration; (2) using multiphase verification that combines statistical extraction with LLM-based contextual validation; and (3) leveraging two complementary domain knowledge sources, a hazard factor ontology and a risk control knowledge base, to address different question types.

CoMPASS operates through two specialized pathways corresponding to two fundamental types of construction safety assessment questions. CoMPASS-H (hazard identification pathway) generates distractors for questions that require examinees to identify the direct causal factor (hazard) of a given accident scenario. CoMPASS-R (risk control pathway) generates distractors for questions that require examinees to select the most appropriate risk control measure for a given accident scenario.

The overall system takes as input a construction accident case record comprising an accident description narrative, identified causal hazard factor, and accident type classification. It produces a five-option MCQ with one correct answer and four domain knowledge-guided distractors as output. Figure 2 shows the overall architecture, the two pathways, and their interactions with external knowledge sources.

Figure 2. Overall architecture of the CoMPASS algorithm.

3.2. Construction Accident Data and Ontology Structure

3.2.1. Accident Case Data Structure

The system uses construction accident data sourced from a Korean construction accident database. Each accident record includes the following fields: (1) accident description narrative—a free-text account of the accident circumstances; (2) causal hazard factor at the mid-level classification—the object or condition directly responsible for the accident, which serves as the correct answer for hazard identification questions; and (3) accident type classification—the category of accident outcome (e.g., fall, struck-by, caught-in). The database contains accident records collected from Korean construction sites, structured according to the Korean Occupational Safety and Health Agency (KOSHA) classification standards. Records are categorized by facility type, work type, accident type, and causal hazard factor following the standardized two-level hazard factor taxonomy.

3.2.2. Hazard Factor Ontology

The hazard factor ontology provides a two-level hierarchical classification of the causal factors in construction accidents. The upper-level groups hazard factors into broad categories (e.g., temporary structures, construction equipment, materials), while the lower level specifies individual hazard factors within each category (e.g., scaffolding, formwork, ladder, crane). This classification is based on the Korean Industrial Accident Classification System maintained by the Ministry of Employment and Labor.

Two metrics derived from this ontology are central to the CoMPASS-H pathway:

• Ontological Similarity: The ontological similarity between two hazard factors A and B quantifies their conceptual proximity within the classification hierarchy.

  $$OntSim(A,B)=\{\begin{array}{cc}1.0\hfill & if A=B\hfill \\ 0.8\hfill & if Category\left(A\right)=Category\left(B\right)\hfill \\ 0.0\hfill & otherwise\hfill \end{array}$$

  (1)

  This discrete metric reflects the principle that hazard factors within the same upper-level category are conceptually related and, therefore, more plausible as distractors, whereas those in different categories are too dissimilar to be convincingly incorrect options. The discrete formulation was adopted because the Korean construction safety hazard classification follows a two-level hierarchy (major category → mid-level factor), which limits the granularity available for continuous distance measures. A tree-distance metric or graph-based similarity would require a deeper taxonomy with multiple intermediate levels to produce meaningful continuous values. The three-level approximation (1.0/0.8/0.0) captures the primary categorical distinction available in this two-level structure. Future work with richer ontological structures could benefit from continuous similarity measures.
• Co-occurrence: The co-occurrence of two hazard factors measures the degree to which they appear in similar construction contexts, defined as shared (facility type, work type) combinations.

  $$CoOcc(A,B)={\displaystyle \frac{\left|C_A\bigcap C_B\right|}{\left|C_A\right|}}$$

  (2)

  where $C_A$ denotes the set of (facility type, work type) context pairs in which hazard factor A appears across all accident records. This data-driven metric captures empirical confusion patterns; hazard factors that frequently co-occur in similar construction contexts are more likely to be mistakenly selected by examinees with incomplete knowledge. The asymmetric formulation was an intentional design choice. It measures the proportion of the correct hazard factor’s accident contexts that also feature the candidate, directly quantifying contextual confusion potential from the perspective of the correct answer. A symmetric alternative (e.g., Jaccard index: |$C_A$ ∩ $C_B$|/|$C_A$ ∪ $C_B$|) would additionally penalize candidates that appear in many unrelated contexts, a property less relevant to distractor plausibility. The asymmetric design prioritizes the recall of shared contexts—identifying candidates that overlap with the correct answer’s contextual footprint—over precision.
• Weighted Combination: In Phase 1 of CoMPASS-H, the two metrics are combined as a weighted sum:

  $${Score}_H\left(c\right)=\alpha ·OntSim\left(correct,c\right)+\beta ·CoOcc(correct,c)$$

  (3)

  where α = 0.6 and β = 0.4. A higher weight is assigned to ontological similarity to prioritize conceptual proximity, whereas co-occurrence provides a data-driven correction that accounts for real-world confusion patterns. The weight assignment reflects a design principle: ontological similarity establishes the primary categorical constraint ensuring that distractors originate from the same safety domain, while co-occurrence provides a secondary contextual adjustment. Importantly, the discrete nature of OntSim ({0.0, 0.8, 1.0}) ensures that candidate rankings within the same ontological tier are mathematically invariant to the choice of α, as within-tier ordering depends solely on co-occurrence values. Cross-tier rank reversals require substantial co-occurrence gaps (e.g., exceeding 0.300 at α = 0.6 for a 0.8-tier candidate to outrank a 1.0-tier candidate), constraining the practical impact of weight perturbation on the final distractor selection.

The hazard factor terms in the Korean classification system are typically short (2–4 characters). This brevity poses challenges for embedding-based similarity computation, which is addressed using a definition-expansion technique. We acknowledge that using LLM-generated definitions introduces a potential circularity, as the evaluation metrics (IS and PS) partially depend on LLM outputs. However, three factors mitigate this concern: (1) the 119 definitions are cached and reused identically across all methods and questions, ensuring consistent treatment between CoMPASS and baseline conditions; (2) the definitions serve a narrowly scoped role—converting short Korean terms into embedding-compatible representations—rather than influencing distractor selection logic; and (3) the definitions were manually inspected for factual accuracy against the official Korean construction safety classification standards.

3.2.3. Risk Control Knowledge Base

The risk control knowledge base stores verified risk control measures in a structured database, each containing control measure text (describing a specific safety action), target hazard factor, and applicable accident type. This knowledge base serves as the retrieval corpus for the CoMPASS-R pathway.

To retrieve the correct answers to the risk control questions, the system uses a Hybrid RAG approach that combines keyword matching with a vector similarity search. Given an accident record, the system constructs a search query from the hazard factor and accident type, generates a query embedding using text-embedding-3-large (2000 dimensions), and executes a hybrid search function that weighs semantic similarity at 0.7 and keyword relevance at 0.3. The top-ranked result was selected as the correct risk control measure. Because the raw guideline texts were lengthy (approximately 1000 characters), they were summarized into single-sentence safety measures (30–60 characters) using GPT-4.1 before being used as MCQ options.

3.3. CoMPASS-H: Hazard Identification Pathway

CoMPASS-H generates distractors for questions in the following form: “What is the direct causal factor of the above accident?”.

3.3.1. Phase 1: Candidate Extraction

Phase 1 identifies the 20 most plausible distractor candidates from the hazard factor ontology. For each hazard factor c in the ontology (excluding the correct answer), the algorithm computes the ${Score}_H\left(c\right)$ and ranks all candidates in descending order. The top 20 are retained for contextual verification in Phase 2.

Selecting 20 candidates provides sufficient redundancy to ensure that at least four high-quality distractors survive the LLM verification in Phase 2 while limiting the number of costly LLM API calls.

3.3.2. Phase 2: LLM-Based Irrelevance Verification

Phase 2 uses GPT-4.1 to verify that each candidate is contextually appropriate as a distractor for a specific accident scenario. Statistical similarity alone cannot guarantee contextual appropriateness: a hazard factor may be ontologically similar to the correct answer, yet entirely implausible given the accident narrative.

The LLM evaluates each candidate against the accident description and classifies it into one of four ACCEPT categories or REJECT:

• Observable + Associated: The candidate is explicitly mentioned in the accident narrative but is not the direct causal factor—creating a compelling distractor for examinees who confuse associated factors with the direct cause.
• Contextual + Proximate: The candidate relates to the work location or task context but does not cause the accident, targeting examinees who mistake contextual proximity for causal responsibility.
• Surface Level: The candidate is a sub-component, super-component, or closely related element of the correct answer, testing the examinees’ precision in hazard identification.
• Misleading: The candidate is inferable from accident circumstances but not explicitly stated, exploiting the tendency to assume unstated but plausible causal factors.

Candidates classified as REJECT, either because they are the correct answer or are too implausible, are filtered out. The LLM is called with a temperature of zero for deterministic output.

3.3.3. Phase 3–4: Final Selection and Assembly

In Phase 3, the top four validated candidates (based on their Phase 1 scores) are selected as the final distractors. If fewer than four candidates pass Phase 2 verification, the algorithm supplements the remaining Phase 1 candidates in descending score order.

Phase 4 assembles the five-option MCQ by combining the correct answer with the four distractors and applying a Fisher–Yates shuffle to randomize the option order, preventing positional bias in correct answer placement.

3.4. CoMPASS-R: Risk Control Pathway

CoMPASS-R generates distractors for questions in the following form: “What is the most appropriate risk control measure to prevent this accident?”. This pathway includes an additional Phase 0 for correct answer retrieval and uses similarity metrics tailored to textual risk control measures.

3.4.1. Phase 0: Correct Answer Retrieval

Unlike hazard identification questions, where the correct answer is directly available from the accident record, risk control questions require the retrieval of the most appropriate control measure from the knowledge base. Phase 0 constructs a search query by combining the hazard factor and accident type (e.g., “scaffolding fall risk control measure safety action”), generates a 2000-dimensional embedding using text-embedding-3-large, and executes the hybrid search described in Section 3.2.3. The top-ranked results serve as the correct answers.

3.4.2. Phase 1: Candidate Extraction for Risk Control

Phase 1 identifies the ten most plausible distractor candidates from the risk control knowledge base using two complementary metrics.

• Sentence Similarity: The Jaccard similarity between the correct answer and each candidate control measure.

  $$\mathrm{S}entSim\left(S_1,S_2\right)={\displaystyle \frac{\left|W_{s_1}\bigcap W_{s_2}\right|}{\left|W_{s_1}\bigcup W_{s_2}\right|}}$$

  (4)

  where $W_s$ denotes the set of word tokens in sentence $S$. This metric identifies control measures with similar linguistic structure, which is an important property for plausible distractors, as they should “read like” valid safety measures. While embedding-based cosine similarity would capture deeper semantic relationships, the Jaccard metric was deliberately chosen for the first-stage candidate ranking as a computationally efficient measure that prioritizes structural overlap among safety regulation texts. The subsequent LLM verification phase (Phase 2) compensates for the semantic limitations of token-level matching by applying full contextual reasoning to validate each candidate. Additionally, OntDist complements SentSim by encoding the ontological relationship that Jaccard cannot capture, ensuring that the composite score reflects both structural and categorical dimensions.
• Ontological Distance: The distance between the target hazard factor of the correct answer and the target hazard factor of each candidate, where the target hazard factors are extracted from control measure texts using rule-based keyword matching.

  $$OntDist\left(A,B\right)=\{\begin{array}{cc}0.0\hfill & if A=B\hfill \\ 0.3\hfill & if Category\left(A\right)=Category\left(B\right)\hfill \\ 1.0\hfill & otherwise\hfill \end{array}$$

  (5)

  The combined score for each candidate is:

  $${Score}_R\left(c\right)=\gamma ·SentSim\left(correct,c\right)+\delta ·OntDist({target}_{correct},{target}_c)$$

  (6)

  where $\gamma =0.7$ and $\delta =0.3$. The higher weight on sentence similarity reflects the design principle that the most effective distractors for risk control questions are those that “sound correct” (similar sentence structure) but “apply to the wrong hazard” (different target). Similarly, the discrete OntDist values ({0.0, 0.3, 1.0}) ensure within-tier ranking invariance with respect to γ, as candidates sharing the same ontological distance are ranked solely by sentence similarity regardless of the weight configuration. A cross-tier reversal at γ = 0.7 requires a sentence similarity gap exceeding 0.300, providing a natural stability boundary for the scoring function.

3.4.3. Phase 2: LLM-Based Invalidity Verification

Phase 2 uses GPT-4.1 to verify that each candidate control measure would not actually prevent the accident in question. The key verification question is: “Would this control measure effectively prevent this specific accident?”. If yes, the candidate is REJECTED as a distractor (because a valid preventive measure is not an incorrect answer).

The LLM classifies accepted candidates into four distractor types:

• Administrative Control: The correct answer is an engineering control, and the candidate is a lower-tier measure in the hierarchy of controls (e.g., PPE, training)—plausible but insufficient as a primary prevention measure.
• Different Hazard Target: The candidate has similar sentence structure and action verbs but addresses a different hazard factor—exploiting the tendency to select measures that “sound right” without verifying the target.
• Same Work Type, Different Risk: The candidate addresses the same construction activity but prevents a different accident type, testing whether examinees can distinguish between multiple risks in the same work context.
• Completely Unrelated: The candidate addresses an entirely different type of construction work, serving as an obvious distractor to anchor the difficulty scale.

3.4.4. Phase 3–4: Final Selection and Assembly for Risk Control

The final selection and assembly process mirrors that of CoMPASS-H; the top four validated candidates are selected, supplemented from Phase 1 if necessary, and combined with the correct answer to form a five-option MCQ with a Fisher–Yates shuffled option order.

4. Evaluation Framework

Evaluating the quality of MCQ distractors requires metrics that capture the multiple aspects that make a distractor effective. Previous studies proposed alignment-based metrics that compare generated distractors to reference sets [13,23], distribution-based metrics that compare selection patterns to actual learner responses [23], and human evaluations of validity and plausibility [13]. However, no comprehensive automatic evaluation framework exists for domain-specific distractor-quality assessments. This section presents four complementary metrics: the IS, PS, DS, and QPR, together with a RAT methodology for automated discriminability evaluation.

Figure 3 shows the overall structure of the proposed evaluation framework. The framework comprises four components: distractor quality metrics for embedding-based assessment, a robotic assessment test for discriminability measurement, a baseline design for comparative evaluation, and an experimental setup detailing the dataset and procedures.

Figure 3. Overview of the evaluation framework.

4.1. Distractor Quality Metrics

The IS measures whether a distractor is sufficiently different from the correct answer to avoid creating ambiguity. It is defined as:

$$IS=1-{\displaystyle \frac{e_{correct}·e_{distractor}}{\Vert e_{correct}\Vert ·\Vert e_{distractor}\Vert }}$$

(7)

where $e_{correct}$ and $e_{distractor}$ are the embedding vectors generated by OpenAI’s text-embedding-3-large model with 1536 dimensions. IS ranges from 0 (identical to the correct answer) to 1 (maximally different).

A technical challenge arises when computing the IS for hazard identification questions, where both the correct answer and distractors are short Korean construction terms (typically two to four characters, e.g., formwork and scaffolding). Short-text inputs produce sparse embedding representations that underestimate true semantic relationships, a well-documented limitation of sentence embedding models [27]. To address this, definition expansion was applied: each short term (fewer than 30 characters) was expanded into a definitional sentence (20–40 characters) using GPT-4.1 prior to embedding. For example, formwork can be expanded to a temporary structure installed to mold concrete into a desired shape. This expansion produces semantically richer embeddings that better capture the conceptual relationships between the construction safety terms. A total of 119 domain-specific definitions were generated and cached for consistent use across all IS and PS computations.

For each question, the IS was computed for each of the four distractors, and the mean $\overline{IS}$ was used for the question-level assessment. The acceptable range was defined as $0.3\le \overline{IS}\le 0.9$. The lower bound follows from Haladyna et al.’s [6] warning against ambiguous options: distractors with an IS below 0.3 are semantically too similar to the correct answer, potentially creating multiple defensible answers and undermining assessment validity. The upper bound guards against irrelevant distractors: an IS above 0.9 indicates that the distractor is so dissimilar to the correct answer that any reasonably knowledgeable examinee can immediately eliminate it, rendering it non-functional [13].

The PS measures whether a distractor is contextually related to an accident scenario, a prerequisite for it to function as a convincing, incorrect option. It is defined as:

$$PS={\displaystyle \frac{e_{context}·e_{distractor}}{\Vert e_{context}\Vert ·\Vert e_{distractor}\Vert }}$$

(8)

where $e_{context}$ refers to the embedding of an accident description narrative. The PS ranges from 0 (completely unrelated to the context) to 1 (highly related). Similar to IS, definition expansion is applied to short-text distractors before computing the PS embeddings.

For risk control questions, CoMPASS-R retrieves correct answers and generates distractors from safety guideline texts that are typically lengthy (approximately 1000 characters). To ensure comparability with the concise options expected in the MCQ format, these texts were summarized into single-sentence safety measures (30–60 characters) using GPT-4.1 before being presented to RAT evaluators and used for PS computation.

The acceptable threshold was set at $PS\ge 0.25$. This threshold was calibrated to the embedding characteristics of the evaluation setup: accident descriptions were extended narratives (typically 200–500 characters), whereas the distractors were concise terms or single sentences, resulting in inherently lower cosine similarity values than comparisons between texts of similar lengths and registers. This threshold ensures that distractors maintain meaningful contextual relevance while accommodating the structural asymmetry between context and option embeddings.

IS and PS are complementary; an ideal distractor achieves both high IS (clearly different from the correct answer) and high PS (contextually relevant to the scenario). A distractor with low IS but high PS is problematic because it may be confused with the correct answer. A distractor with high IS but low PS is nonfunctional because it is obviously irrelevant to the scenario.

The DS measures the extent to which distractors effectively deceive evaluation agents and is operationalized through the RAT (Section 4.2). DS is defined as the error rate of the LLM evaluators across all questions generated by the given method.

$$DS={\displaystyle \frac{1}{N\times M}}\sum _{q=1}^N\sum _{m=1}^{M}1(a_{q,m}\ne c_q)$$

(9)

where $N$ is the number of questions, $M$ is the number of evaluator LLMs, $a_{q,m}$ is the answer selected by LLM $m$ for question $q$, and $c_q$ is the correct answer index. DS ranges from 0 (all evaluators always selected the correct answer, indicating trivially easy distractors) to 1 (all evaluators always selected incorrect answers, indicating ambiguous questions).

DS was computed at the method level (i.e., one DS value per generation method) rather than at the question level, following the principle that individual LLM selections cannot be attributed to specific distractors within a question. An appropriate DS range of approximately 0.3–0.6 was expected, corresponding to an evaluator accuracy of 40–70%. This range aligns with the concept of distribution-based evaluation [23] in which effective distractors attract a non-trivial proportion of selections from capable respondents. While method-level aggregation provides a single summary measure, it masks per-question variability. To address this limitation, per-question accuracy distributions are additionally reported to reveal the full range of distractor effectiveness across individual questions.

The QPR is a composite metric that quantifies the proportion of questions whose distractors simultaneously satisfy all three quality conditions. The three quality conditions and their thresholds were: (1) IS ∈ [0.3, 0.9], ensuring distractors were neither trivially dissimilar nor ambiguously close to the correct answer; (2) PS ≥ 0.25, requiring minimum contextual relevance to the accident scenario; and (3) DS ∈ [0.3, 0.6], corresponding to an evaluator accuracy range of 40–70% that indicates nontrivial but not impossibly difficult distractors.

$$QPR={\displaystyle \frac{\left|\{q:\overline{{IS}_q}\in \left[0.3,0.9\right]\right|\bigwedge \overline{{IS}_q}\ge 0.25\bigwedge DS\in \left[0.3,0.6\right]\}|}{N}}$$

(10)

QPR addresses the limitation of individual metrics; a method may score well on one dimension while failing on others. For example, distractors that are clearly incorrect (high IS) but irrelevant (low PS) would pass the IS criterion but fail the QPR. Following the multi-criteria evaluation approach of Uhm et al. [9], who assessed RAG output quality across seven criteria simultaneously, QPR requires all conditions to be met for a question to be classified as passing.

4.2. Robotic Assessment Test

The RAT uses multiple frontier LLMs as virtual examinees to assess distractor effectiveness at scale. This approach is motivated by two considerations: (1) recruiting human construction safety experts for large-scale MCQ evaluation is prohibitively expensive, and (2) frontier LLMs possess sufficient domain knowledge to serve as reasonable proxies for knowledgeable examinees, enabling the automatic evaluation of whether distractors can deceive capable respondents. The RAT is positioned as a scalable automated screening tool rather than a definitive replacement for human evaluation. The validity of LLM-based assessment rests on two observed properties: first, three architecturally distinct LLMs showed consistent accuracy patterns across all methods, suggesting that the results reflect distractor properties rather than model-specific artifacts; second, the binary discrimination task (selecting the correct answer from five options) represents a lower cognitive demand than the contextual reasoning required of human examinees, making RAT a conservative estimate of distractor difficulty. Validating the correspondence between RAT-based and human-based assessments constitutes a critical next step.

Three evaluation LLMs were used to avoid single-model bias: Claude Sonnet 4.5 (Anthropic), Gemini 3 Pro Preview (Google), and GPT-5.2 (OpenAI). All models were accessed via the OpenRouter API, with the temperature set to zero for deterministic responses.

The test protocol presented each LLM with (a) the accident description narrative, (b) the question stem (e.g., “What is the direct causal hazard factor of the above accident?” for hazard identification; “What is the most appropriate control measure to prevent this accident?” for risk control), and (c) five answer options in randomly shuffled order, instructing the model to output only a single digit (1–5) corresponding to its selected answer. Responses were parsed by extracting the first valid digit; outputs that failed to produce a digit between 1 and 5 were classified as invalid. Each question was independently evaluated by all three LLMs, yielding three response data points per question.

The RAT results serve two functions: (1) to compute the DS metric, as defined in Section 4.1, and (2) to provide per-model accuracy rates that characterize how different LLM architectures respond to domain-guided versus unguided distractors. An appropriate accuracy range of 40–70% indicates that distractors are neither trivially easy (accuracy > 90%) nor ambiguously difficult (accuracy < 20%).

4.3. Baseline Design

The baseline was designed to isolate the contribution of domain knowledge integration by controlling for all other variables. Both the baseline and CoMPASS used the same generation LLM (GPT-4.1), input data (50 accident cases), and evaluation framework (IS, PS, DS, QPR, and RAT). The only difference was the presence or absence of domain knowledge rules.

Baseline-H (Hazard Identification) provides the LLM with the accident description, the correct hazard factor, and a list of 15 candidate hazard factors, and then instructs it to select four plausible distractors from the candidates. Unlike CoMPASS-H, this process does not use ontological similarity scoring (Phase 1) or contextual irrelevance verification (Phase 2); the LLM makes selections based solely on its internal knowledge and the provided context.

Baseline-R (Risk Control) provides the LLM with the accident description, the hazard factor, and accident type, then instructs it to generate one correct risk control measure and four plausible distractors from scratch. Unlike CoMPASS-R, this process does not employ RAG-based correct answer retrieval (Phase 0), sentence similarity and ontological distance scoring (Phase 1), or invalidity verification (Phase 2); the LLM generates all content without external domain knowledge grounding.

This controlled design ensures that any observed quality differences between CoMPASS and the baseline can be attributed specifically to domain knowledge integration rather than to differences in the underlying LLM, prompt complexity, or evaluation methodology. An intermediate baseline incorporating RAG retrieval without ontology-based scoring would further isolate the contribution of each component. However, the primary comparison in this study targets the fundamental question of whether domain knowledge integration improves distractor quality at all. If unguided LLMs produce entirely non-functional distractors, the bottleneck is knowledge grounding itself rather than the specific scoring mechanism. Component-wise ablation studies represent a natural extension of this work.

4.4. Experimental Setup

The evaluation dataset comprised 50 randomly sampled accident cases (seed = 42) from the Korean construction accident database. For each case, hazard identification and risk control questions were generated using each method, yielding 200 questions (50 cases × 2 question types × 2 methods). The sample size of 50 cases was determined by balancing statistical adequacy for paired non-parametric comparisons (N ≥ 20 recommended for stable Wilcoxon signed-rank estimates) with the computational cost of the RAT evaluation pipeline, which required 600 LLM API calls across four methods and three evaluator models. The resulting effect sizes (r = 0.358–0.882; Section 5.3) provide strong evidence of genuine quality differences despite the moderate sample size.

Table 1 summarizes the models and parameters used in the experiments.

Table 1. Models and parameters.

Component	Model	Dimensions	API	Temperature
Question generation	GPT-4.1	-	OpenAI	0
Definition expansion	GPT-4.1	-	OpenAI	0
Guideline text summarization	GPT-4.1	-	OpenAI	0
IS/PS embedding	text-embedding-3-large	1536	OpenAI	-
RAG embedding	text-embedding-3-large	2000	OpenAI	-
RAT evaluator 1	Claude Sonnet 4.5	-	OpenRouter	0
RAT evaluator 2	Gemini 3 Pro Preview	-	OpenRouter	0
RAT evaluator 3	GPT-5.2	-	OpenRouter	0

Paired comparisons between the CoMPASS and the baseline were conducted for IS and PS at the question level (N = 50 per question type). The Wilcoxon signed-rank test was used as the primary statistical test, as the paired differences may not follow a normal distribution. The significance level was set at α = 0.05. Effect sizes were reported using rank-biserial correlation (r). DS and QPR were compared at the method level using descriptive statistics, which are aggregate metrics. RAT accuracy was compared across methods and models using a two-way analysis.

5. Results

This section presents the evaluation results comparing CoMPASS-generated distractors against baseline distractors across four metrics: IS, PS, DS, and QPR. The RAT results were then analyzed to characterize the response of different LLM architectures to domain-guided versus unguided distractors. Statistical significance was assessed using the Wilcoxon signed-rank test, with effect sizes reported as rank-biserial correlations (r).

5.1. Overall Distractor Quality

Distractor quality was evaluated across the four metrics for both question types. For hazard identification questions, both methods produced IS values within the acceptable range of 0.3–0.9 (Baseline-H: 0.537; CoMPASS-H: 0.518). The difference was statistically significant with a medium effect size, indicating that CoMPASS-H generates distractors with a lower IS, meaning that they are semantically closer to the correct answer. This is a desirable outcome: CoMPASS-H selects distractors from the same major ontological category as the correct hazard factor, producing options that are conceptually related yet clearly distinct, which increases their potential to deceive examinees. The PS results revealed a trade-off inherent to the ontology-based selection mechanism. CoMPASS-H achieved a lower PS (0.282) than Baseline-H (0.302) with a statistically significant difference and a large effect size. This trade-off has a clear structural explanation: the ontology selects distractors based on taxonomic proximity to the correct answer (same major category), which ensures semantic similarity (lower IS), but does not guarantee contextual relevance to the specific accident scenario (PS). For example, when the correct hazard factor is “formwork” in a formwork collapse accident, the ontology may select “scaffolding” from the same “temporary structures” category—a semantically close distractor (low IS) that may not be directly relevant to the specific accident context (lower PS). The DS for CoMPASS-H (0.293) narrowly missed the acceptable lower bound of 0.3 (shortfall of 0.007), corresponding to an average RAT evaluator accuracy of 70.7%. Consequently, CoMPASS-H achieved a QPR of 0%, failing to meet its own composite quality criterion. While the ontology-based approach improved distractor effectiveness relative to the baseline (DS: 0.293 vs. 0.100; accuracy: 70.7% vs. 88.0%), this result demonstrates that ontology-guided selection alone is insufficient for meeting all three quality thresholds simultaneously. The Baseline-H DS (0.10 indicates that baseline distractors were trivially easy for LLM evaluators (accuracy 88.0%), confirming that without domain knowledge guidance, LLM-generated distractors failed to challenge knowledgeable respondents.

For the risk control questions, CoMPASS-R achieved a substantially lower IS (0.460) than Baseline-R (0.599), with a highly significant difference and the largest effect size observed in this study. CoMPASS-R distractors were semantically closer to the correct answer, meaning that they represent plausible alternative safety measures that could be confused with the correct risk control measure, rather than obviously irrelevant options. This IS improvement was achieved without sacrificing contextual relevance. The PS values were statistically indistinguishable between CoMPASS-R (0.317) and Baseline-R (0.314). This result demonstrates that the RAG-based approach retrieves distractors from the same accident context (matching facility and construction type), preserving contextual plausibility while simultaneously increasing semantic proximity to the correct answer. CoMPASS-R achieved a DS of 0.56, the highest among all four methods and well within the acceptable range of 0.3–0.6. This corresponded to an average RAT evaluator accuracy of only 30.7%, indicating that the CoMPASS-R distractors are highly effective in deceiving the LLM evaluators. In contrast, Baseline-R achieved a DS of 0.00 (accuracy 100.0%), indicating that all three LLM evaluators answered every baseline risk control question correctly—a complete failure of the baseline distractors to function as effective incorrect options.

CoMPASS-R was the only method to achieve a non-zero QPR (90.0%), with 45 out of 50 questions satisfying all three quality criteria simultaneously (IS ∈ [0.3, 0.9], PS ≥ 0.25, DS ∈ [0.3, 0.6]). The five failing questions fell below the PS threshold of 0.25. All other methods achieved a QPR of 0.0%: Baseline-H and Baseline-R failed the DS criterion (0.10 and 0.00, respectively, both below 0.3), while CoMPASS-H narrowly missed the DS lower bound (0.293 vs. 0.3). Table 2 summarizes the distractor quality metrics for the four methods.

Table 2. Distractor quality comparison between CoMPASS and baseline.

Method	IS (Mean ± Std)	PS (Mean ± Std)	DS [95% CI]	QPR (%)
Baseline-H	0.537 ± 0.073	0.302 ± 0.052	0.100 [0.033, 0.180]	0.00
CoMPASS-H	0.518 ± 0.089	0.282 ± 0.052	0.293 [0.187, 0.407]	0.00
Baseline-R	0.599 ± 0.083	0.314 ± 0.063	0.000 [0.000, 0.000]	0.00
CoMPASS-R	0.460 ± 0.084	0.317 ± 0.052	0.560 [0.453, 0.667]	90.0

Note: IS = Incorrectness Score (acceptable range: 0.3–0.9); PS = Plausibility Score (acceptable threshold: ≥0.25); DS = Discriminability Score (acceptable range: 0.3–0.6); QPR = Quality Pass Rate (percentage of questions satisfying all three criteria simultaneously). IS and PS are reported with definition expansion applied.

5.2. Robotic Assessment Test Results

The RAT evaluation assessed distractor discriminability using the three frontier LLMs as virtual examinees. CoMPASS consistently produced lower evaluator accuracy than the baseline for both question types. For hazard identification, CoMPASS-H reduced the average accuracy by 17.3 percentage points (from 88.0% to 70.7%). For risk control, CoMPASS-R reduced the average accuracy by 69.3 percentage points (from 100.0% to 30.7%). The magnitude of the risk control reduction was particularly notable: Baseline-R achieved perfect accuracy across all three evaluator models, indicating that its distractors were completely non-functional, whereas CoMPASS-R reduced the accuracy to approximately 30%, indicating highly effective distractors. The CoMPASS-H average accuracy (70.7%) fell at the upper boundary of the target range (40–70%), indicating appropriately challenging hazard identification distractors. The average accuracy of CoMPASS-R (30.7%) was slightly below the target lower bound of 40%, suggesting that some CoMPASS-R distractors may be overly similar to the correct answer, potentially creating ambiguity. This observation aligns with the low IS value (0.460) and warrants further investigation in future studies involving human subjects.

All three evaluator models showed remarkably consistent behavior for each method. For Baseline-R, all three achieved exactly 100.0% accuracy. For CoMPASS-R, all three converged near 30% (30.0%, 30.0%, and 32.0%), with GPT-5.2 showing marginally higher accuracy. This cross-model consistency suggests that the quality differences between the methods reflect genuine distractor properties rather than model-specific artifacts. For hazard identification, CoMPASS-H reduced accuracy most substantially for GPT-5.2 (−18.0pp, from 86.0% to 68.0%), followed by Gemini 3 Pro Preview (−18.0pp, from 88.0% to 70.0%) and Claude Sonnet 4.5 (−16.0pp, from 90.0% to 74.0%). The relatively uniform reductions across the models confirm that the domain-guided distractors were effective, regardless of evaluator architecture. Table 3 lists the per-model accuracy rates obtained from the RAT evaluation.

Table 3. Robotic assessment test (RAT) accuracy by evaluator model and generation method.

Method	Claude Sonnet 4.5	Gemini 3 Pro Preview	GPT-5.2	Average
Baseline-H	90.0%	88.0%	86.0%	88.0%
CoMPASS-H	74.0%	70.0%	68.0%	70.7%
Baseline-R	100.0%	100.0%	100.0%	100.0%
CoMPASS-R	30.0%	30.0%	32.0%	30.7%

Note: Accuracy represents the percentage of questions each LLM evaluator answered correctly. Lower accuracy indicates more effective distractors. The acceptable accuracy range is 40–70%.

Figure 4 presents the per-question accuracy distribution for each method. The box plots revealed substantial variability masked by method-level averages. For CoMPASS-R, 27 out of 50 questions (54%) achieved 0% accuracy (all three LLMs selected a distractor), indicating highly effective distractors, while 7 questions (14%) achieved 100% accuracy, suggesting room for improvement. The median CoMPASS-R accuracy was 0%, confirming that the majority of questions featured distractors that were sufficiently plausible to deceive all three evaluator models. In contrast, Baseline-R showed zero variance, with all 50 questions achieving 100% accuracy. For hazard identification, CoMPASS-H produced 8 questions (16%) with 0% accuracy compared to 3 (6%) for Baseline-H, demonstrating increased per-question difficulty.

Figure 4. Per-question RAT accuracy distribution.

5.3. Paired Comparison Results

Paired comparisons using the Wilcoxon signed-rank test revealed distinct patterns between the two types of questions. For the hazard identification questions, CoMPASS-H showed significantly lower IS and PS than Baseline-H, with medium to large effect sizes. For risk control questions, the CoMPASS-R produced a significantly lower IS, with the largest effect size observed in this study, while PS showed no significant difference between the methods. Detailed statistical results are presented in Table 4.

Table 4. Wilcoxon signed-rank test results for incorrectness score (IS), plausibility score (PS) comparisons.

Comparison	W	p-Value	Effect Size (r)	Direction
Hazard IS	409.0	0.027 *	0.358	CoMPASS < Baseline
Hazard PS	251.0	<0.001 ***	0.606	CoMPASS < Baseline
Risk IS	75.0	<0.001 ***	0.882	CoMPASS < Baseline
Risk PS	558.0	0.449	0.125	n.s.

Note: * p < 0.05; *** p < 0.001; n.s. = not significant. Effect sizes: small (r < 0.3), medium (0.3 ≤ r < 0.5), and large (r ≥ 0.5). “CoMPASS < Baseline” indicates CoMPASS produces lower values. For IS, lower values indicate distractors closer to the correct answer (more deceptive). For PS, lower values indicate less contextual relevance.

Bootstrap 95% confidence intervals for DS (10,000 resamples) confirmed that CoMPASS-R’s discriminability was robustly above the 0.3 threshold (DS = 0.560, 95% CI [0.453, 0.667]), whereas CoMPASS-H’s DS confidence interval spanned the threshold (0.293, 95% CI [0.187, 0.407]), consistent with its borderline performance. A McNemar exact test comparing QPR pass rates between CoMPASS-R and Baseline-R yielded p = 5.68 × 10⁻¹⁴, confirming that the 90% vs. 0% QPR difference was statistically significant. Additionally, paired Wilcoxon signed-rank tests on per-question RAT accuracy showed that CoMPASS-R produced significantly lower accuracy than Baseline-R (W = 0.0, p = 3.63 × 10⁻⁹, r = 0.834), and CoMPASS-H produced significantly lower accuracy than Baseline-H (W = 8.5, p = 1.19 × 10⁻³, r = 0.458).

6. Discussion

The evaluation results revealed distinct patterns of domain knowledge integration effectiveness for the two types of questions. For the hazard identification questions, CoMPASS-H improved discriminability by reducing LLM evaluator accuracy from 88.0% to 70.7%, confirming that ontology-guided selection produces more effective distractors than unguided LLM generation. Nevertheless, CoMPASS-H did not meet the composite quality standard (QPR = 0%), with its DS (0.293) falling 0.007 below the 0.3 threshold. This demonstrates that ontology-based selection alone, while improving discriminability relative to the baseline, is insufficient for achieving all three quality criteria simultaneously. However, this improvement introduced an IS–PS trade-off: selecting distractors from the same ontological category as the correct answer increases semantic proximity (lower IS) but does not guarantee contextual relevance to the specific accident scenario (lower PS). Despite this trade-off, the CoMPASS-H distractors remained within the acceptable PS range, and the improvement in discriminability demonstrated a net positive effect on distractor quality. For the risk control questions, CoMPASS-R achieved the ideal outcome—IS improvement with PS preservation—by grounding distractors in the same authoritative knowledge source as the correct answer through hybrid RAG retrieval. The dramatic reduction in LLM evaluator accuracy from 100.0% to 30.7% provides compelling evidence that domain knowledge grounding is essential for generating nontrivial distractors. The cross-model consistency across Claude Sonnet 4.5, Gemini 3 Pro Preview, and GPT-5.2 further validates that these quality improvements reflect genuine distractor properties rather than model-specific artifacts.

The CoMPASS-R average accuracy of 30.7% fell below the target range of 40–70%, and per-question analysis (Figure 4) revealed that 27 out of 50 questions (54%) achieved 0% accuracy, with all three LLMs selecting a distractor. While this indicates highly effective distractors from an automated screening perspective, it also raises the possibility that some distractors are excessively similar to the correct answer, potentially creating ambiguity rather than legitimate challenge. The low IS value (0.460) for CoMPASS-R supports this interpretation. Future refinement could incorporate minimum semantic distance thresholds to prevent excessive similarity between distractors and the correct answer or introduce a post-generation filtering step that ensures a minimum IS floor.

These findings extend existing work on automated distractor generation in important ways. Feng et al. [13] demonstrated the potential of LLMs for distractor generation in general educational domains, and McNichols et al. [23] introduced distribution-based evaluation metrics. Neither approach integrated domain-specific knowledge structures. The multi-stage filtering architecture of CoMPASS—ontology-based candidate extraction followed by LLM verification and composite quality screening—conceptually aligns with recent decision frameworks in safety-critical engineering that distinguish screening, admissibility, and approval as separate stages under uncertainty [28]. This staged approach ensures that each filtering criterion addresses a distinct quality dimension, reducing the risk of false acceptance. CoMPASS addresses this gap by incorporating a construction safety ontology and RAG-based retrieval, achieving 90% QPR for risk control questions compared with 0% for unguided generation. In the construction safety domain, existing LLM applications have focused on information retrieval [9,17] and compliance checking [25], whereas the present work extends LLM applications to assess content generation, a novel contribution that addresses the need for rigorous domain-specific evaluation identified by Sammour et al. [19].

As summarized in Table 5, previous LLM-based construction safety studies focused on three primary application areas: safety information generation [9], visual hazard assessment [18,24], and regulatory compliance checking [25,26]. A common thread across these studies was the objective of identifying or generating correct safety information, such as detecting violations, recognizing hazards, and retrieving applicable regulations. This study addressed a fundamentally different task: generating educationally valid distractors for MCQ-based safety assessments. Unlike the aforementioned applications, which aim to identify what is correct or safe, distractor generation requires the systematic identification of concepts that are plausible yet factually incorrect in the given context. This reversal of the typical LLM objective—from “find correct” to “find plausibly incorrect”—necessitates a distinct methodological approach that CoMPASS provides through its domain knowledge integration. Furthermore, existing studies rely predominantly on human expert evaluations [9] or straightforward accuracy metrics against ground-truth labels [24,25]. The RAT methodology introduced in this study represents a novel evaluation paradigm that employs frontier LLMs as virtual examinees. This approach enables scalable, automated quality assessment while capturing the discriminability dimension—whether distractors appropriately challenge uninformed respondents without misleading knowledgeable respondents —which is unique to educational assessment contexts.

Table 5. Methodological comparison with recent LLM-based construction safety studies.

Study	Application Domain	Knowledge Source	Task Objective	Evaluation Method
Uhm et al. [9]	Safety information generation	RAG + Accident database	Generate accurate safety information	Human evaluation (42 persons)
Kim et al. [24]	Visual safety assessment	Fine-tuned image-text data	Hazard recognition from images	GPT-4V + Human expert evaluation
Guo et al. [25]	Regulatory compliance	BiRAG + Safety regulations	Compliance checking	Accuracy comparison
Chan et al. [26]	Real-time monitoring	Domain-specific ontology	Safety violation detection	Accuracy measurement
CoMPASS (This study)	MCQ assessment	Hazard ontology + Hybrid RAG	Distractor generation	RAT (3 LLMs) + IS/PS/DS/QPR

The practical implications of this study are significant. The CoMPASS algorithm enables the scalable generation of high-quality MCQ distractors, potentially reducing the reliance on expert question writers for construction safety assessments. The modular architecture of ontology-based candidate extraction followed by LLM verification can be adapted to other safety-critical domains with existing knowledge structures, such as healthcare or aviation safety. The RAT methodology also offers a practical alternative to expert evaluation, with consistent results across the three LLM architectures, suggesting its reliability as a preliminary screening tool. From a theoretical perspective, the contrast between CoMPASS-H and CoMPASS-R illustrates that the method of domain knowledge integration, not merely its presence, critically affects distractor quality; RAG-based grounding yields superior results compared to ontology-based selection alone. The current implementation requires approximately 9 LLM API calls per question for CoMPASS (candidate scoring followed by iterative LLM verification) compared to a single call for baseline generation. The complete evaluation pipeline—encompassing question generation, RAT evaluation, and quality scoring for 200 questions across four methods—was completed in approximately 3 h. For practical deployment generating a single question set without RAT evaluation, the estimated processing time is substantially shorter, and further optimization through batch processing and caching can reduce computational overhead.

In practice, CoMPASS could be integrated into existing learning management systems as an automated question bank generator. Safety managers would input accident case descriptions, and the system would automatically generate validated MCQ sets for worker assessments, with the QPR metric serving as an automated quality gate to flag questions requiring manual review.

This study has several limitations. First, the evaluation relied on LLM-based assessment rather than human expert evaluation. While cross-model consistency supports the reliability of this approach, validating the correspondence between RAT-derived and human-based assessments is a critical next step. Second, the dataset was limited to 50 Korean construction accident cases; despite the observed effect sizes (r = 0.358–0.882) indicating genuine quality differences, cross-cultural and cross-domain validation is needed. Third, the IS–PS trade-off in CoMPASS-H remains unresolved, as demonstrated by its borderline QPR performance (DS = 0.293 vs. threshold 0.3). Fourth, certain design choices—including LLM-generated definitions for embedding computation and pre-fixed scoring weights (α = 0.6, γ = 0.7)—may benefit from empirical optimization in future work, although consistent treatment across methods and the discrete structure of ontological similarity provide robustness within the current framework.

7. Conclusions

This paper presents CoMPASS, a domain-knowledge-guided algorithm for generating high-quality MCQ distractors to support safety knowledge assessments on construction sites. The algorithm integrates construction safety ontology, accident co-occurrence data, and hybrid RAG to produce distractors that are simultaneously incorrect, plausible, and discriminating. A four-metric evaluation framework (IS, PS, DS, and QPR) and RAT methodology were developed as an automated screening tool for distractor quality assessment, complementary to human expert evaluation.

An evaluation of 50 Korean construction accident cases demonstrated that domain knowledge integration substantially improved distractor quality. CoMPASS-R achieved 90% QPR with significant improvement in IS while preserving PS, representing the ideal outcome for risk control questions. CoMPASS-H improved discriminability but exhibited an IS–PS trade-off, indicating a clear direction for future refinement. The Baseline results, particularly the 100% LLM accuracy for risk control questions, provide strong evidence that frontier LLMs produce trivially identifiable distractors without external domain knowledge grounding.

The contributions of this work include (1) the CoMPASS algorithm, which demonstrates that domain knowledge grounding can consistently produce high-quality distractors at scale, (2) a four-metric evaluation framework with definition expansion enabling automated assessment of domain-specific distractors, and (3) the RAT methodology, which provides scalable discriminability assessment validated across multiple LLM architectures. Future work will focus on resolving the CoMPASS-H IS–PS trade-off, conducting validation studies with actual construction site personnel and extending the approach to other safety management contexts beyond construction.