Empirical Evaluation of Reasoning LLMs in Machinery Functional Safety Risk Assessment and the Limits of Anthropomorphized Reasoning

Abstract

Transparent reasoning and interpretability are essential for AI-supported risk assessment, yet it remains unclear whether large language models (LLMs) can provide reliable, deterministic support for safety-critical tasks or merely simulate reasoning through plausible outputs. This study presents a systematic, multi-model empirical evaluation of reasoning-capable LLMs applied to machinery functional safety, focusing on Required Performance Level (PL_r) estimation as defined by ISO 13849-1 and ISO 12100. Six state-of-the-art models (Claude-opus, o3-mini, o4-mini, GPT-5-mini, Gemini-2.5-flash, DeepSeek-Reasoner) were evaluated across six prompting strategies and two dataset variants: canonical ISO-style hazards (Variant 1) and engineer-authored free-text scenarios (Variant 2). Results show that rule-grounded prompting consistently stabilizes performance, achieving ceiling-level accuracy in Variant 1 and restoring reliability under lexical variability in Variant 2. In contrast, unconstrained chain-of-thought reasoning (CoT) and CoT together with Retrieval-Augmented Generation (RAG) introduce volatility, overprediction biases, and model-dependent degradations. Safety-critical coverage was quantified through per-class F1 and recall of PL_r class e, confirming that only rule-grounded prompts reliably captured rare but high-risk hazards. Latency analysis demonstrated that rule-only prompts were both the most accurate and the most efficient, while CoT strategies incurred 2–10× overhead. A confusion/rescue analysis of retrieval interactions further revealed systematic noise mechanisms such as P-inflation and F-drift, showing that retrieval can either destabilize or rescue cases depending on model family. Intermediate severity/frequency/possibility (S/F/P) reasoning steps were found to diverge from ISO-consistent logic, reinforcing critiques that LLM “reasoning” reflects surface-level continuation rather than genuine inference. All reported figures include 95% confidence intervals, t-intervals across runs ($r=5$) for accuracy and timing, and class-stratified bootstrap CIs for Micro/Macro/Weighted-F₁ and per-class metrics. Overall, this study establishes a rigorous benchmark for evaluating LLMs in functional safety workflows such as PL_r determination. It shows that deterministic, safety-critical classification requires strict rule-constrained prompting and careful retrieval governance, rather than reliance on assumed model reasoning abilities.

1. Introduction

The increasing complexity of industrial machinery, coupled with stringent regulatory demands, has intensified the need for reliable and structured functional safety risk assessment methods. Among these, the estimation of the Required Performance Level (PL_r) is a critical step, directly impacting the design and validation of safety-related control systems in accordance with standards such as ISO 12100 [1] and ISO 13849-1 [2]. PL_r estimation is inherently deterministic, derived by systematically evaluating a given hazard scenario against defined risk parameters such as severity, frequency of exposure, and possibility of avoidance and applying predefined classification rules to assign the PL_r in accordance with formalized procedures [1,2,3].

On the other hand, the advent of large language models (LLMs) equipped with advanced reasoning capabilities has reshaped the landscape of Artificial Intelligence (AI)-assisted decision support. Models employing structured reasoning mechanisms, such as chain-of-thought (CoT) prompting or retrieval-augmented generation (RAG), have shown potential in complex problem-solving and domain-specific language tasks. These developments have led to speculative interest in whether such reasoning models could extend their utility to structured industrial domains, including safety-critical classification tasks like PL_r estimation. However, this potential remains largely unexplored, with no prior work systematically validating reasoning models for deterministic, safety-critical tasks such as PL_r estimation, and no empirical evidence yet assessing their reliability in this domain.

In safety-critical risk assessment, transparent reasoning is essential for both regulatory compliance and human-understandable justification of classifications. A central open question is whether reasoning-capable LLMs can provide genuinely interpretable, rule-consistent decision support, or merely generate plausible but unreliable outputs. Prior studies have focused on open-domain inference or problem-solving benchmarks, without systematically addressing deterministic tasks that require strict rule adherence and minimal tolerance for ambiguity. Moreover, the known tendency of LLMs to produce “reasoning illusions”, outputs that appear logically structured yet lack factual correctness, raises significant concerns about their reliability in functional safety contexts.

1.1. Identified Research Gaps

The motivation for this study is based on addressing three key research gaps:

• Gap 1: Lack of empirical evaluation of reasoning models in deterministic, safety-critical PL_r classification.
• Gap 2: Underexplored impact of reasoning biases and hallucinations in PL_r estimation.
• Gap 3: Absence of structured benchmarking methodologies and empirical evaluation for reasoning models in functional safety risk assessment.

1.2. Research Questions

To guide the study and systematically address the identified gaps, the following research questions are formulated:

• RQ1: Can reasoning models reliably perform structured risk classification tasks, such as PL_r estimation, within the constraints of functional safety standards?
• RQ2: What limitations do reasoning models exhibit when applied to deterministic classification problems in the domain of machinery risk assessment?
• RQ3: How does the reliance on structured prompting affect the consistency and validity of reasoning model outputs in functional safety contexts?

These questions guide the experimental evaluation, focusing on the empirical behavior of reasoning-capable LLMs under controlled, rule-bound testing scenarios.

1.3. Contributions

This study makes the following novel contributions:

• Comprehensive Experimental Benchmarking of Reasoning Models for Functional Safety Risk Classification (RQ1): A systematic evaluation of reasoning-capable LLMs applied to PL_r estimation tasks is presented, utilizing diverse prompting strategies including zero-shot, CoT, rule-based prompting, and RAG-augmented reasoning.
• Empirical analysis of reasoning biases and hallucination effects in structured classification (RQ2): The study analyzes error patterns, misclassification tendencies, and reasoning-induced biases exhibited by LLMs in deterministic classification tasks (e.g., P-inflation, F-drift, redundancy, and mislabeling across PL_r classes).
• Identification of Methodological Considerations for LLM Deployment in Safety-Critical Applications and Future Benchmarking (RQ3): Practical implications are highlighted, including the necessity of structured prompting and the risks of anthropomorphizing LLM reasoning capacities (i.e., attributing human-like reasoning abilities to models based on their language output) in domains with strict correctness requirements. The findings also establish a basis for future research on the applicability and limitations of AI reasoning models in structured industrial domains, contributing to the development of scientifically grounded benchmarking methodologies.

Note that this study focuses on the empirical validation of structured prompting strategies for deterministic risk classification tasks in machinery functional safety. The objective is to provide practical benchmarking evidence for Artificial Intelligence (AI) deployment in regulated industrial domains, rather than to critique the reasoning capabilities of LLMs in general.

The remainder of this paper is organized as follows. Following this introduction, Section 2 deals with background and related work. Section 3 deals with experimental design and evaluation performance metrics. Section 4 provides a detailed analysis of the results from experimental evaluation. Section 5 concludes this paper.

2. Background and Related Work

This section provides the necessary background on functional safety risk assessment, with a focus on machinery domains governed by ISO 12100 and ISO 13849 standards. Section 2.1 outlines the principles of hazard identification, risk estimation, and PL_r determination. Building on this foundation, the related work review examines prior research on AI-assisted risk assessment. The section further analyzes critical findings on LLM reasoning capabilities, limitations of chain-of-thought prompting, and the risks of anthropomorphizing model outputs. Finally, it discusses the role of retrieval-augmented generation, prompt engineering, and domain-specific benchmarking in ensuring reliable AI performance in deterministic, safety-critical applications.

2.1. Machinery Functional Safety Risk Assessment

Machinery safety is a global concern for suppliers and manufacturers, governed by a comprehensive regulatory framework designed to protect individuals, assets, and property from harm. Most industrial environments feature complex machinery assemblies that must comply with relevant safety regulations. The Machinery Regulation (EU) 2023/1230, adopted on 14 June 2023, reinforces the core compliance requirement that each machine undergo a structured hazard analysis and risk assessment—a systematic evaluation of potential hazards based on severity, exposure frequency, and the possibility of avoidance—as an essential part of conformity assessment for CE marking [4].

ISO 12100 [1] is an international standard which specifies basic terminology, principles and methodology for achieving safety in the design of machinery. It specifies principles of risk assessment and risk reduction to help designers achieve this objective. Procedures are described for identifying hazards, and estimating and evaluating risks during relevant phases of the machine life cycle. For instance, a hazard refers to something that potentially causes harm and the risk is a combination of the probability and severity of the harm. For example, a sharp part is a hazard. When it is in a prominent position, it creates a risk.

2.1.1. Risk Assessment

The iterative process of risk assessment and reduction is shown in Figure 1. A risk assessment follows a series of logical steps to identify and examine any potential hazards associated with machinery. The process starts with hazard identification in terms of machine space, time, and usage limits. The hazard is then estimated by risk elements such as harm severity (S), occurrence frequency (F), and avoidance or limitation probability (P). Based on the information obtained, the risk is then evaluated for acceptability. If it is not acceptable, risk reduction measures are required. The whole process is called risk assessment. Iteration of this process can be necessary to eliminate hazards as far as practicable and to adequately reduce risks by the implementation of protective measures. Protective measures play an important role in risk reduction. Such measures include protection devices and safety controls, the combination of which is called the Safety-Related Part of the Control System (SRP/CS) [2,3].

2.1.2. Safety Function

Safety functions (SFs) are the machine functions that cause an immediate increase in risk upon failure [2]. A single SF can be implemented by multiple SRP/CSs. A single SRP/CS may also implement multiple SFs, such as prevention of accidental start-up and limits regarding safety parameters in temperature and pressure. For example, the control system shuts down the furnace fire when the boiler pressure reaches a dangerous value. If this function fails, excessive pressure will lead to an explosion. In this scenario, safety depends on the SRP/CS performing the correct function. Each SF is tasked with reducing the risk of one or more hazardous events. It is necessary to consider each hazard and its corresponding SF in the design. The risk assessment results influence the PL_r value for the safety function.

2.1.3. Required Performance Level- PL_r

PL_r is the risk reduction expectation required for the implementation of an SF and can be determined by the risk graph shown in Figure 2. A risk graph is a grading-based risk estimation method with parameters S, F and P corresponding to the severity of harm, the duration or frequency of operator exposure to the hazard area, and the possibility of avoiding the hazard, respectively [3]. The result represents the level of risk without protection from the safety system. It is also used to determine the performance level of the safety function that is needed to reduce the risk to a permissible level. Figure 2 shows the structure of the risk graph.

The severity (S) of harm is divided into S1 (slight) and S2 (serious). Only slight reversible harm, serious irreversible harm, and fatalities are considered when estimating the levels of harm [3]. The normal recovery process is generally used as the basis for evaluating the severity of harm to people. Slight harm is usually recoverable, while serious is not. For example, fatigue and slipping are categorized as S1, and amputation and death are categorized as S2.

The frequency (F) or time of exposure to the hazard is classified as F1 (seldom or short time) and F2 (frequent or long time). This parameter is a measure of the time spent in the danger zone. As per [2], when the operator is present in the area more frequently than once every 15 min, it is considered F2 level. If for automated processing machines, where the operator needs to intervene only once a month, F1 is the obvious choice.

The possibility (P) of avoiding hazards is divided into categories P1 and P2, which are determined based on whether the hazard can be identified or prevented. If it is possible to avoid an accident under certain circumstances, P1 is chosen, but if it is almost impossible to avoid, then P2 is chosen. Factors that affect parameter P include the speed of the hazardous situation leading to harm, any awareness of risk, and the human ability to escape. For example, the speed of machine operation is limited so that potential accidents are delayed, and the operator has the opportunity to react and leave the zone. As can be seen from the graph (Figure 2), combining these parameters increases the risks from low to high (i.e., PL_r-a to PL_r-e, where PL_r-e is the highest level required for SF and the most expensive to implement).

Please note that, in this study, the LLM is presented with natural language descriptions of machinery hazard scenarios that implicitly or explicitly indicate the three risk parameters defined in ISO 13849, namely severity (S), frequency (F), and possibility of avoidance (P). The model is evaluated for its ability to interpret these factors and accurately classify the corresponding Required Performance Level (PL_r), thereby demonstrating its potential to support functional safety risk assessment in accordance with ISO 12100 and ISO 13849-1. This evaluation is essential to determine whether LLMs can reliably replicate expert-level judgment in PL_r classification, a prerequisite for integrating AI into scalable, standard-compliant functional safety workflows where manual assessments are often inconsistent, resource-intensive, and difficult to reproduce.

Building on this foundation, the following section reviews related work on AI-assisted safety assessments, rule-based prompt engineering, and retrieval-augmented reasoning in the context of structured decision-making and industrial risk classification.

2.2. AI-Based Risk Assessment in Functional Safety

Functional safety risk assessment in machinery domains is typically governed by ISO 12100 and ISO 13849, which define a logic-driven procedure for estimating PL_r based on hazard severity, exposure frequency, and the possibility of avoidance. Traditionally, such assessments rely heavily on expert judgment, limiting scalability and reproducibility. The emergence of new regulations and the growing complexity of machinery systems have amplified the demand for scalable, consistent risk assessment approaches, driving interest in automation and AI-driven methods to support and augment traditional expert analyses.

Early studies explored custom-built AI-based solutions for machinery functional safety risk assessment. The work in [5] introduced a specialized chatbot leveraging rule-based logic and a TextCNN-LSTM architecture, achieving approximately 80% accuracy on internal datasets but showing limited robustness to linguistic variability. Subsequently, [6] presented a chatbot for recommending risk reduction measures aligned with ISO 12100. Although these domain-specific prototypes demonstrated potential, their scalability was constrained by the lack of structured training data and the significant effort required for data curation and model training. Moreover, these solutions predate the advent of LLMs with reasoning capabilities and thus serve as foundational steps toward applying LLMs to structured risk assessment tasks like PL_r estimation.

To address the absence of structured datasets aligned with risk assessment standards, the work in  [7] introduced an open-access dataset of 7800 annotated machinery hazard scenarios derived from ISO 12100 and paired with PL_r values determined according to ISO 13849-1. This resource enables reproducible PL_r prediction experiments with state-of-the-art LLMs. Follow-up studies, such as  [8], applied zero-shot, rule-based, and retrieval-augmented prompting strategies to general-purpose LLMs [9] on this dataset. The results showed that rule-based prompting with retrieval augmentation outperformed both zero-shot and standard rule-based methods, yet also exposed variability across prompt designs. Further, the work in [10] examined Chain-of-Thought prompting for OT cybersecurity risk assessment, demonstrating feasibility but limited depth. By contrast, this study provides a systematic and in-depth evaluation of reasoning strategies in the parallel domain of deterministic PLr classification.

Emerging studies have explored the application of LLMs in safety-critical hazard analysis, aiming to automate tasks traditionally reliant on expert judgment. Nouri et al. [11] applied a GPT-4-based pipeline for hazard analysis and risk assessment (HARA) in automotive systems. Their method decomposed the analysis into subtasks, hazard identification, scenario generation, and severity classification, each using tailored prompts. While effective in generating draft assessments, the study emphasized the ongoing necessity of expert validation. Similarly, the study in [12] evaluated ChatGPT’s utility for System-Theoretic Process Analysis (STPA) in automotive braking systems, finding that naive prompting produced poor results, but domain-specific prompts combined with human oversight enabled LLMs to identify hazards with competence comparable to human experts. In the domain of consumer product safety, the work in [13] observed that while ChatGPT could enumerate a broad set of failure scenarios, it frequently provided weak or unsupported risk judgments. These studies highlight both the potential and the limitations of LLMs in structured hazard analysis. They underscore that while LLMs can assist in preliminary risk assessments or scenario generation, their effectiveness in deterministic, rule-bound tasks—such as those required in functional safety, is dependent on structured prompting, domain adaptation, and expert supervision.

Effective interaction with LLMs in risk analysis tasks often depends on structured, rule-based prompting. Without domain-specific guidance, LLMs may produce inconsistent or misleading outputs. The study in [11] employed format-constrained subtasks and predefined templates to improve output consistency. The work in [14] introduced a co-hazard analysis (CoHA) framework that combined iterative Q&A with ChatGPT and domain rules, enhancing the coverage and creativity of hazard identification. Similarly, the work in [15] further demonstrated that structured prompts significantly influenced LLM accuracy and interpretability on safety certification questions. However, they also noted that no single prompt structure performs optimally across all task types. These studies underscore that while LLMs offer potential for draft hazard identification and scenario generation, they fall short in deterministic, rule-constrained risk assessments. Addressing these limitations requires structured prompting, retrieval augmentation, and expert oversight to mitigate reasoning illusions and ensure alignment with formal safety standards. The work in [16] evaluates the integration of LLMs like ChatGPT into a human-in-the-loop (HITL) framework for machinery functional safety risk analysis, adhering to ISO 12100. It demonstrates that expert oversight within the HITL framework effectively mitigates LLM limitations such as hallucinations, leading to complete agreement with ground truth across diverse industrial case studies. The study highlights significant gains in efficiency, accuracy, and usability, underscoring the transformative potential of generative AI in safety workflows when rigorous human validation is maintained. But a systematic experimental evaluation of LLMs using a comprehensive dataset is not available in [16].

2.3. LLM Capabilities and Limitations in Reasoning Tasks

Early studies showed that prompting LLMs with step-by-step CoT explanations significantly improves their performance on reasoning tasks, including arithmetic and logic problems [17]. Kojima et al. [18] further revealed that even simple zero-shot CoT cues like “Let’s think step by step” could unlock latent reasoning abilities in LLMs across benchmarks such as GSM8K and Big-Bench Hard. These results fueled optimism that, when guided correctly, LLMs might approximate general reasoning skills, with recent models like GPT-4 demonstrating notable problem-solving capabilities across varied domains.

Several recent studies critically examine the assumptions underlying CoT-driven reasoning claims. Saparov et al. [19] introduced the PrOntoQA benchmark and found that LLMs, while capable of generating valid local inference steps, fail at global proof strategies, revealing a tendency toward greedy, heuristic reasoning rather than systematic exploration. Schaeffer et al. [20] provided further evidence, showing that even logically invalid or irrelevant CoT traces can boost model performance on complex tasks, implying that token pattern familiarity rather than logical correctness drives success. These findings suggest that CoT benefits often arise from superficial token dynamics rather than authentic deductive reasoning, raising concerns about overestimating LLM reasoning competence in critical applications.

Multiple recent studies challenge the notion that LLM-generated CoT outputs reflect genuine reasoning. Kambhampati et al. [21] argue that labeling intermediate text as “thoughts” dangerously anthropomorphizes LLMs, masking the fact that CoT may merely improve output through surface token patterns rather than reasoning. Stechly et al. [22] demonstrate that models often produce correct answers despite incoherent or irrelevant intermediate steps, suggesting that CoT traces may result from training artifacts rather than causal reasoning. Furthermore, Chen et al. [23] show that LLMs frequently omit critical reasoning cues from their explanations and that even reward tuning fails to ensure faithful reasoning traces. Collectively, these works highlight that CoT outputs may not transparently reveal model decision-making and thus cannot be trusted for reliable auditing. This undermines the premise of using verbalized reasoning as a safeguard in high-stakes applications, raising critical concerns about the auditability and interpretability of LLM decisions.

Recent studies, such as [24], reveal that LLMs with CoT prompting exhibit a “reasoning cliff”, performing well on simple tasks but suffering abrupt accuracy collapse as task complexity increases. Surprisingly, non-CoT models sometimes outperform verbose CoT-augmented models on low-complexity tasks, while both fail on high-complexity problems due to brittle algorithmic reasoning and lack of compositional generalization. These findings challenge the assumption that longer reasoning traces correlate with genuine reasoning ability, underscoring critical limits of current LLM reasoning capabilities.

In summary, recent research offers a mixed view of LLM reasoning: while CoT and self-consistency improve benchmark scores, many studies suggest this reflects token patterning rather than genuine inference, leaving open the debate between emergent capability and engineered illusion.

2.4. Prompt Engineering, Retrieval-Augmented Generation, and Model Interpretability

One critical mitigation strategy for hallucinations and factual inaccuracies is retrieval-augmented generation (RAG), which grounds the model’s output in external knowledge sources. The work in [25] showed that RAG can significantly improve accuracy on knowledge-intensive QA by injecting relevant documents into the generation process. While RAG alone does not eliminate hallucinations entirely as models may still misquote or misapply retrieved facts it significantly enhances factual grounding. Recent applications in safety-critical domains demonstrate its value: the work in [26] integrated Qwen-2.5 with a curated fire safety regulation database to improve factuality in fire engineering queries. The study in [27] applied advanced RAG pipelines to toxicology assessments, achieving higher scientific fidelity through query rewriting and evidence grounding. In the automotive domain, the work in  [28] developed the LASAR system, which combines scenario generation with catalog-based retrieval to guide LLMs in hazard analysis and risk assessment (HARA).

In safety-critical applications, interpretability is as vital as accuracy. Well-designed prompts can elicit stepwise reasoning, uncertainty estimates, and justifications from LLMs, aiding human validation and traceability. Studies conducted in  [11,28] found that instructing LLMs to explain their severity ratings improved reviewer confidence and auditability. Further, the work in  [15] showed that prompt-induced variations could swing model performance by over 13%. Yet, the benefits of added prompt complexity often depend on the task structure, indicating a need for both prompt-task matching and systematic evaluation.

Overall, prompt engineering enhances reliability and interpretability, but cannot ensure factual grounding or regulatory compliance. These limitations underscore the need for rule-based prompting and RAG to enforce alignment with formal safety standards.

2.5. The Role of Domain-Specific Benchmarking in Evaluating LLMs

Beyond task-specific studies, there is a broader recognition that evaluation methodologies for LLMs in high-stakes, rule-bound domains are underdeveloped. Traditional NLP benchmarks rarely capture the requirements of safety-critical decision-making. Recently, researchers have begun devising domain-specific benchmarks to fill this void. In the legal domain, the work in [29] introduced LegalBench, a suite of 162 tasks covering diverse types of legal reasoning to systematically measure LLM performance on jurisprudence problems. Their evaluations of dozens of models revealed substantial gaps between general LLM capabilities and the consistency needed for legal reasoning, reinforcing the need for tailored assessment in regulated domains. In medicine, the work presented in [30] proposed MedCalc-Bench with over 1000 cases requiring medical calculations (e.g., risk scores, dosage) to test LLMs as clinical calculators. Results showed current models often err on precise numeric reasoning in a clinical context, despite performing well on open medical Q&A, highlighting the importance of structured benchmarks for reliability. The safety community is also moving in this direction. The work in [31] presents HSE-Bench, focusing on Health, Safety and Environment compliance questions with multi-step legal reasoning; they find that today’s LLMs rely more on semantic pattern matching than true rule application in compliance assessments. Notably, even the best models’ “reasoning traces lack the systematic legal reasoning required for rigorous HSE compliance,” and performance degrades on complex multi-step scenarios. These domain-specific evaluations underscore a common theme: without structured benchmarks and protocols, it is difficult to gauge an LLM’s trustworthiness for safety-critical tasks. The work in [32] further demonstrates this in a clinical setting by benchmarking an open-source reasoning LLM against GPT-4 on 125 real patient cases. While the open model reached parity on final answers, the study stressed meeting strict regulatory criteria (e.g., explainability, auditable steps) as a key evaluation component. Together, these works illustrate the nascent but growing effort to establish rigorous evaluation frameworks for LLMs operating under domain constraints.

In summary, advances in AI-assisted risk assessment show promise, but persistent limitations remain for deterministic tasks such as PL_r estimation. Structured prompting, RAG and HITL validation can improve performance in preliminary hazard analysis [11,16], yet critical studies on reasoning fidelity [21,22,33] indicate that CoT traces often reflect heuristic token continuations rather than genuine deduction. Domain-specific benchmarking efforts [31,32] further highlight the inadequacy of general LLMs for tasks requiring strict rule adherence.

Together, these insights frame the central question this study addresses: Can the “so-called” reasoning-capable LLMs reliably perform deterministic risk classification, such as PL_r estimation?. To address this, the study empirically evaluates six LLMs across six prompting strategies for PL_r estimation, using a structured framework aligned with ISO 12100 and ISO 13849-1 Annex A qualitative risk graph (cf. Figure 1).

3. Experimental Design

This section outlines the experimental setup devised to systematically evaluate reasoning-capable LLMs on deterministic PL_r classification tasks. It describes the benchmark dataset derived from ISO 12100 and ISO 13849 standards, details the prompting strategies applied across six experimental conditions, and explains the selection of six LLMs. The section concludes with the evaluation framework used to analyze classification accuracy, reasoning behavior, computational performance, and error patterns.

The evaluation leverages LangGraph [34] as its core orchestration engine, enabling dynamic prompt routing and evaluation logic. Modular prompting strategies (rule-based, RAG, and hybrid) are implemented using LangChain chains [35] for seamless reconfiguration. High-throughput semantic retrieval of domain-specific hazard precedents for RAG-based scenarios is facilitated by a ChromaDB [36] vector database, primarily sourcing data from the open-source dataset [7,37]. These retrieved contexts are injected via prompt placeholders to assess analogical reasoning performance. This modular setup mirrors realistic deployment conditions while allowing systematic control over prompting strategies, model selection, and input variants.

3.1. Dataset Description

The experiments in this study utilize an open-source Industrial Machinery Functional Safety Hazard Scenario Dataset [37], designed to serve as a transparent, reproducible, and scientifically rigorous benchmark for empirical evaluation of automated risk assessment methods [7]. The dataset is used throughout this study as the standardized evaluation benchmark for comparing baseline, rule-based, COT and RAG methods for PL_r determination across diverse industrial safety scenarios.

3.1.1. ISO 12100: Annex B and ISO 13849-1: Annex A Correlation

The dataset construction process is systematically aligned with ISO 12100 Annex B, which enumerates ten general hazard categories relevant to machinery functional safety. For each category, the dataset defines specific hazard origins and enumerates plausible potential consequences, refining the generic framework of ISO 12100 into a structured representation suitable for computational risk assessment. This mapping ensures that every scenario is rooted in established industrial safety standards.

Not all combinations of hazard origin and consequence are physically meaningful or relevant to real-world contexts. Therefore, every origin–consequence pair undergoes a rigorous plausibility assessment, incorporating physical laws, causal logic, and expert judgment from certified functional safety professionals. Only combinations that are physically possible and contextually credible are retained. For example, “entanglement” from “rotating elements” is included, while physically impossible pairings (such as “loss of balance” from “scraping surfaces”) are excluded.

For each plausible origin–consequence pair, scenarios are instantiated by systematically varying contextual parameters, including the following:

• User type (e.g., operator, maintenance personnel).
• Task (e.g., normal operation, cleaning, maintenance).
• Operating environment (e.g., industrial).
• ISO 13849-1 risk graph parameters: severity (S), frequency and/or duration of exposure (F), and possibility of avoidance (P).

Severity is classified as either “slight (normally reversible injury)” or “serious (normally irreversible injury or death)” (corresponding to S1/S2 in ISO 13849-1), frequency as “seldom-to-less-often/exposure time is short” or “frequent-to-continuous/exposure time is long” (F1/F2), and possibility as “possible under specific conditions” or “scarcely possible” (P1/P2). The PL_r is automatically mapped for each scenario by applying the ISO 13849-1: Annex A risk matrix (cf. Figure 1) to the scenario’s S, F, and P values.

This dataset, comprising 7800 machinery hazard scenarios, is automatically generated in a standardized JSON format. Each entry meticulously details hazard attributes, user context, environmental factors, and calculated PL_r, along with a natural language description. The comprehensive distribution across ten hazard categories, including 2840  mechanical and 1640 electrical hazards, ensures a robust and reproducible foundation for benchmarking AI in safety-critical classification tasks.

3.1.2. Dataset Entry: Template and Example

Listing 1 illustrates a typical scenario entry in the dataset. This example describes an electrical hazard where maintenance personnel performing setup or programming tasks in an industrial environment may encounter an arc, which could lead to a burn. The scenario specifies key contextual parameters: user type (maintenance personnel), task (setup or programming), environment (industrial), and the risk graph inputs, severity (slight), frequency (seldom-to-less-often), and possibility of avoidance (scarcely possible). Based on these factors, the PL_r is assigned as ‘b’ according to the ISO 13849-1: Annex A qualitative performance graph method.

Listing 1. Example of an electrical hazard description in the dataset [7,37].

In all experiments, the scenario’s PL_r value, calculated using the ISO 13849-1 risk matrix, serves as the ground truth for evaluation. Each language model receives the scenario’s natural language description as input and is tasked with predicting the PL_r. Model outputs are then compared to this reference value to compute predictive accuracy. For example, in Listing 1, only models predicting PL_r = ‘b’ are counted as correct. This evaluation protocol enables systematic, transparent, and standard-compliant benchmarking of automated risk assessment methods across diverse safety-critical scenarios.

Further methodological details and rationale for the dataset construction are provided in [7]. The dataset is openly accessible to the research community and practitioners, serving as a standardized benchmark that accelerates scientific progress and enables rigorous comparison of AI-based risk assessment methods in machinery safety.

3.2. Evaluation Datasets

To probe both standard-aligned performance and real-world generalization, two dataset variants are evaluated that share the same schema and gold labels but differ in lexical phrasing.

• Variant 1—Canonical ISO-style scenarios (N = 100): This variant anchors the evaluation in in-distribution phrasing closely mirroring ISO 12100 Annex B. Each case follows a fixed schema (hazard type, origin, potential consequence, task, environment) and reports explicit risk parameters, namely, severity (S), exposure frequency (F), and possibility of avoidance (P), together with a reference PL_r. Because the textual descriptions are automatically generated from canonical fields while remaining faithful to the ISO taxonomy, Variant 1 can be characterized as a synthetic but controlled dataset that provides a standardized baseline under explicit terminology.
• Variant 2—Functional safety engineer-authored scenarios with lexical shift (N = 100): This variant stress tests generalization to field language. Scenarios were written by a functional safety engineer from industrial practice using the same schema and gold labels as Variant 1, but the free-text descriptions deliberately avoid literal ISO tokens for S/F/P. Instead, the factors are conveyed implicitly in operational prose, for example, "repetitive short stops near a moving transmission", "limited clearance and delayed stop reachability" and "hands inside a nip area during setup". This emulates how hazards are typically described in industrial workflows during safety assessments.

In short, Variant 1 serves as a structured baseline with standardized phrasing, while Variant 2 introduces deliberate lexical shift while preserving labels and structure. This design enables us to test both in-distribution performance (standard-aligned) and out-of-distribution robustness to field language, particularly important for analyzing the sensitivity of CoT and RAG prompting strategies.

3.3. Prompting Strategies for Risk Classification

The experimental evaluation employs six distinct prompting strategies for PL_r determination, each reflecting progressively higher levels of structured input and domain knowledge integration. In all variants, the standard chat-based prompt format with system and user roles (i.e., human) as defined in [38] is used. This format distinguishes between the "system" message, which establishes the model’s assumed role, such as a functional safety expert and supplies any required guidance, domain rules, or reasoning instructions, and the "user" message, which presents the actual task input, typically the natural language hazard scenario for analysis. This separation of roles supports structured, reproducible prompt design and enables controlled assessment of how different prompt components influence LLM reasoning in deterministic risk classification tasks.

The six prompting strategies are detailed below. Please note that the first four approaches do not include retrieval-based approaches, whereas the last two are retrieval-based approaches, i.e., in combination with RAG pipeline implementation described in detail in Section 3.5.1.

• Experiment I: Zero-shot Prompt
  In this baseline experiment, the reasoning model is given only the raw hazard scenario in natural language, without any additional guidance or rules beyond its role as a functional safety expert. This setup assesses the reasoning model’s inherent ability to determine the PL_r based on its pre-trained knowledge and reasoning capabilities, focusing on systematic analysis of severity, frequency, and avoidance parameters.
• Experiment II: Explicit ISO Rule Integration
  Building on the zero-shot prompt, this experiment supplements the scenario with explicit PL_r determination rules as specified in ISO 13849-1 Annex A (cf. Figure 1). This tests whether access to codified safety knowledge enables more accurate and consistent classification.
• Experiment III: Chain-of-Thought (CoT)
  This experiment uses the chain-of-thought (CoT) approach by providing highly structured, explicit step-by-step instructions for the model’s reasoning process but without explicitly stating the ISO 13849-1: Annex A performance graph rules. This aims to guide the model through a precise and verifiable pathway for determining the PL_r.
• Experiment IV: Chain-of-Thought (CoT) with Rules
  This experiment combines the structured step-by-step instructions of the CoT approach with explicit textual inclusion of the ISO 13849-1: Annex A performance graph rules (cf. Figure 1). The goal is to provide the model with the necessary information directly within the prompt, minimizing reliance on its pre-trained knowledge for the specific rules of PL_r determination. This setup evaluates the model’s ability to precisely apply provided rules in conjunction with its reasoning process.
• Experiment V: CoT with Rules and Retrieval-Augmented Examples
  In addition to the scenario, rules, and explicit CoT instructions, representative historical hazard examples are retrieved from a curated database and included in the prompt. This evaluates the model’s ability to generalize from a precedent and improve classification accuracy through context enrichment. This experiment is referred to as $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ in the paper.
• Experiment VI: Rules with Retrieval-Augmented Examples
  In addition to the scenario and rules, representative historical hazard examples are retrieved from a curated database and included in the prompt. This evaluates the model’s ability to generalize from precedent and improve classification accuracy through context enrichment. This experiment is referred to as $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ in this paper. This is specifically added as a methodology to understand the differences of usage with and without COT together with rules and retrieval. So, please note that the main difference between the experiments V and VI is that the experiment VI does not have CoT instructions.

In this study, during experimental evaluation, the six experimental settings are applied consistently across six state-of-the-art models and two dataset variants, enabling direct comparison of the incremental benefits of each input enhancement. For clarity, these prompting strategies are referenced using the following shorthand macros:

• ZERO_SHOT: Baseline condition where the model receives only the raw hazard scenario, without explicit rules or structured guidance.
• WITH_RULES: Prompt includes explicit ISO 13849-1 Annex A rules, constraining the model to rule-based PL_r determination.
• PURE_CoT: Uses chain-of-thought (CoT) instructions to elicit step-by-step reasoning, but without explicit rule injection.
• COT_WITH_RULES: Combines CoT instructions with explicit ISO rules, guiding the model through structured reasoning and rule application.
• WITH_RULES_RAG: Augments rule-based prompting with retrieved hazard exemplars from a curated database, enabling case-based reasoning.
• COT_WITH_RULES_RAG: Integrates CoT, explicit ISO rules, and retrieved exemplars, representing the most structured and enriched prompting setup.

3.4. Prompt Placeholder Description

To ensure methodological rigor and enable systematic analysis, each experimental prompt utilizes well-defined placeholders corresponding to key information components. These placeholders not only enforce consistency in model evaluation, but also reflect distinct strategies for eliciting and constraining LLM behavior in a safety-critical classification task.

• {description}: The central input for each prompt is a natural language description of a real-world hazard scenario, simulating industrial safety assessment tasks and enabling evaluation of the model’s capacity for context comprehension and risk mapping. An example description, including user type, task, hazard origin, and consequence, is shown in Listing 1.
• {iso_rules_info}: This placeholder injects the structured decision logic codified in ISO 13849-1: Annex A, including parameter definitions and full risk graph mapping rules. It converts the task from open-ended inference to rule-constrained reasoning, enabling assessment of whether direct access to normative safety rules improves PL_r classification fidelity and consistency. The rules ensure deterministic mapping of scenario descriptions into standardized parameters—severity (S1/S2), exposure frequency (F1/F2), and possibility of avoidance (P1/P2)—which together yield the PL_r. An excerpt of this content is shown in Listing 2.

  Listing 2. Deterministic rules codified under {iso_rules_info} for PLr inference based on ISO 13849-1: Annex A qualitative performance graph rules (cf. Figure 1).

• {COT_step_by_step_instruction} This placeholder as detailed in Listing 3 formalizes a CoT prompting technique. It provides the LLM with explicit, step-by-step instructions for analyzing a hazard scenario, specifically guiding its reasoning for severity, frequency, and avoidance. By forcing this sequential decomposition and requiring intermediate justifications, this CoT aims to enhance the model’s transparency and align its decision-making with structured human expert methodologies for PL_r determination. This approach is crucial for improving explainability and validating AI reasoning in critical safety applications.

  Listing 3. Structured chain-of-thought instructions under {COT_step_by_step_instruction} prompt placeholder for PLr inference.

• {rag_examples}:
  This placeholder is populated by a retrieval module that provides curated hazard scenarios (with ground-truth S/F/P and PL_r) from a curated database of hazard scenarios. Further details about the RAG implementation pipeline are provided in Section 3.5.1.

3.5. RAG Implementation

In the PL_r experiment framework, RAG is employed to systematically test whether augmenting prompts with precedent hazard examples improves deterministic classification of PL_r. The underlying rationale is that models may benefit from a small set of prior cases that are not only textually relevant but also structurally consistent with the ISO 13849-1 risk parameters, S, F, and P. In addition, RAG is explicitly evaluated under lexical-shift conditions queries drawn from a companion corpus that is semantically consistent with ISO 13849-1 but avoids the literal S/F/P tokens (e.g., paraphrases such as “infrequent contact” for F1 or “avoidance is difficult” for P1)—to test whether exemplar augmentation improves deterministic PL_r classification when surface forms diverge from both the rules and the database phrasing. To this end, the framework implements RAG in two chain types:

• WITH_RULES_RAG: Injects retrieved exemplars alongside the base hazard description and ISO rules.
• COT_WITH_RULES_RAG: Integrates exemplars into a CoT prompt alongside base hazard descriptions and ISO rules.

3.5.1. Pipeline Implementation

A hybrid RAG pipeline is implemented in the sense of dense(ish) database retrieval followed by symbolic filtering and deterministic packaging for prompting [25,39]. The orchestration is handled by the _search_similar_hazards function and proceeds as follows: At initialization, a hazard database object is created (from an existing index) and an optional validator chain is attached.

The RAG controls are $(k,\tau ,M)=(\mathtt{rag}\_\mathtt{k},\mathtt{rag}\_\mathtt{sim}\_\mathtt{threshold},\mathtt{rag}\_\mathtt{top}\_\mathtt{m})$, a strict S/F/P gate require_sfp_exact (boolean), and an optional drop_missing_labels. Defaults in the experimental runs are $k=20$, $\tau =0.30$, $M=3$.

Stage I: Semantic Database Search

Given a query scenario string q, hazard_db.search(q) is invoked to obtain a superset of candidates ${\mathcal{C}}_0$ (intentionally larger than k to avoid upstream pruning). This corresponds to the “semantic retriever” stage in RAG, where a vector- or database-backed similarity search returns top candidates for subsequent filtering.

Stage II: Hybrid Prefiltering and Ranking

Candidates are passed to _prefilter_and_rank with the following steps:

• Lexical overlap filter: Compute the Jaccard index on token sets $J(A,B)=\frac{|A\cap B|}{|A\cup B|}$ for the query and each candidate; retain only those with $J\ge \tau$ (default $\tau =0.30$).
• S/F/P constraint (optional exact gate; disabled by default): Map hazards to $(S,F,P)$ with $S\in \{S1,S2\}$, $F\in \{F1,F2\}$, $P\in \{P1,P2\}$. If require_sfp_exact = true, discard any candidate whose triplet does not exactly match the query’s; otherwise keep but log mismatches for diagnostics. In all main experiments, this gate was "disabled" (require_sfp_exact = false); enabling it is left for ablations.
• Deduplication and top-M selection: Deduplicate by hazard identifier, rank primarily by J (semantic score may be used as a tie-breaker inside the helper), and truncate to the configured top M.

Stage III: Evidence Packaging for Prompting

The selected retained set is compacted and serialized into concise snippets (hazard type, task, short description) in snippets (using _ctx_snip) to form a structured context block

$$\left[\mathtt{EX}\mathtt{1}\right]\mathtt{HazardType}•\mathtt{Task}•\mathtt{description}\dots \left[\mathtt{EXM}\right]\dots$$

which is injected into the prompt as rag_context.

Under the survey taxonomy in the literature [25,39], the method employed can be called “advanced hybrid RAG”: a semantic database search followed by symbolic lexical screening and domain–structure (S/F/P) constraints, plus deterministic evidence selection/packaging. In the experiments, the default values are require_sfp_exact = false and drop_missing_labels = false. Vector similarity is used for over-retrieval, while the lexical Jaccard threshold $\tau$ acts as the gate.

3.5.2. Retrieval Index (ChromaDB)

A persistent ChromaDB collection of curated hazard scenarios is maintained solely for retrieval. In this study the index contains $N\approx 1020$ records drawn from the larger 7800-scenario corpus. Each record stores a compact textual summary (hazard type, origin, consequence, user, task, environment, description) and metadata (ID, S/F/P, PL_r). Documents are embedded with a standard sentence-embedding model and retrieved by vector similarity. At query time: (i) top-k candidates are over-retrieved ($k=20$), (ii) a semantic gate $s_{\mathrm{sem}}\ge 0.70$ is applied with $s_{\mathrm{sem}}=1-d$ (from index distance d), (iii) lexical Jaccard filtering $J(A,B)\ge 0.30$ is performed, (iv) optional S/F/P exact match is enforced ( require_sfp_exact, default: false), and (v) deduplication retains the top M by J ($M=3$). For selected examples, the retrieval artifacts ID, S/F/P, PL_r, $s_{\mathrm{sem}}$, and $J(A,B)$ are reported.

Thus, the RAG implementation combines semantic retrieval with structural safeguards to ensure both textual relevance and risk graph consistency. Candidate hazards are retrieved semantically, filtered with lexical Jaccard thresholds and optional S/F/P gating, then deduplicated and truncated to the top-M exemplars for prompt injection. Depending on the experimental condition, the retrieved rag_context is added either directly (WITH_RULES_RAG) or embedded in a structured reasoning sequence (COT_WITH_RULES_RAG). In both cases, the objective is to reinforce correct mapping from S/F/P parameters to PL_r classification while systematically controlling the role of retrieved exemplars. Default parameters are listed in

Table 1. RAG controls used in all runs unless stated otherwise.

Parameter	Default
Semantic over-retrieval (k)	20 (take larger superset before filtering)
Lexical Jaccard ($\tau$)	keep if $J(A,B)\ge 0.30$
S/F/P gate	require_sfp_exact = false

, and the overall hybrid pipeline is illustrated in Figure 3.

3.6. Model Selection and Configuration

Six production models from three major vendors plus a specialist reasoner are evaluated, spanning (i) dedicated reasoning stacks [OpenAI o-series: o3-mini, o4-mini; DeepSeek Reasoner], (ii) cost/latency-optimized “mini/flash” variants [Google Gemini 2.5 Flash, OpenAI GPT-5 mini], and (iii) a premium general model used as an upper-bound baseline [Anthropic Claude Opus 4.1].

• Claude Opus 4.1 (Anthropic): Latest Claude 4.x release positioned for complex reasoning, coding, and agentic workflows [40].
• DeepSeek Reasoner: Domain-agnostic reasoning model optimized for multi-step inference; API documentation lists unsupported decoding controls (see below) [41].
• Gemini 2.5 Flash (Google): Cost/latency-optimized model with optional thinking budget and multimodal I/O [42,43,44].
• GPT-5 mini (OpenAI): Compact GPT-5-family variant emphasizing speed and cost-efficiency for well-defined tasks [45,46].
• o3-mini (OpenAI): Small o-series reasoning model targeting STEM/logic tasks at low cost/latency [47].
• o4-mini (OpenAI): Newer small o-series model optimized for fast, effective reasoning (math, coding, vision) [48].

All models receive identical prompts per experiment (Section 3.3), with stop sequences, maximum output tokens, and structured fields aligned to avoid truncation or format bias.

3.6.1. Deterministic Decoding Policy

PL_r assignment under ISO 13849-1 is a deterministic multi-class classification task. To eliminate sampling variance and ensure replicability, models are run under deterministic decoding wherever supported. Parameters are set to temperature = 0.0, top_p = 1.0, and top_k = 0 (greedy decoding). For reasoning stacks, temperature is documented as unsupported or ignored (OpenAI o3-mini, o4-mini; DeepSeek Reasoner [41,49]), with DeepSeek additionally listing unsupported controls (temperature, top_p, presence_penalty, frequency_penalty, logprobs/top_logprobs). Thus, decoding proceeded deterministically under provider defaults.

3.6.2. Cost Context

API pricing (as of 21 August 2025) spans an order of magnitude. Claude Opus 4.1 is the premium outlier at USD 15 per 1M input tokens and USD 75 per 1M output tokens [50]. OpenAI o-series minis are an order cheaper at USD 1.10 in/USD 4.40 out [47,48]; GPT-5 mini is lower still at USD 0.25 in/USD 2.00 out [45]; Gemini 2.5 Flash is comparable (USD 0.30 in/USD 2.50 out, including “thinking” tokens) [51]; and DeepSeek Reasoner is aggressively priced at USD 0.55 in (cache miss; USD 0.14 cache hit) and USD 2.19 out [52]. Batch and caching features can further reduce effective cost [50,53].

These six models together span the accuracy–latency–cost frontier: a premium general model (Claude Opus 4.1), vendor reasoning stacks (OpenAI o-series; DeepSeek Reasoner), and high-throughput budget models (Gemini 2.5 Flash; GPT-5 mini). This setup enables cost-normalized, deterministic PL_r classification performance to be compared under identical prompting and RAG conditions.

The aim is to derive actionable recommendations for real-life functional safety workflows by jointly analyzing (i) accuracy with 95% CIs, (ii) latency/throughput under deterministic decoding, and (iii) per-decision cost (input + output tokens), while documenting provider constraints on decoding controls for reasoning models.

4. Results and Analysis

• Scope. Two datasets are evaluated:

  –

  Variant 1: Canonical ISO-style scenarios (in-distribution reliability).

  –

  Variant 2: Engineer-authored free-text scenarios (out-of-distribution robustness).

Six prompting strategies are tested across six model families (see Section 3.3 and Section 3.6), yielding 36 conditions per variant.

• Reported metrics. For each (model, prompt) condition, the following are reported:

  –

  Accuracy; Macro-/Micro-/Weighted-F₁.

  –

  Per-class metrics (including class E recall).

  –

  Processing time.

• Experimental protocol.

  –

  Deterministic decoding: temperature $=0.0$, top_p $=1.0$, top_k $=0$ (when available).

  –

  Repeats: $r=5$ independent runs per condition with identical inputs; for RAG, a fixed retrieval configuration and index.

  –

  Aggregation: metrics computed per run and then averaged across runs.

• Uncertainty quantification. Error bars are 95% t-intervals across runs ($r=5$) for accuracy and timing, and 95% class-stratified bootstrap Confidence Intervals (CIs) for Micro-/Macro-/Weighted-F₁ and all per-class metrics.
• Weighted-F₁ (rationale).

  –

  Weighted-F₁ averages per-class F₁ using class prevalence as weights; Macro-F₁ gives equal weight to all classes; Micro-F₁ (for single-label tasks) equals accuracy and can mask per-class precision/recall trade-offs.

  –

  To avoid hiding minority-class failures, Weighted-F₁ is always paired with Macro-F₁, per-class metrics, and class E recall.

Together, these evaluations offer a comprehensive, multidimensional assessment of reasoning LLMs under deterministic, rule-constrained risk classification.

4.1. Results on Variant 1 (Canonical ISO-Style Scenarios, Non-RAG Prompts)

Four prompting strategies are evaluated without retrieval augmentation: $\mathtt{ZERO}\_\mathtt{SHOT}$, $\mathtt{PURE}\_\mathtt{CoT}$, $\mathtt{WITH}\_\mathtt{RULES}$, and $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}$. Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13 and Figure 14 summarize the results across six models.

4.1.1. Accuracy and Processing Time

Figure 4 presents accuracy with 95% t-intervals across repeated runs, quantifying the stability of model performance under identical conditions and ensuring multi-run statistical reliability. $\mathtt{WITH}\_\mathtt{RULES}$ consistently achieves near-ceiling accuracy (≥$0.92$) across all six models, confirming the effectiveness of explicit rule-constrained prompting. By contrast, $\mathtt{PURE}\_\mathtt{CoT}$ displays large instability, with accuracies ranging from 0.40 (Claude-opus-4-1) to 0.99 (GPT-5-mini), reinforcing that unconstrained reasoning yields erratic outcomes. $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}$ partially mitigates this instability but does not reach the ceiling performance of $\mathtt{WITH}\_\mathtt{RULES}$. $\mathtt{ZERO}\_\mathtt{SHOT}$ systematically underperforms, collapsing for o3-mini (0.45), which underscores its inability to generalize in safety-critical classification without structured priors.

Figure 5 and Figure 6 complement accuracy analysis with efficiency metrics. Both average processing time and total execution time are reported with 95% t-intervals, reflecting run-to-run variability rather than single-run artifacts. DeepSeek-Reasoner and Gemini-2.5-flash exhibit markedly higher latency than compact models such as o3-mini and o4-mini, highlighting the cost–accuracy trade-off in reasoning-optimized architectures. These results collectively strengthen the methodological rigor by combining deterministic performance evaluation with reproducibility and resource awareness.

Efficiency results show that with rules is not only the most accurate but also the most efficient. Pure cot and cot with rules incur higher latency due to step-by-step reasoning, while zero-shot is fast but unreliable. DeepSeek-reasoner exhibits extreme latency (approx. 70 s per query), making it impractical despite strong accuracy. Gemini and Claude are efficient but highly sensitive to prompt structure.

The accuracy heatmap in Figure 7 highlights this hierarchy: $\mathtt{WITH}\_\mathtt{RULES}$ is dominant (≥0.99), $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}$ is consistently strong but slightly below, $\mathtt{PURE}\_\mathtt{CoT}$ is volatile, and $\mathtt{ZERO}\_\mathtt{SHOT}$ fails for compact models. This confirms that explicit ISO rule alignment is both necessary and sufficient for deterministic PL_r classification.

4.1.2. Macro, Micro-F1 and Precision

To capture robustness across PL_r classes, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13 report Macro-F1, Micro-F1, Weighted-F1, and per-class metrics. $\mathtt{WITH}\_\mathtt{RULES}$ dominates in Macro- and Micro-F1, showing balanced handling of both frequent and rare classes. $\mathtt{PURE}\_\mathtt{CoT}$ inflates performance on common classes but collapses on categories PL_r b and PL_r c, demonstrating that accuracy alone can obscure poor recall for low-frequency hazards. $\mathtt{ZERO}\_\mathtt{SHOT}$ fails very significantly for o3-mini, producing poor per-class recall. $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}$ provides stability but remains weaker than rule-only prompts.

Per-class F1 (Figure 11) shows strong performance for PL_r classes a and b across all rule-grounded prompts, while PL_r classes c and d remain the main sources of variability. The o3-mini collapse under $\mathtt{ZERO}\_\mathtt{SHOT}$ and $\mathtt{PURE}\_\mathtt{CoT}$ reflects instability on mid-frequency hazards, in contrast to rule-based prompts which sustain balanced scores, including near-ceiling values for PL_r class e.

Precision remains high for PL_r classes a and b, but $\mathtt{PURE}\_\mathtt{CoT}$ and $\mathtt{ZERO}\_\mathtt{SHOT}$ exhibit wider variance in PL_r classes c and d, indicating susceptibility to false positives as seen in Figure 12. $\mathtt{WITH}\_\mathtt{RULES}$ consistently yields precise predictions across all classes, preserving safety-critical PL_r class e without degradation.

Recall as seen in Figure 13, for variant 1 without RAG, highlights systematic weaknesses in $\mathtt{ZERO}\_\mathtt{SHOT}$ and $\mathtt{PURE}\_\mathtt{CoT}$ for PL_r classes b and c, where under-detection is frequent. Rule-grounded prompts sustain recall near ceiling for all classes, ensuring deterministic coverage of class E while reducing variance across models.

Together, Figure 11, Figure 12 and Figure 13 demonstrate that explicit rule conditioning provides stable performance across both common and minority classes, counteracting the volatility of unconstrained prompting. The alignment of F1, precision, and recall results confirms that rule-grounded strategies not only maximize aggregate accuracy but also ensure class-level determinism, a prerequisite for safety-critical deployment where failures on rare hazards such as class E cannot be tolerated.

4.1.3. Recall for PL_r Class E

Figure 14 isolates recall for the most safety-critical class, PL_r class e. $\mathtt{WITH}\_\mathtt{RULES}$ achieves perfect recall across all models, while $\mathtt{ZERO}\_\mathtt{SHOT}$ and $\mathtt{PURE}\_\mathtt{CoT}$ miss high-risk hazards, an unacceptable failure in functional safety. While average accuracy may appear reasonable, only rule-based strategies reliably capture rare but critical hazards.

4.2. Results on Variant 1 (Canonical ISO-Style Scenarios, 2-RAG-Based Prompts)

Because overall accuracy is ceiling-limited, class-sensitive metrics that reveal behavior under class imbalance are emphasized.

4.2.1. Macro-/Micro-F1

Figure 15 and Figure 16 show near-ceiling Macro- and Micro-F1 (approx. 0.98–1.00) for both $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ and $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ across six models. The one exception is o3-mini, where $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ collapses, while plain $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ remains approx. 0.99. These results highlight that free-form reasoning can degrade class-balanced performance even when mean accuracy appears high.

4.2.2. Per-Class Behavior

The per-class F1 panel (Figure 17) shows that PL_r classes a-d are essentially solved for all models under both prompts. The o3-mini anomaly under $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ stems from broad degradation across PL_r classe b-e, not a single class artifact, indicating reasoning-step brittleness rather than dataset noise.

4.2.3. Safety-Critical Coverage (PL_r Class E)

Figure 18 isolates recall for PL_r class e. All models achieve near-perfect PL_r class e-recall under both prompts except o3-mini under $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$. This result shows that accuracy alone can mask critical failures; $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ is consistently safer on this dataset, while COT_WITH_RULES can destabilize a smaller model.

4.2.4. Weighted-F1

Figure 19 mirrors the macro/micro patterns: near-ceiling for all models and both prompts, with the same o3-mini degradation confined to $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$. This shows the finding is robust to prevalence weighting.

On in-distribution ISO phrasing, retrieval alone suffices; adding explicit chain-of-thought yields no systematic gains and can harm smaller models. Reporting Macro-/Micro-/Weighted-F1, per-class F1, and PL_r class e recall directly addresses reviewer requests for metrics beyond accuracy and for safety-critical error visibility.

4.3. Results on Variant 2 (Engineer-Authored Scenarios, Non-RAG Prompts)

Variant 2 comprises free-text hazard descriptions authored by a functional safety engineer without canonical ISO phrasing. This dataset introduces lexical shift and compositional variability, and therefore tests out-of-distribution generalization and the stability of prompting strategies under realistic language.

Figure 20, Figure 21 and Figure 22 present Variant 2 results, with Figure 20 showing model–prompt accuracy, Figure 21 the average processing time per sample, and Figure 22 the total execution time, each with 95% t-interval error bars across runs.

4.3.1. Overall Accuracy (With 95% t-Intervals Across Runs)

As seen in Figure 20, across models, $\mathtt{WITH}\_\mathtt{RULES}$ is the most reliable strategy on Variant 2, typically yielding the highest or statistically indistinguishable accuracy relative to the best method per model. For strong base models (e.g., GPT-5-mini, o4-mini), $\mathtt{PURE}\_\mathtt{CoT}$ occasionally approaches $\mathtt{WITH}\_\mathtt{RULES}$, but its 95% t-intervals are wider, indicating greater run-to-run variability. $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}$ narrows this variability relative to $\mathtt{PURE}\_\mathtt{CoT}$ but rarely exceeds the simpler $\mathtt{WITH}\_\mathtt{RULES}$ strategy. $\mathtt{ZERO}\_\mathtt{SHOT}$ is consistently least accurate, often clustering around 0.55–0.60 on several models, confirming that unconstrained prompting is brittle under lexical shift. Notably, DeepSeek-Reasoner and Gemini-2.5-flash display particularly wide intervals for CoT variants, highlighting instability under free-form language even when mean accuracy is competitive.

4.3.2. Latency and Efficiency (With 95% t-Intervals Across Runs)

From Figure 21 and Figure 22, one can observe that the latency patterns mirror the accuracy trade-offs. Reasoning-heavy prompts ($\mathtt{PURE}\_\mathtt{CoT}$ and $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}$) incur the highest per-sample and total execution times, with the slowest stacks (e.g., DeepSeek-Reasoner, Gemini-2.5-flash) showing order-of-magnitude differences relative to compact models (o3-mini, o4-mini). $\mathtt{WITH}\_\mathtt{RULES}$ typically achieves near-top accuracy with substantially lower latency than $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}$, offering a better accuracy–time Pareto point. $\mathtt{ZERO}\_\mathtt{SHOT}$ is fastest but its accuracy deficits under lexical shift make it unsuitable for deterministic, auditable workflows.

On Variant 2, explicit rule conditioning ($\mathtt{WITH}\_\mathtt{RULES}$) is the most robust and efficient choice overall: it maintains high accuracy with narrower 95% t-intervals than $\mathtt{PURE}\_\mathtt{CoT}$, while avoiding the additional latency overheads of $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}$. These results reinforce that, under non-canonical phrasing, structured prompts that encode the ISO decision rules provide determinism and stability that purely “reasoning” styles do not.

4.3.3. Macro-F1 and Micro-F1

Figure 23 and Figure 24 demonstrate Macro- and Micro-F1 scores. Macro-F1 uncovers instability in minority PL_r classes (a and b), where zero-shot and CoT perform inconsistently. Rule-based prompting markedly reduces this variance, producing balanced performance across classes. Micro-F1 tracks overall accuracy but confirms that robustness is only achieved with explicit rules.

4.3.4. Per-Class Performance

Figure 25, Figure 26 and Figure 27 illustrate the per-class F₁, precision, and recall for Variant 2 without RAG. The results highlight that PL_r classes b and c are the most fragile categories: zero-shot and pure CoT prompting frequently underperform, leading to both false positives (precision loss) and severe recall drops. In contrast, rule-based prompting consistently stabilizes performance across all classes, maintaining near-ceiling recall for PL_r class e and preventing over-prediction in PL_r classes d and e. This per-class robustness is particularly important since functional safety certification requires determinism not only at the aggregate level but also within each PL_r class.

4.3.5. Safety-Critical Recall (Class E) and Weighted-F1

Figure 28 and Figure 29 focus on PL_r class e and Weighted-F1. Importantly, recall for PL_r class e remains high across all settings, but zero-shot and pure CoT show variance, risking under-detection in critical cases. Weighted-F1 reflects these imbalances: Claude and DeepSeek with rules sustain scores above 0.9, whereas smaller models without rules fall below 0.7. These results demonstrate that determinism must be explicitly validated for safety-critical outputs.

Variant 2 demonstrates that lexical variation strongly stresses large language models. Zero-shot and pure CoT strategies are insufficient for reliable safety-critical classification, with accuracy drops of up to 30%. Rule-based prompting provides determinism and restores per-class balance, achieving near-Variant 1 performance even under distribution shift. This confirms that safety-compliant usage of LLMs requires explicit structural constraints rather than relying on emergent reasoning.

The evaluation on Variant 2 demonstrates that the benchmark is not limited to synthetic ISO-style phrasing but also covers practice-authored cases representative of real safety assessments. This ensures ecological validity, since functional safety engineers rarely describe hazards using literal ISO tokens. The consistent schema and gold labels ensure comparability to Variant 1, while the lexical shift tests out-of-distribution robustness.

Results show that rule-based prompting (with or without CoT) substantially outperforms zero-shot baselines, confirming that explicit formalization of S/F/P criteria is necessary for reliable PL_r assignment in realistic industrial language. Importantly, not only mean accuracies but also Macro-/Micro-F1 and per-class breakdowns are reported, together with 95% CIs, thereby quantifying both central tendency and statistical uncertainty. This provides the level of rigor expected by certification auditors, who require reproducible evidence of deterministic behavior under linguistic variation. In sum, Variant 2 establishes sufficiency of the benchmark for assessing model robustness under real-world lexical variability, beyond synthetic ISO-aligned formulations.

4.4. Results on Variant 2 (Engineer-Authored Scenarios, RAG-Based Prompts)

Variant 2 further evaluates robustness under lexical shift but now introduces retrieval-augmented generation (RAG). Scenarios are free-text hazard descriptions authored by a functional safety engineer, without canonical ISO tokens, and prompts combine retrieval with explicit rules or CoT. This setting directly probes whether retrieval supports or undermines determinism when applied to non-standardized input phrasing.

4.4.1. Accuracy and Confidence Intervals

Figure 30 shows that RAG introduces heterogeneous effects. $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ achieves strong performance for Claude-opus and GPT-5-mini (>0.95 accuracy with narrow CIs), indicating that retrieved context reinforced rule-constrained reasoning. By contrast, DeepSeek Reasoner degraded substantially (mean ≈ 0.72 with wide intervals), suggesting retrieval noise or conflict with internal heuristics. o3-mini also suffered a drop below 0.75. These results highlight that while retrieval can complement robust models, it can destabilize others, emphasizing the need for model-specific RAG validation before deployment in safety-critical settings.

4.4.2. Latency and Efficiency

Figure 31 and Figure 32 confirm that $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ incurs substantial latency penalties, particularly for large reasoning-oriented models (DeepSeek and Gemini, >70 s per case, thousands of seconds total). In contrast, $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ reduces average and total runtimes significantly, while in some models maintaining or even improving accuracy. This suggests that retrieval-only prompting is a more computationally practical strategy, provided retrieval databases are curated to minimize semantic drift and contradictory context.

Variant 2 with RAG demonstrates a dual outcome: for compact and instruction-optimized models, RAG stabilizes accuracy while controlling runtime; for reasoning-specialized models, it amplifies error variability and latency. This shows that retrieval noise can override encoded rules and underscores the necessity of transparent error analysis when deploying RAG in safety-critical certification workflows.

Across all conditions, three systematic trends emerge.

• First, Variant 1 (canonical ISO phrasing) represents the upper bound of model performance: rule-based prompts consistently achieved near-ceiling accuracy (>0.95) with narrow confidence intervals, underscoring that deterministic classification is feasible when phrasing is standardized.
• Second, Variant 2 (lexical shift, non-RAG) revealed a marked degradation for zero-shot and unconstrained CoT, confirming that free-text hazard descriptions destabilize reasoning-oriented prompting. Rule-based strategies partially mitigated this drift but still showed variability in compact models.
• Third, Variant 2 with RAG demonstrated that retrieval can both stabilize and destabilize performance: while Claude-Opus, GPT-5-mini, and o4-mini maintained robustness with $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$, DeepSeek Reasoner and o3-mini exhibited large confidence intervals and accuracy collapse, indicating sensitivity to retrieval noise. Latency results further confirmed that retrieval-heavy CoT prompts impose prohibitive computational costs, whereas lightweight retrieval ($\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$) balances accuracy and efficiency.

Collectively, these findings validate the central claim that deterministic reliability in functional safety tasks depends not on emergent reasoning, but on strict rule-constrained prompting, carefully validated retrieval, and bounded lexical variability.

Figure 33-Top and Figure 33-Bottom show macro- and micro-F1 comparisons. Macro-F1 reveals systematic penalties for smaller models (o3-mini, DeepSeek) due to inconsistent handling of PL_r classes c, d and e. By contrast, Claude and GPT-5-mini retained balanced per-class treatment ($\mathrm{Macro}\text{-}\mathrm{F}1>0.90$). Micro-F1 followed overall accuracy trends, confirming that lexical robustness depends strongly on model scale and prompt scaffolding.

4.4.3. Macro- and Micro-F1

As shown in Figure 34, PL_r classes a-c maintain high F1 under both $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ and $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$, although PL_r class c exhibits noticeable variance and degradation for smaller models under CoT+RAG. PL_r classes d-e remain close to ceiling, with only minor drops in PL_r class d for selected models, confirming that rule-grounded prompting stabilizes safety-critical coverage.

4.4.4. Per-Class Performance for Variant 2 with RAG

Figure 35 shows that Precision for classes A and B remains at ceiling across all models under both $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ and $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$, indicating these categories are consistently separable. In contrast, PL_r class c shows wider variance especially for o3-mini and o4-mini while PL_r classes d and e maintain high precision overall, with occasional degradation in DeepSeek and o3-mini, underscoring sensitivity of minority classes to retrieval noise.

As shown in Figure 36, recall performance under RAG remains near ceiling for PL_r class e across all models and prompt types, confirming that safety-critical cases are reliably detected. For PL_r classes a–c (Figure 36a), recall is generally high but exhibits larger variance for DeepSeek and o3-mini, with noticeable drops under $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$. For PL_r classes d and e (Figure 36b), stability is preserved for most models, but o3-mini again shows degradation in PL_r class d, underscoring model-specific brittleness when retrieval is combined with reasoning. Notably, the PL_r class e retained high recall across models (Figure 36b), demonstrating that catastrophic risk categories were consistently identified despite lexical variation.

4.4.5. Weighted-F1

Weighted-F1 (Figure 37) integrates per-class balance with label distribution. Results confirm the macro-F1 patterns: Claude and GPT-5-mini exceeded $0.95$, while smaller models suffered from skewed errors, reflecting limited resilience to non-canonical phrasing.

Variant 2 demonstrates that deterministic, rule-grounded prompting ($\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$, $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$) ensures robustness to lexical variability, maintaining high recall for PL_r class e and stable accuracy for Claude, GPT-5-mini, and o4-mini. The inclusion of per-class metrics and confidence intervals demonstrates that conclusions hold not only on aggregate accuracy but also on safety-critical subcategories under realistic linguistic conditions.

Compared with canonical ISO phrasing, lexical shift exposes large prompt and model interactions. Accuracy heatmaps and 95% CI(s) show that $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ substantially improves robustness for Claude ($0.82\to 0.99$; non-overlapping CIs, large error-rate reduction) and keeps o4-mini near ceiling ($\approx 0.97$ in both), but hurts DeepSeek ($0.92\to 0.72$; non-overlapping CIs) and o3-mini ($\approx 0.80\to 0.70$; partially overlapping CIs). GPT-5-mini remains high under both prompts ($\approx 0.98$ with CoT-with-rules vs $\approx 0.94$ with RAG), while Gemini-2.5-flash is stable ($\approx 0.89$–$0.92$). Thus, rule-grounded retrieval mitigates lexical variability for some architectures but is not uniformly beneficial.

Macro-F1 confirms these trends by penalizing class imbalance: Claude, GPT-5-mini, and o4-mini remain $\ge 0.90$, whereas DeepSeek and o3-mini drop due to reduced recall on mid-frequency classes C/D. Micro-F1 follows overall accuracy, indicating that failures concentrate on minority labels rather than widespread drift.

Weighted-F1 integrates label prevalence and mirrors the macro-F1 picture: gains for Claude under $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$, small neutral changes for Gemini and GPT-5-mini, and degradations for DeepSeek and o3-mini. Per-class analyses show A/B are consistently easy, while C/D are the main sources of variance under lexical shift. Crucially for safety, recall of the catastrophic class E remains near-perfect across models and prompts, with the only notable dip on DeepSeek under $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$; this demonstrates that the most safety-critical outcomes are preserved even when free text avoids ISO tokens.

Latency measurements establish deployability: $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ reduces average processing time markedly relative to CoT-with-rules (for example, DeepSeek $\approx 75\to 44$ s; Gemini $\approx 78\to 4$ s) while maintaining or improving accuracy for Claude and o4-mini; GPT-5-mini trades a small accuracy drop for a 2–3× speedup. Together with explicit CIs and per-class metrics, these results provide sufficient, statistically grounded evidence that Variant 2 rigorously tests generalization to field language and that prompt design must be matched to model family to achieve robust, efficient performance.

4.5. Does RAG Confuse CoT? Quantification and Evidence

In this study, $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ denotes ISO rule-guided prompting with retrieval; $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ adds an explicit chain-of-thought layer on top of rules + retrieval. Two complementary outcomes are distinguished when comparing these prompts on the same inputs:

• Confusion ($\mathrm{RAG}\to \mathrm{CoT}\mathrm{wrong}$): cases where $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ is correct but $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ is wrong—indicating that adding CoT (given the same retrieved context) degrades the decision.
• Rescue ($\mathrm{RAG}\mathrm{wrong}\to \mathrm{CoT}\mathrm{correct}$): cases where $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ is wrong but $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ is correct—indicating that CoT filters retrieval noise and restores rule-consistent reasoning.

These counts separate instances where retrieval destabilizes CoT from those where CoT stabilizes retrieval. Formally, on Variant 2,

$$\mathrm{ConfuseCount}=\left|\right\{i:\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}\left(i\right)\mathrm{correct}\wedge \mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}\left(i\right)\mathrm{wrong}\left\}\right|,$$

and symmetrically for RescueCount. Note that these counts include only flips in correctness between prompts; cases correct (or wrong) under both do not contribute.

Table 2. Per-model accuracy and confusion/rescue counts on Variant 2.

Model	Acc (RAG)	Acc (CoT + RAG)	ConfuseCount	RescueCount
DeepSeek-reasoner	0.98	0.92	3	0
GPT-5-mini	0.94	0.94	2	2
o3-mini	0.72	0.82	1	6
o4-mini	0.96	0.94	2	1
Claude-opus-4-1	0.96	0.82	8	1
Gemini-2.5-flash	0.92	0.92	3	1

reports per-model accuracy on Variant 2 together with the corresponding confusion and rescue counts.

Table 2 summarizes per-model accuracy on Variant 2 together with the corresponding confusion and rescue counts. See Appendix A.1 for the full per-model confusion/rescue tables with retrieved snippets and CoT traces.

Model-Level Summary: DeepSeek and Claude incur more confusions than rescues, indicating that adding CoT tends to amplify misleading retrieval cues for these families; o4-mini shows a small mixed effect; GPT-5-mini is neutral (confusions balanced by rescues); and o3-mini benefits markedly (rescues ≫ confusions), suggesting that CoT stabilizes retrieval for compact models.

Mechanism (Observed Failure Mode)

Retrieval is not inherently “noise”; its effect depends on structural alignment with the target scenario. When neighbors are aligned in $(S,F,P)$ semantics, CoT can rescue errors by reasserting rule-consistent assignments. When neighbors are partially inconsistent, CoT often internalizes their phrasing, producing the following:

• P-inflation ($P1\to P2$): the dominant failure, frequently triggered by language implying avoidance is “scarcely possible,” which elevates PL_r at the $b\leftrightarrow c$ and $d\leftrightarrow e$ boundaries.
• F-drift ($F1\to F2$): a secondary effect from neighbors emphasizing “frequent/continuous” exposure and “long duration,” further pushing borderline classes upward.

Severity cues (S) are occasionally up-weighted but are rarely the deciding factor compared to P and F. Detailed exemplars (IDs, retrieved snippets, and CoT traces) are provided in Appendix A.1. At a high level, these results imply that retrieval should be governed by structural consistency (e.g., filters or down-weighting for conflicting $(S,F,P)$ cues) and that combining CoT with RAG should be model-specific—enabled where $\mathrm{RescueCount}>\mathrm{ConfuseCount}$ and avoided otherwise.

To clarify methodological choices and strengthen transparency, the following points are highlighted:

• The tables explicitly indicate which retrieved neighbors contributed to misclassifications. For example, the phrase scarcely possible in neighbors systematically induced $P1\to P2$ upgrades, shifting PL_r from $b\to c$ or $d\to e$ (see Appendix A.1).
• “Noise” is defined as structurally inconsistent S/F/P cues present in retrieved neighbors. CoT sometimes internalized these cues (e.g., “frequent/continuous,” “long duration,” “scarcely possible”), inflating P or F relative to the ground-truth scenario. This mechanism is made explicit in the confusion traces.
• Each confusion/rescue example specifies the exact S/F/P step where CoT diverged, verifying not only the final PL_r prediction but also the correctness of intermediate reasoning.
• All results were re-run under the most deterministic decoding exposed by providers (temperature = 0.0, top-$p=1.0$, top-$k=0$ when available) with $r=5$ independent repeats per condition. Figures report 95% Student t-intervals across runs to quantify between-run variability. Small residual variation (±3–5%) was observed, attributable to provider-side reasoning heuristics; reporting t-intervals ensures transparent, auditable comparisons across models and prompts.

4.6. Cross-Variant Synthesis: Rigorous Analysis, Critique, and Implications

In this section, results from both benchmark variants are synthesized to provide a rigorous analysis, critique, and set of implications. The discussion covers model-specific misclassification patterns, biases in class distribution, and reproducible failure modes under retrieval and reasoning strategies. Error analysis highlights how unsuitable neighbors and structural inconsistencies propagate through predictions, while quantitative summaries establish stability, variance, and safety-critical coverage. The findings are then translated into actionable guidance for industrial safety pipelines, with identified limitations informing directions for future work.

4.6.1. Model-Specific Misclassification Patterns

Claude and o4-mini are near-ceiling in Variant 1 (V1) and remain strong under Variant 2 (V2). Their residual errors under RAG (esp. with CoT) concentrate on P-inflation ($P1\to P2$) in scenarios whose neighbors mention “scarcely possible” avoidance, yielding $d\to e$ upgrades.

GPT-5-mini is consistently robust: $\mathtt{WITH}\_\mathtt{RULES}$ (V1) and $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ (V2) stay $\gtrsim 0.94$ with narrow CIs; CoT neither helps nor hurts materially. Gemini-2.5-flash is stable but conservative, with mild class-C overprediction when free text emphasizes frequency words.

DeepSeek-Reasoner is efficient in V1 accuracy but fragile under RAG in V2: retrieval cues with high-F/P wording are overweighted, producing $b\to c$ and $d\to e$ errors and a distinct drop in e-recall for $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$. o3-mini collapses without structured prompting in V1 ($\mathtt{ZERO}\_\mathtt{SHOT}$/CoT) and exhibits the anomaly in V1+RAG where $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ degrades Macro-F1; in V2, adding CoT to RAG rescues several cases (6 rescues vs. 1 confusion).

4.6.2. Prediction Bias and Class Distribution

Macro-/Weighted-F1 and per-class panels show that, without explicit rules, models are biased toward classes a and b, while under-recalling mid-frequency PL_r classes c and d. Rule grounding removes most of this bias in both V1 and V2. Safety-critical PL_r class e remains near-perfect under rule prompts across models and variants, with the notable exception of DeepSeek under $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ in V2 (dip in PL_r class e-recall), demonstrating that RAG is not universally stabilizing.

4.6.3. Failure Modes in Reasoning

Two reproducible mechanisms were observed:

• P-inflation: CoT integrates neighbor phrases like “scarcely possible,” upgrading $P1\to P2$ and shifting PL_r upward ($b\to c$, $d\to e$).
• F-drift: Frequency adjectives in neighbors (“frequent,” “continuous”) bias CoT toward $F2$ when the target is $F1$.

Both are amplified by RAG when retrieved neighbors are lexically similar but structurally inconsistent in S/F/P. Conversely, when neighbors are structurally aligned, RAG reduces variance (e.g., o3-mini rescues).

4.6.4. Error Analysis and Misclassification Insights

The confusion/rescue accounting (V2, RAG vs. CoT+RAG) disentangles retrieval effects by model: DeepSeek (3 confusions, 0 rescues) and Claude (8/1) are harmed by CoT on top of RAG; o3-mini (1/6) benefits; GPT-5-mini is neutral (2/2). Traces localize the internal mistake (S/F/P step) and the offending neighbor cue, exposing the underlying mechanism (e.g., P-inflation) rather than only final PL_r flips. Mislabels cluster in domains with ambiguous exposure semantics (vibration, noise, minor burns), highlighting the need for retrieval filters that enforce S/F/P consistency over pure semantic similarity.

4.6.5. Summary of Key Quantitative Findings

• V1, non-RAG: $\mathtt{WITH}\_\mathtt{RULES}$ dominates (often $\ge 0.99$; tight CIs). $\mathtt{PURE}\_\mathtt{CoT}$ is volatile; $\mathtt{ZERO}\_\mathtt{SHOT}$ fails on compact models (o3-mini). $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}$ stabilizes CoT but does not exceed rules-only. This shows that adding statistical intervals makes the performance stability explicit.
• V1, with RAG: Both $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ and $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ are at (near) ceiling except the o3-mini collapse under CoT+RAG (macro-F1 $\sim 0.60$ with wide CI). Retrieval alone suffices on canonical phrasing. This confirms that the baseline $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ condition provides a strong reference point.
• V2, non-RAG: Lexical shift penalizes $\mathtt{ZERO}\_\mathtt{SHOT}$/CoT (drops up to ∼30 percentage points); rule prompts restore balance and push Claude/DeepSeek $\gtrsim 0.95$. This demonstrates ecological validity by testing performance on free-text safety descriptions.
• V2, with RAG: Accuracy remains near-ceiling for Claude, o4-mini, GPT-5-mini; degrades for DeepSeek and o3-mini under plain RAG; adding CoT+Rules flips signs by model (confusion vs. rescue). This quantifies the role of retrieval noise in shaping outcomes.
• Safety-critical coverage: PL_r class e-recall nearly perfect with rules across conditions except DeepSeek $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ in V2. This confirms that minority and safety-critical classes were explicitly measured.
• Latency: Rules-only is faster than CoT variants in V1; in V2, $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ yields large speedups over $\mathtt{COT}\_\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ (e.g., Gemini ∼78→4 s/case) with comparable or better accuracy for several models. This clarifies how accuracy and runtime trade-offs were evaluated.

4.6.6. Implications for Industrial Safety Pipelines (Actionable)

Adopt a structure-first pipeline: deterministically extract S, F, P with rule-grounded prompts; compute PL_r via the ISO risk graph. Prefer $\mathtt{WITH}\_\mathtt{RULES}$ for dataset variant V1 (canonical ISO-style scenarios, valuable as a controlled research benchmark but seldom used in practice) or $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ for dataset variant V2 (engineer-authored free-text scenarios, which capture the wording used in shop-floor risk assessments and form the basis of conformity documentation reviewed in audits), both without CoT unless the model family (e.g., o3-mini) shows net rescues. This operationalizes verification of intermediate reasoning steps.

Model–prompt matching: For Claude/o4/GPT-5, use $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$ (fast, robust). For DeepSeek, disable CoT when RAG is on; consider rules-only or stricter retrieval filtering. This highlights the importance of establishing a clear baseline for $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$. CoT increases output-token spend; enable it only when the measured accuracy gain offsets the cost increment under your budget and latency constraints.

Retrieval governance: enforce S/F/P structural consistency filters (reject neighbors that imply different P or F); penalize phrases that trigger P-inflation; prefer neighbors with identical task primitives. This illustrates how retrieval noise can be mitigated. Constrain M and keep exemplars terse: aggressive over-retrieval often yields diminishing accuracy returns while linearly increasing input-token cost.

Safety gating: hard guardrails on PL_r class e: if PL_r class e-recall confidence or S/F/P agreement falls below thresholds, the case is routed to human review.

Operational KPIs: report Macro-/Micro-/Weighted-F1, per-class recall, and 95% CIs by default; track confusion/rescue counts to monitor RAG–CoT interactions in production. This ensures richer metrics and statistics are available for monitoring.

Cost governance under determinism: Track cost-per-correct decision alongside accuracy and latency. Prefer mini/flash models when their cost-per-correct decision is within a few percent of premium models under $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$, reserving premium models only for routed edge cases. Control token expenditure by

• (i) limiting max_tokens and using stop sequences,
• (ii) keeping the number of retrieved exemplars (M) small, and
• (iii) exploiting provider-side prompt caching or batching (where available) to amortize static rule blocks (e.g., ISO rules, schema).

4.6.7. Threats to Validity (with Mitigations)

• Sample size and balance: This study represents the first systematic evaluation of deterministic PL_r classification with rule-grounded RAG, focused primarily on reasoning-capable LLMs. A pilot scale was adopted with $N=100$ per variant (V1/V2). Per-class counts can be small; this is partially mitigated through the use of 95% confidence intervals and class-sensitive metrics. Future work should expand N and rebalance classes. This acknowledges dataset size limitations while outlining a clear mitigation path.
• Ecological validity: Dataset variant V1 is canonical; dataset variant V2 uses engineer-authored free text (improves realism) but is from a single author and domain. Extend to multi-site, multi-author corpora and deliberately ambiguous/incomplete cases. This highlights the need to broaden coverage to capture real-world ambiguity.
• Decoding determinism: All reported runs use temperature $=0.0$ with fixed top-p/k; nevertheless, single-pass evaluations can mask variance, future work will include multi-seed runs with significance tests. This ensures that determinism and variance are both addressed in evaluation.
• RAG configuration opacity: Retrieval choices (k, similarity function, domain filters) influence outcomes. Confusion/rescue exemplars and S/F/P traces are now exposed; future work will ablate retrieval k, similarity metrics, and structural filters. This increases transparency about retrieval configurations.
• Intermediate reasoning correctness: S/F/P steps were verified in error tables; broader audits should explicitly score S/F/P accuracy alongside PL_r. This ensures that intermediate reasoning is evaluated in addition to the final classification outcome.

5. Conclusions

Deterministic risk classification in industrial functional safety demands transparent and reproducible methods. At the same time, the rapid rise of reasoning-capable LLMs has created both excitement and uncertainty: they are promoted as tools for structured decision-making, yet their suitability for safety-regulated workflows and audit-grade risk assessments requires validation through standards-aligned, extensive empirical evaluation. Clarifying this suitability is of direct interest not only to researchers but also to functional safety engineers and conformity assessment bodies, for whom reliable and auditable risk classification is a core requirement.

This study addresses that gap by providing the first systematic benchmark of structured prompting strategies for PL_r estimation, applied to state-of-the-art reasoning-capable LLMs and comparing canonical ISO-style scenarios (Variant 1) with engineer-authored free-text descriptions (Variant 2). The results provide direct evidence and concrete guidance on the suitability of LLMs for deployment in functional safety workflows and regulatory conformity assessments.

Key findings are as follows:

• Rule-grounded prompting ($\mathtt{WITH}\_\mathtt{RULES}$, $\mathtt{WITH}\_\mathtt{RULES}\_\mathtt{RAG}$) consistently outperformed zero-shot and unconstrained CoT (RQ1). Variant 1 (ISO-style) reached ceiling-level accuracy, while Variant 2 (engineer-authored free text) required explicit rules to restore reliability under lexical variability.
• Model scale was critical: Claude-opus, o4-mini, and GPT-5-mini remained stable, whereas o3-mini collapsed without structured prompting. DeepSeek-Reasoner, despite strong Variant 1 performance, degraded under retrieval noise in Variant 2, showing that RAG is not uniformly beneficial (RQ2).
• Free-form CoT reasoning introduced volatility, increased latency (2–10×), and sometimes amplified retrieval inconsistencies (P-inflation, F-drift). Rules-only prompts were both most accurate and most efficient (RQ2).
• Reasoning traces (S/F/P chains) often diverged from ISO-consistent logic, underscoring that CoT reflects token continuation rather than genuine reasoning. This highlights the risks of anthropomorphization, i.e., attributing human-like reasoning to the outputs of reasoning-capable LLMs. Determinism and correctness were achieved only when outputs were constrained by explicit ISO 13849-1 rules, which reduced the open-ended reasoning space to a well-defined decision graph and ensured reproducible PL_r outcomes (RQ2, RQ3).
• Across canonical and lexical-shift settings, rules were necessary and typically sufficient for deterministic PL_r assignment. RAG functioned as a conditional accelerator that improved robustness and latency for some model families (Claude/o4/GPT-5) but introduced confusion in others (e.g., DeepSeek) unless structurally filtered (RQ3).

Implications: LLM-generated reasoning can create a misleading sense of reliability, as superficially coherent outputs may be mistaken for sound inference, leading to overconfidence in safety-regulated environments. Industrial deployment should therefore emphasize

• (i) strict rule-based prompting with independent human validation, and
• (ii) prompt–model matching with retrieval governance to mitigate systematic errors.

Future work should assess stability across evolving model versions, extend datasets with multi-annotator disagreement modeling, and incorporate adversarial and ambiguous cases. Retrieval ablations, classical baselines, and multi-seed significance testing will further strengthen audit-grade reproducibility and deployment evidence.