Source Page Evaluation of Local RAG Retrieval for Engineering Safety: Implications for Mechanical and Civil Engineering

Abstract

Industrial safety question answering requires access to the exact source material that supports an answer, not merely a related document. This study evaluates the retrieval stage of a local retrieval-augmented generation (RAG) system at the source page level. Eighteen documents on lockout/tagout, machine guarding, and personal protective equipment were prepared in raw, cleaned, and metadata-enriched forms. Best Matching 25 (BM25), latent semantic analysis (LSA), and reciprocal rank fusion (RRF) hybrid retrieval were tested on 54 questions: 36 answerable, 12 safety-related but unsupported, and 6 outside the covered topics. Raw BM25 achieved the highest mean reciprocal rank (0.492) and Hit@5 (0.722), cleaned BM25 achieved the highest Hit@3 (0.556) and answerability AUROC (0.765), and metadata-enriched BM25 achieved the highest Hit@10 (0.833). For raw text, the retrieval method affected reciprocal rank (Friedman χ² = 7.13, p = 0.028). Hybrid retrieval exceeded LSA after Holm correction (p = 0.011), whereas BM25 and hybrid retrieval did not differ (p = 0.302). Text preparation had no overall effect within BM25 (p = 0.220). The results show that document-level evaluation can overstate access to exact evidence. The study provides a repeatable procedure for source tracking, page-level relevance labeling, separate retrieval and generation evaluation, and refusal analysis. These design principles are also relevant to construction safety knowledge systems that combine regulations, accident reports, project data, and BIM-linked information. Because the relevance set is a model-assisted silver standard, the findings remain exploratory rather than expert-certified absolute retrieval accuracy. A controlled same-run extension with all MiniLM L6 v2 and BM25 + MiniLM RRF is reported separately; its observed fusion gains did not reach statistical significance on the 36 answerable questions. The work is intended to support safety engineers, researchers, and developers designing evidence-grounded industrial and construction safety information systems.

1. Introduction

Industrial safety engineers routinely consult regulations, guidance documents, procedures, training material, incident narratives, and lessons learned. The practical value of a large language model (LLM) assistant in this setting depends not only on fluent generation but also on whether the system retrieves the precise and authoritative evidence required for a safety judgment. Unsupported or weakly grounded responses are especially problematic where an apparently minor omission can change the meaning of an energy control, guarding, or personal protective equipment (PPE) requirement.

Retrieval-augmented generation (RAG) combines a large language model with an external collection of source material, so answers can be based on retrieved passages rather than only on information stored in the model [1]. Best Matching 25 (BM25) is a word-based search method and remains a strong baseline for technical document collections that use precise terminology [2]. Embedding-based semantic search methods represent questions and passages in a shared mathematical space so that they can match similar meanings even when the wording differs [3,4]. Hybrid systems combine exact word- and meaning-based search, often through reciprocal rank fusion [5]. However, combining methods does not guarantee better results: a weak semantic ranking can reduce the value of a strong BM25 ranking by adding less useful pages.

These evidence retrieval challenges are also important in civil and construction engineering. Construction safety information is distributed across regulations; accident reports; project records; safety management systems; and, increasingly, BIM- and knowledge graph-based platforms. Recent reviews identify falls, material handling, heavy machinery, temporary support systems, PPE, training, supervision, and site conditions as recurring concerns, while digital construction research highlights the difficulty of integrating heterogeneous safety knowledge into decisions [6,7,8,9]. A retrieval system that returns only a related document rather than the exact supporting page can therefore mislead a site engineer, safety manager, or project manager, even when the generated answer sounds plausible.

Despite growing use of RAG- and LLM-based systems in engineering, the reviewed literature leaves an important evaluation gap. Existing safety studies commonly assess classification accuracy, generated recommendations, or end-to-end system outputs, while regulatory search studies often evaluate retrieval under different document structures and relevance units. As a result, it remains unclear how much reported document-level success translates into access to the exact source page needed to support a safety answer and whether the text preparation or retrieval method changes that page-level evidence access under controlled conditions. To address this gap without confounding retrieval with answer generation, the present study was designed as a controlled source page retrieval experiment. The document collection, text segmentation settings, question set, and relevance labels were fixed across methods. Retrieval quality, whether the documents support an answer, and answer generation were treated as separate stages, and the current paper evaluates only the retrieval and answer-support stages. This design enables direct comparison of word-based, semantic, and hybrid retrieval while preventing differences in generated wording from obscuring evidence of access failures.

The answer generation model is intentionally excluded so that differences in generated wording do not hide retrieval effects.

The study makes four contributions. First, it implements a repeatable retrieval pipeline with stable document identifiers, source details, checksums, and versioned text files. Second, it replaces broad document relevance with labels for the exact source pages that support an answer. Third, it separates retrieval, assessment of whether the documents support an answer, and answer generation into distinct stages while evaluating only the first two stages in the present experiment; this prevents a correct refusal from being counted as answer quality and prevents fluent generation from hiding missing evidence. Fourth, it provides paired statistical comparisons, analysis by safety topic and question type, and a transparent source-evidence labeling structure that supports future external adjudication without changing the archived retrieval results.

The scope is deliberately limited. The paper does not propose a new language model, claim deployment readiness, or evaluate answer generation. Its engineering contribution is the controlled evaluation of the evidence access layer that must operate before a local RAG assistant can produce a defensible safety response.

• The study addresses the following four research questions:
• RQ1. How do BM25, latent semantic analysis (LSA)-based semantic search, and hybrid retrieval differ in retrieving exact industrial safety evidence?
• RQ2. Do raw, cleaned, or metadata-enriched text versions improve retrieval?
• RQ3. Which safety domains and question types are most difficult?
• RQ4. Can retrieval scores distinguish questions that the documents can answer from unsupported and out-of-domain questions?

The remainder of the paper is organized as follows. Section 2 reviews related work in safety text mining, RAG evaluation, regulatory information retrieval, human oversight, and construction safety knowledge systems. Section 3 describes the document collection, text extraction, indexing, retrieval methods, evaluation questions, and statistical analysis. Section 4 reports the retrieval, answer support, domain level, and failure analysis results. Section 5 discusses the implications for industrial and construction safety, Section 6 summarizes limitations and future work, and Section 7 concludes the study. The appendices provide the source document inventory (Appendix A Table A1), the complete evaluation question set, review fields, and a reproducibility checklist.

2. Related Work

2.1. Safety Text Mining and Domain-Specific Language Models

Recent studies have applied language models to unstructured safety records, showing both the value of domain-specific text and the need for expert reference standards. Sabetta et al. [10] used GPT 4 Turbo to extract fatality, injury, and occupation information from OSHA lockout/tagout accident narratives. Ando et al. [11] evaluated four language models on 2619 same-level fall cases using expert manual classifications and Cohen’s kappa. Wang et al. [12] combined Bidirectional Encoder Representations from Transformers (BERT)-based topic modeling (BERTopic), a large language model, and social network analysis (SNA) to identify systemic risk patterns in 745 accident investigation reports. Yang et al. [13] prepared a 10 GB collection of coal mine safety documents and 10,000 high-quality task examples. These studies show that safety language applications benefit from carefully selected domain documents, clear labels, and comparison with expert judgments, but they do not compare word based, semantic, and hybrid evidence search under the same controlled conditions.

Other work focuses primarily on recommendation generation or specialized reasoning. Tumre et al. [14] found that off-the-shelf open-source models had limitations in the specificity and format of preventive action recommendations and reported improvement through domain-specific supervised fine tuning. Shi et al. [15] developed Aegis, a multi-agent architecture with reflective retrieval mechanisms for automotive functional safety tasks such as hazard analysis, safety requirement generation, and test case generation. Shi et al. [16] combined physics-informed graph prediction with a large language model for leakage and explosion emergency decision support. These systems support the broader premise that safety applications require domain grounding, yet their evaluations center on task outputs rather than the independent quality of retrieved evidence.

Recent deep learning work in adjacent engineering diagnostics further illustrates the value of adaptive representations under changing operating conditions. Yan et al. [17] proposed a meta learning-based graph convolutional wavelet network for multi-scale dynamic modeling of high-speed rail subsystems, Wang et al. [18] developed an adaptive fused-domain cycling variational generative adversarial network for fault diagnosis under data scarcity, and Wang et al. [19] proposed a dynamic collaborative adversarial domain adaptation network for unsupervised rotating machinery fault diagnosis. These studies address signal modeling, synthetic data fusion, and domain adaptation rather than document retrieval, so they are cited here as evidence of broader deep learning strategies for engineering generalization rather than as direct comparators relative to the present RAG retrieval experiment.

2.2. RAG- and Evidence-Grounded Safety Guidance

Baek et al. [20] developed a RAG system that retrieved construction accident cases and generated safety risk management guidance. The study is the closest domain-specific precedent in the reviewed library because it couples retrieval with safety recommendations and compares generated guidance with practitioner-developed outputs. Nevertheless, end-to-end guidance quality does not reveal whether retrieval failure, evidence selection, or generation caused a weak response. The present study therefore isolates the retrieval layer and evaluates the exact source page before any generator is scored.

Broader manufacturing studies reinforce the importance of controlled grounding but address different tasks. LLM MANUF, the integrated large language model manufacturing framework proposed by Du et al. [21], combines multiple fine-tuned models and ranking/fusion strategies for manufacturing decisions. SafetyGPT and mLLM CRD (multimodal large language model-based context-aware rule detection) focus on multimodal monitoring of unsafe behavior [22,23], while Wang and Zhou [24] combined a large language model agent with risk-sensitive reinforcement learning for human–robot operation in smart factories. These systems demonstrate that industrial artificial intelligence benefits from domain knowledge and structured evaluation, but they do not provide a controlled source page retrieval comparison.

2.3. RAG Evaluation and Regulatory Information Retrieval

Recent RAG evaluation frameworks examine retrieval and answer generation separately rather than combining them into one score. The Automated RAG Evaluation System (ARES) evaluates whether the retrieved context is relevant, whether the answer stays faithful to that context, and whether the answer addresses the question; it combines automated judges with a limited set of human-reviewed examples [25]. FaithEval tests situations in which the supplied context is unanswerable, inconsistent, or deliberately false and shows that strong language models may still fail to remain faithful to the evidence [26]. These findings support separate evaluation of retrieval quality, whether the documents support an answer, and faithfulness of the generated response in safety-oriented systems.

Research on regulatory information search also shows that exact word and semantic methods can complement each other and that domain adaptation matters. MST R combines domain-tuned semantic search, BM25-style sparse search, reciprocal rank fusion, and a second stage ranking step for regulatory documents [27]. FusionRAG similarly combines embeddings, BM25, Facebook AI Similarity Search (FAISS), and rank fusion in the Regulatory Information Retrieval and Answer Generation (RIRAG) shared task [28]. These studies support a controlled comparison of word-based and semantic search, but they do not isolate exact source page retrieval in industrial safety documents.

A recent review synthesized 78 peer-reviewed studies across robotics, fault diagnosis and maintenance, design and simulation, manufacturing knowledge systems, and related engineering workflows [29]. The review found a broad shift from prompt-only demonstrations toward RAG, knowledge graphs, multimodal grounding, digital twins, validation modules, and systems connected to engineering tools. It also identified inconsistent evaluation sets, weak grounding when information is unclear, incomplete integration with engineering software, and limited real-world validation. These findings directly support the present focus on evaluating the retrieval stage separately, linking every result to a traceable source page, carefully managing the source documents, and retaining human review.

2.4. Human Oversight and Trustworthy Use

Human validation is a recurring requirement in safety-critical artificial intelligence. SETTE, a human-centric operator safety framework, integrates large language models, extended reality (XR), the Internet of Things (IoT), and human in the loop validation for operator safety [30]. Vella et al. [31] concluded that a large language model-based system could support structured occupational health judgments but should complement rather than replace professional decision making. Bonnin et al. [32] discuss data quality, model limitations, variation across repeated runs, bias, and oversight risks in safety engineering and distinguish standalone language models from complete, validated artificial intelligence systems. Johnsen and Steen [33] further show that subcontractor-driven environments create fragmented responsibility, gaps in verification and knowledge transfer, and underreporting when psychological safety is weak. Together, these studies support separate evaluation of retrieval, whether the documents support an answer, and answer generation, and they reinforce the need for source traceability and documented expert review in projects involving multiple contractors.

2.5. Construction Safety and Civil Engineering Knowledge Needs

Construction projects provide a demanding setting for evidence-grounded safety information because hazards vary by project type, phase, work package, and subcontractor interface. Almaskati et al. [6] reviewed 236 publications and identified 18 hazard groups, with falls from height, material handling, and heavy machinery among the most widely reported. Torres et al. [8] combined a literature review, a survey of 52 construction managers, and analysis of 36 residential projects; they identified 29 accident factors and seven prevention strategy lines, including PPE use, temporary support systems, site conditions, workspace planning, and early digital simulation. Pham et al. [7] used 181 professional responses and an exploratory factor analysis–analytic hierarchy process (EFA AHP) framework to prioritize highrise safety factors related to training and inspection, employer responsibility, worker competence, working conditions, and safety equipment. Together, these studies show that civil engineering safety questions are context-sensitive and often require evidence tied to the project type and work condition rather than generic guidance.

Construction knowledge is also becoming more digital and interconnected. Ganiyev et al. [34] found that safety climate, safety behavior, and building information modeling were prominent themes in a bibliometric analysis of more than 15,000 construction safety publications. Yang et al. [9] systematically reviewed 70 studies on knowledge graphs in smart construction safety management and found that regulations, accident reports, project data, and BIM are common knowledge sources for accident analysis, hazard identification, decision support, and safety knowledge management. However, knowledge graph construction remains partly manual and dependent on expert input, while data integration, dynamic risk inference, and generalizability remain challenges. These findings strengthen the case for traceable page-level retrieval as a foundation that can connect unstructured safety documents to future BIM, digital twin, or knowledge graph systems.

Management evidence is equally important. Arévalo Sarrate et al. [35] evaluated a safety management system implementation across 302 construction projects in five countries and more than 23 million work hours, showing the value of longitudinal, standardized, and auditable safety data. Bourahla et al. [36] linked occupational health and safety to project performance and social sustainability, emphasizing legal compliance, training, inspections, safety procedures, and PPE. These studies suggest that a civil engineering RAG system should not be evaluated only on topical relevance; it should preserve the authority, project context, and management purpose of the retrieved evidence.

2.6. Research Gap and Study Position

Within the reviewed industrial and construction safety literature, we found no study that held the document collection, text segmentation settings, and questions constant while comparing word-based, semantic, and hybrid search against exact source page evidence. Construction studies provide hazard taxonomies, factor prioritization, safety management system evaluation, knowledge graphs, and end to end RAG guidance [6,7,8,9,20,35], but they do not isolate exact page retrieval under controlled search settings. General RAG and regulatory search research provides strong examples of component evaluation and method combination [25,26,27,28]. The present study therefore focuses on the level of detail at which retrieval is evaluated, the way source text is prepared, failures by safety topic, and whether the documents support an answer.

2.7. Methodological Synthesis and Design Implications

The literature suggests five design requirements for safety-oriented RAG. First, each system stage should be evaluated separately because an overall answer score can hide whether the problem began in retrieval, selection of the supporting context, or answer generation [25,26]. Second, word-based and semantic search should be compared using the same source pages because terminology-rich regulations may favor exact wording, while paraphrased operational questions may benefit from meaning-based matching [2,3,4,5,27,28]. Third, document selection and version control are part of system validity: issuing authority, jurisdiction, revision date, and document type determine whether a retrieved passage is a regulation, guidance, project record, or training material [32,37,38,39,40]. Fourth, independent expert review strengthens page relevance decisions when those decisions are used to support operational safety claims [30,31,32]. Fifth, construction applications should retain project type, phase, work package, contractor role, and links to BIM or other structured project information because safety priorities vary with the work context [6,7,8,9]. This interpretation is consistent with the broader intelligent manufacturing review, which concluded that many apparent gains from large language models actually come from better text preparation, search, coordination, and validation rather than from the base model alone [29].

These requirements shaped the experiment. The source document collection and text segmentation settings were fixed before the search methods were compared, repeated text segment results were combined into unique source pages, unsupported questions were retained as a separate test of whether the system should answer, and all current relevance decisions are described as model-assisted silver-standard labels. This framing turns the study from a general chatbot demonstration into a testable engineering information problem: whether the retrieval stage can provide the evidence needed for a traceable safety answer. Table 1 summarizes the reviewed studies and positions the present experiment relative to prior work.

Table 1. Comparison of reviewed industrial and construction safety studies relevant to the present work.

Study	Safety Task	How Evidence Was Used	Evaluation	Gap Relative to This Study
Sabetta et al. [10]	LOTO narrative extraction	Prompted LLM; no retrieval comparison	OSHA narratives	Information extraction, not source page retrieval
Ando et al. [11]	Accident classification	No retrieval layer	2619 expert-labeled cases	Classification rather than evidence access
Baek et al. [20]	Construction safety guidance	RAG over accident cases	Guidance compared with practitioners	End-to-end output; retrieval not isolated
Shi et al. [15]	Functional safety engineering	Agent tools and reflective retrieval	Different evidence unit and task definition	Complex agent tasks; not a controlled source page retrieval benchmark
Yang et al. [13]	Coal mine safety LLM	Large-domain document collection	10 GB of documents; 10,000 tasks	Document/model evaluation, not a search method comparison
Bonnin et al. [32]	Responsible safety engineering AI	Document management and validation principles	Conceptual risk analysis	No retrieval experiment
Saad Falcon et al. [25]; Ming et al. [26]	General RAG evaluation	Component-wise relevance, faithfulness, and difficult context testing	Human-validated evaluation sets	Not an industrial safety source page evaluation set
Malviya et al. [27]; Faisal et al. [28]	Regulatory information retrieval	Word-based and semantic search combined with domain adaptation	Shared task retrieval and QA metrics	Different regulatory tasks and source page definitions
Almaskati et al. [6]	Construction hazard identification	Systematic review of hazards and mitigation measures	236 publications; 18 hazard groups	Hazard taxonomy, not retrieval evaluation
Torres et al. [8]	Residential construction accident prevention	Literature, manager survey, and project analysis	52 managers; 36 projects	Context-specific factors; no document retrieval
Pham et al. [7]	Highrise construction safety prioritization	Survey evidence and expert weighting	181 professionals; EFA AHP	Prioritizes factors rather than evidence access
Arévalo Sarrate et al. [35]	Construction safety management systems	Longitudinal management and accident data	302 projects; five countries; >23 million work hours	Management system evaluation, not RAG retrieval
Yang et al. [9]	Smart construction safety knowledge management	Knowledge graphs using regulations, accident reports, project data, and BIM	Systematic review of 70 studies	Metadata-enriched knowledge focus; no exact page retrieval comparison

Taken together, the reviewed studies show strong domain motivation for evidence-grounded safety systems but leave unresolved how reliably a retriever reaches the exact supporting page when the document set, questions, segmentation, and labels are held constant. The methods in Section 3 operationalize this gap by freezing those experimental inputs, evaluating retrieval before generation, and scoring unique source pages rather than broad document relevance.

3. Materials and Methods

3.1. Source Documents and Document Management

The study uses 18 English-language PDF documents: six on lockout/tagout (LOTO), four on machine guarding, and eight on PPE. Sources include the Occupational Safety and Health Administration (OSHA), the UK Health and Safety Executive (HSE), and training or consultation material linked to recognized authorities. Each file was assigned a stable document identifier, source URL, safety topic, issuing authority, jurisdiction, and SHA 256 checksum. Because the documents come from different jurisdictions and years, these fields were retained with each page instead of treating all statements as interchangeable requirements. The collection includes OSHA LOTO and safeguarding guidance [38,39], HSE PPE guidance [37], and OSHA PPE guidance [40]. The unified ranking used in this experiment is a controlled retrieval condition for comparing methods; it is not proposed as an operational rule for combining regulatory requirements across jurisdictions. In deployment, the applicable jurisdiction should normally be established before retrieval and used as a metadata filter or routing constraint. If jurisdiction is unknown or if retrieved US and UK sources conflict in terminology, required procedure, or legal status, the system should display the authority and jurisdiction explicitly and refer the case for professional review rather than synthesize the requirements as though they were equivalent. Table 2 lists the document groups used in the experiment.

Table 2. Safety documents used in the retrieval experiments.

Domain	Documents	Primary Content	Source and Version Consideration
LOTO	6	Hazardous energy control, procedures, fact sheets, and training	OSHA-centered; publication years vary
Machine guarding	4	Safeguarding, mechanical hazards, and machinery safety	Overlapping training handouts and OSHA guidance
PPE	8	Hazard assessment, PPE selection and use, and sector-specific guidance	OSHA and HSE jurisdictions; broader construction documents

Documents were included when they contained substantial information about one of the three target safety topics and were available as stable, readable PDF files. Promotional material, duplicate copies, files with no usable text, and documents limited to unrelated safety topics were excluded. The final document collection contains 417 pages: 120 LOTO pages, 68 machine guarding pages, and 229 PPE pages.

A separate document review record was maintained outside the text used for retrieval. For each document, the record identifies the issuing body, publication or revision year where available, intended role in the experiment, and any concern about jurisdiction or whether the material is current. Regulations and guidance issued by recognized authorities were treated as the main evidence sources. Grant-funded training material and consultation manuals were kept as supporting material but were not described as binding requirements. This distinction matters because a search result can be topically relevant while still coming from a source with less authority or from a different jurisdiction. Figure 1 summarizes the controlled source page retrieval workflow used in the study.

Figure 1. Controlled source page retrieval workflow.

3.2. Text Extraction and Text Versions

Text was extracted separately from each PDF page. The retained corpus contained 18 PDFs and 417 source pages with usable machine-readable text; the retrieval pipeline did not apply image OCR as a preprocessing step. The source set nevertheless remained heterogeneous. It included guidance booklets, regulatory guidance, fact sheets, training manuals and handouts, a sample written program, an assessment guide, a quick card, and a slide deck export. Archived representation statistics recorded 723,217 raw extracted characters, with large differences in page text density across sources. A later independent PDF layout re-extraction produced 726,618 characters in the current software environment; that auxiliary count is used only for layout and runtime diagnostics and does not replace the archived text segment inventory. Three experimental text versions were generated from the same evidence: raw extraction, cleaned text, and metadata-enriched text. The raw version preserved extraction noise; the cleaned version normalized spacing, broken lines, and repeated formatting artifacts; and the metadata-enriched version prepended short document, topic, source, and page fields. The third condition was intentionally included to test whether source metadata helps or interferes with retrieval; it was not imposed on the raw or cleaned conditions.

Extraction and Normalization Quality Controls

Extraction was reviewed at the page level rather than only as a whole document stream. The processing pipeline retained page identifiers, normalized repeated whitespace, repaired line-break hyphenation when a word was clearly split across lines, and removed recurrent headers or footers only when they did not carry evidential meaning. A separate automated layout audit of all 417 retained PDF pages identified 198 pages containing at least one embedded image object, 27 pages with fewer than 100 extracted characters, and 79 pages with fewer than 300 extracted characters. A conservative text fragment heuristic flagged 72 pages as having potential multi-column or side-by-side reading-order risk; these flags are automated risk indicators rather than manually adjudicated layout labels. The corpus audit also identified the 59-page PPE slide deck export (PPE 06) as a low text source requiring inspection and identified spacing artifacts in PPE 07. Raw extracted text density ranged from approximately 192 characters per page in PPE 06 to approximately 4371 characters per page in PPE 08. These differences are reported as corpus properties because they can influence extraction quality, candidate exposure, and retrieval behavior.

The experiment evaluates textual evidence retrieval rather than multimodal document understanding. Tables were retained only as extracted linearized text, and no layout parser was used to reconstruct graphical structure, multi-column reading order, diagrams, or image-only table content. The 54 evaluation questions were designed around textual procedural, definitional, comparative, and scenario evidence; none intentionally required interpretation of information available only in a chart, diagram, or image. Consequently, the reported results should not be generalized to visually dependent technical document questions. No summarization, paraphrasing, or language model rewriting was applied to the indexed source passages. Table 3 summarizes the three text versions.

Table 3. Text versions used in the experiment.

Text Version	Processing	Experimental Purpose
Raw	Page text with minimal processing	Lower-bound version that retains extraction noise
Cleaned	Whitespace, broken line, and repeated format normalization	Test the effect of basic text cleaning
Metadata-enriched	Cleaned content plus document, topic, source, and page information	Test whether added source context improves retrieval

3.3. Text Segmentation and Indexing

All text versions were divided into segments of 1000 characters with an overlap of 200 characters. Segment boundaries were placed at sentence or word breaks where possible. Keeping the segmentation settings fixed allows differences in text preparation and search method to be compared fairly. The 1000-character size and 200-character overlap were fixed before evaluation as a practical balance between preserving procedural context and limiting unrelated material within each segment; they were not tuned on the reported test results.

The fixed files contained 1065 raw text segments, 1025 cleaned text segments, and 1065 metadata-enriched text segments. The small reduction after cleaning reflects removal of repeated formatting and consolidation of broken lines. Each segment kept the document ID and page number so that overlapping results from the same page could be combined before evaluation. Collapsing overlapping segment hits to unique source pages prevents the same page from occupying several evaluation ranks, but it does not completely remove the greater candidate exposure of longer or denser documents. The 1000/200 setting is therefore treated as a controlled baseline rather than as a universally optimal segmentation rule.

3.4. Retrieval Methods

A word-based sparse search method using BM25 was implemented with k1 = 1.5 and b = 0.75. BM25 gives greater weight to exact and near-exact term matches using how often a term appears, how rare it is across the documents, and document length normalization [2].

$$\mathit{BM}25(q,d)={\Sigma }_{\left\{t\in q\right\}} \mathit{IDF}\left(t\right) · f(t,d)(k1+1)/[f(t,d)+k1(1-b+{\displaystyle \frac{b\left|d\right|}{\mathit{avgdl}}})]$$

where f(t,d) is the frequency of query term t in text segment d, |d| is the segment length, and avgdl is the mean segment length.

Here, IDF(t) is the inverse document frequency of term t, k₁ controls term frequency saturation, and b controls document length normalization.

For the latent semantic analysis (LSA) baseline, term frequency–inverse document frequency unigrams and bigrams were projected to 256 dimensions using truncated singular value decomposition, and cosine similarity was used for ranking. LSA is used here as a corpus-derived statistical latent semantic baseline rather than as a proxy for modern neural dense retrieval. Because the decomposition is learned directly from this relatively small 417-page experimental corpus, the latent dimensions can reflect collection-specific term co-occurrence, sparsity, and extraction noise rather than semantic relationships that generalize beyond the corpus. Its purpose is therefore to provide a transparent and reproducible non neural comparison and to exercise the semantic search and fusion pipeline. It is not all MiniLM L6 v2 and is not presented as a modern sentence transformer result. Embedding-based methods such as dense passage retrieval and Sentence BERT provide the basis for the modern neural semantic search extension reported in Section 4.7 [3,4].

$$X \approx U_k{\Sigma }_k{V}_k^T,k=256$$

z_d = x_d V_k, z_q = x_q V_k

Here, X is the TF IDF matrix, $U_k {\Sigma }_k V_k^T$ is its rank-k truncated singular value decomposition, and x_d and x_q are document segment and query vectors projected through V_k.

The LSA model was fitted separately to each text version. Questions and text segments were projected into the same 256-dimensional space and normalized before cosine ranking. The fitted representation is therefore transductive to the fixed experimental corpus: it summarizes statistical co-occurrence within that corpus and is not claimed to provide a transferable semantic encoder. This method remains useful as a transparent non neural baseline, but it is not treated as a substitute for modern sentence embeddings; the separate all MiniLM L6 v2 comparison is reported in Section 4.7.

$$\mathit{cos}(q,d)={\displaystyle \frac{\left(q · d\right)}{\left({\left|\left|q\right|\right|}_2{\left|\left|d\right|\right|}_2\right)}}$$

For hybrid search, BM25 and LSA rankings were combined using reciprocal rank fusion with a constant of k = 60 [5]. The method combines ranking positions rather than raw scores, avoiding an invalid direct comparison between word-based and semantic score scales.

$$\mathit{RRF}(d)={\Sigma }_{\left\{r\in \left\{\mathit{BM}25, \mathit{LSA}\right\}\right\}} {\displaystyle \frac{1}{\left[k+\mathit{ran}{k}_{r\left(d\right)}\right]}},k=60$$

The number of retrieved candidates and the fusion constant were fixed before evaluation. Let C_p denote the set of indexed text segments originating from source page p, and let s_m(c,q) be the segment-level score assigned by retrieval method m to segment c for query q. Segment hits were collapsed to a page-level score using the maximum segment score ($S_{m\left(p,q\right)} = \mathit{ma}{x}_{\left\{c\in C_p\right\}} s_{m\left(c,q\right)}$). Pages were then ordered independently within each retrieval method to obtain the ordinal page rank (rank_m(p,q)). RRF was applied only to these page-level ordinal ranks: $\mathit{RRF}(p,q) = {\Sigma }_{\left\{m\in \left\{\mathit{BM}25,\mathit{LSA}\right\}\right\}}{\displaystyle \frac{1}{\left[60 + \mathit{ran}{k}_{m\left(p,q\right)}\right]}}$. Thus, BM25 scores and cosine similarities were never added, averaged, or otherwise compared on their absolute scales. The maximum score aggregation only determines the within-method ordering of unique pages before fusion; RRF itself remains an ordinal rank fusion operation. Collapsing segments before fusion also prevents the same source page from appearing several times simply because overlapping text segments were retrieved.

3.5. Evaluation Question Set and Silver Standard Page Labels

The evaluation set contains 54 questions: 36 questions that can be answered from the documents, 12 safety-related questions whose requested information is not present in the document collection, and 6 questions outside the covered safety topics. The benchmark was constructed before the final source page comparison, then frozen so that retrieval methods would be evaluated on the same questions. Candidate questions were selected to cover direct lookup, multi-step procedural interpretation, scenario reasoning, comparison, near-domain refusal, and out-of-domain refusal. Answerable questions were tied to mandatory answer elements and candidate supporting pages; unsupported questions deliberately used plausible safety vocabulary while requesting information absent from the corpus. Because the primary benchmark had already been used for method comparison, we did not add new primary questions after observing retrieval failures, which would introduce post hoc test set adaptation. Additional broad and multi-step questions are therefore treated as a prospective extension rather than silently changing the reported benchmark. The construction engineering implications are discussed separately; no construction-specific question is included in the present experiment.

The 36 answerable questions were balanced across the three safety topics. They comprised 18 direct questions, 11 procedural questions, 5 scenario questions, and 2 comparison questions. The procedural group includes multi-step energy control, maintenance, group lockout, PPE assessment, retraining, and guarding questions rather than only single-fact lookups. The unsupported set contained four safety-related questions per topic, while the six questions outside the covered topics tested clear rejection. The safety-related unsupported cases are particularly important because they contain plausible safety terminology and therefore challenge score-based refusal more strongly than unrelated questions. Appendix B, Table A2 summarizes the composition of the evaluation set, while the complete list of 54 questions is provided in Appendix B, Table A3.

For answerable questions, the relevant item is a unique source page identified by document ID and page number. Retrieved text segments were combined into unique source pages before scoring. Each question includes a draft reference answer, the minimum points that the answer should cover, supporting page identifiers, supporting text segment identifiers, and a short evidence excerpt.

Candidate pages were collected from the highest ranked results of the rebuilt retrieval pipeline and checked against the extracted source text. A page was marked relevant when it contained evidence for at least one required part of the answer. A relevant page therefore provides supporting evidence but may not be sufficient by itself for the complete answer. Several pages were retained when the full answer was spread across the documents. The review record stores the question, whether the documents can answer it, the draft reference answer, required answer points, relevant document IDs, relevant page IDs, supporting text segment IDs, and an evidence excerpt.

The present benchmark uses a model-assisted silver standard for source evidence relevance rather than an independently expert-adjudicated safety ground truth. This distinction reflects the task being evaluated: the applicable safety requirements are taken from the fixed source corpus, and the annotation task asks whether a specified source page directly supports predefined mandatory answer elements, not whether an annotator would formulate the safety requirement independently from professional experience. A page is treated as supporting when its text directly supports at least one mandatory answer element; pages that provide useful context but incomplete support are distinguished conceptually from pages sufficient for the claimed element, and multi-page answers are represented by mapping required elements to the corresponding source pages. The question record retains the draft reference answer, required answer points, source page identifiers, supporting segments, and evidence excerpts so that each judgment is traceable and reproducible. Independent dual review adjudication by qualified safety reviewers would provide an additional external validation layer, particularly for ambiguous, scenario-based, multi-page, or jurisdiction-sensitive cases. Accordingly, the current labels are reported transparently as a silver standard and are used for controlled comparative retrieval evaluation; they are not presented as expert-certified safety ground truth. This distinction limits claims about absolute accuracy and deployment validation without invalidating the paired within-benchmark comparison because every retrieval condition is evaluated against the same frozen questions, evidence units, and relevance decisions.

3.6. Retrieval Measures and Whether the Documents Support an Answer

Retrieval performance was measured using Hit@1, Hit@3, Hit@5, Hit@10, Recall@k, mean reciprocal rank (MRR), and normalized discounted cumulative gain at rank 10 (nDCG@10). Hit@k shows whether at least one labeled supporting source page appears within the first k unique pages, Recall@k measures the proportion of all labeled supporting pages retrieved, and MRR rewards early retrieval of the first supporting page. nDCG@10 was calculated with binary gains, assigning a gain of 1 to a page labeled as supporting the reference answer and 0 otherwise; binary gain is therefore the special two-level case of the general graded relevance formulation. This does not imply that partial and fully sufficient evidence are equivalent in practice, and graded independently adjudicated labels would provide a stronger external validation construct. The ability to distinguish answerable from unsupported questions was evaluated using AUROC based on the top retrieval score. Answer generation was not evaluated because the retrieval method and supporting pages should be selected and reviewed before complete answers are scored.

Hit@k answers the practical question of whether at least one supporting page appears early enough to be shown to a user or an answer generation model. Recall@k is stricter when several pages jointly support an answer. MRR emphasizes the first relevant page, while nDCG reflects the order of all labeled pages. Because some questions require several evidence pages, no single measure is sufficient on its own.

$$\mathit{MRR}={\displaystyle \frac{\left(\frac{1}{N}\right){\Sigma }_{\left\{i=1\right\}}^{N}1}{r_i}}$$

$$\mathit{Hit}@k=\left({\displaystyle \frac{1}{N}}\right){\Sigma }_{\left\{i=1\right\}}^{N}I\left(R_i\cap P_i,k \ne \varnothing \right)$$

$$\mathit{Recall}@k={\displaystyle \frac{\left(\frac{1}{N}\right){\Sigma }_{\left\{i=1\right\}}^N\left|R_i\cap P_i,k\right|}{\left|R_i\right|}}$$

$$\mathit{DCG}@k={\displaystyle \frac{{\Sigma }_{\left\{j=1\right\}}^k{g}_j}{\mathit{log}2\left(j+1\right)}},\mathit{nDCG}@k={\displaystyle \frac{\mathit{DCG}@k}{\mathit{IDCG}}}k,g_j \in \left\{0,1\right\}$$

Here, N is the number of evaluated questions, r_i is the rank of the first supporting page for question i, R_i is the set of labeled supporting pages for question i, P_i, k is the set of the top k retrieved pages, and g_j is the binary gain assigned to rank j.

The AUROC for distinguishing answerable and unsupported questions was calculated from each method’s top retrieval score. These scores are method-specific: BM25 values, cosine similarities, and RRF scores are not interchangeable probabilities. The AUROC analysis therefore measures how well each score ranks answerable questions above unsupported questions; it does not define one universal threshold for operational use.

3.7. Statistical Analysis

The same 36 answerable questions formed paired observations across all tested settings. Confidence intervals and paired differences were estimated with 20,000 question-level bootstrap resamples. Friedman tests assessed overall differences among the three search methods within each text version and among the three text versions within BM25. Planned pairwise comparisons used Wilcoxon signed rank tests on reciprocal rank with Holm correction within each comparison family. Exact McNemar tests assessed paired Hit@3 outcomes. The statistical analysis is exploratory because the relevance decisions form a model-assisted silver standard rather than an independently expert-adjudicated gold standard.

3.8. Reproducibility and Implementation

The experiments were implemented as repeatable Python scripts with a random seed of 42. The reproducibility package contains the source document list and SHA 256 checksums, raw/cleaned/metadata-enriched text segment files, evaluation questions, ranked results, detailed relevance scores, settings files, and statistical analysis outputs; Appendix C provides the corresponding reproducibility checklist. Core dependencies were NumPy, SciPy, scikit learn, pypdf, and PyYAML. The original archived experiment did not preserve a complete hardware manifest, so historical CPU, GPU, RAM, and operating system specifications cannot be reconstructed reliably and are not assigned retrospectively. This remains a reproducibility limitation of the original run.

To characterize computational scale without inventing historical hardware, a separate reference benchmark was performed on the same 18-PDF, 417-page corpus using a Linux x86 64 virtual machine with five allocated vCPUs on an AMD EPYC 9V74 host and 5.8 GB RAM; no GPU was used. The reference environment used Python 3.13.5, pypdf 5.9.0, NumPy 2.3.5, SciPy 1.17.0, scikit learn 1.8.0, and PyYAML 6.0.3. In this independently reconstructed BM25 benchmark, extraction of the 417 PDF pages required 9.70 s, chunk preparation required 0.023 s, and BM25 index construction required 0.074 s. Across the 54 evaluation questions after warmup, mean retrieval time was 0.735 ms per query (median if 0.671 ms; 95th percentile of 1.250 ms; maximum of 2.865 ms), and peak process memory was approximately 334 MB. The reconstruction produced 1052 raw chunks rather than the 1065 archived raw segments, so these timing results characterize computational requirements only and do not replace the primary effectiveness results. They exclude LLM answer generation and therefore should not be interpreted as end-to-end response time or as evidence of deployment readiness.

Every result row can be traced to an evaluation question, text version, search method, ranked page ID, score, supporting text segment ID, and relevance decision. The run logs record all settings and input file checksums. This structure allows future sensitivity analyses to substitute independently adjudicated labels without changing the underlying retrieval results. The same evaluation questions and page-level scoring principle were subsequently reused for the controlled MiniLM extension in Section 4.7.

4. Results

4.1. Overall Evidence-Level Retrieval

Overall performance is summarized in Table 4. BM25 produced the strongest pilot results among the tested methods tested. Raw BM25 achieved the highest MRR (0.492, 95% bootstrap CI of 0.368–0.619) and the highest Hit@5 (0.722). Cleaned BM25 achieved the highest Hit@3 (0.556, 95% bootstrap CI of 0.389–0.722) and the highest AUROC for distinguishing answerable questions (0.765). Metadata-enriched BM25 achieved the highest Hit@10 (0.833). These values are based on the model-assisted silver-standard relevance set and should be interpreted as exploratory comparative estimates. Among the 36 answerable questions, nDCG@10 ranged from 0.374 to 0.500 across configurations, with raw BM25 highest at 0.500; Recall@10 ranged from 0.579 to 0.713, with metadata enriched BM25 highest at 0.713.

Table 4. Source page retrieval performance under the silver-standard benchmark (Hit@k and MRR: 36 answerable questions; AUROC: all 54 questions).

Text Version	Method	Hit@1	Hit@3	Hit@5	Hit@10	MRR	AUROC
Raw	BM25	0.333	0.528	0.722	0.806	0.492	0.758
Raw	LSA	0.222	0.417	0.472	0.722	0.350	0.736
Raw	Hybrid RRF	0.278	0.500	0.611	0.778	0.437	0.562
Cleaned	BM25	0.333	0.556	0.639	0.778	0.473	0.765
Cleaned	LSA	0.278	0.444	0.472	0.750	0.398	0.707
Cleaned	Hybrid RRF	0.278	0.472	0.611	0.806	0.432	0.645
Metadata-enriched	BM25	0.333	0.528	0.667	0.833	0.478	0.745
Metadata-enriched	LSA	0.222	0.444	0.528	0.750	0.377	0.705
Metadata-enriched	Hybrid RRF	0.306	0.528	0.611	0.806	0.455	0.548

At rank 1, all three BM25 text versions achieved a Hit@1 of 0.333. Differences appeared at deeper cutoffs: cleaned BM25 produced the best Hit@3, raw BM25 produced the best Hit@5, and metadata-enriched BM25 produced the best Hit@10. This pattern suggests that text cleaning and added source information changed where relevant pages appeared in the ranking rather than producing one consistently better version. The LSA baseline was weaker at early ranks, although it recovered some evidence by rank 10.

The gap between document-level and source page performance was substantial. In the earlier development evaluation, most tested methods retrieved a labeled document within the top three results. Under source page scoring, the best Hit@3 was 0.556. Thus, a retrieval method may reach the correct publication while still missing the exact section that supports the answer.

4.2. Paired Statistical Comparisons

Table 5 summarizes the selected paired comparisons. Within the raw text version, reciprocal rank differed among methods (Friedman χ² = 7.13, p = 0.028). Hybrid search outperformed the LSA baseline after Holm correction (adjusted p = 0.011), whereas the BM25–LSA comparison did not reach significance after Holm correction (adjusted p = 0.058). BM25 and hybrid search did not differ (adjusted p = 0.302). The comparison of text versions within BM25 was not significant (Friedman χ² = 3.03, p = 0.220). No paired Hit@3 comparison was significant under exact McNemar tests.

Table 5. Selected paired reciprocal rank comparisons.

Comparison	ΔMRR	95% Bootstrap CI	Holm Adjusted p	Rank Biserial	Interpretation
Methods within raw: BM25 vs. LSA	0.141	0.036 to 0.251	0.058	0.483	Not significant after Holm correction
Methods within raw: BM25 vs. Hybrid RRF	0.055	0.021 to 0.136	0.302	0.322	No evidence of a paired difference
Methods within raw: Hybrid RRF vs. LSA	0.087	0.038 to 0.143	0.011	0.730	Exploratory difference after correction
Text versions within BM25: raw vs. cleaned	0.019	0.003 to 0.042	0.339	0.463	No evidence of a paired difference

The bootstrap confidence intervals show substantial question-level uncertainty, which is expected with 36 paired answerable items. The statistical findings should therefore be read together with effect sizes and error cases rather than as a simple winner–loser ranking. The strongest corrected comparison was hybrid RRF versus LSA for raw text; this indicates that adding the sparse ranking improved the weak semantic baseline, not that hybrid retrieval surpassed BM25.

4.3. Domain-Level Performance

As shown in Figure 2, raw BM25 performance varied by domain: LOTO (Hit@3 = 0.500, MRR = 0.459), machine guarding (Hit@3 = 0.250, MRR = 0.280), and PPE (Hit@3 = 0.833, MRR = 0.736). Machine guarding was the weakest domain, while PPE was strongest. Possible explanations include overlapping machine-guarding sources, variation in terminology, broad question wording, and stricter page-level labels. The difference also indicates that an overall retrieval average can conceal domain-specific failure modes.

Figure 2. Hit@3 by industrial safety topic and retrieval method for the raw text version.

4.4. Whether the Documents Support an Answer and When the System Should Refuse

The highest archived AUROC for distinguishing answerable from the combined unsupported set was 0.765 for cleaned BM25. This value compares 36 answerable questions with all 18 unsupported questions, including six clearly out-of-domain cases, and should therefore be interpreted as broad answer support discrimination rather than as a realistic estimate of near-domain refusal performance. The operationally more demanding comparison is the 36 answerable questions against only the 12 safety-related unsupported questions because these cases share terminology with the corpus. To quantify this issue without inventing a value from aggregate results, we independently reconstructed the raw text BM25 pipeline from the 18 retained PDFs and frozen 54-question benchmark. The reconstruction reproduced the archived broad raw BM25 discrimination closely (AUROC = 0.756 versus 0.758 in the archived experiment) and yielded AUROC = 0.669 when the negative class was restricted to the 12 safety-related unsupported questions; discrimination against the six clearly out-of-domain questions was substantially higher (AUROC = 0.931). These reconstructed values are reported as an auxiliary robustness analysis rather than substituted for the archived cleaned BM25 result because the historical cleaned text preprocessing cannot be reproduced bit for bit from the aggregate archive. The decrease from broad to near-domain discrimination confirms that plausible safety questions lacking corpus support are materially harder to reject than unrelated questions. Neither the archived 0.765 value nor the reconstructed 0.669 near-domain value is evidence of deployment-ready refusal reliability. This separation remains insufficient for an automatic refusal decision in a safety-critical system. RRF scores represent combined rank positions rather than calibrated probabilities and should not use the same thresholds as cosine similarity or BM25 scores. The diagnostic workflow records both the top retrieval score and the margin between the first and second ranked unique pages for each method because the same numerical threshold is not meaningful across BM25, cosine similarity, and RRF. In an auxiliary silver-standard diagnostic, top-score AUROC exceeded score-margin AUROC for BM25 (0.759 vs. 0.670), LSA (0.736 vs. 0.583), and RRF (0.608 vs. 0.541), indicating that score margin alone is not a sufficient refusal signal. Method-specific thresholds should therefore be calibrated on development data and evaluated on held-out questions. We intentionally do not promote a deployment threshold from the present silver-standard benchmark. Any operational calibration should be re-evaluated on a larger externally validated benchmark and should extend the completed MiniLM analysis to additional modern dense retrievers and rerankers. A practical refusal mechanism should combine method-specific top scores; score margins; agreement between sparse and semantic retrieval; required evidence coverage; support from more than one passage; and, where justified, a separately validated answerability classifier.

Safety-related unsupported questions were the more realistic source of false acceptance because they used the same vocabulary as the source documents. A high exact word score can therefore show topical similarity without proving that the requested fact is actually present. In practice, the system should make a separate decision about whether the documents support an answer and should show the evidence pages, the top score margin, and agreement between methods to a reviewer instead of turning the top score directly into a yes/no decision. The present paper therefore treats refusal calibration as a method-specific validation problem rather than selecting a single global threshold after inspecting the full test set.

4.5. Error and Failure Analysis

Manual inspection of raw BM25 failures identified recurring patterns rather than random misses. Table 6 summarizes representative examples. The first relevant rank is based on the current silver-standard source page labels; “>20” means that no labeled page appeared in the evaluated top 20 results.

Table 6. Representative evidence retrieval failures for raw BM25.

ID	Domain	Failure Pattern	Question	First Relevant Rank	Interpretation
Q005	LOTO	Scenario wording/negation	Can LOTO be skipped when a machine is already switched off?	>20	The query emphasizes the visible machine state, while the evidence concerns isolation of hazardous energy; top results drifted toward general guarding and maintenance text.
Q016	Machine guarding	Cross-domain procedure	What should be done when a guard must be removed for servicing or maintenance?	>20	The answer spans guarding and energy control concepts; rankings split across maintenance, LOTO, and guarding pages.
Q013	Machine guarding	Buried technical definition	What is the point of operation on a machine?	19	The exact definition appears in a specific OSHA section, while other pages repeat broader safeguarding terminology.
Q003	LOTO	Broad multi-step procedure	What steps should be completed before servicing or maintaining equipment?	14	Many pages discuss parts of the procedure, so simple word overlap did not prioritize the page labeled as the consolidated evidence source.
Q018	PPE	Over-broad question	When should personal protective equipment be used?	12	The source documents contain many pages on use, selection, maintenance, and hazard assessment; broad wording produces several plausible but incomplete matches.

The cases suggest three practical improvements. First, evaluation questions should preserve operational context and indicate when the evidence may span more than one safety topic. Second, retrieval evaluation should distinguish a partly relevant page from a page that is sufficient to support the full answer. Third, a second stage ranking method or a broader query step should be tested specifically on terminology differences and broad procedural questions rather than only on overall average scores.

4.6. Robustness and Diagnostic Checks

The PDF layout audit confirms that machine readable text does not imply uniformly simple document structure. Nearly half of the evaluated pages contained embedded image objects, low-text pages were concentrated in several training or slide-based sources, and the automated reading-order heuristic identified a substantial subset of pages with potential column or side-by-side layout risk. The primary experiment therefore remains a text retrieval study and does not claim complete recovery of visually encoded technical information.

As an auxiliary robustness check, BM25 retrieval was independently reconstructed on the raw extracted text and the same 36 answerable silver-standard questions using several segmentation settings: 500/100, 750/150, 1000/200, 1250/250, 1500/300, and 2000/400 characters/overlap, together with whole-page indexing. Across the chunked settings, Hit@3 ranged from 0.472 to 0.583, and MRR ranged from 0.490 to 0.525; whole-page indexing produced Hit@3 = 0.556, Hit@10 = 0.778, and MRR = 0.520. No single segmentation setting dominated every cutoff: for example, 750/150 achieved the highest MRR (0.525), whereas 500/100 achieved the highest Hit@10 (0.861). This confirms that segmentation affects ranking but does not support treating one fixed chunk size as universally optimal or fixed chunking as intrinsically invalid. Because this robustness run was independently reconstructed from the archived settings and produced 1052 chunks at 1000/200 rather than the 1065 archived raw segments, its absolute values are not substituted for the primary results in Table 4.

Additional diagnostic isolation was carried out on the independently reconstructed pipeline for individual whitespace normalization, line-break dehyphenation, repeated header/footer removal, individual metadata fields, whole-page versus two-page indexing, BM25 parameter settings, and LSA n gram/dimensionality settings. Across the preprocessing-only checks, Hit@3 ranged from 0.556 to 0.611 and MRR from 0.478 to 0.502; across the tested metadata field variants, Hit@3 ranged from 0.528 to 0.583 and MRR from 0.471 to 0.479. In the tested BM25 parameter grid, MRR ranged from 0.452 to 0.521, while the tested LSA configurations produced MRR values from 0.375 to 0.468. No individual preprocessing operation or metadata field uniformly dominated all retrieval cutoffs, and parameter changes altered ranking behavior, reinforcing the decision to treat the primary settings as controlled baselines rather than universal optima. Because these auxiliary reconstructions do not exactly reproduce the archived segment inventory, they are retained as diagnostic robustness analyses rather than replacements for the primary Table 4 experiment.

Exploratory raw BM25 stratification by question type on the reconstructed pipeline also showed heterogeneous difficulty. Hit@3/MRR were 0.611/0.498 for direct questions (n = 18), 0.545/0.525 for procedural questions (n = 11), 0.400/0.397 for scenario-based questions (n = 5), and 0.500/0.542 for comparative questions (n = 2). The comparative category is too small for stable inference, while the lower early rank performance of the scenario-based group is consistent with the failure analysis. These results support reporting question type alongside topic rather than relying only on an overall mean.

4.7. Modern Dense Retrieval Extension

To address the limitation of LSA as an interim semantic baseline, we conducted a controlled modern dense retrieval extension using sentence transformers/all MiniLM L6 v2. The 18 retained PDFs were re-extracted into 417 physical pages and segmented using the archived 1000-character window and 200-character overlap, producing 1066 reconstructed raw segments rather than the archived 1065. To avoid mixing non-equivalent runs, BM25 (k1 = 1.5, b = 0.75), MiniLM cosine retrieval, and BM25 + MiniLM reciprocal rank fusion (RRF, k = 60) were all evaluated on this same reconstruction, the frozen 54 question benchmark, and the same silver-standard source page labels. MiniLM embeddings were 384-dimensional, and segment scores were aggregated to pages by the maximum segment cosine similarity, matching the page-level aggregation principle used for the sparse comparator.

On the 36 answerable questions, reconstructed BM25 achieved a Hit@1/3/5/10 of 0.361/0.528/0.694/0.778, MRR of 0.495, and nDCG@10 of 0.487. MiniLM achieved 0.306/0.583/0.722/0.861, MRR of 0.490, and nDCG@10 of 0.505. BM25 + MiniLM RRF produced the highest observed Hit@3 (0.611), Hit@5 (0.806), Hit@10 (0.917), MRR (0.525), and nDCG@10 (0.567), while reconstructed BM25 retained the highest observed Hit@1 (0.361). The omnibus Friedman test on reciprocal rank did not reach the 0.05 significance level (χ² = 5.065, p = 0.079). No pairwise Wilcoxon comparison was significant after Holm correction (BM25 vs. MiniLM p = 0.940; BM25 vs. RRF p = 0.631; MiniLM vs. RRF p = 0.373), and no exact McNemar comparison at Hit@1, Hit@3, Hit@5, or Hit@10 was significant. The fusion gains are therefore reported as descriptive rather than statistically established improvements.

For answer-support discrimination, the MiniLM top-page cosine score achieved an AUROC of 0.853 for the broad 36 versus 18 comparison and 0.780 when the negative class was restricted to the 12 safety-related near-domain unsupported questions, compared with 0.715 and 0.627, respectively, for reconstructed BM25. MiniLM achieved an AUROC of 1.000 against the six clearly out-of-domain questions; this value is interpreted cautiously because that subset is small and deliberately distinct from the covered safety topics. RRF top-score AUROCs were 0.768, 0.709, and 0.884 for the broad, near-domain, and out-of-domain comparisons, respectively; because RRF scores are rank-derived, these values are treated as diagnostic rather than calibrated refusal probabilities. All results in this extension retain the model-assisted silver labels. Independent expert adjudication remains an external validation step for interpreting the benchmark as expert-certified ground truth rather than a prerequisite for reporting the controlled same-label method comparison.

5. Discussion

5.1. Evaluating Exact Pages Changes the Conclusion

The most important methodological finding is that document-level retrieval can substantially overestimate access to supporting evidence. In the earlier development pilot, most methods achieved Hit@3 values close to one when any relevant document counted as a success. When relevance was defined as the exact supporting source page, the best Hit@3 fell to 0.556. A generally relevant document may contain dozens of pages and still fail to provide the passage needed for the answer. Document-level success should therefore be interpreted as successful navigation to a relevant publication, not as proof that the system found the evidence needed to ground an answer.

5.2. Why BM25 Is a Necessary Baseline

BM25 performed strongly because the source documents contain exact regulatory terms, named procedures, lists of energy sources, guarding terms, and recurring PPE phrases. This result is consistent with the continuing value of word-based sparse search in terminology-rich regulatory collections [2,27,28]. The controlled modern dense extension further showed complementary behavior rather than simple replacement: reconstructed BM25 retained a higher observed Hit@1 than MiniLM, MiniLM improved the observed deeper cutoff retrieval and answer support discrimination, and BM25 + MiniLM RRF produced the highest observed MRR and nDCG@10. However, the reciprocal rank omnibus test was not significant, so these differences are descriptive rather than evidence that one method is statistically superior.

5.3. Hybrid Retrieval Is a Hypothesis, Not a Guaranteed Contribution

Reciprocal rank fusion improved the LSA baseline in several settings but did not consistently improve BM25. Combining rankings helps only when the methods contribute different relevant pages. A weak semantic component can reduce the value of a stronger BM25 ranking, so hybrid search should be compared with both component methods rather than assumed to be better. This interpretation is consistent with regulatory search studies that combine word-based and semantic signals but also use domain adaptation, filtering, or a second ranking stage [27,28].

5.4. Text Preparation and Added Source Information

No overall effect of text version was detected within BM25. Cleaning improved Hit@3 but reduced Hit@5 and Hit@10 compared with other text versions, while adding document, topic, source, and page metadata improved deeper retrieval but not early rank success. These mixed results show that adding such information does not automatically improve search. Repeated topic terms may increase the length of a text segment or compete with the actual evidence. The auxiliary field-level diagnostics likewise found no individual metadata field that uniformly dominated all retrieval cutoffs. Confirmatory work should reproduce those ablations on the archived segment inventory and examine whether field effects differ by question type rather than relying only on overall averages.

5.5. Implications for Safety Engineering Assistants

The findings support a staged safety system. First, the retrieval method should return traceable passages and indicate whether the source documents appear to support an answer. Second, the system should refuse to answer or send the case to a reviewer when evidence is missing or conflicting. Third, one fixed answer generation model should prepare an answer only from accepted evidence and cite its source. Finally, experts should separately evaluate correctness, completeness, faithfulness to the source, traceability, and practical usefulness. This staged approach is consistent with calls for human oversight and validated artificial intelligence systems in safety engineering [30,31,32].

The reference benchmark indicates that the retrieval stage, itself, is computationally modest for the present 18-document corpus: CPU-only BM25 retrieval averaged below 1 ms per query in the reference environment, while PDF extraction was an offline preprocessing cost. These measurements do not establish that a local end-to-end assistant is faster or cheaper than a cloud service because LLM inference, network latency, privacy requirements, integration overhead, and service pricing were outside the experiment. The study therefore limits its practical claim to the computational feasibility of the retrieval layer at this corpus scale.

5.6. Implications for Civil and Construction Engineering

Although the experiment used LOTO, machine guarding, and PPE documents rather than a construction-specific collection, the retrieval problem has a close civil engineering analogue. Construction teams must combine regulations, safety plans, accident reports, project records, and increasingly BIM-linked or knowledge graph information. The reviewed construction literature shows that hazards vary across residential, highrise, industrial, and heavy construction settings and that safety priorities depend on site conditions, temporary work, equipment, worker competence, and management systems [6,7,8,32]. A page-level retrieval benchmark can therefore test whether an assistant finds the specific evidence needed for a site decision rather than merely a document about the same hazard.

For civil engineering deployment, the present workflow should be extended with construction-specific source fields such as project type, work package, project phase, contractor, jurisdiction, revision status, and links to BIM objects or knowledge graph entities. Yang et al. [9] show that construction safety knowledge graphs already integrate regulations, accident reports, project data, and BIM but still rely on expert-supported extraction and face challenges in data integration and generalization. The page-level traceability used here can serve as an auditable bridge between unstructured documents and those structured systems.

The transfer should be tested rather than assumed. A civil engineering evaluation set should include falls from height, scaffolding and formwork, excavation, lifting, temporary structures, electrical work, plant movement, and contractor coordination, with questions separated by project type and phase. The current results provide a method for building that test, not evidence that the present rankings will hold in construction.

5.7. Practical Design Recommendations

• Retain BM25 as a mandatory benchmark and practical sparse retrieval fallback for exact regulatory terms, standard identifiers, named procedures, and equipment-specific vocabulary.
• Display document ID, page number, source authority, jurisdiction, and publication date with every retrieved evidence passage.
• For construction documents, retain project type, project phase, work package, contractor, and any BIM or asset identifier with each retrieved passage.
• If refusal thresholds are deployed, calibrate them separately for each search method using development data and evaluate them on held-out questions; do not compare raw BM25, cosine similarity, and RRF scores using one threshold.
• Log retrieval and generation independently so that a correct refusal is not counted as a high-quality generated answer and a fluent answer cannot hide missing evidence.
• Send low-confidence, conflicting, or cross-jurisdiction cases to a safety professional and record the reviewer’s decision so the evaluation set can be improved later.

6. Limitations

• The source page relevance decisions are model-assisted and should be interpreted as a transparent silver standard rather than an independently expert-adjudicated gold standard. This limits claims about absolute retrieval accuracy or operational safety validation, but all compared retrieval conditions use the same frozen relevance set, preserving the internal consistency of the comparative experiment.
• The archived semantic comparison method is LSA rather than a modern neural sentence transformer. A separate controlled MiniLM extension was subsequently added on a reconstructed raw corpus; its labels remain the same model-assisted silver standard, and its observed method differences did not reach statistical significance.
• The evaluation set contains 54 questions and three industrial safety topics, with only 36 answerable cases. Bootstrap intervals are therefore necessarily broad, the comparative category contains only two questions, and topic-level performance varies substantially. The benchmark was frozen before the final source-page comparison, so we did not enlarge the primary test set after observing method failures. A larger prospectively authored challenge set, including additional broad and multi-step procedural questions, is needed for confirmatory evaluation. The current set also includes no construction-specific documents, hazards, or project phases. The civil engineering implications are therefore design propositions based on the reviewed literature, not evidence that the reported rankings generalize to construction or other industries, jurisdictions, languages, or document types.
• The source document collection mixes regulatory guidance, training material, sector-specific documents, and sources from different years and jurisdictions. It also varies substantially in document length and extracted text density, including a low-text slide deck export and sources with known spacing artifacts. BM25 applies segment length normalization through its b parameter, and LSA vectors are normalized before cosine ranking, but these algorithmic normalizations do not remove document-type or layout heterogeneity. The document list, source roles, version control rules, and extraction quality flags should therefore be reported consistently when the benchmark is reused.
• Answer generation was deliberately not evaluated; the study makes no claim about end-to-end answer correctness or deployment readiness.
• The primary AUROC analysis of whether questions can be answered uses the top retrieval score. Auxiliary diagnostics also consider the first to second result score margin, but neither feature constitutes a validated operational refusal mechanism.
• The statistical analysis is exploratory because the evaluation set is small and the relevance decisions form a model-assisted silver standard rather than an independently expert-adjudicated gold standard.

6.1. Threats to Validity

Construct validity. Source-page relevance is a stricter construct than document relevance, but some safety answers require evidence distributed across several pages. The present labels are designed to verify whether a cited source page supports predefined mandatory answer elements; they do not constitute independent expert formulation of the underlying safety requirement. Binary gains were used in nDCG for the primary experiment, which may underrepresent partial evidence, evidence sufficiency, and procedural continuity. An externally validated extension could distinguish no support, partial support, and sufficient support and test the sensitivity of the reported method comparisons to independently adjudicated judgments.

Internal validity. The three text versions were produced using repeatable rules, but cleaning and metadata enrichment can change term distributions, segment boundaries, and candidate exposure. The primary design kept segment size and overlap fixed so that retrieval methods were compared under the same segmentation rule, and retrieved segments were collapsed to unique source pages before scoring. This collapse removes repeated exposure of overlapping segments from the same page in the final ranked list, but it does not eliminate the greater number of candidate pages contributed by long documents. Corpus diagnostics confirmed this exposure imbalance: document page count correlated with chunk count (r = 0.810), and extracted character count correlated strongly with chunk count (r = 0.987). An auxiliary chunk sensitivity audit therefore varied chunk size/overlap and also indexed whole pages. Across the tested settings, MRR ranged from 0.490 to 0.525 for chunked conditions, while whole-page indexing achieved MRR = 0.520; no single segmentation setting dominated all cutoffs. These diagnostics indicate that the main qualitative retrieval pattern is not dependent on one chunk size, but they do not constitute a fully document-balanced re-estimation of Table 4. Auxiliary reconstructions also examined individual preprocessing operations, metadata fields, and selected BM25/LSA parameter settings. Because these reconstructions did not exactly reproduce the archived segment inventory, they are treated as robustness diagnostics rather than replacements for the primary experiment. A future benchmark should additionally equalize or reweight document contributions to isolate residual document length exposure directly.

External validity. The source document collection is limited to English-language material in three safety topics and includes different authorities, years, document types, page layouts, and text densities. All retained PDFs provided extractable machine-readable text, but the pipeline did not evaluate image-only scans, OCR accuracy, layout reconstruction, diagrams, charts, or visually dependent tables. Results may therefore differ for current regulations, organization-specific procedures, multilingual documents, scanned records, multi-column technical manuals, accident narratives, and questions whose answer depends on visual page structure.

Conclusion validity. The evaluation set contains 36 paired answerable questions, and the relevance decisions form a silver-standard benchmark rather than an independently expert-adjudicated gold standard. The comparative results are therefore interpreted conditionally on this frozen benchmark; confidence intervals, effect sizes, and failure analysis are more informative than isolated p values or claims of absolute retrieval accuracy.

6.2. Future Work

Future work should strengthen external validity in three directions. First, an independently adjudicated expert benchmark could be used to test the sensitivity of the present silver-standard comparisons, especially for ambiguous, scenario-based, multi-page, and jurisdiction-sensitive questions. Second, the completed all MiniLM L6 v2 extension should be broadened to additional modern retrievers such as E5 or BGE and, where computationally feasible, a cross encoder reranker. These extensions should preserve the frozen questions and evidence units while using independently adjudicated labels and a larger benchmark to determine whether the descriptive BM25 + dense fusion gains observed here are reproducible. Third, a separate civil engineering extension should build and validate a construction safety document collection and a prospectively authored question set covering project phases, temporary works, work at height, lifting, excavation, plant movement, contractor coordination, and additional broad multi-step procedures. That extension should retain project and BIM-linked source details and test whether the retrieval patterns observed here transfer to construction practice.

7. Conclusions

This study turns a broad local RAG implementation into a focused evaluation of whether retrieval methods can find the exact industrial safety source pages needed for an answer. Under the fixed silver-standard benchmark used consistently across all retrieval conditions, BM25 provides a strong baseline, hybrid search is not automatically better, text preparation does not produce a uniform improvement, and document-level success can conceal source-page failure. These findings are comparative and exploratory rather than claims of expert-certified absolute retrieval accuracy or deployment readiness. The workflow also provides a practical foundation for civil engineering safety systems, where regulations, accident records, project information, and BIM-linked knowledge must remain traceable to the evidence used in a site decision. Future external validation can examine the sensitivity of the reported comparisons to independently adjudicated relevance labels, additional neural retrievers and rerankers, and construction-specific document collections.