Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

1.This work focuses on RAG preprocessing to alleviate LLM hallucinations, yet does not discuss the latest state-of-the-art hallucination mitigation methodologies. Could the authors elaborate on how your framework aligns with these latest advances, and clarify its incremental value relative to them?

2.All experiments are conducted with a fixed LLM and embedding model. Could you provide theoretical proof of the multi-model compatibility of your proposed pipeline, and verify whether its performance advantages generalize to other mainstream LLMs and embedding models?

3.Your evaluation is built on a limited corpus (36 documents) and a 50-question benchmark. Could you clarify whether your proposed framework is adaptable to larger-scale document corpora and more comprehensive evaluation question sets, along with relevant scalability analysis?

Author Response

Manuscript ID: applsci-3908378

Title: From PDF to RAG-Ready: Evaluating Document Conversion Frameworks for Domain-Specific Question Answering

We thank all reviewers for their constructive and thorough feedback. The manuscript has been substantially revised in response to the comments received. Below, we address each comment individually, describing the changes made and their location in the revised manuscript. All changes are visible through tracked changes in the revised manuscript file.

Reviewer 1

Comment 1: Hallucination mitigation — alignment with state-of-the-art

This work focuses on RAG preprocessing to alleviate LLM hallucinations, yet does not discuss the latest state-of-the-art hallucination mitigation methodologies. Could the authors elaborate on how your framework aligns with these latest advances, and clarify its incremental value relative to them?

We thank the reviewer for this important observation. Our work addresses hallucination mitigation upstream, at the data preparation stage, by ensuring that the content fed to the retriever and the LLM is structurally accurate and complete. This is complementary to downstream approaches that operate at the generation or verification stage.

Action taken: We have added a new paragraph to Section 4 (Discussion) that explicitly positions our work relative to three recent hallucination mitigation strategies: self-consistency decoding (Wang et al., 2023), Chain-of-Verification prompting (Dhuliawala et al., 2024), and Retrieval-Augmented Fine-Tuning (Zhang et al., 2024). The paragraph argues that upstream data quality is a prerequisite for effective hallucination mitigation regardless of which downstream strategy is employed, and that these approaches are complementary rather than competing. The three references have been added to the bibliography.

Comment 2: Single LLM/embedding model — generalisability

All experiments are conducted with a fixed LLM and embedding model. Could you provide theoretical proof of the multi-model compatibility of your proposed pipeline, and verify whether its performance advantages generalize to other mainstream LLMs and embedding models?

We acknowledge that all experiments used a single LLM (gpt-4o-mini) and a single embedding model (text-embedding-3-small). This is an inherent limitation of the study and was already noted in Section 4.6 (Limitations).

Action taken: We have expanded the third limitation paragraph in Section 4.6 to note that the pipeline is architecturally model-agnostic: it produces Markdown files and metadata that can be consumed by any LLM and embedding model without modification. Because the accuracy differences we report stem from the quality of the input data rather than from model-specific interactions, we expect the relative ranking of preprocessing pipelines to hold across models, though the magnitude of the differences may vary. A multi-model evaluation is identified as future work in Section 5.

Comment 3: Scalability to larger corpora

Your evaluation is built on a limited corpus (36 documents) and a 50-question benchmark. Could you clarify whether your proposed framework is adaptable to larger-scale document corpora and more comprehensive evaluation question sets, along with relevant scalability analysis?

The corpus, while limited in document count, spans 1,706 pages and nearly 492,000 words, which is a non-trivial volume. The 50-question benchmark was carefully designed to target specific failure modes across all document types.

Action taken: We have expanded the first and second limitations in Section 4.6 to address this concern. Additionally, we note that the system architecture, based on the Medallion Architecture with containerized Docker deployment, is designed to support larger corpora through incremental ingestion and modular processing. A sentence about computational bottlenecks at scale (VLM-based image descriptions, LLM-based hierarchy rebuilding) has been added. The evaluation protocol was also strengthened: each configuration is now evaluated over 50 independent runs (up from 10), with statistical significance assessed through Wilcoxon signed-rank tests and effect sizes reported via Cohen’s d.

Reviewer 2 Report

Comments and Suggestions for Authors

This study aims to systematically evaluate how PDF document preprocessing affects the performance of Retrieval-Augmented Generation (RAG)-based question answering. The topic is timely and interesting. However, several major concerns should be addressed as follows:

1. The Introduction should present quantitative evidence to emphasize the practical importance of RAG systems. For example, numerical statistics on how preprocessing errors affect QA accuracy or performance variations in real-world applications should be included. This would clarify the necessity and urgency of the research.
2. Recent studies across various domains have actively explored RAG pipelines. These latest works should be incorporated to strengthen the literature review, such as (1) Party politics in transport policy with a large language model and (2) Retrieval Augmented Generation-Based Large Language Models for Bridging Transportation Cybersecurity Legal Knowledge Gaps.
3. The current analysis focuses primarily on PDF conversion methods, but input data characteristics should also be considered as variables. These include document type (text-heavy vs. image-heavy), table density, and multi-column layouts. Such factors can directly influence both conversion quality and QA performance, and their relationships should be further analyzed.
4. The chunking strategy requires more detailed design and analysis beyond simple comparison. Key parameters such as chunk size, overlap, and hierarchy depth should be systematically examined. Additional experiments are needed to quantify how hierarchy-aware chunking contributes to performance improvements.
5. While the results indicate that metadata enrichment improves performance, a component-level analysis is needed. The contribution of individual metadata elements, such as section information, document type, and image descriptions, should be separately evaluated. Their relative importance should be quantitatively reported.
6. The evaluation dataset consists of 50 manually constructed questions, but performance differences across question types should be analyzed. For example, results should be compared across table-based, hierarchy-based, and image-based questions. This would clarify how preprocessing methods affect different QA scenarios.
7. The figures are generally informative, but clearer visual differentiation of input and output data across each stage (Extract–Transform–Load) is required. Data flow directions and data formats (e.g., PDF, Markdown) should be explicitly illustrated. In addition, Figure 1 and Figure 2 serve different purposes, so their captions should more clearly distinguish their roles.
8. Table 1 and Table 2 require consistent formatting in terms of units and alignment. Units such as pages, words, and size should be standardized, and the representation of mean and standard deviation should follow a consistent format. Decimal precision should also be unified to improve readability.
9. Table 3 presents a comparison of frameworks, but it is largely qualitative, making direct comparison difficult. The table should be revised to include quantitative metrics such as processing time, accuracy, and structure preservation rate. Evaluation criteria should also be clearly defined for fair comparison.
10. Similar expressions are repeated throughout the manuscript, reducing readability. Phrases such as “this study aims to…” and “it is important that…” appear frequently and should be minimized. Sentences should be simplified and focused on key arguments.
11. Key RAG pipeline terms, such as chunking, hierarchy reconstruction, and metadata enrichment, should be clearly defined at their first occurrence. Consistent terminology should be maintained throughout the manuscript, as some concepts are currently described using varying expressions.
12. Abbreviations such as LLM, RAG, and OCR should be defined in full at first mention. Their usage should remain consistent thereafter. In addition, the naming conventions of frameworks (e.g., Docling, MinerU) should be standardized across the manuscript.
13. Minor grammatical issues are observed, including missing articles and incorrect plural forms. For example, expressions such as “a knowledge graph construction” should be revised for correctness. Long sentences with unclear structure should be divided to improve clarity.

 

Author Response

Manuscript ID: applsci-3908378

Title: From PDF to RAG-Ready: Evaluating Document Conversion Frameworks for Domain-Specific Question Answering

We thank all reviewers for their constructive and thorough feedback. The manuscript has been substantially revised in response to the comments received. Below, we address each comment individually, describing the changes made and their location in the revised manuscript. All changes are visible through tracked changes in the revised manuscript file.

Reviewer 2

Comment 1: Quantitative evidence in Introduction

Add quantitative evidence on how preprocessing errors affect QA accuracy.

Action taken: We have added a paragraph to the Introduction citing two recent studies that demonstrate the impact of document preparation quality on RAG performance: Akbar et al. (2025), who showed that a RAG system built on a curated legal corpus outperformed leading commercial LLMs, and Yun and Lee (2025), who demonstrated that careful filtering and structuring of legislative texts was essential for reliable LLM-based classification. These references strengthen the empirical motivation for our preprocessing-focused approach.

Comment 2: Add recent RAG studies

Add recent RAG studies to the literature review, including transport domain examples.

Action taken: The two references mentioned above (Akbar et al., 2025; Yun & Lee, 2025) have been added to the Introduction. Both address domain-specific RAG applications where document preprocessing quality is critical, directly supporting the paper’s central argument.

Comment 3: Input data characteristics as variables

Analyze input data characteristics (text-heavy vs image-heavy, table density) and their effect on results.

Action taken: We have added a paragraph to Section 4.1 (Discussion) linking the corpus characteristics documented in Section 2.2 (Tables 1–2) to the observed performance patterns. The analysis draws on the new per-question-type results (Section 3.2.6, Table 6) to show that table-dependent questions are the primary driver of accuracy differences, consistent with the corpus-level observation that 94.4% of documents contain tabular structures. A systematic per-category evaluation is identified as future work.

Comment 4: Chunking strategy parameters

Provide a systematic examination of chunk size, overlap, and hierarchy depth.

Chunk size (1,000 characters) and overlap (200 characters) were fixed by design to isolate the effect of the conversion framework and splitting strategy. A factorial analysis of these parameters would multiply the number of configurations beyond feasibility for this study.

Action taken: We have added a fifth limitation to Section 4.6 explicitly acknowledging this constraint. The description of the Hierarchical Recursive splitting strategy in Section 2.1.2 has been clarified to note that, unlike the other two strategies, chunks are bounded by section markers and may therefore be shorter than the configured maximum. The chunk size specification in Section 2.5 has been updated accordingly.

Comment 5: Component-level metadata analysis

Separate the contribution of section info, document type, and image descriptions.

The benchmark design already partially isolates two metadata components: structural metadata (configs 16 vs. 19: +6.5 pp, p < 10⁻³⁴) and VLM-based image descriptions (configs 19 vs. 20: +2.9 pp, p < 10⁻¹⁷). Document type metadata is implicitly encoded in the hierarchical structure and cannot be isolated without additional configurations.

Action taken: The text in Sections 3.2.3 and 4.2 now explicitly highlights these comparisons as component-level evidence. A full factorial decomposition of individual metadata elements is suggested as future work.

Comment 6: Performance across question types

Analyse performance across different question types (table-based, hierarchy-based, image-based).

Action taken: We have added a new Section 3.2.6 (“Performance by Question Type”) with an accompanying Table 6. The 50 benchmark questions were categorized into four types: table-dependent (9 questions), hierarchy-dependent (6), system/SAP (12), and straightforward text (23). The analysis reveals that table-dependent questions are the primary driver of the overall accuracy gap: frameworks using basic splitting score below 70.1% on these questions, while Docling with Hierarchical splitting scores 99.8% — a 33 pp gain on a single question type. A corresponding interpretive paragraph has been added to Section 4.2.

Comment 7: Figures

Improve figure clarity; distinguish Figure 1 and Figure 2 more clearly.

Action taken: Figure 2 (the chatbot screenshot) has been removed from the manuscript, as it served an illustrative rather than analytical purpose. The remaining figures have been renumbered. Figure 1 (ETL pipeline) has been retained with an improved caption.

Comment 8: Tables 1 and 2 formatting

Ensure consistent formatting, units, and decimal precision in Tables 1 and 2.

Action taken: Tables 1 and 2 have been reviewed for formatting consistency. Units, decimal precision, and alignment have been standardized.

Comment 9: Table 3 — add quantitative metrics

Add quantitative metrics to Table 3 instead of only qualitative Pros/Cons.

Action taken: A column showing the best benchmark accuracy achieved by each framework has been added to Table 3. Processing time was not included because external variables (LLM API access tier, server load) confounded the measurements, making fair comparison impossible without a controlled benchmarking setup.

Comment 10: Reduce repetitive phrases

Reduce repetitive phrases such as ‘this study aims to’, ‘it is important that’.

Action taken: A thorough editorial review has been conducted to reduce repetitive phrasing and improve conciseness throughout the manuscript. Phrases such as “this study aims to,” “it is important to note,” and “it is worth mentioning” have been eliminated or replaced with more direct formulations.

Comment 11: Define key terms at first use

Define key terms (chunking, hierarchy reconstruction, metadata enrichment) at first use.

Action taken: Key terms have been verified and defined at first occurrence. “Retrieval depth” is now defined as “the number of chunks retrieved and provided to the LLM as context.” The terminology “hierarchy reconstruction” has been unified to “hierarchy rebuilding” throughout the manuscript for consistency.

Comment 12: Abbreviation consistency

Define abbreviations at first mention and ensure consistent framework naming.

Action taken: All abbreviations are now defined in full at first use in the body text: OCR (Optical Character Recognition), API (Application Programming Interface), REST (Representational State Transfer), and SEM (Standard Error of the Mean). LLM, RAG, and PDF are expanded in the Introduction. An abbreviation table is provided at the end of the manuscript per MDPI style.

Comment 13: Minor grammar issues

Address missing articles, incorrect plurals, and long unclear sentences.

Action taken: The manuscript has been carefully proofread to correct grammatical issues, including missing articles, incorrect plurals, and overly complex sentence structures.

Reviewer 3 Report

Comments and Suggestions for Authors

The manuscript evaluates the quality of PDF processing frameworks by configuring 19 pipelines, including Docling, MinerU, Hand-made Markdowns, and DeepSeek OCR variants. The research is timely, and the experimental design well supports the central argument that data preparation quality is the main factor in the performance of the Retrieval-Augmented Generation system. Generally, the paper is interesting, and the finding that data preparation quality dominates RAG performance is an important practical contribution.

However, I have some recommendations on how to improve the manuscript quality, which are listed below:

1. Is it possible to add clearly formulated research questions to the Introduction part?
2. You already mentioned that the evaluation dataset of 50 questions is relatively small and might have limited statistical power. However, if you can include the confidence interval for 10 evaluation runs in the test results of Table 4, it can provide additional insight into the frameworks.
3. It looks like non-English language challenges are underemphasised in the Conclusion. The discussion of Portuguese-specific character misrecognition is insightful, and, I believe, needs to be added to the conclusion as well.
4. Please correct a small typo, repeated word in conclusion (p.17, line 655) — "basic basic RAG pipeline".

Author Response

Manuscript ID: applsci-3908378

Title: From PDF to RAG-Ready: Evaluating Document Conversion Frameworks for Domain-Specific Question Answering

We thank all reviewers for their constructive and thorough feedback. The manuscript has been substantially revised in response to the comments received. Below, we address each comment individually, describing the changes made and their location in the revised manuscript. All changes are visible through tracked changes in the revised manuscript file.

Reviewer 3

We thank Reviewer 3 for the positive assessment and the constructive suggestions, all of which have been addressed.

Comment 1: Research questions in Introduction

Is it possible to add clearly formulated research questions to the Introduction part?

Action taken: Three explicit research questions have been added to the Introduction, immediately before the paragraph summarising the main findings. RQ1 asks to what extent conversion frameworks differ in downstream RAG accuracy; RQ2 asks which pipeline dimensions contribute most to accuracy; and RQ3 asks whether an automated pipeline can match or exceed manual curation.

Comment 2: Confidence intervals for evaluation runs

Include the confidence interval for evaluation runs in the test results of Table 4.

Action taken: Table 4 has been substantially expanded. Each configuration was re-evaluated over 50 independent runs (up from 10). Accuracy is now reported as Mean ± SD [95% CI]. Statistical significance between configurations is assessed via Wilcoxon signed-rank tests (Table 5), with effect sizes reported as Cohen’s d. Normality of paired differences was assessed using the Shapiro-Wilk test (Appendix B); since 18 of 24 distributions depart from normality, the Wilcoxon test was adopted as the primary non-parametric test.

Comment 3: Non-English language challenges in Conclusion

The discussion of Portuguese-specific character misrecognition is insightful and needs to be added to the conclusion.

Action taken: A sentence has been added to the Conclusions (Section 5) noting that framework selection should account for language-specific robustness, as DeepSeek OCR exhibited systematic misrecognition of Portuguese diacritical characters while Docling handled Portuguese text without noticeable errors.

Comment 4: Typo — “basic basic”

Please correct a small typo, repeated word in conclusion (p.17, line 655) — “basic basic RAG pipeline.”

Action taken: The typo has been corrected to “basic RAG pipeline.”

Reviewer 4 Report

Comments and Suggestions for Authors

The manuscript addresses a practically relevant problem, but in its current form it does not yet reach the level of scientific rigor expected for a research article. The study reads more like an engineering benchmark report or system implementation note than a mature scholarly contribution.

The main concern is the limited strength of the validation design. The conclusions are drawn from a corpus of 36 documents and a manually constructed benchmark of 50 questions, using a single LLM, a single embedding model, and an LLM-as-judge protocol. In addition, no statistical significance analysis is reported, and the “upper baseline” is based on hand-made Markdown files produced by the research team. These choices substantially limit the generalizability and evidential strength of the claims.

The manuscript should clarify whether the corpus contains any sensitive personnel information, the exact access status of the documents, and the basis on which ethical review was considered not applicable.

The manuscript also tends to overstate its conclusions relative to the experimental evidence. The paper makes broad claims about the dominant importance of data preparation quality, metadata, and hierarchy-aware chunking, yet the experiments are confined to one corpus, one language setting, and one main evaluation setup. As a result, the conclusions should be presented much more cautiously.

Presentation is another weakness. The writing is often verbose, repetitive, and at times informal or speculative for a scientific article. Several sections could be tightened substantially. In addition, some figures appear closer to system demonstration material than to analytical scientific evidence. For example, the chatbot interface screenshot adds limited research value and reinforces the impression of a product demo rather than a rigorous paper.

To become publishable, the manuscript would need a much stronger experimental design, broader validation, more rigorous statistical analysis, a more restrained interpretation of the findings, and substantial rewriting for clarity, concision, and scientific tone.

Author Response

Manuscript ID: applsci-3908378

Title: From PDF to RAG-Ready: Evaluating Document Conversion Frameworks for Domain-Specific Question Answering

We thank all reviewers for their constructive and thorough feedback. The manuscript has been substantially revised in response to the comments received. Below, we address each comment individually, describing the changes made and their location in the revised manuscript. All changes are visible through tracked changes in the revised manuscript file.

Reviewer 4

Comment 1: Scientific rigour — engineering benchmark vs. scholarly contribution

The manuscript reads more like an engineering benchmark report than a mature scholarly contribution. The study lacks statistical significance testing and confidence intervals.

We appreciate this candid assessment, which prompted the most substantial revisions to the manuscript.

Action taken: The evaluation protocol has been redesigned. Each of the 21 configurations (expanded from 19) was evaluated over 50 independent runs. Table 4 now reports accuracy as Mean ± SD [95% CI]. Statistical significance between configurations is assessed via Wilcoxon signed-rank tests (Table 5), with Cohen’s d effect sizes. The Shapiro-Wilk test (Appendix B) justified the choice of the non-parametric Wilcoxon test over paired t-tests. Of 24 pairwise comparisons, 23 are statistically significant (p < 0.05), with the single non-significant comparison (Docling K = 50 vs. K = 5, p = 0.090) being consistent with the theoretical argument in Section 3.2.4. A new per-question-type analysis (Section 3.2.6, Table 6) provides further analytical depth. Three explicit research questions have been added to the Introduction. The Results sections (3.2.x) have been rewritten to be purely descriptive, with all interpretive content moved to the Discussion (Chapter 4).

Comment 2: Limited validation design

36 documents, 50 questions, single LLM, single embedding, LLM-as-judge, hand-made baseline by the research team.

We acknowledge these limitations and have addressed them through several improvements.

Action taken: The evaluation now uses 50 runs per configuration (not 10), enabling statistical testing with narrow confidence intervals. Section 4.6 (Limitations) has been expanded to six paragraphs, explicitly acknowledging each limitation (corpus size, question count, single LLM/embedding, LLM-as-judge bias, fixed chunking parameters, and preliminary GraphRAG). The hand-made baseline is presented with appropriate caveats: it represents the best achievable result for the given RAG configuration, not a theoretical ceiling. The full LLM-as-judge evaluation prompt is now provided in Appendix A for reproducibility.

Comment 3: Ethical review / sensitive personnel information

Given the corpus involves personnel-related documents from the Portuguese Army, the ethical review waiver needs clearer justification.

Action taken: The Institutional Review Board Statement and Informed Consent Statement are present in the manuscript (“Not applicable”). The corpus consists exclusively of publicly accessible administrative regulations, procedural manuals, and salary tables. No individual personnel records, personally identifiable information, or classified material is included. The documents describe institutional procedures and regulatory frameworks, not individual cases. This clarification can be added to Section 2.2 (Corpus) if the editor considers it necessary.

Comment 4: Overstated conclusions

The conclusions overstate the generalizability of findings.

Action taken: All conclusion statements have been qualified with hedging language: “in our experimental setting,” “for this corpus,” “we cannot generalise this finding to other document types.” The span claim has been updated from 23 to 15 percentage points to reflect the Round 2 results. The claim about font-based hierarchy superiority has been replaced with a nuanced statement reflecting the inconclusive Round 2 results.

Comment 5: Verbose and repetitive writing

The writing style is verbose, repetitive, and at times informal or speculative.

Action taken: A comprehensive editorial pass has been conducted. Repetitive phrases (“this study aims to,” “it is important to note,” “we should notice”) have been eliminated. The Introduction has been tightened. The Results sections (3.2.x) have been rewritten to be purely descriptive, removing embedded interpretations. Speculative language has been replaced with evidence-based statements supported by the statistical tests.

Comment 6: Chatbot screenshot (Figure 2)

The chatbot screenshot is more appropriate for a product demo than a research paper.

Action taken: Figure 2 (the chatbot screenshot) has been removed. The referencing text has been revised to describe the chatbot functionality without the screenshot. The remaining figures have been renumbered.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors The authors have carried out detailed revisions to address my comments as evident from their response letter and the revised manuscript. I now recommend the manuscript for publication.  

Reviewer 2 Report

Comments and Suggestions for Authors

I am happy with the responses.