Optimizing Document Interaction Using Large Language Models by Integrating Retrieval-Augmented Generation, Facebook AI Similarity Search, and Human-like Performance Metrics ^†

Abstract

We developed an intelligent conversational system that enhances document interaction using advanced language models and embedding techniques. The system integrates retrieval-augmented generation, Facebook AI similarity search-based retrieval, and cosine similarity for efficient information extraction from Portable Document Format documents. It employs three embedding models, namely All-MiniLM L6 v2, All-MPNet Base v2, and Instructor Large, with three large language models including LLaMA 3.3 70B, Gemma 2-9B IT, and Mixtral 8x7B-32768. System performance is evaluated using ROUGE-1, BERTScore, and a novel human-like performance (HLP) metric, showing improved retrieval accuracy, response coherence, and efficiency for academic and enterprise applications.

1. Introduction

Large language models (LLMs) have emerged as a transformative force in natural language processing (NLP), demonstrating strong capabilities in tasks such as summarization, sentiment analysis, text generation, and question answering (QA) [1]. Models such as generative pre-trained transformer and large language model meta AI (LlaMA) have proven effective in generating coherent, contextually aware responses that closely resemble human communication [2]. As LLMs become increasingly integrated into real-world systems, the demand for applying them to unstructured document understanding, particularly for research article analysis, has grown significantly.

LLMs hold significant potential in document-level QA systems, where users require concise answers to complex queries derived from lengthy, technical texts [3]. However, existing implementations often suffer from limitations such as fixed context windows, token size constraints, and the absence of effective retrieval mechanisms that render them insufficient for handling large or multi-topic documents [4]. To address these challenges, researchers have proposed retrieval-augmented generation (RAG) frameworks, which combine the generative power of LLMs with vector-based retrieval techniques to inject relevant context into the answer generation pipeline [4].

Despite growing adoption of RAG, systematic comparative evaluations of LLMs in document QA tasks are lacking. Most existing studies rely on limited evaluation metrics such as recall-oriented understudy for gisting evaluation (ROUGE), which primarily measures surface-level lexical overlap and fails to capture the semantic relevance or human-likeness of responses [5,6]. Furthermore, there is minimal discussion around model latency or real-time performance, which is critical for practical deployment [7]. These gaps underscore the need for robust benchmarks that incorporate diverse models, retrieval strategies, and holistic evaluation metrics.

To address these gaps, we introduce an intelligent conversational system for extracting insights from academic Portable Document Format (PDF) using a RAG-based architecture. The system integrates recursive text segmentation, FAISS vector indexing, and cosine similarity for precise document chunk retrieval. Three state-of-the-art LLMs, which are LLaMA 3.3 70B, Gemma2-9B IT, and Mixtral 8x7B-32768, are employed to generate responses. Each is then evaluated in combination with three high-performing embedding models, which are all-MiniLM L6 v2, all-mpnet-base v2, and Instructor-Large.

Furthermore, system performance is evaluated using a tri-metric strategy, which includes ROUGE-1 for lexical comparison [8], bidirectional encoder representations from transformers (BERT) score for semantic similarity [9], and a novel HLP index, which assesses response fluency, relevance, and naturalness from a human-centered perspective. In addition, response latency is also measured to determine the system’s real-time usability across varying document sizes [5].

In this study, we evaluate the accuracy among the three LLMs used by utilizing evaluation metrics such as ROUGE-1, BERT Score, and HLP and response latency across model-embedding combinations for real-time deployment scenarios. By combining advanced retrieval techniques with a multi-metric evaluation approach, this article offers one of the first systematic reviews of LLM performance in intelligent document interaction. The findings provide practical insights for deploying document QA systems in academic, enterprise, and educational settings.

2. Methodology

Figure 1 illustrates the system flowchart, which outlines the process of extracting and analyzing content from academic articles to generate accurate responses to user queries. The process begins with the user uploading an academic article in PDF format. If the upload is successful, the text is extracted, cleaned, and divided into smaller chunks. These chunks are processed using a selected embedding model to generate numerical representations (embeddings), which are then stored in a Facebook AI similarity search-based retrieval (FAISS)-managed database for efficient similarity searches. Once this is complete, the system transitions to the query-handling phase. Here, the user inputs a query, which is preprocessed and matched against relevant text chunks retrieved from the database. These chunks are combined into a single context, and a language model is loaded to generate a response using a QA pipeline. The system outputs the user query along with the corresponding response, concluding the process. This workflow is modular, leveraging embedding models, FAISS, and a QA pipeline to enable accurate and efficient information retrieval and response generation.

Figure 1. System process.

2.1. Dataset Description

Six research articles were manually selected from reputable academic databases, including Google Scholar, the Association for Computing Machinery (ACM) Digital Library, and ResearchGate (Table 1, Table 2 and Table 3). The selected articles cover a variety of domains such as mental health, education, biology, physics, technology, and sociology, and are categorized into analytic, experimental, and argumentative/persuasive writing styles. Moreover, each article was converted to text and preprocessed for consistency. A set of three manually curated questions was also formulated for each article, resulting in a total of 18 query-answer pairs for evaluation.

Table 1. Analyzed research articles.

Reference	Topic	Description
Social Media and Mental Health (PDF1) [10]	Mental health	This article reviews the benefits and challenges of outdoor learning in early childhood education and care (ECEC). It synthesizes findings from 20 international studies to highlight how outdoor learning supports holistic development, health, and multimodal, hands-on learning opportunities for children.
Outdoor learning in early childhood education: exploring benefits and challenges (PDF2) [11]	Education	This article investigates the causal impact of social media on mental health, focusing on the staggered introduction of Facebook across U.S. colleges during 2004–2006.

Table 2. Experimental research articles.

Reference	Topic	Description
Studies on the effect of caffeine on growth and yield parameters in Helianthus annuus L. variety Modern (PDF3) [12]	Biology	The article investigates the effect of different caffeine concentrations on the growth and yield parameters of Helianthus annuus L. (sunflower variety Modern). This study highlights caffeine’s potential as a mutagen and growth regulator, depending on dosage.
The Effect of Temperature on the Electrical Conductivity and Microstructure Behaviour of Silver Particles (PDF4) [13]	Physics	The article explores the impact of temperature on the electrical conductivity, microstructure, and mechanical properties of silver particles used in conductive inks. This research highlights the role of temperature in optimizing the performance of silver-based conductive inks in printed electronics.

Table 3. Argumentative or persuasive research article.

Reference	Topic	Description
Determining the Effects of Technology on Children (PDF5) [14]	Technology	The article examines the dual nature of technology’s impact on children, addressing concerns about excessive media exposure and the potential benefits of technological integration in education and socialization.
Death Penalty: what keeps it alive and why it should be abolished (PDF6) [15]	Society	The article examines the current use of the death penalty in the United States and contrasts it with the abolition of the death penalty in many other countries, particularly Europe.

2.2. Dataset Description

Text from each article was segmented into overlapping chunks using LangChain’s RecursiveCharacterTextSplitter (v1.0.0 from LangChain) to preserve the meaning of the text by maintaining the flow between sentences and paragraphs. Each chunk was then converted into a vector using one of three embedding models.

• all-MiniLM-L6-v2 was chosen for being fast and lightweight while still providing good semantic understanding, which is useful for real-time applications.
• all-mpnet-base-v2 was selected because it captures word order and context better than older models like BERT, and performs well on question answering tasks.
• hkunlp/instructor-large was used for its ability to follow instructions and generate task-specific embeddings, which makes it ideal for matching user queries with relevant document content.

These vectors were stored in the FAISS index using cosine similarity. When a user asks a question, the system retrieves the top relevant text chunks based on the question before passing them to the LLM.

2.3. Question Answering with LLMs

The retrieved context is passed to a generative LLM alongside the input question. The selection of LLM models was based on a combination of their architectural strengths, public availability, and relevance to real-world applications:

• LLaMA 3.3 70B was chosen for its strong performance in understanding complex queries and its large context window of 128,000 tokens, making it ideal for analyzing long research articles.
• Gemma 2-9B IT offers a good balance between performance and efficiency. It handles reasoning tasks well and uses fewer resources, making it suitable for real-world chatbot use.
• Mixtral 8x7B-32768 is a Mixture-of-Experts model that runs faster by activating only part of the model during use. It supports up to 32,768 tokens, which helps in processing full-length PDFs effectively.

All models were accessed via HuggingFace Inference Endpoints and OpenLLM interfaces, depending on availability and hardware constraints. They were integrated into a unified retrieval-then-generate pipeline, enabling controlled evaluation across all LLM and embedding model pairings.

2.4. Evaluation Metrics

The system was evaluated using four metrics. Each LLM-embedding pair was tested across all 18 queries, resulting in 135 evaluations:

• ROUGE-1: Measures the overlap of generated responses with expected answers.
• BERTScore: A metric for evaluating the quality of text generation based on semantic similarity.
• HLP: A binary metric (Yes/No) for each response to assess whether the response matches a reasonable human answer.
• Latency: Response time was recorded in seconds to assess real-time usability.

3. Result and Discussion

Table 4 presents the performance evaluation of various LLM and embedding model combinations using ROUGE-1, BERTScore, HLP, and response time. It highlights the accuracy, semantic relevance, and efficiency of each configuration for document question answering. Table 4 provides a comparative evaluation of nine distinct model configurations formed by pairing three embedding models—all-MiniLM L6 v2, all-mpnet-base v2, and Instructor Large—with three LLMs: LLaMA-3.3-70B, Gemma2-9B IT, and Mixtral 8x7B-32768. The evaluation considers four key metrics: ROUGE-1, BERTScore, HLP, and average response time.

Table 4. Results.

Model	LLM	Average ROUGE-1	Average BERTScore	Average HLP (%)	Average Response Time
All MiniLM L6 v2	llama-3.3-70b versatile	0.308	0.309	94	1.50
	gemma2-9b-it	1.000	0.443	89	0.53
	mixtral8x7b-32768	0.388	0.439	83	0.64
All Mpnet Base v2	llama-3.3-70b versatile	0.369	0.286	95	1.10
	gemma2-9b-it	1.000	0.437	89	0.23
	mixtral8x7b-32768	0.963	0.480	89	0.92
Instructor Large	llama-3.3-70b versatile	0.291	0.438	89	0.79
	gemma2-9b-it	0.875	0.258	78	0.77
	mixtral8x7b-32768	0.363	0.395	83	0.90

Among the combinations tested, Gemma2-9B IT with all-MiniLM L6 v2 achieved perfect ROUGE-1 and the highest BERTScore within its category (1.000 and 0.443) while also delivering one of the fastest response times (0.53 s), highlighting its strong balance between accuracy and efficiency. The Mixtral 8x7B-32768 with all-mpnet-base v2 pairing reported the highest BERTScore overall (0.480), alongside a strong ROUGE-1 score (0.963), making it well-suited for high-precision retrieval tasks. In terms of HLP, LLaMA-3.3-70B consistently demonstrated top-tier fluency and relevance, particularly when combined with all-mpnet-base v2 (95% HLP). However, this came at the cost of increased latency (up to 1.50 s with all-MiniLM L6 v2). Conversely, Instructor Large embeddings generally resulted in lower latency but showed more variability in HLP and semantic accuracy. Overall, the table underscores the trade-offs between response quality and computational efficiency across different embedding-LLM configurations.

4. Conclusions

We assessed the accuracy of three state-of-the-art LLMs—LLaMA 3.3 70B, Gemma2-9B IT, and Mixtral 8x7B-32768—using ROUGE-1, BERTScore, and the proposed HLP metric and examined their response latency in combination with different embedding models for real-time deployment. The experimental results demonstrate that the Gemma2-9B IT paired with all-MiniLM L6 v2 delivers an optimal balance between accuracy and speed, achieving perfect ROUGE-1 and strong BERTScore performance with low latency. The Mixtral 8x7B-32768 model with all-mpnet-base v2 achieved the highest semantic relevance (BERTScore of 0.4800), showing promise for tasks where semantic precision is critical. Notably, LLaMA 3.3 70B, despite having the highest HLP rating (up to 95%), incurred higher latency, indicating a trade-off between response fluency and efficiency.

HLP as an evaluation metric further enriched the analysis by capturing human-centric qualities such as naturalness and contextual alignment, which are often overlooked by traditional lexical and semantic metrics. This positions HLP as a valuable complement to existing benchmarks in future LLM evaluations. Overall, the findings affirm that embedding model selection significantly influences retrieval precision, while LLM choice directly impacts the quality and naturalness of responses. The results support the deployment of adaptable, high-performing document QA systems across academic, educational, and enterprise domains. Future work will focus on domain-specific tuning, multi-document support, and optimizing real-time performance for broader scalability and robustness.