NLP System for Automation of Document Workflow in a Research and Development Organization—A Case Study

Featured Application

The Large language model (LLM)-based system for autonomous document processing was developed with a view to its application in the GIG National Research Institute (GIG-NRI). The system will help with automatic document classification, and depending on the type, information extraction, summarization, or action triggering. The system is in the Minimum Viable Product (MVP) stage, and its further development depends on pilot assessments.

Abstract

Research and development (R&D) organizations face significant operational bottlenecks due to the manual processing of diverse, unstructured documents. This paper presents the design, implementation, and pilot evaluation of an on-premise, multi-agent natural language processing (NLP) system developed for the GIG National Research Institute (GIG-NRI). Built upon a LangGraph architecture, the system utilizes open-weight large language models (LLMs) to perform zero-shot document classification, dynamic routing, and specialized information extraction. We rigorously evaluated the classification agent across twelve different local LLMs under two distinct testing regimes: first, using a strictly defined dataset of known administrative and scientific document types, and second, introducing a subset of out-of-distribution (unclassified) data to test real-world robustness. Our results demonstrate that the 70-billion parameter model (cogito:70b) achieved a peak accuracy of 97.3% in the first regime and maintained a strong 94.3% accuracy when confronted with out-of-spec data. However, our analysis reveals a critical operational trade-off regarding computational efficiency. The 24-billion parameter (magistral:24b) and 32-billion parameter (qwen3:32b) models emerged as the next best in overall accuracy while requiring less than half the processing time of their 70B counterpart. Notably, magistral:24b proved superior for strictly defined document streams, whereas qwen3:32b demonstrated greater robustness when handling out-of-distribution inputs. Furthermore, we demonstrate the efficacy of heterogeneous model assignments for complex multi-stage tasks, such as Scientific Article summarization via hierarchical Map-Reduce.

1. Introduction

Currently, enterprises, including research and development units, face a dual challenge: economic transformation and digital transformation. Both issues are interconnected, as digital transformation and the application of ML/AI solutions are undoubtedly the key factors increasing competitiveness and efficiency today. Naturally, it must be remembered that the new opportunities associated with technological development also bring new risks, particularly in terms of cybersecurity, which must be effectively mitigated. The application of NLP to support the effective ingestion and circulation of information within an institution is one of the most active areas of applied research. The challenges of natural language processing and automatic understanding were largely overcome by the creation and application of large language models (LLMs). Despite the vast proliferation of this technology in daily life (almost everyone has used or at least heard of ChatGPT, Gemini, Claude, or other solutions), there remains broad potential to use its capabilities to transform the way industry, administration, and R&D organizations conduct their everyday activities.

There are a few pathways in which the use of LLMs is considered: information discovery and extraction, document summarization, the retrieval-augmented generation (RAG), and multistep agentic reasoning [1,2,3]. The LLM-based systems are believed to be able to reduce the workload and time spent on repetitive, standard activities and allow employees to focus on more abstract and demanding challenges [1,4]. However, their implementation and the scope of their support within existing enterprises and organizations can vary significantly. Perhaps the most commonly reported applications are presented in the healthcare sector [5], but other sectors, namely legal [6,7], administration [8], and manufacturing [9], are also reported in the literature. Tripathi et al. demonstrated how LLM support can help in administrative work accompanying healthcare services [10]. The work mentions the important aspect of privacy and security issues, which are particularly important in the case of patient data processing. The importance of security and privacy problems is thoroughly investigated by Harrer [11]. The author emphasizes the ethical, cultural, and even technical challenges that must be addressed before the safe adoption of the LLMs in healthcare is possible. In the review [5], the authors present the wide range of possible applications of multimodal models in healthcare, but point out their limitations: bias, hallucination, and the necessity for privacy and security standards implementation. Similar considerations are also presented in [12]. While the application of AI can bring value and support medical staff, there is still a wide area of improvement, and users have to be aware of potential drawbacks. The enormous rise in research in the field of application of AI in healthcare was reported by the authors of the review [13]. Similar conclusions are also presented in [14,15]. The latter emphasizes the increasing role of multimodal AI as well as the application of LLMs in healthcare institutions’ administration. The support given by AI in the administration is fundamental for research reported by Lorencin et al. [16]. The authors state that the usage of LLMs can be beneficial by taking over the processing of administrative documents, billings, forms, and so on. The medical staff can focus on patient care, no longer wasting their time and expertise. This observation also serves as a fundamental principle for our work. The usage of an on-prem LLM solution (including the model itself) guarantees superior control over data privacy and security. Though it seems healthcare is most present in the research of AI applications, the other areas of activity are also considered. Plonka et al. investigated the optimal splitting of large legal texts for RAG purposes [7]. The thorough review of LLMs’ proliferation in supporting legal document processing is presented in [6]. The authors notice that the linguistic nature of legal documents makes them particularly suitable for natural language processing (NLP). The authors conclude that effective processing of legal documents requires the application of fine-tuned models—the general-purpose ones are not able to correctly understand the nuances present in legal professional language. The LLM-based models are also considered for application in governmental administration document processing [8]. Their role in such a case is focused on information extraction, document classification, and querying. Automation of the management of vast amounts of documentation results in improved effectiveness and a lower error level. Note that though the LLMs in a variety of types (namely ChatGPT, Llama, Claude) are more and more used in business process automation, often the crucial issue today is not the technology capabilities themselves, but rather their efficient, secure, and compliant integration with existing systems [17].

Despite the successes and willingness of the organizations to include LLMs in their everyday operations, there are limitations and threats that anyone has to be aware of. Investigating the performance of three selected models in solving complex tasks, the surprisingly poor performance can be observed [4]. The problem is particularly severe when models are used in a compound, multi-application environment. The above is one of the symptoms showing that, despite enormous progress, there are still many aspects that have to be taken into consideration when LLM-based solutions are implemented. The first and probably most important, not only for the healthcare sector, is data security and privacy [18,19]. The second, and just as commonly mentioned, are the integration issues [20,21]. This problem is particularly severe when integration involves legacy systems. The other commonly reported issue is the significant demand for resources and computational power [22,23]. Apart from the mentioned problems lying mostly at the junction of LLM-based systems and the existing infrastructure and requirements, there are also issues inherently built into the way such models work. One of these phenomena is the “black box” nature of the models, which leads to a lack of explainability and a lack of inherent tools for judging the quality of model output [24,25]. Another problem is the so-called model hallucinations, where the model generates the output that is not grounded in the provided data or is inherently false [26,27]. The mitigation of such dangerous cases in practice usually involves the application of self-monitoring in the developed system, for example, by using an LLM model for assessing the output of another LLM model [28,29,30]. In spite of the mentioned open research and technical questions, the LLMs are expected to play an increasingly important role in information processing in business. Though the technology still needs to settle down and move towards being well-established, with clearly defined processes and tools [31], it is now the best time to initiate exploration in this promising domain.

The implementation of solutions utilizing large language models (LLMs) within a research institute represents a strategic approach to investing in the acceleration of the research cycle and operational optimization. Institutes operate on vast volumes of data, especially text data—from administrative documentation and tender specifications to thousands of scientific publications and patents. LLM-based solutions will enable the automation of the analysis, synthesis, and generation of this content, thereby freeing up the institute’s most valuable resource: the time of its highly qualified scientific and support staff. Instead of manually searching databases or formatting reports, researchers can focus on interpreting results and generating innovations.

The solution presented in this article is an approach that, in the short term, will yield effects such as shortening the time required for correspondence handling, automating the initial categorization of documents, and generating drafts of standard letters. Additionally, it will enable the automatic filtering of tender inquiries for compliance with the institute’s research profile and its research teams, as well as the preliminary identification of key formal requirements. From the perspective of direct support for scientists, the resulting benefit will be the ability to generate summaries of long reports, analyses, or scientific articles in near real-time, which accelerates the decision-making process.

The aspects mentioned above will allow, in the long-term perspective associated with the system’s functional development, a significant reduction in the time needed to analyze the state-of-the-art when writing grant applications and R&D projects. This will directly translate into a higher quality and greater number of submitted applications, ultimately leading to an increase in competitiveness. Another anticipated outcome will be the creation of an internal knowledge base for the institute, enabling the rapid location of experts, procedures, or archived project results, which will be understood as an element of knowledge management. The approach presented in the article, involving the analysis of global scientific trends (based on analyzed data, e.g., articles and patents), will also eventually allow for the identification of new, promising research directions and/or point to areas for potential cooperation.

In the GIG National Research Institute (GIG-NRI), we established the pilot project for developing a solution supporting the processing of a selected set of document types commonly used within the organization. In the paper, we would like to share our experiences, ideas, and problems we encountered while developing the minimum viable product (MVP) version of the system. Our contributions are as follows:

• The architecture and tools for the development of an agent-based system for an R&D organization have been proposed.
• The optimization of LLM model selection for a specific task is presented.
• The pilot deployment of the LLM-based solution was elaborated and presented.

2. Materials and Methods

2.1. General Assumptions

The daily operations of research and development (R&D) institutions necessitate the management of a vast volume of written documents. While a portion of these documents is generated internally, the majority originates from external stakeholders, including contractors, partner research organizations, and governmental bodies. Consequently, these documents are highly unstructured, yet they contain critical information that must be systematically extracted and processed to meet institutional requirements. Although supported by existing information technology (IT) infrastructure, this processing remains inherently manual and heavily reliant on human intervention. The work is largely repetitive, tedious, and time-consuming. Automating this workflow would not only significantly reduce document processing times but also optimize the allocation of human capital, allowing highly skilled employees to focus on more complex, intellectually demanding tasks. Therefore, the objective of this study is to design and implement an autonomous document processing system. Deployed as a minimum viable product (MVP), this solution aims to evaluate the feasibility of such automation within an R&D context. Specifically, it seeks to quantify the potential reduction in manual workload and processing time, and ultimately, to perform a cost–benefit analysis comparing system implementation expenses against projected operational savings.

The following assumptions were formulated upon screening analysis of needs and possibilities:

• The system should help in processing the most tedious and work-consuming documents.
• Most documents may contain sensitive or commercially confidential data.
• The processing workflow depends on the document type, as does the information retrieved from it.
• The system is intended to expose its functionality via API.
• The regulations of AI and, particularly, LLM usage in the organization have to be developed.

The assumptions from 1 to 4 were directly used in system architecture and code development. The details of that process are presented in the following sections. Assumption 5 led to the development of a formal document, of which details are presented in Section 2.5. The main criterion for success is strictly tied to the limitations of human labor while preserving a low level of errors.

2.2. NLP System Architecture and Design

The system was designed to operate as a web service, exposing its functionality via an application programming interface (API) to be utilized by client applications. It was architected as a system of collaborating AI agents that execute their functions using large language models (LLMs) and a suite of custom tools.

The general document processing workflow is implemented as a pipeline consisting of the following steps: data extraction from documents, document classification, type-dependent information extraction or generation based on the document’s content, and the generation of a formatted report. The identified document types and the scope of information extracted and other processing conducted for them are presented in

Table 1. Document types and processing steps.

Document Type	Information Extracted/Processing
Invoice	Extraction of: order value, tax, payment deadline, overdue payment flag, number of items, invoice ID, issue date, date of service/delivery, client identification data, payer identification data.
Contract	Extraction of: parties to the agreement, subject of the agreement, contract date, terms of execution, contractual penalties.
Scientific Article	Extraction of: title, authors, affiliations, publication date, journal, DOI. Generation of abstract and keywords. Determination of Impact Factor—IF.
Tender Specification Document	Extraction of: subject of the order, contracting authority, technical requirements, eligibility requirements, required references, bid bond/tender security, estimated order value based on the bid bond, submission deadline, required formal documents, terms of contract execution, warranty conditions, contractual penalties, service level requirements.
Plagiarism Detection Report	Extraction of: title, authors, Classification of reports outcome (evaluation result) into one of the classes: plagiarism, requires clarification, OK.
ISO Procedure Document	Identification of the relevant procedure and verification of compliance with the document template.
Other	No processing; user decision required.

.

The main workflow consists of a few steps:

• Extraction: In this step, the text information is extracted from the document. Cleaning of the text is performed (removal of empty lines, removal of non-textual chars, etc.).
• Document type classification: The type of document is determined based on its content. The result document is classified into one of the classes described in Table 1.
• Processing: The document text analysis is performed in this step. The exact operations performed are dependent on the document type.
• Final response generation: In this step, the final form of system output is generated. There are two formats considered: Markdown for human-readable output and JSON as a format most convenient for exchanging with other systems working in the institute.

The overall process of document transformation is presented in Figure 1. The input document (PDF, Microsoft Word, or image format) is processed by the first node in the processing chain. The node is responsible for the extraction of plain text from the document. For that purpose, it delegates the conversion from native document format into plain text to a specialized tool—docx reader, PDF reader, or OCR. The tools return the plain text extracted from the documents. Then, the next node is responsible for preprocessing. This step focuses on text cleaning. The raw document text passes through a filtering process to remove “noise” while preserving critical metadata. All footnote/reference markers ([1], [note 5], [edit], etc.) are removed. Moreover, short lines (≤3 words) such as page numbers, headers, and sparse formatting artifacts are eliminated. However, the metadata-rich lines are preserved regardless of word count (DOI references, publication/submission dates, author information, and date formats), ensuring that important document attributes required for downstream extraction pipelines remain intact.

The next step is responsible for document type classification. The node classifies the document into six considered classes (ISO Procedure Document, Plagiarism Detection Document, Tender Specification Document, Scientific Article, Contract, and Invoice). If none of the mentioned classes is recognized for the document, it is labeled as “other” and left for human decision for further processing steps.

Once the classification is done, the language of the document is recognized (it is used for matching prompt language with the document language, so if possible, the prompts used for further processing are intact with the documents. Upon the classification result, the suitable pipeline for data extraction/generation is invoked. The pipelines respond with JSON-formatted output, which is then processed by the formatting node. At that stage, depending on requirements, the JSON is checked and corrected, or the Markdown file is generated. The result of formatting the node is then returned to the user.

The workflow is realized by the use of a multi-agent system. The AI agents compounding the system are organized in the form of a graph with a bunch of conditional edges. The crucial role is played by the orchestration agent. It is responsible for supervising the overall flow of information in the system. All document and further text processing activities are delegated by the orchestration agent to the specialized subagents. Each of the subagents is an independent workflow. Most of them are relatively simple, realizing the information extraction functionality. However, in a few cases (e.g., Scientific Article, Plagiarism Detection Report), a few steps must be taken to complete the agent’s task. The subagents return the results of their work in the form of a well-formatted JSON file. The file is then passed by the orchestrating agent to the formatting agent. The formatting agent transforms the JSON format into Markdown, more suitable for human-based verification and further processing. The JSON format is used for passing the extracted data to other systems working in the institute (ERP, Intranet service, dedicated databases). The idea of document processing management is presented in Figure 2.

First, the document for processing is submitted to the system using the Web API. The document (in Microsoft Word, PDF, or bitmap image format) is then handled by the orchestration agent. The agent delegates the control and document to the extracting agent. In the extracting agent, the document is transformed to plain text and cleaned (see description of Figure 1 for details). The resulting cleaned text of the document is then passed to the classifying agent, where the type of document is recognized. Then, upon identifying the document class, the orchestrating agent passes the document to suitable information-extracting agents. The agents handle the processing routine adjusted to the functionality required for the given document type. The result of specialized agent processing is then returned to the orchestrating agent, which delegates it for final formatting to the formatting agent. The resulting formatted document (in JSON or Markdown format, depending on requirements and document type) is returned to the user.

The system is intended to work in cooperation with the other systems present in the organization. The functionality is available through the web API. In

Table 2. The description of the API exposed by the developed system.

Method	Exposed Functionality
pass(document_file) returns: document ID (guid), status code	The document is uploaded to the system manager repository. The document is given an ID, and the user is informed of the status code of the result (OK/error description).
process_pipeline(document_ID) returns: status code	The complete document processing workflow is executed. As a result, a Markdown or JSON file is stored in the repository with the same ID as the input document. The status code is returned to the caller.
process(document_ID, document_type) returns: status code	The processing workflow dedicated to the selected document type is executed. The true type of document identified by the passed ID is not checked. As a result, a Markdown or JSON file is stored in the repository with the same ID as the input document. The status code is returned to the caller.
classify(document_ID) returns: document type, stats code	The classification part of the workflow is executed. The classification result is returned to the caller along with the status code.
get_result(document_ID) returns: Markdown/JSON result document, status code	The content of the Markdown/JSON file (the result of document processing) for the given document ID is returned to the caller.
delete_document(document_ID) returns status code	All stored in files tied to the given document ID are removed permanently from the repository.

, the description of the designed interface is presented.

The proposed set of methods gives the user flexibility in document processing. The documents can be processed from the input file to the resulting Markdown or JSON file, but the process can be split into parts, allowing execution of each part of complete workflows independently. The possibility has twofold importance. First, it allows enforcing the correct processing pipeline in cases where automatic classification gave the wrong answer or is not able to determine the document type (the ‘other’ type). Second, it provides the possibility to execute parts of the workflow in batch (e.g., the classification stage or Invoice Processing pipeline for all selected documents is processed). Not only does it have more control over document processing, but it also allows efficiency optimization when different models are used for different stages in the processing workflow. In such a case, the model reloading on the server acting as the model host can be limited.

2.3. Infrastructure

One of the crucial requirements for the system was to limit the sensitive documents and information sharing with external systems. While it is possible to use dedicated cloud resources provided by the well-recognized providers, the option of using an on-prem system was chosen for the solution. Such a decision leads to the necessity of ensuring sufficiently powerful servers along with the network infrastructure. For the solution developed, the following infrastructural decisions were made:

• The LLM model (or models) is run at a dedicated server with at least one dedicated GPU card (an LLM server—for detailed configuration see Appendix B).
• The solution is deployed at one of the available servers (solution server—different from the LLM server).
• The communication between the solution server and the LLM server is provided by a wide-band, high-performance network connection.
• The solution server is the only one allowed to connect and use the resources of the LLM server, which is enforced by network configurations.
• The external systems connect to the solution server using the exposed Web API. The security checks apply; the list of servers hosting external systems allowed to connect to the solution server is limited.
• The filesystem resources of the solution server are used for document and Markdown/JSON files’ storage.
• No backup of processed files is done on the solution server; external systems are responsible for this task.

All hardware and network infrastructure necessary for system development and usage are provided by GIG-PIB resources.

For code development, code versioning, and Continuous Integration and Continuous Deployment (CI/CD) operation support, the Azure DevOps platform was used. Project management, as well as the issues reporting/solving process, is entirely based on the tools available on the platform.

2.4. System Development

The environment of the Python (version 3.12.3) language was chosen as a base for system development. This decision is based on the popularity of Python in the development of machine learning and artificial intelligence solutions. The language itself and its supporting libraries are well-documented, easy to obtain, and most are published with licenses allowing for commercial use. Visual Studio Code (version 1.106) was selected as the Integrated Development Environment (IDE) for all software development tasks. The Ubuntu Linux 24.04 LTS was used as a platform for system development and deployment.

One of the crucial design decisions was to separate the server on which the LLM instance is running from the server used for the developed system’s hosting. For simplicity and reported successful applications, Ollama was selected as the on-premises LLM hosting service [32,33].

The internal construction of the system was developed using a multi-agent approach. The functionality was divided into coherent, self-contained workflows, such as “Scientific Paper Processing” or “Invoice Processing.” For each defined document type, a separate processing pipeline was implemented. The overall process of document handling is managed by an orchestration agent, whose role is to delegate control for each step in the processing chain to dedicated, specialized agents.

The agents themselves were designed using a graph abstraction and implemented using the LangGraph library, which is widely used in the development of advanced AI agents [34,35,36]. A key architectural feature is the main Router Graph, which performs an initial, AI-driven classification of the input document. Based on this classification, it uses conditional routing to invoke the appropriate specialized sub-graph for the identified document type. This modular design ensures a clear separation of concerns and allows for scalable expansion of the system’s capabilities.

Interaction with LLMs, prompt organization, and output formatting were implemented using LangChain, another library known for its robustness in the development of LLM applications [37,38,39]. To enforce reliable, structured output from the LLMs, Pydantic [40,41] schemas are used to define the target data structure for each extraction task. The system leverages LangChain’s output parsers to validate that the model’s JSON output conforms strictly to these schemas. For complex, long-document tasks such as the summarization of scientific papers, a hierarchical Map-Reduce strategy (Map-Reduce with an intermediate collapse step) was implemented. This approach overcomes the context window limitations of the models by first summarizing smaller chunks of the document independently (“map”) before recursively combining these summaries (“reduce”) into a final, cohesive output.

For flexibility, prompts are stored in the filesystem outside the code, allowing for easy modification and tuning. The association between prompts and the agents that use them is achieved based on a convention-driven folder structure. It was also decided that prompts should be written in the language that conforms with the processed document’s language, a determination made by an automatic language detection node early in the processing pipeline. Prompts for a given agent are stored in the prompts/<agent-name>/folder, and the filenames follow the format “current_<agent_part_name|’main’>_<language code>.prmpt” (the ‘|’ bar is used as an alternative operator here). Despite the name extension, all prompts are in fact stored in ordinary, UTF-8 text files. An example of the actual prompt template used for the classification node is provided in Appendix A.2.

To allow for optimizing model selection for each processing step, a configurable LLM factory module was proposed and implemented. Its construction is straightforward: based on a central configuration file (in JSONC format), a model manager instantiates the proper LangChain wrapper object for a given agent. This allows for heterogeneous model deployment, where different models—varying in size, specialization, or even hosting location (local or remote)—can be assigned to individual processing steps. It is assumed that if not specified for a particular agent, a default model configuration is used. This solution allows us to systematically examine the robustness and efficiency of different models for various document processing tasks. The exact configuration parameters applied to the LangChain—Ollama connection are detailed in Appendix A.3.

2.5. Organizational and Formal Considerations

Regarding the required organizational and formal actions, it must be remembered that the implementation of LLMs is not only an IT project but, first and foremost, a management and organizational project. Therefore, it is critical to define the necessary actions and regulations that ensure the comprehensive security of the organization, especially in a research and development entity.

In terms of organizational and formal actions, the first step should be to appoint an interdisciplinary team to map the processes that are planned for LLM support and that possess the greatest potential for efficiency gains. It is also necessary to decide on the deployment model, which is critical for security. This step will require a decision on the choice of technology and model. The next element is securing information safety. This is the biggest challenge in a scientific unit that processes trade secrets, sensitive data, creates innovations, protects intellectual property, and generates its own IP/Know-how.

The subsequent step is preparing employees: through communication—providing information about the activities and their purpose; and through training—on tool operation and security awareness (particularly concerning what data must not be entered). A significant element of the implementation process is prioritizing processes and gradually deploying solutions in stages, while having a predefined target action plan.

A very important element is the implementation of a regulation/procedure for using AI tools, established as a formal document defining the permitted tools for use in the institute, the categories of data that must not be processed, the scope of authorizations, and the scope of responsibility, especially regarding responsibility for results generated with the aid of ML/AI tools. It is crucial to establish regulations regarding the definition of key documents generated by AI that cannot be shared with external clients without human substantive verification and acceptance. This will ensure an appropriate level of security and maintain the rule that the AI tool is intended to serve as a supporting, rather than a deciding, instrument.

3. Results

The developed system was evaluated in a controlled environment, but strictly corresponding to the operational, production conditions. The evaluation was focused on the performance and flexibility of the system’s core architectural components. The system’s ability to classify documents, execute specialized extraction pipelines, and optimize performance through configurable model deployment was validated during tests.

3.1. Document Classification and Dynamic Routing Performance

The effectiveness of the AI-driven classification agent, serving as the primary decision node, was evaluated using a rigorous pilot protocol. To ensure the results reflect the specific operational reality of the institute, a balanced dataset of 300 documents was assembled, comprising exactly 50 real-world examples for each of the six identified categories (Invoice, Contract, Scientific Article, Tender Specification Document, Plagiarism Detection Report, ISO Procedure Document). The ground truth classification for this dataset was established through a manual verification process involving two senior administrative staff members. This ensured that the test set was free of label noise and ambiguities, providing a reliable baseline for assessing the agent’s performance.

The workflow is designed to handle both out-of-scope documents and execution failures by clearly distinguishing between semantic rejections and technical errors. To manage irrelevant or unrecognizable documents (e.g., books, manuals, or general correspondence), the prompt explicitly instructs the LLM to assign an “Other” label (see Figure 1). Unlike the six predefined categories, “Other” functions as a controlled fallback mechanism for out-of-scope content, allowing the system to reject the document successfully without breaking the parsing chain.

Conversely, technical anomalies—such as JSONDecodeErrors, schema validation exceptions, or empty text outputs—are no longer conflated with the “Other” category. Instead, the evaluation logic explicitly traps these pipeline failures and assigns them a “system error” label. This approach ensures that any execution failure automatically registers as a misclassification against the ground truth, objectively penalizing the model’s overall accuracy, precision, recall, and F1-scores.

The classification pipeline was evaluated under two distinct testing regimes. The first regime utilized a closed-set evaluation to determine the system’s baseline accuracy on known, predefined document types. The second regime introduced an open-set evaluation to test the system’s robustness against out-of-distribution data. For this phase, the dataset was augmented with 50 diverse, out-of-scope documents (e.g., novel excerpts, CVs, instruction manuals, and personal correspondence). It must be noted that the overall dataset size was constrained by the highly sensitive and proprietary nature of the institution’s operational documents.

A perfectly balanced dataset was deliberately chosen to assess the proposed classification method. While real-world document streams are inherently imbalanced, this methodological choice is scientifically justified for a zero-shot inference paradigm. Because the proposed system does not undergo a traditional training phase, it is immune to the class-imbalance problem, wherein a model develops a statistical bias toward majority classes. Consequently, utilizing a balanced test set isolates and measures the LLM’s inherent discriminative capability for each specific document type. This ensures that performance metrics, such as the macro-averaged F1-score, reflect the model’s competence uniformly across all target classes without skew. Nevertheless, we acknowledge that these metrics represent the model’s idealized discriminative capacity; in a live production environment, the natural imbalance of the document stream will ultimately dictate the absolute operational throughput.

As mentioned, the experimental methodology adopted a zero-shot classification paradigm. A custom batch evaluation script submitted the initial 6000 characters of each document text to the agent via the standardized API. The classification task was executed five times for every model–document pair. The reported results represent the average performance across these runs. The experiment covered 12 distinct open-weight models hosted on an on-premise Ollama server equipped with an NVIDIA L40S GPU with 48 GB video RAM. The details of hardware configuration can be found in Appendix B. The model selection aimed to cover a wide spectrum of sizes (from small 2B parameter models to large 70B models) to identify the optimal trade-off between accuracy and resource consumption. Quantitative results of this comparative analysis are summarized in

Table 3. The document type classification results (only documents of known types).

Model Name	Accuracy	Precision	Recall	F1	Comp. Time [s]
cogito:70b	0.973	0.847	0.834	0.839	2468.40 ± 1.66
deepseek-r1:70b	0.940	0.829	0.806	0.812	2481.70 ± 1.14
devstral:24b-small	0.940	0.832	0.806	0.815	477.28 ± 9.93
gemma3:27b	0.947	0.960	0.947	0.946	657.96 ± 1.14
granite4:small-h	0.693	0.654	0.594	0.588	374.03 ± 0.25
llama3.1:70b	0.953	0.828	0.817	0.819	2405.30 ± 7.33
llama3.1:8b	0.800	0.715	0.800	0.746	262.98 ± 0.16
llama3.3:70b	0.907	0.808	0.777	0.775	2249.39 ± 3.94
magistral:24b	0.967	0.841	0.829	0.833	436.87 ± 1.59
mistral-small3.1:24b-instruct	0.843	0.793	0.723	0.702	690.21 ± 0.44
qwen2.5:72b	0.853	0.792	0.731	0.703	3141.07 ± 13.83
qwen3:32b	0.927	0.829	0.794	0.803	585.43 ± 1.18

. Performance is reported using standard multi-class classification metrics: Accuracy, Precision, Recall, and F1-score (macro-averaged across classes). Additionally, ‘Comp. time’ denotes the total time required to process the entire batch of 300 documents (or 350 documents for the enhanced dataset) sequentially, serving as a proxy for computational cost.

The results of the classification for the first dataset, containing only documents of selected types, are presented in Table 3. The best accuracy of 97.3% was observed for the cogito:70b model. Such accuracy is sufficient for any practical considerations. The confusion matrix for the best case was presented in Figure 3. The model failed only with ISO Procedure documents, of which four were unrecognized (classified as “other”), and four were incorrectly classified as Tender Specification documents. However, the model required an average of over 40 min to complete the task, which might be a bottleneck when larger amounts of documents need to be processed. Upon successful classification, the appropriate specialized sub-graph was invoked for further document processing. The obtained results proved the viability of the conditional routing architecture.

The cogito:70b model was also the best when the extended dataset was used for the experiment. The obtained accuracy (94%) is still high enough for practical application (see

Table 4. The document type classification results, including the “other” class.

Model Name	Accuracy	Precision	Recall	F1	Comp. Time [s]
cogito:70b	0.943	0.946	0.943	0.941	2859.80 ± 1.99
deepseek-r1:70b	0.846	0.839	0.846	0.824	2895.32 ± 1.13
devstral:24b-small	0.877	0.875	0.877	0.868	546.82 ± 6.72
gemma3:27b	0.843	0.873	0.843	0.815	766.61 ± 1.23
granite4:small-h	0.686	0.664	0.686	0.629	436.37 ± 0.27
llama3.1:70b	0.880	0.906	0.880	0.872	2816.19 ± 7.91
llama3.1:8b	0.731	0.705	0.731	0.677	306.81 ± 0.17
llama3.3:70b	0.840	0.878	0.840	0.827	2626.29 ± 4.06
magistral:24b	0.894	0.897	0.894	0.883	509.68 ± 1.71
mistral-small3.1:24b-instruct	0.780	0.844	0.780	0.744	806.24 ± 0.51
qwen2.5:72b	0.794	0.870	0.794	0.757	3684.58 ± 12.93
qwen3:32b	0.917	0.928	0.917	0.913	683.00 ± 1.27

). Again, the most important model disadvantage is the time needed for processing the 350 documents—almost 50 min. The confusion matrix (Figure 4) shows that the model misclassified not only the ISO Procedures document but also failed in classifying the “Other” category.

It has to be stressed that the “Other” class is particularly difficult to be processed by the model correctly, as there are no clear rules on what the other documents should look like. It is rather the option for documents that are not recognized as any other category. But it can, in fact, be composed of text segments that might resemble other document classes. The task in itself has a nature of open-set classification [42]. Therefore, the relatively high level of misclassification is understandable.

To establish a quantitative baseline for document type classification, we employ classical supervised learning pipelines that combine Term Frequency–Inverse Document Frequency (TF-IDF) [43] feature extraction with two well-established classifier families: Support Vector Machines (SVM) [44,45] and Random Forests (RF) [46]. The computations for the baseline were conducted twice: once with known document types and once using the extended set. However, the second case is in fact the open-set problem, which is simulated by explicitly introducing an artificially prepared set of ‘other’ class documents. While in the case of LLM, such an approach is not limiting, as it merely tests the model’s ability to properly recognize documents not belonging to any known class; conversely, the use of classical machine learning approaches involves a training phase, for which the training data belonging to “other” class may not exhaust the variety of the real world.

The stage of text extraction from the documents was exactly the same as in the case of LLM-based classification. Then, the text representations are computed using the TF-IDF weighting scheme. There were three classifier configurations evaluated: Linear SVM, RBF (Radial Base Functions) SVM, and Random Forest (RF).

The datasets were partitioned into a training set (70%, n = 210 for the smaller and n = 245 for the extended dataset) and a held-out test set (30%, n = 90 for the smaller and n = 105 for the extended dataset) using stratified random sampling.

During training, the 5-fold stratified cross-validation was applied exclusively to the training set. Final performance figures are reported on the held-out test (the baseline values). The values obtained for both considered cases are reported in

Table 5. Classification metrics for test set (only documents of known types).

Classifier	Accuracy	Precision	Recall	F1
SVM (Linear)	0.867	0.880	0.867	0.865
SVM (RBF)	0.856	0.866	0.856	0.855
Random Forest	0.867	0.868	0.867	0.864

and

Table 6. Classification metrics for test set (“Other” class included).

Classifier	Accuracy	Precision	Recall	F1
SVM (Linear)	0.914	0.924	0.914	0.916
SVM (RBF)	0.886	0.900	0.886	0.888
Random Forest	0.895	0.902	0.895	0.895

, respectively.

The TF-IDF + SVM baseline achieves an accuracy of approximately 86.7% and a macro F1-score of 0.865 on the closed-world, six-class task, establishing a robust empirical reference for the classification problem. Notably, the majority of the zero-shot LLMs outperformed this baseline, with only four out of the twelve tested models yielding a lower accuracy.

However, interpreting the baseline model’s performance on the extended dataset—which includes an explicit ‘Other’ class—requires careful methodological scrutiny. While the SVM baseline results on the extended dataset were surprisingly strong (outperforming the majority of the LLMs, though remaining inferior to the top-performing models like cogito:70b), these metrics are likely artificially inflated by the constraints of the training data. The addition of an ‘Other’ category transitions the task from standard classification to open-set recognition [47]. Traditional supervised algorithms like SVMs are discriminative models that partition a closed feature space. When trained with an explicit ‘Other’ class, the SVM attempts to draw a decision boundary around the specific outlier examples provided in the training set. This approach provides an adequate approximation only if the ‘Other’ examples used during training are generally representative of the infinite space of out-of-distribution documents that will be targeted during inference. In a production environment, this condition is impossible to guarantee, and discriminative boundaries predictably fail when presented with novel document types [42].

The zero-shot LLM approach fundamentally circumvents this limitation. Rather than partitioning a closed vector space, LLMs approach classification as a task of semantic entailment and instruction following [48]. The LLMs evaluate the semantic alignment between the input document and the explicit definitions of the target classes. Therefore, the ‘Other’ category is not treated as a spatial region in a feature space. It is rather a form of exclusion: ‘the document does not semantically match the definitions of Invoice, Contract, etc.’. This makes zero-shot LLM architectures inherently more robust to true out-of-distribution inputs.

3.2. Multi-Stage Structured Data Extraction from Scientific Papers

While Table 1 outlines the system’s comprehensive processing capabilities across six distinct document categories, the empirical evaluation of the extraction phase in this study is deliberately focused on the Scientific Article pipeline. This scoping decision is rooted in the primary architectural contribution of this work. Standard administrative documents, such as invoices and contracts, possess relatively predictable structures, and their extraction using modern LLMs has been extensively validated in the recent literature (e.g., [49]), making repetitive baseline testing redundant. Conversely, the Scientific Article pipeline represents the system’s most demanding operational scenario. Processing these documents necessitates multi-stage extraction, heterogeneous model assignments, and a hierarchical Map-Reduce strategy to overcome inherent LLM context window limitations. Therefore, while the extraction pipelines for the remaining administrative document categories are fully operational within the institute’s workflow, their specific quantitative extraction metrics fall outside the scope of this manuscript and are left for future domain-specific studies.

The sub-graphs used for tailored document processing vary in complexity and the requirements for model capabilities and computation resources utilization. While processing Invoices or Plagiarism Detection Reports proved to be relatively straightforward (the documents are hardly longer than a few pages), the most difficult for processing was the Scientific Article. Its processing consists of the extraction of the bibliographic, meta-information (title, authors, journal, DOI, etc.), summarization of its contents, and proposition of a set of keywords. Therefore, the process was split into three parts, each dedicated to a different scope of information (metadata, text summarization, keyword generation). Each part was then processed by dedicated subroutines of the Scientific Paper Processing Agent. The subroutines were abstracted as the LangGraph nodes, each responsible for a single task. In the following subsections, a more detailed description of each node is presented.

The metadata extraction was evaluated on its ability to extract fields from 25 sample articles. The articles came from the most popular scientific resources: arXiv, Elsevier, IEEE, MDPI, SpringerLink. There were five papers selected for the tests from each source. The agent demonstrated high effectiveness in information retrieval, though prompt engineering was necessary for preventing the model from hallucinating titles.

The agent demonstrated high effectiveness in information retrieval. Quantitatively, the extraction of the article title and DOI achieved 96% and 80% accuracy (correct extraction) across the test set, respectively (24/25 and 20/25 documents). The problem with title extraction appeared for a paper with a long, multiline title. The part of the title was truncated, so the system returned incomplete information. In the case of DOI, a few behaviors were observed: lack of prefix (the system returned the numbers only), the truncated DOI and (most common) not returning DOI at all.

The extraction of author lists was correct in 24 out of 25 cases (96%). In one case, the authors list was corrupted—the authors list was empty. The publication date and journal name were correctly extracted in 23/25 cases (92%) for both. Once the journal name was not returned at all, and once the word ‘journal’ was returned as the journal name.

As the metadata information is mostly contained on the first page of the paper, only the first 6000 characters were passed to the model’s prompt for that task. Surprisingly, one of the best-performing models here was llama3.1:8b. Despite its small size, there was no observed performance degradation in comparison to other, bigger models. However, prompt engineering was necessary for preventing the model from hallucinating the titles.

The Scientific Paper content summarization node was tested on the same set of papers. The length of papers varied from 20 to over 30 pages. As the attempt at one-pass summarization led to exceeding the context capabilities for most of the used models, a hierarchical Map-Reduce strategy was implemented. It allowed the successful processing of the entire document without encountering context window limitations. The quality of final summaries (generated by llama3.1:70b-instruct) was assessed by senior researchers using a binary utility scale (Acceptable as is/Requires Editing). The researchers were instructed to cross-reference the generated summaries against the source texts. A summary can be marked ‘Acceptable’ only if it contains zero hallucinations (including fabricated numbers, names, or unsupported claims) and accurately reflects the core findings. Should the assessors find any errors, the paper was given a ‘Needs Editing’ category. The described checking protocol, though straightforward, adheres to the practical use of the paper summarization routine. Each document was evaluated by a panel of three scholars, and the final score was determined by a majority vote. The model produced summaries rated as ‘Acceptable as is’ in 24 out of 25 cases (96%). The remaining one case was flagged primarily for minor errors regarding numerical values presented in the summary.

Inter-rater agreement was evaluated for the three-scholar binary assessment of 25 documents by three raters (scholars). Complete agreement between raters (all three scholars assigning the same label) was observed for 19 of 25 documents, corresponding to a rate of 76%. Agreement beyond chance was quantified using Fleiss’ kappa [50]:

$$\mathsf{\kappa }=\left(\overline{P}-\overline{P_e}\right)/\left(1-\overline{P_e}\right)$$

(1)

where $\overline{P}$ denotes the mean observed agreement and ${\overline{P}}_e$ the expected agreement under independent ratings. The mean observed agreement was $\overline{P}$ = 0.840, while the expected chance agreement was ${\overline{P}}_e$ = 0.789. The resulting Fleiss’ kappa was $\kappa$ = 0.242, which corresponds to fair agreement under the Landis and Koch benchmark range of 0.21–0.40 [51]. To assess dyad-specific consistency, pairwise Cohen’s kappa coefficients [52] were computed for each rater pair: ${\mathsf{\kappa }}_{S_1-S_2}$ = 0.107, ${\mathsf{\kappa }}_{S_1-S_3}$ = 0.645, and ${\mathsf{\kappa }}_{S_2-S_3}$ = 0.169, with a mean pairwise value of 0.307. The pattern indicates heterogeneity in rater concordance, with comparatively high agreement between S₁ and S₃ and lower agreement for dyads involving S₂. The symbols S₁, S₂, and S₃ were used to denote the three scholars.

The identification of keywords was assigned to the graph node utilizing the phi4 model. In the assessment, the model generated an average of five keywords per article. Evaluators confirmed that 100% of these keywords accurately reflected the domain, often identifying relevant terms not explicitly listed by the original authors.

At the end of the metadata extraction/summarization pipeline, all partial pieces of information gathered by specialized nodes were assembled into a well-formed JSON document. The document was then processed and reformatted into Markdown.

3.3. The Heterogeneous Model Assignments

A key feature of the architecture is the ability to assign different LLMs to different tasks. Such an approach allowed for choosing the best-performing model for each task. The encouraging results were particularly well visible in the case of Scientific Article document type processing. Each step of the graph was assigned to a different model, thus yielding optimal results. The system implemented per-node model assignment through the JSONC configuration file. The class factory instantiating the Ollama client wrapper checked the model name and additional parameters, which should be used for a particular processing graph node. The mechanism of gracefully falling back to the default model when the configuration file does not have an entry for the node. Such a solution has two advantages. First, it does not enforce the inclusion of a configuration for every node, which is convenient. Second, and more importantly, it makes the development process easier. The newly designed node or nodes do not need to have the configuration provided immediately. Once the most suitable “all-purpose” model is chosen, it is a good choice as a starting point for new functionality.

3.4. Text Mining—Structured Output Generation

The majority of agents comprising the system produce output data in a structured format. To ensure such behavior, the Pydantic library was used. Each output document, along with the web service API, was strictly defined with the use of the library. Then, when LLM is given a task to do, the appropriate formatting instructions are injected into the prompt. The injection is achieved utilizing the functionality given by the LangChain library. The final step in the processing pipeline is to validate the output, to make sure that no formatting errors were made by the LLM. In the case of the presented system, such an approach appeared to be sufficient. Only in the case of the Scientific Article metadata extraction, difficulties were experienced with the model confusing the parts of metadata or trying to confabulate. Two mitigations for this behavior were considered: rewriting the formatting instructions in a customized way or using the Reflection pattern [53,54] to make the model correct the errors it made upon the error output of the format validator. Finally, the first solution was chosen as simpler and sufficient for the given problem.

4. Discussion

The results suggest that a modular, multi-agent architecture utilizing on-premises LLMs offers a viable approach to automating document processing workflows within a research organization.

4.1. Classification Dilemma—Compromise Between Accuracy and Time Efficiency

The high classification accuracy of the agent validates the proposed AI-driven routing paradigm. This architecture enables scaling, allowing new, specialized processing pipelines to be integrated into the central orchestrator without disrupting existing workflows. During the evaluation of model effectiveness for the document classification task, a critical trade-off emerged between model complexity, inference time, and classification accuracy. To illustrate this, scatter plots were generated comparing each model’s accuracy against its average execution time (Figure 5 and Figure 6), where the diameter of each circle represents the model’s parameter count.

As anticipated, the largest model, cogito:70b, yielded superior accuracy; however, the performance of the mid-sized magistral:24b and qwen3:32b models is particularly noteworthy. Despite possessing less than half the parameters of the 70B baseline, these models exhibited only a marginal degradation in performance. Specifically, magistral:24b demonstrated high robustness on the strict document dataset (which excludes out-of-distribution documents belonging to the “other” class), whereas qwen3:32b excelled on the extended dataset. Crucially, both models reduced inference time by more than 50% compared to cogito:70b. Given this highly favorable trade-off between a minimal accuracy reduction and significant computational efficiency, selecting either magistral:24b or qwen3:32b as the foundational model for the classification task is empirically justified. Conversely, the qwen2.5:72b model performed suboptimally; despite exceeding cogito:70b in parameter size, it recorded one of the lowest overall accuracies. Finally, as expected, compact models such as llama3.1:8b prioritized inference speed at the direct expense of classification accuracy.

Based on the empirical results, the following recommendations can be formulated regarding model selection for practical deployment. If computational resources are unconstrained and maximizing classification accuracy is the primary objective, the cogito:70b model remains the optimal choice. Conversely, if inference speed is prioritized over marginal accuracy gains, or if hardware limitations impose a strict bottleneck, both the magistral:24b and qwen3:32b models present highly viable alternatives. The final selection between these two mid-sized models should be dictated by the expected statistical distribution of the input data. Specifically, if the input stream consists predominantly of the predefined document categories, magistral:24b is recommended. However, if the system must process a substantial volume of documents falling outside the defined taxonomy (out-of-distribution inputs), qwen3:32b demonstrates superior robustness and is therefore the preferable option.

Because document classification is a critical node in the document processing workflow, analyzing the most frequent classification errors is essential. Figure 7 and Figure 8 present the confusion matrices for qwen3:32b and gemma3:27b. The latter was selected to represent an average-performing model, as it ranks in the middle of the evaluated cohort. An analysis of the confusion matrices reveals that both models exhibit a shared vulnerability: a high failure rate in classifying the ISO Procedure Document. Interestingly, both models systematically misclassify this category as a Tender Specification Document. The models demonstrate significantly higher accuracy across the remaining, well-defined document types. The only other highly problematic class is the “Other” category. While qwen3:32b predominantly conflates “Other” with Scientific Article, gemma3:27b exhibits a highly dispersed error distribution across multiple classes for this category.

To investigate the root cause of this erroneous behavior, the classification node was modified to implement a self-explanation mechanism, requiring the model to output a reasoned justification alongside its classification. The investigation focused on qwen3:32b model solely. When prompted to articulate their reasoning, the responses from the model revealed a consistent bias. The model correctly identified the initial characteristics of the ISO Procedure Document; however, because the text was highly structured and contained rigid requirements and specifications, it ultimately mapped these features to a Tender Specification Document, outputting the latter as the result. A potential mitigation strategy for this error involves refining the prompt to force the model to anchor its final output on its initial structural observations rather than secondary semantic features. However, while explicitly providing negative constraints in the prompt (e.g., instructing the model specifically not to confuse ISO and Tender documents) might resolve this isolated issue, such an approach should be avoided, as it risks reducing the agent’s overall generalization robustness.

The problem of text classification using LLMs is also addressed in other research. In [55], the authors propose a novel mechanism for text classification called RGPT, which uses a boosting-like approach to enhance the model’s capabilities. Their presented results are exceptional, achieving an accuracy as high as 98.6% on the SST-2 [56] dataset. Interestingly, these results were achieved using Llama 2 as the base model, demonstrating that simpler, smaller models can be sufficient for achieving outstanding outcomes. This conclusion aligns with the findings presented in our study; our model, despite its relatively small size, was able to achieve results that were both high-quality and sufficient for practical application.

However, a direct comparison between the results in our own study is challenging. Although the performance levels are comparable, the problem domains and the nature of the input texts differ significantly. The cited work focuses on identifying the topic (AGNews dataset [57]) or sentiment (SST-2 and MR [58] datasets) of relatively short texts. In contrast, our objective was to classify the category of entire documents. Our input dataset, comprising real documents processed within our institution, often contains a weakly defined structure. This structural element, which can provide additional classification clues to the model, makes our task substantially different from topic or sentiment classification.

The state-of-the-art in text classification was also reviewed in [59]. The authors analyzed various approaches for building text classifiers, including zero-shot prompting, few-shot fine-tuning, and the use of synthetic, LLM-generated data for fine-tuning. They also investigated the influence of language on classification performance. In their conclusion, the authors state that zero-shot prompting performs reasonably for tasks with few classes but that its performance degrades for more complex problems. Our work utilizes a zero-shot prompting approach, and since our task involves only six document categories, this method yielded very good results. Again, our task differed substantially from those analyzed in [59], so a direct comparison of the results is difficult. Nonetheless, for the specific case of classifying a document’s type, zero-shot prompting proved to be an effective approach. This is because document type classification is supported not only by textual content but also by the document’s structure, even if that structure is only weakly defined. This leads to a concluding remark: our results do not undermine the findings of [59]; rather, they help to delimit the scope of their skepticism. While they proved that zero-shot prompting can be fragile for nuanced tasks, our work shows it can still be robust for other applications, such as document type analysis, which is more reliant on structure.

A key conclusion can be formulated from the cited papers and our own results: moderate-sized models (e.g., ~70b parameters) are often sufficient for complex tasks, obviating the need to use much larger ones. This finding has significant consequences for the resources required for model hosting, as well as for time and energy consumption. For practical applications, we demonstrated that a single, well-chosen open-weight model (magistral:24b or qwen3:32b, depending on the expected input data characteristics in our case) managed by an agentic framework (LangGraph) achieved good results (96.7% and 91.7% respectively) for document routing without any fine-tuning. We have shown how paradigms such as zero-shot prompting can perform effectively when adapted to a constrained, real-world research and development (R&D) environment.

4.2. Comparative Perspective—The Challenge of Assessment Without Standards

Assessing the performance of the proposed system requires placing it within the context of current research. However, the task reveals the lack of suitable benchmarks, suitable for the realities of industrial automation. The most well-known dataset, RVL-CDIP [60] (often treated as a standard for document classification), suffers from high label noise. The literature reports that the error level is as high as 8% [61,62]. Using such a dataset makes it impossible to reliably assess the quality of a classifier of expected accuracy at 97–99%. Moreover, the dataset consists of images, so it cannot be directly used with the proposed solution (we focus on text documents solely, either in txt, PDF, or Microsoft Word format). Another problem is caused by the fact that the database consists of just one-page documents, which hardly represent real-world documents (entire files, not isolated single pages) [62]. It seems that the proposed solution partially fills the gap identified by authors in [50,62], handling and routing multi-page inputs.

While there are available high-quality data for scientific domains, there is a scarcity of public, diverse datasets for financial and administrative documents, forcing researchers to create small, private datasets [49,63]. The same approach was chosen in the case of the presented system, when the dataset was composed of a sample of documents processed within the organization.

It seems direct comparison of our solution to other, analogous research is challenging. However, the efficiency, in terms of accuracy, can still be compared, having in mind the limitations of such a comparison. As an example of such a comparison, the results in financial documents [49] can be considered. The authors report the accuracy as high as 99.24% for invoices. Moreover, the result was achieved with minimal computational cost. Achieving such a good result required a large, annotated training set. On the other hand, the agentic LLM approach utilized in this study was capable of achieving similar accuracy using zero-shot reasoning. No prior training was necessary. A similar approach was applied by Park et al. [64]. The transformer-based models struggled to process extensive texts without truncation. The authors report achieving an accuracy at 95.69% and point out that the most difficulties were caused by the relatively small context window of the used models (512 tokens). This difficulty appears to be resolved by the sheer scale of modern LLMs used in our work. Xia et al. present the results of the classification of multi-page, scientific documents, yielding accuracy as high as 98.21%. The research used the DocGenome database—a proposed set of scientific documents [63]. The best results were obtained with the use of ChatGPT 4 V commercial model. It has to be noted that the work focused on scientific documents only—no other types of documents (particularly business) were considered.

The proposed system operates in a space where the lack of a “Business DocGenome” forces reliance on proprietary validation. Consequently, the success of this system serves as a practical counter-argument to current academic constraints: it demonstrates that in the absence of perfect standards, generalist open-weight models orchestrated by agentic logic can outperform specialized architectures on complex, real-world tasks, provided one accepts the higher inference costs.

Summing up, it can be stated that achieving the accuracy in the document classification task in the range of 94–99% is currently viable. However, most of the literature sources focus on a particular task and rarely use a variety of business-related single- and multi-page documents. It can be observed that when the classification task is narrow enough (in terms of generality of documents), a simple yet very effective model can be used with success [49]. The more general tasks require larger models [63], which is also supported by our research.

Table 7. Comparison of the results reported in the documents’ classification tasks.

Research Reference	Accuracy	Method Used
Vajjala et al. [59]	89.0%	Supervised Fine-Tuning (BERT/RoBERTa)
Zhang et al. [55]	97.6%	RGPT-7 (Ensemble + Boosting framework using fine-tuned LLaMA-2-7B)
Xia et al. [63]	98.21%	GPT-4V, proprietary multimodal LLM, Zero-shot
Serbanescu et al. [49]	99.24%	MobileNetV2 (Pre-trained with data augmentation, Supervised)
Park et al. [64]	95.69%	Longformer (Sparse-Attention Transformer, Supervised)
Van Landeghem et al. [62]	94.45%	DiT-Base (Aggregated predictions from first/second/last pages)
Ours	97.3%	Agentic Workflow/Cogito-70B (Zero-shot Open-Weight LLM)

presents the comparison of the best accuracy achieved in current research (including ours).

The presented study has demonstrated that the convergence of large language models (LLMs) and agentic workflows represents an efficient approach in processing administrative and business documents. While other approaches [49,64] offer computational efficiency for specific, well-defined tasks, they need training with extensive annotated datasets and suffer from limited context windows. Our implementation proves that these constraints can be effectively bypassed through the use of LLMs (such as Cogito-70B) orchestrated by a logic-driven router. The system presented in this case study achieves classification accuracy almost matching the performance of proprietary state-of-the-art models like GPT-4V [63], but without the potential privacy risks or costs.

While the accuracy results presented in Table 7 demonstrate the viability of the system, there are also other aspects of the solution worth consideration. One of the most important is architectural trade-offs. Unlike conventional supervised methods demanding extensive datasets [49,64] or proprietary cloud solutions introducing data privacy considerations [63], the proposed approach offers a middle ground.

Table 8. Comparative analysis of the proposed agentic solution, proprietary LLM solutions, and conventional document processing methods.

Comparison Criteria	On-Prem Agentic LLM (Proposed)	Proprietary Cloud LLMs (e.g., GPT-4V [52])	Conventional Approach (Supervised ML/CNN [53])
Data Privacy & Security	High (Data never leaves on-premise infrastructure)	Low (Data processed via external API)	High (Local inference; no external data transfer)
Adaptability & Flexibility	High (Adaptable via prompt engineering)	High (Adaptable via prompt engineering)	Low (Requires retraining or new templates for new document types)
Training Data Requirement	None/Minimal (Relies on pre-trained reasoning and context)	None (Pre-trained)	High (Requires extensive annotated datasets for high accuracy)
Complex Logic & Reasoning	Medium (Agentic graph enables multi-step decision making & routing)	Medium/High (Dependent prompting strategy)	Low/None (Limited to pattern recognition; no semantic reasoning)
Operational Efficiency	Balanced (Moderate to high hardware cost)	Variable (High operational costs at scale)	High (Very fast inference; low compute cost; best for static layouts)

provides a qualitative comparison of these paradigms, highlighting how the proposed on-premise agentic system differs from other mentioned approaches in terms of flexibility and security essential for sensitive research environments. There appears to be no universal solution. However, the best solution can be chosen for the given requirements and limitations.

To conclude the discussion, we draw attention to the possibility of using traditional machine learning and classifiers in document mining. Though the methods are computationally efficient, they were deliberately not selected for this system due to the realities of the target operational environment. First, the processed documents exhibit strong structural variability, originating from diverse external organizations. In this context, rule-based approaches (such as regular expressions) are too brittle, requiring constant manual tuning and updates to handle unforeseen templates. More importantly, training traditional supervised machine learning models (e.g., Support Vector Machines or fine-tuned neural networks) requires massive, comprehensively annotated datasets to generalize effectively. While our manually verified dataset is sufficient for testing, it is prohibitively small for training robust classifiers from scratch.

To circumvent these constraints, the proposed system utilizes an LLM-based agentic architecture that relies on zero-shot reasoning. This paradigm entirely bypasses the data-intensive training phase. Crucially, it provides immense operational flexibility: integrating a new document class requires only (usually minor) adjustments to the system’s text prompts, eliminating the arduous cycle of retraining and redeploying dedicated models. Consequently, the adoption of an agentic LLM approach represents a deliberate architectural trade-off. The system sacrifices the computational frugality of traditional algorithms—accepting higher GPU resource consumption—in exchange for zero-shot agility and robust performance on unpredictable data. It seems that in a dynamic R&D setting, overcoming the bottlenecks of manual data labeling and constant rule maintenance fully justifies this computational investment.

4.3. Data Extraction and Summarization—The Fight for Relevant Context Content

The implementation of the Scientific Paper pipeline highlights two key findings. First, the three-pass extraction strategy is an effective method for handling documents with both semi-structured metadata at the beginning and unstructured content throughout. Second, the hierarchical Map-Reduce chain proved its robustness as a solution for overcoming the inherent context window limitations of LLMs. Using this approach enabled the processing of long-form documents that would be intractable with a naive, single-pass approach. Moreover, we observed that a limited overlap (just a few dozen characters) between the chunks into which the document is split helps in achieving coherent summarization during the Reduce phase. Finally, splitting the process into three parts allowed for the fine-grained selection of models for each stage, optimizing the trade-off between effectiveness and accuracy.

4.4. Heterogeneous Models Usage—The Lessons Learned

A key finding of this work is the effective application of heterogeneous model selection for the given task. Gains in flexibility and effectiveness are made possible by architecting a system where model selection is treated as a configurable parameter. This “right-tool-for-the-job” philosophy proves effective for building practical and cost-effective AI systems, allowing computational resources to be focused only on the tasks that truly require them. However, this approach is not without its challenges. While pairing the appropriate model to a given task is a sound strategy, it comes with certain costs. Ideally, this solution works best when all models can be instantiated on the hardware at system startup. If models must be switched dynamically, the associated latency can be prohibitive for practical applications. During a switch, the old model must release its hardware resources before the new one can be loaded and configured, a process that takes a noticeable amount of time (as long as ~30 s in our case). In practice, two mitigation strategies can be considered: either batching all operations that use a certain model to complete before switching, or selecting a single, universal model suitable for all purposes, thus eliminating the need for switching entirely. Beyond operational latency, hardware requirements represent another significant limitation. For instance, achieving the reported 97.3% accuracy relied on a 70B parameter model. Hosting such a model requires high-performance GPU servers with sufficient VRAM capacity. This requirement introduces a trade-off between initial infrastructure investment and the system’s operational capability. Another significant limitation, which presents a second key area for future work, is the difficulty in assessing model uncertainty. Unlike traditional probabilistic classifiers, LLM-based agents do not inherently output a quantifiable uncertainty metric. This makes it challenging to automatically assess the quality of a response based solely on the model’s output. Developing a reliable uncertainty measure would be of great practical importance, as it could help direct human attention to documents that are most likely to have been processed incorrectly.

While the system showed promising performance, the study also highlighted key areas for future development. A primary area for improvement is the introduction of a self-correction or validation agentic loop. Such a pattern is expected to further enhance the reliability and accuracy of the structured data output, thereby reducing the need for human verification.

5. Conclusions

This study presents the design, implementation, and evaluation of a modular, on-premise multi-agent system for automating document processing workflows within a research and development (R&D) organization. The system’s core AI-driven, LLM-based routing was assessed by evaluating its document classification agent under two testing regimes. The first regime measured the system’s ability to correctly classify a predefined dataset of known document types. The second regime introduced a subset of unclassified documents (not belonging to any predefined type) to test the system’s robustness against out-of-distribution data. Under these conditions, the cogito:70b model achieved an accuracy of 97.3% in the first regime and 94.3% in the second.

Furthermore, our analysis revealed a significant trade-off between performance and computational efficiency. The magistral:24b and qwen3:32b models emerged as strong candidates for practical application; they achieved accuracies of 96.7% (first regime) and 91.7% (second regime), respectively, while requiring less than half the processing time of their 70B counterpart. The architectural flexibility of the system was further validated through the successful implementation of complex, multi-stage pipelines—such as the Scientific Article processor—which leverages heterogeneous model assignments for distinct tasks. Consequently, a right-sized, open-weight model (24B–70B range) hosted on-premise offers an optimal balance of accuracy, speed, and privacy, which is a critical requirement for sensitive R&D environments.

By automating time-consuming tasks like document classification and initial analysis, this system demonstrates a viable method for enhancing operational efficiency. Preliminary estimates suggest potential time savings of 3 to 6 h per week per user, allowing scientific staff to dedicate more attention to core research and proposal development. While future iterations of this architecture could assist in identifying complex connections across research fields, the current scope is successfully limited to workflow automation.

While this case study presents an effective, secure, and scalable solution for standardizing information processing, the approach remains constrained by the substantial hardware requirements and inference latency inherent to hosting large open-weight LLMs locally. Finally, because this system was evaluated in a controlled, production-like environment, these efficiency projections remain theoretical. Rigorous A/B testing and deployment in a live operational setting are required to fully validate the system’s efficacy and quantify its real-world impact.