Comparative Evaluation of GPT-4o, GPT-OSS-120B and Llama-3.1-8B-Instruct Language Models in a Reproducible CV-to-JSON Extraction Pipeline

Abstract

Recruitment automation increasingly relies on Large Language Models (LLMs) for extracting structured information from unstructured CVs and job postings. However, production data often arrive as heterogeneous, privacy-sensitive PDFs, limiting reproducibility and compliance. This study introduces a deterministic, GDPR-aligned pipeline that converts recruitment documents into structured, anonymized Markdown and subsequently into validated JSON ready for downstream AI processing. The workflow combines the Docling PDF-to-Markdown converter with a two-pass anonymization protocol and evaluates three LLM back-ends—GPT-4o (Azure, frozen proprietary), GPT-OSS-120B and Llama-3.1-8B-Instruct—using identical prompts and schema constraints under near-zero-temperature decoding. Each model’s output was assessed across 2280 multilingual CVs using two complementary metrics: reference-based completeness and content similarity. The proprietary GPT-4o achieved perfect schema coverage and served as the reproducibility baseline, while the open-weight models reached 73–79% completeness and 59–72% content similarity depending on section complexity. Llama-3.1-8B-Instruct performed strongly on standardized sections such as contact and legal, whereas GPT-OSS-120B better-handled less frequent narrative fields. The results demonstrate that fully deterministic, auditable document extraction is achievable with both proprietary and open LLMs when guided by strong schema validation and anonymization. The proposed pipeline bridges the gap between document ingestion and reliable, bias-aware data preparation for AI-driven recruitment systems.

1. Introduction

Information extraction (IE) from unstructured or semi-structured text remains one of the most active and impactful applications of Large Language Models (LLMs). This study builds upon our earlier work on AI-driven job–candidate matching. In our previous research, we introduced a zero-shot recommendation framework based on pre-trained language models that enabled efficient and scalable candidate ranking without supervised fine-tuning [1]. That approach was subsequently extended in a supervised setting through Siamese architectures combining GRU, LSTM and Transformer sequential heads to capture contextual dependencies across multilingual recruitment documents [2]. The present paper continues this line of research by evaluating Large Language Models (LLMs) in a reproducible information-extraction pipeline designed for structured CV-to-JSON conversion. From healthcare narratives and financial records to legal documents and recruitment data, automated extraction aims to transform heterogeneous text into structured, machine-readable formats that can be reliably used for analytics, matching and downstream decision-making. With the advent of generative foundation models such as GPT, Llama and Mistral, research attention has shifted from rule-based and task-specific pipelines to universal, prompt-driven frameworks capable of schema-aware reasoning and structured data generation [3,4,5,6].

Related Work

A large body of prior research has explored LLM-based information extraction across clinical, legal, financial and recruitment domains. Recent studies demonstrate the high accuracy of LLMs in structured extraction tasks, including echocardiographic entity identification [7], electronic health record parsing [8] and domain-specific reasoning with instruction-tuned architectures [9]. Further work has examined prompt engineering strategies [3,4], parameter-efficient adaptation such as LoRA and ARD-LoRA [10,11], evaluation frameworks for LLM robustness and completeness [12,13] and reproducibility challenges in generative pipelines [14,15]. Privacy-preserving document workflows have also been investigated [16,17], highlighting the need for on-premises, GDPR-aligned extraction methods. These works collectively motivate the need for deterministic, auditable and privacy-compliant extraction pipelines, while revealing gaps that our study directly addresses.

Recent advances demonstrate that LLMs can achieve high accuracy in domain-specific extraction tasks, rivaling or exceeding traditional NLP systems. For instance, Chi et al. [7] showed that lightweight, open-source models could reach near-expert F1-scores (≈0.96) when extracting echocardiographic entities from clinical narratives, while Ntinopoulos et al. [8] benchmarked 18 Large Language Models for structured extraction from electronic health records, finding that modern architectures such as GPT-4 and Llama 3.1 70B surpassed 0.98 accuracy. Similarly, Kim et al. [9] applied GPT-based architectures to clinical free text and demonstrated performance comparable to supervised BioClinicalBERT models, highlighting that general-purpose LLMs can function effectively as information extractors across specialized domains.

A growing body of research explores the optimization of such models through instruction tuning and efficient adaptation. Methods such as Low-Rank Adaptation (LoRA) [10,18], Rotated LoRA (RoLoRA) [19] and Adaptive Rank-Determined LoRA (ARD-LoRA) [11] enable parameter-efficient fine-tuning of LLMs for extraction and classification tasks, significantly reducing training cost while preserving performance. These methods also support multi-task instruction tuning [20], enabling the same model to handle diverse document formats. Uluirmak et al. [21] showed that LoRA fine-tuning of Llama-family models can adapt them efficiently for domain-specific scenarios such as structured CV analysis, while Loukil et al. [22] demonstrated a LoRA-based, LLM-centric pipeline for invoice information extraction—an example of applying parameter-efficient adaptation directly to enterprise document workflows.

Despite this progress, extraction performance strongly depends on the quality of prompt engineering. Polat et al. [3] systematically tested multiple prompting strategies for knowledge extraction and found that few-shot prompting with schema examples consistently outperformed chain-of-thought or unconstrained instructions. Likewise, Hu et al. [4] improved the F1-score of GPT-4 from 0.80 to 0.86 on clinical Named Entity Recognition by incorporating domain-specific guidelines and iterative prompt refinement. Similar findings were reported by Zambrano et al. [23] in legal text extraction, emphasizing that context-rich instructions and zero-shot schema alignment are crucial for achieving consistency. Beri and Srivastava [24] further summarized advanced prompt engineering methods—such as multi-turn clarification, automatic few-shot retrieval and structured output constraints—as key levers for accuracy and reliability in LLM-driven IE pipelines.

LLM-based IE systems are now also subject to rigorous benchmarking and evaluation. Dedicated frameworks such as LangTest [25], RepEval [12] and HiBench [13] aim to measure completeness, robustness and hierarchical reasoning in generated outputs. Anghel et al. [26] proposed a scalable meta-evaluator for diagnosing instability and bias in LLM evaluation, while Ma et al. [27] introduced revision-distance metrics bridging human and automatic judgments. At the same time, Li and Sun [28] developed a fine-grained evaluation framework combining pointwise and pairwise comparisons to assess structured outputs at multiple levels of granularity. Such methodologies complement traditional precision–recall–F1 measures, which, although widely used, often fail to capture schema-level completeness or semantic fidelity—an observation also highlighted in biomedical extraction reviews by Hsu and Roberts [29] and Hein et al. [30].

Reproducibility and transparency are emerging as fundamental requirements for credible LLM research. Multiple studies have exposed variability in model behavior across runs, configurations or floating-point computations [14,31]. Digan et al. [15] investigated seven clinical NLP suites and concluded that reproducibility failures frequently stem from uncontrolled randomness, inconsistent preprocessing and opaque workflows. Sethi and Gil [32] argued for publishing experiments as semantic workflows with machine-readable provenance to ensure transparency and re-use. The same reproducibility concerns extend to generative pipelines, where small differences in decoding temperature or model versioning can lead to divergent outputs [31,33]. Tahmasebi et al. [14] even questioned the reproducibility of “magical” LLM-based recommenders, calling for standardized reporting and deterministic setups. In response, our study aligns with the reproducibility paradigm by fixing model snapshots, prompts and decoding parameters to achieve byte-identical outputs under repeated runs.

Parallel to technical reliability, privacy and compliance have become decisive factors in real-world deployments. Sensitive documents such as medical records, invoices or CVs contain Personally Identifiable Information (PII) that must be protected throughout processing. Studies in the medical domain [7,8,34] consistently emphasize the risks of using cloud-hosted proprietary LLMs for confidential data and advocate for local, open-weight alternatives. Yoon et al. [16] proposed the LOFI (Language–OCR–Form Independent) pipeline for industrial documents, integrating OCR and LLM extraction in a secure on-premises environment, while Shrikaran et al. [17] developed an on-premises ETL pipeline combining EasyOCR and local LLMs to preserve data sovereignty. These works highlight the growing preference for privacy-preserving LLM pipelines, aligning with data-protection frameworks such as GDPR. Our proposed CV-to-JSON system extends this line of work by introducing a deterministic anonymization protocol at the Markdown level, ensuring that no PII leaves the local processing environment while maintaining semantic coherence.

While proprietary models such as GPT-4 or GPT-4o often set performance benchmarks, open-source models are rapidly closing the gap. Comparative analyses [6,35,36,37] show that open-weight models like Llama 3.x, Mistral 7B and DeepSeek perform competitively across multiple IE tasks, with negligible losses in accuracy but major gains in cost, transparency and deployability. For instance, Lehnen et al. [36] reported that Llama 3.1 405B performs comparably to GPT-4 in extracting data from thrombectomy reports, while Elnashar et al. [38] demonstrated efficient structured data generation using GPT-4o under prompt-style optimization. Open-source evaluations are further facilitated by benchmarking initiatives such as MMBench [39], VRDU [40] and VHELM [41], which evaluate the multimodal and document-understanding capacities of LLMs. Yao et al. [42] similarly applied prompt engineering to engineering report extraction, underscoring that structured prompt design can yield domain-agnostic generalization. Collectively, these studies reinforce the strategic importance of open models in regulated domains such as HR and recruitment, where both transparency and privacy are critical.

Despite these advancements, several persistent challenges remain. First, most prior IE pipelines rely on stochastic generation without deterministic control, which undermines auditability and reproducibility. Second, evaluation metrics typically ignore schema completeness and semantic fidelity, limiting their utility for complex structured outputs. Third, anonymization and GDPR compliance are rarely integrated directly into model pipelines, leaving privacy safeguards to external preprocessing. Finally, few works perform controlled, side-by-side comparisons between proprietary and open LLMs using identical prompts, decoding configurations and evaluation schemas. Addressing these gaps is essential for advancing reliable and accountable information extraction at scale.

Therefore, the present study introduces a fully deterministic, privacy-preserving and reproducible pipeline for transforming real-world CVs into validated JSON. Building upon the prior literature in prompt engineering [3,4,24], model reproducibility [14,32] and structured extraction [16,22], our approach integrates document anonymization, schema validation and multi-model comparison within a single unified framework. It evaluates three distinct back-ends—GPT-4o (a frozen proprietary Azure model), GPT-OSS-120B and Llama-3.1-8B-Instruct—using a shared two-message prompt and near-zero-temperature decoding. Performance is measured on 2280 multilingual CVs using two complementary metrics: schema completeness (coverage of all required fields) and content similarity (semantic fidelity to the source document). The results demonstrate that deterministic, auditable extraction can be achieved with both proprietary and open models when guided by strict schema enforcement and anonymization protocols.

In contrast to prior research, which typically evaluates LLM-based extraction in stochastic, non-reproducible settings, our work introduces a fully deterministic and auditable pipeline that guarantees byte-identical outputs across runs. Moreover, unlike existing studies that treat privacy as an external preprocessing step, we integrate an irreversible GDPR-compliant anonymization procedure directly into the model workflow, ensuring that no personal identifiers ever leave the controlled environment. This study is also, to our knowledge, the first to conduct a strictly controlled, side-by-side comparison of proprietary and open-weight LLMs using identical prompts, decoding configurations, schema constraints and evaluation metrics. Finally, our dual-metric evaluation framework—distinguishing schema completeness from semantic content fidelity—offers a more rigorous and fine-grained assessment than traditional precision–recall approaches. Collectively, these elements constitute a methodological innovation that advances the reproducibility, transparency and privacy standards of LLM-driven information extraction.

This study makes several key contributions in the realm of AI-driven recruitment automation and information extraction from CVs:

• We introduce a reproducible CV-to-JSON extraction pipeline that converts heterogeneous recruitment documents into structured, anonymized data while ensuring GDPR compliance.
• We conduct a controlled comparative evaluation of GPT-4o, GPT-OSS-120B and Llama-3.1-8B-Instruct under identical prompt templates and deterministic decoding conditions.
• We propose two novel evaluation metrics—schema completeness and content similarity—that jointly assess structural coverage and semantic fidelity of extracted outputs.
• We incorporate a deterministic anonymization layer operating directly on Markdown-formatted CVs, preventing any leakage of personal identifiers.
• We ensure full auditability and transparency, with fixed model versions, version-controlled prompts and byte-identical re-executions ensuring verifiable reproducibility.

This study bridges existing research gaps by combining methodological rigor, privacy compliance and open benchmarking into a single, end-to-end framework for LLM-based document information extraction. It contributes not only empirical results but also a reproducible blueprint for future studies seeking to operationalize Large Language Models in regulated domains such as recruitment, healthcare and finance.

2. Materials and Methods

2.1. Dataset Description

The dataset used for this study consists of 2280 CVs in total, collected from a diverse pool of candidates. The CVs are in two main languages: English and Polish. The language distribution of the dataset is presented in

Table 1. Language distribution in the CV dataset.

ISO Code	Language	Count	Percent (%)
en	English	2041	89.52
pl	Polish	239	10.48
Total		2280	100.00

.

As shown in the table, the vast majority of the CVs are in English, accounting for 89.52% (2041 CVs) of the total dataset. A smaller proportion, 10.48% (239 CVs), is in Polish. This language distribution is important as it reflects the multilingual nature of recruitment documents, which may impact the performance of language models that rely on linguistic features for extraction tasks.

The dataset composition supports the evaluation of language models in a multilingual setting, with an emphasis on both widely used (English) and less frequent (Polish) languages. This balance allows us to assess how well the models perform across different linguistic environments, providing valuable insights for the development of robust, multilingual CV extraction systems.

This dataset serves as a foundational resource for evaluating the performance of Large Language Models (LLMs) in extracting structured information from CVs. The subsequent sections of this paper will discuss the preprocessing, anonymization and extraction methodologies applied to these CVs, alongside the comparison of model performances across different languages and data structures.

2.2. Conversion PDF to Markdown Pipeline

In this study, we apply the Docling [43] pipeline to transform unstructured PDF documents into structured Markdown representations, which serves as the foundation for further processing and information extraction. The pipeline leverages state-of-the-art AI techniques to ensure that the conversion process maintains the document’s original structure while making it suitable for downstream analysis. This process is crucial for handling the diverse and complex nature of real-world recruitment data, which often arrives in PDF format, a widely used but inherently unstructured format. The Docling system thus plays a key role in ensuring that the extracted data remains accurate, reproducible and compliant with privacy regulations, especially in GDPR-sensitive environments.

The Docling system is a library for automatically converting PDF documents into structured Markdown representations. It applies a multistage, AI driven pipeline for converting PDF documents into Markdown files. Figure 1 illustrates the general workflow and the following section describes each component in greater technical detail. In this expanded version we provide additional information about the underlying models.

In the context of our study, all source PDFs are first converted into Markdown files using the Docling library before any downstream analysis is performed. This ensures that the data used for training and evaluation has been processed consistently and leverages Docling’s capabilities for accurate document parsing and layout analysis.

2.2.1. Layout Analysis Model

The layout detection model in this study is based on the RT-DETR (Realtime Detection Transformer) [44,45] object detector. RT-DETR is an end-to-end model that uses a convolutional core and a transformer-based encoder–decoder to predict a set of objects on a page without the need for a non-maximum suppression (NMS) algorithm. Compared to earlier versions of the DETR model, Baidu introduced a more efficient hybrid encoder that separates intra-scale feature interactions and cross-scale information fusion.

In RT-DETR, features from the last three layers of the CNN core are passed to two modules: the Intra-Scale Feature Interaction (AIFI) module and the Cross-Scale Feature Fusion (CCFF) module. The AIFI module processes features within a single scale, while the CCFF module combines information across different scales, improving multi-scale representation quality with reduced computational resource usage. Additionally, an uncertainty-minimizing query selection mechanism picks high-quality initial queries instead of random ones, improving detection accuracy and convergence speed.

Our model is based on RT-DETRv2, released in mid-2024. RT-DETRv2 retains the same overall architecture but modifies the deformable attention in the decoder, assigning different numbers of sampling points to each feature level, which can also utilize discrete sampling instead of grid sampling. The training strategy includes dynamic data augmentation and scale-dependent hyperparameter tuning, which improves accuracy without compromising inference speed.

The RT-DETR model was trained on a dataset containing approximately 150,000 pages of documents (including the DocLayNet and WordScape datasets) to recognize paragraphs, headings, lists, captions, images and tables. The updated RT-DETRv2 version improves object detection performance, achieving an average accuracy of 53.1% on the COCO benchmark and operating at 108 frames per second (FPS) on a T4 GPU.

2.2.2. Table Structure Recognition

For tables, Docling uses TableFormer, a vision transformer model designed for reconstructing logical table structures from images. TableFormer extends the encoder dual decoder architecture of PubTabNet by replacing LSTM decoders with transformer decoders and by adding an object detection decoder that explicitly predicts cell bounding boxes. This architectural change allows TableFormer to extract cell content directly from programmatic PDFs rather than relying on OCR and significantly improves the Tree Editing Distance Score (TEDS) from 91% to 98.5% on simple tables and from 88.7% to 95% on complex tables. During inference the predicted cell bounding boxes are aligned with the geometric coordinates of the PDF tokens, enabling Docling to avoid re-transcribing cell contents and to produce precise row/column and merged cell structures.

Internally, TableFormer uses a compact Vision Transformer architecture. The CNN backbone extracts feature maps which are flattened into a sequence and passed through a two-layer transformer encoder with an input dimension of 512 and a feed-forward dimension of 1024 using four attention heads. A four-layer transformer decoder then refines these features, using multi-head attention and feed-forward networks. ResNet blocks are inserted before both the structure decoder and the cell bounding box decoder to balance the learning of structural and spatial predictions. During training, TableFormer uses three Adam optimizers for the backbone, structure decoder and bounding box decoder; the learning rate is initially set to 0.001 for 12 epochs and then reduced to 0.0001 for an additional 12 epochs on the PubTabNet dataset. A dropout rate of 0.5 helps prevent overfitting. The model generalizes well across PubTabNet, FinTabNet, TableBank and SynthTabNet, outperforming state-of-the-art baselines on the TEDS metric across simple and complex tables.

2.2.3. Optical Character Recognition (OCR)

Optical Character Recognition (OCR) is a technology used to recognize and extract text from scanned or image-based documents, where traditional text-extraction methods, such as parsing PDF files, cannot be applied. This process is crucial for converting image-based documents into machine-readable formats.

In this study, we used two different OCR back-ends: Tesseract and EasyOCR. Tesseract is a well-known, open-source OCR engine that has been widely used for decades. Since version 4, Tesseract includes a subsystem based on a neural network, specifically designed for recognizing lines of text. This system is based on the Long Short-Term Memory (LSTM) network, a type of artificial neural network that excels at sequential data, like text. The LSTM in Tesseract helps improve the recognition of vertical text and text in multiple languages.

On the other hand, EasyOCR is a newer library based on PyTorch (version 2.9.1), a popular deep learning framework. It supports over 80 languages and is designed to handle both printed and handwritten text. EasyOCR first locates text regions within an image using a model called CRAFT (Character Region Awareness For Text) detection. Then, it uses a Convolutional Recurrent Neural Network (CRNN) to decode the identified text into readable strings.

We primarily use EasyOCR while we fall back on Tesseract for less common scripts or when EasyOCR struggles. It is important to note that OCR is not needed for every document. It is only applied when the source PDF is not extractable by other means, such as when it is a scanned image.

The choice between Tesseract and EasyOCR depends on the document and its layout. EasyOCR is generally faster and more efficient when working with diverse fonts and layouts, especially for documents that may not have standardized formatting. Tesseract, although older, is still useful for certain types of text and is highly customizable.

2.2.4. Reading Order and Post-Processing

After performing layout analysis and recognizing tables, we apply a reading order model to convert the page elements into a logical sequence that mimics human reading behavior. The reading order predictor used in Docling is based on IBM’s document analysis toolkits. This model takes as input the bounding boxes, labels and text snippets for each element on the page. It computes spatial and geometric features such as positions, sizes and alignments and uses a trained model to sort the elements in a logical sequence, typically from top to bottom and left to right. Additionally, it groups items like lists, headers and footnotes.

The model also predicts relationships between different elements. For example, it identifies links between figures and their captions, footnotes and their markers and it merges adjacent text spans that belong to the same paragraph. After establishing the order of elements, a set of post-processing heuristics is applied. These heuristics help to normalize typography, infer heading levels and detect the language of the text. This stage is crucial for generating Markdown that maintains the human reading order, especially in complex multi-column layouts.

Although we treat the reading order predictor as a black-box, it can be considered a lightweight neural network classifier. The classifier operates on features that are derived from the page’s geometry. For each pair of blocks (which could be text, images, tables, etc.), the model calculates features such as vertical and horizontal offsets, overlap ratios, relative sizes, text alignment along the baseline and whether one block lies within the bounding box of another. These features are fed into a multilayer perceptron (MLP), which outputs probabilities for the correct order and relationship labels (e.g., “same paragraph” or “caption of figure”).

The predictions made by the model are used to construct a directed graph, which represents the reading sequence. A topological sort is then applied to produce the final order of the elements. In practice, this learned ordering model significantly outperforms traditional heuristic rules, especially when dealing with complex, multi-column layouts.

2.2.5. Serialization and Extensibility

The final stage of the pipeline involves serializing the structured elements—such as paragraphs, lists, tables, figures and captions—into Markdown format and, optionally, into JSON for programmatic use. This allows for easy interaction with the document data in a machine-readable format. Since each component of the pipeline (layout detector, table recognizer, OCR, reading-order model) is instantiated through a common interface, Docling is designed with flexibility in mind. This modularity enables researchers to swap out components for alternative architectures or to fine-tune models specifically for domain-specific documents.

In our experiments, the updated pipeline processes programmatic PDFs at approximately 1-to-1.3 pages per second on a modern CPU, with significantly higher throughput when using GPUs. Enabling OCR or table structure recognition reduces the processing speed but enhances the system’s robustness. The RT-DETRv2’s modular decoder provides flexibility, allowing us to adjust the number of decoder layers to balance speed and accuracy. Additionally, TableFormer, which is used for table structure recognition, can be replaced with other table recognizers as needed, depending on the specific document characteristics.

For a more detailed overview of the AI models and their respective roles in the Docling pipeline, please refer to

Table 2. Summary of AI models and their roles in the Docling pipeline.

Model/Component	Primary Function
RT-DETR/RT-DETRv2	Transformer-based detector for page layout analysis; identifies paragraphs, figures, captions, tables.
TableFormer	Vision Transformer that predicts logical table structure (rows, columns, merged cells) aligned with geometric tokens.
EasyOCR/Tesseract	OCR back-ends for scanned or image-based pages.
Reading-Order Model	Determines the reading sequence using spatial/geometric relationships.
Post-Processing Heuristics	Caption linkage, heading hierarchy, language detection, text normalization.
Serialization Module	Converts the assembled document to Markdown and JSON.

. This table summarizes the primary functions of each model, providing clarity on their specific tasks in the overall document processing workflow.

2.3. CV Markdown Anonymization Protocol

Before being processed by any language model, each source CV is first converted into Markdown format using the Docling pipeline (Section 2.2) [43]. Next, we apply a deterministic anonymization step that operates exclusively on the Markdown representation. The goal of this step is to remove or replace all Personally Identifiable Information (PII), such as personal names, email addresses, phone numbers, postal addresses and online profile links. In addition, it reduces the presence of quasi-identifiers, such as exact employer names or detailed street-level locations, while preserving the meaningful contextual elements required for later information extraction, such as section headers, job titles, technologies and month–year timelines.

The anonymized Markdown documents produced in this stage are the only inputs used throughout the study; no raw PDF files or images are ever transmitted to the models. The entire process is designed to comply with the General Data Protection Regulation (GDPR, also known as RODO in Poland) and to minimize any risk of re-identification. Importantly, no mapping is retained between the original data and its anonymized replacements, ensuring that the anonymization is both irreversible and privacy-safe.

2.3.1. Detection Strategy

The detection of Personally Identifiable Information (PII) in the anonymization process relies on two complementary methods: high-precision pattern matching and lightweight Named Entity Recognition (NER).

The pattern-based detection identifies elements such as email addresses, phone numbers, web links (Uniform Resource Locators, URLs), international bank account numbers (IBANs) or similar tokens and postal codes. These are recognized using deterministic regular expressions that provide high accuracy and low false-positive rates.

In parallel, the NER component identifies and labels entities belonging to common categories, including PERSON (individual names), ORG (organizations or employers) and GPE/LOC (geopolitical entities or locations). This step helps capture identifiers that cannot be detected by simple text patterns alone.

To minimize false positives, particularly for technical terms and software names, the system uses a curated lexicon of common technologies such as Java, Kotlin, Jenkins, Docker and Selenium. It also recognizes section headers like Skills, Education and Experience to better distinguish between technical terminology and human or organizational entities.

The overall detection process follows a fixed sequence: contact information and numeric identifiers are processed first, followed by names of persons, organizations and locations. This order prevents nested substitutions and ensures that all sensitive information is accurately detected before anonymization.

2.3.2. Replacement Strategy

Table 3. PII categories and transformation policies applied to CV Markdown.

Category	Policy (Deterministic Within a CV)
Personal names (PERSON)	Replace with synthetic, human-readable pseudonyms drawn from a fixed pool; stable per-entity within the CV. Preserve capitalization (e.g., Alex Novak). No mapping retained.
Emails	Replace local-part with a token derived from salted hash; domain forced to a documentation-only domain (example.com); e.g., first.last@example.com.
Phone numbers	Preserve country code if present; replace subscriber number with a non-dialable digit pattern of the same length (e.g., +48 600 000 000). Spacing retained.
Online profiles (LinkedIn, GitHub, personal sites)	Normalize to documentation/example endpoints while keeping labels (e.g., https://example.com/in/<token>, https://github.com/example-user).
Postal addresses	Remove street/number; keep coarse city/country granularity with synthetic city names (e.g., Warsaw, PL).
Employers/Organizations (ORG)	Replace with synthetic but domain-plausible names (e.g., EuroFin Bank, ThermoTech R&D Center, SkyTrip Group); stable within the CV.
Project/product names	Replace with neutral descriptors (e.g., Mobile Banking App, Travel Booking Platform) or synthetic names; maintain role context.
Dates	Keep month–year granularity (MM.YYYY or Month YYYY). Day-of-month and exact durations are not increased in precision.
Government/unique IDs (e.g., PESEL, passport)	Fully redact and replace with fixed-length masks (XXXXXXXXXXXX).
Images/photos	Remove and replace with an HTML comment placeholder <!– image –> (see Listing A3 in Appendix C).
Free-text quotations	No change unless direct identifiers occur; if so, apply the same policies as above.

summarizes the replacement policy. Replacements are deterministic within a document and preserve document coherence. Concretely, we derive an ephemeral per-document salt from the file checksum and use it to drive stable choices (e.g., the same person name is replaced consistently everywhere in a given CV), but we do not re-use salts across documents. This prevents cross-document linkage while maintaining intra-document consistency. Case, punctuation and coarse formatting are preserved (e.g., ALEX NOVAK stays uppercase).

Two-Pass Sanitation and Quality Assurance (QA)

To ensure that no residual Personally Identifiable Information (PII) remains after anonymization, the pipeline performs a second verification pass on the processed Markdown file. In this pass, the PII detector is executed again on the anonymized text. If any remaining identifiers are found, they are replaced or masked using a stricter redaction policy. Only aggregate statistics are logged, such as the number of entities replaced per category (e.g., names, emails, phone numbers), along with the final checksum of the sanitized document. No original text fragments are ever stored.

For additional verification, a small set of test documents containing known identifiers (seeded entities) is used as a control sample. These are checked manually to confirm that all identifiers were correctly anonymized. If any residual identifiers are detected during this process, the anonymization rules are refined and the pipeline is revalidated.

Information such as job titles, technology names, and month–year timelines is preserved because it provides valuable semantic context for downstream extraction tasks and does not constitute direct personal identifiers under GDPR.

The deterministic anonymization workflow applied to each CV is summarized in Algorithm 1.

Algorithm 1 Anonymize-Markdown for One CV

Require:   $m_{in}$: Markdown from Docling;   salt ← SHA256(checksum$\left(m_{in}\right)$) Ensure:   $m_{out}$: GDPR-compliant anonymized Markdown   1:  $S\leftarrow$ segment($m_{in}$)  ▷ Divide into sections: Contact, Skills, Experience, Education, etc.   2:  $E\leftarrow$ detect_pii(S) ▷ Detect emails, phones, URLs, IDs + light NER for PERSON, ORG, GPE   3:  for entity $e\in E$ in order: email → phone → url/id → PERSON/ORG/GPE do   4:        $r\leftarrow$ make_replacement(e, $salt$) ▷ According to Table 3; preserve case and format   5:        $m_{in}\leftarrow$ replace_all($m_{in}$, e, r)   6:  end for   7:  $E^{\prime }\leftarrow$ detect_pii($m_{in}$)   8:  if $E^{\prime }\ne \varnothing$ then   9:        $m_{in}\leftarrow$ harden($m_{in}$, $E^{\prime }$)      ▷ Fallback masking and conservative redaction 10:  end if 11:  return  $m_{out}\leftarrow m_{in}$

Although organization names and city-level mentions are replaced with synthetic counterparts, highly specific achievements or niche combinations of technologies might still be unique in small labor markets. We therefore avoid cross-document linkage (per-document salts) and never store reversible mappings. The anonymized sample used for illustration (Listing A3 in Appendix C) follows exactly the policy above.

2.3.3. Impact of De-Identification on Extraction Accuracy and Inference Efficiency

The de-identification step was designed to preserve all information necessary for downstream extraction while irreversibly removing personal identifiers. Its impact on model performance was evaluated along two dimensions: (a) extraction quality and (b) inference efficiency.

Extraction Quality

The replacement strategy preserves semantic anchors such as section headers, job titles, employment timelines, technology names and skill descriptors. Because Large Language Models rely primarily on these structural and lexical cues rather than on specific proper names, the removal of identifiers does not degrade the ability to populate the JSON schema. Empirically, anonymized CVs did not exhibit lower completeness or similarity scores relative to a non-anonymized baseline on an internal test subset. Cases where model outputs were incomplete were traced to model-specific behavior (e.g., omission of the source section) rather than to the de-identification stage. By ensuring stable formatting, consistent pseudonyms and preserved chronology, the anonymization process acts as a normalization layer that can, in some instances, reduce noise present in raw CVs.

Inference Efficiency

The anonymization step operates entirely before model inference and does not increase computational load during decoding. The transformations are deterministic, pattern-based and linear in the document length. No additional tokens are introduced beyond short placeholder substitutions (e.g., example.com, synthetic organization names). As a result, the token count seen by the model remains effectively unchanged, and inference runtime is unaffected. Batch-level measurements on 2280 CVs confirmed that inference throughput was statistically identical before and after anonymization, with variations attributable solely to model variability and system-level scheduling, not to preprocessing.

Overall, the de-identification procedure provides GDPR compliance and improves input consistency without negatively affecting extraction accuracy or inference latency. It behaves as a controlled normalization layer rather than a source of perturbation in the pipeline.

2.4. Anonymized Markdown Sample

To illustrate the input format used in our experiments, Listing A3 shows a fully anonymized and GDPR-compliant synthetic CV in Markdown. All names, organizations and links are fictional and provided solely for methodological demonstration.The complete anonymized Markdown CV example is provided in Appendix C.

2.5. Methods: Language-Model Back-Ends

This study applies three Large Language Model (LLM) back-ends for structured information extraction from the Markdown produced by the Docling pipeline (Section 2.2). The models differ in licensing and serving mode: a proprietary Azure-hosted model (GPT-4o) that was frozen for the entire study to mitigate data leakage and two open-source models (GPT-OSS-120B and Llama-3.1-8B-Instruct).

2.5.1. Shared Prompt Template and System Instructions

All three back-ends were driven by the same two-message chat template to ensure comparability across models: (i) a system instruction fixing the model’s role as a deterministic CV-to-JSON extractor and (ii) a user prompt template (stored as Prompt.txt) that injects the CV Markdown and enumerates the target JSON schema and filling rules. The very same strings and decoding settings were used for every back-end.

Determinism and Guardrails

Decoding was configured for near-deterministic extraction: temperature was set to 0.0 for all runs (single-turn exchange; system+user). The prompt forbids hallucination, mandates full key coverage, requires type-appropriate empty values for missing data (empty string/array/object or null) and instructs the model to return only a single JSON object. Downstream validation performs JSON parsing and schema checks with a lightweight repair/retry policy (see “Calling policy and output validation” in Section 2.5).

The deterministic extraction behavior of all evaluated models was governed by a fixed system instruction, presented in Listing A1. The complete system instruction is available in Appendix A.

This prompt defined the model’s role, enforced strict schema compliance and prohibited any hallucinated or extraneous output.

Listing A2 presents the unified user prompt template that was passed to each model during evaluation. It contains the Markdown-formatted CV placeholder, the complete JSON schema to be filled, and explicit rules ensuring structural completeness, normalization, and deterministic extraction behavior across all back-ends. The full version of the prompt template can be found in Appendix B.

Each extraction sent exactly two messages (system+user) with the CV injected at {{CV_MARKDOWN}}. No images or raw PDFs were transmitted. The same maximum output length and stop constraints were enforced for all back-ends; no fine-tuning, memory or document-level context was used. The authoritative template is archived as Prompt.txt.

2.5.2. GPT-4o (Azure)

All information extraction experiments were performed using a dedicated Azure OpenAI deployment referred to internally as GPT-4o. The endpoint was fixed to the model snapshot GPT-4o-2024-11-20 and remained frozen throughout the study to ensure reproducibility. It was hosted within Azure AI Foundry as a private resource with access restricted to a virtual network through a Private Endpoint. Role-based access control was applied with the principle of least privilege, and server-side logging of prompts and completions was disabled. The endpoint operated exclusively in inference mode with no fine-tuning, memory or adapter layers. All transmitted inputs were anonymized Markdown documents generated by the Docling pipeline, never raw PDFs or images.

The system used a two-message conversational pattern consisting of a fixed system instruction and a user prompt containing the anonymized CV in Markdown form together with the target JSON schema. Deterministic decoding was strictly enforced by setting the temperature to zero and disabling sampling, nucleus sampling and function-calling features. The Azure JSON mode was intentionally not used; output correctness relied instead on the strong constraints embedded in the system and user prompts, followed by strict post-generation validation. Each query contained only the text required for extraction, without auxiliary metadata, ensuring minimal data exposure.

Input documents varied widely in length and complexity, sometimes approaching the context window limits of the GPT-4o model. To mitigate truncation and cost overhead, preprocessing heuristics were applied to remove decorative Markdown markers, duplicate or empty sections and placeholder images. Long CVs were optionally split into semantic segments such as Work Experience, Education and Skills, each processed with the same prompt configuration and later merged during validation. This segmentation strategy proved more reliable than truncating entire documents and provided consistent performance across thousands of samples.

All files were processed in batch mode using sequential or lightly parallel execution depending on Azure service quotas. Each request was followed by a short delay of several seconds to avoid rate-limit errors, and transient failures such as HTTP 429 or 5xx responses triggered exponential backoff re-tries with limited repetition. Each batch run was instrumented with timers, producing logs containing file names, full paths, processing status, diagnostic messages, timestamps and checksums of generated JSON outputs. The audit log was stored in structured tabular form to facilitate filtering and further analysis.

The post-processing layer verified that the response consisted of a single valid JSON object. If additional text, such as explanations or code fences, was detected then the parser isolated the longest brace-delimited substring, removed formatting markers, normalized whitespace and eliminated redundant trailing commas before parsing again. The parsed structure was then validated against the predefined schema to ensure key completeness, type correctness and domain compliance for controlled fields such as CEFR (Common European Framework of Reference for Languages) levels or employment types. Missing values were replaced by type-appropriate empty entities and successful outputs were canonicalized and formatted for readability. When validation failed, the raw model response was archived under a dedicated filename for later inspection, while the processing record was marked as failed but retained for transparency.

The entire workflow was designed to maintain observability and traceability. Every processed file generated immutable artifacts: the canonical JSON output and a corresponding log entry. Because the pipeline operated deterministically and the model snapshot was fixed, re-running the same batch under identical configuration produced byte-identical outputs. This reproducibility was reinforced by version control of the prompt template, the validation schema and the anonymization scripts. Processing throughput on a standard CPU machine averaged roughly one to two pages per second for programmatic Markdown inputs, with significantly higher speeds when executed on GPU-enabled environments.

Operationally, the most common failure cases included truncated outputs, partial JSON responses containing commentary and transient service throttling. These were mitigated through defensive parsing, conservative retry strategies and comprehensive result logging. No silent skipping of samples occurred—every input file produced either a validated JSON or a logged failure entry. The combination of a frozen model snapshot, deterministic prompt design, strict validation and auditable batch control made GPT-4o a stable reference baseline for all the experiments reported in this study.

2.5.3. Compute Environment for Open-Weight Models (Llama-3.1-8B-Instruct and GPT-OSS-120B)

All experiments with the two open-weight back-ends—Llama-3.1-8B-Instruct and GPT-OSS-120B—were executed on a single multi-GPU node equipped with eight NVIDIA A100 accelerators.

Table 4. GPU node specification used for open-weight model inference.

Component	Value
GPUs (count × model)	$8\times$ NVIDIA A100-SXM4-40GB
Per-GPU framebuffer (reported)	40,960 MiB (=40 GiB )
Total framebuffer across 8 GPUs	$8\times$ 40,960 MiB $=327,680$ MiB (=320 GiB)
Driver version	570.195.03
CUDA toolkit/runtime	12.8
Persistence mode	On (all GPUs)

summarizes the key hardware and driver/runtime characteristics captured immediately prior to batch inference. Except where noted in Section 2.5.4, both back-ends were served in inference-only mode under the same guardrails (system+user prompts, temperature $=0.0$, disabled nucleus sampling) and with the same schema validation pipeline.

As detailed in Section 2.5.4, the 120B-parameter open-weight model was executed with tensor parallelism across the eight A100 devices, with paged key–value caches pre-allocated to avoid fragmentation. The 8B open-weight model was served under the same calling interface and guardrails, sharing the node and driver/runtime stack above. No MIG partitioning was used; persistence mode remained enabled throughout the runs to stabilize device initialization latency. These choices, together with the frozen decoding configuration and schema validation described earlier, ensure that re-running the same batches on this node yields byte-identical outputs.

2.5.4. GPT-OSS-120B (Open Source)

GPT-OSS-120B denotes an open-weights, ≈120B-parameter, decoder-only Transformer used in this study as a high-capacity open-source baseline. The model was served in inference-only mode on on-premise GPUs with tensor parallelism (TP) and a paged key–value (KV) cache allocator. As described in Section 2.5.1 and the common calling/validation policy in Section 2.5, the exact same two-message prompt, schema, stop constraints and near-deterministic decoding were applied to all back-ends; no fine-tuning, memory or tool-calling features were enabled.

As a glossary for the runtime and serving acronyms used in this subsection, see

Table 5. Acronyms and their meaning in the context of the GPT-OSS-120B deployment.

Acronym	Meaning/Role
TP	Tensor Parallelism: shard large weight tensors of a layer across GPUs; all GPUs execute the same layer in lockstep.
DP	Data Parallelism: replicate the whole model on multiple workers and split the batch; used only for throughput, not within a single decode step.
PP	Pipeline Parallelism: place consecutive layer blocks on different devices; increases inter-stage latency; not used in our default runs.
SP	Sequence Parallelism: split the sequence dimension to reduce activation memory; optional, not required for our batch sizes.
KV cache	Key/Value cache: per-layer per-token tensors retained to enable $O\left(1\right)$ incremental decoding.
RoPE	Rotary Position Embedding: relative position encoding commonly used in modern decoder-only LLMs.
MQA/GQA	Multi-/Grouped-Query Attention: share K/V projections across attention heads to reduce KV memory; support is runtime-dependent.
PagedAttention	Paged KV allocator: manages KV cache in fixed-size pages to limit fragmentation under long contexts.
EOS	End-of-Sequence token used to terminate generation.
VRAM	GPU device memory (a.k.a. framebuffer memory) holding weights and caches.
INTk	k-bit integer quantization of weights (e.g., INT8, INT4); activations remained FP16 in our setup.

.

For comparability and auditability, decoding was configured for near-determinism: temperature $=0.0$, top-$k=1$, nucleus sampling disabled, single pass (no beam/speculative decoding) and streaming off. We enforced a single JSON object as the only output, followed by strict JSON parsing and schema validation; on parse failure a minimalist, local repair step was applied without re-generation (see common policy in Section 2.5).

A 120B-parameter dense model requires the following weights memory, ignoring caches and runtime buffers. We report both decimal gigabytes (GB, ${10}^9$ bytes) and binary gibibytes (GiB, $2^{30}$ bytes).

Table 6. Weights memory for a 120B-parameter model and per-GPU share with tensor parallelism (TP) over 8 GPUs.

Precision	Bytes/Param	Total Weights	Per-GPU (TP = 8)
FP16 (half)	2	240 GB $\approx 223.5$ GiB	$27.94$ GiB
INT8 (weight-only)	1	120 GB $\approx 111.8$ GiB	$13.97$ GiB
INT4 (weight-only)	0.5	60 GB $\approx 55.9$ GiB	$6.99$ GiB

summarizes the theoretical memory footprint of the model weights for a dense 120B-parameter Transformer under different quantization precisions. The table also reports the per-GPU memory share assuming tensor parallelism (TP) over eight GPUs, which corresponds to the configuration used in our inference experiments.

During autoregressive decoding, the per-token KV-cache footprint across all layers can be estimated as shown in Equation (1):

$$\mathrm{KV}\mathrm{bytes}\mathrm{per}\mathrm{token}\approx 2\times L\times H\times b,$$

(1)

where L denotes the number of layers, H the hidden size ($=\mathrm{heads}\times \text{head\_dim}$) and b the number of bytes per element in the KV cache (e.g., $b=2$ for FP16 precision).

Two illustrative dense-LLM regimes are as follows:

• $L=80$, $H=8192$, $b=2$: $2\times 80\times 8192\times 2=2,621,440$ bytes $\approx 2.5$ MiB per token;
• $L=96$, $H=12,288$, $b=2$: $4,718,592$ bytes $\approx 4.5$ MiB per token.

Paged allocators (PagedAttention) store the cache in fixed-size pages to reduce fragmentation under long contexts and fluctuating batch sizes. If supported by the runtime, INT8/INT4 KV caches can halve or quarter these figures at some accuracy cost; our main runs kept FP16 KV for stability.

Algorithm 2 describes the deterministic extraction workflow implemented for the GPT-OSS-120B back-end. The procedure guarantees reproducible and schema-compliant CV-to-JSON conversion by applying fixed decoding parameters, tensor-parallel execution, and canonical post-validation of the generated output [14,32].

Algorithm 2 Deterministic CV-to-JSON extraction with GPT-OSS-120B

Require:  anonymized Markdown m, prompt template p, schema s Ensure:  single JSON object j conforming to s   1:  $x\leftarrow$ tokenize(p with m); shard model across n GPUs via TP; pre-allocate paged KV   2:  set decoding: temperature $=0$, top-$k=1$, nucleus off; disable speculative/beam decoding   3:  $\widehat{y}\leftarrow$ generate(x)                     ▷ single pass; no streaming   4:  $j\leftarrow$ longest_json_substring($\widehat{y}$); if parse(j) fails or $j⊭s$ then $j\leftarrow$ repair_and_retry($\widehat{y},s$)   5:  return canonicalize(j)

2.5.5. Llama-3.1-8B-Instruct (Open Source)

Llama-3.1-8B-Instruct is an open-weights model with approximately 8 billion trainable parameters. It is instruction-tuned, which means it has been further adapted to follow natural-language instructions. Because the weights are open, the model can be deployed on-premises (inside a company network), which is attractive for privacy-sensitive workflows.

We ran Llama-3.1-8B-Instruct locally on the same GPU node described in Section 2.5.3. No tensor parallelism was required for this model class. The calling interface, guardrails and decoding were identical to the other back-ends: two messages (a fixed system instruction plus a unified user prompts; Section 2.5.1), temperature set to $0.0$ (deterministic generation), nucleus/top-p sampling disabled and a strict, schema-based JSON validator after generation (Section 2.5).

Thanks to its size, Llama-3.1-8B-Instruct can be served on a single modern Graphics Processing Unit (GPU) with 16–24 GiB of device memory video RAM). In our experiments, it shared the multi-GPU node but did not require model sharding. This makes it cost-effective for batch CV processing in private environments.

In the CV→JSON extractor, Llama-3.1-8B-Instruct was strong on standardized, low-ambiguity sections such as contact, legal and preferences, and it handled languages well (e.g., mapping language levels to the Common European Framework of Reference for Languages (CEFR) scale). These are sections with stable headings and predictable micro-structure, which the model picks up reliably.

The model can under-fill or omit long-tail and nested sections (for example, analytics, memberships, some projects) and may output fewer items than expected in narrative lists. In our runs we observed a systematic tendency to skip the source section unless it was strongly emphasized in the prompt.

With a pinned model build, temperature $=0$ and frozen prompts (Section 2.5.1), repeated runs produce byte-identical JSON after validation (Section 2.6). Because inputs are anonymized Markdown (Section 2.3) and inference is local, no Personally Identifiable Information leaves the processing environment, supporting compliance with the General Data Protection Regulation (GDPR).

2.5.6. Comparative Summary of the Three LLM Models

This subsection gathers, in one place, the high-level differences among the three language-model back-ends evaluated in our pipeline: the frozen proprietary baseline (GPT-4o), the large open-weights model (GPT-OSS-120B) and the compact open-weights model (Llama-3.1-8B-Instruct). The goal is practical: choose the right engine for a given privacy budget, accuracy target and deployment setting, while keeping the language simple and the terminology explicit.

Before discussing model-specific findings, it is useful to summarize their most important characteristics side-by-side.

Table 7. Key Findings Comparison of the Three Evaluated Language Models [4,5,46,47,48,49,50,51].

Aspect	GPT-4o	GPT-OSS-120B	Llama-3.1-8B-Instruct
Architecture	Proprietary, large, autoregressive transformer	Open-source, large, transformer	Open-source, smaller, transformer
Accuracy (CV-to-JSON extraction)	Highest, robust	High, improvable with fine-tuning	Competitive with prompt engineering
Reproducibility	Challenging (closed)	Moderate (open, containerizable)	High (open, local deployment)
Fine-tuning	Flexible, effective	Effective with LoRA/QLoRA	Highly effective, especially with LoRA
Prompt Engineering Impact	Moderate	Moderate	High (critical for performance)
Cost and Deployment	Expensive, cloud	Moderate, local/cloud	Low, local possible
Tokenization Bias	Present, mitigated by scale	Present, more pronounced	Present, can be mitigated with adaptation
Best Use Case	Highest accuracy, less privacy concern	Customizable, large-scale, privacy-aware	Cost-effective, privacy-critical, adaptable

provides a concise comparative overview of the three evaluated Large Language Models—GPT-4o, GPT-OSS-120B, and Llama-3.1-8B-Instruct. The table highlights key aspects such as model architecture, accuracy, reproducibility, cost and practical deployment considerations. This summary helps contextualize the quantitative results presented later and clarifies the trade-offs between closed-source and open-source approaches in our reproducible CV-to-JSON extraction pipeline.

If you need the best possible accuracy and are comfortable with cloud processing then the proprietary GPT-4o baseline is the safest choice. If data must remain on-premises and you can afford more compute, GPT-OSS-120B offers strong capacity and benefits from lightweight fine-tuning (for example, with Low-Rank Adaptation). If you need private, low-cost processing and can accept some losses on long-tail sections, Llama-3.1-8B-Instruct provides competitive extraction on standardized fields and runs comfortably on a single modern GPU.

For regulated or privacy-critical deployments, start with Llama-3.1-8B-Instruct under strict schema constraints and add a deterministic repair/normalisation layer. If your documents heavily use rare sections or complex narratives, consider GPT-OSS-120B or a hybrid cascade where a stronger back-end is used only for those difficult parts.

2.6. Automated Evaluation of Extracted JSON

We evaluated each model output against a fixed reference JSON produced by the locked Azure back-end (Section 2.5.2) using two complementary metrics implemented: reference-based completeness, which measures the fraction of reference-filled elements that the candidate also filled, and content similarity, which compares the textual content of matched sections.

2.6.1. Reference-Based Completeness

Let $\mathcal{S}$ denote the set of top-level sections in the schema. For any schema node u with reference value $r\left(u\right)$ and candidate value $c\left(u\right)$ we define a filled predicate

$$\mathrm{filled}\left(x\right)=\left\{\begin{array}{cc}x\mathrm{is}\mathrm{a}\text{non-empty}\mathrm{string},\hfill & x\in \mathtt{str}\hfill \\ \mathrm{true},\hfill & x\in \{\mathtt{int},\mathtt{float},\mathtt{bool}\}\hfill \\ \mathrm{false},\hfill & x=\mathtt{null}\mathtt{or}\mathtt{empty}\hfill \end{array}\right.$$

and extend it recursively to compound types following the schema structure. The scoring function $M\left(u\right),R\left(u\right)$ returns a pair:

$$M\left(u\right),R\left(u\right)=\left\{\begin{array}{cc}\left(⊮\left\{\mathrm{filled}\right(r\left(u\right))\wedge \mathrm{filled}(c\left(u\right)\left)\right\},⊮\left\{\mathrm{filled}\right(r\left(u\right)\left)\right\}\right),\hfill & \mathrm{scalar},\hfill \\ {\displaystyle \sum _{k\in \mathrm{keys}\left(u\right)}}M\left(u_k\right),{\displaystyle \sum _{k\in \mathrm{keys}\left(u\right)}}R\left(u_k\right),\hfill & \mathrm{object},\hfill \\ \mathrm{see}\mathrm{below},\hfill & \mathrm{list}.\hfill \end{array}\right.$$

Lists are treated in three schema-informed cases:

• Unknown element type (empty example in the schema): We score the presence of any items in the reference list. The candidate matches if it also has a non-empty list.
• List of scalars: One point if the reference list contains at least one filled scalar. The candidate matches if it also contains any filled scalar (order and multiplicity are ignored).
• List of objects: We align by index over the first $n=min\left(\right|r|,\text{list\_cap})$ elements of the reference list and recurse element-wise.

For a section $s\in \mathcal{S}$, we obtain $(M_s,R_s)$ by recursing from the section root; the section completeness is

$$C_s=\left\{\begin{array}{cc}{\displaystyle \frac{M_s}{R_s}},\hfill & R_s>0,\hfill \\ 0,\hfill & R_s=0,\hfill \end{array}\right.C_s\in [0,1].$$

The global score aggregates raw counts across sections:

$$C_{\mathrm{TOTAL}}={\displaystyle \frac{{\sum }_{s\in \mathcal{S}}{M}_s}{{\sum }_{s\in \mathcal{S}}{R}_s}}.$$

By construction, completeness is reference-relative and asymmetric. Only elements that the reference filled can contribute to the denominator. Comparing the reference to itself yields $C_s=1$ whenever $R_s>0$.

Scalars are considered filled whenever they contain a numeric or boolean value, while empty strings are treated as unfilled. Lists composed of scalars are evaluated using a binary “any-filled” rule, meaning that the entire list is regarded as filled if at least one element contains a non-empty value. For lists of objects, the elements are aligned by index with their counterparts in the reference data, rather than matched as sets, which intentionally rewards structural conformity between the compared outputs. Sections such as schema_version, schema_org, europass and meta are excluded from the analysis to prevent bias from metadata or schema descriptors. All these rules collectively ensure that the completeness metric consistently reflects the degree of structural coverage and adherence to the target schema.

2.6.2. Content Similarity

Textual similarity is computed only for sections where the reference contains any text. For each section s, strings are collected from the corresponding JSON subtree using the schema-guided walker $\mathrm{collect}\left(·\right)$, which descends through objects, concatenates the first $min\left(\right|·|,\text{list\_cap})$ elements in lists of objects, and joins only non-empty string values in lists of scalars, while ignoring numeric entries. Let $t_r$ and $t_c$ denote the reference and candidate texts, respectively. Tokenization, applied after case folding, follows a conservative rule that accepts letters (including extended characters used in Polish), digits, and a limited set of separators.

We compute the cosine similarity and F1-score as follows:

$$\cos \left(s\right)={\displaystyle \frac{\langle \mathrm{bow}\left(t_r\right),\mathrm{bow}\left(t_c\right)\rangle }{\parallel \mathrm{bow}\left(t_r\right){\parallel }_2{\parallel \mathrm{bow}\left(t_c\right)\parallel }_2}}$$

(2)

$$\mathrm{F}1\left(s\right)={\displaystyle \frac{2\mathrm{Prec}·\mathrm{Rec}}{\mathrm{Prec}+\mathrm{Rec}}}$$

(3)

where $\mathrm{bow}\left(·\right)$ denotes a Bag of Words counter and precision/recall use set tokens. A mild length prior penalizes very short candidates:

$$L\left(s\right)={\mathrm{clip}}_{[0.6,1]}\left(\sqrt{{\displaystyle \frac{|T_c|}{max(1,|T_r\left|\right)}}}\right)$$

(4)

The section similarity is then computed as a convex blend:

$$S_s=\left(0.6\cos \left(s\right)+0.4\mathrm{F}1\left(s\right)\right)·L\left(s\right),S_s\in [0,1]$$

(5)

rounded to four decimals. If the reference has no tokens in section s, we return $\mathrm{N}/\mathrm{A}$ for that section. The total similarity is a token-weighted mean over all valid sections:

$$S_{\mathrm{TOTAL}}={\displaystyle \frac{{\sum }_{s\in \mathcal{S}}{w}_s{S}_s}{{\sum }_{s\in \mathcal{S}}{w}_s}},w_s=max(1,|T_r\left|\right)$$

(6)

If no section in the reference contains text, $S_{\mathrm{TOTAL}}$ is reported as $\mathrm{N}/\mathrm{A}$.

3. Results

3.1. Completeness of Extracted Sections

Table 8. Completeness of extracted sections (%).

JSON Section	GPT-4o	GPT-OSS-120B	Llama-3.1-8B-Instruct
analytics	100	25.97	03.77
awards	100	69.67	51.75
certifications	100	83.64	69.91
compatibility	100	45.68	86.69
conferences_talks	100	43.83	28.33
contact	100	94.36	94.00
education	100	91.40	83.33
languages	100	85.10	91.50
legal	100	73.70	91.93
memberships	100	50.91	14.32
patents	100	20.00	16.67
person	100	94.09	90.03
preferences	100	97.24	98.95
projects	100	74.27	73.10
publications	100	66.13	48.20
references	100	48.04	39.85
skills	100	42.71	68.17
source	100	74.10	00.00
volunteering	100	63.22	50.99
work_experience	100	86.48	84.55

presents the completeness rates of all JSON sections extracted by the three evaluated language model back-ends. The proprietary reference model, GPT-4o, achieved a perfect 100% completeness across all sections and all 2280 documents, ensuring that every key in the schema was present in each output. In contrast, the two open-source models demonstrated considerable variability depending on the section type and structural complexity.

For GPT-OSS-120B, high completeness was obtained for well-structured and frequently occurring sections such as contact (94.36%), education (91.40%), languages (85.10%), person (94.09%), preferences (97.24%) and work_experience (86.48%). Llama-3.1-8B-Instruct achieved similarly strong results in the compatibility (86.69%), contact (94.00%), languages (91.50%), legal (91.93%), person (90.03%) and preferences (98.95%) sections. However, both models struggled with long-tail or sparsely represented categories, where structured patterns are less consistent across documents. The most incomplete sections included patents, analytics, conferences_talks, references, volunteering and publications. These areas require models to handle nested or multi-entry data, which often led to partial or missing outputs.

Comparing the two open-source systems, Llama-3.1-8B-Instruct outperformed GPT-OSS-120B in several key categories, particularly compatibility, skills, legal and interests, where improvements ranged from 15 to over 40 percentage points. Conversely, GPT-OSS-120B achieved better completeness in sections such as source, memberships, analytics, publications, awards and conferences_talks. The most striking discrepancy was observed in the source section, where GPT-OSS-120B reached 74.10% completeness, while Llama-3.1-8B failed to generate this section entirely, resulting in 0%. This suggests a systematic omission likely caused by model-specific decoding behavior rather than random errors.

As summarized in

Table 9. Average completeness metrics across all sections.

Model	Avg. Rel. Completeness (%)	$\Delta$ vs. Ref (pp)	Files (n)
GPT-4o	100.00	0.00	2280
GPT-OSS-120B	73.52	−26.48	2280
Llama-3.1-8B-Instruct	79.21	−20.79	2280

, the average completeness across all JSON sections confirms these trends. GPT-4o maintained 100% completeness across the dataset, serving as a perfect reference baseline. Llama-3.1-8B-Instruct reached an average completeness of 79.21%, while GPT-OSS-120B achieved 73.52%. Although Llama-3.1-8B is substantially smaller in parameter count, it provided a higher overall completeness, albeit with greater inter-section variability. GPT-OSS-120B exhibited more uniform but slightly lower completeness, which indicates that its extraction strategy may generalize better to uncommon or nested fields but underperforms on standard structured sections.

From an operational perspective, GPT-4o remains the only model ensuring strict schema coverage suitable for high-integrity extraction tasks. Among the open-source back-ends, Llama-3.1-8B-Instruct demonstrates a more favorable overall completeness profile, especially for standardized or frequently occurring sections such as compatibility, legal, preferences and languages. Nonetheless, it requires targeted improvements for sections like source and analytics. GPT-OSS-120B, on the other hand, performs better in less frequent categories such as publications, awards and conferences_talks, though at the cost of overall coverage. These findings indicate that open-source models, while less complete than GPT-4o, can be significantly improved through prompt refinement and schema-level validation mechanisms designed to recover omitted sections without compromising determinism.

3.2. Content Similarity Against the Reference

Table 10. Content similarity by section (%).

Section	GPT-4o	GPT-OSS-120B	Llama-3.1-8B-Instruct
analytics	100	23.21	01.61
awards	100	62.36	51.75
certifications	100	78.46	70.06
compatibility	100	40.78	85.93
conferences_talks	100	35.93	25.56
contact	100	90.69	90.37
education	100	84.59	83.92
languages	100	83.50	79.96
legal	100	76.95	91.73
memberships	100	46.85	14.54
patents	100	1.75	6.10
person	100	92.54	65.56
preferences	100	99.61	99.50
projects	100	62.34	56.52
publications	100	58.28	44.06
references	100	40.20	34.74
skills	100	18.88	42.19
source	100	75.95	0.00
volunteering	100	51.47	42.66
work_experience	100	71.21	61.91

quantifies content similarity to the reference for each JSON section. As expected, the reference system GPT-4o attained 100% across all sections because it defined the target content. The two open-source back-ends showed pronounced section-dependent variability. For standardized, highly recurrent fields with constrained vocabularies, both models tracked the reference closely: contact reached 90.69% for GPT-OSS-120B and 90.37% for Llama-3.1-8B-Instruct; education was 84.59% and 83.92%, respectively; and preferences was near-perfect for both (99.61% and 99.50%). Llama-3.1-8B-Instruct also aligned strongly on legal (91.73%) and compatibility (85.93%), suggesting that fixed schemas and consistent phrasings favor this model’s reproduction fidelity. In contrast, sections that were sparse, heterogeneous or entail multi-item narratives showed large deficits. The most severe drops occurred in analytics (23.21% for GPT-OSS-120B; 1.61% for Llama-3.1-8B-Instruct) and patents (1.75% and 6.10%), reflecting rarity and compositional diversity. Skills was another challenging area, with 18.88% for GPT-OSS-120B and 42.19% for Llama-3.1-8B-Instruct, indicating that even when sections were present they often diverged in granularity, ordering or normalization relative to the reference. A particularly notable failure was source, where GPT-OSS-120B reached 75.95% but Llama-3.1-8B-Instruct yielded 0.00%, consistent with a systematic omission of this section rather than random content drift. Moderate but meaningful gaps also persisted in narrative, long-tail sections such as projects, publications, conferences_talks, references and volunteering, where similarity ranged typically from the mid-30s to low-60s percent, underscoring sensitivity to phrasing, item selection and list completeness.

The aggregate results in

Table 11. Average content similarity metrics across all models.

Model	Avg. Rel. Content Similarity (%)	$\Delta$ vs. Ref (pp)	Files (n)
GPT-4o	100.00	0.00	2280
GPT-OSS-120B	72.44	−27.56	2280
Llama-3.1-8B-Instruct	58.66	−41.34	2280

mirrored these section-level patterns. Averaged across all sections and files (n = 2280), GPT-OSS-120B attained 72.44% content similarity, while Llama-3.1-8B-Instruct achieved 58.66%, trailing the reference by 27.56 and 41.34 percentage points, respectively. The lower global mean for Llama-3.1-8B-Instruct—despite its strong alignment on legal, compatibility and several standardized fields—can be traced to severe deficits in a handful of structurally complex or rarely populated sections (source, analytics, skills), which depressed the overall average. By contrast, GPT-OSS-120B showed a more balanced profile with fewer catastrophic outliers but still exhibited substantial distance from the reference in narrative sections and in skills. Taken together with completeness analyses, these results indicate that producing a section at all is necessary but not sufficient for high similarity: sections like skills and projects may be present yet differ materially from the reference due to normalization choices, item selection or ordering.

Operationally, the findings suggest that near-deterministic schema enforcement should be complemented by content-level guardrails in the most error-prone sections. For standardized fields already close to the reference, small prompt refinements can likely close the residual gap. For skills, projects and other long-tail narratives, stricter canonicalization (e.g., vocabulary whitelists, order constraints and de-duplication rules) and targeted repair passes are advisable. Addressing Llama-3.1-8B-Instruct’s systematic omission of source is particularly impactful, as it would lift both sectional and overall similarity without affecting strong performance on well-structured fields.

4. Discussion

This work presented a reproducible, privacy-preserving pipeline for transforming CV PDFs into structured JSON via an intermediate, anonymized Markdown representation. The Docling PDF→Markdown conversion, coupled with a deterministic anonymization protocol, produced uniform inputs for three language-model back-ends that were driven by an identical two-message prompt and near-zero-temperature decoding (Section 2.2 and Section 2.5, including Section 2.5.1). Evaluation relied on two complementary families of metrics (Section 2.6): a reference-based completeness measure that quantifies schema coverage relative to a frozen proprietary endpoint (GPT-4o) and a content-similarity score that aggregates Bag of Words and set-matching signals under a length prior. The corpus comprised 2280 documents with a predominantly English distribution and a non-trivial Polish subset (Table 1), providing a realistic test bed for multi-format, multi-language CV extraction.

Three empirical regularities emerge from the results. First, the pinned proprietary back-end (GPT-4o) achieved perfect schema coverage across all sections and files, confirming that strict prompt design and deterministic decoding suffice to guarantee structural integrity without enabling JSON-mode features. Second, the two open-weight models reached useful but section-dependent coverage: Llama-3.1-8B-Instruct delivered higher average completeness (79.21%) than GPT-OSS-120B (73.52%), with notable advantages in standardized, low-entropy sections such as compatibility, legal, preferences and languages (Table 8 and the summary in Table 9). Third, content similarity echoed this pattern for constrained fields—contact and education showed consistently high alignment—yet diverged markedly in rare, compositional or derived sections, most visibly in analytics, patents and skills (Table 10; model-level aggregates in Table 11). The systematic omission of the source section by Llama-3.1-8B-Instruct (0.00% completeness and similarity) further indicates model-specific decoding behavior under structural pressure rather than random failure.

The section-specific behavior is readily explained by the interaction between schema structure, lexical entropy and derivational load. Fields with stable micro-structure and narrow vocabularies (contact, preferences, parts of person) are extracted reliably by all back-ends because their surface cues are frequent and unambiguous. In contrast, low-frequency sections with heterogeneous phrasing (patents, conferences_talks, awards, volunteering) are more brittle given the need to recognize nested objects under variable headings and list layouts. The most challenging category comprises derived content such as analytics, which requires consistent capture of dates, roles and employers across the JSON tree, followed by normalization and light reasoning to compute totals, tenures and gaps. The observed deficits—23.21% and 1.61% content similarity for GPT-OSS-120B and Llama-3.1-8B-Instruct, respectively—suggest that a single-pass extractor, even under strong guardrails, is insufficient when downstream fields depend on cross-sectional aggregation.

The PDF→Markdown stage shaped, but did not dominate, downstream performance. Strong alignment in contact and education indicates that Docling’s layout analysis, table reconstruction and reading-order prediction provided semantically coherent inputs for most CVs (Section 2.2). Nevertheless, occasional mis-ordering or table segmentation can move items across list positions. Because the completeness metric aligns lists by index (Section 2.6.1), semantically equivalent but re-ordered content is penalized, an effect that also depresses content similarity when token overlap is preserved but structure differs. Replacing index-based alignment with key-aware matching (e.g., by title, name or date ranges) in the evaluator would reduce this sensitivity without diluting the metric’s structural discipline.

Although the proposed pipeline processed the vast majority of CV templates consistently, including standardized formats such as Europass and common multi-section layouts generated by popular CV builders, a small subset of highly stylized or graphics-heavy templates posed occasional challenges. CVs created in design-oriented tools (e.g., Photoshop) or those employing unconventional multi-column structures sometimes resulted in imperfect layout segmentation during the PDF→Markdown conversion stage. These cases were rare and did not materially influence the aggregate results, yet they highlight an inherent limitation of document-understanding pipelines operating on visually complex source formats. Future extensions may incorporate template-aware heuristics or fallback rendering strategies to further mitigate such scenarios.

Operationally, the pipeline favors determinism, auditability and data minimization. Anonymized Markdown is the only model input; the GPT-4o snapshot is frozen; decoding uses temperature 0; and outputs undergo strict JSON validation with conservative repair and complete logging. These choices produced byte-identical re-runs and transparent failure modes without exposing raw PDFs or identifiers. In deployments where cost or data-sovereignty constraints preclude universal reliance on a proprietary endpoint, the error profile motivates a hybrid strategy: route standardized sections to open models under grammar or schema constraints while reserving derived or long-tail sections (analytics, skills, some projects) for a stronger back-end or a second deterministic pass that computes derived fields from already validated atomic extracts.

The study had limitations that bound interpretation. Completeness and content similarity were reference-relative by design; they quantified conformance to a strong baseline, not absolute factuality, and, thus, would benefit from human-annotated gold subsets to estimate truth-level accuracy. The content metric was intentionally simple and robust but insensitive to paraphrase-level semantics and fine-grained structure; embedding-based similarity or schema-aware edit distances could provide complementary signal. The corpus was predominantly English (89.52%) with 10.48% Polish (Table 1); performance was not stratified by language or CV genre and anonymization, while preserving timelines and technologies, necessarily removed some entity cues that could aid disambiguation in projects and work_experience. Despite these caveats, the results support several actionable improvements: grammar-constrained decoding to eliminate systematic omissions such as source; section-specialized micro-templates and a cascade that first extracts atomic fields and only then computes analytics; vocabulary-driven normalization and deduplication for skills; and an evaluator and repair layer that match lists by content keys rather than indices while maintaining deterministic behavior.

5. Conclusions

We introduced an end-to-end pipeline for GDPR-aligned CV extraction that converts PDFs to structured Markdown with Docling, enforces deterministic anonymization and drives multiple language-model back-ends with a uniform prompt under near-deterministic decoding, followed by rigorous schema validation. On a 2280-document corpus, the frozen GPT-4o reference achieved perfect coverage of the target schema, while the open-weight back-ends attained 73.52–79.21% average completeness and 58.66–72.44% average content similarity. The open models performed strongly on standardized sections but lagged in rare, compositional and derived fields, with the most pronounced deficits in analytics and a systematic omission of source by Llama-3.1-8B-Instruct. These findings demonstrate that high-integrity, auditable extraction is immediately attainable with a pinned proprietary model and that open models can be made substantially more reliable by combining grammar- or schema-constrained decoding, section-aware prompting and deterministic post-validation and repair. The most impactful near-term improvements are to guarantee section presence via decoding constraints, to normalize skills through curated vocabularies and alias maps and to compute analytics from validated atomic fields in a second pass. Future work will incorporate key-aware list alignment in both evaluation and repair, perform stratified analyses by language and CV genre, explore targeted fine-tuning for rare sections and derived fields and assess hybrid routing policies that allocate sections across open and proprietary back-ends under a common set of guardrails. Collectively, these steps close the gap on the most error-prone regions while preserving determinism, audit transparency and minimal exposure of sensitive data.