Evaluation of Vanilla and RAG-Supported LLM Comprehension of the EU AI Act

Abstract

This paper presents a comparative benchmarking study evaluating the regulatory comprehension of four open-source large language models, TinyLlama-1.1B-Chat, Gemma-2B-Instruct, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3, on the EU Artificial Intelligence Act (Regulation EU 2024/1689). A custom benchmark of 100 multiple-choice questions was constructed and classified across three dimensions, namely cognitive difficulty, knowledge type, and regulatory mechanism, with a balanced answer key distribution. Each model was evaluated under vanilla inference, relying exclusively on parametric knowledge, and retrieval-augmented generation (RAG), in which relevant Act passages are retrieved using a FAISS-indexed sentence embedding pipeline with standardized top three chunk retrieval applied uniformly across all models. Vanilla accuracy ranges from 22.0% for TinyLlama-1.1B to 80.0% for Llama-3.1-8B, indicating that larger models perform better. Under the retrieval configuration employed, RAG improves performance of the evaluated models, with gains from 3.0 percentage points for TinyLlama-1.1B to 17.0 for Gemma-2B and 10.0 for both 7B–8B models. Questions related to procedural knowledge, governance and enforcement emerge as weaknesses in the baseline, vanilla setting, partially mitigated by retrieval. Qualitative analysis identifies two RAG failure modes: retrieval failures, where the embedding mechanism returns informationally insufficient passages, and integration failures, where the correct passage is retrieved but not correctly exploited. The study contributes an empirical characterization of small and medium language model regulatory comprehension and a reusable 100-question benchmark.

1. Introduction

Large language models (LLMs) have advanced natural language processing tasks, including question-answering (QA), summarization, and reasoning. LLM benchmarking is an active area, given the increasingly important role of LLMs in information retrieval and decision support (or even decision-making). Existing benchmark studies evaluate LLMs on general natural language processing (NLP) benchmarks, typically employing large models with extensive computational resources. While LLMs have demonstrated strong performance across diverse benchmarks, there is growing interest in small language models (SLMs) with parameter counts between 1B and 8B parameters, which are particularly suitable for environments with constrained computational resources, including edge deployments and privacy-sensitive systems. However, the performance of small and medium models in domain-specific, question-answering tasks remains insufficiently explored. Small models exhibit limitations when processing specialized domains, such as legal documents, which contain extensive internal cross-references and highly specialized terminology.

In this work, we investigate the performance, in terms of accuracy, of small or medium models in specialized domains. Specifically, we evaluate the performance of small and medium language models (SLM and MLM) on question-answering tasks related to the EU Artificial Intelligence (AI) Act. A set of 100 multiple-choice questions (MCQs), based exclusively on the EU AI Act, of different cognitive difficulty, knowledge type, and referring to different processes, evaluates the model performance on domain-specific tasks. Two LLM-based pipelines for automated question-answering are employed, under two conditions: (a) vanilla inference based on the models’ existing knowledge, and (b) retrieval-augmented generation (RAG).

Language models and retrieval-augmented generation are both active research areas. Evaluating their capacity to comprehend the regulatory frameworks governing their deployment has important technical and societal implications for the future internet ecosystem. The manuscript contributes to the growing body of research on regulatory technology and AI governance, areas that Future Internet has consistently featured, by providing actionable evidence at the intersection of AI systems and internet governance. The reusable MCQ benchmark can be reused for intelligent web systems, automated compliance tools, and the governance of AI-enabled digital services.

The structure of the document is as follows: Section 2, Materials and Methods, describes the state of the art, the selection of LLMs, the materials on which the LLMs are evaluated, the types of MCQs, the methodology, and the experimental environment. Section 3 describes the experimental results by model and question type, while Section 4 presents the discussion and observations. Section 5 concludes the document, including directions for future work. The MCQs are included in the Appendix A, each with a unique identifier, four labelled answer options and the correct answer letter along with classification metadata fields related to cognitive difficulty, knowledge type and regulatory mechanism.

2. Materials and Methods

2.1. Language Models

Language models typically use the Transformer architecture [1,2], in which inputs are converted into vector representations (embeddings) that are subsequently used to generate probabilistic predictions. The original Transformer architecture consists of encoder and decoder components responsible for input representation and sequence generation, respectively. Language models may generate factually incorrect or unsupported responses, commonly referred to as hallucinations. Hallucinations are commonly categorized as intrinsic when the model provides information that contradicts what is already present in the input context, or extrinsic when they include information that cannot be deduced from the input document [3]. Recent taxonomies distinguish between factual hallucinations and faithfulness hallucinations, capturing inconsistencies with real-world facts and divergence from the input context or lack of internal consistency [4,5].

These limitations are critical in document-grounded question-answering (QA), especially in legal and regulatory domains, where correctness depends on precise alignment with source texts. This challenge is reflected in legal NLP benchmarks such as LexGLUE [6] and LegalBench [7], which highlight the difficulty of reasoning over long, structured legal documents requiring cross-referencing and precise interpretation. Surveys of instruction-tuning and supervised fine-tuning indicate improvements in instruction-following but also emphasize remaining gaps in factual precision and reasoning [8,9,10].

Small and medium LLMs (1B–7B) support computationally constrained environments, but their capacity for domain-specific reasoning is limited relative to larger models. However, recent work demonstrates that parameter-efficient adaptation techniques, such as LoRA [11], along with distillation approaches and intermediate reasoning supervision, can significantly improve performance in compact models. Taylor [12] investigates the performance of small models in domain-specific tasks and shows that parameter-efficient adaptation techniques allow compact models to achieve performance comparable to larger systems. This finding is supported by other studies on domain adaptation and distillation, in which the Adapt-and-Distil method suggests that small models can achieve better results when trained on subject-specific texts with adapted vocabularies [13]. Additionally, approaches using the distillation method and exploiting intermediate reasoning steps demonstrate that small models can achieve higher performance than larger architectures [14]. Studies in sentiment analysis have shown that compact models can achieve high accuracy with high-quality datasets, reducing computational costs [15]. Even when trained on weakly labelled or unlabelled datasets, medium language models can perform classification tasks with reasonable accuracy [16].

Based on these considerations, two SLMs and two MLMs have been selected. Specifically, the two small-scale models, TinyLlama-1.1B-Chat [17] and Gemma-2B-Instruct [18], represent instruction-tuned models at sub-2B parameter counts. TinyLlama was trained on 3 trillion tokens using the Llama-2 architecture and fine-tuned with ChatML-style instruction following, making it a capable but parameter-constrained model. OPT (Open Pretrained Transformer) [19] offers controlled training conditions, closely mirroring the structure of GPT-style transformers while being openly documented. OPT was excluded following a pre-experiment evaluation that revealed consistently below-random performance. Gemma-2B-Instruct was used as the second SLM. Gemma-2B-Instruct is Google DeepMind’s instruction-tuned variant of the Gemma architecture designed to maximize capability per parameter. The two medium-scale models, Llama-3.1-8B-Instruct [20] and Mistral-7B-Instruct-v0.3 [21], represent the current state of the art in open-source instruction-tuned models at the 7B–8B parameter scale. Llama-3.1-8B-Instruct is Meta’s most recent small Llama model, trained on over 15 trillion tokens with grouped-query attention and an extended 128K context window. Mistral-7B-Instruct-v0.3 employs sliding window attention. All models are publicly available on Hugging Face Hub.

2.2. Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) combines neural retrieval with sequence generation. RAG combines the parametric knowledge encoded within a model with non-parametric knowledge retrieved from external documents. The original formulation of RAG was introduced in [22] and further extended in subsequent works [23,24]. RAG pipelines use vector search libraries (e.g., Facebook AI Similarity Search (FAISS) [25]) to efficiently retrieve semantically relevant chunks. The retrieved passages are concatenated (or otherwise fused) into the LLM prompt or used as conditioning context during generation. RAG has been shown to reduce hallucination and drastically improve grounded accuracy in knowledge-intensive tasks, making it especially appealing when the target corpus is small, curated, and authoritative. Comparative studies between retrieval-augmented generation and parameter-efficient fine-tuning approaches indicate that retrieval-based methods often yield greater improvements in factual accuracy and adaptability, particularly in settings where knowledge must remain closely aligned with external documents [26]. Evaluations of small open-source models indicate that integrating retrieval components improves the accuracy of generated responses compared to standalone models [27]. In syllabus-based content, where the phrasing of questions may differ from the educational wording, dense retrieval tends to provide conceptually relevant passages more robustly than only lexical matching [28].

However, the effectiveness of RAG depends on retrieval quality and how context is used. In long, structured documents, relevant information may be distributed across multiple sections, and models often struggle to use retrieved context effectively. This limitation is discussed in [29], where language models tend to underutilize information located in the middle of long input sequences. Architectural approaches designed for long-document processing, such as Longformer [30], attempt to address these limitations through sparse attention mechanisms; however, retrieval-based approaches remain necessary when relevant information is distributed across non-contiguous sections. Furthermore, retrieval pipelines may introduce partially relevant or misleading context, which can degrade generation quality [31]. These limitations are problematic in legal texts, where incomplete or improperly scoped information can produce incorrect interpretations.

Recent work has explored extensions of RAG that integrate structured knowledge representations to improve retrieval and reasoning. For example, Pythia-RAG [32] introduces a unified multimodal knowledge graph to enhance question-answering performance, while Talk2Doc [33] employs weighted knowledge graphs to guide retrieval and improve grounding. These approaches reflect broader trends toward combining unstructured retrieval with structured knowledge to improve robustness and faithfulness. Advances in retrieval-augmented generation further extend the paradigm by incorporating iterative and self-reflective mechanisms. For instance, Self-RAG [34] enables models to dynamically decide when to retrieve, generate, and critique their own outputs, improving both factual grounding and reasoning consistency. Similarly, DocPrompting [35] demonstrates the effectiveness of retrieval over structured documentation, highlighting the importance of retrieving contextually relevant and well-aligned textual evidence. These approaches indicate that retrieval should be viewed as an integral component of the reasoning process.

2.3. The EU Artificial Intelligence Act

The primary source document used in this study is the EU Artificial Intelligence Act (Regulation (EU) 2024/1689 of the European Parliament and of the Council [36]). The Act includes 113 Articles and 180 recitals that articulate the legislative intent. The Act exhibits substantial linguistic and structural complexity. Its regulatory logic is risk-based and cross-referential: key obligations in the main Articles are qualified by definitions in Article 3, scoped by the risk categories in Article 6 and Annex III, and further conditioned by sector-specific carve-outs and transition provisions in multiple chapters. Internal cross-referencing means that the meaning of an individual passage is frequently incomplete without resolving references to other parts of the document. This creates a challenge for NLP models operating under retrieval conditions. A retrieved chunk may contain a compliance obligation while omitting the scope condition located in another passage, leading to incorrect interpretation. The vocabulary of the Act is a mixture of technical legal terminology specific to product safety law (e.g., conformity assessment, notified bodies, CE marking) and domain-specific AI terminology (e.g., general-purpose AI model, foundation model, systemic risk).

From a retrieval and comprehension standpoint, the Act presents four distinct sources of difficulty. First, definitional density: Article 3 contains 65 defined terms, and the definitions themselves contain forward references. Second, conditional structure: Obligations are heavily qualified (“unless…”, “without prejudice to…”, “except where…”), requiring a reading system to integrate multiple clauses to determine applicability. Third, list-based annexes: Critical scope-determining content appears in enumerated lists in the Annexes rather than in prose, and these lists are frequently referenced but not reproduced in the main Articles. Finally, numeric thresholds and timelines: The Act contains specific quantitative criteria (e.g., the 10²⁵ FLOP-based compute threshold for systemic risk classification, incident notification windows) that a model must retrieve precisely.

2.4. Benchmarking and Evaluation Questions

Widely used QA benchmarks include the Stanford Question Answering Dataset (SQuAD), which contains questions derived from Wikipedia articles and is designed to evaluate machine reading comprehension [37]. BoolQ [38], CommonsenseQA [39], and MCTest [40] focus on general knowledge or commonsense reasoning [31]. These datasets focus on general language understanding, but they are not designed for specialized technical domains such as legal text [41]. In addition, there is a concern that benchmark datasets are increasingly incorporated into LLM training corpora, making evaluation artificially improved. Previous works have demonstrated that it is possible to assess the comprehension of a text through carefully designed categorization and assessment tools [42].

In this view, the benchmark is designed by the authors to evaluate LLM regulatory understanding in a controlled and systematic manner. The multiple-choice question (MCQ) benchmark comprises 100 original questions, each offering four candidate answers of which exactly one is correct. The MCQ benchmark is designed to ensure broad coverage of the Act while reducing reliance on superficial text matching. Questions avoid reliance on external knowledge sources beyond the Act itself, to ensure that differences in model performance can be attributed to the experimental condition rather than to background knowledge.

Each question was annotated along three independent dimensions:

• Cognitive Difficulty: Questions were assigned one of three levels. Easy questions (n = 23) require direct recall or definition of terms and thresholds explicitly stated in the Act. Questions of medium difficulty (n = 40) demand conceptual understanding and reasoning about regulatory principles. Difficult questions (n = 37) involve multi-step legal and analytical reasoning, such as identifying chains of responsibility across actors and evaluating the interaction of two regulatory provisions.
• Knowledge Type: Questions were further classified into four knowledge-related categories: (a) factual/terminology, covering definitions, numerical thresholds, and named provisions (n = 31); (b) conceptual/theoretical, targeting the rationale and principles underpinning regulatory choices (n = 23); (c) procedural, addressing ordered sequences of compliance actions such as conformity assessment workflows (n = 25); and (d) applied/contextual, presenting realistic deployment scenarios that require mapping a situation to the appropriate regulatory category or obligation (n = 21).
• Regulatory Mechanism: A third perspective captures which structural component of the Act each question addresses: risk classification (questions about the four-tier risk hierarchy and prohibited practices) (n = 21), compliance requirements (obligations concerning documentation (n = 27), data governance, human oversight, and transparency), governance and enforcement (national authorities, the European AI Office, penalties, and market surveillance) (n = 29), and AI system lifecycle (development-time, deployment, post-market monitoring, and modification obligations) (n = 23).

Figure 1 summarizes the distribution across the three categorization dimensions.

The questions are derived from the text of the EU Artificial Intelligence Act and edited by the authors, who have reached consensus on the clarity of possible responses provided as well as the selection of the correct response, reviewed and verified by an independent expert with expertise in EU regulation. All questions have one correct answer without reference to implementing acts or guidance documents. Distractors were constructed to represent plausible misreadings of the regulation instead of easily derived incorrect options, in line with best practice for discriminative MCQ design [43].

While the benchmark cannot be considered a fully validated psychometric instrument, its quality was verified through five verification steps: (1) answer key uniform distributed, i.e., A = B = C = D = 25, (2) no exploitable correct option length cues (bias), as correct-to-wrong length ratio in responses has been measured at 1.04, (3) positional uniformity across difficulty, knowledge type, and regulatory mechanism categories, (4) absence of phrases that could serve as surface-level cues and (5) article-level source references for all 100 questions, independently verified against the official EUR-Lex HTML text of Regulation EU 2024/1689 [36]. Classification labels (on cognitive difficulty, knowledge type, and regulatory mechanism) are based on defined, transparent and documented criteria. Labels were assigned through an annotation process in which each question was independently labelled by all authors and the label was determined by majority. Disagreements were resolved through discussion to reach consensus.

2.5. Workflow

The methodology and workflow are presented in Figure 2. In the preparatory phase, the evaluation benchmark consisting of 100 four-option multiple-choice questions was created. The four open-source instruction-tuned models have been loaded. TinyLlama-1.1B-Chat and Gemma-2B-Instruct have been loaded in full precision (FP32), and Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3 have been loaded with 4-bit NF4 quantization via BitsAndBytes to reduce memory consumption. The HTML version of the EU AI Act was segmented into 786 chunks of 180 words with a 20-word overlap. Each chunk was encoded into a 384-dimensional dense vector using the all-MiniLM-L6-v2 sentence transformer and stored in a FAISS flat inner-product index.

The inference phase evaluated two experimental conditions for each model, i.e., vanilla and RAG, feeding the same benchmark questions and FAISS index into all four models. Each question was encoded using the same embedding model into a query vector. The top three chunks by cosine similarity are retrieved and inserted into the model prompt as labelled excerpts before the question text.

Stochastic sampling has been disabled, setting temperature to zero, and thereby enforcing greedy decoding: at each generation step the model selects the highest-probability next token given the preceding context. A regular expression filter is then applied to extract a valid answer label from the set {A, B, C, D}. If no valid label was detected, the response was recorded as invalid.

The models received the same prompts containing the question text, four answer options labelled A–D, and an instruction to respond with a single letter. The prompt framed the task in the legal domain and the role specified “You are a legal expert specializing in EU AI regulation. Answer the following multiple-choice question by selecting the single most appropriate option.” Model-specific adaption was limited to the chat-template formatting required by each instruction-tuned model. TinyLlama used the Zephyr template, Gemma-2B used start-of-turn makers, Llama-3.1-8B used the Llama-3 header format, and Mistral-7B used the [INST] wrapper. No advanced prompt-engineering techniques, such as chain-of-thought instructions, few-shot examples, or domain-specific elaboration, were applied to any model, ensuring that observed performance differences reflect parametric knowledge rather than prompt sensitivity. The prompt content, including the system role framing as a regulatory domain expert, was identical across all models. Under the RAG condition, the three retrieved excerpts were inserted between the system framing and the question, labelled as [Excerpt 1], [Excerpt 2], and [Excerpt 3], with the instruction “Use the excerpts provided from the EU AI Act to inform your answer.” The excerpt block and accompanying instructions were identical across all four models with different template wrappers.

The results are organized as overall accuracy, accuracy by cognitive difficulty, accuracy by knowledge type, and accuracy by regulatory mechanism. Accuracy values are reported as a percentage of correct questions versus the total number of questions, with the random baseline for a four-option MCQ at 25.0%. The results include the computation of the 95% confidence interval (CI) using the Wilson score interval according to the CI formula:

$$p={\displaystyle \frac{\widehat{p}+\frac{z^2}{2n}\pm z\sqrt{\frac{\widehat{p}\left(1-\widehat{p}\right)}{n}+\frac{z^2}{4n^2}}}{1+\frac{z^2}{n}}}$$

(1)

where $\widehat{\mathrm{p}}$ is the probability of success, n is the number of samples, and the z score for 95% confidence is equal to z = 1.96.

Key aspects of the methodology have been validated. Specifically, an external annotator with expertise in EU digital regulation has independently annotated the multiple-choice questions, comparing with the employed annotation using the Cohen’s kappa. The precision of the excerpts retrieved by all-MiniLM-L6-v2 and BM25 has also been compared for a balanced sample of questions. In addition, a top-k sensitivity analysis has been performed to investigate the influence of the number of selected excerpts.

The implementation used PyTorch 3.13, Hugging Face Transformers v5.9.0, the Accelerate library version 1.13.0 for device mapping, and the Sentence-Transformers library for chunk and query encoding.

3. Results

3.1. Chunking Mechanism

Chunking was initially attempted on the PDF version of the Act. However, word splitting, fragmentation, and merging artefacts proved difficult to control, and the HTML version sourced from EUR-Lex was used to allow for cleaner segmentation. The EU AI Act source document was initially processed using a hierarchical, structure-aware chunking strategy in which each Article was treated as a primary retrieval unit, preserving its internal paragraph and sub-paragraph structure. Articles in Chapters II–VII were retained intact given their moderate length and conceptual density. Article 3 (Definitions) was designated as a standalone context block and appended to every query response as a fixed supplement, independent of semantic similarity, given its universal cross-reference function across the Act. Recitals were grouped into thematic blocks of five to ten consecutive items and stored as auxiliary chunks to support questions targeting the rationale rather than the operative text of provisions. Each Annex was treated as a single chunk. The resulting corpus comprised approximately 280 retrievable units with a mean length of approximately 320 tokens per chunk, well within the context windows of all target models.

During the initial tests in the pre-configuration phase, the RAG pipeline on TinyLlama-1.1B-Chat produced a result below vanilla inference on the same (experimental) benchmark. This result was attributed to a systematic prompt-construction failure caused by context-window overflow. The initial RAG configuration retrieved the five most semantically similar chunks, each containing approximately 400 words, and prepended the Article 3 definition chunk to every prompt. Together, the definitions chunk (approximately 530 tokens), retrieved passages (approximately 2600 tokens), question text, and the prompt template produced a total prompt length of approximately 3300 tokens. This exceeded TinyLlama’s 2048-token context window and in this case the Hugging Face Transformers library truncates the input from the left, discarding the earliest tokens. In this configuration, the retrieved context was largely or entirely removed before the model processed the prompt, leaving a malformed fragment that provided no useful information.

To confront this limitation in the experiment configuration, the chunk size was reduced from 400 to 180 words with an overlap of 20 words, producing a revised index of 786 chunks. The number of retrieved chunks was reduced from top five to top three and the always-include definitions chunk behaviour was disabled while the retrieval query was simplified to use only the question text rather than the concatenation of the question and answer options. Under the revised configuration, the estimated prompt length per question has been in the range of 700–850 tokens, which is within TinyLlama’s limit. This configuration has been applied uniformly to all four models to reflect the differences in LLM integration capacity.

3.2. TinyLlama Results

TinyLlama-1.1B achieves a vanilla accuracy of 22.0%, slightly below the 25% random baseline for a 4-option MCQ. Under the RAG condition, accuracy increases slightly to 25% (Figure 3), placing the performance at the random baseline. Given the sample size (n = 100), the Wilson 95% confidence intervals for these estimates are wide (15.2–30.8% for vanilla, and 18–34.0% for RAG).

Across cognitive difficulty levels, the vanilla condition shows higher performance on easy questions (43.5%) compared to medium (12.5%) and difficult (18.9%). Under the RAG condition, performance declines on easy questions (30.4%) but improves on medium (20.0%) and difficult (27.0%) questions (Figure 4). However, due to small subgroups sizes (n = 23 − 40) Wilson intervals are relatively wide.

By knowledge type, TinyLlama performs poorly across all categories, with low results in conceptual/theoretical (17.4%) and procedural (16.0%) questions under vanilla conditions. With retrieval augmentation, small improvements are observed in procedural (20.0%) and applied/contextual (33.3%) categories, while factual/terminology remains unchanged (32.3%), and conceptual performance declines further (13.0%) (Figure 5).

Regarding regulatory mechanisms, the near-chance overall performance implies that there is no dependable specialization across categories. Given that the confidence intervals cluster around 25%, observed differences have limited confidence (Figure 6).

3.3. Gemma-2B Results

Gemma-2B achieved vanilla accuracy 47.0%, substantially above the random baseline. Under RAG, performance increases to 64.0% (Figure 7). With n = 100, the Wilson confidence intervals for these estimates are narrower than those of TinyLlama, and the magnitude of improvement is approximately 17%.

Across difficulty levels, vanilla performance is balanced, with slightly higher accuracy on difficult questions (51.4%) compared to medium (47.5%) and easy (39.1%). With retrieval augmentation, all categories improve substantially: easy (60.9%), medium (65.0%), and difficult (64.9%) (Figure 8). Although subgroup confidence intervals remain wider than those for overall accuracy, the consistency of improvement across all categories supports a positive effect of retrieval.

Regarding knowledge type, Gemma-2B shows moderate vanilla performance across categories, with weaker results in factual/terminology (41.9%) and procedural (36.0%). Under RAG, all knowledge types improve: factual (61.3%), conceptual (65.2%), procedural (64.0%), and applied/contextual (66.7%) (Figure 9). While subgroup intervals overlap to some extent, the uniform direction and magnitude of such gains indicate effective integration of retrieved information.

Regarding regulatory mechanisms, the improvement pattern indicates that RAG enhances recall of factual details and the ability to apply regulatory concepts, considering also Wilson interval widths (Figure 10).

3.4. Mistral 7B Results

Mistral-7B achieved a vanilla accuracy of 77.0%, improving to 87.0% under RAG (Figure 11). Wilson confidence intervals are relatively tight (approximately 10%).

Across difficulty levels, vanilla performance is high: easy (69.6%), medium (80.0%), and difficult (78.4%). Under RAG, all categories improve: easy (87.0%), medium (92.5%), and difficult (81.1%) (Figure 12). Although the improvement on difficult questions is modest, the trend shows a consistent increase.

By knowledge type, Mistral-7B demonstrates strong vanilla performance, particularly in conceptual/theoretical (91.3%). Under RAG, all categories improve further—factual (80.6%), conceptual (95.7%), procedural (80.0%), and applied/contextual (95.2%) (Figure 13)—with the Wilson intervals being narrow.

Regarding regulatory mechanisms, the model exhibits robust performance across categories in both conditions, with RAG providing consistent, incremental improvements (Figure 14). The lack of observable regressions, combined with comparatively narrow confidence intervals, suggests that the system is operating in a consistent manner, leveraging the retrieved contextual information.

3.5. Llama 3.1-8B Results

Llama-3.1-8B achieves vanilla accuracy of 80.0%, increasing to 90.0% under RAG (Figure 15). Wilson confidence intervals are sufficiently narrow.

Across difficulty levels, vanilla performance is consistently high: easy (73.9%), medium (80.0%), and difficult (83.8%). Under RAG, performance improves on easy questions (91.3%) and especially medium (97.5%), slightly decreasing on difficult questions (81.1%) (Figure 16). The small decline in the difficult question category is accompanied by overlapping Wilson intervals.

By knowledge type, the model performs strongly in the baseline setting, particularly in conceptual/theoretical (95.7%). With retrieval augmentation, all categories improve: factual (87.1%), conceptual (100.0%), procedural (80.0%), and applied/contextual (95.2%) (Figure 17). The conceptual category reaches top performance, where Wilson intervals become asymmetric.

For regulatory mechanisms, Llama-3.1-8B maintains strong performance across categories, with RAG producing further improvements overall (Figure 18). Confidence intervals are compressed near ceiling accuracy.

4. Discussion

4.1. RAG Case Analysis

Retrieval behaviour is investigated for eight questions, for which retrieved chunks are manually assessed for relevance and impact on model performance. The first four cases show RAG-driven corrections while the last four show cases where RAG introduced errors in models that had initially answered correctly. For the latter, two distinct failure modes have been considered: (a) failure of RAG, i.e., retrieval of irrelevant excerpts, and (b) failure of the models to integrate correct excerpts. This analysis constitutes a qualitative retrieval quality evaluation for the analyzed cases while the retrieval quality was not fully and formally evaluated (using information retrieval (IR) metrics such as precision@k and mean reciprocal rank) against a human relevance judgement set.

Q82 concerns the body established by the EU AI Act to advise the Commission, composed of national supervisory authority representatives. It is a factual, easy question on governance and enforcement. The correct answer is the European Artificial Intelligence Board (Article 65); distractors include the European Data Protection Board, the European Standardisation Committee, and ENISA. Under vanilla inference, TinyLlama selected the European Standardisation Committee while Gemma-2B and Llama-3.1-8B selected the EDPB. Of the three retrieved chunks, only Chunk 2 was directly relevant, quoting Article 65 and naming the Board with its composition of one representative per Member State. Chunks 1 and 3 were retrieved on vocabulary overlap (Board, Commission, governance) without containing the relevant provision. All three models corrected under RAG. Mistral-7B had encoded the correct answer parametrically and was unaffected by retrieval.

Q63 asks which body is empowered to conduct audits of GPAI providers suspected of serious violations. It is a medium-difficulty, procedural question on governance. Possible answers are the AI Office, ENISA, Member State market surveillance authorities, and the EDPB. All four models failed in the baseline setting, each selecting a different plausible EU regulatory body. The correct answer, i.e., the AI Office, is based on Recital 162, which places GPAI enforcement under Commission authority administered through the AI Office. Chunks 1 and 2 both cite Recital 162 and are directly relevant; Chunk 3 describes market surveillance authority procedures for compliant AI systems and is only partially relevant. Gemma-2B, Llama-3.1-8B, and Mistral-7B corrected under RAG. TinyLlama persisted with ENISA, representing an integration failure in which the retrieved passages were insufficient to override a strongly encoded parametric prior.

Q100 asks how an AI system assessing recidivism risk in criminal proceedings is classified under the Act. It is a medium-difficulty, applied question on risk classification. Gemma-2B, Llama-3.1-8B, and Mistral-7B all selected “Prohibited” under vanilla inference, incorrectly mapping the fundamental rights concerns raised by recidivism prediction to Article 5 prohibitions rather than to the Annex III high-risk classification. All three retrieved chunks were relevant; Chunk 1 lists recidivism risk assessment verbatim as an Annex III category. The three larger models corrected under RAG. TinyLlama persisted with the wrong response representing an integration failure in which the retrieved evidence was not effectively incorporated into the model’s reasoning.

Q32 asks within how many days a high-risk AI system provider must notify the national competent authority of a serious incident. It is a difficult, procedural question about governance. The correct answer is 15 days (Article 73); distractors are 72 h, 30 days, and 90 days. All four models failed under vanilla inference: TinyLlama selected 90 days; Gemma-2B and Mistral-7B selected 72 h, likely reflecting the GDPR Article 33 personal data breach notification deadline; and Llama-3.1-8B selected 30 calendar days. Chunks 1 and 2 both reference Article 73 and the serious incident notification framework, and TinyLlama, Gemma-2B, and Mistral-7B corrected to 15 days. Llama-3.1-8B persisted with 30 days despite receiving the same chunks, representing a model-specific integration failure (the 30-day timeframe appears in multiple EU regulatory instruments).

The following four cases illustrate RAG-induced regressions, in each case arising from a different failure mechanism.

Q58 asks which procedural step a public authority must complete before deploying a high-risk Annex III AI system for employment-related decisions. It is a difficult, procedural, compliance question. The correct answer is a fundamental rights impact assessment (Article 27); distractors include registration in the EU database, notification of the EDPS, and a GDPR data protection impact assessment. Llama-3.1-8B and Mistral-7B had the correct answer through parametric knowledge. After RAG, both switched to registration in the EU database. The retriever matched keywords such as “public authority”, “Annex III”, and “high-risk” to registration-related passages. As a result, the retrieved passages described deployer registration obligations rather than the Article 27 pre-deployment requirement. Chunk 3 reaches Article 27 but it is truncated before the operative obligation is stated. This is a retrieval failure: the correct article was reached but the decisive passage was not captured within the retrieved excerpt.

Q79 asks which entity affixes the CE marking to a high-risk AI system. It is an easy, factual question on the AI system lifecycle. The correct answer is the provider (Articles 16 and 48); distractors include the AI Office, the national competent authority, and a designated notified body. Llama-3.1-8B had the correct answer parametrically and maintained it under RAG. The two smaller models answered incorrectly under both conditions. Mistral-7B, which also had the correct answer initially, regressed under RAG. Chunks 1 and 2 were partially relevant, confirming that CE marking must be applied without specifying who affixes it. Chunk 3 describes affixing of the notified body’s identification number—a separate requirement from CE marking. This appears to mislead Mistral-7B, which appears to have extracted the notified body as the affixing entity without recognizing the distinction between the two requirements.

Q30 examines who bears provider-like obligations when a deployer customizes a high-risk AI system decision threshold. It is a difficult, applied, compliance question. The correct answer is that the bank (deployer) assumes provider obligations for the modified aspects; distractors include the original technology firm, no additional obligations, and the European Banking Authority. Three of the four models answered correctly under vanilla inference. The first retrieved chunk describes pre-determined learning changes that do not constitute a substantial modification, a provision that applies to provider-determined automatic updates. Llama-3.1-8B and Mistral-7B correctly resisted this misleading passage. Gemma-2B, however, regressed from the correct answer to “no additional obligations”, representing a model-specific integration failure in which a partially relevant chunk displaced a correct parametric prior.

Q92 asks for the maximum fine applicable to an SME committing a minor infringement not involving prohibited AI or GPAI violations. It is a difficult, factual, governance question. The correct answer is €5 million or 1% of global annual turnover (Article 99(6)); distractors offer higher fine tiers. The first retrieved chunk informs on the SME fine calculation principle but requires cross-referencing paragraphs 3–5 of Article 99 to derive the specific amount. Llama-3.1-8B, which had selected the wrong tier under vanilla (€10M/2%), corrected under RAG. Mistral-7B, which had the correct answer under vanilla, switched to an incorrect tier after RAG, apparently misapplying the multi-step inference required by the retrieved text. This case illustrates that even high-quality retrieval can produce contradictory outcomes across models when the correct answer requires multi-step reasoning rather than direct extraction from the retrieved passage.

4.2. Model Comparison

Figure 19 presents the radar profiles for TinyLlama-1.1B and Gemma-2B under both conditions.

Figure 20 presents the radar profiles for Llama-3.1-8B and Mistral-7B. Vanilla polygons are larger and more convex than those of the smaller models, indicating that the 7B-scale models exhibit broader EU AI Act knowledge, reflecting their combined differences in scale, architecture, training, and precision. The indentation at the governance-and-enforcement vertex indicates weaker model knowledge of the Act’s supervisory architecture, enforcement procedures, and penalty structures. Under RAG, Mistral-7B’s polygon expands uniformly outward across all dimensions, reaching near-ceiling on risk classification (95.2%) and applied/contextual (95.2%) categories. Llama-3.1-8B’s RAG profile also expands (with a slight contradiction in difficult questions, 83.8% vanilla to 81.1% RAG). Mistral-7B improves more uniformly with RAG while Llama-3.1-8B has stronger gains on easy questions with top performance on conceptual questions.

Figure 21 presents the radar profiles for the four models. The outward expansion of vanilla polygons from TinyLlama-1.1B through Gemma-2B to the 7B-scale models indicates a positive association between model scale and EU AI Act understanding. Because the models also differ in architecture, training data, and numerical precision, model size cannot be isolated as the sole cause of this difference. The observed effect is non-linear and category-dependent: governance and enforcement accuracy improve substantially from small to medium-scale (24.1% for TinyLlama to 65.5% for both 7B models), while conceptual/theoretical performance approaches its ceiling already at the 2B scale (Gemma-2B vanilla 56.5%, RAG 65.2%, versus Llama-3.1-8B vanilla 95.7%). Llama-3.1-8B under RAG is the highest-performing configuration in terms of overall accuracy (90.0%), while Mistral-7B under RAG achieves the most uniform gains across all four regulatory mechanism categories.

4.3. Statistical Significance Analysis

To investigate whether the observed differences in accuracy are statistically significant, McNemar’s test has been employed to compare the predictions of two classifiers evaluated on identical observations (pair data). The test focuses on the different predictions, that is, the cases in which the two classifiers disagree. The contingency table consists of the following parameters:

• a: The number of questions answered correctly by both the vanilla and RAG versions.
• b: The number of questions answered incorrectly by the vanilla model but correctly by the RAG model.
• c: The number of questions answered correctly by the vanilla model but incorrectly by the RAG model.
• d: The number of questions answered incorrectly by both models.

The discordant pairs (b and c) contribute to the McNemar test statistic. The exact binomial form is used as the total number of discordant pairs is lower than 25 (count of observations). For all models examined, the total number of discordant pairs was less than this number.

Using

Table 1. Contingency tables for all models.

		RAG Correct	RAG Incorrect
TinyLlama-1.1B	Vanilla Correct	15	7
	Vanilla Incorrect	10	68
Gemma-2B	Vanilla Correct	46	1
	Vanilla Incorrect	18	35
Llama-3.1-8B	Vanilla Correct	78	2
	Vanilla Incorrect	12	8
Mistral-7B	Vanilla Correct	73	4
	Vanilla Incorrect	14	9

, the exact McNemar test was applied separately to each model, giving the results as in

Table 2. Statistical summary table.

Model	Vanilla	RAG	b	c	Test Used	McNemar Statistic	p-Value	Significant
TinyLlama-1.1B	22.0%	25.0%	10	7	Exact McNemar	7	0.629	No
Gemma-2B	47.0%	64.0%	18	1	Exact McNemar	1	<0.001	Yes
Llama-3.1-8B	80.0%	90.0%	12	2	Exact McNemar	2	0.013	Yes
Mistral-7B	77.0%	87.0%	14	4	Exact McNemar	4	0.031	Yes

.

For the TinyLlama-1.1B model, the vanilla version has an accuracy of 22.0% and the RAG version an accuracy of 25.0%. Upon comparing the responses generated by the two, there were 10 cases where the RAG version gave a correct response whereas the Vanilla one was wrong (b = 10), as well as seven cases where the vanilla version was right and the RAG version wrong (c = 7). The results of the McNemar test were χ² = 7, p = 0.629. As the p-value is larger than the standard value for alpha of 0.05, the difference is not statistically significant, and the null hypothesis (the marginal probabilities of getting a right answer in each of the versions are equal) is not rejected. Therefore, RAG-based retrieval does not make TinyLlama perform reliably better (performance improvement can be attributed to randomness).

The Gemma-2B model has an accuracy of 47.0% in vanilla mode and 64.0% in RAG mode. The analysis of the discordant pairs indicates 18 instances where only the RAG model gives the right answer (b = 18) and one instance where only the vanilla model provides the right answer (c = 1). The result of the exact McNemar test gives a value of the statistic as 1 and p < 0.001. As the p value is less than 0.05, the null hypothesis is rejected.

Regarding Llama-3.1-8B, vanilla accuracy is 80.0% and RAG-based is 90.0%. There have been 12 cases when only RAG-supported answered correctly (b = 12) and two cases where only vanilla has been right (c = 2). According to the exact McNemar test, the value of statistic is equal to 2 and p = 0.013. The difference is statistically significant and thus the null hypothesis is rejected.

Concerning the Mistral-7B model, the results have been 77.0% for the vanilla approach and 87.0% for the one with RAG. There have been 14 discordant cases where only the RAG-supported correctly (b = 14) and four cases where only the vanilla has been correct in its prediction (c = 4). The McNemar test resulted in a value of 4 and p = 0.031. Since the p value is less than 0.05, the null hypothesis is rejected, indicating a significant difference in performance.

The analysis indicates that the effect of RAG on models is not uniform. Although TinyLlama-1.1B had only a marginal increase in performance, which is statistically insignificant, the Gemma-2B, Llama-3.1-8B, and Mistral-7B models showed a significant improvement after being integrated with RAG.

4.4. Methodology Validation

To validate the methodology, the three key aspects, (a) the multiple-choice question classification scheme, (b) the effectiveness of retriever in comparison with BM25 and (c) the sensitivity of top-k, have been investigated.

First, to assess the reliability of the multiple-choice classification scheme, an independent expert with expertise in EU regulation, having already verified the soundness of the correct responses, labelled the full set of multiple-choice questions based on the criteria defined in Section 2.4. Cohen kappa has been calculated considering the relative observed agreement (i.e., the actual proportion of alignment, p_o) and the probability of agreement by chance (the proportion of times they would be expected to agree by chance, p_c).

$$\mathit{Cognitive} \mathit{difficulty}: \kappa ={\displaystyle \frac{p_o-p_c}{1-p_c}}={\displaystyle \frac{0.81-0.3463}{1-0.3463}}=0.71$$

(2)

$$\mathit{Knowledge} \mathit{type}: \kappa ={\displaystyle \frac{p_o-p_c}{1-p_c}}={\displaystyle \frac{0.88-0.2524}{1-0.2524}}=0.84$$

(3)

$$\mathit{Regulatory} \mathit{mechanism}: \kappa ={\displaystyle \frac{p_o-p_c}{1-p_c}}={\displaystyle \frac{0.82-0.2582}{1-0.2582}}=0.76$$

(4)

Inter-annotator agreement, measured by Cohen’s kappa, has been 0.71 for cognitive difficulty, 0.84 for knowledge type, and 0.76 for regulatory mechanism. Figure 22 presents the confusion matrices comparing the two classifications.

Second, the retriever employed, all-MiniLM-L6-v2 with FAISS cosine similarity, has been compared with the BM25 lexical baseline operating over the identical chunk corpus. For each question, in a 40-question balanced sample (including 10 questions per category in the knowledge type and the regulatory mechanism dimensions) the top three passages returned by the dense retriever and by BM25 were collected. Each retrieved excerpt was assessed for relevance against the question by the independent expert. The excerpt is considered relevant if it contains the provision of the Act on which the correct answer depends.

Precision@3 as the proportion of the three retrieved passages considered relevant, Recall@3 as the proportion of questions for which at least one relevant passage appeared in the top three and the Mean Reciprocal Rank (MRR) as the ranking of retrieved relevant passages have been computed per question for each retriever and averaged. As depicted in Figure 23, the dense retriever (all-MiniLM-L6-v2) achieved a mean Precision@3 of 0.24, a Recall@3 of 0.57, and a Mean Reciprocal Rank (MRR) of 0.39, compared with 0.23, 0.50, and 0.36 respectively for the BM25 lexical baseline. As the responses of a subset of questions depend on a single provision of the Act, only one of the three retrieved passages may be relevant, capping for these questions the Precision@3 metric at 0.33, irrespective of retrieval quality.

The dense retriever outperformed BM25 although the modest absolute values across both retrievers confirm that retrieval remains a limiting factor. Third, while the retrieval depth was fixed at k = 3 for all experiments to allow for a consistent evaluation setting across models, a top-k sensitivity analysis, for Gemma-2B and Mistral-7B models, has been performed to examine the effect of varying the number of retrieved chunks on the accuracy.

The results, as depicted in Figure 24, indicate that retrieval depth can influence performance depending on the model. Mistral-7B benefits from additional retrieved context, with accuracy increasing from 79% at k = 1 to 89% at k = 4. Gemma-2B improves from 62% at k = 1 and k = 2 to 64% at k = 3 and 67% at k = 4. While larger models may benefit more from additional context, and smaller models may easily be disturbed by irrelevant retrieved context, a more comprehensive sensitivity analysis across a wider range of k values is required to determine the optimum number of chunks used for each model.

4.5. Limitations

The study used a single retrieval configuration, all-MiniLM-L6-v2, selected for its wide adoption, open-source availability, and limited compute requirements. all-MiniLM-L6-v2 was pre-trained on general English text without legal-domain fine-tuning, which may reduce retrieval precision on provisions where vocabulary diverges from general use. The qualitative analysis in Section 4.1 investigates retrieval quality for the eight selected questions; however, formal IR evaluation was not conducted against a human relevance judgement set. Legal-domain retrievers and alternative embedding models (such as Legal-BERT, sparse–dense hybrids, and domain-adapted embedders) have not been tested and could affect retrieval quality and the resulting RAG performance.

The retrieval configuration has been constant across all four models with the models receiving three excerpts of 180 words per question from the same FAISS index. This configuration ensured that cross-model differences reflected variation in comprehension and integration capacity rather than differences in retrieval volume. However, as identified in the top-k sensitivity analysis, 2B and 7B models may operate below their potential optimal retrieval configuration considering their larger context windows. RAG hyperparameters, particularly the chunk size and retrieval depth, were selected partially based on their performance (inherent limitations) on the evaluation benchmark. As a result, the RAG setup was not determined completely independently of the test data, which may have introduced a degree of evaluation bias. While no per-question or per-category tuning was performed, and the retrieval configuration was fixed before the final evaluation run, the performance improvements observed for the RAG-enhanced models may be slightly optimistic and may not necessarily be reproduced on a new dataset.

The benchmark set of 100 multiple-choice questions has been authored by the research team. Benchmark classification labels on the three dimensions (cognitive difficulty, knowledge type, and regulatory mechanism) have been assigned through majority vote by the authors (after independently assigning labels). The classification has been validated by an expert with expertise in EU regulation. The benchmark cannot be considered a fully validated psychometric instrument, and results should be interpreted as indicative of relative model performance rather than as an absolute measure of regulatory comprehension. In addition, findings are conditioned on the specific retrieval pipeline and may not generalize to embedding models, indexing schemes, retrieval depths, or chunking strategies.

5. Conclusions

This study evaluated four open-source large language models: TinyLlama-1.1B-Chat, Gemma-2B-Instruct, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3. The models were tested on the EU AI Act under both baseline inference and retrieval-augmented generation conditions using a curated 100-question multiple-choice benchmark with balanced answer distributions and controlled option-length bias. The results confirm that larger models perform better in terms of parametric knowledge of a specialized regulatory document but not uniformly across knowledge dimensions. The two 7B–8B models achieved vanilla accuracy of 77.0% and 80.0% respectively, while sub-2B models scored 47.0% (Gemma-2B) and 22.0% (TinyLlama-1.1B). Conceptual and theoretical questions approached ceiling performance at the 7B scale (Llama-3.1-8B: 95.7%), while governance and enforcement remained at 65.5% for both large models under standard inference configuration.

RAG improved performance for most of the evaluated models, with gains ranging from 3.0 percentage points for TinyLlama-1.1B, a change not statistically reliable given the near-random baseline, to 17.0 for Gemma-2B, and 10.0 for both 7B–8B models. These results indicate that RAG is beneficial for EU AI Act comprehension for the larger models, but without uniform effect as the improvements for Gemma-2B, Mistral-7B, and Llama-3.1-8B are statistically significant, whereas that for TinyLlama-1.1B is not.

Procedural questions were the category most consistently improved by retrieval augmentation. The qualitative case analysis identifies two distinct failure modes: (a) retrieval failures, where the embedding mechanism does not surface the relevant provision, and (b) integration failures, where the correct passage is retrieved but the model cannot correctly exploit it. RAG has improved performance for the evaluated models, in a non-uniform way, under the specific retrieval setup including all-MiniLM-L6-v2 embeddings, FAISS indexing, top-3 retrieval, and the chosen chunking strategy.

Future studies should address the potential evaluation bias arising from benchmark-informed hyperparameter tuning by tuning the RAG configuration using a separate validation set, while keeping an unknown test set exclusively for the final evaluation. This setup would provide a more reliable assessment of whether the observed RAG improvements expand to unseen datasets and different evaluation settings. Furthermore, future work can explore three primary directions. First, optimization of the retrieval mechanism can be investigated involving BM25, sparse–dense hybrids, and legal-domain embedding models, involving a variable number of k. Second, the governance and enforcement performance gap warrants further investigation to determine whether it reflects the lower frequency of EU supervisory architecture content in pre-training corpora or a difficulty in encoding enforcement procedures parametrically. Third, the benchmark methodology and RAG evaluation framework can be applicable beyond the EU AI Act to other EU regulatory instruments, including the General Data Protection Regulation, the Digital Services Act, and the forthcoming implementing measures and delegated regulations of the AI Act itself.