Building a Multilingual AI Legal Assistant Using Retrieval-Augmented Generation: A Case Study on the Legal System of Kazakhstan

Abstract

This study presents the development of an Artificial Intelligence (AI)-based legal assistant using the Retrieval-Augmented Generation (RAG) architecture to provide legal assistance to citizens of the Republic of Kazakhstan. The proposed solution is designed to generate accurate, evidence-based responses to user queries using the regulatory legal acts of the Republic of Kazakhstan as the primary source of information. A legal corpus comprising 101,000 legislative documents and court decisions, with approximately 77 million tokens in Kazakh and Russian, was constructed to support the retrieval component of the system. To identify the most effective semantic retrieval method, three multilingual embedding models—Multilingual-E5-Large, BGE-M3, and KazEmbed-V5—were evaluated for vector search. The experimental results showed retrieval accuracies of 87.6%, 76.8%, and 83.3%, respectively. The GPT-5.4 and Llama-4-Scout-17B-16E-Instruct large language models were used to generate legal reasoning and responses based on documents retrieved through semantic search. The quality of the generated responses was evaluated using two complementary approaches. First, legal experts assessed the factual correctness and legal validity of the answers. Second, automatic evaluation was performed using word-level F1, BLEU, ROUGE, and BERTScore-F1 metrics. Among all evaluated configurations, GPT-5.4 combined with Multilingual-E5-Large achieved the highest overall accuracy (88.5%), whereas Llama-4-Scout-17B-16E-Instruct combined with KazEmbed-V5 achieved an accuracy of 83.6%. Based on the proposed architecture and the selected semantic retrieval and language models, an AI legal assistant was developed and integrated into the “Adal Azamat” legal services platform providing users in Kazakhstan with practical access to AI-assisted legal consultation.

1. Introduction

The widespread use of digital technologies in the legal domain has led to a rapid increase in the volume of legal data. The overwhelming amount of digitized legal information means that, in many jurisdictions, citizens face significant challenges in understanding and navigating complex legislative and regulatory documents because of their length, hierarchical structure, and frequent amendment. As a result, both legal professionals and the general public increasingly rely on automated tools to access, interpret, and use legal information efficiently. However, traditional keyword-based legal information retrieval systems often fail to capture semantic meaning, resulting in incomplete or inaccurate retrieval, particularly when processing complex legal queries formulated in natural language. Consequently, there is growing demand for intelligent tools that facilitate access to relevant legal information for both legal professionals and citizens. Addressing this challenge requires intelligent legal information systems capable of retrieving comprehensive and up-to-date information from legislation and regulatory documents, organizing it in a structured and user-friendly form, analytically processing large volumes of legal data, and understanding natural-language queries.

Legal information system solutions developed to date include legal search engines, specialized applications, legal AI assistants, and other intelligent systems [1]. The capabilities and underlying technologies of these solutions vary considerably. Because the vast majority of legal information is available in textual form, research and development in this field have focused primarily on natural language processing (NLP). Widely used NLP techniques include semantic search, machine learning, Transformer-based models, and large language models (LLMs) [2]. The emergence of LLMs has significantly accelerated the development of intelligent legal applications, including legal question-answering systems and AI legal assistants [3,4]. Recent advances in NLP and LLMs have enabled the development of intelligent legal question-answering systems capable of generating coherent, context-aware answers. However, purely generative approaches have significant limitations in the legal domain, including hallucinated content, lack of explicit grounding in legal sources, and insufficient transparency of reasoning [5]. These limitations create substantial risks of inconsistency and distortion in legal applications, where accuracy, controllability, and consistency with primary sources are essential.

To address these challenges, RAG has emerged as a promising paradigm that integrates information retrieval mechanisms with generative language models. By combining document retrieval with text generation, RAG-based systems improve factual grounding, response consistency, and controllability, all of which are essential requirements for legal applications. Although RAG has demonstrated considerable potential in legal information retrieval and question answering, most existing systems have been developed for high-resource languages and general-purpose corpora, limiting their applicability to low-resource languages and jurisdiction-specific legal domains. This limitation is particularly evident in the Republic of Kazakhstan, where legal documents are primarily published in Kazakh and Russian. The Kazakh language, in particular, remains underrepresented in large-scale NLP resources and pre-trained language models, making the development of reliable AI-based legal systems significantly more challenging. Furthermore, legal information systems in Kazakhstan must process heterogeneous legal sources, including legislative and regulatory documents as well as court decisions, each characterized by distinct structures, terminology, and legal semantics.

The aim of this study is to investigate and develop the intelligent core of a multilingual AI assistant based on the RAG architecture and a jurisdiction-specific legal knowledge base. The proposed solution is implemented as an integrated pipeline that combines multilingual document preprocessing, vector database indexing, semantic retrieval, and response generation using large language models. Particular attention is paid to ensuring legal grounding by explicitly linking the generated responses to the relevant statutory provisions and legislative articles. Unlike existing approaches, which primarily focus on either retrieval accuracy or response generation quality, the proposed methodology adopts a system-level perspective that balances retrieval accuracy, semantic relevance, legal grounding, and practical usability.

The main scientific and experimental contributions of this study are as follows:

• Development of a domain-specific legal corpus based on the regulatory legal acts and court decisions of the Republic of Kazakhstan;
• Evaluation of multilingual embedding models for semantic retrieval of legal texts in Kazakh and Russian;
• Development of a domain-adapted RAG architecture that integrates semantic retrieval and LLM-based response generation for legal question answering;
• Comprehensive experimental evaluation of the proposed approach using both automatic evaluation metrics and expert assessment;
• Development and implementation of an AI legal assistant based on the proposed architecture and the results of the experimental evaluation.

The remainder of this article is organized as follows: introduction; related work; research methodology and models; experimental results and evaluation; discussion and limitations; and conclusion.

2. Related Works

To date, a large number of applications and intelligent systems have been developed that exploit the capabilities of artificial intelligence in the legal domain. Similarly to other fields influenced by digital transformation, the application of information technologies in law has been investigated for decades, resulting in numerous software solutions designed to support legal practice and improve access to legal information and services [1,6]. Early legal AI systems were primarily based on rule-based expert systems, keyword-based information retrieval, and logical inference mechanisms. These approaches required extensive manual knowledge engineering and system training, while their scalability and adaptability to evolving legal knowledge remained limited [7,8]. Subsequently, the emergence of data-driven machine learning, deep learning, and NLP techniques gradually replaced symbolic approaches, enabling more effective processing of unstructured legal texts and significantly improving the performance of intelligent legal applications [2,3,9].

The adaptation of LLMs to the legal domain has become one of the most active areas of artificial intelligence research in recent years. Studies have shown that LLM technologies perform well in tasks such as automatic classification of legal documents, retrieval of legislative information, analysis of court decisions, summarization of legal texts, and answering legal questions [5,10]. However, general-purpose language models do not fully capture legal terminology or specific legal contexts, which reduces their accuracy [11]. Accordingly, many studies recommend using models and legal knowledge bases specifically adapted to the legal domain [12]. Modern AI legal assistant systems aim to answer users’ legal questions accurately by using legal documents, regulatory acts, and case law based on national legislation [13]. Research is also being conducted on creating specialized legal datasets to improve legal question answering. For example, the IndicLegalQA dataset, consisting of 10,000 question–answer pairs based on 1256 cases of the Supreme Court of India, has been developed and shown to be effective for training legal chatbots and LLMs [14]. In addition, approaches have been proposed for explaining legal concepts through cases using LLMs, thereby conveying legal knowledge in a form understandable to non-specialists; their effectiveness in improving legal literacy has been confirmed experimentally [15]. Recent publications indicate that generative models alone are insufficient for legal reasoning and should therefore be combined with deductive, case-based, and abductive reasoning methods [10]. However, many studies show that standalone LLMs are prone to hallucination, may not identify the original source of a document, and can produce legally incorrect or unverifiable statements [13,16]. These limitations pose significant risks to the factual accuracy and correctness of legal applications [16,17]. To address these issues, current research increasingly favors RAG architectures that combine LLMs with external legal knowledge bases.

Retrieval-Augmented Generation is considered one of the principal approaches for improving the accuracy and reliability of legal AI systems. Although large language models perform well in processing legal texts, their static knowledge and tendency to hallucinate can reduce the quality of legal answers [5,18]. To address this problem, an RAG architecture uses semantic search to retrieve regulatory acts, court decisions, and other legal documents relevant to the user’s query, and then generates an answer based on the retrieved context. This improves the accuracy and legal grounding of the answer [18,19]. Recent studies have shown that RAG systems can make effective use of relationships among legal documents by combining vector stores, knowledge graphs, and semantic models [20]. Legal search systems are also shifting from keyword-based methods to semantic search. In this context, embedding models, text chunking, and multi-level vector representation methods significantly improve the retrieval of similar court cases [21]. Hierarchical frameworks combining information at the levels of facts, legal reasoning, and judicial opinions have been proposed to preserve the logical structure of legal documents and have been shown to improve the retrieval of legal precedents [22]. RAG is also used in automatic legal-document generation systems, where relevant sections of previous court documents are used to generate new text while preserving the authority of the lawyer or judge to make decisions under the human-in-the-loop principle [19,23]. Overall, current research indicates that RAG has become one of the key architectural solutions for ensuring the accuracy, interpretability, and reliability of legal AI assistants.

Semantic search techniques are widely used in the retrieval phase of the RAG architecture to identify documents that are semantically relevant to a user query. Unlike traditional keyword-based search, semantic search relies on dense vector representations of text, enabling the system to capture contextual and conceptual similarities between user queries and legal documents [18,24]. Recent studies have shown that embedding-based retrieval significantly improves both recall and relevance in large-scale legal corpora [3,25]. These approaches perform particularly well when processing documents containing complex legal terminology and long-range semantic dependencies [15]. Multilingual embedding models, such as Multilingual-E5-Large and BGE-M3, have demonstrated strong performance in cross-lingual and low-resource language settings, making them well suited for use in jurisdictions with bilingual or multilingual legal frameworks [26,27].

An analysis of the scientific literature shows that artificial intelligence is currently considered not as a technology that will completely replace lawyers, but as an intelligent assistant and decision-making tool that will facilitate the work of legal professionals and judicial authorities. AI systems improve the efficiency of legal professionals and reduce the time required to perform repetitive tasks, such as legal research, document analysis, contract drafting, legal information retrieval, and case law analysis [28,29,30]. At the same time, artificial intelligence is transforming the nature of legal services by complementing, rather than replacing, the professional activities of lawyers and enhancing the quality of legal analysis and decision-making [30,31]. AI-based legal assistants enable citizens to access legal information around the clock, obtain preliminary legal advice based on legislation and case law, automatically generate legal documents, and assess the likely outcome of legal proceedings [32,33,34]. In addition, modern legal AI platforms that integrate knowledge graphs, semantic search, and large language models improve the accuracy of legal information processing and facilitate legal research [18]. However, many studies have highlighted that important issues remain unresolved, including the legal liability of AI systems, algorithmic transparency, personal data privacy, model neutrality, and the allocation of responsibility for legal errors. Therefore, in legal practice, AI tools are regarded as assistive technologies that support the decision-making process under the human-in-the-loop principle rather than replacing the professional judgment of lawyers or judges [28,30,31,34].

The studies reviewed above indicate that a wide range of approaches has been proposed for developing AI legal assistants. Among the most promising are architectures that integrate NLP with RAG, enabling more accurate and context-aware retrieval of legal information and generation of responses [9,17,19]. However, given the specific characteristics of legal texts and the need for reasoning grounded in regulatory documents, question-answering systems and AI assistants based on the legal knowledge base of a particular country have produced better results. LLM technologies have been adapted to the national legal system in Italy [5], legal question-answering systems and specialized legal datasets have been developed in India, where AI has also been applied in the judicial system [14,29], systems have been developed in China for judicial examinations [35], construction law, and legal reasoning [12], legal-information retrieval systems have been developed in Vietnam [36], ontology-based legal assistants have been developed for Arabic legal information retrieval [37,38], and an AI assistant has been developed in the United Kingdom to improve access to legal information and services [39]. In Morocco, AI systems have also been developed to automate criminal court proceedings, demonstrating the potential to improve court administration by classifying court documents and automatically extracting legally relevant data [40].

Research Gap and Contributions of This Study. Recent studies have shown that combining RAG with LLMs can improve the accuracy, interpretability, and reliability of question-answering systems and AI assistants. Many RAG-based legal assistants have been developed for widely spoken, high-resource languages, including Italian, Chinese, English, Hindi, and Arabic. These studies demonstrate the effectiveness of combining semantic search with generative models in legal applications.

Despite these achievements, there are still a number of research gaps in the creation of multilingual legal data corpora and models for low-resource languages such as Kazakh. Particular examples include the following:

First, most legal RAG systems are designed for high-resource languages, and low-resource languages are still poorly represented. The number of studies investigating the use of LLM technologies in the legal sphere for the Kazakh language is limited, including only the work of [41] in internationally indexed publications. Taking into account all the features of natural language, developing or identifying effective algorithms for it, preparing data sets, analyzing the capabilities of new technologies, and developing systems and applications requires a lot of research.

Secondly, the legal environment of Kazakhstan uses both Kazakh and Russian languages. Every legal document is accessible in both languages, and all laws and regulations are published in both languages. Therefore, effective multilingual search and reasoning mechanisms are needed, while most of the existing systems are focused on one language.

Third, there is limited research investigating the impact of search-related components such as document preprocessing, segmentation strategies, embedding models, vector databases, and re-ranking methods on legal corpora. This challenge is particularly important given the complex structure and frequent updates of legal documents. The lack of publicly available legal datasets and comprehensive evaluation studies for legal texts in Kazakh and Russian limits the development and comparison of legal AI systems.

In this regard, the main idea of our proposed solution is to obtain relevant information on laws, regulations, acts, etc., from specialized sites, knowledge bases, and create an effective vector database. Using multilingual semantic search in the database, find data related to the user’s query, select the closest results from the information found, and form an answer in the form of a conclusion based on the data determined to be closest to the query using LLM.

3. Methods

Our research methodology is aimed at creating the intellectual core of a multilingual legal assistant powered by artificial intelligence, as stated in the objective of our work. We employ methods of comparative analysis, design, data storage and processing, software development, and testing. The study examines the legal assistant’s architecture and its internal structures, which are designed to answer questions based on legal documents. Specifically, we create a legal dataset, perform primary data processing, create a vector database, conduct semantic searches across vector databases, reconstruct user queries using an LLM on top of a semantic search system, and justify and explain search results. Priority is given to factual accuracy, control, and legal correctness, beginning with the initial processing of a large volume of legal documents and passing them through several stages of semantic processing, ensuring that the answers obtained are based on legal acts.

3.1. System Architecture

The architecture of the AI assistant consists of several components: a user-interaction Telegram bot, auxiliary utilities, a reasoning and generation layer, a semantic retrieval layer, a vector database, and the initial dataset. The architecture and workflow of the AI assistant are shown in Figure 1.

Figure 1. AI assistant architecture and workflow.

The user interacts with the AI assistant via a Telegram bot. Its task is to send user questions to the assistant utilities and present the received answers.

Auxiliary utilities. After receiving a user request, the auxiliary utilities begin processing it and notify the user of the processing status. The system determines whether the request is text or audio. A text request is passed to the language-detection module. An audio request is passed to the speech-recognition module, where it is converted into text using speech-to-text (STT). The resulting text is then passed to the language-detection module. To identify the user’s language and process the request, the system uses the redis (redis.py) and bot_request (bot_request.py) modules. Redis is used as a fast cache for temporarily storing user data, such as language, the latest message, and session state. Its main functions include writing and retrieving values, storing hash values, working with lists, and clearing the database. It follows the singleton pattern, meaning that only one Redis client instance is used within the bot. The request text is then corrected and cleaned before being passed to the RAG layer.

The reasoning-and-generation layer uses an LLM for contextual reasoning, inference, and natural-language interpretation. Most importantly, the model is restricted from incorporating external knowledge beyond the retrieved documents, thereby reducing hallucinations and increasing legal certainty. In this layer, questions that have undergone initial processing in the auxiliary-utilities layer are reformulated by the LLM in accordance with the prompt-engineering instructions. The LLM also reasons over the results of vector search and generates an explanatory response.

The semantic retrieval layer is implemented using the Multilingual-E5-Large, BGE-M3, and KazEmbed-V5 multilingual embedding models. These models retrieve documents from the vector database that are semantically closest to the user’s query. The resulting legal fragments are passed to the LLM together with a structured prompt that restricts the use of information unrelated to the user’s request during generation.

The data corpus in the AI assistant architecture consists of a collection of legal acts and judicial acts of the Republic of Kazakhstan. The documents undergo initial processing; then, they are divided into chunks according to the criteria of semantic search models, converted into embeddings, and stored in a vector database.

In this study, the main focus is on generating correct judgments and answers using semantic search and large language models. The implementation of semantic search consists of several tasks, such as creating a legal dataset and pre-processing text data, and identifying effective models of multilingual semantic search for the Kazakh and Russian languages. In order to generate correct judgments and answers using large language models, the results of semantic search are fed into a large language model and prompt engineering is used. Also, tests are run on several language models and the accuracy of the results is evaluated. Next, the research activities mentioned and their results are presented.

3.2. Creation of the Legal Data Corpus and Text Preprocessing

Regulatory legal acts and judicial acts of the Republic of Kazakhstan were used to form the retrieval corpus. The Kazakh and Russian versions of these documents were used in parallel. The documents were obtained from official open information resources (https://adilet.zan.kz/ and https://zan.gov.kz/) and from government legal databases. A dedicated security certificate was obtained to access the official state legal database of the Republic of Kazakhstan. To incorporate general-domain textual data alongside legal data, the Kazakh OSCAR corpus [42] and a Russian books corpus [43] were added to the original dataset. The resulting corpus contains 39 million tokens in Kazakh and 38 million tokens in Russian.

Data preprocessing. All documents are preprocessed through text extraction, cleaning, normalization, and tokenization, and are then converted into vector representations for subsequent use by the search engine. The preprocessing steps used to create the semantic-search vector database are shown in Figure 2.

Figure 2. Data preprocessing processes.

3.3. Chunking and Overlap

Regulatory legal acts usually use a strict hierarchical structure consisting of sections, chapters, articles and clauses. To preserve legal semantics and ensure accurate search, documents are stored with additional metadata and pre-processed based on the requirements of the semantic search model. Each legal document is divided into semantically coherent parts that correspond to individual articles or logically complete clauses. If the document length exceeds the size of the parts, it is divided into several parts, the relationship between which is ensured by overlap. In the proposed approach, document segmentation is carried out on the basis of a multi-level (hierarchical) strategy. The algorithm works from top to bottom: first, the text is semantically divided into paragraphs (paragraph-level). At this stage, rule-based segmentation is used based on structural markers characteristic of legal texts (for example, “section”, “chapter”, “article”). If the length of a paragraph exceeds the specified token limit, it is divided at the sentence level (sentence-level). This uses a regex-based method that takes into account punctuation marks (period, exclamation mark, question mark) and linguistic features. If the sentence itself is too long, a word-level split is performed at the final level. This multi-level approach provides stability for long texts.

The chunking process uses the sliding window principle, i.e., overlap is introduced between adjacent chunks. This overlap is determined not by words or symbols, but at the level of sentences and tokens, as a result of which the semantic integrity of the context is preserved. In addition, an adaptive packing strategy is used during chunking: each chunk is filled as much as possible up to the token limit, but not exceeding the specified limit. This approach reduces gaps and allows for more efficient use of embedding space.

This study combines token-based chunking, hierarchical segmentation, sliding-window overlap, and domain-aware rule-based preprocessing. In addition to textual content, each chunk is enriched with metadata such as the document title, article number, jurisdiction, language, and date of the latest amendment. This metadata-based approach supports precise citation in generated responses and more comprehensive search. It also enables the system to distinguish between provisions in different legal codes that have the same article number.

3.4. Semantic Search Methods

Because this study uses multilingual text data, multilingual embedding models were considered for semantic search. Multilingual semantic search is an umbrella term for tasks that involve searching for texts with similar or relevant semantics in another language [44]. Multilingual text-embedding models are specifically designed to represent words and phrases with the same meaning in different languages within a shared vector space. Many such models are currently available; the principal models considered are listed in Table 1. For semantic search, each document fragment is transformed into a dense vector representation using a multilingual embedding model.

Table 1. Multilingual text embedding models.

Model Name	Multilingual Support	Max Input Tokens	Embedding Dimension	Main Purpose of Use
multilingual-e5-large [44,45]	+	~512	1024	Semantic search
bge-m3 [26]	+	~8192	1024	Long texts, hybrid search
LaBSE [26,44]	+	~512	768	Cross-lingual search
text-multilingual-embedding-002 [46]	+	~2048	768	Text search, text analysis
Jina-Embeddings-V2 [27]	+	~8192	1024	Clustering and long-text processing
text-embedding-ada-002 [27,47]	+	~8192	1536	Semantic search, classification
Text2vec Base Multilingual [47]	+	~512	384	Fast search
KazEmbed-V5 [48]	+	~512	768	Semantic search
sentence-transformers/ paraphrase-multilingual-mpnet-base-v2 [44,49]	+	128	768	Semantic search, clustering, machine translation

As a result of the literature review, it was decided to consider and experimentally evaluate the Multilingual-E5-Large, BGE-M3, and KazEmbed-V5 models, which can encode legal texts in Kazakh and Russian into a shared semantic space. The selected models are modern multilingual embedding models designed for semantic search and RAG systems. Multilingual-E5-Large provides a solid foundation for general-purpose multilingual semantic search, KazEmbed-V5 is aimed at improving semantic search for Kazakh-language corpora, and BGE-M3 extends retrieval capabilities through hybrid dense-sparse and multi-vector representations.

3.4.1. Multilingual-E5-Large Embedding Model

Multilingual-E5-Large is a modern multilingual embedding model based on the Transformer architecture. It uses contrastive learning to map texts from different languages into a shared semantic space, enabling effective cross-lingual semantic retrieval. The model is widely used in semantic search systems and RAG architectures for multilingual document collections. It is suitable for various natural language processing tasks, including semantic search, information retrieval, RAG, and semantic textual similarity. Multilingual-E5-Large belongs to the E5 family and supports more than 100 languages, including Kazakh and Russian, which are used in our system [44,45].

The following briefly describes the generation of embeddings for documents in the retrieval corpus using Multilingual-E5-Large, the creation of a final embedding vector through a pooling operation, conversion of the user query into an embedding, and retrieval of similar document embeddings from the vector database. Let the retrieval corpus consist of a set of documents $R_c =\{X_1,X_2,X_3,\dots X_n\}$, each represented as a sequence of input tokens:

$$X_i =(x_1,x_2,x_3,\dots x_m),$$

(1)

where $X_i$ denoted as document and $\{x_1,x_2,x_3,\dots x_m\}$ denoted as tokens included in the document. M is the length of document or number of tokens in the document.

After tokenization, embedding is formed:

$$E_i =(e_1,e_2,e_3,\dots e_m),$$

(2)

The Transformer encoder computes contextual representations:

$$H_i =Trnasformer\left(E_i\right),$$

(3)

where $H_i$:

$$H_i =(h_1,h_2,h_3,\dots h_m),$$

(4)

A pooling operation (usually mean pooling) is used to generate the final embedding vector:

$$v= {\displaystyle \frac{1}{m}}{\sum }_{j=1}^m{h}_j,$$

(5)

The resulting vector represents the semantic content of the text in a multidimensional space $v \in R^{1024}$. For Multilingual-E5-Large, the embedding vector dimension is 1024.

Semantic space and similarity search. The basic idea is that semantically similar texts are located close to one another in vector space. Cosine similarity is used to estimate the similarity between query q and document d [50]:

$$sim(q,d)= {\displaystyle \frac{q\times d}{\left|\left|q\right|\right| \left|\left|d\right|\right|}},$$

(6)

where

• q—query embedding;
• d—document embedding;
• q × d—scalar product;
• |(|q|)| and ||d|| are the vector norms.

The higher the cosine similarity value, the more relevant the document is considered.

Retrieval-training features. Multilingual-E5-Large was trained using contrastive learning with positive and negative pairs. For each training instance:

• query q;
• positive document d+;
• negative document d−.

• the model minimizes the contrastive loss [50]:

$$L= -log {\displaystyle \frac{\exp \left(\frac{sim\left(q,d^{+}\right)}{\tau }\right)}{{\sum }_{j=1}^M\exp \left(\frac{sim\left(q,d_j\right)}{\tau }\right)}},$$

(7)

where

• τ—temperature parameter;
• M—number of documents in the batch.

3.4.2. KazEmbed-V5 Embedding Model

KazEmbed-V5 was developed on the basis of the Multilingual-E5 architecture and further trained on Kazakh corpora to improve retrieval quality for Kazakh-language data. Unlike general-purpose multilingual embedding models, it focuses primarily on semantic retrieval of relevant documents rather than keyword matching. The use of a Transformer encoder, contrastive learning, and cosine similarity enables the model to address multilingual semantic-search and retrieval tasks effectively in RAG systems [48].

Because KazEmbed-V5 is based on the Multilingual-E5 architecture, its token sequence (Formula (1)), embedding formation (Formula (2)), computation of contextual representations by the Transformer encoder (Formula (3)), and pooling operation (Formula (5)) can be described in the same way as for Multilingual-E5-Large. The difference is the dimensionality of the vector representing the semantic content of the text in multidimensional space $v= R^{768}$.

The semantic-search principle and mathematical basis of model training are also the same as for Multilingual-E5-Large (Formulas (6) and (7)).

3.4.3. BGE-M3 Model

BGE-M3 (BAAI General Embedding—Multilingual, Multi-Functionality, Multi-Granularity) is a modern multilingual embedding model developed by the Beijing Academy of Artificial Intelligence (BAAI). It is designed for semantic search, dense retrieval, hybrid retrieval, multi-vector retrieval, Retrieval-Augmented Generation (RAG), and cross-lingual search [26,51]. A distinctive feature of BGE-M3 is that it combines several retrieval methods within a single architecture: dense embeddings, sparse retrieval, and multi-vector retrieval. The model is characterized by the following three combinations—M³ = Multi-Lingual + Multi-Functionality + Multi-Granularity.

The model is based on an XLM-RoBERTa-type Transformer encoder architecture. The input token sequence is represented as embeddings, the Transformer encoder computes contextual representations, and a pooling operation (Formula (5)) is performed in the same manner as for Multilingual-E5-Large. In BGE-M3, the vector representing the semantic content of the text has 1024 dimensions, $v= R^{1024}$. Mean pooling is used to represent the text’s semantic content in a dense vector space. Cosine similarity is used to calculate relevance between query q and document d (Formula (6)). One of the principal features of BGE-M3 is its support for multi-vector retrieval.

Multi-vector retrieval. Instead of a single embedding vector, the model can use a set of local vectors.

$$D=(d_1,d_2,d_3,\dots ,d_k),$$

(8)

where each vector (d_i) describes a separate fragment of the document.

Similarity is calculated as

$$sim\left(q, D\right)=\max sim (q,d_j),$$

(9)

This approach improves the processing of long documents by preserving local context and increasing retrieval accuracy.

Sparse retrieval component. BGE-M3 also supports sparse retrieval, similar to BM25. A weight is calculated for each token:

$$w_j=f\left(x_j\right),$$

(10)

where

• x_j—token;
• w_i—importance score.

Sparse representation makes it possible to account for exact term matches and improve retrieval in legal and technical documents.

Hybrid retrieval. The final relevance can be calculated as a combination of dense and sparse similarity:

$$Score\left(q,d\right)=\alpha \times Dense\left(q,d\right)+\beta \times Sparse(q,d),$$

(11)

where α and β are component weights. This hybrid retrieval combines semantic similarity with lexical matching. Contrastive learning was used to train the BGE-M3 model.

The application of the models considered in semantic search consists of two stages: indexing and retrieval. The embeddings are indexed by a vector-similarity search engine, enabling efficient nearest-neighbor search across millions of legal-text fragments. When a user submits a query, its embedding is calculated and compared with the indexed vectors to retrieve the most relevant fragments. During indexing, documents are first divided into chunks, an embedding is calculated for each chunk, and the embeddings are stored in a vector database. Retrieval comprises three steps: converting the user query into an embedding; calculating similarity between the query embedding and the indexed embeddings; and returning the most relevant chunks, for example, the five nearest documents.

3.5. Transformer Attention Mechanism for Multilingual Models

The multilingual embedding models considered in this study are based on the Transformer encoder architecture, which relies on the self-attention mechanism for contextual representation learning. Below, we provide a general description of the self-attention mechanism common to all the models considered (detailed information on the self-attention mechanism can be found in the literature [52,53]). Given an input sequence: $X_i =(x_1,x_2,x_3,\dots x_m)$ The model computes three vector representations for each token:

• Query—Q;
• Key—K;
• Value—V.

These vectors are obtained through linear transformations:

$$\left\{\begin{array}{c}Q =X{W}_Q\\ K =X{W}_K \\ V =X{W}_V\end{array}\right\},$$

(12)

where W_Q, W_K, W_V—are trainable parameter matrices.

The self-attention mechanism computes attention scores between tokens using scaled dot-product attention [52]:

$$Attention\left(Q,K,V\right)=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V,$$

(13)

where

• d_k is the dimensionality of the key vectors;
• softmax normalizes the attention weights.

The input includes Q-queries and d_k-dimension keys, as well as d_v—dimension values. The layer itself performs the following important operations, calculates the dot product between the query and the keys, divides the obtained values by $\sqrt{d_k}$ and applies the softmax() function to the result. In practice, the Q matrix is built from all queries. The keys and values are also assembled into matrices K and V, respectively, and then the output matrix is calculated using Formula (14) [52].

$$MultiHead\left(Q,K,V\right)=Concat({head}_1,\dots ,{head}_h)W^0,$$

(14)

where ${head}_i=Attention({QW}_i^Q,{KW}_i^K,{VW}_i^V)$, $W_i^Q\in R^{d_{model}\ast d_k},$ $W_i^K\in R^{d_{model}\ast d_k},$ $W_i^V\in R^{d_{model}\ast d_v}$, $W^o\in R^{{hd}_v\ast d_{model}}$—linear transformation matrices, h—number of parallel levels of attention.

Multilingual-E5-Large uses Transformer self-attention to generate dense contextual embeddings optimized for retrieval-oriented semantic search. KazEmbed-V5 inherits the self-attention architecture from multilingual-E5 and further adapts contextual semantic representations for Kazakh-language retrieval tasks. BGE-M3 extends Transformer-based attention mechanisms through dense, sparse, and multi-vector retrieval representations, enabling improved long-document semantic matching.

Semantic search returns the most relevant legal fragments as follows:

$$R=(r_1,r_2,r_3,\dots r_l),$$

(15)

where

• l—number of the most relevant fragments;
• R—set of fragments retrieved by the search.

The results are then processed at the next stage using large language models.

3.6. Reasoning and Explanation Using LLMs and Semantic-Search Results

Large Language models. The most-relevant legal fragments retrieved from the vector database are supplied to LLMs for reasoning and interpretation. In the proposed architecture, GPT-5.4 and Llama-4-Scout-17B-16E-Instruct are used as generative reasoning models. The role of the LLM layer is not to search for information directly, but to synthesize and interpret legal information contextually on the basis of the retrieved evidence. Unlike traditional generative systems that rely solely on parametric knowledge, the proposed architecture uses regulatory legal texts retrieved by the search system and constrains unsupported generation, thereby improving factual accuracy and reducing hallucinations.

GPT-5.4 and Llama-4-Scout-17B-16E-Instruct are based on autoregressive Transformer architectures that use self-attention mechanisms for contextual reasoning. In general, the input sequence to the LLM in this study can be represented as follows:

$$X_{llm} =\{R,Q,S_p\},$$

(16)

The input to the LLM consists of a set of legal fragments (R) obtained as a result of the search, a user query (Q), and system prompt instructions (S_p).

In a large language model, input data are processed through several stages, including the conversion of tokens into embeddings, calculation of contextual representations through self-attention layers, and generation of contextual hidden states that encode semantic dependencies. LLMs also use multi-head self-attention mechanisms [53].

Next, the LLM autoregressively generates an explanation by estimating the conditional probability of each next token on the basis of the preceding tokens and the supplied context. This produces the following output sequence:

$$Y =\left(y_1,y_2,y_3,\dots ,y_T\right),$$

(17)

The model objective is:

$$P\left(Y|X_{LLM}\right)=\prod _{t=1}^{T}P\left(y_t|y<t, X_{llm}\right),$$

(18)

where

• $X_{llm}$—legal fragments from semantic search and prompts;
• $y<t$—previously generated token.

The probability distribution over the vocabulary at each generation step is calculated as follows:

$$P\left(y_t\right)=softmax\left(W_o{h}_t+b\right),$$

(19)

where

• $h_t$—is the hidden representation,
• $W_o$—is the output projection matrix,
• b—is the bias vector

The token with the highest probability, or a token selected according to the decoding strategy, is chosen as the next generated token.

Retrieval-grounded reasoning. In the proposed architecture, reasoning must be based on the retrieved legal evidence. Above, the user query is denoted by Q and the set of legal fragments retrieved by the search is denoted by R. The generated explanation is therefore modeled as:

$$P\left(Y\right|Q,R),$$

(20)

where Y—represents the generated legal interpretation. This formulation ensures that the reasoning process is grounded in external legal sources rather than relying exclusively on the parametric memory of the model.

A special prompt template is used to increase the accuracy of the actual data and limit the generation of the final response. The generation process consisting of the user query (Q), the set of legal fragments obtained as a result of the search (R), and the system prompt instructions (S_p) can be expressed as $Y=f(R,Q,S_p)$.

One of the main challenges of LLM-based AI assistants is the occurrence of hallucinations. The proposed RAG architecture addresses this problem by providing relevant legal passages and applying prompt constraints. The resulting textual response is generated as an interpreted legal explanation based on the retrieved evidence and is then presented to the user. This retrieval-grounded approach allows for increased transparency in the reasoning process while ensuring the reliability and legal compliance of expert systems with the requirements of legal interpretation. Unlike generative systems that rely solely on the parametric knowledge of large language models, the proposed architecture grounds its responses in normative legal sources, which is essential for legal information systems that require a high level of accuracy and interpretability.

4. Results

In this section, we present the experimental tests and the evaluation results of the selected methods. First, we describe the preparation of the dataset, including the segmentation of legal texts into chunks and the overlap strategy used during chunking. The evaluation focuses primarily on the accuracy of semantic search and the correctness of the answers generated by the large language model. The assessment is conducted using a fixed set of legal queries derived from real user scenarios. All experiments are performed under identical hardware and software conditions.

4.1. Dataset

Preprocessing of the source data ensures consistency and compatibility with the models used. Before indexing, the documents are converted into a unified JSON format and preprocessed using a structure-aware segmentation approach. Each segment corresponds to a semantically coherent legal unit, typically an article or a complete clause, and is stored together with metadata such as the document title, article number, language, and revision date. The semantic-search methods and models are then applied. This process includes text chunking, generation of vector embeddings, indexing, document retrieval, and evaluation of the results. Table 2 presents statistics for the legal-document corpus used in this study.

Table 2. Corpus statistics.

Parameters	Kazakh	Russian	Total
Number of documents	52,000	49,000	101,000
Number of total tokens	39,000,000	38,000,000	77,000,000
Average document length	750 tokens	776 tokens	763 tokens
Data storage format	JSON	JSON	JSON

4.1.1. .json Document Structure

Each JSON file is represented as an array of objects, where one object is a record of a regulatory legal act (RLA) or a judicial act. The .json document structure is organized as follows

[

{

"_id": { "$oid": "4efa1f930cf26f517f1db1a8" },

"decompressedText": " Full text of the document in unpacked form…",

"in": "0400",

"st": "new",

"voa": "official name of the document ",

"zg": "short description/title of the document ",

"lg": "kaz",

"ngr": "Z000000123_",

"processed_timestamp": 1741104477.8270261,

"source": "npa"

}

]

Description of keys:

• _id.$oid—unique record identifier (MongoDB ObjectId).
• decompressedText—full text of the regulatory legal act or judicial act in uncompressed form (string).
• in—internal document category code.
• st—record processing status (new, updated, etc.)
• voa—full official title of the act.
• zg—short title or description of the document.
• lg—document language (kaz—kazakh, rus—russian).
• ngr—internal registration number (format: prefix + identifier).
• processed_timestamp—UNIX document processing time (float).
• source—source type (npa-regulatory legal acts, court-judicial acts).

The representation of the collected data in the data frame is shown in Figure 3.

Figure 3. Representation of legal data in a DataFrame (legal dataset in kazakh language).

4.1.2. Implementation of Chunking and Overlap

In this study, chunk sizes for text segmentation were selected on the basis of the input-size limits of the embedding models. In particular, chunk length was determined using the embedding model’s tokenization mechanism rather than the number of characters or words. The structure of text data divided into chunks is shown in Figure 4.

Figure 4. Text data divided into chunks (the text data given in Kazakh).

To determine an effective chunk length, retrieval accuracy was evaluated for chunk lengths of 128, 256, 512, and 1024 tokens supported by the embedding models. In addition, experiments were conducted with overlap values from 0 to 100 tokens (0, 20, 40, 60, 80, and 100) to determine an effective overlap between chunks. The F1-score of the semantic search results was calculated for each chunk–overlap combination. The results are shown in Table 3.

Table 3. Search. The impact of chunk size and overlap parameters on semantic search quality (F1-score).

Embedding Model	Chunk Size	Overlaps
		0	20	40	60	80	100
multilingual-e5-large	128	0.686	0.688	0.690	0.705	0.720	0.722
	256	0.691	0.683	0.700	0.718	0.730	0.742
	512	0.736	0.743	0.789	0.820	0.831	0.847
	1024	0.741	0.764	0.792	0.843	0.860	0.876
BGE-M3	128	0.710	0.710	0.710	0.712	0.720	0.721
	256	0.720	0.722	0.728	0.731	0.737	0.739
	512	0.739	0.740	0.740	0.756	0.765	0.768
	1024	0.702	0.721	0.740	0.750	0.764	0.765
KazEmbed-V5	128	0.695	0.690	0.692	0.695	0.697	0.700
	256	0.695	0.700	0.724	0.739	0.754	0.760
	512	0.723	0.743	0.780	0.800	0.829	0.833

The results of the sensitivity analysis showed that the chunk size and overlap parameters significantly affect the quality of semantic search. In general, as the overlap value increases, the F1-score of all three embedding models tends to increase. This phenomenon can be explained by the preservation of contextual information between neighboring chunks, which reduces the probability that important legal information will be separated at chunk boundaries.

The highest result for the multilingual-e5-large model was obtained with a combination of 1024-token chunks and 100-token overlap (F1 = 0.876). Moreover, when increasing the chunk size from 512 to 1024 tokens, the F1-score increased from 0.847 to 0.876. This indicates that the larger chunk size for this model allows a wider context to be stored, improving the quality of semantic search. In contrast, the highest F1-score for the BGE-M3 model was observed with a 512-token chunk and 100-token overlap (F1 = 0.768). In the case of using a 1024-token chunk, the result decreased to 0.765. Therefore, for BGE-M3, excessively increasing the chunk size does not further improve the search quality. The highest performance for the KazEmbed-V5 model was obtained with a 512-token chunk and 100-token overlap (F1 = 0.833). Compared to the 128- and 256-token chunk sizes, moving to 512 tokens led to a significant increase in search quality. This result indicates that the 512-token chunk size is effective for KazEmbed-V5.

The effect of the chunk size and overlap parameters on the F1-score is graphically shown in Figure 5. The results show that with increasing overlap, the F1-score increases in most cases and the optimal chunk size varies depending on the embedding model.

Figure 5. The effect of chunk size and overlap parameters on F1-score (the highest accuracy values are indicated by asterisks).

4.2. Multilingual Search Results

Evaluating the performance of AI assistants in answering legal questions requires assessing both the quality of information retrieval and the accuracy of the generated answers. In this study, performance is evaluated at two complementary levels: (1) semantic retrieval quality and (2) answer generation quality. This two-level evaluation ensures that improvements in answer generation are based on accurate document retrieval rather than solely on the linguistic capabilities of the language model.

The legal-document fragments retrieved by the search system were evaluated against a set of relevant legal articles annotated by domain experts. The primary objective was to measure how effectively the system retrieved legally relevant provisions in response to a user query.

Precision and recall were calculated according to whether the retrieved fragments contained the correct legal articles or clauses. The F1 score, the harmonic mean of precision and recall, was used to provide a balanced assessment of retrieval effectiveness. This metric is particularly important in legal information retrieval because failure to retrieve a relevant article, or retrieval of an incorrect article, may lead to legally invalid conclusions. The five top-ranked results produced by semantic search were evaluated for accuracy. The average results of the multilingual retrieval models are shown in Table 4.

Table 4. Performance of Embedding Models in Semantic Search.

No	Model Name	1st Top Result Accuracy	2nd Top Result Accuracy	3rd Top Result Accuracy	4th Top Result Accuracy	5th Top Result Accuracy
1	Multilingual-E5-Large	0.876	0.874	0.866	0.863	0.862
2	BGE-M3	0.768	0.733	0.701	0.683	0.681
3	KazEmbed-V5	0.833	0.827	0.721	0.649	0.636

Although the Multilingual-E5-Large, KazEmbed-V5, and BGE-M3 models are all based on Transformer coding architectures and contrastive learning principles, they differ in terms of text chunk length, accuracy, and search strategies.

4.3. LLM Answer-Generation Results

GPT-5.4 and Llama-4-Scout-17B-16E-Instruct were used as the reasoning and interpretation layer for the retrieved search results. Answers were generated from the retrieved context and system instructions defining the model’s role as a legal analytical assistant. The prompt was designed to meet the following requirements:

• Use only the provided legal fragments;
• Cite the relevant articles and regulatory provisions;
• Avoid assumptions and unsupported conclusions;
• Clearly indicate when the available context is insufficient to support an answer.

Figure 6 presents an answer generated by a large language model using semantic-search results for the question “Meдицинaлық көмeк aлғaн кeздe нayқacтың қaндaй құқықтapы бap?” (“What rights does a patient have when receiving medical care?”).

Figure 6. Semantic search results for a given question (user query and response in Kazakh).

A set of 104 questions was specially prepared by legal experts to evaluate the answers generated by the large language models. The final list of questions was compiled through a multi-stage procedure. The first stage involved analyzing legal inquiries from individuals in Kazakhstan (2024–2026). The second stage systematized questions by legal domain: family, civil, labor, housing, tax, and administrative law. The third stage comprised expert screening to eliminate duplicates, correct inaccuracies, and broaden thematic coverage. The questions were submitted as user queries and the resulting answers were collected. Two methods were used for evaluation: assessment by legal experts and automatic evaluation using word-level F1, BLEU, ROUGE, and BERTScore-F1 metrics.

Legal experts classified the answers into three categories: fully correct; correct but requiring additional information; and incorrect. Expert assessment for responses (results) was conducted in two rounds: individual questionnaires based on relevance, significance, and completeness, followed by an in-person meeting to resolve disagreements and reach consensus. Kendall’s concordance coefficient was 0.82, indicating a high degree of unanimity and reliable results. The coefficient is a non-parametric measure of agreement among experts ranking the same set of objects. Its value ranges from 0 to 1, with interpretation based on the Landis and Koch scale: values below 0.20 indicate slight agreement, 0.20–0.40 fair, 0.40–0.60 moderate, 0.60–0.80 substantial, and 0.80 and above almost perfect agreement [54].

The evaluation results for the configurations combining GPT-5.4 with the retrieval models are presented in Table 5.

Table 5. Accuracy of GPT-5.4 configurations with semantic-search models (the numbers of responses are shown in parentheses).

No	Semantic-Search Model	Correct Answer	Incorrect Answer	Requires Supplementation and References to Legal Articles
1	Multilingual-E5-Large	88.5% (92)	9.6% (10)	1.9% (2)
2	BGE-M3	77.9% (81)	18.3% (19)	3.8% (4)
3	KazEmbed-V5	85.6% (89)	14.4% (15)	0% (0)

The evaluation results for the configurations combining Llama-4-Scout-17B-16E-Instruct with the retrieval models are presented in Table 6.

Table 6. Accuracy of Llama-4-Scout-17B-16E-Instruct configurations with semantic-search models (the numbers of responses are shown in parentheses).

No	Semantic-Search Model	Correct Answer	Incorrect Answer	Requires Supplementation and References to Legal Articles
1	Multilingual-E5-Large	82.6% (86)	14.4% (15)	2.88% (3)
2	BGE-M3	77.9% (81)	20.2% (21)	1.9% (2)
3	KazEmbed-V5	83.6% (87)	13.5% (14)	2.88% (3)

Figure 7 presents the collected and verified question-and-answer results.

Figure 7. Dataset compiled from the questions and answers (here are questions and answers in Russian as one of the results).

Several widely used natural language generation metrics, such as F1 (word level), BLEU, ROUGE-1, ROUGE-L, and BERTScore-F1, were used to objectively assess the quality of the responses generated by the RAG system. These metrics allow for comparative analysis and evaluation of the responses generated by the system with the results verified by legal experts.

The word-level F1 metric measures the harmonic mean of precision and recall at the word level and indicates the degree of lexical overlap between generated and reference answers. BLEU evaluates n-gram overlap and is often used to measure the lexical similarity of generated text. ROUGE-1 evaluates unigram overlap, while ROUGE-L evaluates the longest common subsequence and therefore reflects structural similarity between texts. BERTScore-F1 uses contextual embeddings from Transformer-based language models to measure semantic similarity between generated and reference answers. This metric is particularly important for legal question-answering systems because legally correct answers may be expressed using different wording while preserving the same meaning [R48].

The combined use of lexical metrics (word-level F1, BLEU, and ROUGE) and a semantic metric (BERTScore) provides a more comprehensive evaluation of the proposed RAG system. The lexical metrics quantify the extent to which generated answers reproduce key legal terms and concepts, whereas BERTScore evaluates semantic consistency. Together, these metrics provide complementary information on the quality, correctness, and semantic consistency of the generated answers. The results are presented below. Table 7 reports word-level F1, BLEU, ROUGE-1, ROUGE-L, and BERTScore-F1 for GPT-5.4 configurations, and Table 8 reports the same metrics for Llama-4-Scout-17B-16E-Instruct configurations.

Table 7. Results for GPT-5.4 configurations with semantic-search models.

Metrics	Multilingual-e5-Large	BGE-M3	KazEmbed-V5
F1 (word level)	0.231	0.179	0.224
BLEU	0.081	0.034	0.072
ROUGE-1	0.248	0.181	0.231
ROUGE-L	0.164	0.102	0.149
BERTScore_F1	0.887	0.842	0.873

Table 8. Results for Llama-4-Scout-17B-16E-Instruct configurations with semantic-search models.

Metrics	Multilingual-e5-Large	BGE-M3	KazEmbed-V5
F1 (word level)	0.176	0.151	0.201
BLEU	0.033	0.016	0.0615
ROUGE-1	0.182	0.156	0.206
ROUGE-L	0.108	0.08	0.135
BERTScore_F1	0.836	0.82	0.84

In the GPT-5.4 model, reasoning is implemented through multi-stage interpretation of retrieved context, which has a high ability to logically identify regulatory rules and generate structured explanations. This model has shown high stability when processing complex legal queries with cross-references to regulatory acts and ambiguous legal statements.

The Llama-4-Scout-17B-16E-Instruct model was used as an open, multilingual reasoning model with a Mixture-of-Experts (MoE) architecture optimized for long-context inference and the interpretation of large volumes of textual data. Owing to its extended context window, the model efficiently processes multiple legal document fragments while maintaining response coherence and contextual consistency. The use of an open model also enables local deployment and greater control over the processing of legally sensitive data.

Ablation experiment results. To compare the results of the RAG system with the capabilities of individual LLMs, separate tests were conducted on the GPT-5.4 and Llama-4-Scout-17B-16E-Instruct models. The 104 questions used above were set as queries to these LLM models, and the results were evaluated using the F1 (Word Level) BLEU, ROUGE-1, ROUGE-L and BERTScore-F1 metrics. The test results are shown in Table 9.

Table 9. Results of standalone LLMs without semantic retrieval.

Metrics	GPT-5.4	Llama-4-Scout-17B-16E-Instruct
F1 (word level)	0.311	0.204
BLEU	0.061	0.029
ROUGE-1	0.138	0.128
ROUGE-L	0.127	0.121
BERTScore_F1	0.718	0.673

Comparison of the individual LLM results with the results of the RAG configurations presented in Table 7 and Table 8 shows that the semantic search component has a significant impact on response quality. The most consistent change was observed in the BERTScore-F1 metric: compared to the value of 0.7181 in the standalone configuration for the GPT-5.4 model, it increased to values of 0.887, 0.842, and 0.873 in the RAG configurations based on Multilingual-e5-large, BGE-M3, and KazEmbed-V5, respectively. This represents a relative increase of 23.5%, 17.3%, and 21.6%, respectively. For the Llama-4-Scout-17B-16E-Instruct model, the BERTScore-F1 increased from 0.6732 to 0.836, 0.820, and 0.840, respectively, corresponding to a relative increase of 24.2%, 21.8%, and 24.8%. Thus, the addition of the semantic search component provided a consistent improvement in semantic matching across all three embedding models for both LLMs.

These results quantitatively confirm the contribution of the semantic search component in the proposed RAG architecture. In addition, the non-uniformity of changes in lexical overlap-based metrics such as Word-Level F1, BLEU, and ROUGE can be explained by the fact that generated responses may be lexically different from the reference responses, but semantically close.

5. Discussion

According to the main findings of this study, the proposed AI legal assistant for processing legal texts in Kazakh and Russian demonstrated strong performance. The experimental results indicate that the effectiveness of a legal RAG system depends not only on the retrieval model but also on the interaction between the retrieval and generation components.

Evaluation of the semantic-search models. Multilingual-E5-Large achieved the highest retrieval accuracy (0.876), outperforming BGE-M3 (0.768) and KazEmbed-V5 (0.833). These results indicate that Multilingual-E5-Large is more effective at identifying relevant legal fragments in the bilingual Kazakh–Russian legal corpus. Its strong performance may be attributed to its multilingual training strategy and robust semantic-representation capabilities.

Among the combinations of LLMs and semantic-search models, GPT-5.4 with Multilingual-E5-Large achieved the highest overall performance, with an accuracy of 88.7%. By contrast, the Llama-4-Scout-17B-16E-Instruct with Multilingual-E5-Large configuration achieved 82.6%. The highest accuracy among the Llama-4-Scout-17B-16E-Instruct configurations, 83.6%, was achieved with KazEmbed-V5. Although Multilingual-E5-Large retrieved more relevant documents on average, KazEmbed-V5 appears to provide context better aligned with the linguistic characteristics of Kazakh legal texts, enabling the LLM to produce more accurate answers. The automatic evaluation metrics support this observation: among the LLaMA-based configurations, KazEmbed-V5 achieved the highest BLEU (0.0615), ROUGE-1 (0.206), ROUGE-L (0.135), and BERTScore-F1 (0.840) scores. These results indicate strong lexical and semantic similarity between the generated and reference answers.

The results are consistent with those of Rakhimova et al. [41], who showed that GPT-4o mini outperformed LLaMA-based models in Kazakh legal question-answering tasks. The main difference is that their study fine-tuned several large language models, including GPT-4o mini, LLaMA, KazLLM, Gemma, Phi, Qwen, and Mistral, whereas our solution is based on Retrieval-Augmented Generation. In their study, GPT-4o mini achieved ROUGE-1 = 0.309 and ROUGE-L = 0.263 and received the highest expert-assessment score, indicating strong legal-reasoning capability. A similar trend was observed in our experiments. The GPT-5.4 with Multilingual-E5-Large configuration achieved the best automatic evaluation results, with word-level F1 = 0.248, BLEU = 0.081, ROUGE-L = 0.164, and BERTScore-F1 = 0.887, outperforming all Llama-4-Scout-17B-16E-Instruct configurations. Expert evaluation also indicated that GPT-based models performed better: GPT-5.4 combined with Multilingual-E5-Large achieved an answer accuracy of 88.7%, whereas the best Llama-4-Scout-17B-16E-Instruct configuration achieved 83.6%. These findings are consistent with previous studies indicating that GPT-based models currently generate more accurate and semantically relevant answers in legal question-answering tasks, particularly when combined with a RAG architecture and high-quality semantic retrieval.

6. Conclusions

This study demonstrates that the adoption of modern artificial intelligence technologies in the legal domain is an important means of supporting lawyers’ professional activities, expanding access to legal information, and providing legal services to citizens. AI-powered legal assistants can improve the efficiency of legal services by retrieving legal information, analyzing regulatory documents, and responding promptly to legal questions. Accordingly, the study included a comprehensive review of international research and the experience of various countries in applying artificial intelligence in the legal domain. The review helped refine the aims and objectives of the study and identified the need to investigate and develop the intelligent core of an AI legal assistant based on RAG.

During the study, a bilingual corpus of legal texts in Kazakh and Russian was collected from official legal-information sources (39 million tokens in Kazakh and 38 million tokens in Russian). On the basis of this corpus, a dense-vector semantic-search system was implemented using modern embedding models. The study evaluated Multilingual-E5-Large, KazEmbed-V5, which is adapted for Kazakh, and the multilingual BGE-M3 model. The experimental results showed that Multilingual-E5-Large achieved the highest retrieval accuracy (0.876), outperforming BGE-M3 (0.768) and KazEmbed-V5 (0.833). The most relevant documents identified through semantic search were then supplied to GPT-5.4 and Llama-4-Scout-17B-16E-Instruct, which generated answers to legal questions. The evaluation questions were specially prepared by professional legal experts. The answers generated were assessed both by experienced lawyers and automatically using word-level F1, BLEU, ROUGE-1, ROUGE-L, and BERTScore-F1. These two evaluation approaches produced broadly consistent results. Expert assessment is an important condition for ensuring the reliability and accuracy of legal AI systems. On the basis of this research and development, an AI legal assistant was created and integrated into the “Adal Azamat” legal services platform (https://legal-digital.kz/en (accessed on 12 July 2026)), where it is available on the main page under the name “AI Legal Consultant.”

The RAG-based AI legal assistant proposed in this study demonstrated promising performance in multilingual legal question-answering tasks. The experimental results indicate that combining semantic retrieval with the reasoning capabilities of large language models enables effective legal information retrieval and the generation of high-quality answers to legal questions. However, the practical deployment of such a system requires continuous expert oversight, regular updates of the legal knowledge base, and further improvements to both the retrieval and legal reasoning components. Future research will focus on expanding and continuously updating the legal knowledge base, including the integration of legal knowledge graphs, as well as enhancing the retrieval and answer-generation components to further reduce errors. In addition, comprehensive evaluations involving legal experts and realistic legal scenarios should be conducted to assess the system’s performance in practical applications.

7. Limitations

Despite the promising results obtained in this study, several limitations can be noted.

First, the developed legal AI assistant relies on a static legal corpus collected from official legal information resources. Although the corpus contains more than 101,000 legal documents and 77 million tokens in Kazakh and Russian, it may not fully reflect newly adopted legislation, amendments, or emerging legal practices. Therefore, the corpus needs to be continuously updated to maintain the relevance and accuracy of the system.

Second, the answer-generation component uses general-purpose large language models (GPT-5.4 and Llama-4-Scout-17B-16E-Instruct) that were not developed specifically for the legal domain of Kazakhstan. Consequently, despite the use of Retrieval-Augmented Generation, the generated answers may occasionally contain inaccuracies, omissions, or hallucinations.

Finally, due to the time-consuming nature of legal expert evaluation, the expert review was conducted on a limited set of responses. Future research should include a larger dataset for expert evaluation and an analysis of inter-annotator agreement to further assess the reliability and robustness of the system.

These limitations provide directions for future improvements, including the continuous updating of the legal corpus, the integration of diverse legal knowledge sources, the adaptation of large language models to the legal domain, and a more comprehensive involvement of legal experts in evaluating system performance. Another important consideration is that the proposed AI assistant is not a substitute for professional legal advice and should be deployed with appropriate warnings and control mechanisms.