LLM Performance in Low-Resource Languages: Selecting an Optimal Model for Migrant Integration Support in Greek

Abstract

The integration of Large Language Models (LLMs) in chatbot applications gains momentum. However, to successfully deploy such systems, the underlying capabilities of LLMs must be carefully considered, especially when dealing with low-resource languages and specialized fields. This paper presents the results of a comprehensive evaluation of several LLMs conducted in the context of a chatbot agent designed to assist migrants in their integration process. Our aim is to identify the optimal LLM that can effectively process and generate text in Greek and provide accurate information, addressing the specific needs of migrant populations. The design of the evaluation methodology leverages input from experts on social assistance initiatives, social impact and technological solutions, as well as from automated LLM self-evaluations. Given the linguistic challenges specific to the Greek language and the application domain, research findings indicate that Claude 3.7 Sonnet and Gemini 2.0 Flash demonstrate superior performance across all criteria, with Claude 3.7 Sonnet emerging as the leading candidate for the chatbot. Moreover, the results suggest that automated custom evaluations of LLMs can align with human assessments, offering a viable option for preliminary low-cost analysis to assist stakeholders in selecting the optimal LLM based on user and application domain requirements.

1. Introduction

Large Language Models (LLMs) are increasingly powering sophisticated applications, including conversational agents capable of remarkable fluency and comprehension [1]. While these models excel when trained on vast datasets typical of high-resource languages like English, their performance often diminishes significantly for low-resource languages such as Greek [2]. This disparity presents a major obstacle for deploying LLM-based solutions in contexts requiring high linguistic precision and domain-specific knowledge, particularly where under-represented languages are crucial. For instance, supporting migrants navigating complex administrative systems in Greece demands accurate and contextually appropriate communication in Greek, a capability many general-purpose LLMs currently lack due to data scarcity and limited optimization for the language’s unique grammatical and syntactical structures.

The challenge of effectively leveraging LLMs for specialized, low-resource language applications is central to efforts like the SALLY project, which aims to develop an intelligent chatbot providing essential information to migrants in Greece [3]. In Greece, as in many other countries, public services, including those related to healthcare and residency, are predominantly provided in Greek and require some level of communication in the local language [4]. This monolingual approach creates a significant structural barrier for migrants who often lack proficiency in the language. Therefore, migrants often face significant language and bureaucratic barriers, necessitating AI tools that can deliver precise, reliable, and empathetic support in Greek. However, the tendency for LLMs trained predominantly on English corpora to produce inaccurate or irrelevant responses in this specific context, highlights a critical gap. Existing benchmarks often focused on English or general tasks and providing insufficient guidance on LLM suitability for Greek-language migrant support, which underscores the need for targeted evaluations.

This paper undertakes a focused examination of LLM performance in the context of Greek, crucial for migrant integration in Greece. Moving beyond general-purpose benchmarks, the work centers on realistic, domain-relevant tasks reflecting the informational needs of migrants, such as understanding administrative procedures or interpreting official texts. The goal is to identify the most capable LLM for integration into the SALLY project’s framework, ensuring optimal performance before deployment. The evaluation involves both human assessments from experts dedicated to social impact and technological advancements, and structured LLM-based self-evaluation, offering a robust comparison of six State-of-the-Art models. Notably, the alignment between human and machine evaluations suggests that LLMs, when appropriately guided, can serve as effective preliminary evaluators in similar settings. By identifying Claude 3.7 Sonnet as the most suitable model for integration into the SALLY chatbot, the study provides practical insights for deploying LLMs in sensitive, linguistically complex applications, while also contributing to emerging methodologies for model selection in low-resource and socially impactful domains.

More specifically, the contributions of our work can be summarised as follows:

• We present a comprehensive evaluation of State-of-the-Art LLMs on Greek-language tasks situated within real-world migrant integration scenarios. To the best of our knowledge, this is the first study to systematically assess the performance of LLMs in addressing the unique linguistic and contextual challenges faced by migrants in Greek-speaking environments, offering novel insights into their applicability for supporting migrant integration.
• We introduce a dual-layered evaluation framework that combines human expert judgment with structured LLM self-evaluation, allowing for both qualitative depth and scalability.
• We offer practical insights into model performance variability across different dimensions in migrant assistance, using four key task types: reading comprehension, summarization, grammar and syntax correction, and emotion detection.
• We introduce an evaluation dimension that accounts for the emotional and psychosocial aspects of migrant communication, reflecting the need for responses that are not only accurate but also empathetic and contextually appropriate, an often overlooked factor in LLM assessment for socially sensitive domains.
• We explore the viability of LLMs as evaluators themselves, highlighting their potential as low-cost, preliminary screening tools that show strong alignment with human expert assessments.

The rest of the paper is structured as follows: Section 2 presents related work relevant to our research. Section 3 describes the specifics of the methodology followed to assess the performance of LLMs. Section 4 presents the results we obtained, while Section 5 discusses the findings. Finally, Section 6 concludes our work.

2. Background

The growing demand for context-aware, domain-specific AI systems has highlighted the limitations of general-purpose LLMs in low-resource language settings and sensitive application areas [5]. While many LLMs perform well in English and high-resource languages, their effectiveness often deteriorates when applied to linguistically complex or under-represented languages such as Greek. This performance gap becomes particularly problematic in high-stakes domains like migrant support, where communication must be not only accurate but also empathetic and culturally appropriate.

The SALLY research initiative addresses this gap by developing an intelligent chatbot designed to assist Third Country Nationals (TCNs) with accessing information and navigating bureaucratic procedures in Greece. To meet this goal, SALLY integrates generative AI with structured data, domain knowledge, and emotional sensitivity, all operating in Greek. However, the success of such a system hinges on the ability to rigorously evaluate LLM performance under real-world conditions and linguistic constraints. More specifically, deploying generative chatbots in sensitive domains entails several practical and societal risks. These systems can produce inaccurate, misleading, or overly general responses, which may be problematic in high-stakes domains, such as legal, healthcare support or minority groups, such as migrants [6,7]. Furthermore, the data used to train such models often contain embedded social biases, potentially leading to culturally insensitive or discriminatory outputs [8]. To address these concerns, SALLY has been designed as a domain-driven model. At its core, SALLY combines traditional information retrieval with generative AI through Retrieval-Augmented Generation (RAG), pulling data from the Greek Ministry of Migration and Asylum’s official website (migration.gov.gr, accessed on 21 May 2025) to craft informed replies grounded in reliable sources, while the involvement of professionals and experts working directly with migrants allows for more controlled and reliable information delivery.

A defining feature of SALLY is its focus on Greek-language support, a necessity driven by Greece’s role as a primary entry point for TCNs. While some migrants rely on widely spoken languages like English, many require assistance in Greek to navigate administrative processes, access services, and engage with local communities, key steps toward integration. In addition, sociolinguistic studies in migratory contexts have highlighted the complex, multilingual realities that migrants navigate, often relying on a mix of native, host-country, and intermediary languages depending on the setting and purpose of communication [9]. In designing SALLY, these considerations informed the emphasis on clarity, tone, and contextually appropriate host language use, aiming to reduce communicative barriers. To address this, we evaluated six LLMs to assess their ability to process and generate Greek content effectively, providing insights into their suitability for SALLY’s goals.

In the following, we situate our work within recent efforts in multilingual LLM benchmarking, domain-specific chatbot development, and LLM-based evaluation frameworks. These studies inform our methodological choices and underscore the novelty of our contribution to the field.

2.1. Multilingual LLM Evaluation

Evaluating LLMs is a challenging task across the research community, with humans typically being used as evaluators. One example is the Chatbot Arena platform [10] which provides a novel, State-of-the-Art approach through crowdsourced human preferences in a pairwise comparison system. With over 1 million votes, it has become a credible, open-access benchmark in the LLM community, using robust statistical methods to ensure accurate model ranking. Similarly, the BOSS benchmark [11] offers a more comprehensive evaluation suite for NLP models, extending beyond typical tasks like classification to assess models on diverse challenges, including those faced by models like LLaMA and GPT-3.5-turbo. Another study [12] assesses four LLMs across five tasks relevant to undergraduate computer science students, providing qualitative and quantitative evaluations to measure their effectiveness. In [13], the authors evaluate a GPT-powered chatbot trained on COVID-19 data, using TF-IDF, BERT, BioBERT, and USE to filter responses.

While existing evaluation frameworks, such as the Chatbot Arena and the BOSS benchmark, have advanced the assessment of LLMs, they remain mainly focused on high-resource languages and general-purpose scenarios. Similarly, task-specific studies, such as those evaluating LLMs for education or domain-restricted dialogue systems like COVID-19 chatbots, offer useful insights but often rely on predefined metrics or filtered content that may not reflect the linguistic and sociocultural complexity of real-world users. In contrast, the present study situates LLM evaluation within a socially impactful, low-resource context, emphasizing the Greek language and migrant integration needs. By designing tasks rooted in trusted governmental content and combining expert human ratings with LLM self-assessment, the study provides a more nuanced, application-driven perspective on model suitability. This dual evaluation strategy not only complements but also extends prior work by offering a methodology for assessing LLMs in under-represented languages and mission-critical domains, like migration.

2.2. LLMs as Evaluators

Studies have also explored the use of LLMs as evaluators. One study [14] assesses LLM-generated reading comprehension materials by comparing them to human-written counterparts through both automated and human evaluations. The results show that LLM-generated materials are rated higher than human written ones, with similar findings irrespective of the type of the evaluation. Another study [15] examines whether detailed prompting or simpler methods like model perplexity align better with human judgments in natural language generation tasks. This study introduces a taxonomy that encompasses four evaluation categories: content, relevance, integrity, and engagement—tested in multiple benchmark datasets. BIG-bench benchmark [16] evaluates large language models across 204 diverse and challenging tasks, with human expert raters providing a baseline for performance comparison. Additionally, study [17] compares the topic detection and naming capabilities of LLMs to human participants, revealing significant differences and emphasizing the need for more nuanced evaluation metrics. Study [18] tests the sensitivity of LLMs as evaluators and shows that LLM-based evaluators exhibit confusion between evaluation aspects, and in [19], researchers shed even more light on the limitations of LLMs as evaluators, demonstrating their bias towards their own generations. Finally, EvaluLLM [20] is an application that allows users to set up, run, and review evaluations using LLMs, while [21] introduces EvalGen, which is a tool that aids in evaluating LLM outputs by using LLMs to generate and suggest evaluation criteria and assertions.

Recent efforts have also examined the use of LLMs as evaluators of other LLMs. Truong et al. argue for human alignment in LLM evaluation pipelines, showing that model-generated evaluations can diverge from human judgment without careful design [22]. In follow-up studies, Fazzinga et al. and Lee et al. emphasize how structured, context-aware evaluation frameworks, such as those used in ACME, a chatbot supporting migrant counseling in Europe, can improve both system transparency and user satisfaction [23,24]. These insights validate our methodological choice to compare human expert assessments with LLM self-evaluations under a shared rubric, a hybrid approach that enables scalable pre-screening while preserving human oversight. The present study introduces a human–machine evaluation framework tailored to Greek, a low-resource language, and rooted in migrant integration tasks. By applying a shared, expert-informed rubric to both human and LLM evaluators, the study not only confirms strong alignment between the two but also demonstrates how LLMs can reliably support preliminary evaluations in socially critical domains. This dual approach, grounded in realistic content, offers a model for ethically conscious LLM selection in under-represented language contexts.

2.3. Evaluation in Socially Sensitive and Low-Resource Contexts

Recent research has increasingly investigated the performance of LLMs in socially sensitive contexts, shedding light on their strengths and limitations in low-resource environments. Bubaš et al. and Lynch et al. conducted evaluations of conversational AIs and narrative generation using LLMs in under-resourced language contexts, including Croatian and sentiment analysis of ChatGPT (GPT 3.5/4) outputs, uncovering performance inconsistencies, translation artifacts, and reduced narrative fidelity [25,26]. These findings underline the importance of evaluating LLMs not just through general benchmarks but within the actual linguistic and cultural contexts they are intended to serve. Such concerns are especially acute in domains requiring high accuracy and sensitivity, such as migrant integration.

In parallel, domain-specific LLM evaluations have emerged as a critical area of study. Roumeliotis et al. reviewed performance characteristics of OpenAI models across different information domains, noting issues like hallucinations in medically oriented prompts [27]. Similarly, Branda et al. demonstrated that chatbot designs tailored to public health emergencies improve trust and effectiveness, particularly when emotional support and factual accuracy are interdependent, a finding directly relevant to migrant support systems like ours, where regulatory clarity must be delivered with empathetic tone [28].

All in all, the current landscape in assessing the performance of LLMs, emphasizes the necessity of tailored evaluation protocols for LLMs operating in low-resource, socially critical domains. Our work builds upon this foundation by introducing a Greek-language, domain-grounded benchmark for LLM evaluation in migrant support tasks and by validating an emotion-sensitive evaluation dimension that reflects real-world communication needs. In doing so, we address a documented gap in the literature and offer a replicable framework for assessing LLMs in linguistically and socially complex environments.

3. Method

This study evaluated six LLMs to determine their suitability for SALLY (see Section 3.2 for more details). The evaluation utilized authentic content sourced from the Greek Ministry of Migration and Asylum website (https://migration.gov.gr/), ensuring relevance to the real-world scenarios SALLY will encounter. Given Greek’s pivotal role in enabling migrants to access public services, employment, and social opportunities, coupled with its under-representation in LLM training datasets, the assessment prioritized Greek-language performance. To achieve a comprehensive analysis, we employed a dual methodology combining human evaluations with automated LLM self-evaluations, offering both practical and technical perspectives on model capabilities. The research protocol followed in this work has obtained an ethical approval from the Research Ethics Committee of the Aristotle University of Thessaloniki, Greece (101/23-10-2023, 272274/2023).

3.1. Evaluation Scope

The evaluation (Figure 1) focused on four criteria, carefully selected to reflect SALLY’s core functionalities and the essential skills needed for effective migrant support. These criteria, along with their specific aims, are:

• Reading Comprehension: Evaluates the LLM ability to interpret intricate official texts, such as asylum regulations, healthcare eligibility rules, or residency permit requirements, ensuring accurate responses to procedural queries [29]. For instance, a prompt might ask, “What are the eligibility criteria for temporary residency?” based on a 300-word legal excerpt, testing precision in extracting key conditions.
• Text Summarization: Assesses the capacity to distil verbose content into concise, actionable outputs, a critical skill for migrants overwhelmed by dense bureaucratic documents. An example task involved summarizing a 500-word asylum policy into a 50-word overview, preserving essentials like deadlines and contact points.
• Grammar & Syntax: Measures linguistic accuracy and coherence in Greek, vital for credible and understandable responses. This included correcting errors in sentences like “She no have permit” to “She doesn’t have a permit”, focusing on verb conjugation, agreement, and preposition use.
• Emotion Detection: Assesses the model’s ability to recognize and appropriately respond to emotional cues commonly found in migrant communications, such as anxiety, frustration or confusion [24,30]. For instance, a user message expressing distress over prolonged asylum procedures was used to test whether the system could generate a response that conveys understanding, reassurance, and emotional sensitivity. This task was assessed in collaboration with experts from the Center for Research and Technology Hellas (CERTH) and PRAXIS, an independent Civil Society Organisation in Greece, both with extensive experience in supporting vulnerable social groups and migration-related challenges.

The selection of these four evaluation criteria is grounded in empirical research and practical considerations for developing effective AI-powered support systems. The emphasis on reading comprehension and text summarization capabilities stems from the documented challenges in clinical decision support systems, where accurate interpretation and concise presentation of complex information have proven crucial for user trust and adoption [29]. The inclusion of grammar and syntax as key criteria is particularly relevant for bilingual systems, as specialized LLMs have demonstrated superior performance in maintaining linguistic accuracy compared to general-purpose models. The focus on emotion detection is supported by recent research in the domain [24], as well as in healthcare and crisis-response chatbots, where recognizing and responding to users’ emotional states, such as anxiety, stress, or frustration, has been shown to significantly enhance perceived helpfulness and trustworthiness [31]. This highlights the importance of emotional intelligence and tone sensitivity in contexts requiring empathetic engagement. These criteria align with current best practices in LLM development, where specialized models designed with specific user needs in mind consistently outperform general-purpose LLMs across multiple evaluation metrics [32].

In total, 42 tasks were curated by experts in the areas of social sciences and migration (PRAXIS), linguistics (ENL-AUTH) and technology (CSD-AUTH, CERTH) based on representative content from migration.gov.gr. These tasks were designed to mirror potential real-world migrant inquiries and were distributed across the criteria: 12 for Reading Comprehension (e.g., interpreting legal texts), 18 for Grammar & Syntax (e.g., correcting flawed Greek sentences common in migrant communication), 6 for Summarization (e.g., condensing lengthy policy documents), and 6 for Emotion Detection (e.g., identifying emotional tones in user questions). For all evaluations, human and machine, responses generated for these tasks were scored on a standardized 1–5 Likert scale, where 1 indicated poor quality (inaccurate, unclear, irrelevant) and 5 denoted exceptional performance (accurate, clear, fully relevant, and well-formed). Example prompt templates are presented in

Table 1. Example Prompt templates for the evaluation tasks.

Evaluation Task	Prompt Template
Reading Comprehension	Given the following text, answer the question based only on the provided content. Text: [TEXT], Question: [QUESTION]
Text Summarisation	Given the following text, generate a concise and coherent summary that captures the key points and essential details. Text: [TEXT]
Syntax and Grammar	Correct the grammar and syntax of the following text without changing its meaning or tone. Text: [TEXT]
Emotion Detection	Identify the sentiments expressed in the following text (e.g., hopeful, positive, supportive, optimistic, pragmatic, concerned, encouraging, progressive, compassionate, forward-thinking, inclusive, empowering, frustrated, solution-oriented). Return the results starting from the most plausible sentiment. Text: [TEXT]

.

The evaluation took place outside the SALLY chatbot, using domain-relevant content and questions, rather than embedding them in the chatbot framework. This choice was primarily driven by the need to perform a clean, more controlled assessment of the core capabilities of LLMs without being influenced by chatbot-specific factors, such as user interface or system integration constraints. As the primary objective of the study is to assess and compare the LLMs’ capabilities for migrant-related contexts, rather than their performance within a specific end-to-end system, we emphasize the scientific question of model suitability over engineering considerations.

All prompts provided to the language models and the corresponding model-generated answers were in Greek, as the core objective of this study is to evaluate the performance of the LLMs specifically in the Greek language across the defined tasks. To enhance understanding for a broader audience, we provide an illustrative example of the texts and responses in English translation within the main body of the paper. Furthermore, all original Greek texts used in the evaluation, along with their English translations, are available for reference in Appendices Appendix A and Appendix B. For example, one of the instantiations of the Reading Comprehension template (Table 1) was the following (the original questions were in Greek, but for clarity, we present equivalent text in English):

Given the following text, answer the question based only on the provided content.

Text: The accommodation structures provide temporary housing for third-country nationals or stateless persons who have applied for international protection in Greece. They also host family members of asylum seekers, unaccompanied or accompanied minors, and individuals belonging to vulnerable groups. These facilities ensure a stable living environment that meets basic living and social needs, as well as educational and learning requirements. Residents live under a regime of independent living while being required to follow internal regulations to maintain the smooth operation of the facility. Services provided include housing, food, legal and social information, psychological support, access to healthcare, personal hygiene supplies, clothing, Greek language lessons, and educational programs. Residents must leave the structures within 30 days of receiving a decision granting them international or subsidiary protection. For unaccompanied minors, this period starts upon reaching adulthood.

Question: How do people living in these accommodation structures experience their stay?

The prompt is then given to the six LLMs (Section 3.2) in order to collect the responses and use them for human evaluation (Section 3.3) and LLM self-evaluation (Section 3.4). Example responses for this specific instantiation that are to be evaluated are given below (the evaluators are not aware of the sources of the responses).

Response 1: Under a regime of independent living.

Response 2: Those residing in the structures are in temporary accommodation and have the opportunity to become familiar with the host country, its rights and obligations. The accommodation structures provide a stable framework that meets not only the basic living and social needs of residents but also their educational and learning requirements. Residents live independently but must follow internal regulations to ensure the facility’s smooth operation. In the case of unaccompanied minors, their transition period starts upon reaching adulthood. Residents must leave the facility within 30 days of receiving a decision granting them protection.

Response 3: Residents in the structures live under a regime of independent living but are required to follow internal regulations for the smooth operation of the facility.

Response 4: Residents in the structures live under a regime of independent living and must adhere to internal regulations to ensure smooth operation of the facility.

Response 5: People residing in the structures live independently, covering their basic living and social needs as well as their educational and learning needs. They receive services such as housing, food, information on their rights and obligations, psychosocial support, access to healthcare, personal hygiene items, clothing, Greek language lessons, and access to education and skills development programs.

Response 6: According to the text, residents live under a regime of independent living but must follow internal regulations to ensure the smooth operation of the facility. The structures provide a stable framework that meets basic living and social needs, as well as educational and learning requirements. They receive services such as housing, food, information, psychosocial support, healthcare access, essential items, Greek language lessons, and educational programs.

3.2. Selected Large Language Models

Six LLMs were selected for evaluation based on their strong performance on the Chatbot Arena leaderboard [10] at the time of the study and to represent diverse development approaches from leading organizations: (1) GPT-4o, (2) Claude 3.7 Sonnet, (3) Gemini 2.0 Flash, (4) DeepSeek V3, (5) Llama 3.3 70B, and (6) Grok 2. We focused on models that are either widely adopted, representative of different development philosophies, or accessible for academic and non-commercial use. Some, like GPT-4o and Claude 3.7, are already integrated into major platforms and widely tested in user-facing applications. Others, such as DeepSeek and Llama 3.3, offer open access, making them suitable for in-depth evaluation. We also considered variety in architecture and alignment strategies, from highly optimized proprietary systems to community-driven open-source models. The goal was to cover a spectrum of current LLM capabilities while keeping the evaluation grounded in realistic use cases.

To generate the content for evaluation, we interacted with each LLM via its official online playground using default configurations, since no adjustments were made to parameters like temperature or top-p. For each of the 42 tasks, we provided the relevant source text from migration.gov.gr along with a standardized prompt structured to mimic a migrant query (e.g., “Based on this text, how can a migrant apply for asylum in Greece?” or “What are the required documents according to this passage?”). This procedure ensured fairness, as all models responded to the identical inputs and context. The resulting set of 42 generated responses per model formed the corpus that was subsequently assessed in both the human and machine evaluation phases.

3.3. Human Evaluation

The human evaluation was conducted by seventeen native Greek-speaking individuals (11 men, 6 women). Their demographics spanned various age groups (23.5% 18–24, 29.4% 25–34, 41.2% 35–44, 5.9% 45+). Regarding educational background, 52.9% were at the PhD level, 35.3% had postgraduate (Master’s level) education, and 11.8% had an undergraduate degree. These individuals were selected for their extensive practical experience assisting TCNs with asylum processes, residency, and integration challenges, granting them deep domain expertise and familiarity with the specific linguistic needs and common issues faced by migrants. Choosing Greek speakers for this stage was essential, as their command of the language allowed for a detailed assessment of the LLMs’ handling of grammar, syntax, and reading comprehension. Their expertise secured an accurate evaluation of the models to produce well-structured and meaningful responses, which is critical for ensuring that the chatbot works as expected before being used by migrants.

This evaluation phase was conducted with domain experts rather than members of the target migrant population due to practical considerations. The chatbot is still undergoing iterative refinement, and expert users, such as professionals working in migrant support services, were better positioned to provide structured, actionable feedback. Their familiarity with typical user needs and interaction patterns enabled a more focused assessment of the system’s functionality, linguistic clarity, and coverage of relevant topics. Engaging experts allowed for a more efficient and technically informed evaluation process, helping to identify and address early design issues before involving end users in subsequent testing rounds.

Prior to the evaluation, participants attended a one-hour briefing session. This session outlined SALLY’s objectives (delivering accurate, empathetic support in Greek) and provided detailed guidelines on interpreting and consistently applying the 1–5 Likert scale across the four criteria. Sample responses illustrating different quality levels (e.g., a score of 2 for a Reading Comprehension answer missing key details vs. a score of 5 for a fully accurate one) were reviewed to calibrate judgment.

The evaluation itself spanned two weeks. Participants worked independently in a controlled setting, assessing the LLM-generated responses in a blind setup—they were unaware of which LLM produced each output they reviewed. For each response, they compared it against the original prompt and source text (where applicable) and assigned a score from 1 to 5 based on the predefined criteria: checking factual accuracy and completeness (Reading Comprehension), evaluating conciseness and preservation of key information (Summarization), verifying grammatical correctness and coherence (Grammar & Syntax), and judging the appropriateness and sensitivity of tone (Sentiment Analysis). This rigorous, expert-driven process ensured the ratings reflected not only technical accuracy but also practical utility and linguistic appropriateness within the migrant support context.

3.4. Machine Evaluation

Concurrently with the human assessment, we conducted a machine-powered evaluation where each of the six LLMs evaluated the full set of generated responses, including those produced by itself and its peers. The primary goals were to explore the potential of LLMs as evaluators in this specialized, low-resource language domain and to determine the degree of alignment between machine and expert human judgments.

Critically, for this phase, we did not use a simplified, abstract prompt for evaluation. Instead, we provided each LLM with the exact same detailed, structured questionnaire that was given to the human evaluators. This questionnaire contained the overall instructions, the 1–5 Likert scale definition, specific guidance and considerations for scoring within each of the four criteria (Reading Comprehension, Summarization, Grammar & Syntax, Sentiment Analysis), and illustrative examples.

The process involved inputting the full questionnaire text, followed by the specific LLM-generated response to be assessed, directly into the respective LLM’s online playground interface. The LLM was then prompted to follow the instructions in the questionnaire to assign a numerical score (1–5) and, where supported by the model’s interface or output style, provide brief justifications for its score. This was repeated for all responses generated across all 42 tasks for all six models, resulting in over 250 machine-generated scores per evaluating model. This evaluation used the identical set of generated responses provided to the human experts.

This approach rigorously tested the LLMs’ ability to understand and apply a complex, multi-faceted evaluation rubric designed for humans, directly within the target language and domain. By maintaining methodological consistency with the human evaluation, we aimed to obtain directly comparable results and assess the viability of using LLMs for nuanced, criteria-based quality assessment as a potentially scalable and cost-effective preliminary step in LLM selection for demanding applications like SALLY.

3.5. Example

To further illustrate the evaluation methodology, we present an additional example about responses for the Grammar & Syntax task (the original questions were in Greek, but for clarity, we present equivalent text in English). In this task, the evaluators receive a sentence with grammar or syntax errors and need to evaluate LLM responses that try to fix the errors. For the generation of the responses, the third prompt in Table 1 was used. As such, given the incorrect input sentence: “Migrants are goes Greece and not finding any helps for paper and doctors is missing”, the following six responses are generated (as mentioned, the evaluators are not aware of the sources of the responses):

• Response 1: Migrants going to Greece cannot find help with their documents or access medical care.
• Response 2: Migrants who go to Greece are not able to find help about papers and doctors.
• Response 3: Migrants who go to Greece cannot find help with their documents or medical care.
• Response 4: Migrants are going to Greece but cannot find any help with their papers, and medical care is unavailable.
• Response 5: Migrants going to Greece are not able to find help with their papers or medical attention.
• Response 6: Migrants goes to Greece but don’t finding help with their documents and no doctor can be found.

The correct LLM response, based on both human evaluation and the LLM’s self-evaluation, is the first one, generated by Claude 3.7 Sonnet, which properly adjusts verb tense, prepositions, and sentence structure. The other responses still contain errors, such as incorrect verb forms (“are goes” instead of “are going”), incorrect pluralization (“helps” instead of “help”), and unnatural phrasing (“doctors is missing” instead of “doctors are unavailable”).

It is worth mentioning that the human evaluators assessed not only the quality of responses but also their correctness across the four key criteria. For Reading Comprehension and Text Summarization, they evaluated the accuracy by comparing the LLM-generated responses against the original texts, which were extracted from the official website of the Ministry of Asylum in Greece. Specifically, for Reading Comprehension, evaluators read the original text, reviewed the given question, and assessed whether the LLM’s response reflected the content. Similarly, for Text Summarization, they compared the generated summary against the full text to verify its accuracy and completeness. For Grammar and Syntax, correctness was assessed based on the clarity and meaning of the sentences, ensuring grammatical and syntactical accuracy. Lastly, for Sentiment Analysis, evaluators examined the sentiment identified by the LLM in a given text to determine whether it correctly captured the intended emotions.

4. Results

This section presents the evaluation outcomes for the six LLMs, assessing their suitability in the SALLY chatbot. The evaluation encompasses both human assessments and machine-powered self-evaluations by the LLMs, focusing on the four criteria.

In the human evaluation (

Table 2. Human Evaluation Results (RC: Reading Comprehension, TS: Test Summarisation, SG: Syntax and Grammar, SA: Sentiment Analysis).

Model	RC	TS	SG	SA
GPT-4o	4.05 (0.68)	3.64 (0.93)	4.21 (0.60)	2.94 (0.55)
Claude 3.7 Sonnet	4.32 (0.46)	4.11 (0.78)	4.78 (0.28)	4.41 (0.61)
Gemini 2.0 Flash	3.79 (0.81)	3.58 (0.87)	3.64 (0.68)	4.23 (0.90)
DeepSeek V3	3.02 (0.90)	3.7 (0.68)	2.9 (0.83)	2.76 (0.97)
Llama 3.3 70B	2.23 (1.04)	3 (1.06)	1.47 (0.50)	3.11 (0.85)
Grok 2	3.64 (0.86)	3.64 (1.05)	3 (0.77)	3.88 (0.78)

), Claude 3.7 Sonnet led across all criteria with consistent, high-quality performance. It achieved the highest score in Grammar & Syntax (Mean = 4.78, Standard Deviation = 0.28), reflecting its ability to produce grammatically accurate and well-structured Greek responses with minimal variability, a critical asset for a bilingual chatbot serving migrants. Its Reading Comprehension score (M = 4.32, SD = 0.46) further highlights its strength in accurately interpreting complex texts, such as official migration documents. In Sentiment Analysis (M = 4.41, SD = 0.61), it demonstrated a superior capacity to detect and respond to emotional nuances, essential for empathetic interactions, while its Summarization score (M = 4.11, SD = 0.78) indicates effective distillation of information.

GPT-4o followed as a strong contender, delivering robust performance across most criteria. It scored well in Grammar & Syntax (M = 4.21, SD = 0.60) and Reading Comprehension (M = 4.05, SD = 0.68), suggesting reliability in generating accurate and comprehensible responses. However, its slightly higher standard deviations compared to Claude 3.7 Sonnet indicate less consistency, and its Sentiment Analysis score (M = 2.94, SD = 0.55) was notably lower, pointing to a limitation in emotional responsiveness. Gemini 2.0 Flash also performed admirably, particularly in Sentiment Analysis (M = 4.23, SD = 0.90), nearly matching Claude 3.7 Sonnet, though its Summarization score (M = 3.58, SD = 0.87) and higher variability suggest it struggles with consistency in condensing complex content.

Grok 2 outperformed Llama 3.3 70B in several areas, notably in Sentiment Analysis (M = 3.88, SD = 0.78), where it surpassed GPT-4o, and tied with DeepSeek V3 in Summarization (M = 3.64, SD = 1.05). However, its variability, especially in Summarization, indicates instability that could undermine its reliability for SALLY needs. DeepSeek V3 showed a surprising edge in Summarization (M = 3.70, SD = 0.68), outperforming both Grok 2 and Llama 3.3 70B, but its lower scores in Grammar & Syntax (M = 2.90, SD = 0.83) and Sentiment Analysis (M = 2.76, SD = 0.97), coupled with higher variability (e.g., SD = 0.90 in Reading Comprehension), suggest it is less suited for comprehensive tasks. Llama 3.3 70B consistently underperformed, with the lowest Grammar & Syntax score (M = 1.47, SD = 0.50) and significant variability in Reading Comprehension (M = 2.23, SD = 1.04), highlighting substantial stability issues that render it the weakest candidate.

The machine-powered evaluation (

Table 3. Machine Evaluation Results (RC: Reading Comprehension, TS: Test Summarisation, SG: Syntax and Grammar, SA: Sentiment Analysis).

Model	RC	TS	SG	SA
GPT-4o	4.41 (0.49)	4 (0)	3.83 (0.35)	3.33 (0.81)
Claude 3.7 Sonnet	4.5 (0.31)	4.83 (0.40)	4.60 (0.38)	4.66 (0.51)
Gemini 2.0 Flash	4.16 (0.40)	4.33 (0.51)	3.60 (0.25)	4.16 (0.40)
DeepSeek V3	3.58 (0.49)	3.5 (0.54)	2.82 (0.40)	3 (0.63)
Llama 3.3 70B	1.91 (0.80)	2.5 (0.83)	1.72 (0.39)	3.83 (0.75)
Grok 2	3.75 (0.41)	4.16 (0.75)	3.49 (0.34)	3.83 (0.40)

), where each LLM assessed all responses including its own, mirrored the human rankings, reinforcing the reliability of self-evaluation as a preliminary tool. Claude 3.7 Sonnet again led with outstanding scores in Summarization (M = 4.83, SD = 0.40), Sentiment Analysis (M = 4.66, SD = 0.51), Grammar & Syntax (M = 4.60, SD = 0.38), and Reading Comprehension (M = 4.50, SD = 0.31), showcasing its ability to self-assess accurately and consistently. GPT-4o maintained strong performance, particularly in Reading Comprehension (M = 4.41, SD = 0.49) and Summarization (M = 4.00, SD = 0.00), though its Sentiment Analysis score (M = 3.33, SD = 0.81) remained a weak point. Gemini 2.0 Flash excelled in Summarization (M = 4.33, SD = 0.51) and showed solid Sentiment Analysis (M = 4.16, SD = 0.40), aligning closely with its human evaluation results.

Grok 2 performed well in Summarization (M = 4.16, SD = 0.75) and Sentiment Analysis (M = 3.83, SD = 0.40), but its Grammar & Syntax score (M = 3.49, SD = 0.34) trailed the top models. DeepSeek V3 scored moderately in Reading Comprehension (M = 3.58, SD = 0.49) and Summarization (M = 3.50, SD = 0.54), but its lower Grammar & Syntax (M = 2.82, SD = 0.40) and Sentiment Analysis (M = 3.00, SD = 0.63) scores indicate limitations. Llama 3.3 70B ranked lowest, with poor performance in Reading Comprehension (M = 1.91, SD = 0.80) and Grammar & Syntax (M = 1.72, SD = 0.39), despite a decent Sentiment Analysis score (M = 3.83, SD = 0.75), underscoring its challenges in producing reliable outputs.

A comparison of average performance across the four criteria, illustrated in Figure 2, reveals identical rankings between human and machine evaluations, suggesting that LLM self-evaluation could serve as an effective preliminary assessment method. This alignment is supported by statistical analyses: Kendall’s W $(0.561,p<0.001)$ confirmed significant agreement among the 17 human evaluators, while a Mantel–Haenszel test ${\chi }^2\left(1\right)=4.98,$ $p<0.026$ indicated a linear association between human and machine scores, with higher LLM ratings corresponding to higher human ratings.

Both human and machine evaluations rated Claude 3.7 Sonnet response highest for its precise correction of verb tense, prepositions, and sentence structure, while Llama 3.3 70B response retained errors (e.g., “goes” and “don’t finding”), reflecting its lower performance. These results collectively position Claude 3.7 Sonnet as the standout model, with GPT-4o and Gemini 2.0 Flash as viable alternatives, while highlighting the limitations of Llama 3.3 70B, DeepSeek V3, and Grok 2 for SALLY requirements.

5. Discussion

The evaluation of the six LLMs offers critical insights into selecting the most effective model for the SALLY chatbot. Claude 3.7 Sonnet outperformed its peers across all four criteria, making it the standout candidate. Its exceptional proficiency in Grammar & Syntax is particularly vital for a bilingual chatbot, ensuring clear and credible communication in Greek, a language under-represented in many LLM training datasets. Additionally, its strong Reading Comprehension ensures accurate interpretation of complex official texts, while its adeptness in Sentiment Analysis supports empathetic responses tailored to migrants’ emotional needs. Gemini 2.0 Flash also demonstrated notable strengths, particularly in Sentiment Analysis and Grammar & Syntax, positioning it as a viable alternative with well-rounded capabilities suited to diverse tasks. GPT-4o, while robust in most areas, showed a comparative weakness in Sentiment Analysis, though its overall performance remains competitive.

In contrast, Grok 2, DeepSeek V3, and Llama 3.3 70B exhibited more limited suitability for SALLY requirements. Grok 2 showed promise in Sentiment Analysis but lacked consistency across other criteria, suggesting it may not reliably meet the chatbot’s multifaceted demands. DeepSeek V3 excelled in Summarization yet faltered in Grammar & Syntax and Sentiment Analysis, limiting its versatility. Llama 3.3 70B consistently underperformed, particularly in Grammar & Syntax, with significant variability that undermines its reliability for a migrant support application.

Based on these findings, Claude 3.7 Sonnet emerges as the leading choice for integration into SALLY, with Gemini 2.0 Flash and GPT-4o as potential alternatives. The final decision will hinge on balancing these performance outcomes with practical considerations such as computational resources, cost, and ease of integration. Further testing, specifically targeting Greek-language proficiency, may refine this selection to ensure optimal performance in SALLY unique context.

A key observation from this study is the striking alignment between human evaluations and the LLMs’ self-assessments. This consistency suggests that machine-powered evaluation could serve as a valuable preliminary tool, particularly for projects like SALLY that operate in low-resource languages and domain-specific settings. Such an approach could streamline initial model screening, reducing time and resource demands while providing a reliable proxy for human judgment. However, its broader applicability requires further exploration across diverse tasks and languages to confirm its robustness.

Finally, the reliance on evaluations made by experts familiar with asylum processes, residency, and integration challenges, as well as with the linguistic needs and common issues faced by migrants was a deliberate and effective choice for this phase, leveraging their deep familiarity with migrant inquiries and Greek language nuances to assess technical accuracy and practical relevance. This mirrors professional practices in language quality assessment, where native speakers with domain expertise provide authoritative baselines. Yet, this represents only an initial step. Future evaluations plan to involve migrants, the chatbot target users, to gauge usability, effectiveness, and cultural sensitivity from their perspective. This two-phase strategy ensures both linguistic precision and user-centered design, critical for SALLY success.

6. Conclusions

This paper presents an evaluation of several LLMs to explore their suitability for the SALLY chatbot, an AI agent intended to support migrant integration by providing information in Greek. Greek is considered an under-resourced language in this context, and the application domain is socially sensitive. The analysis, which combined human expert assessments with structured LLM self-evaluations, indicated that Claude 3.7 Sonnet performed strongly, with Gemini 2.0 Flash also showing notable capabilities for addressing the linguistic and informational needs relevant to this application.

This research sought to explore several aspects that may contribute to the field. The study attempted a systematic evaluation of LLM performance specifically for the Greek language within the domain of migrant integration support, aiming to address linguistic and contextual challenges faced by migrants. A dual-layered evaluation framework was employed, combining human expert judgment with structured LLM self-evaluation. The application of this hybrid methodology was explored for a low-resource language and a socially impactful field. An aspect examined in this work was an emotion detection dimension, designed with consideration for the psychosocial complexities of migrant communication. This dimension assessed an LLM’s capacity to recognize and respond to certain emotional cues, such as anxiety or frustration, which can be an important factor in developing AI for specific user groups. The research also aimed to complement generic benchmarks by grounding the LLM evaluation in domain-relevant tasks derived from official governmental sources, which may enhance the potential applicability of the findings to real-world scenarios.

This research offers several findings and methodological considerations. A benchmark of selected LLMs on Greek-language tasks, situated within migrant integration scenarios, is provided, offering insights into their applicability. The development and application of the hybrid evaluation framework are detailed, which integrated human expertise with the analytical capabilities of LLMs, guided by a shared, expert-informed rubric. The study also presents insights into the performance variability of these LLMs across four task types relevant to migrant assistance: reading comprehension of official texts, summarization of information, grammar and syntax correction in Greek, and emotion detection. A notable finding is the indication of LLM potential as preliminary evaluators in specialized contexts. The results showed a strong alignment between LLM self-assessments and human expert judgments when LLMs were provided with detailed, human-designed rubrics, suggesting their utility as a scalable screening tool. Empirical evidence is also presented indicating that carefully designed prompts, based on potential real-world migrant interactions and policy documents, may contribute to improving the faithfulness and contextual appropriateness of LLM-generated outputs in such applications.

The deployment of AI in sensitive areas such as migrant support calls for careful attention to ethical considerations, including data privacy, bias mitigation, and transparency concerning AI capabilities and limitations. Therefore, the pre-integration testing conducted in this study is an important step for assessing LLM accuracy and attempting to minimize risks before real-world application. The selected LLM, Claude 3.7 Sonnet, would require further fine-tuning and integration with moderation and fact-checking mechanisms to support the delivery of responses that are accurate, equitable, and trustworthy.

While this study focused on the Greek language due to its role in accessing services and integration processes in Greece, and acknowledging the multilingual capabilities of modern LLMs, practical constraints limited the inclusion of a broader range of languages in this phase. Future work could consider the integration of native language support, guided by user feedback and evaluation. It is also important to note that the LLM landscape evolves rapidly; these findings represent capabilities at the time of testing. Thus, adaptable evaluation frameworks and custom evaluations tailored to specific linguistic and contextual demands, such as those explored here, can be valuable for specialized projects. This work aimed to address a gap in the literature by offering a framework for assessing LLMs in linguistically and socially complex environments, potentially contributing to the development of effective and ethically considered AI solutions for migrant support and similar domains.