Exploring Data Augmentation in a Low-Resource Language Context: A Case Study on Text Generation for Reading Comprehension in Turkish

Abstract

This study presents a controlled empirical and comparative analysis of existing data augmentation techniques for text generation in Turkish, a morphologically rich, low-resource language. A collection of 265 Turkish reading passages for Grades 4 and 5 was augmented using four techniques: paraphrasing with GPT-3.5-turbo (Generative Pre-trained Transformer 3.5 Turbo), back translation (Turkish–English–Turkish and Turkish–French–Turkish) via Google Translate, synonym replacement via GPT-3.5-turbo, and random insertion via GPT-3.5-turbo. Human evaluators assessed the fluency, coherence, grammaticality, logical flow, and naturalness of the augmented datasets. Each augmented dataset, along with the original, was then used to fine-tune a Turkish GPT-2-medium model, which was evaluated using automatic metrics such as BLEU (Bilingual Evaluation Understudy), ROUGE (Recall-Oriented Understudy for Gisting Evaluation), METEOR (Metric for Evaluation of Translation with Explicit ORdering), chrF (CHaRacter-level F-score), BERTScore (Bidirectional Encoder Representations from Transformers Score), and cosine similarity. According to the human evaluation of the original and augmented datasets, the original texts received the highest ratings, followed by those generated through random insertion, paraphrasing, synonym replacement, and back translation variants, with cosine similarity results between original and augmented texts showing a comparable trend; however, the differences between methods were generally small. The results from text generation indicate that models trained on the original dataset generally achieved slightly higher performance across evaluation metrics compared to those trained on augmented datasets. Among the augmented methods, synonym replacement showed marginally better performance, followed by back translation, random insertion, and paraphrasing; however, the differences between methods were small and not statistically significant.

1. Introduction

Reading comprehension is a foundational skill that underpins students’ success across all subjects in K-12 education [1]. It serves as a gateway to academic achievement, enabling learners to engage critically with content across disciplines. Developing reading proficiency relies on students encountering a range of texts that vary in volume, genre, and complexity [2,3,4]. Research has shown that students’ reading comprehension skills benefit significantly from consistent reading practice, allowing students to build fluency, deepen their understanding, and improve their capacity to interpret and analyze information [5,6,7,8]. Teachers play a key role in supporting this development by curating diverse reading materials, scaffolding text-level interactions, and assigning comprehension questions that promote inferential and evaluative thinking. However, for students with emerging reading skills, there is often a heightened need for more accessible, level-appropriate texts and precisely tailored comprehension items. Meeting this need places considerable demands on instructional resources. Thus, teachers often face significant challenges in meeting this need because (1) locating suitable materials is time-consuming, (2) free resources are often limited in quality and relevance, (3) writing original texts and high-quality items is a highly complex task requiring linguistic expertise and pedagogical knowledge, and (4) the modern educational landscape demands the integration of digital reading resources [9].

In recent years, large language models (LLMs) have gained considerable attention in educational research for their ability to generate reading passages and comprehension items at scale, presenting promising and cost-effective alternatives to traditional text and item creation workflows [9]. These capabilities hold particular promise for reducing the burden on educators while expanding access to high-quality instructional materials. However, these models often require additional fine-tuning to align with specific educational goals, as LLMs are pretrained primarily for next-token prediction rather than task-specific educational objectives. This mismatch between pretraining objectives and downstream task requirements means that raw LLM outputs may not reliably conform to grade-level standards, discourse coherence, or pedagogical appropriateness. Furthermore, despite the recent advances, most existing research has centered on high-resource languages, such as English, leaving a substantial gap in understanding how these tools perform in low-resource language contexts, such as Turkish. This gap is particularly consequential given the growing global demand for multilingual educational technologies.

To address these gaps, our study leverages LLMs to enhance the quality of automatically generated reading passages and comprehension items in Turkish in conjunction with multiple data augmentation techniques, including paraphrasing, back translation, synonym replacement, and random insertion. The rationale for incorporating data augmentation is twofold: first, LLMs, although pretrained on large multilingual corpora, may not fully capture the specific characteristics of low-resource languages such as Turkish, particularly in the context of grade-level reading materials. Turkish, as an agglutinative language with complex morphological structures, presents unique modeling challenges that standard pretraining may not fully address [10,11]. Data augmentation can introduce linguistic variation and increase the diversity of training examples, potentially improving model robustness and generalization. Second, the systematic use of multiple augmentation strategies enables a comparative investigation of how distinct types of linguistic modifications—operating at the word, sentence, or discourse level—affect the quality of model-generated content. This comparative approach allows us to evaluate the relative effectiveness of each augmentation technique and identify which methods are best suited for improving model performance in automated text generation for low-resource educational content. Taken together, our study advances the understanding of data augmentation in a morphologically complex, low-resource language setting and offers practical guidance for building automated content generation systems for Turkish-language educational assessment.

2. Literature Review

2.1. Augmenting Textual Data

The quality of training data can play a more critical role in the performance of natural language processing (NLP) applications than the choice of the model itself [12]. Data augmentation is a technique used in NLP to artificially expand the size of a dataset to address over-fitting [13,14] and imbalance issues in datasets [15] by applying various transformations to the original data. Data augmentation refers to a set of techniques used to artificially increase the volume, quality, and diversity of a dataset by generating modified or transformed versions of existing data while preserving its underlying meaning or label [16]. This process helps improve model performance, especially when the amount of available training data is limited [17]. The augmentation techniques are frequently used in areas such as text generation [17], text classification (e.g., to create more diverse training examples) [18], machine translation (e.g., to enhance translation quality) [19], and sentiment analysis (e.g., to generate a variety of expressions for sentiment labels) [20].

2.1.1. Paraphrasing

Paraphrasing for data augmentation involves creating semantically equivalent versions of text input by changing word choices or sentence structure, allowing models to learn from more diverse representations of the same content [21]. This method can be seen as sentence-level data augmentation, where entire sentences are rephrased while preserving the original meaning without introducing new information. It can help improve model robustness by exposing models to different phrasings of the same sentence [22]. It reduces dependence on specific wording, aiding models in generalizing [23].

Paraphrasing techniques often depend on algorithms, including rule-based [24] and transformer-based models [25], to produce sentences that are varied in both context and syntax. Rule-based approaches use predefined linguistic rules, such as substituting synonyms, restructuring clauses, or changing active to passive voice, to create paraphrases. While these methods have limited variability, they can be effective for simpler paraphrasing tasks [26]. On the other hand, transformer-based models—such as OpenAI’s GPT models (Generative Pre-trained Transformer), Google’s BERT (Bidirectional Encoder Representations from Transformers) [27], and the Text-to-Text Transfer Transformer (T5) [28]—utilize deep learning to generate fluent, contextually appropriate paraphrases [29]. These models are trained on large datasets and fine-tuned to produce high-quality paraphrasing, making them well-suited for creating diverse input texts with natural linguistic variety. Paraphrasing can be described as a process that maps an original sentence S to a set of semantically equivalent sentences {S₁, S₂, …, S_n}.

S $\to$ {S_i ∣ semantics (S_i) ≈ semantics (S)}, i = 1, …, n, where each S_i preserves the original meaning but differs in structure or wording.

2.1.2. Back Translation

Back translation is a sentence-level data augmentation technique to create diverse, yet semantically similar, versions of text data [30]. It involves a two-step process: translating text from the source language into a target language (e.g., from English to French) and then translating it back to the original language (e.g., from French to English). During this process, some words and phrases may be modified due to the inherent variability in language translation, resulting in a slightly altered version of the sentence. This process introduces natural linguistic variation through differences in syntactic and lexical choices across languages, enabling models to learn from a more extensive and diverse set of examples without altering the core meaning [31]. In contexts where labeled data is limited (e.g., low-resource languages), back translation serves as a valuable augmentation tool, helping to improve model accuracy by synthetically expanding the dataset [19].

Back translation techniques typically employ neural machine translation models like Google Translate, MarianMT, or custom-trained translation models to carry out the two-step process [32]. These models improve the fluency and contextual appropriateness of the back-translated text, minimizing errors and preserving meaning. Back translation can be formalized as a composition of translation functions.

S $\to$ S_tgt $\to$ S_bt, where S_tgt represents the translation from the source language to the target language, and S_bt represents the translation back from the target language to the source language.

2.1.3. Synonym Replacement

Synonym replacement substitutes selected words in a sentence with synonyms, introducing lexical diversity into the dataset [18]. By randomly selecting words and replacing them with similar words, this technique allows the model to become less sensitive to specific word choices, thereby improving its ability to generalize. Synonym replacement is widely applied in text classification [18] and sentiment analysis [33], as it helps the model learn to detect key patterns across different vocabulary and reduces overfitting on particular words. Synonym replacement is word-level and thus relatively simple to implement compared to other augmentation methods, often requiring only a dictionary or word embeddings [18].

In synonym replacement, a word is selected from a sentence, and a synonym is chosen from a pre-defined lexicon (e.g., WordNet) [34] or generated using word embeddings that capture semantic similarity, such as GloVe or Word2Vec [35]. The selected synonym is then substituted for the original word in the sentence, ensuring the grammatical and syntactic integrity of the sentence while avoiding alterations that might change the intended meaning. For efficiency, pre-trained models like BERT or GPT can be fine-tuned to select contextually appropriate synonyms, ensuring fluency and preserving meaning [36]. Synonym replacement can be formulated as follows:

S = (w₁,w₂,…,w_m) $\to$ S′ = (w₁,…,w_i′,…,w_m), where w_i′ is a synonym of w_i.

2.1.4. Random Insertion

Random insertion is a word-level data augmentation technique that involves inserting additional words into a sentence without significantly altering its original meaning [18,37]. This method introduces lexical variability by adding synonyms, adjectives, or contextually relevant words at random positions in the sentence, thus increasing the diversity of sentence structures presented to the model. The inserted words are selected based on semantic similarity using predefined synonym sets (e.g., WordNet) or vector-based models (e.g., Word2Vec) [38]. The number of insertions per sentence can be tuned based on the desired augmentation strength, and words are inserted at random positions [39]. Random insertion can be written as follows:

S = (w₁,w₂,…,w_m) $\to$ S′ = (w₁,…,w_i,w_new,w_i+1,…,w_m), where w_new is a contextually appropriate word (e.g., synonym) and the insertion position i is selected randomly.

Random insertion is particularly useful in low-resource language scenarios such as Turkish, where training data is limited, as it enables the generation of more diverse training samples without manual annotation [40,41]. While effective, excessive or poorly chosen insertions may distort sentence meaning or introduce syntactic errors, reducing the overall quality of augmented data [42].

2.2. Automated Text Generation in Turkish

Automated text generation refers to the use of computational models to create human-like text, such as sentences, paragraphs, or documents, either from scratch or conditioned on prompts, inputs, or existing data [43]. In recent years, advancements in pre-trained LLMs have enabled significant progress in text generation tasks across multiple languages [28,44,45]. There have been studies expanding the scope of automated-text-generation research to low-resource languages. Previous studies on automated text generation in the Turkish language have focused on tasks such as question generation [46,47], summarization [48,49], and sentence creation [50], reflecting the need to address specific linguistic challenges.

Question generation has been a popular area due to its applications in educational assessment, where tools are designed to produce relevant questions based on given texts or datasets. For example, Akyon et al. [46] used different LLMs (e.g., GPT-4-Turbo and Llama-2-7b-chat-hf) to generate questions and answers. Another research study focused on generating summary texts using Markov Models and Long Short-Term Memory (LSTM) neural networks, applied to datasets from Wikipedia TR and academic publications [48]. The researchers evaluated the models’ performance in producing syntactically accurate and semantically meaningful texts. These studies reported promising results despite issues such as repetitions and the closeness of the text to the language. Another study focused on generating Turkish sentences using a class-based n-gram model [50]. While bigram models produced a larger number of sentences, they were less semantically coherent. In contrast, the 3-gram model generated fewer but more meaningful sentences.

Even though there has been some progress in Turkish text generation in recent years, the availability of high-quality datasets and pre-trained models remains a challenge for low-resource languages like Turkish [11]. Turkish is an agglutinative language with complex morphology and extensive use of suffixes [10]. Words in Turkish can have numerous inflected forms due to the Turkish agglutinative nature. For example, a single root word can generate dozens of variations with added suffixes, making vocabulary representation and modeling more challenging compared to languages with simpler morphology. For example, the root word “ev” (house) can generate multiple forms such as “evler” (houses), “evlerde” (in the houses), and “evlerimizden” (from our houses). In addition, compared to English and other high-resource languages, Turkish has fewer large-scale, high-quality datasets for training language models [11]. The unavailability of high-quality data limits the ability to fine-tune pre-trained models effectively for text generation tasks in Turkish, introducing unique challenges in automated text generation in Turkish. To overcome these challenges, data augmentation techniques can enhance the training datasets by generating diverse and contextually relevant variations in existing text.

3. Materials and Methods

This study presents a controlled empirical and comparative analysis of commonly used data augmentation techniques for LLM-based text generation in a low-resource, morphologically rich language setting (Turkish). Turkish Grade 4 and Grade 5 teachers provided the researchers with a collection of 265 reading passages. As the materials were sourced directly from practicing teachers, the collection reflects classroom-relevant, grade-appropriate content and supports the overall quality of the dataset. Passage lengths ranged from short texts (~21 words) to longer passages of up to 347 words, with an average length of about 85 (M = 85.51, SD = 48.96). This distribution reflects the variability expected in Grade 4–5 instructional materials, including both short and extended reading materials. The corpus includes a range of genres commonly used in primary education. Narrative texts constitute the majority (n = 214, 81%), followed by fables and animal stories (n = 24, 9%), informational/historical texts (n = 18, 7%), and a smaller proportion of fairy tales (n = 9, 3%). This distribution reflects typical classroom reading materials, where narrative comprehension is emphasized while still exposing students to diverse types of text. The corpus spans multiple topics (e.g., daily life, nature, historical events, and moral stories), contributing to broad lexical coverage.

We applied four data augmentation methods—paraphrasing, back translation, synonym replacement, and random insertion—to create augmented datasets. Subsequently, we fine-tuned a separate text generation model on each augmented dataset, as well as on the original data, and compared their performance to assess the impact of each augmentation technique on model quality. Although the dataset comprises only 265 passages, prior work in low-resource languages has shown that pre-trained language models can be effectively fine-tuned with limited data when the fine-tuning set is representative of the target domain and task [51]. This supports the feasibility of using our teacher-sourced text collection for model fine-tuning.

3.1. Data Augmentation

To investigate the different data augmentation strategies on Turkish text generation, four augmentation techniques were applied to the original dataset: paraphrasing, synonym replacement, random insertion, and back-translation. Appendix A provides two sets of sample texts by four data augmentation techniques. All methods were implemented using a combination of Python-based preprocessing tools and the OpenAI API, specifically GPT-3.5-turbo (OpenAI Python SDK version 0.28), with the analysis conducted in Google Colab. While semantic preservation was implicitly encouraged through system- and user-level prompts instructing the model to maintain the meaning of each reading passage, explicit constraints on reading level (Grades 4 and 5), factual accuracy, and morphological or suffix adaptation were not enforced in the code. The exact system- and user-level prompts used for each augmentation method are provided in Appendix B for reproducibility. The augmentation process relied on the contextual understanding of GPT-3.5-turbo to generate plausible paraphrases, synonyms, and insertions, with the assumption that the model would maintain semantic coherence and overall readability. This design prioritizes scalability and simplicity, while acknowledging potential limitations in linguistic precision.

Paraphrasing was performed using the GPT-3.5-turbo model via the OpenAI ChatCompletion API. Each input story was processed individually, where a structured prompt in Turkish instructed the model to rewrite the story while preserving its meaning. A moderate temperature value (0.7) was used as a commonly adopted middle ground in text generation tasks to introduce controlled variability in lexical and syntactic form while maintaining semantic consistency. Lower temperature values (e.g., <0.5) tend to produce more deterministic outputs with limited variation, while higher values (>0.8) may introduce excessive randomness and risk altering the original meaning. Exception handling was implemented to manage potential API-related errors and ensure robustness of the augmentation pipeline.

Back translation was implemented as a two-step process using Google Translate, where the reading passages were first translated into English and then back into Turkish (TR–ENG–TR). Additionally, the same reading passages were translated into French and then back into Turkish (TR–FR–TR). English was selected as an intermediate language due to its widespread use in low-resource translation [52], while French was chosen as a pivot language due to its frequent use in multilingual NLP and back-translation research [53].

Synonym replacement was implemented using a language-model-based approach, where GPT-3.5-turbo was prompted to rewrite each story by replacing selected words with semantically similar alternatives. The selection of words and morphological adaptation were handled implicitly by the model, rather than through explicit rule-based processing. The system prompt guided the model to maintain semantic equivalence while introducing lexical variation. Given the morphological complexity of Turkish, synonym replacement was applied at the sentence level, relying on the language model to implicitly handle context-appropriate word forms.

Random insertion was implemented using a hybrid approach combining rule-based preprocessing and language-model-assisted synonym generation. Each story was first tokenized using the NLTK library, and stopwords were removed to identify candidate content words. A non-stopword token was then randomly selected from the text. A semantically related word (synonym) for the selected token was generated using GPT-3.5-turbo via the OpenAI API. Instead of replacing the original word, the generated synonym was inserted at a random position within the token sequence.

Evaluation Criteria for Augmented Data

Human Evaluation: We conducted a human evaluation of the original and augmented texts with two evaluators, based on five criteria—fluency, coherence, grammar, logical sequencing of events, and naturalness. Each text was rated on a 5-point scale with the following categories: 1 = Significant errors with no discernible meaning; 2 = Significant errors, making the meaning difficult to understand; 3 = Moderate errors but the meaning is reasonably clear; 4 = Minor errors with reasonably clear meaning; and 5 = Minor or no errors with easily understandable meaning. For this evaluation, a subsample of 50 texts of each model (a total of 300 texts) was selected. To ensure consistency and minimize bias, human evaluators read the same text from each model before proceeding to the next one, allowing for a direct comparison of outputs across models. Across the 50 texts, we first averaged the ratings provided by each evaluator for each criterion and then computed the overall average across the two evaluators.

Automatic Similarity Evaluation: To quantitatively evaluate the semantic fidelity of augmented texts, we employed an embedding-based similarity approach using multilingual sentence representations (paraphrase-multilingual-MiniLM-L12-v2). Both the original and augmented texts were encoded using a pretrained multilingual Sentence-BERT model, and cosine similarity was computed between corresponding text pairs. Prior to similarity computation, all embeddings were L2-normalized to ensure unit length. Cosine similarity was calculated as the dot product between corresponding normalized embeddings, where higher values indicate stronger semantic alignment. The results were then summarized using descriptive statistics.

3.2. Text Generation

The goal of the text generation was to evaluate model performance in a low-resource language using different augmented datasets. For this purpose, each dataset (i.e., original and augmented) was independently split into 80% training and 20% testing, and each training set was used to fine-tune a separate model instance. A fixed random seed (42) was used to ensure reproducibility of the data split. Input texts were tokenized using the model’s tokenizer with padding and truncation to a fixed maximum sequence length. As GPT-2 does not include a default padding token, the end-of-sequence token was used for padding. We chose a Turkish GPT-2-medium model (https://huggingface.co/ytu-ce-cosmos/turkish-gpt2-medium, accessed on 11 May 2026) as the base for fine-tuning due to its open source availability, language-specific architecture, and adaptability to our task. Unlike proprietary LLMs such as GPT-4, which are general-purpose and offer limited transparency and fine-tuning control, the Turkish GPT-2 model is specifically pre-trained on extensive Turkish corpora, making it more suitable for capturing Turkish syntax, morphology, and semantics.

The model was fine-tuned using a causal language modeling objective (next-token prediction), consistent with the GPT-2 architecture. The Hugging Face Trainer API was used to train the model and monitor validation loss during training. After fine-tuning, each model generated text continuations for the test set using the Hugging Face text-generation pipeline with a maximum output length of 100 tokens. Appendix C shows the training arguments used for fine-tuning the GPT-2-medium. The hyperparameters were selected based on prior studies involving transformer-based LLMs [9]. Specifically, a learning rate of 5e-5 and weight decay of 0.01 were used to ensure stable convergence, while a batch size of 8 reflects hardware constraints. The number of training epochs (10) was selected to balance sufficient learning with the risk of overfitting. Warmup steps (500) were included to stabilize early training, and evaluation at regular step intervals (every 500 steps) enabled monitoring of training progress. These settings align with commonly used defaults in the Hugging Face Transformers framework and prior research on transformer fine-tuning.

Evaluation of Generated Text

Automatic Evaluation Metrics. To assess the performance of the fine-tuned models, we used multiple evaluation metrics, including BLEU (Bilingual Evaluation Understudy), ROUGE (Recall-Oriented Understudy for Gisting Evaluation), METEOR (Metric for Evaluation of Translation with Explicit ORdering), chrF (CHaRacter-level F-score), BERTScore (Bidirectional Encoder Representations from Transformers Score), and cosine similarity. Thus, we evaluated the quality of generated text against the reference test samples using a combination of lexical, semantic, and embedding-based metrics. For each model, text continuations were generated from the test set inputs, and the resulting outputs were compared to their corresponding reference texts. Generated outputs include the input prompt followed by the model-generated continuation.

All evaluation metrics were computed using established libraries, including the Hugging Face evaluate framework. Metrics were computed using standard tokenization procedures provided by their respective evaluation libraries. All evaluation scores were computed at the sentence level and aggregated by averaging across the test set for each model. Higher scores across these metrics generally indicate stronger alignment between generated and reference texts, though each metric captures different aspects of similarity, including lexical overlap and semantic correspondence.

Lexical overlap and n-gram-based similarity were measured using BLEU, ROUGE (ROUGE-1, ROUGE-2, and ROUGE-L), METEOR, and chrF [54,55,56]. These metrics quantify the degree of surface-level similarity between generated and reference texts at the word and character levels. The BLEU metric is primarily precision-focused, measuring the overlap of n-grams between the generated and reference texts. METEOR balances precision and recall by computing an F-score and incorporates synonyms, stemming, and paraphrases to assess the similarity between the generated and reference texts. ROUGE prioritizes recall by calculating the overlap of n-grams, word sequences, or longest common subsequences. Specifically, ROUGE-n measures the overlap of n-grams, and ROUGE-L captures the longest shared subsequence between the generated text and reference text, reflecting sentence-level structural similarities. chrF is a character n-gram-based evaluation metric that measures the similarity between generated and reference texts by comparing overlapping sequences of characters, making it particularly effective for morphologically rich languages.

To capture semantic similarity beyond exact token overlap, we further used BERTScore, which evaluates the alignment between generated and reference sentences using contextualized embeddings. BERTScore was computed with Turkish language settings, and the F1 score was used as the primary indicator. In addition, we computed cosine similarity between sentence embeddings of generated and reference texts. Sentence representations were obtained using the multilingual Sentence-BERT model, which provides semantically meaningful vector representations across languages. Cosine similarity scores were calculated for each prediction–reference pair and averaged across the test set.

Significance Testing. To assess whether differences in performance metrics between the baseline model and models trained on augmented datasets were statistically meaningful, we conducted significance testing using a bootstrap resampling procedure. For each augmentation method, model outputs were compared against the baseline model using the same set of reference test samples. Specifically, for each evaluation metric, we repeatedly sampled with replacement from the test set to create bootstrap samples of equal size to the original dataset. For each resampled set, metric scores were computed independently for the baseline predictions and the corresponding augmented model predictions using the same reference texts. The difference between augmented and baseline scores was then calculated for each iteration. This process was repeated over multiple bootstrap iterations (n = 500), with the number of iterations selected based on computational constraints.

4. Results

4.1. Results for Data Augmentation

Human Evaluation: The results of the human evaluation showed that the original texts were consistently rated the highest across all criteria (see

Table 1. Average human evaluation scores for each data augmentation using 50 texts per dataset.

Method	Fluency	Coherence	Grammar	Logical	Naturalness	Overall
Paraphrased	4.50	4.25	4.55	4.35	4.45	4.42
Synonym replacement	4.30	4.40	4.35	4.35	4.30	4.34
Back translation (TR–ENG–TR)	4.39	4.33	3.80	4.28	4.14	4.19
Back translation (TR–FR–TR)	4.18	4.22	3.75	4.25	4.05	4.09
Random insertion	4.30	4.40	4.75	4.65	4.70	4.56
Original	4.65	4.70	4.75	4.70	4.75	4.71

Note. 1 = Significant errors with no discernible meaning; 2 = Significant errors, making the meaning difficult to understand; 3 = Moderate errors, but the meaning is reasonably clear; 4 = Minor errors with reasonably clear meaning; and 5 = Minor or no errors with easily understandable meaning.

). Most texts from the original dataset received scores of 4 or 5 on the 5-point scale, demonstrating minimal or no errors and clear meaning. Among the augmented datasets, texts generated through random insertion received the most favorable evaluations, followed by those created using paraphrasing and synonym replacement. Back-translated outputs received the lowest ratings overall.

For the random insertion, evaluators noted occasional disruptions due to the arbitrary insertion of words. The human evaluation revealed that these insertions sometimes disrupted fluency. Despite occasional fluency issues due to the random insertion of words, the model produced generally coherent, grammatical, logical, and natural texts. For paraphrasing, evaluators noted that some sentences felt overly general or awkward, resulting in somewhat lower ratings. The synonym replacement model was fairly strong, but it faced challenges with occasional inappropriate word choices. These disruptions, though not overly frequent, impacted the fluency of the augmented texts, making some sentences out of place. This also affected the output’s overall naturalness, as some words did not fit the context as well, though the overall meaning remained intact.

Texts augmented by the model trained on the back-translated datasets (both TR–ENG–TR and TR–FR–TR) exhibited similar performance in the human evaluation. Evaluators reported that subtle semantic shifts and occasional awkward phrasing resulted in moderate reductions in naturalness. The TR–ENG–TR model maintained decent fluency, coherence, and logical sequence, but its grammar was noticeably weaker. Evaluators reported issues with tense usage, which detracted from the overall readability of the text. The TR–FR–TR model shared similar issues with the TR–ENG–TR variant. While it performed adequately in fluency, coherence, and logical sequencing, it struggled more with grammar and naturalness. The disruptions were noticeable enough to impact the overall quality of the augmented texts. To ensure the reliability of the human evaluation, inter-rater consistency was assessed, yielding a Cohen’s kappa value of 0.82, which indicates strong agreement among annotators and supports the robustness of the human evaluation results.

Automatic Similarity Evaluation: To evaluate the extent to which each augmentation method preserves the original meaning, cosine similarity scores were computed between original and augmented sentences using the multilingual Sentence-BERT model (

Table 2. Semantic similarity (cosine similarity) between original and augmented texts across augmentation methods.

Method	M	SD	Min	Max
Paraphrased	0.884	0.073	0.583	0.981
Synonym replacement	0.885	0.088	0.201	1.000
Back-translation (TR–ENG–TR)	0.951	0.039	0.734	0.998
Back-translation (TR–FR–TR)	0.943	0.039	0.727	0.996
Random insertion	0.995	0.006	0.957	1.000

). Overall, all augmentation methods produced relatively high cosine similarity scores, indicating that semantic content was largely preserved across transformations. Random insertion achieved the highest mean similarity (0.995) with very low variability, suggesting minimal semantic deviation from the original texts. Back-translation methods also maintained strong semantic alignment (0.951 for ENG and 0.943 for FR), although with slightly higher variability, indicating occasional shifts in meaning. Synonym replacement and paraphrasing resulted in somewhat lower mean similarity scores (0.885 and 0.884, respectively). However, for synonym replacement, the higher standard deviation and lower minimum values indicate that some instances may lead to more noticeable semantic drift.

4.2. Results for Text Generation

Automatic Evaluation Metrics:

Table 3. Evaluation metrics for Turkish text generation with original and augmented texts.

Method	BLEU	ROUGE-1	ROUGE-2	ROUGE-L	METEOR	chrF	BERTScore-F1	Cosine Similarity
Paraphrased	0.4990	0.6477	0.6451	0.6487	0.8616	84.2164	0.8428	0.9697
Synonym replacement	0.6030	0.7406	0.7382	0.7406	0.9076	89.3047	0.8735	0.9864
Back translation (TR–ENG–TR)	0.5713	0.7083	0.7056	0.7089	0.8936	88.0683	0.8665	0.9840
Back translation (TR–FR–TR)	0.5681	0.6974	0.6944	0.6981	0.8864	87.5477	0.8567	0.9781
Random insertion	0.5700	0.7010	0.6988	0.7014	0.8924	87.7833	0.8601	0.9791
Original stories	0.6072	0.7319	0.7296	0.7326	0.9089	89.4137	0.8770	0.9844

presents the performance of models fine-tuned on the original dataset and various augmented datasets, evaluated using BLEU, ROUGE (ROUGE-1, ROUGE-2, ROUGE-L), METEOR, chrF, BERTScore, and cosine similarity based on sentence embeddings. Overall, the model trained on the original dataset achieved the most consistent performance across all evaluation metrics, serving as the baseline for comparison. Its scores were slightly higher than those of the augmented datasets across most lexical and semantic measures, indicating a modest advantage in generating outputs that align more closely with the reference texts.

Among the augmented datasets, performance differences were generally small. The paraphrased dataset showed slightly lower scores across most metrics compared to the other augmentation strategies, indicating a mild reduction in alignment with reference texts. However, the differences were not substantial, suggesting that the model still retained a reasonable level of performance under this augmentation. Synonym replacement performed slightly better than the other augmentation methods, with results that were marginally closer to the baseline across both lexical and semantic metrics. Random insertion produced comparable results, with only minor variations across evaluation measures, indicating a relatively stable effect on model performance. The back-translation datasets (TR–ENG–TR and TR–FR–TR) also yielded closely aligned results, with only slight differences between the two language pairs. Both variants performed similarly across all metrics, suggesting that back-translation preserves much of the original semantic structure while introducing minimal variation.

Overall, the model that was fine-tuned with the original dataset demonstrated somewhat superior performance compared to the models that utilized fine-tuning with the augmented datasets. This indicates that, while augmentation can enhance model capabilities, the original dataset retains a unique value that facilitates better outcomes in specific contexts. Among the various augmentation techniques employed, synonym replacement emerged as the most effective method with a small difference. Thus, the data augmentation techniques produced results comparable to the original dataset, preserving the overall quality of text generation while maintaining consistency and balance across various performance metrics.

METEOR scores were generally higher and exhibited less variation across augmentation methods compared to BLEU and ROUGE scores. This behavior is expected, as METEOR incorporates soft matching mechanisms such as synonym alignment and stem-based matching, making it less sensitive to surface-level lexical differences. As a result, METEOR tends to remain relatively stable even when augmentation strategies such as paraphrasing, synonym replacement, and random insertion introduce variation in lexical form and sentence structure. In contrast, more surface-oriented metrics such as BLEU, ROUGE, and chrF showed slightly greater sensitivity to these variations, reflecting differences in word overlap and character-level similarity between generated and reference texts. These metrics highlight the impact of augmentation on lexical realization rather than meaning preservation. Semantic evaluation metrics, including BERTScore and cosine similarity based on sentence embeddings, showed relatively stable trends across all augmentation methods. This suggests that despite variations in surface form, the generated outputs maintained a broadly consistent semantic relationship with the reference texts.

Significance Testing: To determine whether the observed performance differences between the baseline and augmented models were statistically meaningful, we conducted bootstrap-based significance testing across all evaluation metrics, including BLEU, ROUGE, METEOR, chrF, BERTScore, and cosine similarity. The results indicated that none of the augmented models achieved statistically significant improvements over the baseline. Thus, the small performance differences observed between the baseline and augmented models are not statistically significant. This suggests that, across the bootstrap samples, the baseline model performed as well as or better than the augmented models in nearly all cases. The bootstrap confidence intervals indicated low variability across resampling iterations, suggesting that the estimated performance differences are mostly stable and consistent across metrics (see

Table 4. Bootstrap 95% confidence intervals for augmentation methods relative to the baseline model (500 iterations).

Method	BLEU	ROUGE-1	ROUGE-2	ROUGE-L	METEOR	chrF	BERT Score-F1	Cosine Similarity
Paraphrase	[−0.54, −0.47]	[−0.31, −0.26]	[−0.52, −0.45]	[−0.44, −0.37]	[−0.66, −0.60]	[−43.35, −36.70]	[−0.25, −0.21]	[−0.13, −0.09]
Synonym Replacement	[−0.18, −0.14]	[−0.12, −0.09]	[−0.19, −0.15]	[−0.14, −0.10]	[−0.22, −0.16]	[−13.21, −9.52]	[−0.08, −0.07]	[−0.14, −0.10]
Back translation (TR–ENG–TR)	[−0.43, −0.37]	[−0.20, −0.17]	[−0.38, −0.33]	[−0.26, −0.22]	[−0.47, −0.42]	[−27.82, −24.14]	[−0.13, −0.11]	[−0.06, −0.03]
Back translation (TR–FR–TR)	[−0.46, −0.40]	[−0.24, −0.20]	[−0.43, −0.38]	[−0.31, −0.27]	[−0.53, −0.48]	[−31.35, −28.42]	[−0.15, −0.13]	[−0.06, −0.04]
Random Insertion	[−0.06, −0.04]	[−0.05, −0.03]	[−0.05, −0.04]	[−0.05, −0.03]	[−0.03, −0.02]	[−2.70, −1.62]	[−0.03, −0.02]	[−0.01, 0.00]

Note: Values represent 95% bootstrap confidence intervals for performance differences relative to the baseline model. Negative intervals indicate lower performance compared to the baseline.

for confidence intervals). Augmentation methods showed negative shifts relative to the baseline, with confidence intervals that remain below zero. Random insertion showed comparatively smaller and closer-to-zero confidence intervals even though it still did not improve performance relative to the baseline.

5. Discussion

Consistent with prior studies [18,19,31,40,41], our results indicate that commonly used data augmentation techniques—back translation, synonym replacement, and random insertion—can provide useful but generally modest effects in LLM-based text generation for Turkish, a low-resource, morphologically rich language. Back translation (TR–ENG–TR and TR–FR–TR) showed similar performance across pivot languages, suggesting that the choice of intermediate language has limited influence on overall outcomes and offering practical flexibility in low-resource settings where specific pivot languages may be more accessible than others. However, it also introduced occasional semantic shifts and unnatural phrasing, particularly given the challenges of preserving Turkish morphological structure. Synonym replacement produced comparable performance while maintaining semantic content, although some limitations were observed in ensuring morphologically and contextually appropriate substitutions. Random insertion yielded similar results by introducing surface-level variation without substantially altering meaning, although its effectiveness depends on careful token selection to avoid disrupting fluency [42]. While all three methods contribute to data diversity, their improvements over the baseline remain limited under the current experimental conditions, indicating that they primarily maintain rather than substantially enhance generation performance. Bootstrap-based significance analysis confirms that these observed differences are not statistically significant, suggesting that the improvements over the baseline are modest rather than systematic.

In contrast, our results diverge from studies that found paraphrasing to be a consistently effective augmentation strategy [21]. During the data augmentation evaluation phase, paraphrasing received relatively favorable human ratings, suggesting that the surface quality of paraphrased texts is reasonably high. However, paraphrasing yielded the weakest performance among all augmentation methods in the text generation phase, revealing a small but consistent gap between human-perceived text quality and model training utility. To further investigate this discrepancy, we conducted a qualitative inspection of the generated outputs. The paraphrased versions tend to preserve the overall narrative meaning of the original texts while substantially compressing structural and discourse-level information (Appendix A). In particular, intermediate events, expressive elements, and narrative transitions are frequently omitted or generalized. For example, detailed descriptions of human–animal interactions and emotional expressions in the original story are reduced to more abstract and concise formulations in the paraphrased output (see

Table 5. Qualitative comparison of original and paraphrased versions of the “Kartal” story.

Original Text	Paraphrased Output	Observed Issue
Bir kartal denizden uzak …	Not included	Omission of narrative detail and setting
Bir gün pençesinde … Yuva yaptığı ağacın … Balığı gördüklerinde …	Not preserved	Loss of temporal progression, spatial specificity, and interactional detail
Balık sonunda … Adamlar balığı … Kartal yuvasının …	Not included	Omission of narrative detail and setting
Yavruları ise … Oysa kartal çok yorulmuş … Yuvasına iyice … Onları sevdi, okşadı ve … Sanki … Fakat yavrular … Kartal uçtu ve … Yavrular anneleri … Sonunda kartal çaresizlik …	Not preserved	Loss of emotional and expressive content
Anne kartal akşam … Yine pençelerinde … Ağaca yaklaşırken … Güven içinde … Yavru kartallar … Anne kartal …	Not preserved	Meaning distortion/weakened emotional intensity

). While this results in lower n-gram overlap with the reference texts—thereby reducing BLEU scores—the core storyline remains largely intact. This explains the observed divergence between human judgments, which emphasize semantic adequacy and readability, and automatic metrics, which are sensitive to surface-level structural differences. Furthermore, this pattern is consistent with the bootstrap results, which show no statistically significant change in paraphrasing over other augmentation strategies or the baseline model. In Turkish, such transformations can disrupt the morpho-syntactic regularities that the model relies on, potentially degrading rather than enriching the training signal. Future work should examine whether paraphrasing techniques specifically designed to preserve Turkish morphological constraints can improve training utility in this context.

5.1. Limitations and Future Work

This study has several limitations worth noting. First, this study is designed as a controlled comparative analysis, focusing on systematic evaluation of four different augmentation techniques in text generation under a fixed experimental setup, rather than assessing their scalability or real-world deployment performance. Second, the relatively small dataset size (265 samples), which reflects the low-resource nature of the Turkish language setting, limits the generalizability of the findings and constrains our ability to evaluate scalability or cross-domain robustness. Third, we evaluated only four data augmentation techniques—paraphrasing, back translation, synonym replacement, and random insertion—limiting the scope of our analysis. Other augmentation strategies, such as contextual embedding-based methods or syntactic transformations, may offer additional insights. Furthermore, we examined each augmentation technique in isolation; combining methods may lead to different outcomes. In addition, the use of GPT-3.5-Turbo and Google Translate can introduce a potential confounding factor, as the observed benefits may partially reflect the capabilities of these pre-trained models. However, these tools were used solely to operationalize the augmentation techniques, contributing to the application of these augmentation methods to the Turkish reading corpus. Future studies could systematically evaluate these strategies with non-proprietary models to further verify replicability and robustness.

Another limitation is that augmentation procedures were not constrained with respect to grade-level appropriateness, morphological agreement, or factual consistency, which may have introduced noise into the generated texts. Future research should explore controlled augmentation strategies that enforce such constraints to improve the quality and reliability of augmented data. Developing augmentation techniques specifically tailored to the morphological and syntactic properties of Turkish, such as morphology-aware paraphrasing or morpheme-level augmentation, may help explore whether the effectiveness of synthetic data generation can be improved.

We focused on a single LLM available for Turkish (i.e., GPT-2-medium) and did not compare performance across different text generation models, which affects the generalizability. Future research should explore a wider range of augmentation strategies and model architectures to further examine whether such approaches produce meaningful improvements in low-resource language settings. Future work could explore the effect of data augmentation across larger and more diverse language models, including decoder-only, encoder–decoder, and instruction-tuned architectures, to assess whether the observed trends generalize beyond GPT-2-based systems. Furthermore, the research reports concerns regarding the reliability of automatic evaluation metrics in low-resource and morphologically rich languages [57]; therefore, the results of this study should be interpreted with caution. Moreover, future studies could employ more extensive statistical testing with larger bootstrap sample sizes to improve the robustness and stability of significance estimates. Another limitation of this study was that no formal sensitivity analysis was conducted to evaluate the impact of hyperparameter choices (e.g., temperature and training configurations) on model performance, and, thus, the robustness of the results to alternative parameter settings cannot be fully assessed. Future research could incorporate systematic sensitivity analyses to examine the impact of different hyperparameter settings on model performance, thereby improving the robustness and generalizability of the findings.

The close alignment we observed between human evaluations of augmented datasets and automatic evaluation metrics of generated texts suggests that combining both approaches may be useful, especially in low-resource settings. This may indicate the potential utility of mixed-method evaluation frameworks. That said, our human evaluation, though rich in qualitative data, was limited to only 50 texts per model. Expanding the evaluation sample would strengthen the robustness of the findings. Future studies could also incorporate more extensive human evaluation alongside automatic metrics such as BLEU, ROUGE, and BERTScore, to better capture fluency, coherence, and semantic adequacy in generated outputs. Furthermore, extending evaluation to task-specific settings such as summarization or question generation in Turkish could provide further insights into the potential applicability of augmentation strategies across different NLP tasks. Finally, our focus on Turkish limits the generalizability of the results. Future work could replicate the study across other morphologically complex or low-resource languages to explore broader applicability.

5.2. Conclusions

The purpose of this study was to systematically examine four data augmentation techniques—paraphrasing, back translation, synonym replacement, and random insertion—in the context of Turkish text generation, a low-resource language that remains underexplored in current NLP research. We employed human as well as automatic similarity evaluation to assess the intrinsic quality of the augmented datasets. In terms of text generation, we applied a combination of lexical, character-level, and semantic evaluation metrics (e.g., BLEU, ROUGE, METEOR, chrF, BERTScore, and cosine similarity) to evaluate the performance of text generation models fine-tuned on these augmented datasets.

The original, non-augmented dataset generally achieved comparable or slightly higher results across most metrics, which is consistent with the expectation that naturally occurring text preserves linguistic properties that are difficult to replicate through transformation. Among the augmentation methods, synonym replacement produced results most similar to the original dataset, while other techniques showed slightly larger deviations. Statistical significance testing using bootstrap resampling indicated that none of the observed differences between augmentation strategies and the baseline were statistically significant across evaluation metrics, including both lexical and semantic measures. Although the observed differences were not statistically significant, this result is itself meaningful, as it provides evidence that widely used augmentation techniques do not necessarily improve performance in a low-resource language setting. In NLP research, such null results can help refine assumptions about method transferability across low-resource languages and prevent overestimation of the benefits of standard augmentation strategies.