Instruction Fine-Tuning for Low-Resource Indigenous Languages: Kichwa-to-Spanish Neural Machine Translation ^†

Abstract

Low-resource languages remain challenging for neural machine translation because of the limited availability of parallel corpora and computational resources. This work investigates instruction fine-tuning for Kichwa–Spanish neural machine translation using FLAN-T5-Base. A bilingual corpus was constructed by aligning Kichwa transcriptions with Spanish translation material through normalization, fuzzy matching, and linguistic validation, resulting in 2875 instruction-formatted sentence pairs. The corpus was partitioned into training, validation, and test subsets for model adaptation and evaluation. The experimental evaluation compares instruction-fine-tuned FLAN-T5 with the unadapted FLAN-T5 baseline and fine-tuned mT5-Base using BLEU and chrF++. The task-adapted models were evaluated across independent runs initialized with different random seeds, with performance reported as mean ± standard deviation. Instruction-fine-tuned FLAN-T5 achieved the highest scores, with BLEU of $3.46\pm 0.18$ and chrF++ of $21.57\pm 0.38$, compared with $0.50\pm 0.21$ and $8.59\pm 2.72$ for mT5-Base; the unadapted FLAN-T5 baseline obtained BLEU and chrF++ scores of 1.32 and 15.46, respectively. The results indicate that task-specific adaptation increases lexical and character-level correspondence with the reference translations, while the absolute scores reflect the difficulty of the evaluated low-resource setting. An interactive Streamlit prototype complements the text-based translation system with an optional speech-input workflow presented as an interface-level integration. Overall, the study provides an experimental reference for further research on Kichwa–Spanish neural machine translation.

1. Introduction

Neural machine translation (NMT) has significantly advanced in recent years due to the development of transformer-based architectures and large-scale pretrained language models [1,2,3]. These approaches have demonstrated strong performance in high-resource languages where large parallel corpora are available. However, low-resource and indigenous languages still face major limitations because of the scarcity of annotated datasets, linguistic resources, and speech technologies.

Kichwa is an indigenous language that is spoken in several regions of South America, particularly in Ecuador, where it represents an important component of cultural identity and linguistic heritage. Despite its social and cultural relevance, the availability of computational resources for Kichwa remains limited. The existing translation systems often struggle to generalize due to insufficient training data, dialectal variability, and the lack of specialized language models [4,5]. Recent efforts such as the Killkan dataset have contributed to the development of speech recognition and morphosyntactic resources for Kichwa, highlighting the growing interest in computational tools for indigenous languages [6]. Furthermore, recent studies have explored low-resource automatic speech recognition (ASR) approaches for Kichwa using architectures such as Wav2Vec2, demonstrating the feasibility of integrating speech technologies into indigenous language applications [7]. Nevertheless, the quantity of publicly available parallel corpora and annotated speech data remains significantly lower compared to high-resource languages, limiting the performance and generalization capabilities of modern neural architectures.

Recent advances in instruction fine-tuning have shown that pretrained transformer models can adapt more effectively to downstream tasks through prompt-guided learning [1]. Models such as FLAN-T5 incorporate instruction-based training strategies that improve generalization and task understanding, even under limited-data conditions. This paradigm has become particularly attractive for low-resource natural language processing (NLP) applications because it reduces the need for extremely large domain-specific datasets.

In this work, we explore the application of instruction fine-tuning for Kichwa–Spanish neural machine translation using the FLAN-T5 architecture. A bilingual dataset was adapted into an instruction-based format to train and evaluate the model. In addition, a prototype interface was developed to demonstrate text and speech-based translation workflows through an interactive application. The experimental evaluation compares instruction-fine-tuned FLAN-T5 with the pretrained FLAN-T5 baseline and mT5-Base using BLEU and chrF++. The task-adapted models were evaluated across five independent runs initialized with different random seeds, with performance reported as mean ± standard deviation. The processed corpus, fixed data partitions, and implementation code are publicly released to support reproducibility and future Kichwa–Spanish model comparisons.

The main contributions of this work can be summarized as follows:

• The construction of an instruction-formatted Kichwa–Spanish translation corpus by aligning and filtering bilingual resources through fuzzy matching techniques.
• The instruction fine-tuning and evaluation of FLAN-T5-Base for Kichwa–Spanish NMT, including comparison with the pretrained FLAN-T5 baseline and mT5-Base across five independent runs using BLEU and chrF++.
• An interface-level integration of speech transcription with the text-based Kichwa–Spanish translation model, with quantitative evaluation focused on the NMT component.
• The public release of the processed corpus, fixed training, validation, and test partitions, and implementation code to support reproducibility and future model comparisons.

2. System Architecture and Methodology

This section describes the proposed translation framework, including dataset preparation, instruction fine-tuning, speech processing, system implementation, and evaluation methodology. The complete pipeline was designed to support low-resource Kichwa–Spanish translation through a transformer-based architecture integrated with speech processing components.

2.1. Dataset Preparation

A bilingual Kichwa–Spanish parallel dataset was compiled for training and evaluation. Given the limited availability of digital resources for Kichwa, constructing the corpus required multiple stages of preprocessing, alignment, validation, and normalization. The Kichwa source sentences were derived from 3753 transcriptions in the Killkan dataset [6], which is distributed under the CC BY 4.0 license. Spanish counterparts were obtained from the bilingual material used during corpus construction and associated with the Killkan transcriptions through the normalization and similarity-based alignment procedure described below.

The initial corpus consisted of 3753 Kichwa transcriptions. Since these transcriptions were not directly paired with Spanish translations, a sentence alignment procedure was required to construct a parallel corpus that is suitable for neural machine translation. To achieve this, text normalization was first applied by converting sentences to lowercase, removing punctuation symbols, and standardizing formatting to reduce orthographic variability.

After normalization, approximate string matching was performed using RapidFuzz. Each normalized Kichwa transcription was compared against candidate sentences from the bilingual Kichwa–Spanish material, and the candidate with the highest ratio-based similarity score was selected. Alignments with a similarity score of 100% were accepted directly, whereas candidates scoring between 85% and 100% were manually reviewed before inclusion. Candidates below the 85% threshold were excluded from the parallel corpus and preserved separately for subsequent corpus expansion.

The fuzzy matching process reduced the corpus from 3753 initial transcriptions to 2928 candidate Kichwa–Spanish sentence pairs. Consequently, 825 transcriptions were discarded because no sufficiently similar bilingual counterpart could be identified. A subsequent validation stage removed entries containing incomplete translations or placeholder entries indicating missing translations. This filtering step eliminated an additional 53 sentence pairs, resulting in a final corpus of 2875 retained Kichwa–Spanish translations. Approximate alignments were then manually reviewed by a Kichwa-speaking member of the research team. The linguistic review examined the correspondence between each Kichwa source sentence and its associated Spanish translation and removed incomplete, placeholder, or unrelated translations. Validation decisions were made by this linguistic reviewer; consequently, the protocol did not involve multiple independent annotations or an inter-annotator agreement measure.

In addition to the aligned corpus, a separate file was generated to store Kichwa sentences for which no reliable match could be identified. During the alignment stage, each normalized Killkan sentence was compared against the bilingual dictionary using the RapidFuzz similarity metric. When the highest similarity score was below the acceptance threshold of 85%, the sentence was not included in the parallel corpus. Instead, it was stored in an auxiliary dataset containing the original transcription and normalized sentence, while the translation field remained empty and no similarity score was assigned.

This procedure prevented the inclusion of low-confidence alignments that could introduce noise into the training data. As a result, only sentence pairs satisfying the similarity threshold and subsequent validation criteria were retained for instruction fine-tuning, while unmatched sentences were preserved separately for future manual annotation and corpus expansion. Table 1 shows the complete procedure for creating the dataset.

Table 1. Dataset construction process.

Stage	Samples
Initial Killkan transcriptions	3753
Aligned using fuzzy matching (≥85%)	2928
Unmatched sentences stored separately	825
Removed due to missing translations	53
Final retained corpus	2875

The final distribution was 2300 training samples, 287 validation samples, and 288 test samples.

Examples of accepted and discarded alignments are presented in Table 2. Exact matches generally obtained similarity scores of 100, whereas approximate matches captured orthographic variations, punctuation differences, and word segmentation inconsistencies. Some pairs were later removed because their associated Spanish translations were incomplete.

Table 2. Examples of sentence alignments obtained through the fuzzy matching process.

Killkan Sentence	Matched Kichwa Sentence	Spanish Translation	Score	Status
Pandemiamanta rimaykuna	pandemiamanta rimaykuna	Historias de la pandemia	100.0	Accepted
Ari, wakimpi rikcharini shinapash na krinichu Matria Librewan tukuy rurashkakunata, Eusebio	ari wakinpi rikcharini shinapash na krinichu matria librewan tukuy rurashkakunata eusebio	Sí, a veces me despierto y no puedo creer lo mucho que hemos logrado con Matria Libre, Eusebio.	98.88	Accepted
Ama, ima nikunkitakyari	ama ima nikunkitakyari	—	100.0	Removed due to missing translation

After corpus construction, additional preprocessing steps were applied, including punctuation standardization, sentence cleaning, consistency verification, and duplicate removal. Corpus cleaning and duplicate removal were completed before the train–validation–test partition was generated so that dataset partitioning was performed on the final retained corpus. Particular attention was given to spelling inconsistencies and orthographic variations, which are common challenges in low-resource indigenous language datasets.

The validated sentence pairs were subsequently transformed into an instruction-based format that is suitable for instruction fine-tuning. Instead of using plain parallel sentences, each example was reformulated as a structured prompt explicitly describing the translation task, following recent instruction-tuning methodologies for large language models [1]. A typical training example is shown in Table 3:

Table 3. Example of the instruction-formatted translation instance.

Field	Content
Instruction	Translate the following sentence from Kichwa to Spanish.
Input	Mana entendenichu.
Target output	No le entiendo.

The final dataset contained 2875 instruction-formatted Kichwa–Spanish sentence pairs. The corpus predominantly represents Ecuadorian Kichwa, with a particular emphasis on the Otavalo dialect, providing a relatively consistent linguistic setting for experimentation.

To evaluate the generalization capability of the model, the dataset was partitioned into training (80%), validation (10%), and testing (10%) subsets. The training subset was used for parameter optimization, the validation subset for monitoring model convergence, and the testing subset for final evaluation. To reduce the possibility of cross-partition leakage, corpus cleaning and duplicate removal were completed before dataset partitioning. The resulting training, validation, and test subsets were then kept fixed and used consistently across all the model comparisons.

The resulting corpus provides the fixed Kichwa–Spanish dataset used throughout the translation experiments reported in this study. The processed corpus and the exact training, validation, and test partitions used in the experiments are publicly available in the accompanying GitHub repository: https://github.com/maarckch2308/kichwa_translate.git (accessed on 1 July 2026).

2.2. Speech-to-Text and Translation Pipeline

The prototype includes an optional speech-input branch that illustrates how a transcription stage can provide textual input to the Kichwa–Spanish translation model. Figure 1 shows this interface-level workflow together with the text-based translation path evaluated in this study.

Figure 1. Interface-level workflow connecting Kichwa speech input, transcription, and text-based Kichwa–Spanish translation.

The pipeline begins with an audio input in Kichwa, which is processed by an ASR module to generate a text transcription. The resulting transcription is then passed to the fine-tuned FLAN-T5 translation model, which produces the corresponding translation in Spanish. Finally, the translated output is displayed through the graphical user interface. The speech branch is included to illustrate component connectivity and user interaction within the prototype. It is not used to characterize the accuracy of speech recognition or end-to-end speech translation. No fixed ASR checkpoint and experimental configuration were included in the controlled evaluation; consequently, the reported quantitative results should be interpreted exclusively as text-based machine translation results. A future extension of the system could incorporate a fixed Kichwa ASR model and evaluate the transcription and end-to-end paths using WER, CER, and translation metrics computed from ASR-generated transcriptions.

2.3. Instruction Fine-Tuning

Instruction fine-tuning was applied to adapt the pretrained FLAN-T5-Base model to the Kichwa–Spanish translation task. Unlike traditional input–output mapping, this paradigm represents tasks as natural language instructions [1]. By embedding each Kichwa sentence within a structured prompt template, the model is explicitly conditioned on the translation task while leveraging representations acquired during pretraining [2]. To provide a multilingual comparison, mT5-Base was included as an additional sequence-to-sequence baseline. mT5 extends the T5 framework through multilingual pretraining and was trained and evaluated on the same Kichwa–Spanish partitions used for the FLAN-T5 experiments.

Transformer-based architectures excel in sequence-to-sequence learning due to self-attention mechanisms, which efficiently capture contextual dependencies [3]. This capability is particularly critical for morphologically rich and low-resource languages like Kichwa, where word-level meaning is heavily context-dependent. The training pipeline was implemented using the Hugging Face Transformers library [8]. Input sequences were tokenized using the native FLAN-T5 tokenizer with a maximum sequence length of 128 tokens. Standard padding and truncation strategies were applied to ensure uniform batch processing. During training, the model processed the tokenized prompts and autoregressively generated the expected Spanish translations, optimizing its parameters to minimize the sequence generation loss.

Table 4 details the specific hyperparameter configuration used for the experiments. Due to computational memory constraints typical of local hardware setups, a batch size of 4 was selected for both models. Both FLAN-T5-Base and mT5-Base were trained with the Hugging Face Trainer API using a freshly initialized copy of the corresponding pretrained checkpoint for each of five independent training runs. A different random seed was used for each run through the seed and data_seed parameters in TrainingArguments, allowing model initialization and data shuffling to vary across runs. The FLAN-T5 model was optimized using the AdamW algorithm with a learning rate of $5\times {10}^{-5}$ over five epochs, while mT5-Base used a learning rate of $2\times {10}^{-5}$ over the same number of epochs. Checkpoint saving and validation monitoring were executed once per epoch to track model convergence and mitigate overfitting. The corresponding training-specific hyperparameters for mT5-Base are reported alongside the FLAN-T5 configuration in Table 4. For the comparative evaluation, FLAN-T5-Base and mT5-Base were trained using the same training, validation, and test partitions. Each configuration was independently trained and evaluated across five runs initialized with different random seeds, and the resulting BLEU and chrF++ scores are reported as mean ± standard deviation.

Table 4. Training and evaluation configuration for FLAN-T5-Base and mT5-Base.

Parameter	FLAN-T5-Base	mT5-Base
Base Model	FLAN-T5-Base (google/flan-t5-base)	mT5-Base (google/mt5-base)
Architecture	Transformer encoder–decoder	Transformer encoder–decoder
Framework	Hugging Face Transformers	Hugging Face Transformers
Tokenizer	FLAN-T5 tokenizer	mT5 tokenizer (SentencePiece)
Maximum Input Length	128 tokens	128 tokens
Training Epochs	5	5
Batch Size	4	4
Learning Rate	$5\times {10}^{-5}$	$2\times {10}^{-5}$
Optimizer	AdamW (Hugging Face Trainer default)	AdamW (Hugging Face Trainer default)
Padding Strategy	Max length	Max length
Truncation	Enabled	Enabled
Checkpoint Saving	Every epoch (save_total_limit = 1)	Every epoch (save_total_limit = 1)
Evaluation Strategy	Every epoch	Every epoch
Logging Frequency	Every 10 steps	Every 10 steps
Training Samples	2300	2300
Validation Samples	287	287
Test Samples	288	288

Decoding/inference settings (beam size, no-repeat n-gram size, early stopping, and maximum generation length) are shared between both models and reported separately in Table 5.

Table 5. Training, decoding, and evaluation settings used in the repeated experiments.

Parameter	Value
Independent training runs	5
Random-seed protocol	One distinct random seed per independent run
Maximum generation length	80 tokens
Number of beams	5
No-repeat n-gram size	2
Sampling	Disabled
Early stopping	True
SacreBLEU version	2.6.0
SacreBLEU signature (BLEU)	nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.6.0
SacreBLEU signature (chrF++)	nrefs:1|case:mixed|eff:yes|nc:6|nw:2|space:no|version:2.6.0

2.4. Experimental Setup and Reproducibility

All the model comparisons used the same training, validation, and test partitions. Training was repeated independently across five runs, with a different random seed used for each run. Model initialization and the stochastic components of the training procedure therefore varied across runs, while the test partition and evaluation protocol remained fixed. The test partition was used only for final evaluation. The scripts used for training and evaluation are provided in the accompanying public repository.

During inference, translations were generated using beam-search decoding with max_new_tokens = 80, num_beams = 5, do_sample = False, no_repeat_ngram_size = 2, and early_stopping = True. Length penalty was left at its default value. The same decoding configuration was applied to all the task-adapted models to ensure that performance differences were not introduced by model-specific generation settings. Because decoding was deterministic (beam search without sampling), the random seed affected only model initialization and training dynamics and had no direct effect on the inference procedure itself.

Although the dataset size remained relatively limited, the experiments showed that task-specific adaptation increased the automatic translation scores relative to the unadapted model. Given the low absolute BLEU and chrF++ values, these improvements are interpreted as evidence of increased surface-level correspondence with the reference translations rather than reliable semantic learning.

2.5. Translation Model

The translation system was based on the FLAN-T5 architecture, a transformer encoder–decoder model specifically designed for instruction-following and text-to-text learning tasks [1]. FLAN-T5 extends the original Text-to-Text Transfer Transformer (T5) framework proposed by Raffel et al. [2] by incorporating large-scale instruction-tuned datasets during pretraining, significantly improving task generalization and zero-shot learning capabilities across multiple NLP applications.

Transformer architectures have become the dominant paradigm in modern machine translation because of their ability to capture long-range contextual dependencies using self-attention mechanisms [3]. Unlike recurrent neural networks, transformer-based models process sequences in parallel, enabling more efficient contextual representation learning and improved scalability for sequence-to-sequence tasks.

In the proposed system, the model receives a textual instruction containing the Kichwa input sentence and generates the translated Spanish output autoregressively. During inference, the Kichwa sentence is first tokenized using the pretrained FLAN-T5 tokenizer and then processed by the encoder component, which produces contextualized latent representations. Subsequently, the decoder generates the Spanish translation token by token until the output sequence is completed.

The instruction-based formulation of the task enables the model to interpret translation as a guided language generation problem rather than a simple token mapping process. This characteristic is particularly beneficial in low-resource settings because it allows the model to leverage prior linguistic knowledge acquired during large-scale pretraining.

FLAN-T5 was selected because of its relatively low computational requirements and strong adaptability in low-resource environments compared to larger transformer architectures. In addition, its instruction-tuned design makes it especially suitable for multilingual and domain-adaptive applications where training data availability is limited. The model was implemented using the Hugging Face Transformers framework [8], which facilitated tokenization, fine-tuning, and inference integration within the proposed system.

Under the evaluated low-resource setting, task-specific adaptation increased lexical and character-level correspondence with the Spanish references relative to the unadapted baseline. The absolute metric values indicate that these gains primarily reflect improved reference correspondence, while semantic adequacy requires complementary evaluation beyond surface-overlap metrics.

2.6. Speech Processing Pipeline

The prototype includes an optional audio-input branch that connects speech-derived text with the Kichwa–Spanish translation model within the same interface. The workflow follows the sequence: audio input, speech transcription, text translation, and Spanish output. The quantitative experiments in this study focus on the text-based NMT component, while the audio branch serves to demonstrate component integration.

The proposed workflow follows the sequence:

$$\mathrm{Audio}\mathrm{input}\to \mathrm{Speech}\mathrm{transcription}\to \mathrm{Text}\mathrm{translation}\to \mathrm{Spanish}\mathrm{output}$$

Audio samples and their associated transcriptions were obtained from the Killkan dataset [6] to exercise the speech-input workflow. Because this branch is used as an integration component rather than as an ASR benchmark, WER, CER, and end-to-end speech-translation scores are not reported as part of the present quantitative evaluation.

A dedicated speech benchmark using a fixed Kichwa ASR configuration, reference transcriptions, WER/CER, and translation metrics computed from ASR-generated text constitutes a natural extension of the current interface architecture.

2.7. System Interface

An interactive prototype was developed using Streamlit in order to demonstrate the functionality of the proposed translation system. The interface allows users to provide textual or speech-based inputs and visualize the generated translation results in real time.

The application integrates the translation model, speech processing pipeline, and inference workflow into a single user-friendly environment. The interface was designed to simplify experimentation and provide a practical demonstration of low-resource machine translation technologies.

2.8. Evaluation Metrics

The proposed translation system was evaluated using BLEU and chrF++, two widely adopted metrics in NMT research. These metrics provide complementary perspectives on translation quality by measuring the similarity between generated translations and reference sentences at both the word and character levels.

BLEU (Bilingual Evaluation Understudy) measures translation quality through modified n-gram precision, incorporating a brevity penalty to discourage excessively short translations [9]. Following the standard formulation with $N=4$ and uniform weights, BLEU is defined as

$$\mathrm{BLEU}=\mathrm{BP}·\exp \left({\displaystyle \sum _{n=1}^4}{\displaystyle \frac{1}{4}}\log p_n\right)$$

(1)

where $p_n$ denotes the modified precision for n-grams of order n, and $\mathrm{BP}$ is the brevity penalty,

$$\mathrm{BP}=\left\{\begin{array}{cc}1,\hfill & c>r\hfill \\ e^{(1-r/c)},\hfill & c\le r\hfill \end{array}\right.$$

(2)

where c represents the length of the generated translation corpus and r corresponds to the length of the reference corpus. BLEU was computed at the corpus level using the SacreBLEU library. The evaluation used SacreBLEU version 2.6.0, with the BLEU configuration encoded by the signature nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.6.0. The same references, preprocessing state, and SacreBLEU configuration were used for all the evaluated models and experimental runs.

To complement BLEU, chrF++ was employed to evaluate character-level similarity [10]. Unlike BLEU, which relies on strict word overlap, chrF++ combines character n-gram and word n-gram precision and recall through an F-score formulation,

$${\mathrm{chrF}}_{\beta }=(1+{\beta }^2)·{\displaystyle \frac{\mathrm{Precision}·\mathrm{Recall}}{{\beta }^2·\mathrm{Precision}+\mathrm{Recall}}}$$

(3)

where precision and recall are computed over character and word n-grams. This metric is exceptionally well-suited for morphologically rich and low-resource languages like Kichwa as it captures partial lexical similarities and root matches even when exact morphological conjugations differ. The chrF++ configuration is encoded by the signature nrefs:1|case:mixed|eff:yes|nc:6|nw:2|space:no|version:2.6.0.

BLEU and chrF++ provide complementary measures of surface-form similarity. While BLEU emphasizes word n-gram correspondence, chrF++ captures character- and word-level overlap. For each task-adapted model, BLEU and chrF++ were computed independently across five experimental runs and are summarized as mean ± standard deviation. The two metrics provide complementary measures of word- and character-level correspondence with the reference translations and define the quantitative translation evaluation used in this study. Human judgments of adequacy and fluency provide a complementary evaluation dimension that can be incorporated in subsequent work.

3. Results and Discussion

This section presents the performance evaluation of the Kichwa–Spanish neural machine translation models. The quantitative BLEU and chrF++ results are analyzed together with representative translation outputs to characterize the effect of task-specific adaptation and the behavior of the evaluation metrics.

3.1. Translation Performance

To evaluate the efficacy of the fine-tuning process, the proposed model was compared against the original pretrained FLAN-T5-Base model operating under zero-shot conditions (without task-specific adaptation). The evaluation was further extended with mT5-Base as a multilingual baseline. The task-adapted models were evaluated over five independent runs initialized with different random seeds while maintaining the same training, validation, and test partitions.

The reported standard deviations characterize run-to-run variability across the five independently trained configurations while the corpus partitions and evaluation procedure remained fixed. For the non-fine-tuned FLAN-T5 baseline, no model training was performed and the deterministic decoding configuration produced identical evaluation scores across runs, resulting in a standard deviation of 0.00. Among the task-adapted models, instruction-fine-tuned FLAN-T5 achieved the highest mean performance with comparatively low variability (BLEU $=3.46\pm 0.18$; chrF++ $=21.57\pm 0.38$). Fine-tuned mT5-Base obtained lower mean scores (BLEU $=0.50\pm 0.21$; chrF++ $=8.59\pm 2.72$) and exhibited greater variability, particularly for chrF++. Under the evaluated training configuration, mT5-Base therefore did not outperform either the instruction-fine-tuned FLAN-T5 model or the unadapted FLAN-T5 baseline.

The initial demonstrations of the prototype showed that the pretrained model frequently failed to execute the translation task, often defaulting to hallucination or simply echoing the input prompt. For instance, given the Kichwa input “Mana entendenichu”, the base model reproduced the prompt structure instead of generating a Spanish translation, producing “Traduce del Kichwa al Español: Mana entendenichu”.

Conversely, the fine-tuned model generated “No entendemos”. While the reference translation is “No le entiendo”, the generated output preserves the negation and shares the lexical stem entend- with the reference. However, the change in person and number alters the intended meaning, illustrating that lexical overlap does not necessarily imply semantic adequacy.

The qualitative example illustrates the different sensitivities of the two automatic metrics. Across the repeated experiments, BLEU remained low in absolute terms, whereas chrF++ indicated greater character- and subword-level overlap between generated and reference translations. The comparison with mT5 provides an additional reference for determining whether these improvements are specific to the evaluated FLAN-T5 adaptation or are also obtained by a multilingual pretrained model under the same data constraints: under the present configuration, mT5-Base obtained lower mean BLEU and chrF++ scores than both the fine-tuned and the non-fine-tuned FLAN-T5 models and also exhibited substantially higher variance across seeds (Table 6). This suggests that the multilingual mT5 checkpoint did not transfer effectively to this low-resource pair under the training budget used here and that its fine-tuning was comparatively unstable across random initializations.

Table 6. Comparison of Kichwa–Spanish translation performance across the evaluated models. Results for the task-adapted models are reported as mean ± standard deviation over five independent runs initialized with different random seeds.

Model	BLEU	chrF++
FLAN-T5 Base (without task-specific fine-tuning)	1.32 ± 0.00	15.46 ± 0.00
FLAN-T5 Base (instruction fine-tuning)	3.46 ± 0.18	21.57 ± 0.38
mT5-Base (fine-tuned)	0.50 ± 0.21	8.59 ± 2.72

Note: Bold values indicate the best performance for each metric.

Although task-specific adaptation produced measurable improvements relative to the unadapted FLAN-T5 model, the absolute scores remain limited. Therefore, the results should be interpreted as an experimental comparison under low-resource conditions rather than evidence of reliable or practically deployable Kichwa–Spanish translation.

3.2. Speech Interface Demonstration

The speech-related functionality was examined only at the level of interface integration. The prototype illustrates the sequence from audio input to transcription, text translation, and display of the Spanish output, but the speech-recognition stage was not included in the quantitative experimental protocol. Consequently, the present results do not identify an ASR bottleneck or quantify the effects of pronunciation, acoustic conditions, or transcription errors on translation performance.

The implemented prototype therefore serves as an interface-level demonstration of component connectivity. A controlled assessment of the complete speech-to-translation path would require a fixed ASR model, reference speech transcriptions for WER/CER computation, and end-to-end translation metrics calculated from the ASR-generated text.

3.3. Graphical Interface Prototype

A graphical user interface was developed using Streamlit to demonstrate the complete functionality of the proposed system, as shown in Figure 2. The interface allows users to input either text or audio and obtain translations in real time.

Figure 2. Demo of the graphical user interface of the proposed translation system.

The interface was designed to simplify interaction with the translation pipeline and improve accessibility for non-technical users. The visual organization of the application facilitates understanding of the system workflow, particularly during demonstrations and validation experiments.

The interface demonstrates the interaction between audio input, transcription, text translation, and result visualization. Its purpose is to illustrate software integration and user interaction; ASR accuracy and end-to-end speech-translation quality are outside the quantitative scope of the present evaluation.

3.4. Interpretation of Evaluation Metrics

To better understand the behavior of the evaluation metrics, Figure 3 presents an example comparing the reference translation with the output generated by the fine-tuned model.

Figure 3. Comparison between the reference translation and the translation generated by the model.

In this example, the original Kichwa sentence “Mana entendenichu” should be translated into Spanish as “No le entiendo”. However, the model generated the sentence “No entendemos”. Although both sentences share the lexical stem associated with the verb entender, the generated translation contains an important grammatical error. The model changed the meaning from a first-person singular interpretation (“I do not understand him/her”) to a first-person plural form (“we do not understand”), altering the intended meaning of the sentence. This example illustrates the complementary information provided by the evaluation metrics: lexical and character-level similarity capture partial correspondence with the reference, whereas the person and number mismatch reveals a semantic difference that is not fully represented by surface overlap alone.

This example highlights the difference between the BLEU and chrF++ evaluation metrics. BLEU relies on exact word-level matching and therefore strongly penalizes translations that differ from the reference sentence, even when some semantic information is preserved. Consequently, translations such as “No entendemos” receive a low BLEU contribution because the generated output does not exactly match the expected translation. In contrast, chrF++ evaluates character-level similarity through character n-grams. As illustrated in Figure 4, the generated word “entendemos” shares several lexical and morphological components with the reference word “entiendo”. As a result, chrF++ captures this partial overlap and assigns a higher similarity score than BLEU.

Figure 4. Illustration of how BLEU and chrF++ evaluate translation similarity.

The mean chrF++ score of $21.57\pm 0.38$ across the five runs indicates measurable but still limited character-level correspondence between the generated and reference translations. The score should therefore be interpreted together with BLEU and the qualitative example above, which shows that surface overlap can coexist with morphosyntactic differences that alter sentence meaning. The score is therefore interpreted only as evidence of partial surface-form correspondence. It does not establish semantic adequacy, and the observed grammatical errors show that the generated translations can alter the intended meaning even when lexical or character-level overlap is present.

Future work will focus on:

• Expanding the parallel corpus with additional validated sentence pairs.
• Exploring data augmentation and generative AI techniques for dataset expansion.
• Conducting an independent human evaluation of generated translations with multiple Kichwa–Spanish speakers, including adequacy and fluency judgments and inter- rater agreement.
• Fine-tuning and evaluating speech recognition models directly on Kichwa audio datasets using WER and CER metrics.
• Integrating Low-Rank Adaptation (LoRA) methods to improve training efficiency.
• Developing a complete audio-to-audio translation framework.

These improvements could significantly enhance both transcription accuracy and translation quality, enabling more robust applications for indigenous and low-resource languages.

The repeated-run analysis characterizes the stability of BLEU and chrF++ across different training seeds, while the qualitative example provides additional context for interpreting the automatic scores. In particular, it illustrates that measurable lexical and character-level overlap can coexist with morphosyntactic differences that affect sentence meaning. This distinction motivates the combined use of automatic overlap metrics and, in future evaluations, human adequacy and fluency judgments.

4. Conclusions

This study investigated the application of instruction fine-tuning to Kichwa–Spanish neural machine translation using the FLAN-T5-Base architecture under a low-resource experimental setting. To support this task, a bilingual dataset containing 2875 instruction-formatted sentence pairs was constructed by aligning Kichwa transcriptions with Spanish translation material using a fuzzy matching strategy.

The experimental results show that task-specific instruction fine-tuning improved translation performance relative to the original pretrained FLAN-T5 model. Across five independent runs initialized with different random seeds, the instruction-fine-tuned FLAN-T5 configuration achieved mean BLEU and chrF++ scores of $3.46\pm 0.18$ and $21.57\pm 0.38$, respectively, compared with $0.50\pm 0.21$ and $8.59\pm 2.72$ for fine-tuned mT5-Base. The non-fine-tuned FLAN-T5 baseline obtained BLEU and chrF++ scores of $1.32\pm 0.00$ and $15.46\pm 0.00$, respectively. Instruction-fine-tuned FLAN-T5 therefore achieved the highest mean scores among the evaluated configurations and showed comparatively low run-to-run variability. Fine-tuned mT5-Base produced lower mean scores and greater variability, particularly for chrF++, under the evaluated training configuration. Although the repeated-run analysis provides a more robust characterization than the original single-run experiment, the absolute metric values indicate that substantial room for improvement remains. The comparison with mT5-Base provides a multilingual reference under matched low-resource conditions, while a prompt-free FLAN-T5 ablation remains a targeted extension for separating the contribution of instruction formatting from supervised task adaptation.

Overall, this work establishes a reproducible experimental reference for studying task-specific adaptation in Kichwa–Spanish neural machine translation. The processed corpus, fixed data partitions, and implementation code are publicly released to support independent replication and future comparisons. The observed gains over the unadapted baseline demonstrate measurable adaptation to the translation task, while the absolute metric values highlight clear opportunities for further corpus expansion and complementary human evaluation.

Future work will focus on expanding and validating the bilingual corpus, incorporating native Kichwa speech datasets for speech recognition, evaluating larger and parameter-efficient fine-tuning approaches such as LoRA, and exploring multilingual transfer learning and synthetic data generation techniques. These directions will be evaluated to determine whether they can improve translation quality beyond the limited performance observed in the present study.