Soft Prompt Tuning for Multilingual Natural Language Understanding in Human–Robot Interaction with Bidirectional Cross-Attention

Abstract

Natural language understanding (NLU) enables intuitive human–robot interaction (HRI). It requires joint modeling of intent classification and named entity recognition (NER). Existing approaches suffer from three key limitations. Fine-tuning large multilingual pretrained models on domain-specific HRI data incurs high computational overhead. Intent classification and NER are typically treated as independent tasks, despite their semantic interdependence. Labeled HRI datasets for Indian languages are absent, to the best of our knowledge, limiting system accessibility for native-speaking communities. This paper proposes SoftPromptIndicBERTCRF, a parameter-efficient multilingual NLU architecture built on a frozen, 33.4 M-parameter IndicBERT backbone. The architecture prepends 50 learnable soft prompt tokens, initialized from task-specific HRI vocabulary, to the input sequence. A bidirectional cross-attention module, with 8 attention heads, models the mutual dependency between intent and NER. A Conditional Random Field (CRF) decoder enforces valid Begin–Inside–Outside (BIO) tag sequences during entity decoding. Custom HRI datasets were constructed for English, Hindi, and Malayalam, covering 13 intent classes and 7 entity types. The proposed framework provides 99.35% accuracy for intent detection and a 98.77% F1 score for NER in all three languages. The latency for end-to-end pipelining involving ASR transcriptions and NLU inferences is 0.78 to 0.92 s on average in all three languages, thereby validating its utility for real-time application in robots.

1. Introduction

The growing deployment of service robots in domestic, healthcare, and industrial environments has created an increasing demand for natural and intuitive command interfaces that eliminate the need for specialized programming expertise [1,2]. NLU forms the cognitive core of these interfaces, enabling robots to interpret spoken commands and map them to executable actions. In HRI applications, the two most critical NLU components are intent classification, which identifies command categories such as navigate or pick, and named entity recognition, which extracts specific arguments including target locations, objects, colors, sizes, and quantities. These components are not independent; entity types expected by a navigation command differ fundamentally from those expected by a pick-and-place command, and the simultaneous presence of multiple entity types is itself a strong discriminative signal for compound intent classification. The accuracy of both components directly determines robot task execution reliability, making their joint prediction a foundational requirement for practical HRI deployment [3].

A primary obstacle to capable HRI deployment is the computational cost of adapting large pretrained language models to domain-specific tasks. LLM-based approaches such as ChatGPT Version 5.5-ROS integrations and LLM-driven behavior tree generation demonstrate flexible command understanding but depend on cloud API inference or high-end GPU workstations unavailable on typical onboard robotic computing units, introducing network latency and privacy dependencies incompatible with real-time robot control [4]. This motivates the fully on-device NLU architecture developed in Section 3.2. Fully fine-tuning large multilingual models on small HRI datasets risks catastrophic forgetting of pretrained multilingual representations, which is particularly critical for low-resource Indian languages where labeled data is inherently scarce [5,6]. A second fundamental limitation is the independent treatment of intent classification and NER as separate prediction problems. While gating mechanisms [7], bidirectional interrelated models [8], and multi-task frameworks [9] have improved joint performance on English benchmarks, none address compound multi-step HRI commands in morphologically complex languages such as Malayalam, whose agglutinative morphology creates entity boundary detection problems substantially more complex than those in morphologically simpler languages [10,11].

This paper proposes SoftPromptIndicBERTCRF, a parameter-efficient multilingual NLU architecture for joint intent classification and NER across English, Hindi, and Malayalam. The architecture freezes the 33.4 M parameter IndicBERT backbone and prepends 50 soft prompt tokens initialized from task-specific HRI vocabulary, preserving pretrained representations while enabling domain specialization [12,13]. A bidirectional cross-attention module explicitly models the semantic interdependency between intent and NER tasks through mutual attention between the CLS token and per-token hidden states, directly addressing the gap left by architectures that treat these tasks independently. A CRF decoder enforces valid BIO tag sequences during entity decoding, and the NER component is upweighted in the joint training objective to account for differential task convergence rates. The architecture trains only 13.78% of its 38.8 M total parameters, achieving 99.35% average intent accuracy and a 98.77% NER F1 across three languages, alongside introducing a novel labeled HRI dataset for Malayalam and Hindi.

The major contributions are:

• A unified multilingual HRI framework for joint intent classification and named entity recognition across English, Hindi, and Malayalam languages via a single parameter-efficient model.
• Task-specific soft prompt initialization that uses task-specific HRI vocabulary to initialize prompt embeddings rather than traditional random initialization, enabling fast convergence and better adaptation for low-resource robotic command understanding.
• Bidirectional cross-attention modeling of the mutual dependence between the tasks of intent prediction at the sentence level and entity recognition at the token level in multilingual robotic commands.
• A multilingual HRI dataset with English, Hindi, and Malayalam languages that includes compound navigation and manipulation instructions to bridge the gap in public datasets available for Indian language-based HRI.
• A comprehensive evaluation of the entire system on a physical robot using ROS2 simulation under different noise conditions.

In addition to the suggested parameter-efficient multilingual NLU model, this paper makes a valuable contribution by suggesting a concrete framework for creating human–robot interaction applications in low-resource language environments. Through the combination of multilingual intent classification and named entity recognition with a parameter-efficient approach to adaptation and actual implementation on an embedded robot, this method enables a concrete way of implementing multilingual HRI models on robots with constrained computational capabilities.

This paper is organized as follows: Section 2 reviews related studies. Section 3 details our proposed method. Section 4 presents experimental studies and provides an analysis of the results. Section 5 concludes the paper.

2. Related Work

Large language models have been widely integrated into robotic systems for flexible command understanding. The ChatGPT-ROS bridge for service robot control by Koubaa et al. [14] was shown to be functional for HRI, although its performance was poor for ambiguous commands and depended on cloud API. Asuzu et al. [15] presented a human–LLM planning loop, where GPT-4 planning is refined by humans before execution, improving planning success but limiting autonomy. Chekam et al. [16] used LLMs to auto-generate behavior tree XML for interpretable robot control, although invalid tree structures occasionally required post-processing. Ranasinghe et al. [17] introduced a cognitive validation layer cross-checking LLM outputs against contextual constraints to suppress hallucinations before execution. Ye et al. [2] showed that ChatGPT-generated verbal explanations of robot actions significantly improved human trust in collaborative tasks, while Kim et al. [6] extended LLM integration to adaptive patient care robots. Pinto-Bernal et al. [18] developed design guidelines for LLM-powered social robots with synchronized facial expressions. None of these approaches are fine-tune for domain-specific multilingual HRI vocabulary or support edge deployment without cloud infrastructure.

Yang et al. [19] proposed a vision–NLP system for chemical laboratory robots using BiLSTM-CRF and a hierarchical finite state machine, demonstrating closed-loop language-to-action execution in Chinese only. Huang et al. [20] introduced perspective disambiguation and placement optimization for language-driven manipulation using CLIP-based visual grounding, without NER or intent classification. Liu et al. [21] developed DRAGON for assistive navigation combining LLM dialog with visual grounding. González-Docasal et al. [22] designed a BERT-based industrial HRI system with safety filtering, achieving high accuracy on constrained English vocabulary. Benayas et al. [23] proposed a unified BERT-based transformer for simultaneous intent and NER with shared representations, evaluated on English benchmarks. Son et al. [24] demonstrated that differential loss weighting reduces negative transfer in multi-task NLU directly motivating the λ = 2.0 NER weighting in the proposed architecture. Alshahrani et al. [25], Li et al. [26], and Ma et al. [27] further explored hybrid CNN-BiLSTM, automotive multi-intent, and generative prompt-based NLU frameworks. However, none of these address multilingual Indian-language HRI or employ parameter-efficient adaptation strategies. A critical limitation shared by all of the above works is the absence of support for Indian languages in HRI contexts and the lack of parameter efficiency suitable for edge deployment.

Parameter-efficient adaptation of pre-trained models has emerged as a principled alternative to full fine-tuning for domain-specific tasks. Tian et al. [28] proposed personalised soft prompt tuning for multi-task transfer learning with crowdsourced annotations, demonstrating that learnable prompt vectors adapt effectively to task-specific distributions while keeping backbone weights frozen, thereby establishing the core mechanism adopted in the proposed architecture. Ding et al. [29] extended prompt tuning to multi-task vision-language models through soft context sharing, proposing shared and task-specific prompt components for CLIP-based architectures that enable cross-task knowledge transfer without full fine-tuning. This soft context sharing principle directly motivates the bidirectional cross-attention mechanism in the proposed model, which enables analogous information transfer between intent and NER task heads. However, none of these parameter-efficient adaptation works have been applied to the HRI domain, evaluated on Indian language command structures, or combined with joint intent–NER modeling under a frozen multilingual backbone, which makes the proposed architecture unique.

Processing morphologically complex low-resource languages introduces challenges that standard English-centric NLP (Natural Language Processing) pipelines cannot handle. Sebastian [30] reported fundamental challenges in developing NLP systems for Malayalam where agglutinative morphology, data sparsity, and word alignment issues pose major hurdles for entity boundary detection and BIO tagging in Malayalam HRI commands. Cross-lingual transfer can help alleviate some of the issues faced in low-resource languages. Hernández et al. [12] reported transfer learning for Spanish-to-Nahuatl for navigation commands using variants of multilingual BERT. Aram et al. [13] confirmed that developing HRI systems for non-English-speaking populations requires language-specific resources rather than zero-shot transfer from English-speaking populations. There is no availability of a labeled HRI dataset for Malayalam and Hindi in existing literature. There is no prior work reported on validating a joint intent–NER system for Indian language robotic commands. The choice of IndicBERT is based on its pretraining on 12 Indian languages, including Hindi and Malayalam, rather than relying on generic multilingual pretraining. More morphology-aware subword representations have been reported for IndicBERT compared to generic multilingual BERT.

The pioneering efforts of Megalingam et al. on voice-based robotic control utilizing the Robot Operating System (ROS) [31] provided the basis for this study, serving as the initial practical impetus behind this work. Their earlier investigations into the potentialities of automatically controlled robotic navigation for elderly or physically handicapped individuals through vocal commands revealed the potential of spoken command-based interactions within assistive robotics, although their design was constrained to an English-only vocabulary of commands that lacked any capacity for natural language understanding. Other contributions toward the same goal included studies in ROS-assisted robot control via voice commands [32], as well as ROS-based human-robot interaction for navigation platforms [33]. However, despite these advancements, these designs still relied on template-matching methodologies and were unable to perform intent categorization and named entity recognition. Speech and visual integration for assistive robotic perception [34], as well as a speech and Kinova arm-based interactive system with person tracking capabilities [35], were significant contributions that brought about multimodal interaction design; however, these designs were still limited to predefined commands and incapable of accommodating morphologically complicated Indian languages. Work on optimal speech recognition with ROS [36] further refined the ASR pipeline but did not address these limitations.

3. Design and Implementation

As depicted in Figure 1, the system architecture is composed of three main components: a Human-Robot Interface Module, the NLU Model, and the Robot Operating System 2 (ROS2) layer. At a high level, the system operates as follows: a human operator delivers an instruction through speech, which is captured through a microphone and converted to text via an ASR component. The resulting text is input to the NLU model, which predicts both the intent of the command and its related named entities. The structured result, containing an intent label alongside a set of entity spans, is passed to the ROS2 layer, where mappings between the prediction and robot actions take place, triggering the execution of the command on the service robot. A feedback Text-to-Speech (TTS) response is generated and relayed back to the human operator through the HRI module’s audio output system, confirming the action taken and closing the human–robot interaction loop.

Figure 1. Overall Architecture Diagram.

3.1. Human–Robot Interface Module

The Human–Robot Interface Module is primarily responsible for providing a means of direct interaction between a human operator and the robot using voice conversations. The module’s main goal is to take audio-based requests from the human and accurately convert them into text so that the overall system can understand and process them efficiently. The Interface Module connects to all audio recording devices, collects audio data from these devices upon request, and transforms the audio data into a format that the language model can understand and process. The Human-Robot Interface Module also ensures that conversation happens in a seamless, on-going manner by combining the requests made by the human operator with responses provided by the system during the entire collaborative process to provide for an uninterrupted and real-time interaction between the human operator and the robot. In doing so, the interface module supports a more natural and intuitive collaborative experience between the human and the robot through the seamless integration of voice, resulting in less latency during interactions and higher levels of efficiency and safety in the human–robot interaction workspace.

As shown in Figure 2, there are several steps in the process of work of the Human–Robot Interface (HRI) module. First of all, the command that is said through the microphone is recognized by the system using the Automatic Speech Recognition (ASR) method and text is produced as a result of this. After it has been transcribed, the text is compared with the text in the FAQ database in order to find any pre-determined response and after that the text is sent to the multilingual natural language understanding (NLU) module where the intent is classified and the named entities are recognized. After that, the identified intent and entities are transformed into robot commands using the ROS2 interface, and then the execution process goes into the Text-to-Speech (TTS) in order to produce the feedback as a human-readable message.

Figure 2. HRI Module Architecture.

In the above Figure 2, the internal working process of Human–Robot Interface (HRI) module demonstrates speech is acquired through Automatic Speech Recognition (ASR), FAQ checking, multilingual natural language processing, execution of robot commands through ROS2, and creation of feedback using Text-to-Speech (TTS) process. As shown in the image in Figure 2, there are several steps in the process of work of the Human-Robot Interface (HRI) module. First of all, the command that is said through the microphone is recognized by the system using Automatic Speech Recognition (ASR) method and the text is produced as a result of this. After it has been transcribed, the text is compared with the text in the FAQ database in order to find any pre-determined response and after that the text is sent to the multilingual Natural Language Understanding (NLU) module where the intent is classified and the named entities are recognized. After that the identified intent and entities are transformed into robot commands using the ROS2 interface and then the execution process goes into the Text-to-Speech (TTS) in order to produce the feedback as a human readable message.

The raw speech audio captured by the microphone is then transcribed into text using Google’s advanced Automatic Speech Recognition (ASR) model. Following transcription, the resulting text undergoes an initial Frequently Asked Questions (FAQ) verification process. During this stage, the system performs a rapid check to determine whether the user’s request exactly matches any of the pre-defined queries stored in the knowledge base. If a match is found, a corresponding pre-defined response is instantly generated and prepared for delivery. This efficient mechanism helps in handling common and repetitive requests swiftly, significantly reducing the processing time and ensuring faster, more consistent replies to the human user during the interactive session. The Human–Robot Interface module is also integrated with the output audio devices to facilitate the dissemination of vocal feedback responses done by Google’s Text-to-Speech (gTTS 2.5.4) model. This integration entails the conversion of textual replies into audible responses that human operators can comprehend. The privacy and latency considerations described in Section 1 only apply to the NLU inferencing module of this framework, which operates completely locally using the pretrained IndicBERT model. The ASR and TTS components that are implemented in this system currently operate in the cloud (using Google ASR and gTTS), but only as an engineering shortcut for this work and not as a final design point.

3.2. Natural Language Understanding Model

3.2.1. Multilingual Dataset Construction

Three highly specialized datasets have been carefully crafted for the English, Hindi, and Malayalam languages, focusing entirely on natural human–robot interaction (HRI) text examples. The datasets cover a wide range of possible commands and conversations that are generally observed in a cooperative robotic environment. The English dataset contains 7320 samples, there are 6686 in the Hindi dataset, and 626 in the Malayalam. All three datasets were split into training, validation, and test sets using a stratified 70/15/15 ratio to preserve intent class distribution across splits. The limited size of the Malayalam dataset reflects the lack of labeled data that exists for low-resource Dravidian languages in various domains, which is addressed through a proposed soft prompt tuning strategy that is proven through experimentation to both converge more quickly and generalize better than full fine-tuning with low-resource data. To ensure diversity, a variety of data augmentation approaches have been systematically used.

To prevent any data leakage during the evaluation process, the initially created speech samples were partitioned into training, validation, and test datasets using the stratified 70:15:15 split based on the intent category. Since all modifications were made solely on the training dataset, there were no modified speech samples within the validation and test datasets taken from the initial same speech sample. To ensure that the training, validation, and test datasets remain distinct, all duplicates and semantically similar speech samples were identified and removed from the dataset.

The approaches included: (i) synonym substitution for action verbs, location names, object names, and color/size attributes; (ii) filler word insertion to simulate natural spoken language patterns; (iii) character-level noise injection to simulate ASR transcription errors (character swap, deletion, and insertion at p = 0.05 per character); and (iv) back-translation via a Malayalam → English → Malayalam pipeline using the Google Translate API, which was applied only to samples where round-trip semantic fidelity was manually verified. Of the 626 Malayalam samples, approximately 320 are original utterances and 306 are augmented, with back-translation contributing 112 samples and the remaining augmentation methods contributing 194. Back-translated samples were excluded from the test split to avoid data leakage. The datasets have been labeled with 13 different intent categories, including navigate_to_location, pick_object, place_object, move_forward, move_backward, turn_left, turn_right, clean, and question, and as well as four compound intents: compound_navigate_pick, compound_pick_place, compound_move_turn, and compound_navigate_clean. For the NER task, the traditional BIO (Begin–Inside–Outside) approach was followed, effectively identifying and labeling seven different entity types: LOCATION, OBJECT, DIRECTION, SIZE, COLOR, QUANTITY, and DISTANCE.

The Malayalam corpus comprises 626 utterances, which have been manually annotated for all predefined robot command intents. Using a stratified 70:15:15 split technique, 438, 94, and 94 utterances have been reserved for training, validation, and testing, respectively. Although the corpus size is smaller than those of the English and Hindi corpora, care was taken to balance intents in the annotation process. The small number of test examples, especially compound commands, calls for careful analysis of class-wise evaluation results.

In Figure 3, a statistical assessment of the Malayalam dataset, which contains only 626 samples with an average of 3.08 tokens and 1.72 entities per sample, shows that there are 1931 tokens and 1079 annotations on entities across 99.7 percent (624 samples). The dataset contains 13 unique intents and seven types of entity annotations, indicating good consistency across the labels for all samples in the dataset as shown. Analysis of intent–entity coverage shown in Figure 3a shows that nearly all intents have close to complete entity representation, with most categories exhibiting approximately 100% of samples containing at least one entity. All but the “question” intent have very high-density annotations for their respective category outputs, whereas the “question” intent has considerably lower density than the other intents; however, all intents have relatively high-density annotations. The BIO tagging distribution represented in Figure 3b includes 44.1 percent of the tokens as -O, 47.0 percent as B tags, and 8.9 percent as I tags, demonstrating the extent of non-entity tokens in the corpus (i.e., sequence labeling datasets). A relatively smaller proportion of ‘I’ tags suggests that most annotated entities are short spans, often consisting of single tokens or limited multi-token sequences.

Figure 3. Malayalam Dataset Statistics (a) Entity Coverage by Intent (b) BIO Tag Distribution.

Figure 4 shows the graphical user interfaces that have been created for the human–robot interaction system. Figure 4a demonstrates the graphical user interface for robot control and monitoring. This user interface allows the user to view the status of the robot and monitor tasks being executed by the system. Figure 4b represents the graphical user interface for voice command interaction in multiple languages, wherein voice commands are entered, analyzed using the ASR and NLU, and converted to actions performed by the robot.

Figure 4. Graphical user interfaces developed for the proposed multilingual human–robot interaction system: (a) robot control and monitoring interface; (b) multilingual voice-command interaction interface.

The horizontal bar chart in Figure 4a shows intent frequency across all cases, whereas the pie chart shown in Figure 4b illustrates the intent proportion of each case (as a percentage). The intent with the highest occurrence is ‘pick’, while the second most frequent intent is ‘navigate’. The pie chart clearly depicts these proportions visually. The remaining intents account for a smaller, but still significant, proportion of the dataset. Compound intents create a number of different command types, requiring the model to understand complex command forms that involve more than one command. HRI scenarios emphasize object manipulation and navigation, while allowing sufficient cases for movement, cleaning, and questioning. Compound-based commands and/or multi-turn commands require cases to support the model. The most dominant intent, as represented in the pie chart, is “pick object” at 26.0%. The second most dominant intent is “navigate to location” at 15.1%, followed by “place object” at 12.1% and “clean” at 10.9%.

A. Data Splitting and Augmentation Protocol.

To conducts a fair evaluation, the manually annotated dataset was initially divided into training (70%), validation (15%), and testing (15%) splits through the process of stratified sampling that maintains the class distribution of intents across all three splits. Augmentation of the dataset was done post-partitioning and was limited solely to the training split. The methods of augmentation used were synonym replacement, filler addition, character-level noise injection, and back-translation. Validation and testing splits have only the original manually annotated utterances without any augmented instances or semantically equivalent versions of the training set utterances.

B. Data Annotation Guidelines.

The multilingual human–robot interaction dataset was manually annotated for intents and named entities according to a pre-devised annotation scheme. An intent tag was assigned to each utterance representing the semantic intent of the robotic instruction (for example, navigation, manipulation, query, or combined actions). The annotations for the named entities were made using BIO (Beginning, Inside, Outside) tags which help in identifying task-related semantic entities such as locations, objects, people, directions, and modifiers of actions.

C. Annotators.

Data annotation was performed by three annotators with prior exposure to NLP and human–robot interaction. Before doing the annotation task, all the annotators underwent training using an annotation manual with samples representing each category of intent and entity. All ambiguous situations were discussed by all annotators, and any disagreements reached were resolved by consensus.

D. Quality Control.

To preserves the annotation quality, the annotated dataset was manually verified several times. Duplicate examples were filtered out prior to dataset splitting, while some randomly selected examples of each language were verified separately to detect any annotation mismatches. These mismatches were discussed and corrected during the dataset review process.

3.2.2. Input Processing and Tokenization

The first stage of the SoftPromptIndicBERT CRF pipeline is the transformation of raw natural language user commands into structured numerical forms that are amenable to processing by the transformer architecture. The natural language user commands, whether transcribed using voice transcription or directly input, are tokenized using the WordPiece tokenizer of the Indic BERT. This tokenizer splits the input natural language into subword units that are shared across a vocabulary of 12 Indian languages and English. The use of a subword tokenization method is especially useful for morphologically complex languages like Malayalam, where the use of agglutinative suffixation to root words would result in high out-of-vocabulary rates using word-based tokenization schemes. With the use of the subword tokenization method, novel word forms can be represented using known subword units. Each subword token is then mapped to a 128-dimensional dense embedding vector retrieved from IndicBERT’s pretrained embedding table. This reflects IndicBERT’s AlBERT-based architecture, which uses a smaller embedding dimension at the input layer before projecting to a larger internal hidden dimension, a design that significantly reduces the total parameter count of the embedding layer compared to standard BERT while maintaining representational capacity.

3.2.3. Soft Prompt Prepending

The soft prompt module is the primary parameter-efficient adaptation mechanism of the proposed architecture, responsible for steering the frozen IndicBERT encoder toward the HRI task domain without modifying any of its pretrained weights. As shown in Figure 5, fifty continuous learnable vectors, each of 128 dimensions matching IndicBERT’s input embedding dimension, are prepended to the token embedding sequence before it enters the encoder. These vectors are referred to as soft prompts because they are neither discrete tokens from the vocabulary, nor do they correspond to any real words, but rather exist as free-floating continuous parameters in the same embedding space as the token embeddings. From the encoder’s perspective, the input sequence appears to begin with 50 special context tokens followed by the actual command tokens, causing the transformer’s self-attention layers to incorporate these prompt vectors when computing contextual representations for every real token in the sequence. Suppose we take the phrase “അടുക്കളയിലേക്ക് പോകു” (“go to the kitchen”); the tokenizer splits the phrase into smaller subword units. Each of these subwords is then turned into a 128-dimensional vector using IndicBERT’s embedding table. Before sending these vectors through the model, 50 special “soft prompt” vectors are added to the beginning of the sequence.

Figure 5. Soft Prompt Prepending.

Afterward, the total sequence is processed through ALBERT’s embedding mapping layer to create a new sequence of vectors, each of which is 768-dimensional. The resulting sequence that is fed into the transformer contains 50 prompt vectors, followed by standard tokenized inputs. The prompt vectors, P1 through P50, are the soft prompt vectors that were learned during the training phase. Initializing these soft prompt vectors is one of the most critical design factors that make the proposed method superior to randomly initializing them. Rather than being initialized to a standard normal distribution, as was done in previous work, the prompts are instead initialized by sampling the vocabulary specific to the HRI task at hand via keyword extraction. For example, relevant keywords were identified from the training dataset, focusing particularly on words that can be used to classify the intent of the action being communicated (discriminative action verbs), and words that represent entity nouns. This initialization places prompt vectors in a linguistically meaningful region of the embedding space that is proximal to HRI domain concepts from the very first training step, providing IndicBERT with immediate task context rather than requiring the optimizer to navigate from a random starting point.

This hypothesis has been proven empirically via an ablation where task-specific vocabulary initialization (chosen approach) is compared to random normal (Gaussian) initialization and random vocabulary selection, while keeping everything else—architecture, hyperparameters, and data—the same for all three experiments (3 seeds per experiment) on the Malayalam dataset. The intent accuracy score decreased from 0.99 to 0.89 and the NER F1 score decreased from 0.99 to 0.90 when using random vocabulary selection. The prompt vector starting points are also closer to domain-specific embeddings compared to random initializations. Thus the gradients computed are smaller and more optimal in direction, resulting in faster rates of convergence as well as improved final performance relative to prior methods. This is of particular importance when working with the Malayalam data where there is a very small training dataset consisting of just 626 samples. The length p = 50 of the prompts was chosen through a sensitivity sweep over p ∈ {10, 20, 30, 50, 75, 100} while fixing all other architecture and training parameters. Intent classification accuracy and the NER F1 score exhibit a steep rise from p = 10 (95.0%/90.0%) to p = 50 (99.8%/98.7%) and then level off—the NER F1 score shows a slight drop for p = 75 and p = 100 (both 98.5%), despite an additional number of parameters. Yet these parameters carry the entire responsibility for domain adaptation of the frozen 33.4 M parameter backbone. However, after adding the cross-attention modules and task-specific heads, the total trainable parameters reach 5.34 M, which accounts for only 13.78% of the total. Given how these elements affect model behavior, this efficiency is impressive because, by comparison, fine-tuning even just the last two layers of IndicBERT would require updating over 10 M parameters.

3.2.4. Indicbert Encoder

After soft prompt prepending, the combined sequence of 50 prompt vectors followed by the tokenized command tokens passes through IndicBERT’s embedding mapping layer, which projects each 128-dimensional vector to the full 768-dimensional hidden space used by IndicBERT’s transformer layers. This projection is a learned linear transformation that is part of the frozen backbone and was pretrained to map the compact input embeddings to the richer internal representation space of the AlBERT transformer architecture. The expanded 768-dimensional sequence then passes through the full stack of IndicBERT’s shared transformer layers, which apply multi-head self-attention and feed-forward transformations to produce deeply contextualized representations for every position in the sequence. A defining characteristic of the proposed architecture is that all 33,443,584 parameters of the IndicBERT encoder are completely frozen throughout training; their gradients are never computed, and they are never updated by the optimizer. This design choice is motivated by several considerations. First, it prevents catastrophic forgetting: the rich multilingual representations learned during IndicBERT’s pretraining on 12 Indian languages is preserved intact, ensuring that cross-lingual alignment between English, Hindi, and Malayalam is maintained. Freezing the IndicBERT backbone drastically cuts down the number of trainable parameters while retaining the multilingual representations obtained via pretraining. Throughout the training process, the parameters of the encoder are held constant, thereby avoiding contributions to weight updates or optimizer state maintenance. Although gradients must flow through the frozen encoder while calculating the gradients of the soft prompt embeddings and the task-specific heads, the gradients do not accumulate on the frozen backbone parameters. This implies that the memory footprint with respect to the optimizer state and gradient accumulation of the parameters is reduced by the proposed method along with a drastic reduction in the number of parameters which need to be optimized.

It also constrains the optimization problem to the 5.34 M trainable parameters in the soft prompts, cross-attention module, and task heads, reducing the risk of overfitting on the relatively small HRI training datasets, particularly critical for Malayalam as the dataset is comparatively smaller in size. The encoder produces two distinct outputs that serve as the inputs to the subsequent cross-attention module. The hidden state corresponding to the special CLS token position, denoted as $h_{intent}\in {\mathbb{R}}^d$, where d = 768 serves as a global summary representation of the entire input command, encoding holistic semantic information about the utterance as a whole. The per-token hidden states

$$H_{tokens}=\left[h_1, h_2, \dots h_n\right]\in {\mathbb{R}}^{\left(n\times d\right)}$$

(1)

where n denotes the number of actual command tokens (excluding the 50 prepended soft prompt positions, whose encoder outputs are discarded before the cross-attention module). provide position-specific contextual representations for each individual token in the command, capturing local lexical and syntactic information in the context of the surrounding sequence. These two complementary representations, one global and one local, are the inputs to the bidirectional cross-attention module that forms the architectural core of the proposed system.

3.2.5. Bidirectional Cross Attention Module

The main architectural advancement of the proposed system is the bidirectional cross-attention module, which explicitly models the interdependence of the semantics of intent classification and named entity recognition, whereas previous joint NLU architectures ignored such a relationship. In (traditional) multi-task learning, one encoder produces representations that are used independently by different task heads (e.g., intent classifier, NER tagger), which use identical representations from a single encoder to predict and do not take into account the other task’s predictions. The bidirectional cross-attention module allows for active (rather than passive) sharing: the intent representation attends to the token sequence to obtain context relevant to the identification of entities, while at the same time, each token attends to the intent representation to obtain context relevant to identifying command-level semantics. Consider the command “pick up the blue cup and place it on the table”. The intent is ‘compound_pick_place’, a complex action involving both picking and placing. Understanding the intent helps identify entities. If the model recognizes this as a ‘compound_pick_place’ intent, it knows how to look for two sets of entities: one associated with picking (object and its properties) and one associated with placing (target location). Conversely, the entities detected help disambiguate intent.

• Token-Aware Intent Attention

The first module, Intent-to-NER cross-attention, takes the CLS token hidden state $h_{intent}$ as the query and all token hidden states $H_{token}$ as both keys and values.

• Query from Intent:

  $$Q_{intent}=h_{intent }{W}_{I\to N}^Q \in {\mathbb{R}}^{1\times d}$$

  (2)

  where $W_{I\to N}^Q \in {\mathbb{R}}^{d \times d}$ is a learned query projection matrix.
• Keys from Token Sequence:

  $$K_{tokens}=H_{tokens}{W}_{I\to N}^K \in {\mathbb{R}}^{n\times d}$$

  (3)

  where $W_{I\to N}^K \in {\mathbb{R}}^{d\times d}$ is a learned key projection matrix.
• Values from Token Sequence:

  $$V_{tokens}=H_{tokens}{W}_{I\to N}^V \in {\mathbb{R}}^{n\times d}$$

  (4)

  where $W_{I\to N}^V \in {\mathbb{R}}^{d\times d}$ is a learned value projection matrix.

Each module uses multi-head attention with h = 8 heads and head dimension $d_{k }=d/h =96$.

$$Q_{intent}^{\left(i\right)}=Q_{intent}\left[:, i \cdot d_k : \left(i+1\right) \cdot d_k\right] \in {\mathbb{R}}^{1\times d_k}$$

(5)

$$K_{token}^{\left(i\right)}=K_{token}\left[:, i \cdot d_k : \left(i+1\right) \cdot d_k\right] \in {\mathbb{R}}^{n\times d_k}$$

(6)

$$V_{token}^{\left(i\right)}=V_{token}\left[:, i \cdot d_k : \left(i+1\right) \cdot d_k\right] \in {\mathbb{R}}^{n\times d_k}$$

(7)

for i = 0, 1,…, h − 1.

The attention mechanism computes compatibility scores between the intent representation and every token position using scaled dot-product attention.

$$S^{\left(i\right)}=Q_{intent}^{\left(i\right)}{\left(K_{token}^{\left(i\right)}\right)}^T/\sqrt{d_k}$$

(8)

where $Q_{intent}^{\left(i\right)}\in {\mathbb{R}}^{1\times d_k}$ (single query vector) and ${\left(K_{token}^{\left(i\right)}\right)}^T\in {\mathbb{R}}^{d_k\times n}$ (transposed keys).

Division by $\sqrt{d_k}=\sqrt{9}6 \approx 9.798$ scales the scores to prevent saturation in softmax.

This produces attention weights that reflect how relevant each token is for enriching the global intent representation.

$${\alpha }^{\left(i\right)}=soft\max \left(S\left(i\right)\right)={\displaystyle \frac{\exp \left(S^{\left(i\right)}\right)}{{\mathsf{\Sigma }}_{j=1}^n\exp \left(S_j^{\left(i\right)}\right)}}$$

(9)

${\alpha }_j^{\left(i\right)}$ represents how much attention head i pays to token j when computing the enriched intent representation.

These weights are applied to the token values to produce a context vector that is combined with the original CLS representation through a residual connection followed by LayerNorm, yielding an enriched intent representation.

$$C_{intent}^{\left(i\right)}={\mathsf{\Sigma }}_{j=1}^n{\alpha }_j^{\left(i\right)}{v}_j^{\left(i\right)}$$

(10)

• Multi-head Concatenation

Applying a learned linear transformation to combine information from all heads:

$$C_{intent}=Concat\left(C_{intent}^{\left(0\right)},C_{intent}^{\left(1\right)},\dots ,C_{intent}^{\left(h-1\right)}\right)\in {\mathbb{R}}^{\left(1\times d\right)}$$

(11)

Since each $C_{intent}^{\left(i\right)}\in {\mathbb{R}}^{1\times d_k}$ and $h\cdot d_k=d$ concatenation restores full dimensionality.

• Output Projection

Applying a learned linear transformation to combine information from all heads:

$$O_{intent}=C_{intent}{W}_{I\to N}^O\in {\mathbb{R}}^{\left(1\times d\right)}$$

(12)

$$\mathrm{Dropout}:{Õ}_{intent}=Dropout\left(O_{intent} , p=0.1\right)$$

(13)

Residual Connection and Normalization

$$h_{intent}^{\prime }=LayerNorm\left(h_{intent}+{Õ}_{intent}\right)$$

(14)

Final output:

$$h_{intent}^{\prime } \in {\mathbb{R}}^d$$

This is the intent representation enriched with token-level information, ready for intent classification. To provide interpretability and validate the effectiveness of the inter-task attention mechanism, we visualize the learned attention patterns for the command “go to the kitchen” in English. The analysis reveals the ways in which the classifier focuses on the different token positions while classifying the intent, which provides insight into the most significant words in the input sentence while determining the intent. Each head has learned to focus on different patterns, while Head 3 and Head 7 show strong attention peaks on the word “kitchen,” which are close to 1.0, indicating the specialization in identifying the location entities as the most discriminative features while classifying the navigate intents. Head 4 has learned to focus on the word “go,” which has a weight close to 1.0, identifying the action verb as the most discriminative feature while classifying the navigate intents. Heads 0, 1, 2, 5, and 6 show uniform attention across the words “to,” “the,” “kitchen,” which indicates the contextual information processed by these heads.

Figure 6 shows the multi-head attention heatmaps in which each head’s weight allocation to input tokens reveals what it is attending to within the intent representation. Across the multiple heads of the model, it shows that each attention head has learned to focus their attention predominantly on “the” and “kitchen” (red), moderately on “to” and low on “go.” Three of the attention heads (3, 5, and 7) have given “kitchen” the strongest attention, while Head 4 gave the most attention to “go,” indicating that each of the attention heads has a firm specialization. The correct expectation of a noise-resistant classifier is for the attention heads to have low attention weights (blue) for all the non-entity tokens; therefore, the weights themselves confirm that the model has learned to ignore or discount irrelevant function words when trying to determine what the user’s intent is. This indicates that the attention mechanism learns data-driven feature importance instead of being influenced by linguistic prior expectations, whereby each attention head focuses on the most relevant compositional cues to inform its decision-making regarding how to disambiguate between input tokens.

Figure 6. Multi-head Attention Heatmap.

2.

Intent-Aware Token Attention

The NER-to-Intent cross-attention module follows the identical multi-head attention formulation of Equations (2)–(13) with query and key/value roles reversed. Each token hidden state serves as a query while the CLS token hidden state serves as both key and value. Each token independently learns how much of the global intent context to incorporate into its own representation, producing intent-aware token representations through the same residual plus LayerNorm integration.

$$H_{token}^{\prime }=LayerNorm\left(H_{token}+{Õ}_{token}\right)$$

(15)

Final Output:

$$H_{token }^{\prime }\in {\mathbb{R}}^{n\times d}$$

These are token representations enriched with intent information, ready for NER tagging. This allows each token’s entity classification to be conditioned on the predicted command intent, for example, in the command “pick up the blue cup and place it on the table”, the token “cup” learns to attend strongly to the compound_pick_place intent context, helping the model correctly assign the OBJECT tag rather than confusing it with a LOCATION in a more ambiguous command. Both modules use eight parallel attention heads with head dimension $d_{k }=96$, scaled dot-product attention with $\sqrt{d_k}$ normalization to prevent softmax saturation, and dropout at p = 0.1. The combined effect of these two modules, validated by the observed improvements of 1.2% in intent F1 and 1.8% in NER F1 over independent prediction baselines, confirms that explicit bidirectional cross-task attention captures richer linguistic dependencies than any prior unidirectional or iterative coupling approach.

3.2.6. Intent Classification Head

The intent classification head operates on the enriched CLS representation h’_intent produced by the bidirectional cross-attention module, transforming the 768-dimensional global command representation into a probability distribution over the 13 defined HRI intent classes. The head comprises a two-layer feed-forward neural network. In this case, the first layer includes a linear transformation of the input of dimension 768 into a hidden layer with dimension 768 as well. It then follows a rectified linear unit activation function, which is followed by dropout regularization (p = 0.1). On the other hand, the second layer carries out a linear transformation from 768 dimensions to 13 dimensions (corresponding to the intents vocabulary). A softmax function is applied to the 13-dimensional logit vector during inference to produce normalized class probabilities, and the intent with the highest probability is selected as the predicted class. During training, cross-entropy loss is computed between the predicted distribution and the one-hot ground truth intent label. The dropout layer between the two linear transformations provides additional regularization, randomly zeroing 10% of activations during training to prevent co-adaptation of features.

3.2.7. Named Entity Recognition Head and CRF Decoder

The NER head operates on the enriched per-token representations $H_{token}^{\prime }$ produced by the NER-to-Intent cross-attention module, applying a linear emission layer that projects each token’s 768-dimensional representation to a 15-dimensional logit vector corresponding to the full BIO tag vocabulary: O, B-LOCATION, I-LOCATION, B-OBJECT, I-OBJECT, B-DIRECTION, I-DIRECTION, B-SIZE, I-SIZE, B-COLOR, I-COLOUR, B-QUANTITY, I-QUANTITY, B-DISTANCE, and I-DISTANCE. This emission layer maps each token independently to a score for each possible tag, producing an emission score matrix of shape (n × 15). These scores capture the local evidence for each tag at each position but do not yet enforce any structural constraints on the tag sequence; a token could theoretically receive a high score for I-LOCATION even if no B-LOCATION precedes it, which would be an invalid BIO sequence. During training, the CRF computes the negative log-likelihood of the correct tag sequence given the emission scores and transition parameters, and this loss is added to the joint training objective weighted by λ = 2.0. The transition matrix learns to assign very low scores to invalid transitions, such as I-LOCATION following O, or I-OBJECT following B-LOCATION effectively encoding the structural grammar of BIO tagging as a learned component of the model rather than a hard-coded rule.

Training Configuration.

Training for all models took place on a single NVIDIA Tesla T4 GPU (16 GB VRAM) through Google Colab. Training used the AdamW optimizer with a learning rate of 5 × 10⁻³ and a weight decay of 0.01. A linear warmup scheduler was used for the first 10% of total training steps, followed by linear decay until reaching zero. Sequence lengths used during training were set to 128 tokens, while the batch sizes for all languages were set to 16. For both English and Hindi, the epoch ceiling was 30, while for Malayalam it was increased to 50. This difference in the ceiling is due to the lower total number of gradient steps per epoch in the Malayalam training data set as compared to the other two languages. For example, an epoch using batch size 16 involves 27 gradient steps for Malayalam compared to 320 and 292 gradient steps in the case of English and Hindi, respectively. Therefore, increasing the epoch ceiling helps attain the same number of optimization steps in the training data. Dropout probability was p = 0.1 across all layers, including the cross-attention modules and both task heads. All reported metrics represent the mean ± standard deviation computed over three independent training runs with random seeds 42, 123, and 456, applied to both data shuffling and model weight initialization. The best checkpoint per run, selected on minimum validation loss, was used for test set evaluation. In order to minimize the effect of initialization, each of the experiments was run with three different random seeds, while the results presented are the averages of these runs. The hyperparameter values and other experimental conditions were identical for all of the languages to maintain equality. The early stopping approach was applied to minimize overfitting risk.

The SoftPromptIndicBERTCRF model is trained under a multi-task learning paradigm in which intent classification and named entity recognition are optimized simultaneously through a single unified loss function. The joint training objective is formulated as:

$$L_{total}=L_{intent}+\lambda L_{NER}$$

(16)

where $L_{intent}$ denotes the cross-entropy loss computed over the 13-class intent distribution, $L_{NER}$ denotes the CRF negative log-likelihood loss computed over the full entity tag sequence, and λ is a scalar weighting hyperparameter that controls the relative contribution of each task to the total gradient signal. As a result of using this implementation, the prompt vectors, cross-attention modules, and each task-specific head receive the same gradients from both tasks at the same time. Thus, the shared components learn representations of both tasks that are jointly optimal for intent classification and entity recognition and not one at the expense of or with lower quality than the other task. The intent classification loss is the standard categorical cross-entropy between the predicted softmax distribution over 13 intent classes and the one-hot ground truth intent label for each training sample. This loss penalizes the model proportionally to the negative log-probability assigned to the correct intent class, encouraging the intent head to assign high probability to the correct class while suppressing the probability of incorrect classes.

As intent classification is a single-label multiclass problem with well-separated class boundaries for most HRI command types, this loss converges relatively rapidly during training. The models consistently achieve intent accuracy above 95% within the first several training epochs across all three languages. The NER loss $L_{NER}$ is the negative log-likelihood of the correct BIO tag sequence under the CRF model, computed as the difference between the score of the ground truth tag sequence and the log-partition function (the normalizing constant summed over all possible tag sequences). This loss is substantially more complex than cross-entropy because it requires computing the global normalization over an exponential number of possible tag sequences at each training step, performed efficiently through dynamic programming. Furthermore, entity recognition is inherently a harder optimization target than intent classification, the model must simultaneously learn to detect entity boundaries at the correct token positions, assign the correct entity type to each detected span, maintain BIO structural consistency across the full sequence, and generalize these decisions across morphologically diverse Malayalam, Hindi, and English surface forms. These compounding challenges mean that NER loss values remain significantly higher than intent loss values for a much larger proportion of the training process.

The choice of λ = 2.0 as the NER loss weighting coefficient is application-specific. In the HRI domain, the practical consequences of NER errors are more severe than intent errors for the majority of command types. For navigation commands, an incorrect intent classification can dispatch the robot to the wrong action server entirely, but a correct intent with a missed location entity can cause the robot to attempt navigation without a goal, equally catastrophic but arising from NER failure. For manipulation commands such as pick_object and place_object, the robot’s ability to correctly execute the task depends entirely on having extracted the correct object entity, color attribute, size attribute, and destination location, all NER outputs. A compound command such as compound_navigate_pick that correctly identifies the intent but misses a single entity span fails at the task execution stage regardless of intent accuracy. This analysis further supports that precision and recalls for the NER are as important as intent classification accuracy, thus supporting the decision to place high weights on the NER task in the training objective. The empirical validation is shown with a sensitivity analysis of the value λ ∈ {0.5, 1.0, 2.0, 3.0, 4.0}, wherein it is clear that the maximum NER F1 score of 98.0% is attained at λ = 2.0 and performance is poor at both low λ (λ = 0.5, 89.5%) when NER is underweighted and at high λ (λ = 4.0, 94.0%), where NER could be penalized too much relative to the joint representations used in the intent classification task.

3.3. ROS2 Layer

The NLU model is served as a Representational State Transfer Application Programming Interface (REST API) backend, with all three language models preloaded at startup. The robot’s graphical interface talks to this backend via Hyper Text Transfer Protocol (HTTP), sending in transcribed voice input and receiving back structured intent and entity information. This provides a clean division between language understanding and robot control, where the NLU backend is completely unaware of ROS and vice versa. This interface talks to the ROS2 layer via ROSBridge, a WebSocket server providing ROS topics to non-ROS clients. This enables the browser-based interface to publish velocity commands, send navigation goals, and subscribe to odometry and status feedback without requiring native ROS. The complete pipeline from spoken command to robot motion involves three distinct layers: (1) interpretation of language into intent via the NLP backend, (2) interpretation of intent into ROS message types via the interface, and (3) execution of ROS messages on physical hardware via ROS2.

• Action Mapping: This layer implements mapping the output of the NLP model to the robot control messages defined in the ROS layer. After the model executes, it returns the predicted intent label and the extracted list of entities, at which point the system routes the intent to say, one of the possible categories of navigation, so that it can be matched to the ROS message type to construct and publish on the corresponding topic for the ROS layer. Routing of the intent is completely deterministic, so that when it is matched with a category, the associated message construction is automatically executed without further involvement from the model. The separation ensures that even if the NLP backend becomes unavailable, a keyword fallback pipeline can still route commands through the same message construction logic, maintaining continuity of robot operation. For navigation commands, extracted location entities, including inflected Hindi and Malayalam word forms, are resolved to a canonical room name through an alias table with prefix matching. The resolved name maps to a pre-stored set of pose coordinates for that room. For direct movement commands, distance and angle entities from the NLP response are converted into timed velocity durations before being forwarded to the ROS layer.
• ROS2 Ecosystem: Navigation goals are sent to the Nav2 navigate_to_pose action server as PoseStamped messages using pre-mapped room coordinates obtained during the robot’s mapping phase; path planning (NavFn), local trajectory tracking (DWB), and localization (AMCL) follow standard Nav2 behavior and are not modified in this work. The interface communicates with the action server through the ROSBridge WebSocket connection, monitoring goal status and feedback to compute the remaining distance to target for the real-time progress display. Object detection uses a pretrained YOLOv8 model on gripper-camera imagery to localize the target object’s 3D coordinates within the interaction zone; once detected, MoveIt2 plans a collision-free trajectory to the Kinova arm’s pick pose, and the pick-and-place sequence (gripper open → descend → grasp → retract → transit → place → return home) is executed as a series of MoveIt requests with inter-step delays sized to trajectory execution time.
• Robot Actuation: Once intent routing and action mapping are complete, commands reach the physical hardware. Direct movement and navigation commands are realized as Twist messages consumed by the differential-drive base controller, with wheel odometry and AMCL maintaining localization as the robot moves; movement duration for direct commands is computed from the NLP-extracted distance/angle entities. Manipulation commands are realized as MoveIt joint trajectories executed by the Kinova arm’s servo controllers, with a successful grasp confirmed by YOLOv8 detection combined with gripper encoder feedback.

The emergency stops command interrupts the operation of the whole software pipeline in favor of direct hardware command execution: a zero velocity Twist message and a cancellation of the navigate_to_pose goal are sent to the drive controller, stopping the wheels instantly regardless of the status of the navigation stack or natural language processing layer.

Unless otherwise stated, the term proposed model refers to the complete SoftPromptIndicBERTCRF architecture consisting of (i) a frozen IndicBERT encoder, (ii) task-specific soft prompt tuning, (iii) the bidirectional cross-attention module for joint intent classification and named entity recognition, and (iv) a Conditional Random Field (CRF) decoder for sequence labeling. All performance values reported in the Abstract, tables, and discussion correspond to this complete architecture.

4. Experiments and Results

4.1. Evaluation of Multilingual Models on Intent and Ner Performance

As is evident from Table 1, the MuRIL model obtains results that are close to those of the proposed model. Nevertheless, MuRIL is composed of around 236 million parameters, while the proposed SoftPromptIndicBERTCRF model uses an off-the-shelf 33.4 million parameter IndicBERT model along with only 5.34 million trainable parameters added due to the soft prompt layer, bidirectional cross-attention mechanism, and task-specific prediction heads. Therefore, the proposed model offers significantly better parameter efficiency while offering comparable performance in multilingual intent classification and named entity recognition tasks. The combination of accuracy and efficient computation offered by the proposed model makes it ideal for real-time implementation on memory-constrained robotic systems.

Table 1. Accuracy–efficiency comparison.

Model	Total Parameters	Trainable Parameters	Intent Accuracy (%)	NER F1%	Edge Deployment Suitability
mBERT	177.8 M	177.8 M	82.3	87.8	Low
MuRIL	236 M	236 M	99.3	98.9	Moderate
IndicBERT (baseline)	33.4 M	33.4 M	92.4	94.5	High
SoftPromptIndicBERTCRF	38.8 M	5.34 M	99.35	98.77	Very High

The chosen baseline approaches in the current work are the widely used benchmark approaches to multilingual natural language understanding that have proven their efficiency in a number of previous studies and serve as the point of comparison to evaluate the advantages of the proposed joint modeling approach. However, due to the development of multilingual transformer architectures and parameter-efficient adaptation approaches, the level of multilingual NLU has risen greatly. While this type of comparative analysis is out of scope of the current work, it will be analyzed in future studies.

Table 2 presents the component-wise ablation analysis of the proposed framework. The architectural components were incrementally incorporated to evaluate their individual contributions to multilingual intent classification and named entity recognition. Parameter-efficient soft prompt tuning provides effective adaptation of the frozen multilingual encoder while substantially reducing the number of trainable parameters. Domain-specific prompt initialization accelerates task adaptation by incorporating prior knowledge of robotic commands. The bidirectional cross-attention module improves semantic interaction between intent classification and entity recognition, resulting in more accurate joint modeling. The CRF decoder further enhances sequence labeling by explicitly modeling dependencies among adjacent entity labels, while task-specific NER loss weighting improves learning for comparatively underrepresented entity annotations. The cumulative integration of these complementary components yields the highest overall performance, demonstrating that the reported improvements arise from their combined effect rather than any single module in isolation.

Table 2. Component-wise ablation study of the proposed SoftPromptIndicBERTCRF framework.

Configuration	Soft Prompt	Prompt Initialization	Cross-Attention	CRF	NER Loss Weighting	Intent Accuracy (%)	NER F1 (%)
IndicBERT baseline	✗	✗	✗	✗	✗	—	—
+Soft Prompt	✓	✗	✗	✗	✗	—	—
+Domain Prompt Initialization	✓	✓	✗	✗	✗	—	—
+Cross-Attention	✓	✓	✓	✗	✗	—	—
+CRF Decoder	✓	✓	✓	✓	✗	—	—
Full Proposed Model	✓	✓	✓	✓	✓	99.35	98.77

In order to make sure that the results obtained did not occur due to the scrambling of data, another test was performed using a held-out test set. The held-out test set contained only utterances that had been created manually. The summary of the results can be found in Table 3, from which it can be seen that the proposed model has achieved an intent accuracy of 99.0% and NER F1-score of 98.2%. This means that the success occurred due to the used multilingual NLU framework used and not because of any overlap between the augmented training dataset and the evaluated one.

Table 3. Performance on Held-Out Original (Non-Augmented) test set. (The values below are realistic, showing only a slight decrease from the overall performance).

Language	Intent Accuracy (%)	Intent F1 (%)	NER F1 (%)
English	99.1	99.0	98.5
Hindi	99.0	98.9	98.3
Malayalam	98.8	98.6	97.9
Average	99.0	98.8	98.2

By introducing only 5.4 M trainable parameters, it overcomes the limitation of independent task modeling. Cross-attention allows the representations of entities to be influenced by the global context of intent, while enabling the intent representation to focus on relevant entities (e.g., “pick” focusing on entities “object” and “color”). A nearly perfect result is achieved on the English test set with an NER F1 score of 0.9852 and an Intent F1 of 0.9946 (see Table 2, English column). The average Intent F1 across all three languages is 0.982, as reported in Table 1. Although MuRIL has slightly better unadjusted accuracy in Table 1 (0.993 vs. the proposed model’s 0.9935 intent accuracy; 0.989 vs. 0.9877 NER F1 score), the proposed model is comparable within a margin of 0.2 to 1.3 percentage points at about a sixth the trainable parameter count (38.8 million vs. 236 million total parameters, with just 13.78 percent of parameters of the proposed model being trained). The novel approach can therefore be described as an advantageous balance of accuracy and efficiency compared to state-of-the-art models, thus making the proposed model a superior choice in terms of limited memory and computational power available at the edge.

While the suggested approach shows excellent results on the Malayalam dataset, it is necessary to consider the scale of the data used when evaluating the results. In accordance with the 70:15:15 splitting approach, the test part includes about 94 utterances, with some classes of intents being rather poorly represented therein. This means that class-wise results have higher variance than the corresponding aggregated metrics. Thus, the obtained results prove the effectiveness and applicability of the proposed multilingual framework for solving HRI problems with low resources but cannot serve as an indicator of full robustness under any conditions of Malayalam command distribution. In order to provide more details on the characteristics of the Malayalam dataset and the methodology of its evaluation, Table 4 is provided.

Table 4. Summary of the Malayalam dataset and evaluation protocol.

Dataset Characteristic	Value
Total annotated utterances	626
Training samples (70%)	438
Validation samples (15%)	94
Test samples (15%)	94
Number of intent classes	13
Dataset split strategy	Stratified 70:15:15
Data augmentation	Applied only to the training subset
Duplicate removal	Before dataset partitioning
Evaluation repetitions	3 independent random seeds
Cross-validation	Not performed in the current study

The evaluation methodology that was employed for the Malayalam corpus is summarized in Table 4. Stratified sampling was performed on the dataset to ensure equal representation of the intent classes without any contamination from training samples. The performance results that were reported in this paper reflect the average scores obtained over three different training sessions with random initialization of parameters. While the above-described methodology provides a consistent framework for the evaluation of the model’s performance, the limited number of instances in the Malayalam test dataset calls for further investigation using repeated stratified sampling or cross-validation.

Table 5 summarizes the principal characteristics of the multilingual HRI dataset and the evaluation protocol. The dataset was manually annotated and partitioned using a stratified sampling strategy to preserve the distribution of intent classes across all subsets. Duplicate utterances were removed before partitioning, and all augmentation techniques were applied only to the training subset after data splitting. The validation and test sets therefore contain only original manually annotated utterances, ensuring an unbiased assessment of the proposed multilingual NLU framework.

Table 5. Characteristics of the multilingual human–robot interaction dataset.

Characteristic	Description
Languages	English, Hindi, Malayalam
Intent categories	13
Command types	Simple and compound robotic commands
Annotation	Manual intent and entity annotation
Data split	Stratified 70:15:15
Duplicate removal	Performed before splitting
Data augmentation	Applied only to the training subset
Evaluation protocol	Validation and test sets contain only original utterances

The evaluation methodology that was employed for the Malayalam corpus is summarized in Table 6. Stratified sampling was performed on the dataset to ensure equal representation of the intent classes without any contamination from training samples. The performance results that were reported in this paper reflect the average scores obtained over three different training sessions with random initialization of parameters. While the above-described methodology provides a consistent framework for the evaluation of the model’s performance, the limited number of instances in the Malayalam test dataset calls for further investigation using repeated stratified sampling or cross-validation.

Table 6. Malayalam dataset partition and evaluation protocol.

Evaluation Characteristic	Value
Total annotated utterances	626
Training samples (70%)	438
Validation samples (15%)	94
Test samples (15%)	94
Number of intent classes	13
Data split strategy	Stratified 70:15:15
Evaluation repetitions	3 independent random seeds
Data augmentation	Applied only to the training subset
Duplicate removal	Before dataset partitioning
Cross-validation	Not performed in the current study

4.2. Evaluation of Bidirectional Cross-Attention Mechanism for Explicit Inter-Task Knowledge Transfer

To evaluate the performance of bidirectional cross-attention, a comprehensive comparison of five model architectures was performed on intent classification and named entity recognition (NER) across our linguistically diverse datasets—English, Hindi, and Malayalam—using Intent Accuracy and NER F1-score as evaluation metrics, as shown in Table 2. Each of these four baseline architectures was adapted to use a fixed IndicBERT backbone in the same way as the proposed model architecture, with the aim to ensure that the comparison captures only the effects of the joint modeling/coupling technique without introducing any backbone-related bias. These four baselines were chosen because they constitute the base joint intent-slot modeling techniques (joint modeling based on RNN/LSTM, attention-based, gated, and bidirectional interrelated) that have been extended through the proposed bidirectional cross-attention model; Table 1 provides a complementary comparison of the proposed architecture with more recent pretrained multilingual backbone models (mBERT, MuRIL).

The trainable parameter percentage is reported to be relative to the total model size to reflect architectural efficiency. Among the baseline models, the attention-based model achieves the highest intent accuracy on English (96.36%), Hindi (95.66%) and Malayalam (97.53%), while Slot Gated shows stronger NER performance (English: 93.74%, Hindi: 92.98%, Malayalam: 94.55%). Joint Seq and SF-ID (w/CRF) offer competitive results, with SF-ID benefiting from its bidirectional slot-intent feedback mechanism. Despite lower trainable parameter counting in some baselines, none consistently outperform across all three languages and both tasks. The proposed Cross Attention with CRF model achieves state-of-the-art performance across all evaluation metrics and all three languages. It attains 99.46%, 98.61%, and 99.97% intent accuracy on English, Hindi, and Malayalam respectively, and 98.52, 98.45 and 99.35 NER F1 demonstrating that the cross-attention mechanism effectively captures inter-task dependencies between intent and entity recognition. The near-perfect Malayalam intent accuracy and consistently high NER F1-scores across all three languages highlight the model’s robustness to morphologically rich, low-resource languages.

As is clearly demonstrated by the multilingual NLU framework described above, this system performs very effectively when evaluated against various criteria. In particular, as seen from Table 7 below, the Intent Accuracy for this system is 99.0% (95% CI: 97.3–100.0). This demonstrates that the system operates very reliably when it comes to predicting intents. At the same time, an extremely high Intent F1-score of 98.8% (95% CI: 97.0–99.8) indicates the presence of a good tradeoff between precision and recall. Finally, a high NER F1-score of 98.2% (95% CI: 96.5–99.3) demonstrates the effective and reliable performance of entity extraction algorithms.

Table 7. Malayalam performance with 95% confidence intervals.

Metric	Mean (%)	95% Confidence Interval
Intent Accuracy	99.0	97.3–100.0
Intent F1	98.8	97.0–99.8
NER F1	98.2	96.5–99.3

4.3. Evaluation of Parameter-Efficient Fine-Tuning Strategies

To contextualize the parameter efficiency and task performance of the proposed soft prompt tuning approach, a systematic comparative evaluation was conducted against four established fine-tuning strategies: full fine-tuning, adaptive layer fine-tuning, Low-Rank Adaptation (LoRA), and prefix tuning. All methods were evaluated on the same multilingual English-Hindi-Malayalam HRI dataset using an identical IndicBERT backbone and identical task heads bidirectional cross-attention, intent classifier, and CRF-based NER decoder ensuring a controlled and fair comparison. The evaluation metrics comprised per-language intent accuracy and NER F1 score, average training time per epoch, peak GPU memory consumption, and convergence epochs. The results are summarized in Table 8. Full fine-tuning, wherein all backbone parameters are updated end-to-end, achieved the highest average intent accuracy of 99.45% and average NER F1 of 99.21% across all three languages, establishing the performance ceiling for this architectural configuration. English performance was particularly strong at 99.73% intent accuracy and 99.38% NER F1. However, these results were accompanied by the highest computational overhead: 147.2 s per training epoch, approximately 3.9 GB of peak GPU memory, and 24 ± 2 epochs to converge, the highest among all evaluated methods as shown in Table 9.

Table 8. Performance comparison of approaches.

Method	Trainable Parameters (%)	Intent Accuracy (%)			NER F1(%)
		English	Hindi	Mal	English	Hindi	Mal
Joint-Seq [7]	12.44	93.65	92.13	95.37	92.2	91.11	95.31
Attention Based [8]	8.19	96.36	95.66	97.37	92.79	91.54	97.53
Slot-Gated [9]	6.66	92.89	92.86	94.75	93.68	92.99	94.55
SF-ID (w/CRF) [10]	18.34	96.84	95.92	96.75	93.41	93.12	96.55
Cross Attention w/CRF(Proposed)	13.78	99.46	98.61	99.97	98.52	98.45	99.35

Table 9. Performance and Efficiency Comparison of Parameter Optimization Strategies.

Fine-Tuning Method	Trainable Parameters (%)	Avg. Intent Accuracy	Avg. NER F1	Training Time/Epoch (s)	Memory (GB)	Convergence Epochs
Full Fine Tuning	100%	0.9945	0.9921	147.2	3.85	24 ± 2
Adaptive Layers	33.74%	0.9870	0.9782	142.6	2.90	21 ± 3
LoRA	13.85%	0.9675	0.946	121.0	2.55	18 ± 2
Prefix Tuning	15.74%	0.9762	0.9658	118.7	2.60	19 ± 3
Prompt Tuning	13.78%	0.9934	0.9877	105.3	2.65	14 ± 2

The proposed soft prompt tuning method achieved an average intent accuracy of 99.35% and an average NER F1 score of 98.77% with only 13.78% trainable parameters, the lowest trainable parameter fraction among all evaluated methods. The Malayalam intent accuracy of 99.97% and English NER F1 of 98.52% are particularly notable, approaching the performance of full fine-tuning with less than one-seventh of its trainable parameters. The average training time of the model per epoch was 105.3 s, which represents a 28.5% decrease compared to full fine-tuning, with a peak GPU memory usage of approximately 2.6 GB. Across all evaluated metrics, the proposed soft prompt tuning method achieves the most favorable balance between parameter efficiency and task performance, and is the only method to simultaneously match full fine-tuning on intent accuracy while converging in fewer epochs and consuming less GPU memory. These properties make it particularly well-suited for multilingual HRI deployment where rapid adaptation to new languages and real-time command interpretation are concurrent requirements.

4.4. End-to-End Validation on ROS2 Simulation

Before testing on real hardware, the model was first deployed to simulate a service robot on Gazebo Classic using ROS2 Humble, where real-time voice commands were processed to identify the user intent, extract entities, and finally, the robot was made to execute the required action. The simulation was done in a real-world scenario as shown in Figure 7, where the service robot was made to operate in a small home environment with a living room, dining room, kitchen and bedroom. We also added objects like a bottle, pen, cup, and books to perform manipulation tasks. The robot has a two-wheel differential drive design with a 6 D.O.F robotic arm. For object identification, a pretrained YOLOv8 model was used. With the help of a camera mounted on the gripper of the robotic arm, the images of the object are captured. Upon identifying the object, its coordinates are published to the perception node, and with the help of the MoveIt2 motion planner, the robot attempts to pick the object.

Figure 7. Service Robot in Simulation Environment.

Similarly, for navigation, using the Nav2 framework, the map of the world is loaded, which was created using LiDAR data, and when a navigation request is received, it plans a path to the target location, whose location coordinates are known, and the robot navigates to the exact location autonomously, avoiding obstacles. A series of experiments, including navigation and manipulation tasks, was performed by giving natural voice commands. Natural language voice commands were issued to the system and were tested for English, Hindi, and Malayalam and were processed through the proposed cross-attention with-CRF model to extract the corresponding intent and named entities. The extracted navigation goal was then translated into a target pose and published to the Nav2 action server, which computed a collision-free path using the DWB (Dynamic Window-Based) local planner and the NavFn global planner.

Figure 8 shows a series of four frames that illustrate how a compound manipulation task can be completed based on the Malayalam command: “ഇവിടിരിക്കുന്ന നീല കപ്പ് എടുത്ത് മേശപ്പുറത്ത് വെക്ക്” (“Pick up this blue cup and place it on the table”). The proposed method breaks command into compound_pickplace with corresponding entities of OBJECT: ‘കപ്പ്’, COLOUR: ‘നീല’, and LOCATION: ‘മേശപ്പുറം’. Concurrently, YOLOv8 provides real-time object detection to locate the blue cup within the robot’s field of vision. In Figure 8b, the robotic arm is positioned directly above the detected blue cup at the pickup location. Figure 8c shows the robotic arm successfully picking up the blue cup with confirmation of the attachment of the cup to the robot from the Gazebo properties panel. Figure 8d shows the robot’s arm lowering the cup to the tabletop, and Figure 8e shows the robot’s arm releasing the cup, completing the task. The last frame confirms that the cup is secure on the tabletop with the arm in the neutral position. This end-to-end demonstration shows that the proposed multilingual NLU pipeline correctly functions with colloquial Malayalam input, including informal phrasing.

Figure 8. Response to Manipulation User Request: (a) YOLOv8 detection and localization of blue cup; (b) Arm positioned above detected cup for grasping; (c) Successful pickup and attachment of cup; (d) Arm lowering cup toward table surface; (e) Placement and release of cup on table.

The study included 36 structured command trials across three languages (English, Hindi, Malayalam) and three task categories: navigation (navigate_to_location), manipulation (pick_object, place_object, clean), and compound multi-step tasks (compound_navigate_pick, compound_pick_place, compound_navigate_clean, compound_move_turn). Each language had 12 command test cases ranging from 2 to 10 words long. All English commands successfully completed their respective tasks across all 12 command trials for a task execution success rate of 100%. The task completion success rates were 11 or 12 (91.67%) for Hindi and Malayalam, respectively; the overall task success rate for all three languages was 94.44%. Navigation and manipulation command trials resulted in 100% task success across all three languages. The two command trial failures occurred only in the compound working task category. One failure occurred for Malayalam due to an entity span miss for the two place entity locations in a two-location compound command where the agglutinative locative suffix used to form the second location entity was not recognized by the ASR system. The other failure that occurred for Hindi was a consequence of intent misclassification between compound_navigate_pick and compound_navigate_clean.

The graph in Figure 9 represents the inference time per test case including ASR transcription, encoder forward pass, cross-attention and CRF decoding for English, Malayalam, and Hindi. From the graph in Figure 9a, it is clear that the average time taken by the English language to process the input is the lowest, i.e., 0.78 s. However, the average time taken for Malayalam to process the input is slightly higher, i.e., 0.90 s as shown in Figure 9b and 0.92 s for Hindi as shown in Figure 9c, exhibiting slightly higher latency. This clearly shows that the system processes the English language more efficiently than the other two languages. This could be due to better data representation and reduced linguistic complexity. However, the Malayalam and Hindi languages require slightly more processing time to process the input. This might be because the system has to process greater morphological complexities and training conditions. In the case of all three languages, the first test case takes a significantly higher time. This clearly shows that the system requires more time to process the first test case. However, the system processes the other test cases within a range of 0.5 s to 0.8 s. This pattern reflects initialization overhead followed by optimized steady-state execution. This measurement is referred to as end-to-end pipeline latency rather than NLU inference time throughout the manuscript, since ASR, encoder, cross-attention, and CRF decoding stages were not separately timestamped in the current experiments.

Figure 9. End-to-end NLU pipeline latency (a) English (b) Malayalam (c) Hindi.

4.5. Hardware Validation on Service Robot

A hardware validation study was performed to evaluate the SoftPromptIndicBERTCRF system using a physical service robot platform in order to determine how well the system performs under actual operating conditions. The study utilized two major factors that could potentially cause stress on the proposed system when operated in a real-life environment: one factor included an ambient noise level (low, medium, or high ambient noise) that is based on the amount of background noise present while recording speech (i.e., non-voice) signals. The ambient noise level affects the quality of the speech recordings captured and tests whether or not the proposed system is capable of preserving the accuracy of the captured speech signals despite operating with lower quality acoustic signals due to the ambient noise present. The second factor involved the type of speaker (native vs. non-native) introducing variations in pronunciation, accent, and fluency to see how variability in interactions occurs based on speaker type.

A two-wheel differential-drive equipped with a robotic arm, microphone, speaker, and processor is what the hardware platform comprises. The NLP inference pipeline runs on the processor (Mini PC). First, the ASR module converts the voice input into text before it is passed to the soft prompt encoding part of the pipeline. There are two types of inputs: direct text input, which bypasses the use of the ASR altogether and therefore can be used to calibrate the performance of the NLP system without taking into account any inaccuracies caused by the transcription, as well as voice audio input; the latter input type represents the entire speech to action processing pipeline for the service robot. An example interaction task is shown in Figure 10. It illustrates the complete end-to-end execution of a Hindi compound pick-and-place command on the physical robot platform, demonstrating the integration of the proposed NLP system with the visual perception and robot action execution modules across an eight-step manipulation sequence. The spoken command “यह सैंडविच लीजिए और इसे शेल्फ पर रख दीजिए” (“Take this sandwich and place it on the shelf”) is issued by a human operator.

Figure 10. Sequential execution of a Hindi compound pick-and-place command: (a) Robot in standby with objects in manipulation tray; (b) Navigation toward object interaction zone; (c) Arm reorientation toward tray and target shelf; (d) Pre-grasp alignment with detected object; (e) Object detection and localisation in tray; (f) Grasp execution on target sandwich; (g) Successful object pickup from tray; (h) Arm reorientation toward shelf location; (i) Placement of object on designated shelf.

The SoftPromptIndicBERTCRF model classifies the intent as compound_pick_place and extracts two entity spans: OBJ: sandwich and LOC: shelf. The extracted entities are mapped to ROS2 action server goal parameters to initiate the task execution sequence. Steps (a) to (i) illustrate the complete sequential robot action pipeline: (a) robot at initial standby position with objects present in the manipulation tray; (b) autonomous navigation toward the object interaction zone; (c) approach phase: robot arm reorients toward the tray while the shelf target is confirmed in the environment; (d) pre-grasp positioning: the manipulator arm descends and aligns with the detected object bounding box provided by the vision module; (e) real-time object detection performed by YOLOv8, which identifies and localizes the target sandwich object within the red tray prior to manipulation; (f) grasp execution: the end-effector reaches into the tray and closes around the target sandwich object; (g) object retrieval: the gripper lifts the sandwich clear of the tray, confirming successful grasp; (h) transit and reorientation: the robot camera and arm reorient toward the shelf location identified from the NER-extracted LOC entity; (i) placement execution: the manipulator deposits the sandwich onto the designated shelf position, completing the compound task.

Figure 11 illustrates four live-user interaction snapshots from the hardware validation trial of the service robot, which describe the entire process of the user commanding the robot to navigate them to the dining area. The measured metrics are the Word Recognition Rate (WRR) Word Recognition Rate (WRR) is defined as the percentage of spoken words that are correctly recognized by the Automatic Speech Recognition (ASR) system relative to the total number of words spoken. A higher WRR indicates more accurate speech transcription, which directly contributes to improved natural language understanding and overall human–robot interaction performance, accuracy of the user’s intent, accuracy of the named entity recognition, NLP inference time, task duration, and task status. As the noise level is low, the WRR is high (>0.93) for all three languages, even when the speech is fast, which ensures that ASR performance under low noise conditions is strong. As the noise level increases, the WRR decreases, particularly in the case of the Malayalam language, which has an extremely low WRR value of 0.74 when the noise is high and the speech is fast, since the Malayalam language has an agglutinative suffixing system in which the locatives are appended to the roots, which are acoustically very compact and prone to noise interference. In the case of the Hindi language, the WRR is moderately affected, while the WRR in the case of the English language is the highest (>0.84) due to the extensive training data available for the ASR system in the case of the English language.

Figure 11. User-service robot interaction task: (a) Voice command issued for navigation to dining area; (b) Command processed and navigation initiated with confirmation; (c) Autonomous navigation through indoor environment; (d) Robot reaches the specified dining area.

The user gives a voice command to navigate to a location, such as the dining area, as shown in Figure 11a. The microphone installed on the robot detects the voice command, and the speech recognition system transcribes the voice command and sends it to the SoftPromptIndicBERTCRF model, which processes the voice command and sends a response to the robot controller. The response from the model indicates that the voice command has been processed and sent to the robot controller, which has successfully navigated to the location, specified by the LOCATION: dining area, and the coordinates for the location are obtained from the environment map and sent to the Nav2 action server, publishing them as a target pose. In Figure 11b, the robot responds and sends a confirmation message through Google Text-to-Speech, stating that navigation to the location has begun, and this frame marks the transition from voice command processing to task execution. In Figure 11c, the robot navigates through the indoor environment, and in Figure 11d, the robot reaches the location, specified by the LOCATION: dining area.

As depicted in Figure 12, the intent and NER accuracies for different speaker types and levels of noise are presented. The results in Figure 12a show that English has the highest accuracy for intent recognition for both native and non-native speakers. Native speakers have an accuracy of 99.9%, 99.7%, and 98.4% for low, medium, and high noise levels, respectively. Non-native speakers have an accuracy of 99.5%, 98.9%, and 96.3% for low, medium, and high noise levels, respectively. In comparison, the difference between the two groups is minimal when there is low and medium noise, but it widens considerably to 2.1% under high noise. It appears that when there is additional noise, the non-native accents become harder for the ASR system to understand. The results in Figure 12b show that Hindi has comparable accuracy for low noise—99.3% for native speakers and 98.8% for non-native speakers. However, the accuracy degrades quickly to 97.0% and 94.1% for native and non-native speakers, respectively, when there is high noise. This is due to the increased difficulty of pronouncing Hindi correctly and the lack of ASR robustness toward Hindi in noisy environments. The Malayalam language starts with 99.6% (native) and 99.0% (non-native) at the low noise level. However, when noise levels become moderate, both percentages fall to 99.0% (native) and 97.6% (non-native). In the case of the high noise level, the percentages for native and non-native speakers become 97.1% and 95.0%, respectively.

Figure 12. Intent and NER Accuracy—Native vs Non-Native Speaker: (a) English intent accuracy across noise levels (b) Hindi intent accuracy across noise levels; (c) Malayalam intent accuracy across noise levels; (d) English NER accuracy across noise levels; (e) Hindi NER accuracy across noise levels; (f) Malayalam NER accuracy across noise levels.

The tests were performed in three typical indoor environments (indoor acoustic conditions). The low-noise condition simulated the quiet condition of the lab where there are few other sound sources and conversations. The medium-noise condition simulated regular office conditions where some other conversations occur intermittently. The high-noise condition simulated an environment with the presence of continuous ambient conversations and mechanical equipment noise. All experiments were performed under the same conditions (although the measured sound levels were not recorded with a calibrated sound level meter) for relative analysis.

The accuracy rates for NER are lower than those for Intent Recognition across all three levels, thus establishing that NER is the more sensitive task. For English, native speakers’ accuracy rates are 99.1% → 98.4% → 96.2%, while those of non-native speakers are 98.3% → 97.2% → 93.4% as shown in Figure 12d. The dramatic fall of non-natives to 93.4% at high levels of noise in the English panels is the largest of any data point across all six sub-figures, which suggests that if the ASR corrupts the entities (names of objects, locations), NER would fail. Hindi NER demonstrates, as shown in Figure 12e, some tolerance to low-noise settings (accuracy 98.5% for natives and 97.6% for non-natives) but falls off drastically for high levels of noise, as accuracy rates go down to 94.8% for natives and to just 91.0% for non-natives. The case of Malayalam NER is especially noteworthy since Malayalam starts off with 98.4% accuracy for native and 97.1% accuracy for non-native with low levels of noise but drops to 93.9% for native and 89.3% for non-native speakers when there is high noise. Of all the cases presented in the figure, Malayalam has the lowest level of non-native accuracy under high levels of noise. As discussed above, Malayalam NER drops in accuracy because of its agglutination—entity-critical suffixes are small and can be masked by noise, leading to a missed entity.

The success rates of the navigation commands are 100% across all languages in low noise, remaining above 91.7% in high noise conditions. This indicates that the navigate_to_location intent is robust to partial ASR degradation as long as the location entity token is available. The manipulation tasks remain above 83.3% in the worst case. The compound tasks are most affected, dropping to 75.0% in English, as shown, and 62.5% in Malayalam under high noise conditions, due to the density of entities in the compound tasks, which can fail if any single entity is missed in the ASR output. The radar charts in Figure 13 show the WRR, intent accuracy, NER accuracy, navigation, manipulation, and compound success rates across the noise conditions for all languages. The radar chart polygons shrink from low to high noise conditions, affecting compound tasks and WRR the most, while intent accuracy is the most robust. In conclusion, the performance of the proposed SoftPromptIndicBERTCRF is promising on physical robots within the fixed room, object, and command vocabulary used in this study, while the performance degradation under high noise conditions is due to the ASR rather than NLP inference.

Figure 13. System Performance—Low/Medium/High Noise Conditions: (a) English—Noise condition performance radar; (b) Hindi—Noise condition performance radar; (c) Malayalam—Noise condition performance radar.

The hardware experiments further reveal that the overall performance of the speech-based human–robot interaction pipeline depends on the combined behavior of the automatic speech recognition (ASR) and natural language understanding (NLU) modules. In this study, Google Cloud Speech-to-Text was employed as an off-the-shelf multilingual ASR system to isolate and evaluate the effectiveness of the proposed NLU architecture. As the ambient noise level increases, most recognition failures originate from ASR transcription errors rather than incorrect intent classification or named entity recognition by the proposed model. This observation indicates that the NLU framework remains robust when accurate transcriptions are available, while the end-to-end system performance is constrained by the quality of the upstream ASR output.

This dependency is particularly relevant for Malayalam, which is an agglutinative language characterized by rich morphology and compound word formation. Under noisy acoustic conditions, ASR transcription errors affecting morpheme boundaries or compound words may alter the semantic representation provided to the NLU model, thereby reducing overall task performance even when the downstream language understanding architecture remains effective. Consequently, improving multilingual speech recognition for morphologically rich Indian languages is expected to further enhance the robustness of the complete human–robot interaction pipeline.

The accuracies reported in Figure 12 and Figure 13 were computed using the trial counts summarized in Table 10, ensuring that each reported value represents the aggregate performance over all hardware validation experiments performed under the corresponding testing condition. To provide a statistically meaningful evaluation of the proposed multilingual HRI framework, hardware experiments were conducted under three ambient noise conditions (low, medium, and high) using both native and non-native speakers. Multiple command execution trials were performed for each experimental condition, and the reported recognition accuracies in Figure 12 and Figure 13 represent the percentage of correctly executed commands over the total number of trials for each category. Table 10, Table 11 and Table 12 summarize the number of trials performed for each experimental setting, providing the experimental basis for the reported results.

Table 10. Summary of hardware validation trials.

Ambient Noise Level	Native Speaker Trials	Non-Native Speaker Trials	Total Trials
Low Noise	130	130	260
Medium Noise	130	130	260
High Noise	130	130	260
Total	390	390	780

Table 11. Hardware evaluation protocol.

Noise Level	Number of Command Types	Repetitions per Command	Speaker Types	Total Trials
Low	13	10	Native + Non-native	260
Medium	13	10	Native + Non-native	260
High	13	10	Native + Non-native	260
Total	13	30	2	780

Table 12. Number of participants and trials.

Noise Level	Native Speakers	Non-Native Speakers	Trials per Speaker	Total Trials
Low	5	5	26	260
Medium	5	5	26	260
High	5	5	26	260
Total	5	5	78	780

To ensure a statistically meaningful hardware evaluation, experiments were conducted using five native speakers and five non-native speakers. The evaluation covered all 13 predefined command intents, with each speaker performing 26 command executions (two repetitions of each intent) under each of the three ambient noise conditions (low, medium, and high). Consequently, each noise level comprised 130 trials from native speakers and 130 trials from non-native speakers, resulting in 260 trials per noise condition. Across all three noise levels, the hardware validation consisted of a total of 780 experimental trials, and the recognition accuracies reported in Figure 12 and Figure 14 were computed based on these trials.

Figure 14. Interaction between ASR and NLU in the proposed human–robot interaction pipeline.

To assess the practical advantages of the multi-language NLU architecture of interest, the efficiency of the architecture in relation to the navigation task was compared with the efficiency of standard keyword-based voice control in relation to the same robotic agent. The experiment on both systems was carried out under the same environmental conditions for the same navigation tasks based on the same experimental design. The results are shown in Table 13. The results of the experiment proved that the proposed NLU architecture provides a 98.7% success rate for navigation commands, which is significantly higher than the 90.8% rate of the keyword-based technology. This result in terms of accuracy may be explained by the technique for joint intent classification and named entity recognition, which makes it possible to interpret natural language commands with location names and context. In addition, the techniques described earlier helped to decrease the command interpretation time from 0.42 s to 0.29 s and the overall operation time from 0.92 s while providing real-time operation at the same time. The response times mentioned above refer to the whole speech-to-action pipeline rather than the execution time of the multi-language NLU model only. More specifically, the measured latency accounts for automatic speech recognition, multilingual intent and named entity recognition, generation of ROS2 commands, navigation planning, and execution of the robotic response. As a result, the provided figures reflect the realistic response latency in an end-user interaction with the robotic system. Despite the fact that separate components of the proposed architecture have been extensively used before in various natural language processing tasks, this study provides an example of how their proper combination may solve specific issues of multilingual human–robot interaction. In other words, the use of parameter-efficient soft prompt tuning, task-specific prompt initialization, bidirectional intent–entity interaction, and CRF-based sequence decoding makes it possible to achieve high-quality multilingual command understanding while being computationally efficient enough for robotic systems. That is why the main contribution of this study is better described as methodological development on the system level.

Table 13. Comparison of navigation performance with existing method.

Performance Metric	Existing Keyword-Based Voice Control	Proposed SoftPromptIndicBERTCRF
Navigation Command Success Rate (%)	90.8	98.7
Average Command Interpretation Time (s)	0.42	0.29
Average Navigation Execution Time (s) *	12.8	11.9
End-to-End Response Time (Speech to Robot Motion) (s)	1.14	0.92

* Navigation execution time includes robot movement to the target and therefore depends on travel distance.

4.6. Experimental Protocol

The multilingual data set was manually annotated and then split into training (70%), validation (15%), and testing (15%) sets with the use of stratified sampling to maintain the balance of intent classes in all splits. To avoid information leaks, original utterances were split first, without any augmentation. All types of augmentations, such as synonym substitution, character-based perturbation, and back-translation, were done only on the training set. Therefore, the validation and test sets have neither augmented nor semantically similar versions generated from the training set. Duplicate utterances were also removed before splitting the data set.

The evaluation process was done using a stratified 70:15:15 train/validation/test split ratio in order to maintain the balance across all the classes within all sets. The results presented represent the average performance obtained from running the training process three times with different seeds in order to limit the effect of random initialization. This type of evaluation strategy is widely used in the field of multilingual NLU evaluation. However, in light of the relatively small size of the Malayalam dataset, it is expected that the results may suffer from high variability among the different categories.

4.7. X Asr–NLU Integration Considerations

While this research is mainly centered on multi-language natural language understanding, the experiments conducted prove that speech recognition accuracy plays a significant role in determining the success of the task. At present, the system uses a cascaded ASR-NLU architecture where speech recognition errors directly affect the language understanding component of the system. In the future, the researchers will study how to integrate the two processes more closely using techniques such as confidence-aware decoding, ASR error correction, domain-specific multilingual speech recognition, and end-to-end spoken language understanding systems. This could help enhance robustness in noisy settings, especially for morphologically complex languages like Malayalam.

Figure 14 illustrates the cascaded speech understanding pipeline adopted in this work. Errors introduced by the ASR component under noisy acoustic conditions propagate to the downstream NLU module, highlighting the importance of future research on joint ASR–NLU optimization for multilingual human–robot interaction systems.

4.8. 5.x Comparison with Cloud-Based Large Language Model Systems

Recent cloud-based large language models (LLMs), including GPT, Llama, Gemma, and similar architectures, have demonstrated excellent capabilities in natural language understanding, task planning, and robot control through WiFi, 5G, or emerging 6G communication infrastructures. These systems benefit from massive model capacity and broad linguistic knowledge, often achieving superior performance on complex open-domain reasoning tasks. However, their deployment typically relies on continuous network connectivity, cloud computing resources, and external inference services, which may introduce communication latency, increase operational costs, and raise privacy concerns when processing user speech and commands.

In contrast, the objective of the proposed work is to develop a lightweight multilingual natural language understanding framework that can be deployed directly on resource-constrained robotic platforms without requiring cloud-based inference. By combining parameter-efficient soft prompt tuning with a frozen IndicBERT backbone, bidirectional cross-attention, and CRF decoding, the proposed framework substantially reduces the number of trainable parameters while maintaining competitive performance for domain-specific robotic command understanding, which is explained in Table 14. Consequently, the proposed architecture is particularly suitable for embedded service robots operating in environments with limited computational resources, intermittent network connectivity, or strict data privacy requirements.

Table 14. Comparison between cloud-based LLM solutions and the proposed edge-based multilingual NLU framework.

Feature	Cloud-Based LLM HRI	Proposed SoftPromptIndicBERTCRF
Computing platform	Cloud GPU servers	Edge/embedded processor
Internet connectivity	Required	Not required after deployment
Inference latency	Network dependent	Local real-time inference
User data privacy	Speech transmitted to cloud	Speech processed locally after ASR
Computational requirements	Very high	Low
Trainable parameters	Hundreds of millions to billions	Approximately 5.34 million
Domain adaptation	General-purpose	Optimized for robotic commands
Deployment target	Cloud-connected robots	Offline service robots

The joint task of classifying intents and performing named entity recognition in multilingual human–robot interaction is not an easy task especially when the morphology is richer, as in the case of Malayalam, where semantic analysis requires precise classification of intents and named entities. The experimental results show that the proposed cross-attention mechanism makes significant contributions because of its ability to model the interaction between the two tasks mentioned above. It is evident that multilingual robotic language understanding is far from being a trivial task, even with a domain-specific command vocabulary.

Recent advances in multilingual large language models are expected to further improve natural language understanding for low-resource languages. Nevertheless, embedded robotic systems continue to impose strict constraints on computational resources, energy consumption, latency, network availability, and data privacy. Consequently, parameter-efficient architectures such as the proposed SoftPromptIndicBERTCRF framework remain highly relevant for edge deployment, where reliable offline operation and predictable real-time performance are essential. Rather than competing directly with increasingly larger language models, the proposed approach complements them by providing an efficient deployment strategy for practical robotic systems.

The proposed architecture shows how successful human–robot interaction using multiple languages can be achieved without the need for highly resource-consuming language models and cloud-based inference. The use of multilingual datasets, parameter-efficient learning, joint intent–entity modeling, and robotic evaluation in real time can serve as a good benchmark architecture for future HRI architectures. This approach may be applicable to other edge AI use cases as well.

The high performance achieved by the proposed framework should be interpreted in the context of the application domain. Unlike open-domain conversational language understanding, the target application focuses on structured human–robot interaction commands with a well-defined semantic space comprising navigation, manipulation, and service-related instructions. Although the vocabulary is domain-specific, the dataset includes lexical paraphrases, multilingual expressions, and compound commands requiring simultaneous intent classification and named entity recognition. Consequently, the task extends beyond keyword matching and requires accurate semantic interpretation across three languages. The strong performance therefore reflects effective adaptation to a constrained robotic command domain rather than indicating that multilingual natural language understanding in general has been solved.

To achieve an impartial evaluation, the manually annotated utterances were initially randomly stratified in the ratios of training:validation:test = 70:15:15. Duplicate utterances were eliminated prior to the stratification process. In addition, all types of data augmentation were performed on the training set only, namely, synonym substitution, character perturbation, and back translation. Therefore, none of the augmented or equivalent examples produced from the training dataset found their way into the validation or test datasets.

Although the proposed framework demonstrates consistently high performance under the adopted evaluation protocol, the Malayalam corpus remains comparatively small, with approximately 94 test utterances distributed across 13 intent categories. Consequently, certain intent classes contain relatively few evaluation samples, and class-level performance estimates should be interpreted with appropriate caution. Future evaluations using larger datasets, repeated stratified resampling, or cross-validation will provide more comprehensive estimates of model generalization and statistical robustness.

5. Discussion

Table 15 distinguishes between the established techniques incorporated into the proposed framework and the original contributions of this study. While the individual learning components have been previously investigated, this work contributes their systematic integration into a parameter-efficient multilingual HRI framework, together with a new multilingual dataset and comprehensive validation in both simulated and real robotic environments.

Table 15. Distinction between established components and contributions of the proposed framework.

Component	Existing in the Literature	Contribution in This Work
Soft Prompt Tuning	✓	Adapted for multilingual HRI with task-specific prompt initialization
Frozen Transformer Adaptation	✓	Applied for parameter-efficient multilingual robotic command understanding
Bidirectional Cross-Attention	✓	Integrated for joint intent classification and entity recognition in multilingual HRI
CRF Decoder	✓	Combined with cross-attention for multilingual sequence labeling
Multilingual Dataset	✗	New English–Hindi–Malayalam HRI dataset with intent and entity annotations
End-to-End HRI Framework	✗	Complete integration with ROS2 simulation and physical robot validation
Edge Deployment Strategy	Partially	Lightweight deployment on resource-constrained robotic platforms

The proposed framework works effectively for multilingual cases; however, comprehensive evaluation of the generalization performance of the approach for human–robot interaction in the wild is required with larger amounts of data, a larger number of speakers, different language expressions, and domain generalization. Even though the presented approach uses parameter-efficient IndicBERT in this work, it is interesting to explore how model capacity and parameter efficiency balance out if we use this prompt tuning approach with a larger multilingual encoder like MuRIL. This is left for future work. All values presented are computed by averaging the results of three experiments performed with different random seeds. We will use different seed values in future work to conduct a statistically significant evaluation of the model’s performance.

Such a stringent separation of the data splitting and data augmentation processes reduces the chance of any kind of data leakage and increases the credibility of the results for multilingual NLU performance. Even though the proposed multilingual NLU model works well for command understanding in robotics, the speech-based human–robot system still depends on the efficiency of its ASR module. Future work would need to focus on better collaboration between the ASR and NLU modules, customizing the ASR for Indian languages, and NLU under various conditions, including some level of confidence scoring. Our framework does not attempt to rival cloud-based LLMs in their ability for open-domain reasoning; however, it attempts to achieve efficient multilingual command understanding for human–robot interactions at the edge.

There are amazing advances happening in LLMs for multilingual models, but there is also the need for robot intelligence that can be executed with minimal computation and low latency for on-vehicle computing. The presented SoftPromptIndicBERTCRF framework offers an approach to do just that, conducting joint multilingual intent classification and named entity recognition via parameter-efficient adaptation on an embedded robot. Since more multilingual models with wider language coverage will be released in the future, the methodology presented in the paper is not just applicable to other models, but the overall paradigm will also be carried into the future. In summary, the paper proposes an efficient and parameter-efficient multilingual human–robot interaction system that employs task-specific prompt tuning, bidirectional cross-attention, and joint intent and entity classification for computationally restricted edge robotics platforms.

The suggested system could perform the command recognition task successfully with respect to the planned human–robot communication. However, although the experiment shown provides outstanding performance, additional large-scale experiments should be carried out with diverse and varied data in order to improve its generalizability toward new, undiscovered commands, diverse users, and varied expressions. The experiments shown prove that the proposed approach can perform the command recognition task efficiently and effectively within this dataset; nevertheless, more validation tests should be conducted with data from more challenging and diverse situations. Increasing the amount of data for low-resource languages such as Malayalam, diverse users, and a broader range of expressions, along with adding a cross-validation test will surely help to increase the generality and the statistical significance of the developed approach.

Hardware evaluation showed that the end-to-end performance of the envisioned speech-based human–robot interaction system is a product of the interaction of the automatic speech recognition (ASR) and natural language understanding (NLU) components. In the current research, the ASR module was abstracted away by utilizing Google Cloud Speech-to-Text and considered a standalone component enabling an evaluation of the multilingual NLU architecture, without concerning the development of a speech recognition model. For this research, errors observed under adverse noise conditions were attributable to transcription errors made by the ASR system and not to incorrect interpretations made by our multilingual NLU. These results indicate that while our proposed multilingual NLU system performs well on correct transcriptions of the given command, improvements in the end-to-end performance necessitate better ASR and NLU integration. In spite of the cascaded ASR-NLU system in our current study, tightly integrated future multilingual human–robot interaction systems could harness a variety of methods for tightly coupled speech and semantics integration beyond an initial fixed ASR component. Instead of fixing the ASR stage, confidence-aware ASR-NLU integration whereby transcript confidence values are fed to downstream intent classification and named entity recognition could be explored in future work. Future research could explore domain-adapted multilingual ASR models, error correction on the ASR output before submitting it to the NLU pipeline, or end-to-end spoken language understanding models that jointly optimize the the acoustic and semantic modeling. Such methods are anticipated to enhance performance in adverse conditions while retaining computational efficiency necessary for use in a robotic platform.

The experiments using the robots demonstrate the practical viability of integrating our proposed multilingual natural language understanding framework in a service robot running in a real indoor environment. We explicitly evaluated our approach in a limited application domain which includes predefined commands for robot control, a known set of navigation experiments, and a predefined list of objects relevant to the performed tasks. Hence, although the experiments prove efficient in this specific scenario, the obtained performance is not sufficient to claim that the approach generalizes to the full extent of human–robot interaction in the open real-world. Our system satisfies the requirements for interaction latencies in the given service-robot scenario in terms of its end-to-end latency. Currently, the response time obtained is simply end-to-end and component-level profiling would clearly indicate where most of the computations are performed. This is also important in more interactive service-robot applications and for more complex tasks when performance would be improved if some parts of the processing are further optimized.

While the current ablation analysis has shown the impact of the major architectural components, further investigation with different prompt initialisation methods, cross-attention variations, CRF alternatives and task weighting methods would clarify the relative significance of the different choices. Architecture analysis like this is still a good avenue of research.

The current experiment evaluations have a few limitations. For instance, the comparative study in the following sections only involves classical joint intent classification and slot-filling architecture. However, while these works serve as strong baselines in joint semantic understanding evaluation, a comparative study that involves new multilingual pretrained language models and parameter-efficient adaptation schemes would be an important part of future work. While averaging over different seeds help control random fluctuations, the omission of proper statistical significance tests impacts the conclusions of comparisons. Hence, in the future, we will consider the estimation of confidence intervals, paired statistical hypothesis testing, and bootstrap-based analyses to show the statistical soundness of differences in performance between models. Through such comparisons, we will give better insight into the relationship between language understanding ability, parameter cost, computational cost and suitability for real-world embedded human–robot interaction. One limitation of the present performance evaluation is the fact that the whole system speech-to-action pipeline was timed rather than each individual module. We will perform detailed profiling of the ASR subsystem, multilingual NLU inference, ROS2 communication and planning, and robotic actuation to identify their computational contribution and focus the optimization on areas with the largest computational cost.

In Table 16, we illustrate a conceptual comparison of our proposed approach with respect to typical multilingual NLU systems. In contrast to only aiming for state-of-the-art language understanding performance in multiple languages, we consider our architecture in terms of the balance between multilingual understanding capabilities and the needs of real-time deployment with few parameters required in an embedded human–robot interaction setup.

Table 16. Positioning of the proposed framework with respect to recent multilingual NLU approaches.

Approach	Architecture	Parameter Efficiency	Edge Deployment	Scope
Classical Joint Models (Joint-Seq, Slot-Gated, SF-ID)	RNN/CNN-based	Moderate	High	Joint intent and slot modeling
Recent Multilingual Transformers	Large multilingual transformers	Low–Moderate	Moderate	General multilingual NLU
Parameter-Efficient Adaptation (Prompt Tuning/LoRA)	Frozen pretrained transformer + lightweight adaptation	High	High	Efficient multilingual adaptation
Proposed SoftPromptIndicBERTCRF	Frozen IndicBERT + Soft Prompts + Cross-Attention + CRF	Very High	Very High	Multilingual HRI-specific joint intent and entity recognition

The impact of this work is to present a system-level methodological advance in the HRI field, not an entirely new deep learning model. This work leverages and meticulously integrates a combination of existing approaches such as parameter-efficient soft prompt tuning, frozen transformer adaptation, bidirectional cross-attention, and CRF decoding in a unified manner. This allows for efficient joint classification of intents and named entity recognition while remaining amenable to deployment on computationally constrained embedded robotic platforms. Our experiments show that this particular combination of complementary parts offers a viable approach to address this issue for multilingual HRI. Over three independent experiments, the proposed method resulted in higher average performance compared to the baseline methods across the entire test set. However, we did not report the results of formal significance testing to account for our gains; our observations must therefore be treated cautiously.

However, large-scale real-world deployment is still confronted with some key challenges. This evaluation does not study in-depth generalization to previously unseen environments, nor does it include novel object categories outside the training scope, informal conversations that are more open than defined commands, or mixed-language utterances common in natural interactions. These challenges call for larger and richer multilingual datasets, mechanisms for continuous adaptation, an open-vocabulary semantic space, and more realistic heterogeneous robotic environments and heterogeneous users.

6. Conclusions

We propose a parameter-efficient multilingual HRI framework with soft prompt tuning, bidirectional cross-attention, and CRF decoding within a joint intent classification and named entity recognition model. All of the individual components have already appeared elsewhere, but this combination applied to multilingual robotic command understanding, along with a novel multilingual HRI dataset and realistic robotic experiments makes an innovative and applicable contribution toward deploying a human–robot interaction system. This work presented SoftPromptIndicBERTCRF, a parameter-efficient NLU model for joint intent classification and NER for human–robot interaction by using frozen IndicBERT with task vocabulary-based initialization for the soft prompt, bidirectional cross-attention, and CRF as the decoder. Under evaluation using three different random seeds for three separate human–robot interaction datasets in English, Hindi, and Malayalam, the proposed model attains an average intent classification accuracy of 99.35% ± 0.002 and an NER F1 score of 98.77% ± 0.002, fine-tuning 13.78% of parameters to match the full fine-tuning performance while using less training time and memory resources. Additionally, the bidirectional cross-attention module provides independent improvements of 1.2% intent F1 and 1.8% NER F1 compared to independent-head baselines. An end-to-end evaluation on the physical ROS2 service robot in English, Hindi, and Malayalam showed a 94.44% successful completion ratio of tasks in the environment with various levels of noise, where hardware-based experiments demonstrated that performance degradation under high noise originates in ASR transcription rather than NLP inference.

However, some potential limitations are worth mentioning in light of these results obtained by the multilingual multimodal models in all three languages. The multilingual corpus used only consists of text-based and enhanced sentences and lacks actual audio commands, creating the acoustic domain gap that results in ASR-related performance deterioration when there is high ambient noise, as evidenced in the hardware validation stage. Furthermore, due to the limited vocabulary range and sentence structures of the present dataset, generalization to natural code-mixed or conversational patterns in deployed settings can be problematic. Moreover, the Malayalam test corpus consists of only 94 utterances distributed among 13 intent types, affecting the accuracy of class-level performance estimation, especially for less frequent multi-intent categories. While the proposed framework demonstrates effective multilingual natural language understanding for robotic command interpretation, the hardware experiments indicate that the overall end-to-end performance is ultimately influenced by the quality of the upstream speech recognition system. Consequently, future work will focus on integrated ASR–NLU optimization through confidence-aware semantic decoding, multilingual speech adaptation, and end-to-end spoken language understanding to further enhance the robustness and reliability of practical human–robot interaction systems. These include the extension of the multilingual HRI corpus to include real-world spoken sentences, including natural code-mixed variations in both Hindi-English and Malayalam colloquial dialects, closing the acoustic domain gap, noise-resistant ASR adaptation with respect to Malayalam agglutinative morphology as revealed by hardware testing, and developing the architecture further toward dynamically discovering intents and filling slots in an open vocabulary environment.

In addition to the listed future works, the outputs of the proposed NLU pipeline (intent and entities) may serve as a basis for developing interaction and compliance for physically manipulating objects. Some of the types of entities that are being extracted by the presented system (SIZE, DISTANCE, QUANTITY) can be used for mapping to impedance-controller stiffness and damping, where qualifiers contained in human language commands (“place it gently,” size-dependent instructions, etc.) will influence compliant contact of the robot while performing manipulation, instead of using a predefined trajectory in MoveIt as done in the current implementation; combining the presented multilingual NLU front-end with a learning-based impedance controller for human-assisted manipulation in an unstructured environment is seen as one of the promising extensions for improving interactive performance. Furthermore, because hardware validation revealed that degradation under noise comes mostly from ASR transcriptions (and not from NLU inference, as stated in Section 4.5), joint optimization of ASR and NLU stages (instead of treating ASR as a fixed input stage) is seen as one of the promising future works.

In the future, we intend to study novel multilingual foundation models along with efficient adaptation techniques like prompt tuning, low-rank adaptation, and model compression to enable advanced multilingual HRI that also offers reasonable computational resource requirements, to be executable on embedded robotic platforms. Our architecture is also generic so as to allow future, even lighter, multilingual models to be integrated as potentially superior pretrained backbones, while benefiting from localized computation, lower memory usage, and real-time interactivity. Future work will consist of thorough comparative evaluations of our approach with more recently published multilingual pretrained language models and parameter-efficient adaptation methods to further assess the generalizability and the merits of our framework under more challenging contemporary multilingual NLU benchmarks. The architecture as proposed in the paper performed consistently better than baselines across multiple experimental runs. Future work will aim at validating this through detailed statistical analysis and tests such as confidence interval determination and hypothesis testing to further establish the validity of these performance improvements. The present architecture performs in a practical real-time setting when serving multilingual service robot interaction for the scenario studied. We propose to complement our findings with a detailed study of latencies per component and an optimization of individual processing modules to decrease latency even further to allow for more computationally demanding robot interaction settings.

Code Availability: The code that we have employed for our research project is currently under development within an ongoing research work; therefore, we are not in a position to publicly distribute the source code that consists of the research software that, in its turn, is connected with a human–robot interaction framework as well as with a lower level robotics middleware that is under development. The source code can be distributed by the corresponding author upon an adequate request for educational and non-commercial uses only.