Reasoning-Enhanced Query–Service Matching: A Large Language Model Approach with Adaptive Scoring and Diversity Optimization

Abstract

Query–service matching in customer service systems faces a critical challenge of accurately aligning user queries expressed in colloquial language with formally defined services while balancing business objectives. Traditional keyword-based and embedding approaches fail to capture complex semantic nuances and cannot provide interpretable explanations. We address this problem by proposing a novel reasoning-enhanced framework that leverages large language models (LLMs) for structured multi-criteria evaluation. Our key innovation is a reasoning-first scoring architecture where the model generates detailed explanations before numerical scores, reducing score variance by 18% through conditional mutual information. We introduce a controlled stochastic perturbation mechanism with theoretically derived optimal parameters that balance diversity and relevance, alongside a knowledge distillation pipeline enabling 960× model compression (480B→0.5B parameters) while retaining 94% performance. Rigorous theoretical analysis establishes Pareto optimality guarantees for multi-criteria evaluation, information-theoretic entropy reduction bounds, and PAC learning guarantees for distillation. Experimental validation on real-world telecommunications data demonstrates 89% Precision@1 (15.3% improvement over baselines), 23% diversity enhancement, and 96× latency reduction, with deployment cost decreasing 1200× compared to direct LLM inference. This work bridges the gap between LLM capabilities and production deployment requirements through principled mathematical foundations and practical system design.

1. Introduction

Customer service systems process millions of queries daily, requiring accurate matching between user intentions and available services. This matching process critically impacts user satisfaction, operational efficiency, and business outcomes. The challenge lies not merely in semantic similarity but in understanding the complex interplay among user needs, business objectives, and service capabilities.

Recent advances in large language models (LLMs) have demonstrated remarkable capabilities in understanding and reasoning about natural language. However, effectively leveraging these capabilities for production systems presents unique challenges. First, the computational requirements of large models make direct deployment prohibitive for real-time applications. Second, ensuring consistent and interpretable scoring requires careful prompt design and output structuring. Third, balancing relevance with diversity in recommendations remains an open problem in practical systems.

The telecommunications industry exemplifies the complexity of query–service matching. Users express needs using colloquial language (“cancel my account”), while services are defined in business terminology (“service termination procedure”). This semantic gap, combined with the need to consider business objectives and user satisfaction simultaneously, creates a multi-faceted optimization problem.

Traditional approaches face several limitations. Keyword-based methods lack semantic understanding, missing queries with different surface forms but identical intents. Embedding-based approaches, while capturing semantic similarity, fail to incorporate business logic and cannot provide interpretable explanations for their rankings. Fine-tuned BERT models require extensive domain-specific training data and struggle with reasoning about business constraints.

Our key insight is that LLMs can bridge this gap by performing structured reasoning about query–service matches. By carefully designing prompts that encode evaluation criteria and business logic, we can leverage the model’s reasoning capabilities while maintaining interpretability and control over the matching process.

We present a comprehensive framework for LLM-based query–service matching with three key innovations:

Reasoning-First Scoring Architecture: We restructure the traditional scoring approach by requiring models to generate explanatory reasoning before numerical scores. This leverages the autoregressive nature of language models, where later tokens benefit from the context established by earlier ones. Our experiments show that this ordering improves scoring consistency by 18% compared to score-first approaches.

Multi-Criteria Evaluation Framework: We decompose the matching problem into five weighted criteria: semantic relevance (10%), practical utility (30%), attractiveness (30%), precision matching (10%), and business understanding (20%). Each criterion addresses specific aspects of the matching problem, from user intent alignment to business value optimization.

Knowledge Distillation Pipeline: To enable production deployment, we develop a systematic approach for distilling the reasoning capabilities of large models into efficient student models. Our pipeline generates high-quality training data using 480B parameter models and trains specialized 0.5B parameter models that retain 94% of the performance while reducing inference costs by three orders of magnitude.

This paper makes the following contributions:

1. Theoretical Framework: We formalize query–service matching as a multi-objective optimization problem and prove that our weighted evaluation scheme provides Pareto-optimal solutions under reasonable assumptions about criterion independence.

2. Algorithmic Innovations: We introduce the reasoning-first prompt architecture and adaptive perturbation algorithm, providing theoretical analysis of their convergence properties and empirical validation of their effectiveness.

3. System Design: We present a complete system architecture for production deployment, including parallel evaluation pipelines, caching strategies, and monitoring frameworks that ensure reliable performance at scale.

4. Empirical Validation: Through extensive experiments on real-world telecommunications data, we demonstrate significant improvements over state-of-the-art baselines across multiple metrics, including 89% Precision@1 and 4.3/5.0 user satisfaction scores.

5. Reproducibility: We provide detailed implementation guidelines, hyperparameter configurations, and ablation studies that enable reproduction and extension of our work.

The remainder of this paper is organized as follows. Section 2 reviews related work on query understanding, large language models, information retrieval, diversity optimization, knowledge distillation, and prompt engineering. Section 3 presents our methodology, including the problem formulation, multi-criteria evaluation framework, reasoning-first architecture, adaptive perturbation mechanism, knowledge distillation pipeline, and system architecture. Section 4 describes the experimental setup and presents comprehensive results and analysis. Appendix A provides the theoretical analysis with formal proofs. Section 5 offers discussion on insights, limitations, and future directions. Finally, Section 6 concludes the paper.

2. Related Works

2.1. Query Understanding and Intent Recognition

Query understanding has evolved from simple pattern matching to sophisticated neural approaches that capture semantic and contextual nuances. Early foundational work by Salton and Buckley [1] established TF-IDF as a cornerstone technique for term weighting, while Robertson and Zaragoza [2] formalized the probabilistic relevance framework with BM25, which remains widely used in production systems today.

The introduction of distributed word representations marked a paradigm shift in natural language understanding. Mikolov et al. [3] demonstrated that word embeddings could capture semantic relationships through vector arithmetic, enabling similarity computation beyond exact lexical matches. This was further advanced by Pennington et al. [4] with GloVe embeddings, which incorporated global corpus statistics for improved representations.

BERT and its variants have significantly improved intent recognition across various domains. Liu et al. [5] demonstrated through RoBERTa that careful hyperparameter tuning and training procedures could substantially improve performance. Clark et al. [6] introduced ELECTRA, showing that discriminative pre-training could be more efficient than traditional masked language modeling. Sanh et al. [7] explored knowledge distillation for BERT, creating smaller models while preserving much of the original performance.

Domain-specific applications have shown remarkable success with fine-tuned models. Kenton and Toutanova [8] provided comprehensive guidelines for BERT fine-tuning across various NLP tasks. However, these approaches require substantial labeled data and struggle with reasoning about complex business constraints, motivating the exploration of few-shot and zero-shot approaches.

2.2. Large Language Models for Evaluation

The emergence of large-scale language models has fundamentally changed the landscape of natural language evaluation. The GPT series, beginning with Radford et al. [9] and culminating in GPT-3 [10], demonstrated unprecedented few-shot learning capabilities. Brown et al. showed that GPT-3 could perform various evaluation tasks through in-context learning without explicit fine-tuning.

Prompt engineering has become a critical research area for leveraging LLMs effectively. Shin et al. [11] explored automated prompt generation methods. Schick and Schütze [12] demonstrated that carefully crafted prompts could elicit strong performance from smaller models.

Chain-of-thought prompting, introduced by Wei et al. [13], showed that explicit reasoning steps significantly improve performance on complex tasks. This was extended by Wang et al. [14] with self-consistency decoding, which aggregates multiple reasoning paths for more robust predictions. Kojima et al. [15] demonstrated that even simple “Let’s think step by step” prompts could elicit reasoning in large models.

More sophisticated reasoning techniques have emerged, including least-to-most prompting by Zhou et al. [16], which decomposes complex problems into simpler sub-problems. Yao et al. [17] introduced ReAct, combining reasoning and acting for more dynamic problem-solving approaches.

Recent work has specifically explored LLMs as evaluators for various tasks. Chiang and Lee [18] investigated whether large models could serve as reliable judges for generation quality. Liu et al. [19] introduced G-Eval, showing that GPT-4 could provide more consistent and nuanced evaluations than traditional metrics. However, Zheng et al. [20] highlighted potential biases and limitations in LLM-based evaluation.

Our work extends this evaluation paradigm by introducing a structured multi-criteria assessment specifically designed for query–service matching, incorporating business logic that traditional similarity metrics cannot capture.

2.3. Information Retrieval and Matching Systems

Classical information retrieval has provided foundational techniques that remain relevant today. Rocchio [21] introduced relevance feedback mechanisms.

Dense retrieval methods have gained prominence with the success of neural models. Karpukhin et al. [22] demonstrated that dense passage retrieval could outperform traditional sparse methods for open-domain question answering. Xiong et al. [23] explored approximate nearest neighbor search for efficient dense retrieval at scale.

Hybrid approaches combining sparse and dense retrieval have shown promising results. Luan et al. [24] demonstrated that combining BM25 with dense retrievers could achieve better performance than either method alone. Ma et al. [25] explored zero-shot dense retrieval, showing that pre-trained models could generalize to new domains without additional training.

Recent work has focused on improving retrieval through better training procedures. Karpukhin et al. [22] introduced hard negative mining for dense retrieval training, while Qu et al. [26] demonstrated the importance of data augmentation for robust retrieval models.

Cross-lingual and multilingual retrieval has become increasingly important. Conneau et al. [27] introduced cross-lingual language models, while Zhang et al. [28] explored multilingual dense retrieval for diverse language pairs.

2.4. Diversity in Recommendation Systems

Balancing relevance and diversity represents a fundamental challenge in recommendation and ranking systems. Early work by Carbonell and Goldstein [29] introduced Maximal Marginal Relevance (MMR), which greedily selects documents that are both relevant to the query and dissimilar to already selected documents.

Subsequent research has explored various diversity metrics and optimization techniques. Zhai et al. [30] studied the risk–return tradeoff in information retrieval, showing that diversity could improve user satisfaction even at the cost of some relevance. Chen and Karger [31] explored the theoretical foundations of diversity, proving certain optimality guarantees for MMR-style algorithms.

Modern approaches have incorporated more sophisticated diversity measures. Radlinski et al. [32] introduced learning-to-rank methods that explicitly optimize for diversity. Agrawal et al. [33] formalized the diversity problem as a bi-criteria optimization task, providing a theoretical analysis of the relevance–diversity tradeoff.

Kunaver and Požrl [34] provided a comprehensive survey of diversity techniques in recommender systems, categorizing approaches into content-based, collaborative, and hybrid methods. Antikacioglu and Ravi [35] explored post-processing methods for improving diversity without retraining underlying models.

Recent work has explored diversity in neural ranking models. Wilhelm et al. [36] demonstrated techniques for incorporating diversity constraints into neural rankers.

Our perturbation-based approach draws inspiration from randomized algorithms in optimization, particularly the work of Motwani and Raghavan [37] on randomized algorithms. By introducing controlled randomness, we achieve diversity while maintaining relevance guarantees, a property we formally analyze in Section 4.

2.5. Knowledge Distillation and Model Compression

Knowledge distillation, pioneered by Hinton et al. [38], enables the transfer of knowledge from large teacher models to smaller student models. The core insight is that the soft predictions of teacher models contain more information than hard labels, allowing students to learn from the teacher’s uncertainty and confidence patterns.

Early applications focused on computer vision tasks, but the technique has been successfully adapted to natural language processing. Tang et al. [39] explored task-specific distillation for BERT, while Sun et al. [40] introduced patient knowledge distillation with additional intermediate supervision.

Sanh et al. [7] demonstrated that BERT’s knowledge could be effectively distilled into models half the size while retaining 97% of performance. This work showed the practical potential of distillation for deploying large models in resource-constrained environments. Jiao et al. [41] further pushed the compression boundaries with TinyBERT, achieving even more aggressive size reductions.

Recent work has explored distillation for large language models specifically. Wang et al. [42] introduced MiniLM, which distills knowledge from the self-attention distributions of teacher models. Gu et al. [43,44] explored distillation techniques for instruction-following capabilities in large models.

Multi-teacher distillation has emerged as a way to combine knowledge from multiple models. You et al. [45] demonstrated that students could benefit from diverse teacher models with different architectures or training procedures. Mirzadeh et al. [46] introduced teacher assistant models to bridge large capacity gaps between teachers and students.

Task-specific distillation has shown promise for specialized applications. Mukherjee and Awadallah [47] explored extreme compression for task-specific models, while Turc et al. [48] studied the relationship between model capacity and distillation effectiveness.

Our contribution lies in developing a distillation pipeline specifically optimized for reasoning-based scoring tasks, where we preserve not just the final numerical scores but the intermediate reasoning processes that generate them. This differs from traditional distillation by maintaining interpretability alongside efficiency.

2.6. Evaluation Metrics and Benchmarks

Evaluation methodology in information retrieval has evolved significantly over the past few decades. Cleverdon et al. [49] established the Cranfield paradigm, which remains foundational for IR evaluation. Järvelin and Kekäläinen [50] introduced normalized discounted cumulative gain (nDCG), which has become a standard metric for ranking evaluation.

Modern benchmarks have expanded beyond traditional IR tasks. Thakur et al. [51] introduced BEIR, a heterogeneous benchmark for zero-shot evaluation of information retrieval models across diverse domains. This benchmark highlighted the challenge of generalization in retrieval systems.

User-centric evaluation has gained importance with the recognition that automatic metrics may not fully capture user satisfaction. Kelly and Teevan [52] explored implicit feedback signals, while Joachims et al. [53] demonstrated how click-through data could be used for evaluation.

Recent work has explored the evaluation of generation tasks, which pose unique challenges. Papineni et al. [54] introduced BLEU for machine translation evaluation, while Lin [55] developed ROUGE for summarization. However, these metrics have limitations when applied to open-ended generation tasks.

Liu et al. [19] introduced G-Eval, showing that large language models could provide more nuanced evaluation than traditional automatic metrics. This work demonstrated that LLMs could capture aspects of quality that traditional metrics miss, such as coherence and informativeness.

2.7. Prompt Engineering and In-Context Learning

Prompt engineering has emerged as a critical skill for effectively utilizing large language models. Early work by Radford et al. [56] demonstrated that language models could perform various tasks when provided with appropriate context and instructions.

Brown et al. [10] systematically studied in-context learning with GPT-3, showing that models could adapt to new tasks with just a few examples in the prompt. This capability eliminated the need for task-specific fine-tuning in many scenarios.

Systematic approaches to prompt design have been developed. Reynolds and McDonell [57] explored prompt programming as a new paradigm for leveraging language models.

Automated prompt optimization has become an active research area. Shin et al. [11] introduced gradient-based methods for automatic prompt generation. Li and Liang [58] explored prefix-tuning as an alternative to full fine-tuning, while Lester et al. [59] investigated the effectiveness of soft prompts.

Chain-of-thought prompting represents a significant advance in eliciting reasoning from language models. Wei et al. [13] showed that explicit reasoning steps dramatically improve performance on complex reasoning tasks. This work inspired numerous follow-up studies exploring different reasoning paradigms.

Our reasoning-first prompt architecture builds on these foundations while introducing task-specific innovations for evaluation scenarios, where the order of reasoning and scoring significantly impacts consistency and interpretability.

2.8. Recent Advances in LLM-Based Reranking and Retrieval-Augmented Generation

The past two years (2023–2025) have witnessed significant advances in leveraging large language models for sophisticated ranking and retrieval-augmented generation (RAG) tasks. Recent works have demonstrated that LLMs can serve as effective rerankers, going beyond traditional sparse and dense retrieval methods. Gao et al. proposed a comprehensive LLM-based reranking framework that systematically models various aspect requirements as distinct nodes, automatically incorporating multi-faceted reranking considerations and ensuring scalability and personalization. Similarly, contemporary work has highlighted that LLMs have gained significant traction in multi-stage retrieval systems, offering strong semantic understanding and reasoning capabilities for document ranking tasks. In the RAG domain, recent research has proposed two-step retrieval and ranking frameworks that address both the challenge of retrieving relevant documents and ensuring collective diversity among retrieved passages, recognizing that passage-level relevance alone is insufficient for high-quality generation [60]. Additionally, studies emphasize that retrieval in RAG systems must ensure not only individual relevance but also collective coherence, proposing mechanisms to optimize inter-passage relationships. These approaches typically focus on improving generation quality or precision of single-stage retrieval without explicitly handling business logic, multi-criteria constraints, or the interpretability–efficiency trade-off through knowledge distillation.

Our work positions itself distinctly in this landscape by introducing a theoretically-grounded, multi-criteria evaluation framework specifically designed for query–service matching in commercial settings. While recent LLM-based reranking methods excel at capturing semantic nuances through implicit reasoning, our reasoning-first scoring architecture. explicitly generates detailed explanations before numerical scores, fundamentally enhancing both interpretability and consistency. Furthermore, our framework explicitly integrates multiple competing criteria (semantic relevance, practical utility, attractiveness, precision matching, and business understanding) using Pareto-optimal scalarization, offering a principled approach to multi-objective ranking that goes beyond typical reranking paradigms. Most critically, we address the practical deployment bottleneck through reasoning-preserving knowledge distillation, which compresses LLM capabilities into efficient student models while maintaining interpretability—an often-overlooked aspect in pure reranking or RAG literature that prioritizes generation quality over deployment feasibility. By combining theoretical rigor with practical efficiency, our framework offers a more holistic solution for enterprise-level query—service matching than existing LLM-based reranking or RAG approaches.

3. Methodology

This section presents our comprehensive framework for reasoning-enhanced query–service matching. We begin by formalizing the problem (Section 3.1) and introducing our multi-criteria evaluation approach (Section 3.2). The reasoning-first prompt architecture (Section 3.3) forms the core innovation, followed by our adaptive perturbation mechanism for diversity (Section 3.4) and knowledge distillation pipeline (Section 3.5). The system architecture for production deployment is described in Section 3.6. Figure 1 illustrates how these components integrate into an end-to-end system. Figure 1 presents a comprehensive overview of our proposed framework, illustrating the integrated architecture that combines reasoning-enhanced LLM evaluation with knowledge distillation for efficient deployment.

3.1. Problem Formulation

Let $\mathcal{Q}$ denote the space of user queries and $\mathcal{S}$ the space of available services. Given a query $q\in \mathcal{Q}$, our objective is to produce a ranking $\pi :\mathcal{S}\to \mathbb{N}$ that optimizes both relevance and diversity.

We formalize this as a multi-objective optimization problem:

$${\pi }^{*}={\mathrm{argmax}}_{\pi }\left[{\lambda }_r·R(q,\pi )+{\lambda }_d·D\left(\pi \right)-{\lambda }_c·C\left(\pi \right)\right]$$

(1)

where $R(q,\pi )$ measures relevance, $D\left(\pi \right)$ measures diversity, $C\left(\pi \right)$ represents computational cost, and ${\lambda }_r,{\lambda }_d,{\lambda }_c$ are weighting parameters.

3.2. Multi-Criteria Evaluation Framework

We decompose relevance into five criteria, each capturing different aspects of query–service alignment:

$$R(q,s)=\sum _{i=1}^5{w}_i·r_i(q,s)$$

(2)

The selection of five criteria in Equation (2) derives from systematic domain expert consultation with telecommunications specialists. Through structured interviews with 12 customer service managers and product owners, we identified these five criteria as necessary and sufficient to comprehensively capture query–service alignment without redundancy: $r_1$ (semantic relevance) measures whether the service addresses the user’s stated need; $r_2$ (practical utility) evaluates whether the user can actually access and use this service given eligibility constraints; $r_3$ (attractiveness) assesses user engagement and satisfaction potential from a marketing perspective; $r_4$ (precision matching) quantifies exactness of the match to avoid false positives; and $r_5$ (business understanding) evaluates alignment with company revenue objectives and strategic priorities. The number five specifically was chosen because preliminary experiments with fewer criteria (3 or 4) missed important business dimensions, while more criteria (6 or 7) introduced redundancy and annotation complexity without performance gains. Empirical validation through ablation studies (Section 4.7) confirms that each criterion contributes meaningfully to overall performance, with no single criterion being redundant or dominated by others.

Where $r_i$ represents individual criterion scores, and $w_i$ represents their weights, with ${\sum }_{i=1}^5{w}_i=1$.

The criteria include $r_1$ representing semantic relevance, which measures semantic alignment between query and service, $r_2$ for practical utility that evaluates the service’s ability to address the user’s need, $r_3$ for attractiveness, which assesses user engagement potential, $r_4$ for precision matching that quantifies exactness of match, and $r_5$ for business understanding, which evaluates alignment with business objectives.

3.3. Reasoning-First Prompt Architecture

Our key innovation is structuring prompts to generate reasoning before scores. This leverages the autoregressive nature of language models:

$$P\left(s\right|q,c)=P(r|q,c)·P\left(s\right|q,c,r)$$

(3)

where s is the score, r is the reasoning, q is the query, and c represents the context (criteria and instructions).

The prompt template follows this structure (Listing 1):

Listing 1. Reasoning-First Prompt Template.

3.4. Adaptive Perturbation for Diversity

To enhance diversity while preserving relevance, we introduce controlled perturbations:

$$s_i^{\prime }={\widehat{s}}_i-{ϵ}_i,{ϵ}_i\sim U(0,{ϵ}_{max})$$

(4)

where ${\widehat{s}}_i$ is the normalized score, and ${ϵ}_i$ is the random perturbation.

The perturbation parameter ${ϵ}_{max}$ controls the diversity–relevance trade-off:

$${ϵ}_{max}^{*}=\underset{ϵ}{arg\; max}\left[\alpha ·D\left(ϵ\right)-(1-\alpha )·L_R\left(ϵ\right)\right]$$

(5)

3.5. Knowledge Distillation Pipeline

We employ a teacher–student framework, where a large model (480B parameters) generates training data for a smaller model (0.5B parameters). Our approach implements response-based knowledge distillation (KD), combined with intermediate feature matching, where the student learns from both the teacher’s output distributions and internal reasoning representations. This differs from relational knowledge distillation (RKD), which focuses on inter-sample relationships, and instance knowledge distillation (IKD), which preserves individual instance characteristics [38].

$${\mathcal{L}}_{student}={\mathbb{E}}_{(q,s)\sim \mathcal{D}}\left[KL(P_{teacher}(·|q,s)\parallel P_{student}(·|q,s))\right]$$

(6)

The distillation process preserves both numerical scores and reasoning patterns through a composite loss function that combines score prediction accuracy, reasoning text similarity (using embedding-based matching), and ranking consistency preservation, enabling the student model to maintain both performance and interpretability.

3.6. System Architecture

3.6.1. Overview

Our system implements a comprehensive pipeline architecture designed to handle large-scale query–service matching with real-time performance requirements. The architecture integrates four main components that work collaboratively to process user queries, evaluate service matches through our reasoning-enhanced framework, and deliver ranked results optimized for both relevance and diversity. The system is built with production scalability in mind, incorporating advanced caching mechanisms, parallel processing capabilities, and comprehensive monitoring to ensure reliable operation under high-load conditions.

The overall design follows a microservices pattern where each component can be independently scaled and maintained. This modular approach enables flexible deployment configurations, allowing organizations to adjust computational resources based on their specific requirements and traffic patterns. The architecture supports both synchronous and asynchronous processing modes, with the ability to handle batch requests for offline analysis and real-time requests for interactive applications.

3.6.2. Model Deployment Layer

The deployment layer represents the foundation of our system, managing model serving with carefully optimized inference configurations. Model loading utilizes bfloat16 precision to achieve optimal memory efficiency while maintaining numerical stability for our scoring computations. This precision choice reduces memory footprint by approximately 50% compared to float32 while preserving the dynamic range necessary for accurate evaluation scores.

Batch processing capabilities are implemented to group multiple requests for improved throughput, particularly beneficial when handling concurrent queries from multiple users. The batching mechanism dynamically adjusts batch sizes based on current system load and available computational resources, ensuring optimal utilization of GPU memory and processing capacity. Request grouping strategies consider both temporal proximity and query complexity to minimize overall processing latency.

The caching subsystem implements a sophisticated LRU (Least Recently Used) cache for frequently accessed query–service pairs, significantly reducing redundant computations. The cache maintains both raw scores and reasoning explanations, enabling immediate response for repeated queries while preserving the interpretability features of our framework. Cache invalidation policies ensure that outdated evaluations are refreshed when service definitions or business rules change.

3.6.3. Parallel Evaluation Engine

To efficiently handle the evaluation of multiple services against a single query, our system implements a parallel evaluation engine that distributes computations across multiple processing threads. The engine initializes a ThreadPoolExecutor with a configurable number of workers, typically set based on available CPU cores and memory constraints. This parallel approach significantly reduces overall response time, which is particularly important when evaluating queries against large service catalogs.

The evaluation process begins with the preparation of prompt templates for all candidate services, utilizing our reasoning-first architecture described in the methodology section. Each worker thread generates specialized prompts by combining the user query with individual service descriptions and our multi-criteria evaluation framework. The prompt generation process incorporates contextual information such as user history, business priorities, and temporal factors to ensure comprehensive evaluation.

The response processing involves extracting both numerical scores and reasoning explanations from the language model outputs. The system implements robust parsing mechanisms to handle variations in model responses while maintaining consistency in score interpretation. Error handling procedures ensure that individual service evaluation failures do not compromise the overall ranking process, with fallback mechanisms providing reasonable default scores when necessary.

Result aggregation combines individual service evaluations into a coherent ranking while applying our adaptive perturbation algorithm for diversity enhancement. The aggregation process considers not only raw scores but also confidence measures and reasoning quality indicators to produce robust final rankings.

3.6.4. Monitoring and Quality Assurance

Our system incorporates comprehensive monitoring capabilities designed to ensure consistent performance and early detection of potential issues. Latency tracking monitors P50, P95, and P99 latencies for each system component, providing detailed insights into performance bottlenecks and enabling proactive optimization. The monitoring system maintains historical performance data to identify trends and seasonal patterns in system behavior.

Score distribution monitoring continuously analyzes the statistical properties of generated scores to detect potential model drift or systematic biases. The system tracks distribution parameters such as mean, variance, and skewness across different query categories and service types. Anomaly detection algorithms flag unusual score patterns that might indicate model degradation or configuration issues.

Reasoning quality assessment involves automated and manual review processes to ensure the interpretability and coherence of generated explanations. The system samples a percentage of reasoning outputs for detailed analysis, checking for logical consistency, factual accuracy, and alignment with scoring decisions. Quality metrics include reasoning length, coherence scores, and alignment with expected evaluation criteria.

Performance optimization features include automatic scaling mechanisms that adjust resource allocation based on current demand and predictive models that anticipate traffic spikes. The system maintains detailed logs of all evaluation decisions, enabling comprehensive auditing and facilitating continuous improvement of the evaluation framework.

4. Experiments and Results

4.1. Dataset

We conduct our evaluation on a comprehensive telecommunications service dataset that reflects the complexity and diversity of real-world customer service scenarios. The dataset comprises 10,000 user queries written in Chinese, representing authentic customer requests collected from actual service interactions during a six-month period from January to June 2024. These queries span various service categories: account management (25%), billing inquiries (20%), technical support (30%), service modifications (15%), and promotional information (10%), providing comprehensive coverage of typical customer needs.

Table 1. Dataset summary statistics.

Characteristic	Value
Total Queries	10,000
Collection Period	January–June 2024
Language	Chinese
Service Catalog Size	500 unique services
Annotated Query–Service Pairs	5000
Annotators	15 domain experts
Inter-Annotator Agreement (Cohen’s $\kappa$)	0.78
Query Distribution by Category
Account Management	25%
Billing Inquiries	20%
Technical Support	30%
Service Modifications	15%
Promotional Information	10%
Complexity Characteristics
Ambiguous Queries	32%
Synonym Variations	28%
Terminology Mismatch	40%

provides a structured summary of the dataset characteristics, and

Table 2. Representative dataset examples.

Query (Translated)	Matched Service	Relevance (0–5)	Business Value (0–5)
“I want to cancel my account”	Account Termination Service	5.0	2.0
“Why is my bill so high?”	Billing Inquiry Service	4.5	3.5
“Can’t connect to internet”	Technical Support-Connectivity	5.0	3.0
“Upgrade my data plan”	Service Plan Upgrade	4.8	4.5
“What promotions available?”	Current Promotions Catalog	4.2	4.0

shows representative query–service pair examples.

The service catalog contains 500 unique telecommunications services, each with detailed descriptions, eligibility criteria, pricing information, and business priority rankings. Each service entry includes structured metadata such as service category, target customer segments, processing complexity, and revenue impact, enabling multi-faceted evaluation beyond simple textual similarity. Human annotators, consisting of domain experts with telecommunications industry experience, provided relevance scores on a 0–5 scale for query–service pairs, with inter-annotator agreement measured at k = 0.78, indicating substantial agreement.

Business value annotations complement the relevance scores by incorporating real-world business considerations such as profit margins, strategic importance, and customer retention impact. These annotations enable evaluation of our system’s ability to balance customer satisfaction with business objectives, a critical requirement for commercial deployment. The dataset captures real-world complexity through ambiguous queries that could match multiple services, synonym variations that require semantic understanding, and business terminology mismatches between customer language and official service descriptions.

4.2. Baseline Methods

Our experimental comparison includes four carefully selected baseline approaches that represent different paradigms in query–service matching. BM25 serves as our classical keyword-based retrieval baseline, implementing the probabilistic relevance framework with standard parameter tuning for Chinese text processing. This method represents traditional information retrieval approaches that rely primarily on term frequency and document frequency statistics without semantic understanding.

Sentence-BERT provides our dense embedding similarity baseline, utilizing pre-trained multilingual models fine-tuned for Chinese text. We employ the sentence-transformers library with the paraphrase-multilingual-MiniLM-L12-v2 model, which has demonstrated strong performance on Chinese semantic similarity tasks. This baseline represents modern neural approaches that capture semantic relationships through dense vector representations.

Our fine-tuned BERT baseline uses a domain-specific model trained on telecommunications service data, representing supervised approaches that require substantial labeled training data. We fine-tune the Chinese-BERT-wwm-ext model on our domain-specific dataset using standard classification objectives, providing a strong neural baseline that incorporates domain knowledge through supervised learning. Our fine-tuned BERT baseline uses a domain-specific model trained on telecommunications service data. Specifically, we fine-tune the Chinese-BERT-wwm-ext model (110M parameters) using mean squared error (MSE) regression loss between predicted scores and expert annotations. Training configuration: learning rate 2 × 10⁻⁵, batch size 32, maximum sequence length 512, with early stopping based on validation performance. The model converges within 5 epochs, achieving Precision@1 = 0.63 on the held-out test set. This supervised baseline represents the state-of-the-art for small-model approaches requiring substantial labeled data.

GPT-3.5 serves as our large language model baseline, implementing direct prompting without our specialized framework. This baseline uses straightforward prompts that request numerical scores for query–service pairs, representing the naive application of large language models to evaluation tasks without architectural innovations or specialized prompt design.

4.3. Evaluation Metrics

Our evaluation employs a comprehensive suite of metrics designed to capture multiple dimensions of system performance.

Relevance Metrics: We use standard information retrieval metrics including Precision@k, Recall@k, and normalized Discounted Cumulative Gain (nDCG@k) for $k\in \{1,3,5\}$. Specifically, Precision@k measures the proportion of relevant services among the top-k ranked results: $P@k={\displaystyle \frac{\left|\right\{\mathrm{relevant}\mathrm{services}\}\cap \{\mathrm{top}-\mathrm{k}\mathrm{ranked}\left\}\right|}{k}}$. For our primary metric Precision@1, we evaluate whether the top-ranked service is labeled as highly relevant (relevance score ≥ 4 on the 0–5 scale) by human annotators, representing the probability that the single topmost recommendation satisfies the user’s query. Recall@k measures the proportion of all relevant services retrieved within top-k: $R@k={\displaystyle \frac{\left|\right\{\mathrm{relevant}\mathrm{services}\}\cap \{\mathrm{top}-\mathrm{k}\mathrm{ranked}\left\}\right|}{\left|\right\{\mathrm{all}\mathrm{relevant}\mathrm{services}\left\}\right|}}$. nDCG@k accounts for ranking position with discounted gains: $\mathrm{nDCG}@\mathrm{k}={\displaystyle \frac{DCG@k}{IDCG@k}}$ where $DCG@k={\sum }_{i=1}^k{\displaystyle \frac{2^{re{l}_i}-1}{{log}_2(i+1)}}$ and $IDCG@k$ is the ideal DCG obtained by perfect ranking.

Diversity measurement employs Intra-List Diversity (ILD) calculations that quantify the average pairwise dissimilarity between services in the top-k results. We compute ILD using both semantic embeddings and categorical service classifications, ensuring comprehensive diversity assessment. Higher ILD scores indicate better diversity, reflecting our system’s ability to provide varied options that meet different aspects of user needs.

Business value assessment incorporates domain-specific metrics that reflect real-world deployment considerations. Conversion rate measures the percentage of recommended services that users actually select or purchase, while revenue per query quantifies the average business value generated by our recommendations. These metrics ensure that our system optimizations translate to tangible business outcomes rather than merely improving abstract similarity measures.

User satisfaction evaluation employs 5-point Likert scale ratings collected from human evaluators who assess the overall quality and usefulness of ranked service lists. Evaluators consider factors such as result relevance, diversity, and practical utility, providing a holistic assessment of user experience. We collect satisfaction ratings from 20 domain experts who evaluate 500 randomly sampled query–result pairs, ensuring statistically significant assessment of user-perceived quality.

4.4. Implementation Details

Our experimental implementation utilizes carefully optimized configurations designed to balance performance, efficiency, and reproducibility.

Teacher Model Architecture (Qwen-480B): The teacher model is based on the Qwen-480B transformer architecture with the following specifications: 480 billion parameters distributed across 100 transformer layers, 128 attention heads per layer, hidden dimension size of 16,384, and intermediate FFN dimension of 65,536. The model uses multi-head self-attention with rotary position embeddings (RoPE), RMSNorm for layer normalization, and SwiGLU activation functions. Pre-training was performed on a diverse multilingual corpus exceeding 10 trillion tokens with strong emphasis on Chinese language data, enabling robust semantic understanding and reasoning capabilities. For our application, we use the pre-trained checkpoint without additional fine-tuning—the model serves purely as a zero-shot evaluator through carefully designed prompts. Model serving uses bfloat16 precision to optimize memory usage while maintaining numerical stability for score computations, enabling efficient deployment with pipeline parallelism across 8 NVIDIA A100 GPUs (80 GB each).

Student Model Architecture (Qwen-0.5B): The student model uses a substantially compressed architecture with 500 million parameters: 24 transformer layers, 16 attention heads per layer, a hidden dimension of 1024, and an intermediate FFN dimension of 4096. This 960× parameter reduction (from 480B to 0.5B) is achieved while preserving the same architectural patterns (RoPE, RMSNorm, SwiGLU) as the teacher, facilitating effective knowledge transfer. The student model is initialized from the pre-trained Qwen-0.5B checkpoint and then fine-tuned using our knowledge distillation pipeline for 30 epochs on the teacher-generated dataset of 10,000 training samples. Training uses AdamW optimizer with learning rate 5 × 10⁻⁵, linear warmup over 10% of steps, cosine decay schedule, batch size 32, gradient accumulation steps 4, and maximum sequence length of 512 tokens. The total training time is approximately 8 GPU-hours on a single NVIDIA A100 GPU.

Performance Analysis: Despite the 960× size difference, the student model retains 94% of the teacher’s Precision@1 performance (0.84 vs 0.89), validating the effectiveness of our reasoning-preserving distillation approach. The performance gap manifests primarily in complex queries with semantic ambiguity (32% of dataset), where the teacher’s larger capacity enables more nuanced reasoning. For straightforward queries with clear intent, the student achieves near-parity with the teacher (P@1 difference < 3%). The student model achieves 96× faster inference (25 ms vs 2400 ms per query) and 1200× lower deployment cost ($0.001 vs $1.20 per 1 K requests), making it the practical choice for production deployment while reserving the teacher model for offline dataset generation and periodic quality audits.

Batch processing configurations use size 32 for optimal GPU utilization, with dynamic batching adjustments based on query complexity and system load. These configurations were determined through extensive hyperparameter optimization experiments.

Perturbation parameters are set to a range of 0.001, providing subtle diversity enhancement without significantly degrading relevance scores. This value was selected through systematic analysis of the diversity–relevance tradeoff across different perturbation magnitudes, ensuring optimal balance for our application domain. Criteria weights follow the distribution of 0.1, 0.3, 0.3, 0.1, 0.2 for semantic relevance, practical utility, attractiveness, precision matching, and business understanding, respectively, based on domain expert consultation and empirical validation.

All experiments use consistent random seeds for reproducibility, with statistical significance testing performed using paired t-tests across multiple random splits of the evaluation data. Computing infrastructure consists of NVIDIA A100 GPUs with 80 GB memory, enabling efficient processing of large model inference and parallel evaluation workflows.

4.5. Reproducibility and Practical Considerations

4.5.1. Teacher Model and Knowledge Distillation Strategy

We acknowledge that the direct training and deployment of a 480B-parameter model (Qwen-480B) is not practically reproducible for most research groups due to computational resource constraints. Our approach specifically addresses this limitation through knowledge distillation: the large teacher model serves exclusively as a knowledge source for generating high-quality training data, not for direct deployment in production systems. The teacher model was trained and hosted on a cluster of 8 NVIDIA A100 GPUs with 80 GB memory each, using distributed training with pipeline parallelism. This configuration was used only for the offline stage of generating reasoning explanations and scores on the 10,000 queries in our dataset, which took approximately 48 GPU-hours. The generated dataset (comprising query–service pairs, scores, and reasoning texts) is fixed and can be made available to facilitate reproducibility without requiring access to such large-scale computational resources.

The core contribution lies in the student model (Qwen-0.5B), which is designed for practical deployment and is readily accessible. This 0.5B model can be trained on commodity hardware (a single NVIDIA A100 GPU with 80 GB memory or even smaller GPUs with gradient accumulation). Our distillation training process required approximately 8 GPU-hours for 30 epochs over the 10,000 training samples (after an 80–20 train–test split).

4.5.2. Dataset Collection and Annotation Details

To strengthen the reproducibility and transparency of our experimental setup, we provide detailed dataset specifications:

Query Collection: The 10,000 user queries were collected from authentic customer service interactions spanning a six-month period (January to June 2024) from a major Chinese telecommunications provider. Queries were anonymized and de-identified to comply with privacy regulations. The queries span diverse service categories: account management 25%), billing inquiries (20%), technical support (30%), service modifications (15%), and promotional information (10%).

Service Catalog: The 500 unique services were extracted from the provider’s official service catalog and business documentation. Each service entry includes: (1) service name and description (average length 150 tokens), (2) eligibility criteria (structured constraints such as minimum contract duration, geographic availability, customer tier), (3) pricing information (subscription cost, usage-based fees), (4) business priority ranking (assigned by product managers on a scale of 1–10 based on revenue and strategic importance), and (5) categorical tags (service type, target demographic).

Label Generation Process: Human annotation was performed by 15 domain experts from the telecommunications provider, each with at least 3 years of customer service or product management experience. For each of the 5000 query–service pairs selected for annotation (a stratified sample ensuring balanced representation across service categories), annotators provided:

• Relevance Score: On a 0–5 scale, where 0 = completely irrelevant, 3 = moderately relevant, 5 = perfectly relevant. The average inter-annotator agreement (Cohen’s kappa) was 0.78, indicating substantial agreement and validating annotation quality.
• Business Value Score: On a 0–5 scale, capturing profit margins, strategic importance, and customer retention potential.
• Confidence: Annotators indicated their confidence in the annotation (low/medium/high), allowing us to weight disagreements appropriately during training.

To address potential annotation bias, we implemented multiple quality assurance mechanisms: (1) inter-annotator agreement calculations with automatic flagging of pairs with kappa < 0.60 for re-annotation, (2) expert review of 10% of annotations by senior product managers, and (3) cross-validation against historical customer service interaction logs to ensure consistency with real user behavior patterns.

Noise Characteristics and Dataset Complexity: The dataset intentionally captures real-world complexity through three dimensions of challenge:

• Ambiguous Queries (32% of dataset): Queries that could legitimately match multiple services, such as “I want to upgrade my service”, which could refer to device upgrades, data plan increases, or service tier enhancements.
• Synonym Variations (28% of dataset): Queries expressing the same intent through different terminology, such as “all meaning account termination with subtle semantic differences”.
• Terminology Mismatch (40% of dataset): Queries using colloquial or regional language that differs from formal service descriptions, such as “can’t make calls” versus formal “call functionality exception”.

4.5.3. Hardware and System Configuration

Our experimental infrastructure utilizes NVIDIA A100 GPUs with 80 GB memory, enabling the efficient processing of large model inference and parallel evaluation workflows. For the student model deployment, we also validated compatibility with smaller GPUs (NVIDIA RTX 4090 with 24 GB memory) using gradient accumulation and model quantization (bfloat16 precision), demonstrating practical accessibility for researchers with limited computational budgets. All experiments were conducted on a system with 256 GB CPU RAM and NVMe SSD storage (1 TB) for efficient batch loading and caching.

4.5.4. Deployment Latency Correction

We clarify a previous communication discrepancy: the student model achieves 25 ms latency per query (not 15 ms) on a single NVIDIA A100 GPU with batch size 32. This 25 ms latency includes end-to-end processing: prompt construction (2 ms), model inference (18 ms), output parsing (3 ms), and result aggregation (2 ms). For real-time production deployment with multiple concurrent queries, batching substantially improves throughput, achieving approximately 1280 queries per second per GPU.

4.6. Main Results

Table 3. Performance comparison across methods.

Method	P@1	P@3	nDCG@3	ILD	Satisfaction
BM25	0.42	0.38	0.39	0.48	2.1
Sentence-BERT	0.57	0.52	0.54	0.52	2.7
Fine-tuned BERT	0.63	0.59	0.61	0.55	3.0
GPT-3.5	0.75	0.71	0.73	0.61	3.8
Ours (Teacher)	0.89	0.83	0.85	0.68	4.3
Ours (Student)	0.84	0.79	0.81	0.69	4.2

presents the comprehensive evaluation results across all baseline methods and our proposed approach, demonstrating substantial improvements in multiple performance dimensions. Our teacher model achieves exceptional performance with Precision@1 of 0.89, representing a remarkable 41% improvement over the strong Fine-tuned BERT baseline and an 18% improvement over GPT-3.5’s direct prompting approach. The Precision@3 metric shows consistent superiority, with 0.83 for our teacher model compared to 0.71 for GPT-3.5, indicating that our reasoning-first architecture maintains high precision across multiple retrieved results. Figure 2 provides a comprehensive visualization of these performance metrics across all evaluation dimensions, clearly illustrating the substantial advantages of our approach over baseline methods.

As illustrated in Figure 2, our proposed approach consistently outperforms all baseline methods across four evaluation dimensions: precision, ranking quality, diversity, and user satisfaction.

In terms of retrieval precision, our Teacher model achieves the highest scores across all cut-off levels, attaining P@1 = 0.89, P@3 = 0.83, and P@5 = 0.78, representing substantial improvements over the strongest baseline GPT-3.5 (P@1 = 0.75, P@3 = 0.71, P@5 = 0.68). Notably, our Student model also delivers competitive performance with P@1 = 0.84, P@3 = 0.79, and P@5 = 0.75, surpassing GPT-3.5 by considerable margins while maintaining a much lighter computational footprint. Traditional retrieval methods such as BM25, S-BERT, and F-BERT lag significantly behind, with BM25 yielding the lowest precision scores (P@1 = 0.42, P@3 = 0.38, P@5 = 0.33).

A similar trend is observed for nDCG@3, where our Teacher model reaches 0.85 and the Student model achieves 0.81, both substantially exceeding GPT-3.5 (0.73). The performance gap between our models and the conventional baselines is even more pronounced: BM25, S-BERT, and F-BERT obtain nDCG@3 scores of only 0.39, 0.54, and 0.61, respectively, indicating that our approach not only retrieves more relevant items but also ranks them in a more desirable order.

Beyond relevance-oriented metrics, our method also excels in recommendation diversity. The Teacher model achieves an intra-list diversity score of 0.68, and the Student model reaches 0.69—both meaningfully higher than GPT-3.5 (0.61) and the remaining baselines (BM25: 0.48, S-BERT: 0.52, F-BERT: 0.55). This demonstrates that our approach successfully avoids redundancy in the recommended lists while maintaining high relevance, striking an effective balance between accuracy and diversity. It is worth noting that the Student model slightly surpasses the Teacher model on this metric, suggesting that the distillation process may introduce a mild regularization effect that benefits diversity.

Most importantly, the human evaluation results further corroborate the superiority of our approach. Our Teacher model receives an average user satisfaction rating of 4.3 out of 5, and the Student model achieves 4.2, both comfortably surpassing the “Good” threshold (4.0) and significantly outperforming GPT-3.5 (3.8), which only marginally exceeds the "Good" line. In contrast, BM25 (2.1), S-BERT (2.7), and F-BERT (3.0) all fall at or below the "Neutral" threshold (3.0), indicating that users perceive a tangible qualitative difference between our method and conventional approaches. The narrow confidence intervals observed for our models further suggest that the improvements are robust and consistent across different users and queries.

The nDCG@3 scores reveal that our approach excels at ranking quality, with the teacher model achieving 0.85 compared to traditional methods that struggle to exceed 0.61. This improvement reflects our multi-criteria evaluation framework’s ability to capture nuanced relevance relationships that single-criterion approaches miss. The diversity metrics show interesting patterns, with our student model achieving the highest Intra-List Diversity (ILD) score of 0.69, slightly outperforming the teacher model at 0.68, suggesting that the perturbation-based diversity enhancement is particularly effective in the distilled model. User satisfaction scores demonstrate practical value, where our teacher model reaches 4.3 on a 5-point scale, representing a 43% improvement over Fine-tuned BERT.

4.7. Ablation Study

Figure 3 illustrates the systematic contribution analysis of each architectural component to overall system performance. Our comprehensive ablation analysis reveals that the reasoning-first prompt ordering emerges as the most significant contributor, providing an 18% improvement in Precision@1 when compared to standard score-first prompting approaches. This substantial gain validates our hypothesis that leveraging the autoregressive nature of language models through structured reasoning leads to more consistent and accurate evaluations.

The multi-criteria evaluation framework contributes a 15% improvement over single-criterion approaches, demonstrating the value of decomposing complex relevance judgments into interpretable components. When we remove individual criteria from the evaluation framework, semantic relevance and practical utility show the largest impact, with business understanding and attractiveness contributing more modestly but still meaningfully. Perturbation-based diversity enhancement provides an 8% improvement in user satisfaction scores, with particularly strong effects on queries where multiple viable service options exist. The knowledge distillation component analysis reveals that preserving both scores and reasoning patterns during distillation maintains 94% of teacher performance while enabling dramatic efficiency gains. Ablation Study Methodology: To rigorously evaluate the contribution of each architectural component, we conduct systematic ablation experiments where individual components are removed while maintaining all other elements constant. All ablation studies evaluate the student model (Qwen-0.5B parameters) to ensure fair comparison and practical relevance to production deployment, as the student model represents our deployable solution. Each variant is trained and evaluated using identical procedures: training for 30 epochs with the same hyperparameters (learning rate 5 × 10⁻⁵, batch size 32), evaluation on the same held-out test set of 2000 query–service pairs, and averaging metrics across three random seeds to ensure statistical reliability.

The ablation variants are constructed as follows:

• w/o Reasoning: Replaces the reasoning-first prompting architecture with score-first prompting. Instead of generating explanatory reasoning before numerical scores, this variant directly prompts the model to output numerical scores without intermediate reasoning steps. The prompt template is modified to request only $\{“score”:[0.0to1.0\left]\right\}$ without the reasoning field, testing whether the reasoning generation step provides information-theoretic benefits beyond direct scoring.
• w/o Multi-Criteria: Uses a single weighted aggregate score instead of decomposing evaluation into five separate criteria. The model receives a simplified prompt requesting an overall relevance score without explicit consideration of semantic relevance, practical utility, attractiveness, precision matching, and business understanding as distinct dimensions. This tests whether explicit multi-criteria decomposition improves evaluation quality compared to holistic scoring.
• w/o Perturbation: Disables diversity enhancement by setting perturbation parameter $ϵ=0$. Rankings are produced using unperturbed scores directly from the model without any stochastic perturbation applied during result aggregation. This isolates the contribution of controlled randomness to diversity while measuring relevance degradation.
• w/o Distillation: Trains the student model (Qwen-0.5B) directly on the human-annotated query–service pairs without leveraging teacher-generated reasoning and scores. The model is fine-tuned using supervised learning with mean squared error loss between predictions and expert annotations, representing a conventional supervised baseline without knowledge transfer from the large teacher model.

Results represent averaged metrics (Precision@1, Precision@3, nDCG@3, ILD, User Satisfaction) across three random seeds (42, 123, 456), with paired t-tests confirming statistical significance (p < 0.05) for all reported performance differences. The full model with all components enabled serves as the reference baseline, and performance degradation for each ablation variant quantifies that component’s contribution to overall system performance.

4.8. Multi-Criteria Analysis

Figure 4 presents a detailed analysis of how different evaluation criteria contribute to overall performance. The weight distribution reflects domain expert consultation, with practical utility and attractiveness receiving the highest weights at 0.3 each, recognizing their critical importance in real-world service matching scenarios.

The performance contribution analysis shows that practical utility contributes 0.25 to overall performance, validating its high weight assignment. Semantic relevance, despite its lower weight of 0.1, contributes 0.08 to performance, demonstrating efficient utilization of this fundamental criterion. Business understanding with a weight of 0.2 contributes 0.18 to performance, reflecting the importance of commercial considerations in telecommunications service matching.

4.9. Reasoning Architecture Impact

Figure 5 demonstrates the substantial benefits of our reasoning-first architecture compared to traditional score-first approaches. The score distribution analysis reveals that reasoning-first prompting produces more confident and accurate predictions, with a higher mean score of 0.74 compared to 0.42 for score-first approaches.

The consistency analysis across different query types reveals dramatic improvements in prediction stability. For account cancellation queries, standard deviation decreases from 0.18 in score-first approaches to 0.08 in reasoning-first, representing a 56% reduction in variance. Similar patterns hold across all query categories, with consistency improvements ranging from 45% to 60%, demonstrating the robustness of our reasoning-enhanced approach.

4.10. Diversity–Relevance Trade-Off

Figure 6 demonstrates the carefully balanced relationship between perturbation strength and performance metrics across our evaluation dataset. The systematic analysis reveals that at very low perturbation levels below 0.0005, diversity gains are minimal with less than 10% improvement in ILD scores, while relevance remains virtually unchanged. The optimal operating point occurs at ${ϵ}_{max}=0.001$, where diversity increases by 24% while maintaining 97% of original relevance performance.

Beyond this optimal point, perturbation strengths above 0.002 begin showing diminishing returns, with diversity gains plateauing while relevance degradation accelerates. At ${ϵ}_{max}=0.005$, diversity improvement reaches only 28% while relevance drops to 89% of baseline performance, representing an unfavorable trade-off for practical applications. The analysis across different query types reveals interesting variations in optimal perturbation levels, with simple queries benefiting from lower perturbation around 0.0008 while complex queries achieve better satisfaction with slightly higher levels up to 0.0012.

4.11. Model Efficiency Analysis

Table 4. Comprehensive model performance and efficiency comparison.

Model	Size	P@1	P@3	nDCG@3	Latency	Cost
	(B)				(ms)	($/1 K)
Baseline Models
BERT-base	0.11	0.52	0.48	0.49	15	0.0005
BERT-large	0.34	0.59	0.55	0.57	28	0.0008
RoBERTa-large	0.36	0.61	0.57	0.59	32	0.0009
Fine-tuned BERT	0.34	0.63	0.59	0.61	30	0.0008
Large Language Models
GPT-3 (Davinci)	175	0.72	0.68	0.70	1800	0.60
GPT-3.5	175	0.75	0.71	0.73	1200	0.40
GPT-4	480	0.87	0.82	0.84	3500	1.50
Our Approach
Ours (Teacher)	480	0.89	0.83	0.85	2400	1.20
Ours (Student)	0.5	0.84	0.79	0.81	25	0.001
Performance-Efficiency Metrics
Best Baseline		0.75	0.71	0.73	1200	0.40
Improvement (Teacher)		+18.7%	+16.9%	+16.4%	$2.0\times$	$3.0\times$
Improvement (Student)		+12.0%	+11.3%	+11.0%	$48\times$	$400\times$

presents a comprehensive comparison of model efficiency across different scales, demonstrating the superior performance–efficiency trade-off achieved by our approach. The analysis spans models from 0.11B to 480B parameters, revealing critical insights about the scaling characteristics of query–service matching systems.

Our teacher model demonstrates the highest absolute performance across all precision metrics, achieving P@1 of 0.89 and surpassing even GPT-4’s 0.87 despite similar parameter counts. This 2.3% improvement validates our reasoning-first architecture’s effectiveness. More remarkably, our student model with only 0.5B parameters achieves 0.84 P@1, outperforming GPT-3.5 (175B parameters) by 12% while being 350× smaller and 400× more cost-effective.

The efficiency analysis reveals compelling trade-offs across the model spectrum. Traditional fine-tuned models like BERT-large achieve reasonable latency (28 ms) and cost ($0.0008/1 K) but suffer from a limited performance ceiling at 0.63 P@1. Large language models achieve better performance but impose prohibitive computational costs, with GPT-4 requiring 3.5 s per query and $1.50 per thousand requests. Our distilled student model uniquely occupies the optimal region of this trade-off space, delivering near-state-of-the-art performance (0.84 P@1) with small-model efficiency (25 ms latency, $0.001/1 K cost).

4.12. Knowledge Distillation Effectiveness

Table 5. Performance retention in distilled models.

Model	Size	P@1	Latency (ms)	Cost
Teacher (480B)	480B	0.89	2400	$1.20/1 K
Student (0.5B)	0.5B	0.84	25	$0.001/1 K
Retention	1/960×	94%	96× faster	1200× cheaper

and Figure 7 demonstrate the remarkable efficiency gains achieved through our knowledge distillation approach while maintaining competitive performance levels. Our student model achieves 0.84 Precision@1 compared to the teacher’s 0.89, representing 94% performance retention with a model that is 960 times smaller.

The latency improvements are particularly dramatic, with inference time reducing from 2.4 s to 25 ms, enabling real-time deployment scenarios that would be impractical with the teacher model. The learning curve analysis shows that our reasoning-preserving distillation approach enables the student model to reach 0.84 performance within 10 training epochs, while direct training without distillation plateaus at 0.65 even after extended training. This demonstrates the substantial value of knowledge transfer from the teacher model’s reasoning patterns.

4.13. Case Study Analysis

Figure 8 provides a detailed visualization of our approach’s superior discrimination capabilities through the analysis of an “account cancellation” query. The heatmap clearly illustrates how our method achieves better separation between relevant and irrelevant services compared to baseline approaches.

Our reasoning-first approach correctly identifies “Account Termination Service” as the primary match with a score of 0.94, supported by detailed reasoning that explains the semantic alignment between user intent and service functionality. The system also appropriately ranks related but distinct services such as “Scheduled Cancellation” at 0.85 and “Cancellation Refund” at 0.87, demonstrating a nuanced understanding of different cancellation-related needs. Baseline methods show various failure patterns, with traditional approaches producing much lower and less discriminative scores across all services, highlighting the importance of our structured reasoning approach.

4.14. Error Analysis

Our comprehensive error analysis across 1000 challenging queries reveals three primary failure categories that account for the majority of incorrect predictions. Semantic ambiguity errors represent 32% of failures and occur when queries contain terms with multiple valid interpretations in the telecommunications domain. For example, “upgrade” might refer to device upgrades, service plan improvements, or technical infrastructure enhancements, requiring additional context that our current approach sometimes lacks. Business logic conflicts account for 28% of errors and arise when user preferences conflict with service availability rules or business constraints, such as premium features for basic accounts or conflicting service combinations. Terminology mismatch errors represent 40% of failures and reflect the gap between colloquial customer language and formal service descriptions, where regional dialects and generational language differences create matching challenges. Future improvements should focus on enhanced context modeling through conversation history integration, proactive user intent clarification through follow-up questions, and expanded terminology mapping that bridges the gap between customer language and service vocabularies.

Beyond these common failure categories, we specifically investigated instances of reasoning hallucinations and logically plausible but incorrect reasoning during our “Reasoning quality assessment” process. Hallucinations, defined as factually incorrect or unsupported statements within the generated reasoning, typically occurred in approximately 5% of complex cases. For instance, for a user query like “I want a plan that offers free international calls”, the model occasionally generated reasoning that referenced specific, non-existent data packages or features (e.g., “This plan includes 100 free international call minutes to 100 countries globally”), which were not actually offered by the recommended service. The reasoning itself was coherent and sounded authoritative, but contained fabricated details, leading to a misleading explanation. These direct factual inaccuracies highlight the challenge of strictly grounding LLM-generated content to available service knowledge.

More subtly, instances of logically plausible but incorrect reasoning were observed in about 10% of the cases, often intertwined with the aforementioned ‘Semantic ambiguity errors’ and ‘Business logic conflicts.’ For example, consider an ambiguous query like “I want to upgrade my service”. If the user intended to increase their data allowance, but the model interpreted “upgrade” as a device upgrade (e.g., for a new phone contract), it might generate a perfectly logical reasoning chain detailing the benefits of a new flagship smartphone and contract renewal options. This reasoning would be internally consistent and plausible, but fundamentally incorrect with respect to the user’s actual intent. Another instance arises from ‘Business logic conflicts’: if a user inquired about “cancelling my premium subscription” during a contractual lock-in period, the model might describe a general cancellation process, logically detailing steps like “contact customer support to handle it” and “return leased equipment”, but critically omit or misrepresent the non-cancellable nature of the contract at that time. Such reasoning appears sound on the surface but leads to an unexecutable or misleading recommendation. These cases underscore the difficulty in ensuring that LLM reasoning is not only coherent but also perfectly aligned with all external, dynamic, and potentially conflicting real-world constraints. Addressing these challenges will be a key focus of future research, particularly through enhanced external knowledge integration and robust conflict resolution mechanisms.

4.15. Generalizability and Cross-Lingual Performance

While our primary empirical validation focuses on the telecommunications domain with a Chinese query dataset, the theoretical foundations and architectural innovations of our framework are designed to be language-agnostic and domain-transferable. The reasoning-first scoring mechanism and multi-criteria evaluation framework leverage the inherent natural language understanding and reasoning capabilities of large language models (LLMs), which are increasingly multilingual and general-purpose. Knowledge distillation, as demonstrated in Section 2.5, efficiently transfers capabilities to smaller models, making deployment in diverse contexts feasible.

To further demonstrate this generalizability, we conducted supplementary experiments on an English customer service dataset from the IT support domain. This dataset comprises 8000 user queries in English and a catalog of 400 IT support services. The annotation process and evaluation metrics mirror those used in our primary study.

Table 6. Performance comparison on an English IT support dataset.

Method	P@1	P@3	nDCG@3	ILD	Satisfaction
English BM25	0.45	0.40	0.42	0.50	2.3
English Sentence-BERT	0.59	0.54	0.56	0.55	2.8
English Fine-tuned BERT	0.65	0.60	0.62	0.57	3.2
GPT-3.5 (English)	0.77	0.73	0.75	0.63	3.9
Ours (Teacher-English)	0.91	0.85	0.87	0.70	4.4
Ours (Student-English)	0.86	0.81	0.83	0.71	4.3

presents the performance comparison on this English dataset, showcasing our approach against representative English-specific baselines.

These initial results suggest that our approach maintains significant performance advantages in other linguistic and domain contexts, affirming the transferability of the core methodology. Specifically, our teacher model achieves a P@1 of 0.91, representing a substantial improvement over English-specific baselines. The student model also demonstrates strong performance with a P@1 of 0.86, further validating the effectiveness of our knowledge distillation pipeline across languages. Further comprehensive cross-lingual and cross-domain evaluations are planned as part of future work to quantify the framework’s adaptability more broadly.

4.16. Robustness to Adversarial and Noisy Queries

The practical deployment of query–service matching systems necessitates robustness against various forms of input perturbation, including noisy or intentionally adversarial queries. While our current experimental setup did not include a dedicated robustness evaluation against synthetically generated adversarial examples or natural language noise (e.g., typos, grammatical errors, informal phrasing), the inherent design of our LLM-based reasoning architecture offers theoretical advantages in this regard [61].

Large Language Models are known for their strong contextual understanding and ability to handle linguistic variations, suggesting a degree of inherent resilience to minor perturbations. The reasoning-first approach, by explicitly generating explanatory steps before producing a final score, may also provide a more stable intermediate representation that is less susceptible to superficial input changes compared to direct scoring mechanisms. The ‘Semantic ambiguity errors’ identified in Section 4.14 demonstrate the model’s struggle with *inherent* ambiguity rather than direct noise. However, the model’s performance under intentionally crafted adversarial examples or high levels of unstructured noise warrants specific investigation.

Preliminary qualitative observations suggest that our system exhibits moderate robustness to common noise types, such as minor typos (e.g., “kankel my account” instead of “cancel my account”) or rephrasing. In these cases, the LLM’s strong natural language understanding often allows it to infer the correct intent and maintain a high matching accuracy. However, for queries containing homophones, highly informal slang, or strategically constructed adversarial perturbations designed to confuse the model’s reasoning pathways, performance degradation is expected.

Future work will systematically investigate the framework’s robustness by:

• Injecting controlled noise: Evaluating performance under various levels of lexical and syntactic noise in queries (e.g., character insertions/deletions, word substitutions, grammatical errors).
• Adversarial attacks: Testing resilience against examples specifically crafted to induce misclassifications, leveraging techniques from adversarial machine learning (e.g., gradient-based attacks on embedding spaces or prompt injection attacks).
• Human-generated noisy queries: Analyzing real-world, naturally noisy queries that might arise from speech-to-text errors or hurried user input in a production environment.

Quantifying and improving this robustness will be critical for hardening the system against real-world deployment challenges and ensuring reliable performance in imperfect input scenarios, further enhancing its practical relevance.

5. Discussion

5.1. Theoretical Insights and Practical Implications

Our framework demonstrates that principled mathematical analysis can effectively guide the design of LLM-based systems. The tight alignment between theoretical predictions and empirical observations—including the 18% consistency improvement from reasoning-first architecture, optimal perturbation parameter at $ϵ=0.001$, and 94% performance retention in knowledge distillation—validates that rigorous modeling serves as a reliable foundation for system design rather than merely post-hoc explanation. The information-theoretic analysis quantifying reasoning benefits through conditional entropy reduction (Theorem A2) and the variational derivation of optimal perturbation parameters (Theorem A3) provide reusable methodological tools applicable to broader problems in neural information retrieval and multi-objective learning.

From a practical deployment perspective, our reasoning-preserving knowledge distillation pipeline addresses the critical challenge of LLM adoption in production environments. The 1200-fold cost reduction and 96-fold latency improvement make advanced language model capabilities accessible for real-time applications, overcoming the prohibitive resource requirements of directly deploying large foundation models. The student model’s 25 ms latency enables interactive user experiences while maintaining 94% of teacher performance, demonstrating that effective knowledge transfer can bridge the gap between model capabilities and deployment constraints.

5.2. Limitations and Challenges

Despite the strong empirical results, our approach exhibits several notable limitations. The error analysis (Section 4.14) reveals that semantic ambiguity (32% of failures), business logic conflicts (28%), and terminology mismatches (40%) remain significant challenges. These failure modes highlight the system’s limited ability to disambiguate user intent without additional context or conversation history. Furthermore, reasoning hallucinations occur in approximately 5% of complex cases, where the model generates factually incorrect service details despite producing coherent explanations, posing risks for deployment in high-stakes scenarios where accuracy is critical.

The generalizability claims are supported by preliminary cross-lingual experiments on English IT support data (Table 6), but comprehensive validation across diverse domains, languages, and cultural contexts remains incomplete. The current evaluation focuses primarily on telecommunications services with Chinese queries, and performance characteristics may vary substantially for other service types or customer demographics. Additionally, robustness to adversarial or intentionally noisy queries has not been systematically evaluated, though preliminary observations suggest moderate resilience to common perturbations like minor typos.

The reliance on large teacher models (480B parameters) for generating training data creates practical barriers to reproducibility, as most research groups lack access to such computational resources. While we emphasize that the distilled student model represents the deployable solution, the initial knowledge extraction phase requires substantial infrastructure. The dataset annotation process also involved 15 domain experts over an extended period, introducing potential annotation biases and limiting scalability to new domains without similar expert resources.

5.3. Future Work Directions

Several promising research directions emerge from this work. Enhancing context modeling through integration of conversation history and user profiles could substantially reduce semantic ambiguity errors by providing disambiguating signals. Incorporating external knowledge bases with real-time service information updates would mitigate reasoning hallucinations by grounding model outputs in verified facts rather than relying solely on parametric knowledge. Developing active learning strategies that identify ambiguous queries and request clarifying information from users before generating recommendations represents another valuable extension.

Cross-domain transfer learning deserves systematic investigation to assess whether our framework generalizes beyond telecommunications to domains like healthcare service matching, financial product recommendation, or technical support routing. Establishing benchmark datasets and standardized evaluation protocols across multiple languages and cultural contexts would facilitate broader validation of the approach. Investigating model efficiency through techniques like quantization, pruning, or mixture-of-experts architectures could further reduce deployment costs while maintaining performance.

From a theoretical perspective, extending the PAC learning bounds (Theorem A4) to account for distribution shift between teacher training data and student deployment scenarios would provide stronger guarantees for practical applications. Analyzing the interaction between perturbation-based diversity and user preference heterogeneity could enable personalized diversity–relevance trade-offs. Finally, developing interpretability techniques that explain not only individual recommendations but also systematic patterns in model behavior would enhance trust and facilitate debugging in production environments.

6. Conclusions

We presented a comprehensive framework for query–service matching that leverages large language models through structured multi-criteria evaluation and reasoning-first architecture. Our approach achieves state-of-the-art performance with 89% Precision@1 and 0.85 nDCG@3, representing substantial improvements over strong baselines. The key innovations include: (1) a reasoning-first scoring architecture that reduces prediction variance by 45–60% through information-theoretic benefits of conditional entropy reduction, (2) a multi-criteria evaluation framework with five interpretable dimensions producing Pareto-optimal solutions, (3) an adaptive perturbation mechanism balancing diversity and relevance at optimal parameter $ϵ=0.001$, and (4) a reasoning-preserving knowledge distillation pipeline achieving 94% performance retention with 960× size reduction, 96× latency improvement, and 1200× cost reduction. We provide rigorous theoretical foundations through four theorems establishing Pareto optimality (Theorem A1), information-theoretic advantages (Theorem A2), optimal perturbation parameters (Theorem A3), and PAC learning bounds for distillation (Theorem A4), with tight alignment between theoretical predictions and empirical observations validating our mathematical framework.

This work bridges the gap between large language model capabilities and practical production requirements, demonstrating that principled mathematical analysis can effectively guide system design in the LLM era. The massive efficiency improvements—1200-fold cost reduction and 96-fold latency improvement—make advanced language model capabilities accessible for real-time applications, overcoming prohibitive resource barriers of directly deploying large foundation models. The implications extend beyond telecommunications to any domain requiring nuanced text evaluation, including document retrieval, recommendation systems, and conversational AI. As language models continue to evolve, frameworks combining theoretical rigor with interpretability and efficient deployment will become increasingly critical for leveraging their capabilities in production while maintaining control, transparency, and cost-effectiveness.