RL–Fusion: The Large Language Model Fusion Method Based on Reinforcement Learning for Task Enhancing

Abstract

Model fusion is a technique of growing interest in the field of machine learning, which constructs a generalized model by merging the parameters of multiple independent models with different capabilities without the need to access the original training data or perform costly computations. However, during model fusion, when the number of parameters in a large language model is high, the dimension of the parameter space increases, which makes it more challenging to find the optimal combination of weights. Meanwhile, there is considerable potential for further development in sustainable optimization schemes for task-specific performance enhancement through model fusion in this area. In this paper, we propose a large-scale language model fusion approach based on task-enhanced reinforcement learning (RL–Fusion) to efficiently explore and optimize model fusion configurations. The key innovation of RL–Fusion lies in its use of reinforcement learning to guide parameter selection during model fusion, enabling a more intelligent and adaptive exploration of the parameter space. Additionally, RL–Fusion introduces a dynamic evaluation mechanism that adjusts the evaluation dataset in real-time based on feedback from SOTA models, ensuring continuous enhancement of domain-specific capabilities. RL–Fusion outperforms the baseline model by improving 1.75% in the MMLU benchmark test, 1.8% in the C-eval test, and 1.8% in the Chinese Named Entity Recognition (NER) test on the Yayi NER dataset by 16%. The results show that RL–Fusion is an effective and scalable model fusion solution that improves performance without increasing the computational cost of traditional optimization methods and has a wide range of applications in AI research and practice.

1. Introduction

Large Language Models (LLMs) are advanced AI systems, containing billions or even trillions of parameters, that leverage deep learning and Natural Language Processing (NLP) techniques to understand and generate natural language text [1]. Trained on extensive textual datasets, these models capture intricate language structures and semantic relationships, enabling a wide range of tasks such as language comprehension, text generation, knowledge-based question answering, code generation, and conversational systems [2,3]. Recent advancements in models such as GPT [4] and the LLaMA series [5] have resulted in significant performance improvements, while also addressing challenges related to security, efficiency, and interpretability. A key optimization strategy is model fusion, which combines parameters from multiple independent models to form a more versatile, general-purpose model. This approach, akin to ensemble learning, allows for knowledge transfer without requiring access to the original training data or incurring substantial computational costs. Model fusion enhances both accuracy and robustness, thereby expanding the applicability of LLMs to complex tasks.

Model fusion techniques are broadly categorized into pre-fusion and in-fusion methods [6]. Pre-fusion methods, such as Linearization Fine-tuning [7], improve model weight decoupling in the tangent space of pre-trained models. Architecture Transformation [6] standardizes models with varying architectures into a unified structure, enabling parameter-level fusion. Techniques such as GAN Cocktail [8] map models into a shared parameter space, followed by averaging and fine-tuning for domain-specific applications. Nguyen et al. [9] fused heterogeneous neural networks of varying depths using cross-layer alignment, without increasing network size or requiring access to original training data. In contrast, in-fusion methods [6] dynamically merge models during the fusion process, providing greater flexibility and adaptability, particularly for varying tasks or data samples. These methods adjust weight importance during fusion, allowing more precise control over the merging process compared to basic averaging or linear combinations. Akiba et al. [10] introduced an evolutionary algorithm to identify optimal merging combinations for various open-source models, producing new foundational models with specific capabilities. Liu et al. [11] employed Bayesian optimization to determine the optimal merging weights during pretraining, aiming to reduce resource consumption while improving performance. Zhou et al. [12] conceptualized model merging as a multi-task learning problem, deriving optimal merging coefficients through local linearization and task vector orthogonality. Subspace-based Merging Methods [6] mitigate task interference by transforming models into sparse subspaces. Yadav et al. [13] addressed parameter interference by pruning fine-grained parameters, resolving symbol conflicts, and merging consistent parameters, thus enhancing fusion efficiency. Despite their advantages, in-fusion methods continue to face challenges in cross-domain applications and in determining the optimal fusion model weights and parameters, especially for complex or heterogeneous model structures.

An important avenue in model fusion research involves enhancing domain-specific capabilities during the fusion process. This approach emphasizes tailoring models to optimize performance in specialized domains, as different tasks may benefit from distinct parameters. For instance, Aiello et al. [14] introduced the Joint Autoregressive Mixture (JAM) framework, which fuses large autoregressive text-to-image and language models into a unified system capable of generating high-quality multimodal outputs. Chen et al. [15] presented a method for fusing multimodal LLMs spanning image, audio, video, and point cloud modalities, reducing cross-modal interference through parameter decoupling and adjustment of modal fusion coefficients. Sung et al. [16] proposed guiding principles for model merging, emphasizing the importance of using models with the same pre-training starting point, preferring simpler models with consistent performance, and favoring full-model merging over partial-layer merging. Shukor et al. [17] developed a unified architecture capable of processing four modalities—image, video, audio, and speech—by converting tasks into a “sequence-to-sequence” format, enabling the use of a unified feature extractor and classifier across all modalities. While these approaches show promise, one of the key challenges in model fusion remains to select the appropriate parameters during the fusion process, as different parameter configurations may be advantageous for different tasks. As a result, developing a sustainable framework for advancing model fusion in specific domains is still an ongoing area of research.

This paper explores the challenges of searching a high-dimensional parameter space during model fusion, as well as task capability enhancement issues. A large language model fusion method based on Reinforcement Learning [18] for task enhancing (RL–Fusion) is proposed. The core innovation of RL–Fusion lies in its use of reinforcement learning to dynamically guide the selection of fusion parameters, enabling an intelligent and adaptive exploration of the parameter space to identify optimal combinations. Additionally, RL–Fusion introduces a novel dynamic evaluation mechanism that adjusts the evaluation dataset in real time based on feedback from large language model evaluations, ensuring continuous refinement and enhancement of task-specific performance. This method identifies the optimal combination of fusion parameters to maximize model performance, particularly in enhancing performance for specific tasks, while preserving the model’s original capabilities. Performance evaluation metrics on datasets such as MMLU [19], C-eval [20], and Yayi [21] demonstrate superior results compared to the base models.

The main contributions of this work are outlined as follows:

• Reinforcement learning-based model fusion is employed to optimize parameter selection during fusion by rewarding the results, which allows for effective exploration of optimal parameter combinations and enhances model performance.
• The evaluation feedback module of the LLM provides real-time dynamic feedback, adjusting the evaluation dataset according to the reinforcement learning results, thereby effectively enhancing the fusion model’s performance.
• Effectiveness testing on multiple datasets demonstrates that the method provides modeling capabilities for task enhancement.

The remainder of this paper is organized as follows: Section 2 reviews related work. Section 3 introduces the main steps of RL–Fusion. Section 4 presents the evaluation results of the fused model on multiple datasets. Section 5 concludes the paper and outlines directions for future research.

2. Related Works

The large language model fusion method based on Reinforcement Learning [18] for task enhancing (RL–Fusion) involves weight alignment in the model fusion method and task enhancing for Model fusion.

2.1. Weight Alignment

With the rise of large language models (LLMs), model fusion techniques have garnered significant attention as a means of enhancing model performance. Early works by Goddard et al. [22] and Tang et al. [23] aimed to simplify the fusion process and establish experimental benchmarks; however, their research did not encompass the full spectrum of fusion techniques. As the field advanced, various methods were proposed to mitigate task conflicts and interference when merging models.

Singh et al. [24] performed soft alignment of neurons using optimal transport, while Tatro et al. [25] introduced a heuristic algorithm for approximating neuron alignment. Xu et al. [26] proposed a method for model alignment that operates concurrently in both the weight space and the activation space, reducing interference and conflicts during fusion. Jordan et al. [27] addressed variance collapse after interpolation in neural networks by proposing REPAIR, which readjusts pre-activation values in the interpolated network, significantly improving performance. Crisostomi et al. [28] globally optimized neuron arrangement across all layers, ensuring cyclic consistency during merging, though this method may require significant computational resources. Basic merging methods, such as parameter-weighted averages or task arithmetic operations, perform model merging using simple approaches; however, they may fail to fully capture the complex relationships between different models, resulting in poor performance, particularly when significant differences between tasks exist [6]. Weighted merging methods combine models based on their importance weights. Akiba et al. [10] employed evolutionary algorithms to efficiently search for merging coefficients, identifying optimal weights for complex fusion scenarios. Liu et al. [11] used Bayesian optimization to determine merging coefficients that minimize model entropy on unlabeled test data, suitable for large-scale datasets but computationally intensive. Zhou et al. [12] demonstrated that linear connections (LMC) exist between parameter spaces and feature maps, though finding optimal merging coefficients can be costly with many models.

Despite their advantages, these fusion methods continue to face challenges in cross-domain applications and in selecting optimal fusion model weights and parameters, particularly when dealing with complex or heterogeneous model structures. To address this issue, this paper proposes using reinforcement learning, in conjunction with large language model evaluation, to guide parameter selection during model fusion.

2.2. Task Enhancing for Model Fusion

A recent focus of model fusion research is hybrid domain fusion, which seeks to enhance domain-specific capabilities, such as generating harmless responses or facilitating engaging interactions [29]. This is particularly relevant in applications that require fine-grained behavior. Aiello et al. [14] introduced a joint autoregressive hybrid framework that fuses text and image generation models, enhancing multimodal capabilities through data-efficient instruction-tuning strategies. Chen et al. [15] optimized model performance through parameter decoupling and tuning, achieving multimodal scaling without requiring additional training. Sung et al. [16] explored merging different modal models using linear interpolation and task arithmetic to improve multitasking performance; however, they did not fully address parameter interference between modes. Shukor et al. [17] proposed a unified model capable of handling image, video, audio, and language tasks, demonstrating robust performance for multimodal fusion with small-scale parameters.

These approaches have enhanced model capabilities in specific domains through various fusion methods; however, a sustainable framework has yet to be proposed to address the impact of varying parameters on performance across different tasks during the fusion process. In this paper, we dynamically adjust the evaluation dataset based on the results of a large language model evaluation to improve the model’s performance on the named entity recognition task.

3. Method

The model fusion module employs TIES [13] and Slerp [30] to efficiently combine multiple base language models, dynamically adjusting the fusion parameters through reinforcement learning based on the characteristics of the base models. The SOTA model evaluation module assesses the performance of the fusion model on a test dataset using a Large Language Model (LLM) and provides accuracy scores as key performance indicators. Reinforcement learning optimizes the fusion parameters by balancing exploration with exploitation. Based on the current model configuration and the performance feedback obtained from the SOTA model evaluation, reinforcement learning tunes the parameters to gradually approach the optimal fusion configuration, thereby enhancing performance. The new fusion parameters resulting from the reinforcement learning adjustments are iteratively applied to the model fusion process to obtain the optimal fusion configuration. The overall architecture of the framework is presented in Figure 1.

3.1. Model Fusion

In this paper, RL–Fusion employs two model fusion methods (TIES [13] and Slerp [30]) as the basic fusion methods improved by reinforcement learning to optimize the fusion parameters for model fusion. Unlike traditional SLERP, which is limited to interpolating between two models and requires predefined interpolation factors, RL–Fusion dynamically adjusts these factors using reinforcement learning to achieve optimal fusion configurations. Similarly, while TIES focuses on resolving parameter redundancy and sign conflicts through fixed density and weight parameters, RL–Fusion enhances TIES by leveraging real-time feedback to refine density and weight settings, ensuring better alignment with task-specific objectives. By integrating reinforcement learning, RL–Fusion not only overcomes the limitations of these methods but also introduces a dynamic evaluation mechanism that continuously adapts the fusion process based on performance feedback. This approach ensures a more intelligent, adaptive, and efficient model fusion process compared to static methods like SLERP and TIES.

3.1.1. Slerp

Spherical Linear Interpolation (Slerp) [30] is a method for smoothly interpolating between two vectors, maintaining a constant rate of change and preserving the geometric properties of spherical space. First, the input vectors are normalized to unit length, ensuring they represent direction rather than magnitude. Next, the angle is computed using the dot product. When the vectors are nearly parallel, linear interpolation is applied for efficiency; when the angle is large, SLERP is used, with scale factors computed based on the interpolation factor t and the angle. These factors weight the original vectors and are then summed to obtain the interpolated vector. Slerp can be described as follows:

$$\mathrm{S}\mathrm{l}\mathrm{e}\mathrm{r}\mathrm{p}(\mathrm{p},\mathrm{q},\mathrm{t})=\frac{\sin \left[\right(1-\mathrm{t}\left)\mathsf{\theta }\right]·\mathrm{p}+\sin (\mathrm{T}\mathsf{\theta })·\mathrm{q}}{\sin \mathsf{\theta }},$$

(1)

p, q are the two vectors to be interpolated; T is the interpolation factor, a value between 0 and 1 that specifies the degree of interpolation. Reinforcement learning dynamically adjusts T based on the results of the model evaluation; $\mathsf{\theta }$ is the angle between the two vectors. SLERP is applied to each layer of the model to achieve overall model fusion. Reinforcement learning is used to obtain a new fusion model by adjusting the T.

3.1.2. TIES

TIES [13] aims to efficiently merge multiple task-specific models into a single multi-task model and optimizes model fusion in three steps. First, the Trimming (Trim) phase retains the top k% of parameters with large variations in each task’s parameters, while the remaining parameters are either reset to 0 or restored to the pre-trained model values to reduce interference from redundant parameters on model performance. Next, in the Elect phase, an election symbol vector is created for parameters with opposite signs across different models, and the final sign of each parameter is determined by summing the parameters across models. Finally, the Disjoint Merge phase is performed to compute the average of model parameters with the same final sign, thereby avoiding interference caused by sign inconsistency.

3.2. SOTA LLM Evaluation

In this section, we detail the LLM-based model evaluation method (the model used in this experiment is GPT-4o), which is specifically designed for evaluating model performance in the Named Entity Recognition (NER) task. The evaluation process follows a comprehensive framework that considers several key factors. The overall architecture of the SOTA LLM evaluation is shown in Figure 2.

• Generate Evaluation Prompt

The LLM is employed to generate the prompts required for the assessment. The prompts will direct the model to combine the prediction results from the model ${\mathrm{M}}_{\mathrm{o}}$ and the dataset ${\mathrm{D}}_{\mathrm{o}}$ to calculate the contextual relevance (CR) by evaluating whether the NER results of model and dataset align and whether model and dataset are linguistically congruent. The corresponding weight parameters, $\mathsf{\alpha }$ and $\mathsf{\beta }$, are also set for both aspects in the prompt.

2.

Contextual relevance

Calculate the contextual relevance (CR) between the model output and the ground truth output. This step involves evaluating two dimensions: entity matching and linguistic consistency.

$$\mathrm{M}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}({\mathrm{M}}_{\mathrm{o}},{\mathrm{D}}_{\mathrm{o}})={\displaystyle \frac{\left|\left\{{\mathrm{o}}_{\mathrm{i}}\in {\mathrm{M}}_{\mathrm{o}}|\exists {\mathrm{g}}_{\mathrm{i}}\in {\mathrm{D}}_{\mathrm{o}},{\mathrm{o}}_{\mathrm{i}}={\mathrm{g}}_{\mathrm{i}}\right\}\right|}{\left|{\mathrm{M}}_{\mathrm{o}}\right|}}.$$

(2)

The Match function measures how accurately the model identifies entities compared to the ground truth, based on exact type and content matching. ${\mathrm{M}}_{\mathrm{o}}$ is the model prediction result, ${\mathrm{D}}_{\mathrm{o}}$ is the true value of the dataset, and ${\mathrm{o}}_{\mathrm{i}}$ is the type and value corresponding to each entity in the set of entities identified by the model. ${\mathrm{g}}_{\mathrm{i}}$ is the type and value corresponding to each entity in the set of entities in the true value of the dataset.

$$\mathrm{L}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{u}\mathrm{a}\mathrm{g}\mathrm{e}({\mathrm{M}}_{\mathrm{o}},{\mathrm{D}}_{\mathrm{o}})={\displaystyle \frac{{\mathrm{V}}_{\mathrm{M}}·{\mathrm{V}}_{\mathrm{D}}}{‖{\mathrm{V}}_{\mathrm{M}}‖‖{\mathrm{V}}_{\mathrm{D}}‖}}.$$

(3)

The Language function measures the linguistic consistency between the model’s output and the dataset, using a vector-based similarity measure (cosine similarity). ${\mathrm{V}}_{\mathrm{M}}$ is the linguistic feature vector of the model output ${\mathrm{M}}_{\mathrm{o}}$, and ${\mathrm{V}}_{\mathrm{D}}$ is the linguistic feature vector of the dataset result ${\mathrm{D}}_{\mathrm{o}}$; $‖{\mathrm{V}}_{\mathrm{M}}‖‖{\mathrm{V}}_{\mathrm{D}}‖$ is the modulus of the vector, respectively.

$$\mathrm{C}\mathrm{R}\left({\mathrm{M}}_{\mathrm{o}},{\mathrm{D}}_{\mathrm{o}}\right)=\mathsf{\alpha }·\mathrm{M}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}\left({\mathrm{M}}_{\mathrm{o}},{\mathrm{D}}_{\mathrm{o}}\right)+\mathsf{\beta }·\mathrm{L}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{u}\mathrm{a}\mathrm{g}\mathrm{e}\left({\mathrm{M}}_{\mathrm{o}},{\mathrm{D}}_{\mathrm{o}}\right).$$

(4)

$\mathsf{\alpha }$, $\mathsf{\beta }$ are weight coefficients that balance the importance of entity matching and language consistency in the final assessment. These coefficients can be adjusted based on specific task requirements to reflect the relative influence of each factor.

3.

Performance Evaluation

n denotes the number of evaluation datasets. If we have n samples with scores $\mathrm{C}\mathrm{R}$, Then the average score $\mathrm{E}(\mathrm{P})$ can be expressed as:

$$\mathrm{E}(\mathrm{P})={\displaystyle \frac{1}{\mathrm{n}}}\sum \mathrm{C}\mathrm{R}({\mathrm{M}}_{\mathrm{o}},{\mathrm{D}}_{\mathrm{o}}),$$

(5)

$\mathrm{E}\left(\mathrm{P}\right)$ is the overall performance evaluation score of the model.

3.3. Reinforcement Learning to Optimize Fusion Parameters

3.3.1. State Representation

Reinforcement learning-based strategies can effectively optimize parameter configurations during model fusion. This feedback mechanism allows RL–Fusion to adjust the strategy in real time based on environmental feedback, thus continuously improving the performance and efficiency of the fused model in a given task. The state $s_t$ consists of model parameters $P_t$, performance metrics $M_t$, historical feedback $H_t$, and fusion history $F_t$.

$P_t$ denotes the model parameters at time t, including the weights of the base model and the fusion method, it can be described as follows:

$$P_t=\left[w_t,f_t\right].$$

(6)

$w_t$ is the weight value used for model fusion at the current moment. $f_t$ indicates the fusion method at the current moment.

$M_t$ denotes the performance metric at time t, generated by the SOTA LLM evaluation module (calculated from Equation (5)) to provide real-time feedback;

$H_t$ records the parameters and their corresponding rewards for each past model configuration. It can be described as follows:

$$H_t=\left\{P_t,r_t\right\}.$$

(7)

$P_t$ denotes the model parameters at time t (calculated from Equation (6)), and $r_t$ denotes the reward value obtained after the fusion of the current parameter (calculated from Equation (7)).

$F_t$ records the evolution of the model’s parameter configurations, helping the intelligentsia to decide whether to continue to utilize previously successful configurations or to explore new ones. It can be described as follows:

$$F_t=\left\{P_1,P_2,\dots P_t\right\}.$$

(8)

The state $s_t$ can be expressed by the following Equation:

$$s_t=\left[P_t,M_t,H_t,F_t\right].$$

(9)

3.3.2. Action Definition

The action space comprises operations that directly impact the model fusion process. These actions encompass adjusting the weights of the base models to control their contribution to the final output, selecting fusion methods (TIES [13] or Slerp [30]) for model combination, and optimizing hyperparameters, such as learning rates, to refine the fusion process. Additionally, the agent balances exploration (searching for new configurations) with exploitation (optimizing existing configurations) to enhance the fusion results.

An action represents the various model fusion parameter tuning strategies. The following are the available actions and their corresponding instructions:

3.3.3. Q-Learning in RL–Fusion

Q-Learning computes the reward value by comparing the new result $\mathrm{E}\left(\mathrm{P}\right)$, from the SOTA LLM evaluation (calculated using Equation (5)), with the previous result stored in the Q-table (calculated using Equation (10)). Based on this reward, the algorithm proceeds to action selection and subsequently updates the Q-value accordingly. The main steps of Q-learning in RL–Fusion are shown in Figure 3.

• Reward Feedback

The reward function offers feedback to the agent, directing its decision-making process. A positive reward is assigned when a new model configuration improves performance (accuracy increase), with the reward magnitude proportional to the improvement (+10 for a 10% increase in accuracy). Conversely, a negative reward is applied when performance declines (accuracy decrease), penalizing the agent for suboptimal choices (−10 for a 10% reduction in accuracy). The reward function can be formally described as follows:

$$r_t=\theta ·∆P·1_{∆P},$$

(10)

where $r_t$ is the reward received after the t-th decision; $\theta$ is a scaling factor for the reward, which can be set to 10 to reflect a 10% accuracy gain; $∆P$ is the performance change $\mathrm{E}\left(\mathrm{P}\right)$ new and $\mathrm{E}\left(\mathrm{P}\right)$ old (calculated using Equation (5)); $1_{∆P}$ is the indicator function when performance improves which is 1 if $∆P$ > 0 and −1 otherwise.

The reward function directs the agent’s model configuration decisions by assigning positive or negative rewards based on performance improvement or degradation. The reward magnitude correlates with the change in performance, and a model comparison-based mechanism ensures that the agent is optimizing in the correct direction.

2.

active selection

By selecting different actions (

Table 1. Action description.

Action Category	Action Description	Parameter Range	Specific Adjustment Method
Weight adjustment	Reconfigure the weights of each base model in the fusion model to enhance or attenuate the impact of a particular model.	${\mathrm{w}}_{\mathrm{i}}\in [0,1],\sum {\mathrm{w}}_{\mathrm{i}}=1$	Increase or reduce the weight of a base model, the step size is $∆\mathrm{w}=0.05$, For example, the current weight ${\mathrm{w}}_1=0.5,{\mathrm{w}}_2=0.5$, the action will adjust ${\mathrm{w}}_1$ to 0.55 and ${\mathrm{w}}_2$ to 0.45.
Fusion method selection	Different model fusion methods were chosen, Slerp, TIES	{Slerp, TIES}	Choose a new fusion method. For example, switch from the current TIES to Slerp.
Learning rate adjustment	The learning rate during the model fusion is dynamically adjusted to control the pace of parameter updates.	$\mathsf{\eta }\in [{10}^{-5},{10}^{-1}]$	A linear or exponential adjustment based on the current learning rate, for example, from $\mathsf{\eta }$ = 0.01 was increased to $\mathsf{\eta }$ = 0.02 Or reduced to $\mathsf{\eta }$ = 0.005.

), the agent progressively approaches the optimal fusion configuration by choosing under-explored actions in conjunction with the exploration factor, based on the updated Q-value optimization parameters. The specific action selection process is described in Equation (11):

$$\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}=\left\{\begin{array}{c}\mathrm{random}\mathrm{choice}\mathrm{with}\mathrm{probability}\mathsf{\epsilon }\\ {\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}}_{\mathrm{a}}\mathrm{Q}(\mathrm{state},\mathrm{a})\mathrm{with}\mathrm{probability}1-\mathsf{\epsilon }\end{array}\right..$$

(11)

Through $\mathsf{\epsilon }-\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{e}\mathrm{d}\mathrm{y}$ policy, the agent always selects the action with the highest current Q value, aiming to maximize the expected reward. With probability $\mathsf{\epsilon }$, a random action is selected, and with probability 1 − $\mathsf{\epsilon }$ (Determination based on $r_t$ Equation (10)), the action with the highest Q value is chosen. This method balances exploration and exploitation, allowing the agent to try different actions during the learning process and avoid getting stuck in local optima. The algorithm will, with a certain probability, select parameter configurations that have performed well (greedy selection), and with a certain probability, randomly explore new parameter configurations to prevent getting stuck in local optima.

3.

Q-value Update

At each time step, the agent selects an action based on the current state of the model. The Q-function is subsequently updated using the current Q-value and the observed reward, refining the estimates of different parameter configurations. Through repeated iterations, Q-learning progressively optimizes the base model parameters to maximize long-term rewards, thereby improving the performance of the final fused model. This update is governed by the following Equation (12):

$$\begin{array}{c}\mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}\right)\leftarrow \mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}\right)+\mathsf{\alpha }\left[{\mathrm{r}}_{\mathrm{t}+1}+\mathsf{\gamma }\mathrm{m}\mathrm{a}\mathrm{x}\mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}+1},\mathrm{a}\right)-\mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}\right)\right],\end{array}$$

(12)

where $\mathrm{Q}({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}})$ represents the Q value of the at taking the action in state st; α is the learning rate and controls the magnitude of the Q value update; ${\mathrm{r}}_{\mathrm{t}+1}$ is the immediate reward after executing the action in the current state; $\mathsf{\gamma }$ is the discount factor indicating the effect of the future reward on the current decision; ${\mathrm{s}}_{\mathrm{t}+1}$ is the new state after the action is executed; max$\mathrm{Q}({\mathrm{s}}_{\mathrm{t}+1},\mathrm{a})$ represents the maximum Q value of all possible actions in the new state.

As the Q-values are updated, the agent can progressively optimize the model fusion performance by continuously adjusting the model parameters (model weights, fusion methods, etc.), ultimately converging to the optimal configuration.

3.4. Algorithmic Process

The algorithm optimizes the model fusion process using Q-learning. The base models, parameters, and Q-table are initialized. In each iteration, the models are fused using TIES/Slerp, evaluated using a SOTA LLM module, and rewards are calculated. Actions are selected, and Q-values are updated to refine the parameters. The best-performing model is tracked and returned after all iterations are completed. The flow of the algorithm is illustrated in Algorithm 1.

Algorithm 1: Model Fusion and Optimization with Q-Learning

Input: Base models BaseModels = {$M_1,M_2,\dots ,M_n$} Number of epochs for training: epochs Output: Best_Model and Best_Performance 1: Initialize model parameters and metrics $\mathrm{E}\left(\mathrm{P}\right)$ (The base model before fusion was obtained using SOTA LLM evaluation). 2: Initialize Q-table Q for reinforcement learning. 3: Set Best_Model = $\mathrm{E}\left(\mathrm{P}\right)$, Best_Performance = −∞ Training and Optimization: 4: for episode = 1 to epochs: 5:         Fuse BaseModels using TIES/Slerp method, adjust model weights. 6:       Evaluate the fusion model using SOTA LLM evaluation module for current parameter configuration $\mathrm{E}\left(\mathrm{P}\right)$ Calculated from Equation (5). 7:        Calculate r using Equation (10) 8:        Select action a using Equation (11) 9:        Update Q-value using Equation (12) 10:       Update Q-value for selected action a in state s. 11:        if $\mathrm{E}\left(\mathrm{P}\right)$ > BestPerformance: 12:             BestModel = current model configuration 13:             BestPerformance = $\mathrm{E}\left(\mathrm{P}\right)$

4. Experiments and Results

In order to validate the effectiveness of RL–Fusion and its performance improvement on specific tasks, this study evaluated model fusion through ablation experiments (using reinforcement learning alone, RL–Fusion, and RL–Fusion+Finetune by fine-tuning the pre-fusion pedestal model on named entity recognition data, respectively). Experiments were conducted using the MMLU [19] and C-Eval [20] datasets to validate RL–Fusion’s improvement in cross-linguistic capabilities, and the Yayi [21] dataset was used to test its performance improvement in the named entity recognition task. In addition, to compare the effectiveness of RL–Fusion with the benchmark model, we conducted comparative experiments on the above datasets. To demonstrate the advantages of RL–Fusion in enhancing the capabilities of specific tasks and its efficiency in resource utilization, we also conducted a resource consumption comparison with the traditional fine-tuning approach and a side-by-side comparative analysis with other models with the same amount of parameters on the MMLU and C-Eval datasets.

4.1. Datasets

To validate the effectiveness of the proposed RL–Fusion model fusion method, we conducted performance evaluations on the Yayi [21], MMLU [19], and C-Eval [20] datasets. Importantly, the training set used for model fusion does not overlap with the test sets from these datasets, ensuring an unbiased evaluation. The Yayi dataset is a large-scale corpus containing millions of samples, with 54% in Chinese and 46% in English. It covers data from 12 domains, including finance, society, biology, business, industrial manufacturing, chemistry, vehicles, science, medical diseases, personal life, safety, and general knowledge, spanning a wide range of scenarios. In our experiments, named entity recognition (NER) was specifically chosen as the task to evaluate the model’s performance before and after fusion, as it is a critical area where we aim to achieve performance enhancement through the RL–Fusion approach. To this end, we utilized the NER data from Yayi, which includes 28 Chinese entity types (e.g., person, geopolitics, organization, body parts, drugs) and 130 English entity types (e.g., animal, weapon, conference, book). NER was selected as the key task for performance evaluation in our experiments, and model performance was evaluated using the SOTA LLM evaluation module.

MMLU (Massive Multitask Language Understanding) is a multi-task, multi-language evaluation dataset proposed by OpenAI, designed to assess the performance of large language models across a variety of fields and tasks. This dataset aims to assess the model’s general knowledge and reasoning abilities, encompassing 57 tasks across various disciplines, including history, mathematics, computer science, economics, medicine, and law. The difficulty of the tasks range from high school to undergraduate and graduate levels, with all questions presented in a multiple-choice format with four options. Additionally, it supports multiple languages, enabling the evaluation of the model’s cross-lingual capabilities.

C-Eval is a comprehensive Chinese evaluation suite designed to assess the advanced knowledge and reasoning abilities of base models within the Chinese context. It is the first evaluation suite to comprehensively cover multiple difficulty levels (middle school, high school, university, and professional levels) in Chinese, with multiple-choice questions across 52 distinct subjects, spanning a broad spectrum of fields from the humanities to science and engineering.

4.2. Model Evaluation Metrics

The evaluation metrics for the model primarily focus on classification accuracy across various tasks. The model’s performance on these tasks is evaluated by calculating the average classification accuracy across all examples and tasks. The corresponding performance evaluation metrics include classification accuracy (which quantifies the proportion of correct predictions made by the model across all tasks) and calibration error (which measures the discrepancy between the model’s predicted confidence and the actual accuracy).

The evaluation metrics for the MMLU and C-eval dataset can be calculated using the following steps:

$D$ is the number of all disciplines; for each discipline $\mathrm{d}\in \mathrm{D}$, ${\mathrm{Q}}_{\mathrm{d}}$ is the set of questions for that discipline and $\left|{\mathrm{Q}}_{\mathrm{d}}\right|$ is the number of questions; for each question q $\in {\mathrm{Q}}_{\mathrm{d}}$, ${\mathrm{A}}_{\mathrm{q}}$ is the correct answer and $\overline{{\mathrm{A}}_{\mathrm{q}}}$ is the result predicted by the model.

• For each discipline $\mathrm{d}$, calculate the accuracy for that discipline ${\mathrm{A}}_{\mathrm{d}}$
  $$A_d={\displaystyle \frac{1}{\left|Q_d\right|}}{\sum }_{q\in Q_d}\delta (\overline{A_q}=A_q),$$

  (13)

  where $\mathsf{\delta }(\overline{{\mathrm{A}}_{\mathrm{q}}}={\mathrm{A}}_{\mathrm{q}})$ is 1 when $\overline{{\mathrm{A}}_{\mathrm{q}}}={\mathrm{A}}_{\mathrm{q}}$ otherwise 0.
• Calculate the average accuracy across all disciplines $\mathrm{A}$
  $$A={\displaystyle \frac{1}{\left|D\right|}}{\sum }_{d\in D}{A}_d.$$

  (14)
• Calculate weighted average accuracy $A_w$ across all disciplines:

  $$A_w={\displaystyle \frac{\sum _{d\in D}\left|{\mathrm{Q}}_{\mathrm{d}}\right|·A_d}{\sum _{d\in D}\left|{\mathrm{Q}}_{\mathrm{d}}\right|}}$$

  (15)

Evaluation metrics on NER tasks are calculated from the description in Section 3.2.

4.3. Model Parameter Setups

MergeKit [22] is an open-source model fusion toolkit that offers a comprehensive set of APIs and strategies for integrating pretrained language models. The toolkit supports a variety of fusion techniques, including, but not limited to, weighted averaging, voting mechanisms, and advanced parameter optimization strategies. With MergeKit, model fusion can be efficiently performed, and parameters can be automatically adjusted during the fusion process. This paper employs methods integrated into MergeKit, such as Slerp [30] and TIES [13], for model fusion. The GPUs employed in the experiment are two NVIDIA GeForce RTX 4090 units, and the deep learning framework used is PyTorch v2.3.2 [31].

The base model consists of llama-3-chinese-8b-instruct-v3 and llama-3-8b-Instruct with the total number of training rounds set to 50, the initial learning rate of Q-learning is set to 0.01, the discount factor is set to 0.9, the exploration rate is set to 0.8, and the rate of exploration rate is set to decay by 5% every 5 epochs. For the initial fusion method, Slerp [30] was chosen, and the initial interpolation factors were set to [[0, 0.5, 0.3, 0.7, 1], [1, 0.5, 0.7, 0.3, 0]] These factors define the interpolation ratios between the two models at different layers, controlling how much each model contributes to the final fused model at various points in the interpolation process. For example, t = 0.5 means an equal contribution from both models, while t = 0.3 and t = 0.7 represent skewed contributions. Additionally, the initial model weights in the TIES [13] method were set to [0.5, 0.5], which define the contribution of each model to the final merged model. These weights ensure a balanced integration of the models during the fusion process.

4.4. Experiments

In our experiments, we compare the new model obtained from RL–Fusion with the two base models, llama-3-chinese-8b-instruct-v3 and llama-3-8b-Instruct, on the aforementioned datasets.

4.4.1. Ablation Experiment

To validate the effectiveness of the proposed RL–Fusion approach, we conducted a series of ablation experiments to evaluate the contribution of each component to the overall performance by progressively disassembling each part of RL–Fusion. The ablation experiments used a reinforcement learning (RL)-based selection strategy. In addition, two model fusion strategies were implemented: one using only RL for parameter selection (at which point the evaluation dataset was in the top 5% of the Yayi dataset), and the other combining reinforcement learning with dynamic model evaluation feedback (dynamically adjusting the number of evaluation datasets based on NER evaluation results). To further evaluate the effectiveness of these strategies, we fine-tuned the base model on the Chinese Named Entity Recognition (NER) dataset and evaluated its performance on the MMLU and C-Eval datasets (all sub-tasks; results are shown in Figure 4 and Figure 5), as well as on the Yayi NER dataset (results are shown in Figure 6).

In order to validate the effectiveness of the proposed RL–Fusion approach, we conducted a series of ablation experiments to evaluate the contribution of each component to the overall performance by progressively decomposing the parts of RL–Fusion. The ablation experiments used a reinforcement learning (RL)-based selection strategy. In addition, two model fusion strategies were implemented: one using only RL for parameter selection with a fixed evaluation dataset, and the other combining reinforcement learning with dynamic model evaluation feedback to dynamically adjust the evaluation dataset through evaluation feedback. To further evaluate the effect of RL–Fusion on fine-tuning the model, we fine-tuned the pre-fusion base model, llama-3-chinese-8b-instruct-v3, on the Chinese Named Entity Recognition (NER) dataset and performed model fusion with RL–Fusion on both the MMLU and C-Eval datasets (across all subtasks). As an additional baseline, we also compared the performance of RL–Fusion against a randomly parameterized fusion model, where the model parameters were fused without any optimization strategy, serving as a comparison to assess the impact of RL-based adjustments. The results are shown in Figure 4 and Figure 5, and the Yayi NER dataset (results are shown in Figure 6) to evaluate its performance.

The experimental results, presented in

Table 2. Results of the ablation experiments with the RL–Fusion method.

Method	MMLU	C-Eval	Yayi(NER)
Baseline	68.2%	76.7%	2.1
RL	70.1%	77.9%	2.4
RL–Fusion	70.8%	78.6%	2.5
RL–Fusion+FineTune	69.5%	80.2%	3.8

Bolded content is the experimental results of this paper.

, clearly demonstrate that the RL–Fusion method outperforms other single-parameter selection strategies across all evaluation metrics. Furthermore, after fine-tuning the base model on the Chinese NER data, applying the RL–Fusion fusion strategy led to significant performance improvements, particularly on the Yayi (calculated by Equation (5)) and C-Eval (calculated by Equation (15)) datasets. The relatively lower performance on the MMLU (calculated by Equation (15)) dataset can be attributed to the fine-tuning enhancing the model’s knowledge of NER but also causing the fine-tuned model to underperform on more generalized data. These results underscore the superiority of the RL–Fusion method in optimizing model performance for Named Entity Recognition (NER) tasks.

Based on the experimental results, RL–Fusion demonstrates notable performance improvements across the MMLU, C-Eval, and Yayi (NER) datasets. On the MMLU dataset, RL–Fusion outperforms the baseline by 2.6%, increasing from 68.2% to 70.8%, and also surpasses RL by 0.7%, rising from 70.1% to 70.8%. Compared to RL–Fusion+FineTune, RL–Fusion’s performance is slightly lower by 0.3%. On the C-Eval dataset, RL–Fusion shows a significant improvement, outperforming the baseline by 1.9%, rising from 76.7% to 78.6%, and surpassing RL by 0.7%, from 77.9% to 78.6%. However, RL–Fusion performs 1.6% worse than RL–Fusion+FineTune. On the Yayi (NER) dataset, RL–Fusion shows a marginal improvement of 0.1 over RL, increasing from 2.4 to 2.5, and outperforms the baseline by 0.4, rising from 2.1 to 2.5. However, compared to RL–Fusion+FineTune, RL–Fusion exhibits a 1.3-point decrease, with RL–Fusion+FineTune achieving 3.8.

Through this series of ablation experiments, we not only validate the effectiveness of the RL–Fusion approach but also demonstrate that the combination of RL and SOTA LLM real-time evaluation feedback for parameter selection significantly enhances model performance on Chinese NER tasks. Specifically, on the MMLU dataset, RL–Fusion shows a 0.7% improvement over RL, and on the C-Eval dataset, it improves by 0.7%; although the improvement on the Yayi (NER) dataset is smaller (only 0.1), RL–Fusion still outperforms RL–Fusion+FineTune in comparison. These findings provide compelling evidence for the future application of model fusion strategies in complex NLP tasks.

4.4.2. Comparison with the Original Model

Table 3. The overall results of RL–Fusion LLM and the baseline for the MMLU 7 subtask inference evaluation.

Task	llama-3-8b-Instruct	llama-3-Chinese-8b-Instruct-v3	RL–Fusion LLM
formal_logic	54.0%	50.8%	53.2%
high_school_government_and_politics	92.2%	90.2%	94.8%
international_law	81.0%	80.2%	86.8%
jurisprudence	79.6%	76.9%	82.0%
logical_fallacies	74.8%	74.2%	76.1%
management	84.5%	77.7%	82.5%
marketing	91.0%	90.2%	91.5%
Weighted_accuracy	79.6%	77.2%	81.0%

Bolded content is the experimental results of this paper.

presents a comparison of the overall results between the RL–Fusion LLM and baseline models on the MMLU benchmark. To evaluate the model’s language processing capabilities, we selected seven sub-tasks from the MMLU benchmark: Formal Logic, Jurisprudence, International Law, Logical Fallacies, High School Government and Politics, Marketing, and Management. These sub-tasks require a significant level of information extraction and processing ability. The model must extract key information from the given text, understand the context, and conduct logical reasoning and analysis. We observe that the three LLMs exhibit varying performance across the seven MMLU tasks. On average, the RL–Fusion LLM demonstrates a 1.75% relative performance gain over the original best-performing llama-3-8b-Instruct across all seven tasks. In specific tasks, the enhancement achieved by RL–Fusion LLM is substantial, such as an increase from 81.0% to 86.8% on the International Law task. There are two possible reasons for the degradation of the performance of RL–Fusion LLM in tasks such as formal logic and management. First, the poor performance of the base model in these tasks affects the fusion results. Second, continuous pre-training of Chinese instructions and their relevance to downstream tasks on the llama-3-chinese-8b-instruct-v3 model also leads to performance degradation.

Table 4. The overall results of RL–Fusion LLM and the baseline for the C-eval inference evaluation.

Task	llama-3-8b-Instruct	llama-3-Chinese-8b-Instruct-v3	RL–Fusion LLM
stem	67.9%	63.2%	65.2%
average(Hard)	59.8%	57.4%	60.7%
Social Science	82.8%	84.5%	85.0%
Humanity	78.4%	82.2%	83.9%
Other	70.8%	71.1%	72.3%
Average	75.1%	77.2%	78.6%

Bolded content is the experimental results of this paper.

presents a comparison of the overall results between the RL–Fusion LLM and baseline models on the C-Eval benchmark. We observe that the three LLMs exhibit varying performance across the C-Eval tasks. On average, the RL–Fusion LLM demonstrates a 1.8% relative performance gain over the original best-performing llama-3-chinese-8b-instruct-v3 across all tasks. In specific tasks, the improvement achieved by RL–Fusion LLM is substantial, with an increase from 57.4% to 60.4% on the Average (Hard) task.

Table 5. The overall results of RL–Fusion LLM and the baseline for the Yayi NER data inference evaluation.

Task	llama-3-8b-Instruct	llama-3-Chinese-8b-Instruct-v3	RL–Fusion LLM
NER	1.8	2.1	2.5

Bolded content is the experimental results of this paper.

presents a comparison of the overall results between the RL–Fusion LLM and baseline models on the Named Entity Recognition (NER) data from Yayi. The evaluation results were computed using SOTA LLM evaluation. We observe that the three LLMs exhibit varying performance on the NER task. The RL–Fusion LLM demonstrates a 16% relative performance gain in Chinese NER compared to the original best-performing llama-3-chinese-8b-instruct-v3.

In summary, the results of this study unequivocally validate the effectiveness of the RL–Fusion fusion method. By integrating reinforcement learning with real-time model evaluation feedback to optimize the model fusion process, RL–Fusion not only improves model performance across diverse tasks but also provides a robust framework for enhancing the performance of large language models, particularly for tasks requiring complex reasoning and entity recognition. These findings confirm that RL–Fusion is an effective method for enhancing the capabilities of LLMs, positioning it as a valuable tool for a wide range of natural language processing applications.

4.4.3. Comparison with Fine-Tuned Models on Computational Resources

In order to assess the advantages of the method proposed in this study in terms of resource utilization, an experiment was conducted to compare it with the model fine-tuning method in terms of time, hardware resources, and model generalization capabilities. The results of the experiment are presented in

Table 6. Overall results of the comparison of RL–Fusion LLM and fine-tuning methods.

	Time	Hardware Resource	MMLU	C-Eval
FineTune	60 min	RTX4090 24G	68.5%	79.2%
RL–Fusion	15 min	cpu	70.8%	78.6%

Bolded content is the experimental results of this paper.

.

The fine-tuning method uses Lora [32] and the amount of data involved in the experiment is 1000.

The experiment comparing RL–Fusion and fine-tuning highlights RL–Fusion’s advantages in resource efficiency and performance. RL–Fusion completes tasks in just 15 min using only CPU resources, while fine-tuning takes 60 min and requires an RTX4090 GPU. Despite the fine-tuning method being trained on a Chinese dataset, which improves its C-Eval performance (79.2%), RL–Fusion outperforms it on MMLU (70.8% vs. 68.5%) and remains competitive on C-Eval (78.6%). These results demonstrate that RL–Fusion offers a more time- and resource-efficient approach, achieving comparable or even slightly better performance in key benchmarks.

4.4.4. Comparison with the Other Model

We compare several other large models using the same parameters and dataset. To validate the effectiveness of the proposed method, we use the model combining Slerp and TIES as the baseline for comparison.

Table 7. Overall results of the comparison of RL–Fusion LLM and Other LLMs.

Model	MMLU	C-Eval
RL–Fusion	70.8%	78.6%
Slerp [30]	68.7%	74.1%
TIES [13]	66.9%	76.2%
Llama 3.1-8B [33]	73.0%	81.2%
LLaMA 2-7B [5]	45.3%	75.2%
Mixtral 8x7B [34]	70.6%	74.7%
Qwen 7B [35]	56.7%	59.6%
GPT-3 6.7B [19]	43.2%	54.4%
RWKV v5 Eagle 7B [36]	31%	46.7%
Falcon 7B [37]	28.0%	39.2%

Bolded content is the experimental results of this paper.

presents a comparison of RL–Fusion with Other large language models with a similar number of participants using the MMLU and C-eval datasets.

Table 7 compares RL–Fusion with several large models using the MMLU and C-Eval datasets. RL–Fusion achieves 70.8% on MMLU and 78.6% on C-Eval, outperforming models like Mixtral 8x7B (70.6%, 74.7%) and Qwen 7B (56.7%, 59.6%). Notably, RL–Fusion also surpasses the performance of Slerp (68.7%, 74.1%) and TIES (66.9%, 76.2%), demonstrating the effectiveness of the proposed fusion approach. While it lags behind Llama 3.1-8B (73.0%, 81.2%), it shows significantly better performance than smaller models such as LLaMA 2-7B (45.3%, 75.2%) and GPT-3 6.7B (43.2%, 54.4%). Overall, RL–Fusion demonstrates competitive performance with larger models while being more resource-efficient.

5. Conclusions

This study extensively explores the significance of parameter selection in the model fusion process, with a focus on developing high-performance fusion models. We propose an innovative approach, RL–Fusion, which significantly improves model performance by leveraging reinforcement learning (RL) algorithms in the fusion process, combined with real-time model evaluation feedback. By dynamically adjusting the fusion parameters and evaluating the dataset, RL–Fusion effectively navigates the parameter space to discover optimal combinations. Notably, RL–Fusion enhances model performance in tasks such as Named Entity Recognition (NER), significantly improving accuracy while preserving the base model’s original functionality.

Through a series of well-designed experiments, we validate the significant advantages of the RL–Fusion approach over a single large language model (LLM) and established benchmarks. The experimental results demonstrate that RL–Fusion outperforms traditional methods across several key metrics, owing to its intelligent optimization of parameter selection. Specifically, we employed multiple LLMs with the same structure as the base model in our experiments to evaluate RL–Fusion’s performance under different model configurations. This setup enables a comprehensive evaluation of RL–Fusion’s applicability and effectiveness across various model fusion scenarios.

However, deploying RL–Fusion in real-world applications presents several challenges. First, the computational overhead of reinforcement learning during the fusion process, particularly for high-dimensional parameter spaces, may limit its scalability for extremely large models. Second, the dynamic evaluation mechanism, while effective, requires access to high-quality, domain-specific evaluation datasets, which may not always be readily available. Third, the real-time feedback loop, though innovative, introduces additional latency, which could be a bottleneck in time-sensitive applications. Finally, the generalizability of RL–Fusion across diverse tasks and domains needs further investigation, as its performance may vary depending on the complexity and specificity of the target task.

In summary, RL–Fusion leverages the adaptive learning capabilities of RL to offer a novel and powerful solution in model fusion. Our research not only advances model fusion techniques but also offers new insights into more efficient parameter optimization and improved model performance across a range of AI applications. We anticipate that RL–Fusion will be widely adopted in future research and practice to address more complex model fusion challenges and contribute to the advancement of AI technology. Future work will focus on addressing these deployment challenges to make RL–Fusion more accessible and scalable for real-world use cases.