Enhanced Random Ensemble Mixture: Weight Referring and Merging

Featured Application

The proposed DQN-based Enhanced Random Ensemble Mixture with Weight Referring and Merging was evaluated on the Catch Game environment. In this experimental setting, it showed more stable learning behavior than a standard DQN, suggesting that the method may be useful for reinforcement learning tasks where stable training is important.

Abstract

Reinforcement learning (RL) is widely used to learn sequential decision-making policies in complex environments. Deep Q-network (DQN) extends Q-Learning using deep neural networks, enabling learning in high-dimensional state spaces. However, conventional DQN-based approaches can exhibit variability in learning stability and convergence speed even under similar training conditions. Random Ensemble Mixture (REM) has been introduced to improve stability by combining multiple Q-value estimates, but it typically requires running multiple models simultaneously, which increases computational cost. This paper proposes an enhanced DQN method that integrates REM with a Weight Referring and Merging (WRM) mechanism to improve training stability and efficiency. The proposed approach updates a single primary agent using standard DQN learning while maintaining diversity among auxiliary agents by selectively referring to and partially merging weights from the primary network. Q-values from the primary and auxiliary agents are then combined through REM to produce the final value estimate for action selection. Experiments in the Catch Game environment indicate that the proposed method reaches stable performance earlier than a baseline DQN and reduces training time under the tested configuration (approximately 78%). While the results are encouraging in this environment, further evaluation on additional benchmarks is required to assess broader applicability.

1. Introduction

Reinforcement learning (RL) has garnered significant attention as an effective approach for learning sequential decision-making in complex environments. The deep Q-network (DQN), introduced by Google’s DeepMind research team [1], integrates deep neural networks with Q-Learning [2], enabling effective learning even in extraordinarily complex environments. This approach has demonstrated strong empirical performance in Atari games. However, it still suffers from sample inefficiency and unstable learning. In some relatively low-dimensional settings, we observed that SGD can yield smoother TD-loss dynamics than Adam under comparable tuning [3,4]. Therefore, we use SGD in our experiments and focus on whether WRM improves stability in settings where SGD already provides comparatively stable learning behavior but remaining variance still exists.

Research on RL, including DQN variants that derive diverse Q-values, has introduced approaches to enhance learning stability and performance [2,5]. The bootstrapped DQN [6] improves the balance between exploration and exploitation by implementing diverse exploration strategies via independent Q networks. Maxmin Q-Learning [7] addresses the problem of Q-value overestimation by selecting the minimum Q-value, whereas the averaged DQN [8] enhances the stability of the learning process by averaging Q-values. Multi Q-Learning [9] has been proposed to mitigate overestimation and learning instability in conventional Q-Learning and double Q-Learning. More broadly, ensemble algorithms in RL [10] may improve generalization performance by combining multiple value estimators.

Furthermore, the evolution of deep RL has led to the integration of multiple independent improvements, as seen in the Rainbow DQN [11], which combines complementary components to achieve strong performance. To further enhance sample efficiency and training stability, unified ensemble frameworks, such as SUNRISE [12], have been proposed, and related work also explores randomized ensembles for optimistic exploration [13].

However, one of the common problems of these approaches is uncertainty in Q-values. In bootstrapped DQN, sampling errors can lead to inaccurate Q-values, and in maxmin Q-Learning, selecting the minimum Q-value may hinder the identification of optimal actions. Although averaged DQN enhances learning stability by averaging Q-values, it cannot entirely eliminate the uncertainty associated with Q-value estimation.

Random ensemble mixture (REM) [14] can be applied to address these limitations. REM generates Q-values by combining the outputs of multiple models using randomly sampled weights. This approach aggregates Q-values from various DQN models using diverse information learned by each model, which can help improve learning stability in some settings. Using diverse prediction combinations from various DQN models, REM may reduce estimation uncertainty and lead to more consistent predictions. However, REM also has drawbacks. Due to the random weight selection, it is difficult to guarantee an optimal combination, which may affect prediction stability. Furthermore, multiple models are employed simultaneously. Thus, computational cost and memory usage increase. As the number of models grows, the risk of overfitting to specific models becomes more pronounced.

This paper proposes an enhanced REM that applies a Weight Referring and Merging (WRM) technique to DQN agents. Each agent starts from an independently initialized DQN with different initial parameters and shares the same stream of environment interactions collected during training. However, only the primary agent is updated by standard DQN learning, while auxiliary agents are updated only when selected for the proposed Weight Referring and Merging operation. Then, the Q-values for each action are calculated by considering the predicted partial Q-values of randomly selected agents, each multiplied by a random ratio between zero and one.

The remainder of this paper is organized as follows: Section 2 reviews related work. Section 3 describes the proposed methods. Section 4 presents the experimental setup and environment. Section 5 reports and analyzes the experimental results. Section 6 discusses the implications of the findings. Finally, Section 7 concludes this paper.

2. Related Work

The evolution of Distributional RL has shifted the focus from learning expected values to estimating the full distribution of returns. Key milestones include C51 [15], Quantile Regression DQN (QR-DQN) [16], and Implicit Quantile Networks (IQN) [17]. This paradigm shift is important because it provides a richer characterization of return uncertainty by modeling the full distribution of returns rather than only the expectation [18]. While REM can be paired with distributional value estimators (e.g., multi-head designs similar to QR-DQN), REM itself is fundamentally an ensemble Q-value mixing strategy that forms a random convex combination of multiple Q estimates. In this work, we improve the diversity and effectiveness of this mixing via our Weight Referring and Merging (WRM) mechanism, which selectively propagates and partially merges the primary agent’s weights into a subset of auxiliary agents. This idea is conceptually related to weight-space averaging/merging methods (e.g., SWA and Model Soups) that are known to improve optimization stability and generalization.

• Deep Q-Network (DQN). DQN combines deep neural networks with Q-Learning to enable effective learning even in complex environments and has achieved strong performance on Atari games. The critical innovations of DQN include experience replay [19] and the target network [5], both of which significantly enhance training stability. Experience replay stores and reuses past experiences, reducing correlations between training samples, whereas the target network provides stability in Q-value updates. In addition to these fundamental techniques, recent studies have further optimized the replay process. For instance, an enhanced DQN framework was proposed to manage massive non-executable actions by selectively updating the replay memory [20]. However, DQN still suffers from sample inefficiency, slow learning speed, and Q-value overestimation.
• Random Ensemble Mixture (REM). REM is an ensemble-based approach that uses multiple Q-networks. REM models Q-values via a dynamic ensemble mechanism with random weights, enhances exploration capabilities, and maintains computational efficiency. The core idea of REM is to implement multiple Q-function heads in a single network and generate diverse Q-values by combining them using randomly sampled weights. REM employs this mechanism to address limitations of conventional approaches. It can enhance learning stability in various settings, depending on ensemble diversity and the mixing strategy. Moreover, REM has been reported to perform well in certain offline RL scenarios and may facilitate generalization and computational efficiency, although outcomes may vary across tasks. However, REM also exhibits limitations, such as the limited diversity of ensemble members and the inability to guarantee optimal combinations due to the stochastic nature of weight selection. Beyond output-level mixing, the field of deep learning has seen a surge in Weight Averaging techniques. Methods such as Stochastic Weight Averaging (SWA) [21] and Model Soups [22] suggest that merging model parameters in weight space can lead to wider optima and may improve generalization in some settings compared with simple ensemble averaging [23,24,25,26]. These findings provide a theoretical foundation for our proposed Weight Referring and Merging (WRM) strategy, moving beyond simple output aggregation to structural knowledge integration.
• Bootstrapped DQN. Bootstrapped DQN introduces a parallel learning framework using multiple Q-value heads instead of a single neural network. This approach enhances exploration by randomly partitioning samples via bootstrapping and training each head on different subsets of experiences. A crucial advantage of bootstrapped DQN is its ability to model uncertainty, enabling the agent to explore unvisited regions and learn better policies in complex environments. However, maintaining and training multiple heads increase computational cost, and performance may degrade when the heads are inconsistent.
• Maxmin Q-Learning. Maxmin Q-Learning is an ensemble-based RL method that uses multiple Q-value estimates and selects the minimum to estimate the Q-value conservatively. This approach aims to mitigate overestimation problems and promote stable learning, making it effective in environments where risk aversion is crucial. A key advantage of Maxmin Q-Learning is that it can reduce overconfidence in Q-value estimates and may help maintain more conservative value targets under uncertainty. However, its conservative nature may slow convergence to the optimal policy, and the increased computational complexity can lower sample efficiency.
• Averaged DQN. Averaged DQN aims to achieve stable learning by averaging Q-values from multiple target networks. Instead of relying on a single target network, this method combines the outputs of several past target networks, reducing the variance of Q-value estimates and alleviating overestimation problems. The advantages of averaged DQN are its simplicity of implementation, improved learning stability, and more consistent performance than standard DQN. However, the delayed incorporation of recent information due to the averaging process may limit its exploration capability in complex environments.
• Ensemble DQN. Ensemble DQN trains multiple independent Q networks and combines their predictions. By learning from different initializations and experiences, each network helps estimate the uncertainty in Q-values and improves the balance between exploration and exploitation. By aggregating diverse value estimators, ensemble DQN can help mitigate Q-value overestimation and may provide more consistent performance across different initializations. However, this approach also incurs high computational costs due to the simultaneous training of multiple networks, and performance improvements may be limited without sufficiently diverse ensemble members.
• Transfer Learning. Transfer learning [27] enhances learning efficiency by applying a pretrained model to a new task and is widely used in RL. This method accelerates early-stage learning and reduces data requirements by transferring knowledge acquired in similar environments to new ones. The primary advantage of transfer learning is that effective learning can be achieved with fewer samples, particularly when there is a high degree of similarity between the source and target domains. However, if the discrepancy between the original and target tasks is considerable, the transferred knowledge may negatively influence performance. Additionally, designing an appropriate transfer strategy often requires extra effort and careful consideration.
• Genetic Algorithm. The genetic algorithm (GA) [28], inspired by evolutionary theory, is a population-based, gradient-free optimization method widely used to search neural network architectures and tune hyperparameters. It iteratively improves a population of candidate solutions by applying operations, such as selection, crossover, and mutation. In practice, GA is commonly used to adjust neural network weights or hyperparameters, especially when the search space is discrete, non-differentiable, or highly non-convex. Its advantages include strong exploratory capability in discontinuous and complex search landscapes compared with gradient-based methods, and the ability to escape poor local optima. However, it often requires substantial computational resources, may converge slowly, and can suffer from inconsistent performance due to the stochastic evolutionary operators.

In our experiments (see Section 5), the proposed method yielded a more stable loss trajectory than the baseline under the evaluated setup. We present these results as empirical evidence for training stabilization, while broader applicability remains for future work.

3. Methods

3.1. Overview

This paper proposes an enhanced method that combines REM with a weight (network weight) referring and merging strategy to improve the learning stability and training efficiency of DQN. The core idea of the proposed method is to refer to the network weights of a primary agent and merge them into other non-learning agents. This reduces variability during training and improves overall stability, while enabling agents to quickly reflect the learning outcomes of the primary agent, which may help reduce training time under certain conditions.

As shown in Figure 1, the primary agent performs the same function as conventional DQN, updating Q-values during learning. After completing training at each step, the primary agent’s weights $w_t$ (network weights) are referred to and merged into selected other agents, termed merging agents, to maintain a balance between agent diversity and training stability.

Figure 1. Weight Referring and Merging (WRM) along with the Q-value mixing process.

The weights are not uniformly applied to all agents. Instead, they are selectively propagated only to a merging agent subset based on a predefined ratio and applied with different proportions to each selected agent. In this paper, all non-primary agents are referred to as auxiliary agents, and among them, the subset that receives the primary agent’s weights at a given update step is called the merging agents. This approach aims to retain estimator diversity comparable to independently trained agents. However, auxiliary agents are not fully trained independently and are updated only through the proposed referring and merging mechanism.

More specifically, all merging agents generate their Q-values alongside the primary agent. These Q-values are passed to a Q-value mixer, where REM is employed to assign random weights to the Q-values of randomly selected agents, and the final Q-value is computed by averaging the weighted results.

The primary agent performs the standard DQN Q-value update. By contrast, the auxiliary agents share the same network architecture but are updated by referring to and partially integrating the primary agent’s weights, rather than being trained independently. At each update step, only a subset of auxiliary agents is selected as merging agents and receives the merged weights. This mechanism helps preserve diversity within the agent population while allowing the ensemble to exploit the primary agent’s learning progress.

The proposed method iteratively performs the following steps throughout the training process: Learning, Network Weight Referring and Merging, Q-Value Mixing, and Action Selection.

In the Learning step, as in conventional DQN, the primary agent estimates the Q-value $q_t^P$ based on the current state $s_t$. In the Network Weights Referring and Merging step, merging agents refer to and merge the network weights from the primary agent. During the Q-Value Mixing step, the Q-value mixer receives Q-values from the primary agent and merging agents and determines the final Q-value $q_t^X$. In the Action Selection step, the action selector derives the action $a_t$ based on the mixed Q-value and delivers it to the environment.

3.2. Network Weight Referring and Merging Step

The referring and merging step in the primary agent considers the following factors. First, the merging agents to be updated are randomly selected from those excluding the primary agent, and they differentially refer to the network weights of the primary agent. Specifically, random proportions are generated for each merging agent and normalized such that their sum equals one. New weights are then computed by performing linear interpolation [29] between the weights of the primary agent and those of each agent to be updated.

These steps can be summarized as follows:

Merging-agent selection: A subset of merging agents is randomly selected for updating.

Merging-weight generation: For the selected merging agents, random merging proportions are generated and normalized so that their sum equals one:

$$p_i^D={\displaystyle \frac{r_i}{{\sum }_{j=1}^k{r}_j}}$$

(1)

where $r_i\sim \mathrm{Uniform}(0,1)$, and the constant k denotes the number of selected merging agents to be updated.

Network-weight update: Each merging agent updates its network weights via linear interpolation between the primary agent’s network weights and its current network weights:

$$W_t^D=p_i^D·W_t^P+(1-p_i^D)·W_{t-1}^D$$

(2)

where $W_t^P$ denotes the network weights of the primary agent, and $W_{t-1}^D$ represents the current network weights of a merging agent. For clarity, in Algorithm 1, we explicitly denote the weights of the i-th merging agent as $W_{t,i}^D$ to emphasize per-agent updates, and this indexed form is consistent with Equation (2), where $W_t^D$ represents the weights of a generic merging agent. Here, p is a randomly generated mixing proportion for that specific step, normalized among the selected agents.

By transferring the primary agent’s network weights to only a subset of agents, the remaining agents that do not receive weights can implement different exploration strategies, improving the balance between exploration and exploitation. In contrast, merging agents reuse the primary agent’s parameters as a reference, which can support more stable value updates and may help reduce estimation uncertainty.

3.3. Q-Value Mixing Step

Randomly selected actions are transformed into one-hot encoding vectors to reinforce the primary agent’s selection [30]. This approach prioritizes the primary agent’s decisions, balancing exploration, and exploitation to enable more effective policy learning. Mix-proportions are generated so that they sum to one. These mix-proportions are applied to the Q-values of each randomly selected agent [14], and an exploration term is added. Then, the average of the Q-values with the mix-proportions is calculated, after which the exploration rate is updated.

This step can be summarized as follows:

Random action selection: A random action a is selected and converted into a one-hot vector denoted as $e_a$ [29]:

$$e_a=[0,\dots ,1,\dots ,0],\mathrm{where}1\mathrm{is}\mathrm{at}\mathrm{the}a-\mathrm{th}\mathrm{position}.$$

(3)

Agent selection and mix-proportion generation: An arbitrary number h determines the number of randomly selected agents. h agents are randomly selected, generating a set of random mix-proportions $m=[m_1,m_2,\dots ,m_h]$, normalized such that their sum equals one:

$$m_j={\displaystyle \frac{r_j}{{\sum }_{k=1}^h{r}_k}}$$

(4)

where $r_j\sim \mathrm{Uniform}(0,1)$, $j=1,2,\dots ,h$.

Algorithm 1 WRM (Weight Referring and Merging) + REM (Q-value Mixing)

Note: Symbols follow Equations (1)–(7). Input: $W_t^P$, $W_{t-1,i}^D$, $ϵ$, $decay\_factor$, k, h Output: updated $\left\{W_{t,i}^D\right\}$, combined Q-value $q_{t,a}^c$, selected action $argmax\left(q_{t,a}^c\right)$   1: (1) Learning step (standard DQN update)   2: Update only the primary agent by standard DQN learning and obtain updated weights $W_t^P$.   3: (2) WRM step: weight referring and merging for a subset of auxiliary agents   4: Randomly select k auxiliary agents as merging agents.   5: Generate random merging proportions and normalize them as in Equation (1).   6: for $i=1$ to k do   7:     $r_i\sim Uniform(0,1)$   8: end for   9: for $i=1$ to k do 10:     $p_i^D\leftarrow {\displaystyle \frac{r_i}{{\sum }_{j=1}^k{r}_j}}$ 11: end for 12: for $i=1$ to k do 13:     Update the weights of the i-th merging agent by linear interpolation Equation (2): 14:     $W_{t,i}^D\leftarrow p_i^D·W_t^P+(1-p_i^D)·W_{t-1,i}^D$ 15: end for 16: (3) REM step: Q-value mixing and action selection 17: Select a random action a and convert it into a one-hot vector $e_a$ (Equation (3)). 18: Randomly select h agents for Q-value mixing and generate mix-proportions (Equation (4)). 19: for $i=1$ to h do 20:     $r_i\sim Uniform(0,1)$ 21: end for 22: for $i=1$ to h do 23:     $m_i\leftarrow {\displaystyle \frac{r_i}{{\sum }_{k=1}^h{r}_k}}$ 24: end for 25: for  $i=1$ to h do 26:     Compute Q-values $\left\{q_{t,i,a}\right\}$ and apply the mix-proportion with an exploration term (Equation (5)): 27:     $q_{t,i,a}^m\leftarrow m_i·q_{t,i,a}+ϵ·e_a$ 28: end for 29: Compute the combined Q-value by averaging (Equation (6)): 30:  $q_{t,a}^c\leftarrow {\displaystyle \frac{1}{h}}{\sum }_{i=1}^h{q}_{t,i,a}^m$ 31: Select the final action using $argmax\left(q_{t,a}^c\right)$ and update $ϵ$ (Equation (7)): 32:  $ϵ\leftarrow ϵ·decay\_factor$

Q-value calculation and mix-proportion application: For each selected agent, the Q-value $q_{t,i,a}^m$ is computed. The corresponding mix-proportion $m_i$ is applied, and the exploration term $ϵ·e_a$ is added:

$$q_{t,i,a}^m=m_i·q_{t,i,a}+ϵ·e_a.$$

(5)

Combined Q-value calculation: The average mixed Q-value is computed:

$$q_{t,a}^c={\displaystyle \frac{1}{h}}\sum _{i=1}^h{q}_{t,i,a}^m.$$

(6)

Exploration rate $ϵ$ update: The exploration rate $ϵ$ is decreased:

$$ϵ=ϵ·\mathrm{decay}\_\mathrm{factor}$$

(7)

In this experiment, decay_factor was set to 0.9999. The combined Q-value $q_{t,a}^c$ is employed to select the action with the highest Q-value using $\mathrm{argmax}\left(q_{t,a}^c\right)$.

4. Experiments

4.1. Experimental Environment: Catch Game

This study uses a grid-based Catch Game environment (Figure 2) inspired by prior catching-game formulations studied in multi-agent reinforcement learning [31], consisting of a 10 × 10 grid. The objective of the game is for the player to catch falling fruits, with the available actions being to move left, stay, or move right. The reward is defined as $+1$ for successfully catching a fruit, $-1$ for failing to catch it, and 0 for simply moving without catching it. This environment was employed to evaluate the agent’s decision-making and learning performance based on the proposed method.

Figure 2. Catch Game environment. The rectangle represents the basket, and the circle represents a falling fruit. (a) failure case: the fruit falls outside the basket’s reach; (b) success case: the basket successfully catches the fruit.

4.2. Network Architecture and Initialization

• Network Structure: The DQN model consists of an input layer (100 nodes representing the 10 × 10 grid), two fully connected hidden layers with 128 nodes each (ReLU activation), and an output layer with 3 nodes (left, stay, and right).
• Weight Initialization: To ensure stable starting conditions, the network weights are initialized using a truncated normal distribution (mean = 0.0).
• Standard Deviation: Calculated as $1/\sqrt{n}$, where n is the number of input units in the weight tensor.
• Bias: Initialized with a truncated normal distribution (stddev = 0.01).
• Auxiliary Agents: At the beginning of training, all auxiliary agents are independently initialized from the same truncated normal distribution as the primary agent so that they start from diverse random parameter configurations. During training, only the auxiliary agents selected as merging agents update their network weights by referring to the primary agent, while the other auxiliary agents retain their previous parameters.
• Optimizer choice (SGD vs. Adam): In this study, we focus on training stability rather than maximizing reward performance across optimizers. We used SGD as the primary optimizer (Table 1) because, in this environment, we observed that Adam tends to produce larger TD-loss oscillations and less stable convergence behavior under comparable tuning. To avoid overstating generality, we interpret our results as evidence that WRM can stabilize REM-based ensembles in settings where SGD provides more stable learning dynamics than Adam. Prior studies have reported that SGD can show better generalization behavior than Adam and that closing this gap often requires careful tuning of adaptive optimizers [3,4].
  Table 1. Hyperparameters for training.

  Hyperparameter	Value	Description
  Grid size	10 × 10	Size of the Catch Game grid
  Number of actions	3	Move left, stay, move right
  Hidden layer size	128	Number of units in each fully connected hidden layer
  Discount factor $\gamma$	0.9	Discount factor for future rewards
  Batch size	32	Mini-batch size for training from replay memory
  Replay memory size	80	Maximum number of stored transitions
  Learning rate	0.1	Learning rate of the SGD optimizer
  Number of episodes	30,000	Total number of training epochs/episodes
  Initial $ϵ$	1	Initial exploration rate for $ϵ$-greedy/ensemble perturbation
  Minimum $ϵ$	$5\times {10}^{-4}$	Lower bound of $ϵ$
  $ϵ$ decay factor	0.9999	Multiplicative decay factor applied to $ϵ$ at each step
  Number of agents	1 + 5	One primary agent and five auxiliary agents
  Agent rate	0.6	Ratio for selecting merging/ensemble agents

5. Results

This section analyzes learning performance by comparing the training loss, average win rate during training, and validation performance between the baseline DQN and the proposed method.

5.1. Training Loss

Figure 3 shows that the baseline model maintains relatively high loss values for a longer period and exhibits intermittent loss spikes even after 10,000 epochs. In contrast, the proposed method shows larger fluctuations in the early training stage but converges within approximately 5000 epochs and maintains a stable loss close to zero afterward. The baseline model also displays irregular increases in loss during later stages, whereas the proposed method maintains a more consistent loss trajectory over time. These trends suggest improved training stability for the proposed method in the tested environment.

Figure 3. Loss comparison during training.

5.2. Average Win Rate During Training

Figure 4 presents the average win rate during training. The baseline model gradually increases its win rate during early training and tends to remain above 90% after approximately 10,000 epochs, but it continues to fluctuate and frequently does not reach a sustained 100% win rate. In comparison, the proposed model increases its win rate more rapidly, reaches 100% within approximately 5000 epochs, and maintains this level until the end of training. This win-rate trend suggests earlier stabilization under the evaluated setup. However, convergence should be interpreted together with the loss trajectory reported in Section 5.1.

Figure 4. Win rate comparison during training.

5.3. Validation Performance

Figure 5 reports validation performance using fully trained models. The baseline model shows segments where the win rate does not reach 100% and exhibits irregular fluctuations, suggesting that it occasionally selects suboptimal actions even after training. In contrast, the proposed model maintains a 100% win rate throughout validation in the Catch Game environment. This outcome suggests that the learned action-selection behavior is applied more consistently during evaluation for this specific task, and generalization to other environments remains to be examined in future work.

Figure 5. Win rate history during verification.

Table 2 summarizes the training efficiency gains, showing substantial reductions in both epochs and time required for convergence.

Table 2. Performance comparison between traditional DQN and WRM methods.

Method	No	Epochs	Time
Traditional DQN	1	30,000	5:14:29
	2	7900	1:22:16
	3	14,600	2:33:30
	4	4400	0:45:19
	5	30,000	5:13:16
	Average	17,380	3:01:46
WRM	6	3300	0:38:33
	7	2900	0:33:17
	8	4400	0:50:24
	9	3200	0:37:03
	10	3600	0:41:43
	Average	3480	0:40:12
Improvement		79.98%	77.77%

The WRM method demonstrated significant performance improvements compared to the traditional DQN: Epochs reduction of 79.98% (approximately 80% reduction)—from 17,380 to 3480 epochs on average; Time reduction of 77.77% (approximately 78% reduction)—from 3:01:46 to 0:40:12 on average. Note: For baseline runs only, when a 100% win rate was not achieved within 30,000 epochs, we conservatively treated the epochs-to-threshold as 30,000 for comparison consistency. In contrast, all WRM runs reached a 100% win rate within 30,000 epochs.

6. Discussion

The experimental results in the Catch Game environment suggest that the proposed method improves learning stability and convergence behavior compared to the baseline model. Under the tested configuration, the proposed method reaches a stable win-rate regime with fewer training iterations, while the loss curves provide a clearer indication of stabilization. This observation is consistent with prior work highlighting that reducing variance in value estimation can support more consistent training dynamics. By incorporating weight-level merging, the WRM mechanism may help reduce variability across value estimates and contribute to smoother training trajectories. The loss analysis supports this interpretation. While the baseline model shows intermittent loss spikes even in later training stages, the proposed method converges earlier and maintains a more stable loss near zero after convergence. In addition, the proposed method maintains a stable win rate during later training and validation in this environment, whereas the baseline model shows occasional drops. Overall, these results suggest that combining WRM and REM can be a practical approach to improving training stability under the evaluated setting.

7. Conclusions

This paper presented an enhanced DQN approach that integrates Weight Referring and Merging (WRM) with Random Ensemble Mixture (REM) and evaluated it in the Catch Game environment. The method is designed to use a single primary agent for standard DQN updates while selectively propagating and partially merging weights into auxiliary agents to maintain diversity for Q-value mixing. In the conducted experiments, the proposed method exhibited a smoother and more stable training-loss trajectory and an earlier stabilization of the win-rate curve compared to the baseline DQN, achieving consistent validation behavior within the tested environment. The measured training time was also reduced under the same configuration (approximately 78% compared to the baseline). While these results are encouraging for the evaluated setting, additional studies on broader benchmarks and alternative configurations are required to assess general applicability. In addition, future work will evaluate robustness under increased environmental stochastic and non-stationary settings (e.g., randomized fruit-falling patterns and dynamic difficulty adjustments) to further test adaptability beyond the current controlled environment.