Behavioral Analysis of Transfer Learning in DQN-Based Pedestrian Agents Using Single-Agent Pretraining

Abstract

Reinforcement-learning-based pedestrian agents can acquire adaptive behaviors without hand-crafted motion rules, but training from scratch in each environment is computationally expensive and often environment dependent. This paper investigates a two-stage transfer-learning framework for Deep Q-Network (DQN)-based pedestrian agents and analyzes how single-agent pretraining changes subsequent learning and behavior formation in multi-agent environments. A shared Q-network is first pretrained over four single-agent source tasks using Conservative Q-Learning with adaptive source-task allocation and is then transferred as the initial network of every agent in four multi-agent target tasks. Unlike fixed curriculum training, the source tasks are not traversed in a predetermined order; instead, experience collection is reallocated according to current collision and goal-progress performance. Compared with random initialization, the transferred network improves early-stage learning in intersection, bi-flow, and bi-door environments and significantly reduces target-task training time in these three tasks. In the bottleneck environment, the improvement is limited and the reduction in target-task training time is not statistically significant. Behavioral analysis further shows that pretraining changes early exploration: in the bi-door task, transferred agents reach goal-side and door regions earlier while spending less time near walls. Post-training trajectories also reveal collision avoidance and diverse route choices. These results clarify both the benefits and limitations of single-agent pretraining for efficient and interpretable behavior acquisition in multi-agent pedestrian simulation.

1. Introduction

Pedestrian simulation is widely used in application areas such as crowd-flow analysis, evacuation planning, architectural design, and virtual environments. Deep-learning-based pedestrian-motion simulation has been discussed in simulation-application contexts [1]. Simulation-based studies have also examined route planning and evacuation behavior in metro and transit-station environments [2,3], as well as hospital-flow simulation and building-design support [4,5]. Recent review work on metro-station evacuation further shows that this area includes simulation-based safety assessment, risk analysis, fire and smoke effects, interchange-station layouts, and behavioral studies [6]. Recent review and mapping studies have also summarized pedestrian evacuation simulation and crowd-simulation research from broader methodological perspectives [7,8,9]. Microscopic pedestrian models are particularly useful when the objective is to represent interactions among individual pedestrians.

Conventional approaches include the Social Force Model (SFM) [10] and velocity-based collision-avoidance models such as ORCA [11]. These methods are computationally efficient and provide explicit motion rules, but agent behavior is largely determined by predefined interaction mechanisms. Consequently, it is difficult to represent learning-dependent adaptation and diverse behavioral strategies using a single fixed rule set.

Reinforcement learning (RL) offers a different approach in which pedestrian agents acquire policies through repeated interaction with the environment. Previous studies have shown that RL agents can learn goal-directed navigation, obstacle avoidance, and multi-agent pedestrian behaviors [12,13]. Deep Q-Networks (DQNs) extend value-based RL to high-dimensional observations by approximating action values with neural networks [14]. Recent evacuation studies have also used deep RL to learn evacuation routes, exit choices, and obstacle-avoidance behavior from simulated experience [15,16]. However, two practical limitations remain important for pedestrian simulation. First, training from scratch in every new pedestrian environment requires substantial computational effort. Second, a policy learned in one environment may be strongly dependent on that environment and may not provide useful initialization for another environment.

Recent RL-based pedestrian-simulation studies have addressed these limitations mainly through curriculum design and knowledge reuse. Curriculum methods use scenarios of increasing difficulty to acquire pedestrian movement competences [17], and RL-Godot extends this curriculum-based line of research [18]. Knowledge transfer has also been investigated in multi-agent pedestrian learning and in DQN-based transfer learning for dynamic fire-evacuation planning [12,19]. Our setting is related but more specific: basic navigation knowledge is learned in multiple single-agent environments, combined in one reusable Q-network without a fixed source-task order, and then transferred to several multi-agent target environments. This separation leaves pedestrian–pedestrian interaction to target-task learning, allowing us to examine which parts of single-agent knowledge remain useful once such interaction is introduced. More generally, transfer learning reuses knowledge acquired in source tasks in related target tasks. Recent work has reviewed such mechanisms specifically in deep reinforcement learning [20]; in reinforcement-learning domains, transferred value functions or network parameters can improve initial performance and learning speed when source and target tasks share useful structure, whereas mismatched knowledge can cause negative transfer [21].

In this paper, we investigate a two-stage transfer-learning framework for DQN-based pedestrian agents. The first stage uses multiple single-agent source tasks with different obstacle layouts to learn basic navigation knowledge, particularly goal-directed movement and static-obstacle avoidance. The second stage transfers the resulting Q-network to all agents in a multi-agent target environment, after which ordinary DQN learning is continued. The source-task pretraining process uses CQL-regularized Q-learning updates on an accumulating source-task replay dataset and adaptively reallocates the per-iteration exploration budget according to collision and goal-progress performance.

The central question is not only whether pretraining reduces target-task training time, but also how transferred knowledge changes the behavior that emerges during target-task learning. Neural-network policies are difficult to interpret directly from their parameters. We therefore use behavior-level diagnostics to make the effect of transfer observable: learning curves, collision and goal-reaching metrics, spatial visitation heatmaps during early exploration, and pedestrian trajectories after training. This analysis provides an interpretable account of when and why single-agent knowledge accelerates multi-agent learning, as well as an explicit failure case in an interaction-dominated bottleneck environment.

The main contributions of this paper are as follows. First, we formulate a two-stage learning framework that explicitly separates reusable single-agent navigation pretraining from subsequent multi-agent adaptation, transferring one shared Q-network to several structurally different target tasks. Second, we introduce performance-driven source-task allocation that redistributes the experience-collection budget toward source environments with poorer collision-avoidance or goal-progress performance, thereby avoiding a predetermined curriculum order. Third, we quantify the transfer effect across four target environments in terms of early learning performance, transfer efficacy, and target-task training time. Fourth, rather than treating the transferred network as a black box, we connect the observed efficiency gains to behavior-level evidence: spatial visitation analysis shows how pretraining changes the experiences collected during early learning, and post-training trajectories and bottleneck failure cases clarify both the useful transferred knowledge and its limits.

The remainder of this paper is organized as follows. Section 2 reviews the background and related work on pedestrian simulation, DQN, CQL, and transfer learning. Section 3 presents the proposed two-stage transfer-learning framework. Section 4 presents the simulation experiments, including the experimental setup, source-task pretraining, target-task learning performance, target-task training time, behavioral analysis of transferred knowledge, and discussion. Section 5 concludes this paper.

2. Background and Related Work

2.1. Pedestrian Simulation

Pedestrian simulation models can be broadly classified as macroscopic, mesoscopic, and microscopic approaches. Macroscopic models describe pedestrian crowds as continuous flows and are suitable for large-scale density and flow analysis [22]. Mesoscopic approaches retain individual entities while using aggregate relations such as density–speed characteristics [23]. Microscopic models explicitly represent individual pedestrians and are therefore better suited to analyses of local interaction and individual decision making.

Among microscopic models, SFM represents pedestrian motion using attractive and repulsive force terms [10], whereas ORCA determines collision-free velocities through reciprocal velocity constraints [11]. SFM has continued to serve as a basis for recent microscopic pedestrian-simulation studies. Yang et al. [24] extended SFM by introducing an avoidance-force component to represent pedestrian collision-avoidance behavior in counterflow. Siddharth and Perumal [25] developed a social-force model considering pedestrian characteristics and behavior, incorporating factors such as age, gender, and pedestrian behavior into bidirectional walking simulation. Zhao et al. [26] proposed an enhanced SFM for pedestrian grouping and avoidance behavior by incorporating visual perception constraints, group-type labeling, and collective avoidance mechanisms. These SFM-based studies show that rule-based microscopic models remain active in recent pedestrian-simulation research, although their behavior is still described through explicitly designed interaction mechanisms. RL-based models instead allow agents to acquire actions from experience. Martinez-Gil et al. [12] developed multi-agent pedestrian navigation schemes based on Q-learning and explicitly evaluated variants with and without knowledge transfer, making their study a directly relevant early example of reinforcement-learning-based pedestrian simulation with knowledge reuse. Ravichandran et al. [13] formulated pedestrian simulation as a multi-objective RL problem. More recent evacuation studies have introduced deep RL into path learning: Zhang et al. [15] trained agents in a particle-dynamics environment with obstacle and multi-exit cases, while Huang et al. [16] used DDQN with ray-scanning observations to represent first-person environmental perception. Vizzari and Cecconello [17] used a curriculum of scenarios of increasing difficulty to acquire general pedestrian movement competences, and this curriculum-based research line has recently been extended in the RL-Godot pedestrian-simulation framework [18]. In relation to transfer learning, Sharma et al. [19] used transferred Q-matrix knowledge to initialize a DQN before target-environment learning in a dynamic fire-evacuation setting. Their approach transfers shortest-path information represented in a graph-based evacuation model, whereas the present study pretrains a shared neural Q-network in multiple single-agent pedestrian source tasks and transfers the network itself to several multi-agent target environments. The present paper therefore builds on related work in RL-based pedestrian and evacuation simulation but adopts a different transfer boundary: pretraining is restricted to multiple single-agent navigation tasks, whereas interaction-specific behavior is intentionally left to subsequent multi-agent learning. This design allows the contribution of reusable navigation knowledge to be examined separately from the adaptation required for pedestrian–pedestrian interaction.

Recent review studies provide a broader perspective on pedestrian and crowd simulation. Mandal et al. [6] reviewed metro- and subway-station evacuation research and organized the literature into major themes, including simulation-based evacuation assessment, risk analysis, fire and smoke effects, interchange-station evacuation, and behavioral studies. Chen et al. [7] reviewed pedestrian evacuation simulation for indoor emergencies from the perspectives of simulation approaches, models, and tools, highlighting limitations in model adaptability and verification and validation. Senanayake et al. [8] systematically reviewed agent-based pedestrian evacuation studies and identified continuing challenges such as simplified rule-based decision-making, computational complexity, and limited data for calibration and validation. More broadly, Felemban et al. [9] conducted a recent systematic mapping study of crowd simulation research, organizing recent work across modeling paradigms, behavioral abstractions, simulation settings, implementation tools, and evaluation practices. Together, these reviews provide a recent overview of the broader pedestrian-simulation literature and complement the method-specific studies discussed above.

2.2. DQN and Conservative Q-Learning

Let the state at time step t be $s_t$, the selected action be $a_t$, and the reward be $r_t$. A DQN approximates the action-value function $Q(s,a)$ using a neural network with parameters $\theta$. A target network with parameters ${\theta }^{-}$ is used to stabilize learning. For a transition $(s_t,a_t,r_t,s_{t+1})$, the target is

$$y_t=\left\{\begin{array}{cc}{r}_t,\hfill & \mathrm{if}{s}_{t+1}\mathrm{is}\mathrm{terminal},\hfill \\ r_t+\gamma {\max }_{a^{\prime }}Q(s_{t+1},a^{\prime };{\theta }^{-}),\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(1)

and the DQN loss is

$$L_{\mathrm{DQN}}\left(\theta \right)={\displaystyle \frac{1}{B}}{\displaystyle \sum _{b=1}^B}{\left(y_b-Q(s_b,a_b;\theta )\right)}^2,$$

(2)

where B is the minibatch size and $\gamma$ is the discount factor [14].

CQL was originally developed for offline RL, in which a policy is learned from a previously collected static dataset without further environment interaction [27]. It augments the standard Bellman-error objective with a conservative Q-value regularizer designed to suppress overly optimistic values for actions that are weakly supported by the available dataset.

The source-task pretraining in this study is not a conventional fully offline RL setting. New experience is collected from the source tasks between pretraining iterations and appended to a shared replay dataset. During each update phase, however, the Q-network is trained from the accumulated dataset available at that iteration. We therefore use CQL as a conservative Q-value regularizer for batch updates over an expanding replay dataset, rather than regarding the entire pretraining procedure as offline RL. The motivation is to reduce the risk of overly optimistic Q-value estimates for state–action combinations whose representation in the finite accumulated dataset may be limited during an update phase. Because CQL augments rather than replaces the underlying Q-learning objective, the pretrained Q-network can subsequently be transferred to the target-stage DQN without changing the network architecture or the form of the action-value function.

2.3. Transfer Learning in Reinforcement Learning

Transfer learning reuses knowledge acquired from one or more source tasks in a related target task. Zhu et al. [20] provide a recent survey of transfer mechanisms in deep reinforcement learning. In value-based RL, a common form of transfer initializes the target value function or network parameters with knowledge obtained from the source task. Such initialization can improve initial performance, accelerate learning, or improve asymptotic performance [21]. Positive transfer, however, is not automatic: when source and target tasks differ substantially, transferred knowledge may be irrelevant or harmful. In pedestrian and evacuation-related RL, prior work has already examined knowledge transfer and curriculum-based reuse [12,17,18,19]. The novelty claimed here is therefore not transfer learning itself or a new RL update rule. Instead, this paper combines (i) multi-task single-agent pretraining of one reusable Q-network, (ii) adaptive source-task allocation based on current task performance, (iii) explicit transfer of that network to multiple multi-agent target tasks, and (iv) behavior-level diagnostics that explain how the transferred initialization changes early experience acquisition. We evaluate both learning efficacy and the behavioral pathway through which transfer becomes useful or ceases to be useful across target tasks with different interaction structures.

3. Two-Stage Transfer-Learning Framework

3.1. Overview

The proposed framework separates the acquisition of basic pedestrian navigation knowledge from adaptation to a specific multi-agent environment. Let

$$\mathrm{Envs}=\{{\mathrm{env}}_1,\dots ,{\mathrm{env}}_n\}$$

(3)

denote the set of single-agent source tasks. A shared Q-network $Q_{{\theta }^{\mathrm{tr}}}$ is pretrained using experience collected from these tasks. The source tasks differ in obstacle layout, start position, and goal position but share the need to approach a goal while avoiding static obstacles. The pretrained Q-network is then used as the initial network for every pedestrian agent in each multi-agent target task.

The key hypothesis is that static-obstacle avoidance and goal-directed motion learned in the source tasks reduce unproductive random exploration at the beginning of target-task learning. The transferred network is not expected to solve multi-agent interaction directly because no other pedestrians are present during pretraining. Instead, it provides a navigation-oriented initialization that is subsequently adapted through ordinary DQN learning in the target environment.

The proposed two-stage transfer-learning framework is illustrated in Figure 1.

3.2. Adaptive Multi-Task Pretraining

At the beginning of pretraining, the shared Q-network is initialized randomly. Because no task-specific performance information is available at this point, the per-iteration exploration budget is initially divided equally among the n source tasks, i.e., $p_i=1/n$. In each subsequent pretraining iteration, the fixed exploration budget N is allocated among the source tasks according to $p_i$, and the collected experiences are appended to the shared replay dataset. CQL updates are then performed using the accumulated dataset available at that iteration; no additional environment interaction occurs during the update phase. After the updates, the current network is evaluated in every source task, and the allocation probabilities for the next iteration are recomputed from the current performance. Thus, the replay dataset is fixed within each update phase but expands between pretraining iterations.

Two performance measures are used for source-task evaluation. The first is a collision-related score,

$$R_{\mathrm{collision}}={\displaystyle \frac{{\sum }_{j=1}^{n_{\mathrm{eval}}}{n}_{\mathrm{collision}}^j}{{\sum }_{j=1}^{n_{\mathrm{eval}}}\min (\mathrm{maxstep},{\mathrm{goalstep}}_j)}},$$

(4)

where $n_{\mathrm{eval}}$ is the number of evaluation rollouts and $n_{\mathrm{collision}}^j$ is the number of collisions in evaluation rollout j. This quantity is calculated as the total number of collisions divided by the total number of steps over the evaluation rollouts. The second is the goal-progress score,

$$R_{\mathrm{goal}}={\displaystyle \frac{1}{n_{\mathrm{eval}}}}{\displaystyle \sum _{j=1}^{n_{\mathrm{eval}}}}{r}_j^{\left(\mathrm{goal}\right)},$$

(5)

where $r_j^{\left(\mathrm{goal}\right)}$ is the final value of the goal-progress variable in rollout j, as defined in Section 4.1.1.

Let $f_i=f(R_{\mathrm{collision}}^i,R_{\mathrm{goal}}^i)$ denote the raw performance score defined in Equation (7), and let ${\tilde{f}}_i=\max (0,f_i)$ denote the corresponding safeguarded score. The allocation probability $p_i$ for source task i is defined as

$$p_i=\left\{\begin{array}{cc}{\displaystyle \frac{1}{n}},\hfill & \mathrm{if}{\displaystyle {\sum }_{k=1}^n}{\tilde{f}}_k\le \epsilon ,\hfill \\ {\displaystyle \frac{{\tilde{f}}_i}{{\sum }_{k=1}^n{\tilde{f}}_k}},\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(6)

where $\epsilon ={10}^{-8}$. The allocation probability $p_i$ determines the share of the fixed per-iteration exploration budget assigned to source task i and is used for deterministic budget allocation rather than for randomly selecting a source task. The raw performance score is defined as

$$f(R_{\mathrm{collision}}^i,R_{\mathrm{goal}}^i)=\beta R_{\mathrm{collision}}^i+(1-\beta )(1-R_{\mathrm{goal}}^i).$$

(7)

The lower bound is applied to the composite score rather than to $R_{\mathrm{goal}}^i$ itself. Therefore, when the raw score $f_i$ is positive, the relative weighting between collision and goal-progress performance is retained, whereas a non-positive raw score cannot produce a negative allocation weight. If the sum of the safeguarded scores is zero or sufficiently small, the first branch of Equation (6) restores uniform allocation and prevents division by zero.

Thus, source tasks with poorer collision-avoidance or goal-progress performance generally receive a larger share of the exploration budget in subsequent iterations. Uniform allocation would continue to assign the same share of the exploration budget to every source task regardless of differences in current performance. In contrast, the adaptive rule is designed to redistribute the fixed exploration budget according to the current performance of the shared Q-network.

At initialization, equal allocation is used because no relative performance information is yet available. As pretraining proceeds, the allocation responds to emerging performance differences among the source tasks. A task that performs relatively well tends to receive a smaller share, whereas a task whose performance remains poor or subsequently deteriorates can receive a larger share again. Unlike a fixed curriculum, no predetermined task order is imposed; any source task can be re-emphasized in a later iteration according to the current performance of the shared network. Pretraining terminates when all source tasks satisfy $R_{\mathrm{goal}}^i\ge \tau$ or when the maximum number of iterations is reached.

3.3. Transfer to Multi-Agent Learning

After pretraining, the same Q-network $Q_{{\theta }^{\mathrm{tr}}}$ is copied to every pedestrian agent in a target environment. The network architecture and target-task learning algorithm are otherwise unchanged. Each agent then continues learning using standard DQN. The transfer therefore affects only the initial network parameters, making it possible to isolate the effect of the single-agent knowledge on subsequent multi-agent learning.

4. Simulation Experiments

4.1. Experimental Setup

4.1.1. Pedestrian Agent and Learning Parameters

Experiments were conducted in a two-dimensional virtual environment. Each pedestrian is represented as a circle with a radius of $0.3$ m. At each step, the agent selects a combination of speed and direction. Six speeds, $(1.8,1.44,1.08,0.72,0.36,0)$ m/s, and seven relative directions, $(\pm 3\pi /4,\pm \pi /2,\pm \pi /4,0)$ rad, yield 42 discrete actions.

The observation vector contains 25 variables: goal progress, the relative angle to the goal, changes in speed, direction, and goal distance, and visual information about walls and other agents. The visual field extends ±80° from the current heading up to 3 m and is divided into angular sectors. The reward includes $+100$ for reaching the goal, the change in distance to the goal, a $-0.1$ step penalty, and a $-1$ collision penalty. An episode terminates when the goal is reached or the episode length exceeds 100 steps.

The goal-progress variable is defined as the signed displacement along the initial goal direction, normalized by the initial distance to the goal:

$$r_t^{\left(\mathrm{goal}\right)}={\displaystyle \frac{{\mathbf{g}}_0·({\mathbf{p}}_t-{\mathbf{p}}_0)}{\parallel {\mathbf{g}}_0{\parallel }^2}},$$

(8)

where ${\mathbf{p}}_0$ and ${\mathbf{p}}_t$ denote the initial and current pedestrian positions, respectively, and ${\mathbf{g}}_0$ is the vector from the initial position to the goal. Accordingly, $r_t^{\left(\mathrm{goal}\right)}=0$ at the initial position and $r_t^{\left(\mathrm{goal}\right)}=1$ at the goal position. The variable is not clipped to the interval $[0,1]$; it can become negative when the pedestrian displacement has a component opposite to the initial goal direction and can exceed 1 when the signed projected displacement exceeds the initial distance to the goal. In source-task evaluation, $R_{\mathrm{goal}}$ in Equation (5) is the mean of the final $r_t^{\left(\mathrm{goal}\right)}$ values over $n_{\mathrm{eval}}$ evaluation rollouts. The pretraining termination threshold is set to $\tau =0.8$; therefore, early termination requires every source task to attain an average final goal-progress value of at least 0.8.

The main learning and pretraining parameters are summarized in

Table 1. Main learning and pretraining parameters.

Parameter	Value
Maximum pretraining iterations	100
New experiences collected per pretraining iteration (N)	2000
CQL regularization parameter ($\alpha$)	1.0
Gradient updates per pretraining iteration (K)	100
Evaluation rollouts per source task ($n_{\mathrm{eval}}$)	100
Source-task weighting parameter ($\beta$)	0.5
Pretraining termination threshold ($\tau$)	0.8
Maximum target-task episodes	2000
Exploration rate ($ϵ$)	0.1
Discount factor ($\gamma$)	0.98
Learning rate	0.0001
Initial random exploration	1000 steps
Replay-buffer size	500,000
Minibatch size	128
Target-network update frequency	100 steps

.

The experiments were performed on a Windows 11 Home workstation with an Intel Core i7-14700 CPU, 32 GB RAM, and an NVIDIA GeForce RTX 4070 Ti GPU.

4.1.2. Source and Target Tasks

Four single-agent source tasks were used: O-shaped, U-shaped, S-shaped, and multi-pillar environments. They were designed to require different combinations of detouring and static-obstacle avoidance while sharing the same fundamental goal-reaching objective.

Four multi-agent target tasks were considered: intersection, bi-flow, bi-door, and bottleneck. Each target environment contains eight agents. The intersection task requires two groups to cross in the center of the environment; the bi-flow task requires two groups to pass each other while moving in opposite directions; the bi-door task requires route selection through one of two openings; and the bottleneck task requires agents to coordinate while passing through a narrow passage.

For each target task, DQN training with the pretrained initialization was compared with DQN training from random initialization. Each condition was trained for 2000 episodes over 15 independent trials. During learning, wall collisions, collisions with other agents, steps required to reach the goal, goal-arrival ratio, and episode reward were recorded.

The four single-agent source tasks are shown in Figure 2.

The four multi-agent target tasks are shown in Figure 3.

4.2. Source-Task Pretraining

Pretraining terminated at iteration 30 because all four source tasks exceeded the goal-progress threshold $\tau =0.8$.

Table 2. Performance of the shared Q-network at the end of pretraining.

Source Task	Goal Progress ${\mathit{R}}_{\mathbf{goal}}$	Wall-Collision Rate ${\mathit{R}}_{\mathbf{collision}}$
O-shaped	0.991	0.049
S-shaped	0.980	0.055
U-shaped	1.016	0.070
Multi-pillar	0.963	0.125

shows the final source-task evaluation. All source environments achieved high goal-progress scores and comparatively low wall-collision rates, indicating that the shared network had acquired basic goal-directed and obstacle-avoidance behavior before transfer.

Source-task performance did not improve monotonically. Because the allocation was recomputed after every evaluation, a task with relatively good current performance could receive a smaller share of the subsequent exploration budget and later receive a larger share again if its performance deteriorated relative to the other tasks. Figure 4c illustrates this repeated reallocation during pretraining. The adaptive rule therefore does not enforce monotonic improvement or a predetermined progression through the source tasks; instead, the allocation is repeatedly revised according to the current performance of the shared Q-network.

Because the goal-progress variable is not bounded above, $1-R_{\mathrm{goal}}$ can become negative. We therefore examined the raw allocation scores over the complete pretraining run. Across the 31 evaluation points and four source tasks, corresponding to 124 task–iteration pairs, $1-R_{\mathrm{goal}}$ was negative in 11 pairs, with $R_{\mathrm{goal}}$ reaching a maximum value of 1.039. Nevertheless, the composite raw score $f_i$ remained strictly positive in every pair, with a minimum value of $1.0\times {10}^{-3}$. Consequently, the non-negativity safeguard in Equation (6) would have remained inactive for every recorded allocation, i.e., ${\tilde{f}}_i=f_i$ throughout the reported pretraining run. The allocation sequence used in the reported experiments is therefore identical under the safeguarded formulation. The uniform-allocation branch in Equation (6) was also never activated in the reported run.

4.3. Target-Task Learning Performance

In the intersection environment, the transferred agents showed fewer wall collisions, fewer collisions with other agents, fewer steps to the goal, a higher goal-arrival ratio, and a higher reward during approximately the first 250 episodes. The two methods approached similar performance by the end of 2000 episodes. This indicates that the main effect of transfer is improved early-stage learning rather than a large change in asymptotic performance.

The bi-flow and bi-door environments showed the same overall tendency: the proposed initialization improved performance immediately after learning began, whereas both methods converged to similar final performance. In contrast, the bottleneck task did not show a clear improvement in the initial learning phase. Both methods converged to a goal-arrival ratio of approximately 92%, indicating that some agents did not consistently learn to reach the goal in this highly interactive environment.

The episode-reward learning curves for the four target tasks are shown in Figure 5.

To quantify overall transfer efficacy over the complete learning curve, we use the normalized AUC difference corresponding to the transfer-ratio formulation described by Taylor and Stone [21]:

$$t_e={\displaystyle \frac{{\mathrm{AUC}}_{\mathrm{transfer}}-{\mathrm{AUC}}_{\mathrm{no}\mathrm{transfer}}}{{\mathrm{AUC}}_{\mathrm{no}\mathrm{transfer}}}}.$$

(9)

Table 3. Transfer efficacy based on episode reward.

Target Task	Transfer Efficacy ${\mathit{t}}_{\mathit{e}}$
Intersection	0.0231
Bi-flow	0.0198
Bi-door	0.0285
Bottleneck	0.0233

shows positive transfer efficacy for all four target tasks. In the bottleneck environment, the positive value is mainly associated with improved reward during the middle portion of training rather than a clear acceleration at the beginning or a higher final performance.

4.4. Target-Task Training Time

Target-task training time was measured for 15 independent trials in each target task. The measurement starts after network initialization and covers only the DQN training performed in the target environment; the one-time source-task pretraining cost is not included in

Table 4. Target-task training time and one-sided Mann–Whitney U-test results ($n=15$). Values are mean ± standard deviation in seconds. The one-time source-task pretraining cost is not included.

Target Task	No Pretraining	Proposed Method	p-Value
Intersection	$2599.92\pm 377.53$	$2242.32\pm 258.95$	$2.73\times {10}^{-3}$
Bi-flow	$1284.81\pm 85.79$	$1069.46\pm 79.30$	$6.64\times {10}^{-6}$
Bi-door	$2884.31\pm 188.89$	$2498.15\pm 260.26$	$1.54\times {10}^{-4}$
Bottleneck	$5234.22\pm 2916.51$	$4703.32\pm 3147.93$	$1.70\times {10}^{-1}$

. This follows the target-task-time scenario distinguished by Taylor and Stone [21], in which previously acquired source knowledge is treated as reusable knowledge and evaluation focuses on the time needed to learn a new target task. Since the same pretrained Q-network can be reused across all four target tasks, the reported values quantify the adaptation cost for each target environment, not the total cost of first pretraining the source network and then adapting it. A one-sided Mann–Whitney U test at a significance level of 0.05 was used because the proposed method was hypothesized to reduce target-task training time. Table 4 shows that the mean target-task training time was lower with pretraining in all four tasks. Statistically significant reductions were obtained in the intersection, bi-flow, and bi-door tasks. The bottleneck task showed a lower mean but a large variance and no statistically significant difference.

These results indicate that the single-agent initialization can translate into a measurable reduction in target-task multi-agent training cost when early navigation knowledge remains useful in the target environment. The lack of significance in the bottleneck task is consistent with the stronger dependence of this environment on interactions among simultaneously learning agents.

4.5. Behavioral Analysis of Transferred Knowledge

4.5.1. Spatial Exploration During Early Learning

To examine why pretraining improves early target-task performance, we analyzed the regions visited by agents during the first 500 episodes. For each method, the coordinates of all eight agents were recorded over five independent trials. Each spatial axis was divided into 50 intervals, producing a $50\times 50$ grid. The mean number of visits to each grid cell was calculated over the five trials. We then computed the difference between the proposed method and the no-pretraining baseline. Positive differences indicate cells visited more frequently by transferred agents, whereas negative differences indicate cells visited more frequently by agents trained from scratch.

The 500-episode interval was divided into five windows of 100 episodes. The strongest and most interpretable difference was observed in the bi-door task. During episodes 1–100, transferred agents visited the goal-side region more frequently, while baseline agents spent relatively more time on the starting side of the wall. During episodes 101–200, the proposed method showed greater visitation around the two passable door regions and beyond the wall, whereas the baseline showed greater visitation near wall regions. The difference gradually decreased over episodes 201–500, consistent with the baseline agents gradually acquiring similar route-selection behavior.

To complement the spatial heatmaps with quantitative measures, we analyzed early navigation behavior using the same five independent bi-door trials. Episodes 1–100 and 101–200 were evaluated separately, matching the first two heatmap windows. Metrics were first calculated within each trial, and the five trial-level values were summarized as mean ± standard deviation. Goal-arrival ratio was calculated over all agent–episodes, whereas steps to goal and recorded trajectory length were calculated for successful agent–episodes. The recorded trajectory length was defined as the sum of Euclidean distances between consecutive valid logged positions, with terminal sentinel coordinates excluded. Because the trajectory logs do not contain the step-0 coordinate or the physical terminal goal coordinate, this measure represents an approximate recorded path length and is used for relative comparison between conditions.

The corresponding quantitative results are summarized in

Table 5. Quantitative early-learning behavior in the bi-door task using the same five independent trials as the spatial heatmaps. Values are mean ± standard deviation of trial-level metrics.

Metric	Episodes 1–100		Episodes 101–200
	Pretraining	No Pretraining	Pretraining	No Pretraining
Goal-arrival ratio	$0.532\pm 0.065$	$0.336\pm 0.037$	$0.954\pm 0.016$	$0.847\pm 0.018$
Steps to goal	$49.82\pm 2.05$	$56.39\pm 1.56$	$30.39\pm 1.03$	$38.58\pm 1.46$
Recorded trajectory length	$40.25\pm 1.07$	$45.51\pm 1.12$	$27.58\pm 0.46$	$32.95\pm 0.57$

Steps to goal and recorded trajectory length are calculated for successful agent–episodes.

.

The quantitative measures are consistent with the heatmap-based interpretation of accelerated early navigation. During Episodes 1–100, transferred agents reached the goal more frequently and required fewer steps and shorter recorded trajectories when successful. These differences remained evident during Episodes 101–200, when the goal-arrival ratio exceeded 0.95 with pretraining and the mean steps to goal and recorded trajectory length remained lower than under random initialization. These results provide a quantitative complement to the spatial visitation patterns in Figure 6, while the heatmaps retain the role of identifying where the differences in early exploration occurred.

The intersection task also showed that transferred agents visited the goal-side region more frequently during episodes 1–100, although the constrained geometry made later differences more difficult to interpret. In the bi-flow task, the open geometry produced less localized differences in the heatmaps. These observations suggest that the value of the spatial diagnostic depends on the structure of the target environment.

The bi-door analysis provides a behavioral interpretation of the reduction in target-task training time. The spatial heatmaps and quantitative measures indicate that transferred agents collect different experiences during the critical early phase of learning, including more frequent goal attainment and shorter successful trajectories. These patterns are consistent with the transferred initialization providing useful goal-directed and static-obstacle-avoidance behavior, allowing experiences relevant to successful route selection to be accumulated earlier than under random initialization.

4.5.2. Post-Training Pedestrian Trajectories

After target-task training, greedy policies were used to visualize pedestrian trajectories. In all four environments, trained agents generally reached their goals while avoiding walls and other pedestrians. The resulting routes were not identical across agents, indicating that the independently learned policies produced diverse avoidance and route-selection patterns.

In the bi-flow environment, agents moving in one direction often followed relatively direct routes, whereas agents moving in the opposite direction sometimes used detours to avoid oncoming pedestrians. In the bottleneck environment, successful trials showed agents following preceding pedestrians through the narrow passage, producing an emergent coordinated flow without explicitly programming a following rule.

Representative post-training pedestrian trajectories are shown in Figure 7.

The bottleneck environment also revealed an important failure mode. In some trials, one or more agents remained near their initial positions and failed to acquire an appropriate policy. A representative example is shown in Figure 8.

This phenomenon was observed both with and without pretraining, indicating that it is not specific to the transfer mechanism. Non-stationarity in independent multi-agent learning is one plausible mechanism. In independent Q-learning with experience replay, changes in other agents’ policies can make older replayed experiences less representative of the current interaction dynamics, which has been identified as a known difficulty in deep multi-agent reinforcement learning [28,29]. However, the present experiments did not quantify policy divergence among agents or replay-buffer staleness. We therefore do not identify non-stationarity as the demonstrated cause of the observed bottleneck failures; it is treated only as a plausible interpretation consistent with the observed learning instability and prior literature.

4.6. Discussion

4.6.1. Mechanism of Learning Acceleration and Transferable Knowledge

The experiments demonstrate that single-agent pretraining can provide useful initialization for multi-agent pedestrian learning when the target task still relies heavily on goal-directed movement and static-obstacle avoidance. The most consistent improvements were observed in the intersection, bi-flow, and bi-door environments. In these tasks, the transferred network improved the early learning curves and significantly reduced target-task training time, while final performance remained similar to training from scratch. Thus, the main observed benefit is improved sample efficiency and computational efficiency during target-task adaptation rather than a fundamentally different final policy quality.

The exploration analysis helps explain this effect at the behavioral level. In particular, the bi-door heatmaps show that the transferred agents reach the door and goal-side regions earlier. This makes the transfer mechanism more interpretable than a comparison based only on final reward or training time. The analysis identifies a concrete behavioral pathway: single-agent pretraining creates a navigation-oriented initial policy; the initial policy changes early state visitation; the changed visitation exposes the agents to task-relevant regions earlier; and those experiences accelerate subsequent target-task learning.

The strength of the proposed framework should therefore be interpreted as a combination of transfer design and empirical explanation rather than as a new reinforcement-learning optimizer. Prior pedestrian RL studies have already used knowledge transfer or curriculum learning [12,17]. The distinctive aspect examined here is the deliberate separation of single-agent navigation knowledge from multi-agent interaction learning, together with adaptive allocation of source-task experience and a direct behavioral analysis of what changes after transfer. This formulation makes it possible to state not only whether transfer improves learning, but also what is transferred successfully—goal-directed motion and static-obstacle avoidance—and where that knowledge becomes insufficient when interaction-specific behavior dominates.

4.6.2. Limitation in Interaction-Dominated Environments

The preceding interpretation also clarifies the limitation observed in the bottleneck environment. The source tasks contain no inter-agent interaction, and therefore the transferred network provides little direct knowledge about yielding, following, or resolving competition for a narrow passage. The bottleneck task is consequently dominated by behavior that must still be learned in the target environment. The high variance in target-task training time and occasional learning stagnation indicate that this interaction-dominated task remains difficult under independent learning. Because policy divergence and replay-buffer staleness were not quantified, however, the present experiments do not determine whether non-stationarity is the specific mechanism responsible for these failures.

4.6.3. Interpretability, Limitations, and Future Work

From an explainable-AI perspective, the present analysis focuses on behavioral transparency rather than internal neural-network attribution. We do not claim that the Q-network itself is globally explainable. Instead, we make the effect of transfer observable through learning dynamics, spatial visitation, collision behavior, goal-directed exploration, and post-training trajectories. These diagnostics are particularly suitable for pedestrian simulation because the policy ultimately acts in a physical two-dimensional space in which state visitation and trajectories can be inspected directly. Future work could complement the present behavior-level analysis with value-function attribution or sensitivity methods to connect internal network representations to specific navigation decisions.

Several limitations should be noted. First, all target tasks contain exactly eight agents. Thus, the present evaluation is limited to a fixed crowd size, and scalability to larger agent populations or higher crowd densities has not yet been evaluated. Future work should examine whether the observed transfer effects persist as the number and density of interacting agents increase. Second, the baseline is DQN training from random initialization. This matched comparison isolates the effect of transferred initialization while keeping the target-task network architecture and DQN learning procedure unchanged. Therefore, the present results do not establish superiority over curriculum-based training, other transfer-learning strategies, or alternative multi-agent RL methods, and broader baseline comparisons remain for future work. Third, the contribution of adaptive source-task allocation has not been isolated through an ablation against uniform allocation. Therefore, the present results should be interpreted as evidence for the complete source-task pretraining and transfer framework rather than as evidence that adaptive allocation itself outperforms uniform allocation. A controlled comparison using the same source tasks, total exploration budget, update procedure, and transfer setting while replacing the adaptive rule with uniform allocation is required to quantify the independent contribution of this component. The sensitivity to $\beta$ and other pretraining parameters also remains to be studied. In addition, although CQL is used as a conservative regularizer during source-task pretraining, its incremental benefit over standard DQN updates has not been isolated in the present experiments. CQL should therefore be regarded here as a design choice for source-task pretraining rather than as an independently validated source of the observed transfer gains. Fourth, the source tasks include only single-agent navigation and therefore cannot supply interaction-specific knowledge. Finally, the exploration analysis uses state visitation as a behavioral diagnostic and does not directly identify the internal features used by the Q-network.

These limitations suggest several extensions. More diverse source tasks could include controlled interaction with other pedestrians while retaining the computational advantage of reusable pretraining. Knowledge-sharing mechanisms or centralized-training approaches could be introduced to reduce non-stationarity in the target task [28]. Ablation studies comparing adaptive and uniform source-task allocation, together with analyses of source-task diversity, termination criteria, and transfer timing, would clarify which components are responsible for the observed efficiency gains.

5. Conclusions

This paper analyzed a two-stage transfer-learning framework for DQN-based pedestrian agents in which a shared Q-network is first pretrained on multiple single-agent navigation tasks and then used to initialize agents in multi-agent pedestrian environments. The transferred initialization improved early-stage performance in the intersection, bi-flow, and bi-door tasks and significantly reduced target-task training time in these three environments. The bottleneck task showed no statistically significant reduction in target-task training time and no clear improvement in the initial learning phase, although harmful negative transfer was not observed under the present evaluation.

Beyond aggregate learning performance, the behavioral analyses clarified how transfer affects policy acquisition. In the bi-door environment, transferred agents explored goal-side and door regions earlier, indicating that pretraining changes the distribution of early experience. Post-training trajectory analysis further demonstrated collision avoidance and diverse route choices, while the bottleneck failure cases exposed the limitations of transferring knowledge learned without inter-agent interaction. The main contribution is therefore not merely the use of transfer learning, but the separation and reuse of basic navigation knowledge across single- and multi-agent settings and the behavioral identification of the mechanism and boundary conditions of that reuse. Overall, the results show that reusable single-agent navigation knowledge can improve target-task learning efficiency in multi-agent pedestrian simulation and that spatial exploration and trajectory analysis provide useful behavioral explanations of when the transfer is effective.