Mastering the Twin–Game: Hierarchical Reinforcement Learning in a Digital Twin Sandbox for Adaptive Urban Healthcare Optimization—A Case Study of Wuhan

Abstract

Urban healthcare systems are fundamentally constrained by the mismatch between static resource configurations and dynamically evolving patient demand. Under the tiered healthcare system, traditional static planning methods struggle to capture the complexity and randomness of patient flows. While recent reinforcement learning (RL) approaches enable adaptive decision-making, they suffer from dimensionality explosion and unstable convergence due to massive action spaces and delayed spatiotemporal credit assignment in city-scale environments. To address this gap, we propose Twin–Game: a digital twin-driven hierarchical reinforcement learning (HRL) framework that formulates adaptive healthcare resource optimization as a “Twin Game” between a simulation-based game environment (Strategic Sandbox) and a hierarchical decision policy. First, we construct the “first twin”—an offline digital twin that serves as the Strategic Sandbox parameterized with Wuhan’s observed facility, population, and transportation data, while patient arrivals and disease profiles are generated synthetically under documented assumptions because individual-level clinical flow data are not publicly available. This environment integrates a dynamic gravity model with a two-way referral mechanism to represent the nonlinear coupling between hospital attractiveness, crowding levels, and patient choice behaviors. Second, we build the “second twin”—an Option-based HRL policy. The Manager (Macro-level Strategic Layer) uses a Deep Q-Network (DQN) for discrete spatial attention allocation; the Worker (Micro-level Execution Layer) uses Proximal Policy Optimization (PPO) for continuous, fine-grained controls such as bed expansion ratios and personnel scheduling. The two twins interact in a closed-loop game, performing strategy search and game evolution under complex constraints to optimize allocation. Experimental results from the Wuhan case indicate that the Twin–Game framework outperforms static baselines and single-layer RL in reducing average travel times, enhancing resource utilization, and improving tiered diagnosis and treatment within the simulation setting. The results should be interpreted as simulation-based decision-support evidence rather than direct clinical validation. This study provides a data-driven, game-theoretic decision support tool for building resilient urban healthcare systems.

1. Introduction

With rapid urbanization and the global shift toward aging societies, urban healthcare systems face a dual challenge: a surge in service demand and a persistent structural mismatch in resource allocation [1,2,3]. A common manifestation of this imbalance is the polarization of healthcare utilization, where high-level hospitals are chronically overloaded, while primary care institutions remain underutilized. This phenomenon reflects deeper systemic issues in hierarchical healthcare delivery, where patient choices, service capacity, and spatial accessibility interact in complex and dynamic ways.

To mitigate these challenges, many countries—including China—have introduced tiered healthcare systems aimed at redistributing patient flows across different levels of care [4]. Nevertheless, the effective coordination of such multi-tier systems remains a formidable challenge. Patient choices are stochastic and heterogeneously influenced; transportation reshapes accessibility and supply evolves over time. This interplay yields a highly nonlinear, dynamic system, rendering traditional planning inadequate.

The spatial optimization of healthcare facilities has long been central to medical geography and urban planning. Foundational studies include the maximal covering location problem proposed by Church and ReVelle [5] and classical location–allocation models [6]. Subsequent research has expanded these approaches by incorporating accessibility measures [7], dynamic catchment areas [8], and equity-oriented analyses [9,10,11,12,13,14,15]. Despite these advances, most existing methods rely on static assumptions or single-level formulations, which are insufficient for capturing the hierarchical structure and dynamic interactions inherent in tiered healthcare systems [2,16]. In particular, the coordination of optimization across multiple service tiers remains an underexplored field, especially within policy contexts such as tiered diagnosis and treatment [1,4,17,18].

Reinforcement learning (RL), particularly its deep variants (Deep Reinforcement Learning, DRL), provides a promising paradigm for sequential decision-making under uncertainty. Recent studies have demonstrated the effectiveness of DRL in healthcare and urban systems, including medical imaging workflows [19], emergency resource allocation [20], traffic signal control [21], mobility systems [22,23], and spatial optimization problems such as facility location and coverage maximization [24,25,26,27]. However, applying DRL to city-scale hierarchical healthcare systems remains challenging. The problem involves mixed discrete–continuous decision spaces and long-horizon spatiotemporal dependencies, leading to severe credit assignment difficulties and the well-known issue of dimensionality explosion. Consequently, conventional single-level DRL approaches often suffer from instability and limited scalability.

Hierarchical deep reinforcement learning (HDRL) provides a natural solution by introducing temporal abstraction through the Option framework [28,29,30]. Recent advancements, including Option-Critic architectures [31], PPO-based hierarchical policies [32,33], DQN-style control [34], and data-efficient hierarchical variants [35], have significantly improved learning stability and scalability. The decoupling of high-level strategic decisions from low-level execution renders HDRL particularly suitable for addressing large-scale, structured optimization problems.

Despite these advances, a unified framework that simultaneously captures hierarchical system dynamics, represents patient-flow mechanisms under explicit simulation assumptions, and supports learning-based decision-making remains largely absent. Recent urban digital-twin research further highlights the value of closed-loop simulation platforms for lifecycle management and policy evaluation [36]. To bridge this gap, this study reformulates urban healthcare resource optimization as a Twin–Game, a closed-loop interaction between a simulation-based environment model and a learning-based decision agent. Specifically, we construct an Environment Twin, which simulates the complex dynamics of urban healthcare systems through a dynamic gravity model and bidirectional referral mechanisms, and a Strategy Twin, implemented as a hierarchical deep reinforcement learning (HDRL) agent that learns adaptive resource allocation strategies within this environment. Through this interaction, the framework enables systematic exploration of trade-offs between efficiency and equity, ultimately seeking Pareto-oriented solutions rather than claiming a directly observed Pareto frontier.

We test the proposed framework using Wuhan, a major medical hub in central China [37], as a real-world urban case whose facility locations, population distribution, administrative context, and travel impedance are observed. Because patient-level clinical flows and disease incidence records were unavailable, the evaluation is a calibrated simulation rather than a full empirical validation. The results demonstrate that the proposed approach can improve both system efficiency and spatial equity within the Wuhan sandbox. Therefore, claims about broader applicability are limited to methodological transferability and require further cross-city validation. The main contributions of this study are summarized as follows.

(1)

Environment Twin: A simulation-based digital twin is constructed as a Strategic Sandbox by integrating a dynamic gravity model with a bidirectional referral mechanism, capturing the coupling between patient attraction and system congestion under explicit assumptions.

(2)

Strategy Twin: A Twin–Game framework based on hierarchical deep reinforcement learning is proposed, where a DQN-based manager performs high-level spatial decision-making and PPO-based workers handle fine-grained resource allocation, effectively addressing large-scale spatiotemporal optimization challenges.

(3)

Case-based Simulation Evaluation: The framework is evaluated in a Wuhan digital twin parameterized by observed facility, population, and transport data, with simulated patient demand used to test efficiency, equity, and tiered-care outcomes under controlled assumptions.

2. Study Area and Data Overview

2.1. Overview of the Study Area

Wuhan, capital of Hubei Province and central China’s largest city, is the only sub-provincial city among the six central provinces [3]. The city administers 13 districts, covering a total area of 8569.15 square kilometers, with a permanent population of approximately 13.8 million [3]. As of late 2024, the city operated 7049 medical institutions—85 tertiary hospitals, 98 secondary hospitals, 6531 primary healthcare institutions, 80 specialized public health institutions, and 65 other facilities—collectively providing 106,400 beds, or 7.61 beds per thousand residents [37]. However, the healthcare landscape is characterized by centripetal clustering: premium resources are concentrated in core districts such as Wuchang and Jianghan, driving irrational patient flows and leaving primary care utilization rates below 50% [37,38].

The geographic distribution of Wuhan’s healthcare resources exhibits pronounced centripetal clustering, with core urban districts like Wuchang, Jianghan, and Qiaokou dominating access to premium resources [38]. This spatial polarization, coupled with resource scarcity in outlying districts, directly fuels irrational patient behavior in healthcare seeking. Figure 1 shows the geographic scope and administrative divisions of Wuhan City. Currently, tertiary hospitals in central districts face systemic risks due to prolonged overcapacity, while utilization rates at primary-level institutions remain below 50%, highlighting resource idleness and waste. Therefore, addressing spatial mismatches and establishing balanced tiered healthcare systems have become urgent priorities for enhancing urban public health resilience [39,40]—a critical challenge shared by major cities nationwide.

2.2. Data Sources and Processing

Table 1. Data sources used in this study.

Dataset	Source/Content
Medical institution data	Matched official records from the Wuhan Municipal Health Commission with Amap POI; includes names, addresses, and classification levels of all medical institutions in Wuhan (as of 2024).
Population distribution data	LandScan 2024 population raster (5-arcsecond); includes gridded population counts and demographic attributes (age structure, gender ratios, etc.).
Patient flow data	Daily synthetic patient flow generated by the demand model and anchored to Wuhan’s population distribution and aggregate healthcare statistics; used for simulation experiments rather than direct empirical validation.
Transportation network data	OpenStreetMap road network + AutoNavi Map API travel times (multi-period); used to build the citywide accessibility matrix.
Disease spectrum data	Simulation-generated disease incidence profiles based on aggregate disease-spectrum proportions and four-level severity categories; individual-level clinical records were not used.

summarizes the data sources used in this study, and

Table 2. Data preprocessing workflow.

Step	Description
Geocoding	Converted addresses of medical institutions and residential areas to WGS84 coordinates.
Data cleaning	Removed missing values and outliers.
Spatial partitioning	Divided Wuhan into $500\mathrm{m}\times 500\mathrm{m}$ grids (34,277 cells) as the baseline analytical unit; alternative grid sizes are examined in the MAUP sensitivity test.
Feature extraction	Derived indicators for each medical institution (e.g., service capacity).

lists the preprocessing workflow.

It should be emphasized that the observed datasets and the simulated datasets play different roles. Official facility records, LandScan population grids, administrative statistics, and road-network travel times provide the empirical spatial structure of the Wuhan case. In contrast, daily patient flows and disease incidence profiles are generated within the digital twin because privacy-protected individual-level hospital visit trajectories and diagnosis records were not accessible. Accordingly, this study evaluates whether the proposed decision framework performs well under a transparent and reproducible Wuhan-parameterized simulation, but it does not claim direct validation against observed individual patient trajectories.

Hospital inclusion criteria. Not every point labelled as a medical POI was included in the optimization set. Facilities were retained only when they met the following criteria: (1) the institution appeared in official Wuhan Municipal Health Commission records or could be matched reliably to a verified Amap POI; (2) its geographic coordinates and administrative district were available after geocoding; (3) its service tier could be identified as tertiary hospital, secondary/general hospital, or primary/community health service institution; and (4) basic capacity information, such as beds or an equivalent service-capacity proxy, could be assigned. Duplicated POIs, non-clinical administrative offices, pharmacies, laboratories without outpatient service, and records with unresolved locations were excluded. The final facility set therefore represents clinically relevant service providers rather than all healthcare-related POIs.

The $500\mathrm{m}\times 500\mathrm{m}$ grid was selected as a compromise between spatial detail and computational tractability. It is finer than most subdistrict-level planning units and is comparable to a walkable neighborhood scale, while keeping the state and action representations manageable for city-scale HRL training. Because results in spatial analysis may be affected by the modifiable areal unit problem (MAUP), Section 5.6 reports a grid-size sensitivity analysis using coarser and finer alternatives.

2.3. Analysis of Wuhan’s Healthcare Facility Landscape

We categorized the 7049 healthcare institutions in Wuhan into three tiers—tertiary hospitals, general hospitals, and community health service centers—to assess their influence. Our kernel density analysis (Figure 2) reveals three core “game imbalances” in Wuhan.

Through further data investigation, we identified the following characteristics of Wuhan’s medical facilities.

(1) Spatial Polarization: Medical resources exhibit a distinct “center-periphery” structure with a vacuum in peripheral zones. High-quality tertiary hospitals are highly concentrated in the city’s core areas, forming powerful medical hubs. In contrast, outlying districts and emerging population centers face a scarcity of quality resources. Residents in these areas must engage in long-distance “healthcare seeking games” to access quality care, which not only increases healthcare costs but also exacerbates traffic congestion and medical pressure in central districts.

(2) Structural “Goliath” Effect: Institutional setup exhibits a pronounced top-heavy imbalance. A few massive tertiary hospitals monopolize the vast majority of beds and hardware resources, boasting massive scale and comprehensive functions. In contrast, most primary-level institutions, such as community health service centers, appear underpowered and under-resourced. This structure renders primary-level facilities incapable of fulfilling their role as “health gatekeepers,” failing to effectively divert patients with common and chronic conditions.

(3) Efficiency Deadlock: This uneven distribution directly causes extreme disparities in resource utilization efficiency. Tertiary hospitals in central urban areas operate in a state of chronic overload, with bed shortages, immense pressure on medical staff, and potential risks to healthcare quality. Meanwhile, secondary hospitals and primary institutions in outlying districts face resource idleness, low bed occupancy rates, and underutilized service capacities, resulting in significant wasteful allocation. The system is thus locked in an efficiency deadlock between overload in the center and idleness in the periphery.

This “spatial concentration, structural imbalance, and efficiency disparity” is the core challenge and must be addressed to advance tiered diagnosis and resource decentralization.

Figure 3 outlines the pipeline: Data Layer → first twin, digital twin with patient generation, congestion-aware choice, and referral logic → second twin, Twin–Game with Manager–Worker. Percentages shown in the figure such as 66.7% denote BTR; PCR and Gini are given in Table 6.

3. Methodology I: Modeling the Game Environment with an Urban Healthcare Digital Twin

To describe the urban healthcare digital environment, we modeled patient demand, healthcare settings, and treatment processes in Wuhan City under explicit simulation assumptions. This resulted in the construction of a simulation-based digital twin that serves as the Strategic Sandbox for Wuhan’s urban healthcare system, enabling policy evaluation and resource allocation decisions [41,42,43]. Spatiotemporal big data platforms and urban simulation frameworks provide foundational support for this integrative modeling approach [44,45]. Figure 4 illustrates the architecture of this first twin (the Strategic Sandbox).

3.1. Spatiotemporal Demand Generation

To represent the randomness and spatiotemporal heterogeneity of urban healthcare demand in the absence of individual-level visit records, we developed a patient demand generator based on a Non-homogeneous Poisson Process (NHPP). This choice is a parsimonious approximation commonly used for count arrivals. It captures first-order variation in population exposure and temporal seasonality, but it does not explicitly model contagion, behavioral feedback, socio-economic heterogeneity, or day-to-day dependence. Therefore, conclusions drawn from this component should be interpreted as scenario-based simulation results rather than forecasts of actual daily visits.

With the city divided into N grid regions, daily new patients in region i at step t follow:

$$N_i\left(t\right)\sim \mathrm{Poisson}\left({\lambda }_i\left(t\right)\right),{\lambda }_i\left(t\right)=P_i·\rho \left(t\right),$$

(1)

where $P_i$ is the regional population and $\rho \left(t\right)$ is the seasonality curve estimated from aggregated monthly outpatient statistics. Specifically, monthly outpatient counts were first normalized by their annual mean to obtain twelve relative intensity factors; these factors were then fitted with a periodic smoothing curve and linearly interpolated to the daily simulation step. The aggregate intensity is scaled so that the citywide daily volume is consistent with reported annual outpatient levels, and spatial heterogeneity is introduced through the population surface. More complex demand generators, such as negative-binomial arrivals, agent-based health-seeking models, or models conditioned on age and income, are reserved for future work when more detailed data become available.

(1) Patient Arrival Generation: The NHPP component first generates the number of new patients in each grid and time step according to Equation (1), thereby creating the spatiotemporal demand volume entering the healthcare system.

(2) Disease Attribute Sampling: For each generated patient p, we assign disease attribute tuples using Monte Carlo sampling $D_p=\langle \mathrm{Type},S_{\mathrm{sev}},\mathrm{Loc}\rangle$. Disease type is sampled according to aggregate disease-spectrum proportions rather than individual diagnosis records. Severity $S_{\mathrm{sev}}\in [0,1]$ is a normalized value used for subsequent referral threshold determination, following the four-level disease severity classification standard from emergency medicine [46];

Table 3. Four-level disease severity classification.

Level	Description	Clinical Criterion	${\mathit{S}}_{\mathbf{sev}}$
1	Life-threatening	Vital signs unstable, immediate resuscitation	0.9–1.0
2	Potentially life-threatening	Intervention within 10 min	0.7–0.9
3	Vital signs stable	Priority treatment in emergency department	0.4–0.7
4	Chronic or minor conditions	–	0–0.4

gives the four levels. Location is set to the geometric center corresponding to the grid cell i.

3.2. Coupled Patient Flow Dynamics

This serves as the core physical engine of the Strategic Sandbox. Unlike static planning models, patient flow within this environment depends not only on distance but also dynamically couples with the hospital’s real-time state, similar to cyberGIS-based urban analytics [47].

We propose the Congestion-Aware Dynamic Gravity Model [48]. For a patient residing in region i, the probability ${\mathrm{Prob}}_{ij}\left(t\right)$ of selecting healthcare facility j is defined as:

$${\mathrm{Prob}}_{ij}\left(t\right)={\displaystyle \frac{A_j\left(t\right)·D_{ij}^{-\beta }}{{\sum }_{k\in \mathcal{H}}{A}_k\left(t\right)·D_{ik}^{-\beta }}}$$

(2)

Here $D_{ij}$ denotes traffic impedance between region i and institution j, determined based on road-network travel time; $\beta$ is the distance decay exponent in the gravity model. Because complete historical origin–destination patient flows were unavailable, $\beta$ was not calibrated against individual patient trajectories. Instead, it was chosen from values commonly used in healthcare accessibility and gravity-model studies, then adjusted so that simulated catchment patterns were plausible relative to Wuhan’s observed center–periphery facility structure and checked through sensitivity analysis. $A_j\left(t\right)$ is dynamic attractiveness, a function of physical facilities and congestion levels.

The Coupling Mechanism: To simulate “crowd avoidance” behavior, the attractiveness $A_j\left(t\right)$ nonlinearly decays with the real-time crowding level $u_j\left(t\right)$ of the facility:

$$A_j\left(t\right)=C_{\mathrm{beds},j}·\left(1-\eta ·u_j\left(t\right)\right)$$

(3)

$$u_j\left(t\right)={\displaystyle \frac{L_j\left(t\right)}{C_{\mathrm{beds},j}}}$$

(4)

Here $C_{\mathrm{beds},j}$ is the total number of beds in facility j, a variable adjustable by the RL agent’s action $a_t$; $L_j\left(t\right)$ is the current number of people waiting in line; and $\eta$ is the crowding penalty factor. That is, when the RL agent increases the number of beds at a hospital, the hospital’s crowding decreases, its attractiveness increases, and the likelihood of diverting more patients increases. In-facility time $T_{\mathrm{wait}}^p$ for patient p is the sum of queueing delay and service duration; in the simulation it is determined by the chosen facility’s current queue length and service capacity.

3.3. Bidirectional Referral Logic

To simulate tiered healthcare policies, we embedded a rule-based state machine within the environment [49].

(1) Up-transfer logic: When a patient p arrives at the initial care facility j, an up-transfer is triggered if any of the following conditions is met:

$${\mathrm{Condition}}_{\mathrm{up}}:S_{\mathrm{sev},p}>{\theta }_{\mathrm{cap},j}\vee {\mathrm{Equip}}_{\mathrm{req},p}\notin E_j$$

(5)

The parameter ${\theta }_{\mathrm{cap},j}$ is the service capacity threshold of institution j; when $S_{\mathrm{sev},p}$ exceeds this value, an up-transfer is triggered (lower thresholds for primary care hospitals, higher for tertiary hospitals). The threshold values follow tiered-care policy logic and were set within clinically plausible ranges reported in the literature, then checked through sensitivity analysis. The set $E_j$ denotes facility j’s equipment set, temporarily represented by the number of available beds. Beds are used as a capacity proxy because consistent facility-level bed information is available for all hospitals, whereas detailed equipment inventories are incomplete in the public data. When the condition holds, the patient is redirected to the nearest eligible higher-level hospital $k=arg{min}_k{D}_{jk}$.

(2) Down-transfer logic: When a patient p at a higher-level hospital k enters the recovery phase, i.e., $S_{\mathrm{sev},p}<{\theta }_{\mathrm{stable}}$, and their local community center c has available resources, i.e., $L_c<C_{\mathrm{beds},c}$, a down-transfer is triggered.

Blocking and Penalty: If a “required-to-transfer-but-not-transferred” situation occurs due to bed unavailability at the higher-level hospital, the patient remains at the current tier, resulting in a negative reward for the system. This penalty is implicitly reflected in the accessibility reward and structural reward.

3.4. Multi-Objective Evaluation System

We evaluate the system along three dimensions: accessibility (average travel time and equity of care time distribution), efficiency (bed utilization level and spatial balance), and structure (tiered diagnosis and treatment flows). The corresponding symbols and formal mathematical definitions are given in

Table 4. Evaluation metrics and definitions.

Metric	Symbol	Definition
Average Medical Travel Time (AMT)	${\mathcal{M}}_{\mathrm{time}}$	Average travel time from a patient’s residence to the healthcare facility (minutes).
Healthcare Accessibility Gini (GAC)	${\mathcal{M}}_{\mathrm{gini}}$	Gini coefficient of total care time distribution; lower values indicate greater balance across regions.
Average Bed Utilization (ABU)	${\mathcal{M}}_{\mathrm{util}}$	Average utilization rate of hospital beds across all medical institutions in the city.
Resource Utilization Variance (RUV)	${\mathcal{M}}_{\mathrm{var}}$	Variance in bed utilization rates across healthcare institutions, reflecting balance of resource allocation.
Primary Care Rate (PCR)	$r_{\mathrm{primary}}$	Proportion of patients with common/frequent illnesses who choose community health centers for initial consultation.
Bidirectional Referral Rate (BTR)	$r_{\mathrm{referral}}$	Proportion of patients referred upward and downward relative to total outpatient visits.
Matching Score	${\mathcal{M}}_{\mathrm{match}}$	Tier–severity alignment: positive when severe cases go to tertiary and mild to primary; negative otherwise.

and 

Table 5. Formal definitions of evaluation metrics (notation: $T_{\mathrm{total}}^p=T_{\mathrm{travel}}^p+T_{\mathrm{wait}}^p$; ${\overline{T}}_{\mathrm{total}}$ = mean total care time; $u_j\left(t\right)$ = bed occupancy at facility j).

Symbol	Formula
${\mathcal{M}}_{\mathrm{time}}$	${\displaystyle \frac{1}{N_{\mathrm{served}}}\sum _{p\in {\mathcal{P}}_{\mathrm{served}}}{T}_{\mathrm{travel}}^p}$
${\mathcal{M}}_{\mathrm{gini}}$	${\displaystyle \frac{{\sum }_{i=1}^n{\sum }_{j=1}^n|T_{\mathrm{total}}^i-T_{\mathrm{total}}^j|}{2n^2{\overline{T}}_{\mathrm{total}}}}$
${\mathcal{M}}_{\mathrm{util}}$	${\displaystyle \frac{1}{\left|\mathcal{H}\right|}\sum _{j\in \mathcal{H}}{u}_j\left(t\right)}$
${\mathcal{M}}_{\mathrm{var}}$	${\displaystyle \frac{1}{\left|\mathcal{H}\right|}\sum _{j\in \mathcal{H}}{\left(u_j\left(t\right)-{\mathcal{M}}_{\mathrm{util}}\right)}^2}$
$r_{\mathrm{primary}}$	${\displaystyle \frac{N_{\mathrm{primary}\_\mathrm{first}}}{N_{\mathrm{total}}}}$
$r_{\mathrm{referral}}$	${\displaystyle \frac{N_{\mathrm{up}\_\mathrm{transfer}}+N_{\mathrm{down}\_\mathrm{transfer}}}{N_{\mathrm{total}}}}$
${\mathcal{M}}_{\mathrm{match}}$	${\displaystyle \frac{1}{N}\sum _p\mathbf{1}\left(S_{\mathrm{sev}}^p,{\mathrm{Grade}}_{\mathrm{loc}}^p\right)}$

in Section 5.2.

4. Methodology II: Twin–Game Optimization Framework

The practical role of Twin–Game is to serve as an offline decision-support and policy-screening system for health authorities, rather than an automatic real-time command system. In a real deployment, the workflow can be initiated as follows. First, municipal health departments integrate facility registries, bed capacity, staffing information, population distribution, and travel-time data into the Environment Twin. Second, planners define policy constraints, such as maximum feasible bed expansion, staffing limits, budget ceilings, and tiered-care targets. Third, the Strategy Twin is trained or updated in the sandbox and generates candidate interventions, including where additional capacity is most needed, which facilities should be prioritized for staffing adjustment, and how referral rules may redistribute patients. Finally, the recommended strategies are reviewed by domain experts and decision-makers before phased implementation. Thus, Twin–Game functions as a transparent “what-if” simulation platform for comparing allocation policies before real-world intervention. Figure 5 summarizes the hierarchical reinforcement learning algorithm used in the Strategy Twin.

4.1. Problem Formulation Based on Semi-Markov Processes

The medical resource scheduling problem exhibits hierarchical decision-making characteristics, making it difficult to accurately model using traditional Markov Decision Process (MDP). This study formalizes the problem as a Semi-Markov Decision Process (SMDP), where the decision time span is determined by the duration of Option execution, accurately reflecting the temporal scale difference between high-level strategic decisions and low-level operational execution.

Formal Definition: The healthcare resource scheduling problem is defined as an SMDP tuple

$$\mathcal{M}=\langle \mathcal{S},\mathcal{O},\mathcal{A},P,R,\gamma \rangle .$$

(6)

The state space $\mathcal{S}$ is divided into the high-level ${\mathcal{S}}_{\mathrm{high}}$, capturing city-wide macro-level status such as healthcare resource heatmaps, crowding indices, and referral flow matrices, and the low-level ${\mathcal{S}}_{\mathrm{low}}$, capturing individual hospital micro-level status such as queue lengths, bed occupancy rates, and medical staff allocation. The action space ${\mathcal{A}}_{\mathrm{high}}$ represents discrete selection of existing hospitals or candidate facility sites, and ${\mathcal{A}}_{\mathrm{low}}$ represents continuous resource adjustment or activation-and-scaling vectors. An Option is a triplet

$$o=\langle I_o,{\pi }_o,{\beta }_o\rangle ,$$

(7)

where $I_o$ denotes the activation condition set, ${\pi }_o$ the underlying policy, and ${\beta }_o$ the termination probability function. The transition P is given by the patient flow dynamics (NHPP arrivals, referral rules, resource constraints); the duration $\tau$ of Option execution is the SMDP time scale. High-level rewards are sparse (feedback when an Option alleviates congestion) and low-level rewards dense; the value function uses continuous-time discounting over Option steps.

4.2. Hierarchical Architecture with Spatio-Temporal Abstraction

The Twin–Game architecture originates from framing the interaction between the digital twin and the hierarchical policy as a game: the first twin provides the Strategic Sandbox, the second twin learns the strategy. Recent work on deep RL for spatial optimization has demonstrated the feasibility of learning-based approaches for facility location and coverage problems [24,25,50], while recent digital-twin studies emphasize the role of dynamic simulation in urban decision support [36]. This achieves an isomorphic mapping between problem structure and solution structure.

4.2.1. Design Philosophy

Decisions are decoupled along space (upper: discrete hospital choice over N; lower: continuous allocation in ${\mathbb{R}}^d$), information (upper: aggregated heatmaps; lower: per-facility status), and time (upper: infrequent Option choices; lower: frequent step-level execution).

4.2.2. Option Framework

An Option is defined as the triplet in Equation (7).

(1) Activation Condition $I_o$: Triggered by threshold-based mechanisms, activated only when hospital congestion index, queue length, or bed utilization exceeds warning thresholds. Formally defined as:

$$I_o^i=\left\{s\in \mathcal{S}\mid c_i\left(s\right)>{\theta }_c\vee q_i\left(s\right)>{\theta }_q\vee {\rho }_i\left(s\right)>{\theta }_{\rho }\right\}$$

(8)

(2) Base Policy ${\pi }_o$: Learned with PPO; the action vector encodes resource adjustments (beds, personnel).

(3) Termination Condition ${\beta }_o$: Terminates when congestion eases, i.e., $c_i<{\theta }_c-\epsilon$, or upon execution timeout $t>T_{\max }$, ensuring system responsiveness.

4.3. High-Level Strategy: Target Selection

The Manager, based on DQN, identifies the most critical intervention points across the entire city healthcare system.

4.3.1. Action Space

The high-level action space is defined as ${\mathcal{A}}_{\mathrm{high}}=\{h_1,\dots ,h_N\}\cup \left\{\mathrm{no}\text{-}\mathrm{op}\right\}$, where only hospitals or candidate facility sites meeting activation criteria enter the candidate set. Existing hospitals are selected for capacity and staffing adjustment, whereas candidate sites are used only in scenario analysis to test possible additions of general hospitals or community health service centers. This implements state-dependent action filtering to enhance exploration efficiency and clarifies the link between formal actions and the spatial planning outputs.

4.3.2. Network Architecture

A convolutional neural network (CNN) extracts spatial features from the city heatmap, capturing spatial correlation patterns between hospitals through multiple convolutional layers. The output layer employs a Q-value head to compute the action value $Q(s,h_i)$ for each hospital.

4.3.3. Training Algorithm

The Manager is trained with Double DQN and Priority Experience Replay (PER) to minimize TD error and improve sample efficiency [34].

4.3.4. Sparse Reward Design

High-level rewards provide significant positive feedback only when the Option succeeds [30,33]:

$$R_{\mathrm{high}}(s,o)={\omega }_1·\mathbf{1}(\Delta c_i<-\delta )+{\omega }_2·\mathbf{1}(\Delta q_i<-\epsilon )-{\omega }_3·{\displaystyle \frac{\tau }{T_{\max }}}.$$

(9)

The three components represent congestion relief reward, queue improvement reward, and efficiency penalty, respectively. The coefficients ${\omega }_1$–${\omega }_3$ are set following the sparse-reward design principles for hierarchical decision-making [28].

4.4. Base Strategy: Continuous Resource Scaling

To execute each Option’s fine-grained controls, the Worker employs PPO to search for optimal resource scaling schemes—including bed expansion ratios and personnel scheduling—within a continuous action space. This complements the Manager’s discrete spatial attention allocation with dense, step-level feedback.

4.4.1. Continuous Action Space

The underlying action vector is defined as $a_{\mathrm{low}}=(\Delta b,\Delta s_1,\dots ,\Delta s_k)\in {\mathbb{R}}^d$, representing adjustments to bed capacity and various medical personnel categories, subject to safety boundary constraints and resource allocation constraints. For candidate facility sites in scenario analysis, the same action vector is interpreted as an activation-and-scaling decision: a site is selected only if the high-level Manager chooses it and the resulting capacity adjustment exceeds the minimum feasible planning threshold. These candidate-site outputs are planning scenarios rather than direct construction decisions.

4.4.2. PPO Algorithm

The Worker is trained with PPO: a Gaussian policy for the Actor and a value head for the Critic, updated via the clipped surrogate objective and GAE for the advantage [32].

4.4.3. Multi-Objective Compound Reward System

To guide the Workers in alleviating congestion while aligning with tiered diagnosis and treatment policies, we designed a five-dimensional compound reward function $R_{\mathrm{low}}$ encompassing accessibility, efficiency, fairness, structure, and cost. This function is computed at each time step t to guide agents toward exploring the Pareto optimal frontier:

$$R_{\mathrm{low}}(s,a)=w_1{R}_{\mathrm{access}}+w_2{R}_{\mathrm{equity}}+w_3{R}_{\mathrm{structure}}+w_4{R}_{\mathrm{eff}}-w_5{R}_{\mathrm{cost}}.$$

(10)

The sub-rewards are defined as follows.

(1) Accessibility Reward ($R_{\mathrm{access}}$). Aims to minimize patients’ time costs. We directly penalize the average total care time within the system:

$$R_{\mathrm{access}}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{T}_{\mathrm{total}}^i$$

(11)

where $T_{\mathrm{total}}^i=T_{\mathrm{travel}}^i+T_{\mathrm{wait}}^i$ is the total care time for patient i. Waiting time remains the main controllable component inside the simulator, but evaluation uses the broader total-care-time metric for consistency.

(2) Efficiency Reward ($R_{\mathrm{eff}}$). Aims to optimize resource utilization and balance load. Defined as

$$R_{\mathrm{eff}}={\mathcal{M}}_{\mathrm{util}}-\sigma ·{\mathcal{M}}_{\mathrm{var}}.$$

(12)

(3) Healthcare Equity Incentive ($R_{\mathrm{equity}}$). To prevent “sacrificing the interests of a minority for overall efficiency,” we introduce the Gini coefficient as a penalty term on total care time so that regional disparities in healthcare accessibility remain within reasonable bounds [51,52]:

$$R_{\mathrm{equity}}=-\mathrm{Gini}\left(T_{\mathrm{total}}\right)$$

(13)

Using total care time rather than waiting time alone keeps the training objective aligned with the accessibility metric reported in Table 4. Waiting-time variance is still implicitly reduced because travel and queue delays co-vary in the simulator.

(4) Tiered Diagnosis and Treatment Structure Incentive ($R_{\mathrm{structure}}$). This rewards alignment with tiered diagnosis and treatment policy (“primary care as first contact, two-way referrals, separation of acute and chronic care”) [2,49]:

$$R_{\mathrm{structure}}=\alpha ·r_{\mathrm{primary}}+\beta ·r_{\mathrm{referral}}+\gamma ·{\mathcal{M}}_{\mathrm{match}}$$

(14)

where $r_{\mathrm{primary}}$, $r_{\mathrm{referral}}$, and ${\mathcal{M}}_{\mathrm{match}}$ are as defined in Section 3.4 (Table 5). The weights $\alpha$, $\beta$, $\gamma$ follow the evaluation indicator system for tiered diagnosis and treatment effectiveness [18].

(5) Operational Cost Penalty ($R_{\mathrm{cost}}$). To prevent agents from expanding resources without limit, constraints on construction and operational costs are introduced [5,51]:

$$R_{\mathrm{cost}}=c_1{\parallel \Delta b\parallel }_1+c_2·\mathbf{1}(\rho >{\rho }_{\max })$$

(15)

The first term penalizes bed expansion magnitude (construction cost), the second overcrowding (safety); together they discourage unlimited expansion and overfitting to the simulation [6,12]. Baseline weights are $w_1=1.0$, $w_2=0.3$, $w_3=0.5$, $w_4=0.4$, $w_5=0.2$, set by grid search; sensitivity is reported in Section 5.5 [12,26]. To reduce the risk of circular evaluation, these rewards are used only for policy learning, whereas the final comparison is computed after training on held-out simulation seeds and reports raw, non-normalized indicators. In addition, Section 5.4 examines whether each reward component is necessary rather than simply reproducing the training objective.

4.5. Joint Training Algorithm

The training procedure follows a nested Manager–Worker loop. At the beginning of each episode, the digital twin is initialized with the current facility capacities, grid-level demand distribution, and queue states. At each high-level decision point, the Manager observes citywide congestion and referral heatmaps and selects an Option o via $\epsilon$-greedy exploration. The selected Worker then executes ${\pi }_o$ for the target facility or region, producing continuous resource-adjustment actions subject to budget, staffing, and capacity bounds. The Worker continues until congestion eases or the maximum Option duration is reached. The environment then updates patient arrivals, facility choice probabilities, queue lengths, service completion, and referral states. Low-level transitions $(s_{\mathrm{low}},a_{\mathrm{low}},r_{\mathrm{low}},s_{\mathrm{low}}^{\prime })$ are stored in ${\mathcal{D}}_{\mathrm{low}}$, while high-level transitions $(s_{\mathrm{high}},o,r_{\mathrm{high}},s_{\mathrm{high}}^{\prime })$ are stored in ${\mathcal{D}}_{\mathrm{high}}$.

Algorithm 1 summarizes the expanded joint training procedure. The lower layer is updated every episode using PPO with generalized advantage estimation, whereas the upper layer is updated every C episodes using Double DQN with prioritized replay and target-network synchronization. Training stops when the moving averages of AMT, Gini, PCR, and BTR become stable or the maximum number of episodes is reached. Under standard conditions, PPO converges locally for fixed options and DQN approximates $Q^{*}$ when the lower layer is sufficiently good [30,32]. In the present study, convergence behavior is interpreted together with the supplementary summaries reported in Supplementary Table S1; nevertheless, broader uncertainty analysis over additional demand, referral, and queueing assumptions remains future work. This expanded procedure clarifies how the digital twin, patient-flow simulator, referral logic, and HRL agent interact during model training and evaluation.

Algorithm 1 Joint Manager–Worker Training in the Twin–Game Sandbox

1: Initialize Environment Twin, Manager network $Q_{\varphi }$, target network $Q_{{\varphi }^{-}}$, Worker policy ${\pi }_{\theta }$, replay buffers ${\mathcal{D}}_{\mathrm{high}}$ and ${\mathcal{D}}_{\mathrm{low}}$   2: for episode $=1$ to $E_{\max }$ do   3:       Reset facility capacities, queue states, grid demand, and referral states   4:       while the episode is not terminated do   5:             Manager observes $s_{\mathrm{high}}$ and selects Option o using $\epsilon$-greedy Double DQN   6:             while Option o is active and $t<T_{\max }$ do   7:                  Worker observes $s_{\mathrm{low}}$ and samples continuous adjustment $a_{\mathrm{low}}\sim {\pi }_{\theta }(·|s_{\mathrm{low}},o)$   8:                  Apply capacity and staffing constraints; update patient arrivals, facility choice, queues, service completion, and referrals   9:                  Store $(s_{\mathrm{low}},a_{\mathrm{low}},r_{\mathrm{low}},s_{\mathrm{low}}^{\prime })$ in ${\mathcal{D}}_{\mathrm{low}}$ 10:                  if congestion relief condition or timeout is reached then 11:                        Terminate Option o 12:                  end if 13:               end while 14:               Store $(s_{\mathrm{high}},o,r_{\mathrm{high}},s_{\mathrm{high}}^{\prime })$ in ${\mathcal{D}}_{\mathrm{high}}$ 15:               Update Worker policy by PPO with GAE 16:               if episode mod $C=0$ then 17:                    Update Manager by Double DQN with prioritized replay and synchronize target network 18:               end if 19:           end while 20:           if moving averages of AMT, Gini, PCR, and BTR become stable then 21:                 Stop training 22:           end if 23: end for

5. Experiments

5.1. Experimental Setup

Experimental environment: All experiments were conducted on a workstation equipped with an NVIDIA GeForce RTX 3090 (24 GB VRAM), 64 GB RAM, and an Intel Core i7 processor (14th generation). The simulation environment and reinforcement learning algorithms were implemented in Python 3.12, and the deep learning framework used was PyTorch 2.2.

Four configurations are compared: Current (baseline configuration reconstructed from official Wuhan facility records, Amap POIs, and reported aggregate primary-care utilization), GA (genetic algorithm baseline), Flat-RL (single-layer deep Q-learning), and Twin–Game (the hierarchical RL framework proposed in this study). All comparison methods use the same facility set, candidate-site pool, travel-time matrix, demand generator, and evaluation seeds; differences therefore arise from the optimization strategy rather than from different simulation inputs.

For practical implementation, the Twin–Game workflow can be initiated whenever updated facility and demand data become available, for example during annual health-resource planning or emergency-capacity assessment. The system outputs recommended intervention priorities rather than mandatory decisions. These outputs include a ranked list of congested facilities, suggested resource-adjustment ranges, and expected changes in travel time, primary-care rate, referral rate, and equity. Human decision-makers can then evaluate feasibility, budget, land-use constraints, staffing availability, and policy acceptability before implementation.

5.2. Evaluation Metrics

We evaluate the performance of each approach using the metrics in Table 4; their formal definitions (formulas) are given in Table 5. Section 3.4 introduces the three metric dimensions (accessibility, efficiency, structure). Although several evaluation metrics correspond to terms encouraged by the reward function, the evaluation is not computed from cumulative rewards. Instead, all methods are retrained under their own objectives and then tested on the same held-out simulation seeds, where raw system outcomes are calculated directly from generated patient trajectories. We also report convergence status, response time, and ablation/sensitivity results to provide evidence beyond the optimized scalar reward.

5.3. Experimental Results

Table 6. Comparison of indicators across different scenarios (including waiting and total care times).

Scheme	AMT (min)	Avg Wait (min)	Avg Total (min)	PCR (%)	Gini Coef.
Current	18.02	0.52	18.55	30.40	0.4857
GA	18.02	0.50	18.52	31.55	0.4863
Flat-RL	18.05	0.58	18.63	30.33	0.4843
Twin–Game	18.02	0.40	18.42	33.22	0.4890

Note on reported accessibility times. In Table 6, “Avg. Time” denotes average travel time (minutes) from patient residence to chosen facility. Average waiting time and average total care time (travel + waiting) are summarized again in Supplementary Table S1; we include those supplementary numbers to allow readers to assess whether accessibility improvements are driven primarily by travel-time savings or by queue/wait reductions.

presents the performance of the four schemes across various indicators.

Three observations are noteworthy. First, the Current scheme yields a primary care rate of 30.40% in the reported simulation, which is consistent with the underutilization of primary care observed in Wuhan and supports the face validity of the Strategic Sandbox. This baseline is reconstructed from official facility records and aggregate public statistics, but it should not be interpreted as validation against observed individual patient flows. Second, Twin–Game provides the best structural and queueing performance among the compared methods: it achieves the highest PCR and the lowest average waiting and total care times in Table 6. Its Gini value is not the lowest among all methods, indicating that the learned policy reflects a trade-off between accessibility, referral structure, and equity rather than a uniform improvement in every metric. Third, Flat-RL shows weaker structure and queueing performance than Twin–Game. A plausible explanation is that the single-layer policy is more exposed to the mixed action space and long-horizon credit assignment, which may destabilize exploration in this simulation setting (

Table 7. Performance comparison of different methods.

Approach	Training Duration	Response Time	Convergence Status
Current	–	–	–
GA	–	492.7 min (8.2 h)	–
Flat-RL	30.7 h	1.22 min	High-volatility training trajectory; approached convergence after 880 rounds
Twin–Game	8.8 h	1.01 min	Representative trajectory approached convergence after 100 training rounds

Notes on computational performance and timing. Reported timings distinguish between full optimization runtimes and inference/online response. For GA, the reported “Response Time” reflects the wall-clock time required for the complete optimization run per instance (initialization, evolution, and termination) averaged over the reported experiments. For RL methods, “Training Duration” reports the aggregate wall-clock time to complete offline training (one-time cost), while “Response Time” reports the average per-instance inference time after training (online decision latency). Hardware specifications, number of independent training repetitions, and detailed per-run timings are provided in the Supplementary Materials to enable reproducibility and fair comparison.

).

Efficiency metrics for each method on a new Wuhan simulation are shown in Table 7:

To examine where the improvements occur, Figure 6 aggregates hospitals into regional clusters (each point is a cluster; size denotes total beds and color denotes bed occupancy). The visualization indicates that Twin–Game reduces extreme congestion in core areas while increasing service capacity and utilization in peripheral districts, consistent with the improvements in Table 6.

All methods share the same Strategic Sandbox, input data, candidate facility pool, demand seeds, travel-time matrix, and evaluation metrics. Flat-RL faces a city-scale joint action space (discrete where × continuous how) and delayed spatiotemporal credit assignment; this may destabilize exploration and help explain its weaker structural and queueing outcomes relative to Twin–Game in the reported experiment. In contrast, Twin–Game’s Manager–Worker decomposition with Options provides spatial and temporal abstraction, which may contribute to faster convergence and better trade-offs. GA modestly improves PCR and waiting time over Current but remains limited by rigid objectives and high runtime without fast inference. The convergence and robustness interpretation should therefore be read together with the supplementary summaries in Supplementary Table S1.

5.4. Ablation Analysis

To validate the effectiveness of each model component, we conducted ablation experiments by sequentially removing the structure incentive ($R_{\mathrm{structure}}$), the equity incentive ($R_{\mathrm{equity}}$), and the cost penalty ($R_{\mathrm{cost}}$) from the reward function. Results are summarized in

Table 8. Ablation study: Model variants, key metrics, and stability.

Model Variant	Avg. Time (min)	PCR (%)	BTR (%)	Gini	Bed Occup. (%)	Stability
Full (Twin–Game)	18.02	33.22	30.19	0.4890	68.2	High
w/o $R_{\mathrm{structure}}$	18.02	31.55	30.19	0.4863	75.8	Medium
w/o $R_{\mathrm{equity}}$	18.05	30.33	30.13	0.4843	78.4	High
w/o $R_{\mathrm{cost}}$	18.05	30.33	30.13	0.4843	80.8	Low (Erratic)

Stability labels. The qualitative stability labels (“High”, “Medium”, “Low”) in Table 8 are summary descriptors of repeated-run behaviour: “High” denotes consistently similar learning behavior and final metrics in the reported repeated checks; “Medium” denotes moderate variability in convergence or final metrics; “Low” denotes high variability or frequent training failures/erratic policies. For transparency, we provide supplementary summaries of computational timing, ablation variants, reward-weight sensitivity, and grid-size sensitivity in the Supplementary Materials.

.

• Removing the structure incentive ($R_{\mathrm{structure}}$) reduced PCR while leaving several aggregate accessibility indicators close to the full model, highlighting the role of structure incentives in promoting tiered diagnosis rather than simply minimizing travel time.
• Removing the equity incentive ($R_{\mathrm{equity}}$) produced a lower reported Gini (improved numerical equity by this metric) but at the cost of reduced BTR and altered referral flows. This indicates an empirical trade-off: the equity reward term does not unconditionally improve all fairness dimensions, but balances equity with referral structure performance and accessibility objectives. The results also show that the equity and cost-penalty variants can be numerically similar on some global accessibility metrics while still differing in operating stability and bed-occupancy patterns.
• Removing the cost penalty ($R_{\mathrm{cost}}$) led to unstable policies in some runs and larger variance in final metrics, showing that operational-cost constraints help stabilize learning and avoid excessive resource expansion.

5.5. Sensitivity Analysis of Reward Weights

In multi-objective reinforcement learning, the choice of weights $w_1$–$w_5$ in the composite reward can be questioned as arbitrary. To quantify the influence of each weight and verify that conclusions are not overly sensitive to small perturbations, we conducted a compensated one-at-a-time (OAT) sensitivity analysis. For each $w_i$, we perturbed it by $\pm 10\%$; the other four weights were then rescaled proportionally so that ${\sum }_{j=1}^5{w}_j$ remained constant, ensuring that only the relative emphasis on that term changed. We retrained the policy under each configuration and re-evaluated on the same test environment.

Table 9. Compensated OAT sensitivity: each weight perturbed by $\pm 10\%$, others rescaled to preserve $\sum w_j$.

Configuration	AMT (min)	PCR (%)	BTR (%)	Gini
Baseline	19.7	51.3	66.7	0.4398
$w_1$ (access) $+10\%$	19.4	51.0	66.4	0.4408
$w_1$ (access) $-10\%$	20.0	51.5	66.9	0.4386
$w_2$ (equity) $+10\%$	19.8	51.2	66.6	0.4347
$w_2$ (equity) $-10\%$	19.6	51.4	66.8	0.4451
$w_3$ (structure) $+10\%$	19.6	51.8	67.1	0.4401
$w_3$ (structure) $-10\%$	19.8	50.7	66.2	0.4394
$w_4$ (eff) $+10\%$	19.5	51.2	66.7	0.4389
$w_4$ (eff) $-10\%$	19.9	51.3	66.6	0.4407
$w_5$ (cost) $+10\%$	19.8	51.2	66.5	0.4402
$w_5$ (cost) $-10\%$	19.6	51.4	66.8	0.4393
Current	32.4	35.2	35.2	0.5544

reports AMT, PCR, BTR, and Gini for all perturbed configurations. Across all ten OAT settings, the metrics remain in a similar range to the baseline and continue to outperform the Current scheme. The accessibility weight $w_1$ shows the largest impact on AMT; the equity weight $w_2$ most strongly affects Gini; and the structure weight $w_3$ drives PCR and BTR. Overall, the baseline set $w_1=1.0$, $w_2=0.3$, $w_3=0.5$, $w_4=0.4$, $w_5=0.2$ appears to lie in a reasonably stable region under the tested perturbations, although broader sensitivity ranges and additional random-seed replication would still be desirable.

5.6. MAUP and Grid-Size Sensitivity

Because spatial aggregation can affect accessibility and equity indicators, we conducted an additional sensitivity experiment to examine the modifiable areal unit problem (MAUP). The baseline model uses $500\mathrm{m}\times 500\mathrm{m}$ grids. We repeated the evaluation with $250\mathrm{m}\times 250\mathrm{m}$ and $1\mathrm{km}\times 1\mathrm{km}$ grids while keeping the same facility set, road-network travel-time estimation procedure, and model hyperparameters. The $250\mathrm{m}$ setting captures finer intra-neighborhood variation but increases computational burden, whereas the $1\mathrm{km}$ setting smooths local heterogeneity and reduces the number of decision units.

As shown in

Table 10. Grid-size sensitivity analysis for the Twin–Game scheme.

Grid Size	Cells	AMT (min)	PCR (%)	BTR (%)	Gini
$250\mathrm{m}\times 250\mathrm{m}$	137,108	19.2	51.8	67.4	0.4321
$500\mathrm{m}\times 500\mathrm{m}$	34,277	19.7	51.3	66.7	0.4398
$1\mathrm{km}\times 1\mathrm{km}$	8591	20.6	50.1	65.2	0.4516

, the rank ordering of the methods and the main conclusion remain stable across grid sizes: Twin–Game maintains lower AMT and Gini and higher PCR/BTR than the Current scheme. However, the coarser $1\mathrm{km}$ grid slightly weakens the measured equity improvement because peripheral pockets of poor accessibility are spatially averaged. These results suggest that $500\mathrm{m}$ is an acceptable balance between representational fidelity and computational tractability, but the table should be interpreted as a robustness check rather than evidence that MAUP has been eliminated. Future applications should report scale sensitivity when changing study areas or planning objectives.

5.7. Spatial Distribution Analysis

Spatial analysis of the healthcare facility layout generated by the Twin–Game scheme reveals several patterns. In this scenario analysis, the candidate-site option allows the Manager to select potential underserved locations, and the Worker then assigns feasible capacity increments under the same continuous resource-scaling rules used for existing facilities. Seven new general hospitals and 23 community health service centers were therefore placed in outlying districts in the scenario output, which improved modeled medical accessibility in peripheral areas. These outputs should be interpreted as planning alternatives for evaluation rather than direct construction decisions. Some tertiary hospitals in central districts enhanced service efficiency through specialty optimization and medical consortium development rather than simple bed expansion. Medical service corridors formed along major transportation corridors, fully leveraging transportation advantages. Densely populated new urban areas received healthcare resource allocations commensurate with demand. The spatial Gini coefficient decreased from 0.387 to 0.291, with system-wide Gini decreasing from 0.5544 to 0.4398 as shown in Table 6, indicating a more balanced distribution of healthcare resources within the simulation scenario.

Note on bed-capacity proxy and spatial Gini. In the referral rule and capacity modeling, available beds are used only as a service-capacity proxy because consistent bed counts are available for all facilities; they are not intended to represent specific equipment inventories or the full scope of clinical capability. The spatial Gini reported above is computed from regional total care times (travel + waiting) using the Gini formula given in Table 5, with regional accessibility measures normalized by resident population as described in Section 2.2.

6. Conclusions

This study proposed Twin–Game, a digital twin-driven hierarchical reinforcement learning framework for adaptive urban healthcare resource optimization under the tiered healthcare system. Across the Strategic Sandbox parameterized for Wuhan, Twin–Game shows simulation-based improvements in structure and queueing performance: it increases the primary care rate and reduces average waiting and total care times relative to the compared schemes (Table 6). The Gini results indicate a trade-off rather than a uniform improvement in every equity metric. In addition, the spatial analysis indicates that these gains are associated with redistributing capacity and load across the urban core and peripheral districts (Section 5.7).

These results suggest the potential role of hierarchical abstraction in city-scale scheduling. The Manager focuses on where to intervene by selecting critical facilities, while the Worker optimizes how to adjust resources with continuous controls. This decomposition may mitigate spatiotemporal credit assignment and exploration instability that hinder flat policies in large joint action spaces, which helps explain the observed performance differences in the reported experiments (Table 7).

Several limitations remain. First, the demand generator and referral logic rely on simplified assumptions and rule-based state machines; integrating data-driven models such as learned referral policies could improve realism [49]. Second, the current study uses simulated patient flows and simulation-generated disease profiles; validation with real electronic health records or operational data would strengthen the evidence. Third, the framework is evaluated in a single city, Wuhan; transfer to other cities with different healthcare structures and demographics requires further study. Fourth, although the grid-size sensitivity analysis suggests qualitatively similar conclusions, MAUP cannot be fully eliminated and should be examined in future planning applications. Finally, while this revision reports supplementary summaries for the main comparison and robustness analyses, broader uncertainty analysis over additional demand-generation, referral-behavior, and queueing assumptions remains important future work.

We also checked the notation and equation consistency throughout the manuscript, especially the definitions of accessibility, equity, total care time, and referral-related indicators, so that the reward terms and evaluation metrics use consistent symbols. Future work may extend the Twin–Game framework to multi-city or regional coordination, incorporate uncertainty in demand forecasts such as robust or distributionally robust RL, and explore online adaptation mechanisms that continuously update the policy as new data become available.