Federated Decision Transformers for Scalable Reinforcement Learning in Smart City IoT Systems

Abstract

The rapid proliferation of devices on the Internet of Things (IoT) in smart city environments enables autonomous decision-making, but introduces challenges of scalability, coordination, and privacy. Existing reinforcement learning (RL) methods, such as Multi-Agent Actor–Critic (MAAC), depend on centralized critics and recurrent structures, which limit scalability and create single points of failure. This paper proposes a Federated Decision Transformer (FDT) framework that integrates transformer-based sequence modeling with federated learning. By replacing centralized critics with self-attention-driven trajectory modeling, the FDT preserves data locality, enhances privacy, and supports decentralized policy learning across distributed IoT nodes. We benchmarked the FDT against MAAC in a mobile edge computing (MEC) environment with identical hyperparameter configurations. The results demonstrate that the FDT achieves superior reward efficiency, scalability, and adaptability in dynamic IoT networks, although with slightly higher variance during early training. These findings highlight transformer-based federated RL as a robust and privacy-preserving alternative to critic-based methods for large-scale IoT systems.

1. Introduction

The rapid growth of smart city technologies has increased the reliance on IoT devices to support transportation, surveillance, energy management, and other critical urban operations [1,2,3,4,5,6,7]. These distributed systems generate large amounts of sequential data and require intelligent and adaptive methods for real-time decision-making. RL offers a promising framework for such tasks, yet traditional actor–critic and value-based approaches struggle with two persistent challenges: capturing long-term temporal dependencies and scaling across large heterogeneous IoT networks [8,9,10,11].

MAAC methods were developed to improve coordination among agents in complex settings [12,13,14]. However, MAAC retains a centralized critic that constrains scalability and creates a single point of failure. Its reliance on recurrent architectures further limits effectiveness as vanishing gradients hinder long-horizon reasoning. These shortcomings reduce the suitability of MAAC for high-dimensional dynamic smart city applications where both privacy and adaptability are essential [15,16].

Recent advances in deep learning, particularly transformer architectures, have reshaped sequential modeling. Using self-attention, transformers capture long-range dependencies more effectively than RNNs or LSTM [17,18,19]. Decision Transformers (DTs) extend this capability to reinforcement learning by framing trajectory optimization as a sequence modeling problem [20,21,22]. DTs generate actions conditioned on return-to-go, enabling efficient, value-free policy learning and improved generalization across tasks [20,23,24]. However, most DT frameworks assume centralized training and do not address privacy or scalability in federated contexts [25,26,27].

In this work, we propose an FDT that integrates transformer-based sequence modeling with a federated learning paradigm [28,29]. This framework eliminates the centralized critic, enabling multiple agents to collaboratively learn policies while keeping data local. By combining self-attention with federated aggregation, the FDT improves scalability, robustness, and privacy preservation in distributed IoT environments [30,31].

Our contributions are threefold:

• We designed a federated reinforcement learning framework that embeds Decision Transformers in place of MAAC’s centralized critic, capturing long-horizon dependencies without relying on recurrent networks [32,33].
• We demonstrate how the FDT supports horizontal scalability and privacy by enabling decentralized policy learning through federated aggregation of local updates [34].
• We empirically benchmarked the FDT against MAAC in mobile edge computing simulations, showing improved reward efficiency, adaptability, and scalability, with a trade-off of slightly higher variance during early training [35,36].

The remainder of this paper is organized as follows. Section 2 reviews previous work on reinforcement learning in smart city applications. Section 3 introduces the proposed Federated Decision Transformer framework and its algorithms, followed in Section 4 by a conceptual comparison with the MAAC baseline. Section 5 outlines the experimental environment and evaluation protocol, and Section 6 presents and discusses the empirical results. Finally, Section 7 summarizes the main findings and outlines future research directions.

2. Related Work

2.1. Federated Learning in Smart Cities

Federated learning (FL) has emerged as a practical alternative to centralized training for privacy-sensitive urban analytics across domains such as transportation, energy, and public safety. Surveys highlight key challenges, statistical heterogeneity, client drift, and communication costs that affect real deployments [3,4,5,6]. Beyond surveys, concrete case studies illustrate the promise of FL: Bao et al.  [7,37] applied federated RL for traffic signal control, achieving reduced congestion and improved travel times while preserving privacy by keeping intersection data local. Similarly, Yu et al. [11] demonstrated how federated RL supports resource allocation in multi-access edge computing, while Wang et al. [10] highlighted the role of in-edge AI for joint caching, communication, and learning. Recent work also integrates fairness and adversarial robustness into edge FL frameworks [15], underscoring the need for scalable and secure collaborative training in IoT-driven smart cities.

2.2. Reinforcement Learning for Autonomous Decision-Making

Reinforcement learning (RL) has proven to be an effective methodology for autonomous decision-making across a range of dynamic domains. Recent advances demonstrate that RL can adapt to complex, non-stationary environments such as traffic control [38], energy optimization [39], resource allocation in edge-IoT networks [40], and collaborative intelligence for IoT systems [35]. These studies highlight RL’s capacity for continuous adaptation, distributed coordination, and reward-driven optimization, key attributes for smart city applications. Recent Works on multi-agent RL [33] and transformer-enhanced coordination [36] further confirm its growing relevance in achieving scalable, cooperative, and context-aware decision-making in large, interconnected environments.

2.3. Transformers for Reinforcement Learning and MARL

Transformers have recently expanded from natural language processing into RL, where their self-attention enables long-horizon credit assignment. Several surveys map this emerging landscape, covering representation learning, policy optimization, and dynamics modeling, while also noting challenges such as computation cost and data hunger [18,19]. In multi-agent reinforcement learning (MARL), attention-based designs such as the multi-agent transformer (MAT) [21] have achieved state-of-the-art performance in cooperative benchmarks. More recent work on Multi-Agent Decision Transformers (MADTs) demonstrates efficient trajectory modeling across agents, supporting transfer and adaptation from offline to online [22]. Extensions in 2024 propose selective or structured attention to reduce complexity and latency when scaling to large agent populations [41]. Applied studies also show the benefits of attention-based MARL in traffic control and dynamic environments [42], strengthening their suitability for smart city applications.

2.4. Federated Reinforcement Learning and Federated Transformers

Another research thread adapts reinforcement learning and transformers to federated or distributed settings. Qi et al. [32] provided one of the first surveys on federated reinforcement learning (FRL), highlighting techniques and open challenges. More recently, Li et al. introduced FedTP, a federated personalized transformer architecture that uses server-side hypernetworks to mitigate non-IID challenges [25]. Other works explore federated prompt- or split-based Decision Transformers for mobile edge computing, reducing bandwidth demands and improving personalization [43,44]. In security-sensitive contexts, Parra et al. [29] propose interpretable federated transformers for intrusion detection, aligning with privacy mandates in critical infrastructures. Robustness remains a central concern: recent surveys and reviews highlight threats such as poisoning and backdoors while proposing secure aggregation and anomaly detection as defenses [16,30,34,45]. Broader surveys on efficient FL with foundation and transformer models [28] further emphasize parameter-efficient tuning and update scheduling strategies, both relevant for resource-constrained IoT devices.

Compared with centralized-critic MAAC approaches and prior federated/multi-agent transformers, our contribution is threefold: (i) removing the centralized critic by embedding Decision Transformers per agent node, (ii) integrating transformer-specific federated personalization strategies, and (iii) evaluating scalability, variance, and communication trade-offs under MEC-like smart city constraints.

3. Methodology

We propose an FDT framework that replaces the centralized critic of MAAC with decentralized Decision Transformers trained with a federated learning loop. Each IoT agent maintains its own DT, trained locally on private trajectories, while the server aggregates model updates without ever accessing raw data. This section details the model design, the client-side procedure, and the aggregation of the servers.

3.1. Model Design

Each client agent encodes sequences of $(s_t,a_t,r_t,{\widehat{G}}_t)$ tokens, where ${\widehat{G}}_t$ is the normalized return-to-go (RTG).

• Embeddings. States, actions, and RTGs are projected into a common hidden dimension and concatenated with positional encodings.
• Transformer encoder. Multi-head self-attention captures long-range dependencies, enabling agents to model temporal correlations and interaction patterns without recurrence.
• Policy head. Outputs logits for discrete actions or mean values for continuous actions, conditioned on the contextualized sequence representation.
• RTG normalization. A running normalizer stabilizes training in heterogeneous client environments. Normalization is performed locally at each client, while aggregated statistics across clients maintain consistency at the federated level.

In the federated multi-agent setting, the RTG acts as the implicit coordination signal across agents. Rather than depending on a centralized critic to exchange gradient information, each agent conditions its decision process on a tokenized sequence $[{\widehat{G}}_t,s_t,a_{t-1}]$, where ${\widehat{G}}_t$ represents the normalized cumulative future reward from step t. This conditioning allows agents to infer how their local actions contribute to the global objective through predicted future returns, rather than through explicit critic feedback. As all agents learn to optimize toward consistent normalized RTG expectations, coordinated behaviors emerge implicitly even without parameter sharing or direct communication of trajectories. This decentralized mechanism supports scalability and privacy preservation, while the self-attention structure ensures that long-horizon dependencies and inter-agent dynamics are effectively captured.

This design eliminates the need for a centralized critic by conditioning action prediction directly on returns-to-go.

3.2. Client-Side Training

Each client interacts with its environment to collect trajectories, computes discounted RTGs, normalizes them, and stores the data in a replay buffer. Training proceeds in subsequences of length L, padded with masks to handle variable episode lengths. Teacher forcing is applied: the DT is conditioned on observed states, actions, and RTGs and predicts the next action token. The objective is as follows.

$$\mathcal{L}=\left\{\begin{array}{cc}\mathrm{CE}(a_t,{\pi }_{\theta }(a_t\mid s_{1:t},a_{1:t},{\widehat{G}}_{1:t}))\hfill & \mathrm{discrete}\mathrm{actions},\hfill \\ \parallel a_t-{\mu }_{\theta }{\parallel }_2^2\hfill & \mathrm{continuous}\mathrm{actions},\hfill \end{array}\right.$$

with optional RTG consistency and weight regularization terms. For every K local episodes, the client transmits its model delta ${\Delta }_c$ and sample count $n_c$ to the server.

Complexity

The per-client cost per round is

$$O\left(E·\frac{|{\mathcal{D}}_c|}{B}·L^{2}dH\right),$$

where $|{\mathcal{D}}_c|$ is the number of trajectory tokens, L is the sequence length, d the hidden dimension, and H the number of transformer layers. Communication is $O\left(\right|\theta \left|\right)$ parameters for every K episodes.

3.3. Server-Side Aggregation

The server receives $({\Delta }_c,n_c)$ from participating clients and applies sample-size weighted aggregation:

$${\theta }^{\left(r\right)}={\theta }^{(r-1)}+\sum _{c\in {\mathcal{S}}^{\left(r\right)}}{w}_c{\Delta }_c,w_c={\displaystyle \frac{n_c}{{\sum }_j{n}_j}}.$$

Optional preprocessing includes decompression, unmasking for secure aggregation, adapter masking for PEFT, and norm clipping for differential privacy. Robust alternatives, such as coordinate median, trimmed mean, or Krum mitigate adversarial or corrupted updates.

3.3.1. Complexity

The server cost is $O\left(\right|{\mathcal{S}}^{\left(r\right)}|·|\theta \left|\right)$ for FedAvg. Robust methods such as Krum require $O\left(\right|{\mathcal{S}}^{\left(r\right)}{|}^2)$ distance computations. Communication is $O\left(\right|{\mathcal{S}}^{\left(r\right)}|·|\theta \left|\right)$ per round, reducible with compression or PEFT.

The proposed methodology integrates client-side Decision Transformer training with server-side federated aggregation, enabling scalable and privacy-preserving reinforcement learning across heterogeneous IoT environments. Algorithms 1 and 2 formalize the decentralized workflow, while complexity analysis highlights the trade-offs between computation ($O\left(L^{2}dH\right)$ self-attention per sequence) and communication ($O\left(\right|\theta \left|\right)$ parameters per upload). To validate the effectiveness of this framework, we evaluated the FDT against the MAAC baseline in controlled MEC simulations. The following section describes the experimental setup, including environments, hyperparameters, and evaluation metrics, designed to assess convergence, scalability, communication efficiency, and robustness.

Algorithm 1 FDT Client-Side Training

Require: Clients $\mathcal{C}$; global params ${\theta }^{\left(0\right)}$; rounds R; local episodes E; cadence K; batch size B; sequence length L; learning rate $\eta$ Ensure: Updated global params ${\theta }^{\left(R\right)}$   1: for $r=1$ to R do   2:     Server sends ${\theta }^{(r-1)}$ to clients   3:     for all $c\in {\mathcal{S}}^{\left(r\right)}$ in parallel do   4:         ${\theta }_c\leftarrow {\theta }^{(r-1)}$; $n_c\leftarrow 0$   5:         for $e=1$ to E do   6:            Collect trajectory $(s_t,a_t,r_t)$   7:            Compute discounted RTG; normalize ${\widehat{G}}_t$   8:            Store $(s_t,a_t,r_t,{\widehat{G}}_t)$ in buffer   9:            Sample subsequences of length L with padding/masks 10:            Train DT with teacher forcing; update ${\theta }_c$ 11:            $n_c\leftarrow n_c+$ episode length 12:            if $emodK=0$ then 13:                Send $({\Delta }_c,n_c)$ to server; reset $n_c\leftarrow 0$ 14:            end if 15:         end for 16:     end for 17:     Server aggregates (Algorithm 2) 18: end for

Algorithm 2 Server Aggregation with Privacy and Robustness

Require: Global params ${\theta }^{(r-1)}$; client updates $\left\{({\Delta }_c,n_c)\right\}$ Ensure: Updated params ${\theta }^{\left(r\right)}$   1: for all$({\Delta }_c,n_c)$do   2:     Optionally decompress, unmask, clip, or mask adapters   3: end for   4: Compute weights $w_c=n_c/{\sum }_j{n}_j$   5: if robust aggregator enabled then   6:     $\tilde{\Delta }\leftarrow \mathsf{Robust}(\left\{{\Delta }_c\right\},\left\{w_c\right\})$   7: else   8:     $\tilde{\Delta }\leftarrow {\sum }_c{w}_c{\Delta }_c$   9: end if 10: ${\theta }^{\left(r\right)}\leftarrow {\theta }^{(r-1)}+\tilde{\Delta }$ 11: Broadcast ${\theta }^{\left(r\right)}$ to clients

3.3.2. Explanation of Algorithm 2

Algorithm 2 describes the aggregation on the server-side in each federated round. Upon receiving client updates ${\Delta }_c$ and their sample counts $n_c$, the server first applies optional preprocessing: decompression of quantized/sparse updates, unmasking for secure aggregation, masking to retain only adapter parameters in PEFT, and ${\ell }_2$-norm clipping to bound sensitivity. Differential privacy (DP) accounting can be maintained if Gaussian noise is added on the client side. The server then computes sample size weights $w_c=n_c/{\sum }_j{n}_j$ to balance non-IID client contributions. If robustness is required, the server replaces FedAvg with robust aggregators such as coordinate-wise median, trimmed mean, or Krum, thus mitigating malicious or outlier updates. Finally, the global model is updated to ${\theta }^{\left(r\right)}={\theta }^{(r-1)}+\tilde{\Delta }$, with an optional exponential moving average (EMA) to reduce variance. The updated parameters are broadcast to the clients for the next round.

3.3.3. Complexity

The server-side complexity is linear in the number of participants,

$$O\left(|{\mathcal{S}}^{\left(r\right)}|·|\theta |\right),$$

since each round requires aggregating updates on $|{\mathcal{S}}^{\left(r\right)}|$ clients for a model of size $\left|\theta \right|$. Optional robust aggregation (e.g., Krum) introduces an additional $O\left(\right|{\mathcal{S}}^{\left(r\right)}{|}^2)$ distance computation but remains practical for typical client counts. The communication cost is $O\left(\right|{\mathcal{S}}^{\left(r\right)}|·|\theta \left|\right)$ per round (downlink and uplink), which can be reduced by compression or PEFT.

4. Comparative Analysis with MAAC

To evaluate the effectiveness of our proposed framework, we compare it with the MAAC algorithm, a widely adopted baseline in MARL. MAAC supports cooperative learning across agents through a centralized critic, but this design introduces several limitations: (i) communication bottlenecks due to frequent gradient exchange, (ii) privacy risks arising from centralized data aggregation, and (iii) limited ability to capture long-term temporal dependencies as the critic is typically coupled with recurrent encoders.

Our proposed FDT addresses these issues by embedding self-attention-based sequence modeling within a federated learning loop. Unlike MAAC, which relies on a global critic for coordination, the FDT enables each agent to learn decentralized trajectory-based policies while keeping raw data local. This architecture improves scalability, robustness, and privacy preservation while eliminating the single point of failure associated with centralized critics.

From a coordination perspective, MAAC relies on explicit synchronization through its centralized critic, which computes joint value estimates based on global observations. This explicit mechanism requires agents to share intermediate representations or gradients, creating potential bottlenecks and privacy concerns. In contrast, the FDT achieves coordination implicitly: each agent conditions its policy on locally normalized return-to-go (RTG) trajectories that reflect cumulative future rewards. Because these normalized returns are aligned across agents through federated aggregation, coordination emerges naturally without centralized value propagation. This implicit synchronization mechanism allows the FDT to maintain coherent cooperative behavior even in non-IID and privacy-constrained environments, providing a key advantage over critic-based architectures.

Baseline Comparison Scope

Our quantitative evaluation focuses on the MAAC baseline, implemented using the publicly available codebase of the authors to ensure reproducibility with identical hyperparameter settings. Although other attention-based MARL methods, such as Actor–Attention–Critic (AAC) [12], the multi-agent transformer (MAT) [21], and the Multi-Agent Decision Transformer (MADT) [22], represent important advances; they are considered here at a conceptual level rather than re-implemented. These approaches generally assume centralized or offline training pipelines, which differ from the federated MEC environment targeted in this work. A detailed review of these methods and their limitations is provided in Section 2. Beyond MAAC, recent attention-based baselines such as AAC [12], MAT [21], and MADT [22] represent important advances in multi-agent learning. However, these approaches generally assume centralized or offline training pipelines and do not directly address federated coordination or privacy-preserving constraints. For this reason, we discuss them conceptually but do not re-implement them in this work. A direct empirical comparison with these methods remains an interesting avenue for future investigation.

Table 1. Comparison of MAAC (centralized) and FDT (federated, decentralized) across algorithmic and system-level aspects.

Dimension	MAAC (Centralized)	FDT (Federated, Decentralized)
Learning Architecture	Actor with centralized critic; coordination depends on critic access	Critic-free; decentralized trajectory modeling via self-attention
Temporal Dependencies	Captured through RNN/LSTM encoders; prone to vanishing gradients	Captured through multi-head self-attention; robust to long horizons
Training Paradigm	Centralized training with global critic gradients	Federated local training with periodic model aggregation
Scalability	Critic bottleneck limits performance in large networks	Horizontally scalable; resilient to node/server drop-outs
Privacy and Data Flow	Raw observations may be shared with critic	Raw data remains local; only encrypted model updates exchanged
Communication Overhead	Frequent critic gradient synchronization	Lightweight periodic updates (every K episodes)

summarizes the key differences between MAAC and our proposed FDT on the algorithmic and system-level dimensions.

In the following sections, we detail our experimental setup, evaluation metrics (decision accuracy, convergence speed, communication overhead, and security robustness), and comparative results against MAAC baselines.

5. Experimental Setup

To provide a fair and reproducible evaluation, both MAAC and the proposed FDT were trained under identical environmental conditions. The MAAC baseline was implemented using the publicly available code from the original paper’s GitHub repository (Original MAAC implementation: https://github.com/shariqiqbal2810/MAAC, accessed on 15 October 2025), while our FDT model was adapted to operate under the same conditions, allowing a direct comparison between the two approaches.

Computational Environment. Experiments were carried out on an Apple Mac system equipped with an Apple M1 Max processor, 64 GB of unified RAM and running MacOS Sequoia 15.3.1. The software environment included Python 3.9, TensorFlow 2.9, and PyTorch 2.0.

Training Procedure. The MAAC model is trained using its original implementation, whereas the FDT model employs our modified Decision Transformer integrated with federated learning. Both models were trained in identical environments and number of episodes, ensuring that performance differences arise solely from architectural innovations.

Hyperparameters. To eliminate confounding factors, both models use identical hyperparameter settings, as summarized in

Table 2. Experimental parameters for MAAC and transformer-based RL training.

Parameter	MAAC	Transformer RL
Number of Agents	10	10
Training Episodes	50,000	50,000
Batch Size	64	64
Discount Factor ($\gamma$)	0.99	0.99
Learning Rate (Actor)	0.0005	0.0005
Learning Rate (Critic)	0.0005	0.0005
Optimizer	Adam	Adam
Exploration Strategy	$ϵ$-greedy (decay from 1.0 to 0.1)	$ϵ$-greedy (decay from 1.0 to 0.1)
Target Update Frequency	Every 100 steps	Every 100 steps
Communication Frequency (FL)	N/A	Every 10 episodes
Replay Buffer Size	1,000,000	1,000,000
Environment Type	MEC Simulation	MEC Simulation

. These include learning rates, discount factors, batch sizes, and training durations.

5.1. Multi-Agent Environment

We employ a synthetic multi-agent environment inspired by distributed IoT/MEC systems, where each agent controls localized decisions and interacts with others through a shared global state. States include abstracted features such as task queues, waiting times, current decision phase, and short-horizon demand flows. Actions correspond to either maintaining the current policy or switching to an alternative configuration under safety constraints. Rewards penalize inefficiency in terms of delay, queue buildup, and communication overhead while encouraging balanced throughput across agents. For the FDT, trajectories are tokenized as $[{\mathrm{RTG}}_t,s_t,a_{t-1}]$ with sequence length $L\in \{20,30\}$. The evaluation reports performance in terms of reward dynamics, convergence speed, scalability, and communication cost (bytes/round and bytes-to-convergence), averaged over five seeds.

5.2. Federated Setup

Agents are grouped by region into M clients (default $M=8$; also 16/32). Each round, clients run $K=10$ local episodes and upload model deltas; the server aggregates with FedAvg (FedProx $\mu =0.001$ in ablation). We studied non-IID demand by assigning distinct load profiles per region and varying the network scale (16/36/64 agents).

With this unified setup, we ensure that any observed performance differences reflect the architectural characteristics of MAAC and the FDT rather than experimental inconsistencies. The next section presents the results of this comparative evaluation and discusses their implications for scalability, privacy, and robustness in smart city environments.

6. Results and Discussion

The experiments evaluated how MAAC and the FDT process sequential decision-making tasks and adapt to dynamic multi-agent environments. Performance was assessed in terms of decision accuracy, convergence speed, scalability, communication overhead, and robustness.

6.1. Learning Behavior

Figure 1 illustrates the distribution of agent decision strategies. The FDT model promotes broader exploration, with agents spread throughout the environment, indicating higher adaptability. In contrast, MAAC exhibits denser agent clustering around predefined coordination zones, reflecting stronger cooperative stability but limited flexibility in dynamic settings.

6.2. Reward Efficiency and Stability

Figure 2 compares reward distributions. MAAC achieves stable performance, with most rewards concentrated in the 10–12 range, consistent with its critic-driven cooperative learning. The FDT surpasses MAAC in peak performance, achieving rewards above 22 but with wider variance. This highlights a trade-off: the FDT improves adaptability and long-horizon learning but introduces greater variability during early training.

The high performance variance observed in the FDT can be attributed to its trajectory-based credit assignment: long-horizon RTG conditioning introduces stochasticity in sequence sampling and gradient updates. In early training, this leads to greater fluctuations compared to the centralized critic in MAAC. However, variance gradually decreases as more trajectories are aggregated. This highlights a trade-off between adaptability and stability, suggesting that future optimizations could integrate variance-reduction strategies such as reward normalization, adaptive RTG scaling, or hybrid critic-transformer models.

6.3. Scalability

Figure 3 reports the mean reward for episodes as the number of agents increases. The FDT consistently maintains higher performance across larger agent populations, demonstrating robustness to increasing system complexity. MAAC performance degrades as the number of agents increases, confirming that its centralized critic becomes a bottleneck in large-scale settings. These findings validate the scalability advantage of federated self-attention models in smart city contexts where agent participation is highly dynamic.

The consistent improvements observed across 16, 36, and 64 agents indicate that the FDT scales effectively as system complexity grows. These trends suggest that the FDT can extend naturally to even larger agent populations, and future studies could investigate networks with hundreds of agents to further substantiate scalability in more complex environments.

6.4. Comparative Summary

In general, the results reveal complementary strengths. MAAC excels in structured cooperative tasks, where stability and low variance are paramount. In contrast, the FDT provides superior scalability, adaptability, and privacy preservation, albeit with greater performance variance. This trade-off suggests that hybrid approaches that combine the stability of MAAC with the adaptability of the FDT could be a promising direction for future research.

6.5. Implications for Smart Cities

For practical deployment in smart city environments, the FDT offers significant advantages: (i) horizontal scalability supports dynamic IoT infrastructures, (ii) privacy is preserved by design through federated aggregation, and (iii) robustness is improved through decentralization, reducing single points of failure. These characteristics make the FDT particularly suitable for mobility, energy management, and anomaly detection tasks where adaptability and privacy are critical. It should be noted that the reported results stem from a synthetic multi-agent environment rather than domain-specific traffic datasets. Validating the FDT on real-world applications such as CityFlow or SUMO remains an important direction for future research.

• Feasibility and Deployment Considerations. Practical IoT deployment introduces several operational constraints. Communication overhead is mitigated by transmitting compressed model deltas instead of raw gradients, combined with quantization and structured pruning to further reduce bandwidth requirements. Computational latency can be alleviated by partially offloading transformer inference to regional servers in a hierarchical FL architecture. Energy consumption challenges in battery-powered IoT devices are addressed through lightweight transformer variants (e.g., TinyBERT and MobileFormer) and adaptive client participation strategies. Together, these mechanisms enable scalable and energy-efficient implementations of the FDT across resource-constrained environments.
• Scalability and Component Sensitivity. To better understand the FDT’s performance sources, future work should include ablation studies on the transformer encoder, RTG conditioning, and aggregation mechanisms. This would clarify the contribution of each component, improve interpretability, and reinforce the FDT’s scalability under diverse network and workload conditions.
• Resilience and Cybersecurity. Beyond efficiency, the FDT inherently mitigates several key challenges in federated multi-agent learning. Client heterogeneity is handled through local RTG normalization and adaptive weighting, ensuring stable convergence across non-IID clients. Communication bottlenecks are alleviated through compressed delta updates, partial participation, and compatibility with quantization and structured pruning for bandwidth efficiency. Security and privacy vulnerabilities are mitigated using differential privacy, secure aggregation, and robust aggregation methods such as trimmed mean and Krum, which protect against poisoning and Byzantine attacks. Recent studies also emphasize blockchain-based auditability and trust management to strengthen security in smart city infrastructures [2,16,30,45,46]. Together, these strategies enhance the reliability, robustness, and trustworthiness of large-scale FDT deployments in urban IoT ecosystems.

7. Conclusions

This study compared the MAAC model with a transformer-based reinforcement learning framework integrated into a Federated Decision Transformer (FDT) for smart city applications. The experimental results showed that MAAC provides stable cooperative decision-making through centralized critics but suffers from scalability limitations in dynamic environments. In contrast, the FDT leverages self-attention and decentralized training to achieve higher adaptability, scalability, and reward efficiency, although at the cost of increased performance variance.

The key contribution of this work lies in the integration of Decision Transformers within a federated multi-agent framework, enabling decentralized coordination without a centralized critic. Unlike existing federated reinforcement learning or transformer-based MARL studies, the FDT unifies temporal sequence modeling with privacy-preserving learning, addressing scalability, non-IID data, and communication efficiency simultaneously. This dual innovation in architectural design and training paradigm advances the state of the art in scalable, privacy-aware decision intelligence for smart city IoT systems.

The findings highlight a clear trade-off: MAAC is well-suited for structured cooperative tasks requiring stability, whereas the FDT excels in decentralized, evolving environments where scalability and privacy are critical. These complementary strengths suggest that hybrid approaches that combine the stability of MAAC with the adaptability of the FDT represent a promising avenue for future research.

Future work will extend the FDT with variance-reduction strategies, modular aggregation schemes, and real-world benchmarking on traffic and energy datasets to further validate its generalizability and deployment readiness in smart city environments [47,48,49].