Is More Always Better? Fundamental Trade-Offs Between Information Richness and Dynamic Stability for Heterogeneous Multimodal Communication in 6G Distributed Systems

Abstract

In distributed wireless systems enabled by multimodal sensing and semantic communication, heterogeneous modalities provide richer task-relevant information but simultaneously introduce asynchronous updates due to modality-dependent sensing, processing, and transmission delays. How such asynchronous multimodal communication affects the dynamic stability of the overall system remains insufficiently understood. This paper investigates the fundamental relationship between semantic information richness and system stability in distributed multimodal wireless communication. To this end, a unified system model is developed in which multimodal semantic streams are embedded into the continuous-time task state evolution, explicitly accounting for heterogeneous delays and MIMO transmission effects. The resulting system is formulated as a delay-coupled dynamic model, and Lyapunov–Krasovskii analysis is employed to characterize the stability conditions under asynchronous semantic updates. An explicit stability perturbation metric is derived to quantify how delayed multimodal information influences system dynamics. The analysis reveals a fundamental trade-off between semantic information richness and dynamic stability: incorporating additional modalities does not always improve system performance and may destabilize the system when asynchronous delays accumulate. Based on this insight, a stability-aware optimization framework is proposed to regulate modality participation and precoding design. The results highlight the necessity of stability-driven multimodal communication design for future distributed wireless and 6G intelligent systems.

1. Introduction

With the deep integration of 5G/6G technologies and the digital economy, wireless communication systems for Artificial intelligence of Things (AIoT) are rapidly evolving toward space–air–ground–sea integrated architectures [1]. In emerging application scenarios such as smart grids, low-altitude transportation, and cooperative unmanned systems, distributed devices are required to continuously perceive their surrounding environments and execute complex sensing, decision-making, and control tasks [2,3]. These tasks are inherently multimodal, as they rely on heterogeneous sensing sources—including vision, acoustics, radar, and wireless signals—each capturing complementary aspects of the environment [4,5].

Therefore, the performance of a single device no longer depends solely on its local sensing capabilities, but increasingly on its ability to exchange task-related modal information with other devices via wireless links [6]. Multimodal wireless communication technology arose precisely to address the growing demand for high-quality multimodal information from devices [7], thus undoubtedly becoming a fundamental enabling technology for large-scale intelligent systems. In multimodal wireless communication, wireless transmission is tightly coupled with multimodal sensing and task execution, rather than merely serving as a transparent data conduit [8].

In recent years, semantic communication has emerged as a promising paradigm, breaking away from traditional bit-level transmission modes and focusing on delivering task-related modal information [9]. This significantly reduces data volume and enables real-time communication, thus ensuring the effectiveness of multimodal data [10,11]. Increasing research is dedicated to multimodal semantic communication and cross-modal collaboration, encoding observation data from different sensing modalities or devices into compact semantic representations and transmitting them via wireless links [12]. Most existing semantic communication work employs end-to-end trained semantic encoding and decoding frameworks, focusing on receiver model architecture design, semantic representation learning, and cross-modal feature alignment [13,14,15]. This research paradigm typically assumes that once semantic information from multiple modalities or devices is successfully received and aligned, its integration will continuously yield positive benefits, thereby improving task performance or user experience.

However, most of the studies implicitly treat semantic communication as a one-shot or quasi-synchronous information fusion process, where the primary concern lies in how to align and fuse semantic representations that have already arrived at the receiver [16,17,18]. In practical distributed systems, semantic communication instead operates as a continuous process, in which task-related semantic updates are repeatedly exchanged and injected into the system decision loop [19]. Due to the inherent randomness of wireless communication, such semantic updates inevitably experience heterogeneous transmission delays caused by channel variations, queuing, retransmissions, and resource scheduling [20]. Meanwhile, multimodal sensing systems themselves evolve over different time scales and update frequencies [21]. When semantic updates originating from heterogeneous sensing and communication processes are continuously incorporated into system decision-making, the resulting behavior can no longer be captured by static semantic alignment alone, but must be understood as the evolution of a dynamic system driven jointly by multimodal states and delayed communication updates.

This observation gives rise to a more fundamental yet underexplored question:

1.

The overall system communication latency introduced by multimodal wireless semantic communication is not dynamically characterized.

2.

When semantic information, affected by non-negligible communication latency, is repeatedly injected into the system state evolution process, will this communication-induced asynchronicity accumulate over time and affect the dynamic stability of the system?

3.

In long-term distributed tasks, how should communication and device control systems balance the richness of semantic information and dynamic stability, and determine which semantic updates should be utilized promptly, delayed, or even suppressed?

These questions reveal a fundamental shift in design requirements: the core challenge in such distributed systems lies in the dynamic coupling between multimodal semantic flows and system-level control laws. Conventional semantic communication paradigms prioritize static reconstruction fidelity; however, in continuous feedback loops, unmanaged heterogeneous delays can trigger error accumulation and, ultimately, system divergence. This necessitates a shift toward a stability-centric design that internalizes communication impairments into task dynamics, transforming the resource allocation problem into a real-time scheduling task within a rigorous stability boundary.

Guided by this logic, we propose the Stability-Aware Multimodal Semantic Communication (SA-MSC) framework. This framework treats semantic communication as a closed-loop dynamic process, with its core contributions summarized as follows:

1.

Stability-Centric System Modeling: We establish a unified framework that embeds semantic latencies and device resource states into system dynamics, providing a theoretical foundation for the stability analysis of multimodal semantic systems.

2.

Stability-Richness Trade-off Analysis: We derive a stability-aware perturbation metric to quantify the impact of asynchronous multimodal injection, enabling the joint optimization of modality participation and precoding matrices.

3.

Adaptive Implementation Protocol: We enhance the LA-MSC architecture with a stability-aware control module and a history-augmented Knowledge Base (LKB), allowing the system to dynamically prioritize or suppress semantic updates based on real-time task requirements.

By integrating these components, the SA-MSC framework effectively bridges the gap between multimodal semantic extraction and dynamic stability control. It ensures that the system can maximize semantic richness while maintaining rigorous stability guarantees, providing a robust solution for resilient 6G AIoT applications in complex, heterogeneous environments.

2. Related Works

To position this work within the evolving landscape of 6G-AIoT, this section examines research from three dimensions: distributed communication architectures, collaborative semantic alignment, and the spatio-temporal heterogeneity of multimodal data. Our goal is to build upon these established foundations while addressing the specific challenges of system-level stability illustrated in Section 1.

2.1. Distributed Multimodal Wireless Architectures

The deployment of AIoT systems relies heavily on distributed communication frameworks. Reference [7] first conceptualized multimodal wireless paradigms, and reference [22] successfully optimized distributed semantic systems for resource-constrained devices. Building on this, decentralized networks in [23,24] have significantly improved sensing coverage and deployment feasibility. While these architectures have achieved remarkable transmission efficiency and QoS improvements [25], their primary focus remains on data-level performance. A promising further direction, which this paper explores, is the integration of stability modeling to ensure that communication gains directly translate into control-loop reliability in feedback-driven tasks.

2.2. Collaborative Semantic Feature Alignment

To enable effective multimodal interaction, research has transitioned from unimodal JSCC [26,27,28,29] toward complex feature alignment. Notable progress includes unifying multimodal semantics into joint spaces [30] and leveraging LLMs for high-dimensional semantic mapping [31]. To further integrate diverse functionalities, the SIMAC framework [32] introduces a semantic-driven integrated sensing and communication architecture, utilizing cross-attention and LLM-based encoding to achieve task-oriented multimodal fusion. Similarly, in specialized forecasting tasks, Reference [33] utilizes an optimized GPT-2 architecture to align temporal load data with textual information, demonstrating the superior generalization of LLMs in cross-modal fusion. Furthermore, in the domain of distributed collaborative learning, Reference [34] incorporates fuzzy logic into neural architectures to represent data ambiguities, while Reference [35] proposes a lightweight decoupled modeling approach to address the constraints of edge platforms. These studies provide essential tools for semantic validity; however, they typically operate under a “quasi-synchronous” assumption where features are processed as static inputs. As highlighted by the alignment methods in [36], while the semantic accuracy of individual packets is well-optimized, the impact of asynchronous arrival on system-wide dynamic stability remains an open area of investigation that our work aims to address.

2.3. Spatio-Temporal Heterogeneity of Multimodal Data

Complementary to the algorithmic alignment discussed in Section 2.2, this subsection examines the physical characteristics of multimodal streams. Current surveys [37] provide a comprehensive overview of fusion methods, and reference [38] has introduced innovative solutions for temporal alignment. Specifically, in the field of video processing, Reference [39] emphasizes the importance of capturing joint spatio-temporal information across neighboring frames to restore details in dynamic scenes. These methods have laid a solid foundation for multimodal processing. Nevertheless, in continuous AIoT environments, the interplay between the inherent periodicity of sensing and the randomness of wireless reception creates a complex temporal heterogeneity. Current research is increasingly moving toward handling such continuity, and our SA-MSC framework contributes to this trend by treating temporal reception as a driving factor for state-triggered control.

2.4. Summary and Comparison

To synthesize the research landscape discussed above, we contrast our SA-MSC framework with mainstream research paradigms in Table 1.

Table 1. Comparison of the proposed SA-MSC framework with existing literature.

Research Focus	Multimodal	Heterogeneous Delay	Stability	Representative Works
Distributed Architecture	Partial	Low	×	[7,22,23,24,25]
Collaborative Semantic Communication	Partial	Medium	×	[26,27,28,29,30,31,33,34,35,36]
Multimodal Data Fusion	✓	High	×	[37,38,39]
Ours (SA-MSC)	✓	High	✓	This study

3. System Model

3.1. Distributed Multimodal Communication Model and Problem Statement

As shown in Figure 1, we consider a distributed wireless system consisting of a group of cooperating devices $\mathcal{K}=\{1,2,\dots ,K\}$. In many high-precision cooperative tasks, such as drone obstacle avoidance or the force control of electro-hydrostatic actuators [40], the system must maintain strict operational accuracy despite environmental disturbances. To quantify this, each device $k\in \mathcal{K}$ is associated with a time-varying state vector ${\mathit{q}}_k\left(t\right)\in {\mathbb{R}}^{n_k}$, which represents the task deviation or control error that needs to be minimized at time t. The key notations, parameters, and variables used throughout this paper are summarized in Table A1 of Appendix A.

Figure 1. Distributed AIoT system task execution scenarios and multimodal wireless communication methods between devices.

Initially, the task dynamics of device k are driven by environmental information received by its own multimodal receiver. Let ${\mathcal{M}}_k^{\mathrm{loc}}$ denote the set of local sensing modalities available to device k. The local multimodal information at time t is represented as ${\left\{{\mathit{m}}_{k,i}\left(t\right)\right\}}_{i\in {\mathcal{M}}_k^{\mathrm{loc}}}$. In many practical scenarios, such as complex industrial environments or occluded urban areas, local information may be insufficient to maintain the stability or convergence speed required for ${\mathit{q}}_k\left(t\right)$.

In complex scenarios, such as severe environmental occlusion, the local information set ${\left\{{\mathit{m}}_{k,i}\left(t\right)\right\}}_{i\in {\mathcal{M}}_k^{\mathrm{loc}}}$ alone may fail to satisfy the observability requirements for system stability, leading to the potential divergence of the task state ${\mathit{q}}_k\left(t\right)$. To counteract this, device k incorporates remote semantic assistance from neighboring devices. Specifically, we adopt the LA-MSC model [31] which consists Multimodal Alignment (MMA) and LLM-based Knowledge Based (LKB) as the foundational architecture for semantic interaction, and use ${\mathcal{E}}_{\mathcal{l}k,i}(·)$ to represent this process. For each modality $i\in \mathcal{M}$, the remote device ℓ extracts a compact semantic embedding via the LA-MSC-empowered encoder:

$${\mathit{s}}_{\mathcal{l}k,i}\left(t\right)=a_{\mathcal{l}k,i}{\mathcal{E}}_{\mathcal{l}k,i}\left({\mathit{m}}_{\mathcal{l},i}\left(t\right)\right)\in {\mathbb{R}}^{d_{\mathcal{l}k,i}},$$

(1)

where $a_{\mathcal{l}k,i}\in \{0,1\}$ is the modality selection variable, and $d_{\mathcal{l}k,i}$ denotes the semantic dimension which determines the information richness.

3.1.1. Multimodal Wireless Communication and Latency Characterization

We refer to the above process of exchanging task-oriented multimodal semantic information ${\mathit{s}}_{\mathcal{l}k,i}\left(t\right)$ between devices ℓ and k as multimodal wireless communication.

To achieve multimodal wireless communication as well as ensuring the fundamental stability of ${\mathit{q}}_k\left(t\right)$, we employ Multiple-Input Multiple-Output (MIMO) communication, utilizing spatial multiplexing to simultaneously transmit different semantic streams on parallel spatial channels. Therefore, consider a MIMO link where device ℓ and device k are equipped with $N_t$ transmit antennas and $N_r$ receive antennas, respectively. The wireless environment is characterized by the MIMO channel matrix ${\mathit{H}}_{\mathcal{l}k}\in {\mathbb{C}}^{N_r\times N_t}$, and we use $r_{\mathcal{l}k}=\mathrm{rank}\left({\mathit{H}}_{\mathcal{l}k}\right)$ to denote the number of available spatial streams. By applying the linear precoding matrix ${\mathit{W}}_{\mathcal{l}k}\in {\mathbb{C}}^{N_t\times r_{\mathcal{l}k}}$ and the combiner ${\mathit{U}}_k\in {\mathbb{C}}^{N_r\times r_{\mathcal{l}k}}$, the effective semantic channel matrix is defined as [41]:

$${\mathbf{H}}_{\mathrm{eff}}={\mathit{U}}_k^{\mathrm{H}}{\mathit{H}}_{\mathcal{l}k}{\mathit{W}}_{\mathcal{l}k}\in {\mathbb{C}}^{r_{\mathcal{l}k}\times r_{\mathcal{l}k}}.$$

(2)

To transmit the extracted semantics over the MIMO link, the remote device ℓ maps the selected semantic embeddings ${\left\{{\mathit{s}}_{\mathcal{l}k,i}\right\}}_{a_{\mathcal{l}k,i}=1}$ into the complex baseband domain. Following the end-to-end semantic-to-physical mapping framework [42], and by applying the linear precoding matrix ${\mathit{W}}_{\mathcal{l}k}\in {\mathbb{C}}^{N_t\times r_{\mathcal{l}k}}$, the transmitted signal is given by:

$${\mathit{x}}_{\mathcal{l}k}={\mathit{W}}_{\mathcal{l}k}{\mathcal{G}}_{\mathcal{l}k}\left({\left\{a_{\mathcal{l}k,i}{\mathit{s}}_{\mathcal{l}k,i}\right\}}_{i\in \mathcal{M}}\right)\in {\mathbb{C}}^{N_t},$$

(3)

where ${\mathcal{G}}_{\mathcal{l}k}(·)$ is the joint semantic mapper. Upon transmission through the wireless channel ${\mathit{H}}_{\mathcal{l}k}$, the received signal ${\mathit{y}}_k$ at device k is expressed as:

$${\mathit{y}}_k={\mathit{H}}_{\mathcal{l}k}{\mathit{x}}_{\mathcal{l}k}+{\mathit{n}}_k={\mathit{H}}_{\mathcal{l}k}{\mathit{W}}_{\mathcal{l}k}{\mathcal{G}}_{\mathcal{l}k}\left({\left\{a_{\mathcal{l}k,i}{\mathit{s}}_{\mathcal{l}k,i}\right\}}_{i\in \mathcal{M}}\right)+{\mathit{n}}_k,$$

(4)

where ${\mathit{n}}_k\sim \mathcal{CN}(\mathbf{0},{\sigma }^2\mathit{I})$ is the additive white Gaussian noise. To recover the semantic information, as illustrated in (2), device k applies ${\mathit{U}}_k\in {\mathbb{C}}^{N_r\times r_{\mathcal{l}k}}$, yielding the processed semantic stream:

$${\widehat{\mathit{s}}}_{\mathcal{l}k}={\mathit{U}}_k^{\mathrm{H}}{\mathit{y}}_k={\mathit{U}}_k^{\mathrm{H}}{\mathit{H}}_{\mathcal{l}k}{\mathit{W}}_{\mathcal{l}k}{\mathcal{G}}_{\mathcal{l}k}(·)+{\mathit{U}}_k^{\mathrm{H}}{\mathit{n}}_k.$$

(5)

To derive the achievable transmission rate, we first analyze the composition of the processed signal ${\widehat{\mathit{s}}}_{\mathcal{l}k}$ in (5). Given the normalized semantic signal covariance [43] ${\mathit{R}}_g=\mathbb{E}\left[{\mathcal{G}}_{\mathcal{l}k}{\mathcal{G}}_{\mathcal{l}k}^{\mathrm{H}}\right]=\mathbf{I}$ and the filtered noise covariance ${\mathbf{R}}_{\tilde{n}}=\mathbb{E}\left[{\tilde{\mathit{n}}}_k{\tilde{\mathit{n}}}_k^{\mathrm{H}}\right]={\sigma }^2\mathbf{I}$, the effective signal-to-noise ratio (SNR) matrix of the MIMO semantic channel is expressed as:

$${\mathbf{SNR}}_{\mathcal{l}k}={\mathbf{R}}_{\tilde{n}}^{-1}\left({\mathbf{H}}_{\mathrm{eff}}{\mathbf{R}}_g{\mathbf{H}}_{\mathrm{eff}}^{\mathrm{H}}\right)={\displaystyle \frac{1}{{\sigma }^2}}{\mathbf{H}}_{\mathrm{eff}}{\mathbf{H}}_{\mathrm{eff}}^{\mathrm{H}}.$$

(6)

Based on the Shannon-Hartley theorem for multi-stream transmission, the achievable rate $R_{\mathcal{l}k}$ is determined by the log-determinant of the SNR-augmented identity matrix, representing the aggregate capacity of $r_{\mathcal{l}k}$ parallel spatial layers:

$$\begin{array}{cc}\hfill R_{\mathcal{l}k}& =B{log}_2\det \left(\mathbf{I}+{\mathbf{SNR}}_{\mathcal{l}k}\right)\hfill \\ \hfill & =B{log}_2\det \left(\mathbf{I}+{\displaystyle \frac{1}{{\sigma }^2}}{\mathbf{H}}_{\mathrm{eff}}{\mathbf{H}}_{\mathrm{eff}}^{\mathrm{H}}\right),\hfill \end{array}$$

(7)

where B denotes the communication bandwidth.

For the entire process described above, we believe that the end-to-end latency for modality m formed as $T_{\mathcal{l}k,m}$ is caused by the interaction is due to device l’s processing latency $T_{\mathcal{l}k,m}^{\mathrm{proc}}$ for the modality and the communication latency $T_{\mathcal{l}k,m}^{\mathrm{comm}}$. $T_{\mathcal{l}k,m}^{\mathrm{comm}}$ can be determined based on the transmission information dimension $d_m$ in (1) and the transmission rate $R_{\mathcal{l}k}$ in (7). Therefore, $T_{\mathcal{l}k,m}$ can be expressed as:

$$T_{\mathcal{l}k,m}=T_{\mathcal{l}k,m}^{\mathrm{proc}}+{\displaystyle \frac{a_{\mathcal{l}k,m}{d}_{\mathcal{l}k,m}}{R_{\mathcal{l}k}\left({\mathit{W}}_{\mathcal{l}k}\right)}}.$$

(8)

Consequently, the task state ${\mathit{q}}_k\left(t\right)$ is updated through the integration of real-time local sensing and these delayed remote semantics. Specifically, the available information set at time t is given by

$${\mathcal{I}}_k\left(t\right)={\left\{{\mathit{m}}_{k,i}\left(t\right)\right\}}_{i\in {\mathcal{M}}_k^{\mathrm{loc}}}\cup {\left\{{\mathit{s}}_{\mathcal{l}k,m}(t-T_{\mathcal{l}k,m})\right\}}_{\mathcal{l}\in \mathcal{K}\setminus \left\{k\right\},a_{\mathcal{l}k,m}=1}.$$

(9)

And the augmented task dynamics are governed by the following system:

$${\dot{\mathit{q}}}_k\left(t\right)=f_k\left({\mathit{q}}_k\left(t\right),{\mathcal{I}}_k\left(t\right)\right).$$

(10)

3.1.2. Problem Analysis and Statement

The evolution of the task state ${\mathit{q}}_k\left(t\right)$ in (10) reveals a fundamental coupling between communication resources and control stability. To clarify the underlying physical insights before formulating the optimization problem, we provide the following remark.

Remark 1.

The interaction between multimodal sensing and MIMO transmission leads to two critical observations:

• Richness and Freshness: Activating more remote modalities ($a_{\mathcal{l}k,m}=1$) increases the information richness available to the task, but simultaneously consumes more bandwidth, leading to larger asynchronous delays $T_{\mathcal{l}k,m}$.
• Stability Sensitivity: Modalities with high sensitivity $C_{\mathcal{l}k,m}$ are “double-edged swords”; they provide critical updates for stability but also induce severe perturbations if delayed beyond the system’s tolerance.

Based on these insights, the collaborative system faces two intertwined challenges:

• C1: At the control layer of the distributed system, we need to think how to dynamically select the optimal modality set $\left\{a_{\mathcal{l}k,m}\right\}$ to maximize information gain without violating the stability boundary.
• C2: At the wireless communication layer, we need to think How to design the precoding matrices $\left\{{\mathit{W}}_{\mathcal{l}k}\right\}$ to adaptively compress the latencies of the most sensitive semantic streams under power constraints.

To address these two intertwined challenges, we propose a joint modality-and-precoding optimization framework. Specifically, we develop a stability-aware modality selection scheme to tackle C1 by pruning non-essential information, and a delay-optimal MIMO precoding strategy to resolve C2 by adaptively shaping the communication resources.

3.2. Stability-Oriented Dynamic System Representation

To address two Challenges as illustrated in Section 3, we first need to focus on two key areas:

1.

How to quantify the stability boundaries of the entire distributed system.

2.

After quantifying the boundaries, how to implement modal control.

Therefore, this section will present a complete model of a distributed control system and quantify its disturbance boundary. Based on this boundary, a stability control method will then be proposed.

As shown in Figure 2, We first characterize the stability boundaries of the system. Let the aggregated task state vector of all K devices at time t be defined as:

$$\mathit{q}\left(t\right)={\left[{\mathit{q}}_1^T\left(t\right),{\mathit{q}}_2^T\left(t\right),\dots ,{\mathit{q}}_K^T\left(t\right)\right]}^T\in {\mathbb{R}}^{nK}.$$

(11)

Figure 2. Schematic diagram of global dynamics analysis of distributed system.

In our distributed multimodal framework, semantic communication acts as a form of delayed feedback within the task execution loop. Due to the asynchronous transmission delays $\left\{T_{\mathcal{l}k,i}\right\}$ induced by heterogeneous modalities and MIMO resource constraints, the update of $\mathit{q}\left(t\right)$ depends not only on instantaneous local sensing but also on these delayed semantic streams. When such delayed updates are repeatedly incorporated, they may accumulate and fundamentally threaten the dynamic stability of the system. Consequently, the distributed multimodal task execution process can be viewed as a continuous-time delay differential system, compactly represented as:

$$\dot{\mathit{q}}\left(t\right)=\mathcal{F}\left(\mathit{q}\left(t\right),{\left\{\mathit{q}(t-T_{\mathcal{l}k,i})\right\}}_{\mathcal{l},k,i}\right),$$

(12)

where $T_{\mathcal{l}k,i}$ is the specific delay for modality i from device ℓ to k, and the function $\mathcal{F}(·)$ is determined by the task dynamics $f_k(·)$ and the semantic mappings ${\mathcal{G}}_{\mathcal{l}k}(·)$ established in the previous sections.

Directly analyzing the stability of the nonlinear delay system in (12) is challenging due to the intricate coupling between the high-dimensional task states and the heterogeneous semantic delays. To bridge the gap between the physical semantic communication process and the system’s dynamical behavior, and to facilitate a tractable stability analysis, we adopt the following standard assumptions that bound the variations and the impacts of the semantic feedback:

Assumption 1.

The task dynamics $f_k(·)$ are Lipschitz continuous with respect to the task state and semantic inputs.

Assumption 2.

For each semantic embedding ${\mathit{s}}_{\mathcal{l}k,i}\left(t\right)$ defined in (1), we introduce a stability sensitivity coefficient $C_{\mathcal{l}k,i}>0$. This coefficient is formally defined as the Lipschitz constant for the semantic abstraction mapping ${\mathcal{E}}_{\mathcal{l}k,i}(·)$, i.e.,

$$\parallel {\mathcal{E}}_{\mathcal{l}k,i}\left({\mathit{m}}_1\right)-{\mathcal{E}}_{\mathcal{l}k,i}\left({\mathit{m}}_2\right)\parallel \le C_{\mathcal{l}k,i}\parallel {\mathit{m}}_1-{\mathit{m}}_2\parallel ,\forall {\mathit{m}}_1,{\mathit{m}}_2.$$

(13)

The coefficient $C_{\mathcal{l}k,i}$ quantifies the importance of modality i by scaling how semantic variations—potentially distorted by communication delays—propagate into the task execution loop.

Assumption 3.

All semantic delays are finite and bounded, satisfying $0<T_{\mathcal{l}k,i}\le T_{max}<\infty$.

In practical AIoT scenarios, these assumptions represent the physical boundaries of the system: $C_{\mathcal{l}k,i}$ and $T_{max}$ characterize the AI encoder’s sensitivity and the network’s latency ceiling, respectively. These conditions can be empirically validated through module-level sensitivity testing and network telemetry. Furthermore, the stability conditions derived from these assumptions are inherently robust to AI uncertainties, such as training instability or model mismatch. By incorporating a perturbation margin ${\Delta }_k$ in our criteria, the framework ensures that as long as the local semantic reconstruction errors remain within this safe boundary, global dynamic stability is preserved even if the AI modules exhibit fluctuations.

Stability Perturbation Induced by Heterogeneous Semantic Delays

Specifically, we define ${\mathit{q}}^{\star }={[{\mathit{q}}_1^{\star T},\dots ,{\mathit{q}}_K^{\star T}]}^T$ as the steady-state equilibrium point of the global delay system (12). At this point, the rate of change of the system state vanishes, satisfying the following equilibrium condition:

$$\dot{\mathit{q}}\left(t\right)=\mathcal{F}\left({\mathit{q}}^{\star },{\left\{{\mathit{q}}^{\star }\right\}}_{\mathcal{l},k,i}\right)=\mathbf{0}.$$

To evaluate the system stability under small perturbations, we define the local error state as ${\mathit{e}}_k\left(t\right)={\mathit{q}}_k\left(t\right)-{\mathit{q}}_k^{\star }$. While the global dynamics in (12) provide a comprehensive description of the interconnected system, their non-linear nature—compounded by heterogeneous delays—precludes a direct derivation of closed-form stability criteria.

To achieve analytical tractability and isolate the impact of individual modality parameters, we decompose the global dynamics $\mathcal{F}$ into nodal interactions and apply a first-order Taylor expansion around the equilibrium ${\mathit{q}}^{\star }$. This linearization allows us to approximate the complex semantic feedback as a linear time-delay system, where the evolution of the error state for device k can be derived as follows:

$$\begin{array}{cc}\hfill {\dot{\mathit{e}}}_k\left(t\right)& ={\dot{\mathit{q}}}_k\left(t\right)-{\dot{\mathit{q}}}_k^{\star }\hfill \\ \hfill & =f_k\left({\mathit{q}}_k\left(t\right),\left\{{\mathit{s}}_{\mathcal{l}k,i}\left(t\right)\right\}\right)-f_k\left({\mathit{q}}_k^{\star },\left\{{\mathit{s}}_{\mathcal{l}k,i}^{\star }\right\}\right)\hfill \\ \hfill & \approx {\displaystyle \frac{\partial f_k}{\partial {\mathit{q}}_k}}{|}_{{\mathit{q}}^{\star }}({\mathit{q}}_k\left(t\right)-{\mathit{q}}_k^{\star })+\sum _{\mathcal{l}\in \mathcal{K}}\sum _{i\in \mathcal{M}}{\displaystyle \frac{\partial f_k}{\partial {\mathit{s}}_{\mathcal{l}k,i}}}{|}_{{\mathit{q}}^{\star }}({\mathit{s}}_{\mathcal{l}k,i}\left(t\right)-{\mathit{s}}_{\mathcal{l}k,i}^{\star })\hfill \\ \hfill & ={\mathbf{A}}_k{\mathit{e}}_k\left(t\right)+\sum _{\mathcal{l}\in \mathcal{K}}\sum _{i\in \mathcal{M}}{a}_{\mathcal{l}k,i}{\mathbf{C}}_{\mathcal{l}k,i}\left[{\mathcal{E}}_{\mathcal{l}k,i}\left({\mathit{q}}_{\mathcal{l}}(t-T_{\mathcal{l}k,i})\right)-{\mathcal{E}}_{\mathcal{l}k,i}\left({\mathit{q}}_{\mathcal{l}}^{\star }\right)\right],\hfill \end{array}$$

(14)

where the Jacobian matrices are defined as ${\mathbf{A}}_k={\displaystyle \frac{\partial f_k}{\partial {\mathit{q}}_k}}{|}_{{\mathit{q}}^{\star }}$ and ${\mathbf{C}}_{\mathcal{l}k,i}={\displaystyle \frac{\partial f_k}{\partial {\mathit{s}}_{\mathcal{l}k,i}}}{|}_{{\mathit{q}}^{\star }}$. While this first-order approximation characterizes the stability within the linear regime, the potential impact of high-order nonlinearities under extreme disturbances will be detailed in Section 4.3.3.

Crucially, the feedback gain matrix ${\mathbf{C}}_{\mathcal{l}k,i}$ provides the mathematical foundation for the stability sensitivity coefficient $C_{\mathcal{l}k,i}$ in (13). Specifically, $\parallel {\mathbf{C}}_{\mathcal{l}k,i}{\parallel }_2\le C_{\mathcal{l}k,i}$, where $C_{\mathcal{l}k,i}$ serves as a physical scaling factor: it quantifies the sensitivity of device k’s state derivative to semantic deviations in modality i. This mapping ensures that $C_{\mathcal{l}k,i}$ captures the potential of delay-induced semantic errors to be amplified as they propagate through the nodal feedback loop. Consequently, the magnitude of the feedback error in (14) is jointly determined by the sensitivity coefficient $C_{\mathcal{l}k,i}$ and the state deviation ${\mathit{q}}_{\mathcal{l}}(t-T_{\mathcal{l}k,i})-{\mathit{q}}_{\mathcal{l}}^{\star }$. This structural coupling indicates that the asynchronous semantic delays $T_{\mathcal{l}k,i}$ affect the error dynamics through the scaling effect of $C_{\mathcal{l}k,i}$, which provides a rigorous basis for quantifying the stability margin in the subsequent analysis.

Utilizing the Lipschitz continuity in (13), the semantic perturbation satisfies:

$$\parallel {\mathcal{E}}_{\mathcal{l}k,i}\left({\mathit{q}}_{\mathcal{l}}(t-T_{\mathcal{l}k,i})\right)-{\mathcal{E}}_{\mathcal{l}k,i}\left({\mathit{q}}_{\mathcal{l}}^{\star }\right)\parallel \le C_{\mathcal{l}k,i}\parallel {\mathit{e}}_{\mathcal{l}}(t-T_{\mathcal{l}k,i})\parallel .$$

By substituting this bound into (14) and applying the triangle inequality, the evolution of the error norm is bounded by:

$$\parallel {\dot{\mathit{e}}}_k\left(t\right)\parallel \le \parallel {\mathbf{A}}_k\parallel \parallel {\mathit{e}}_k\left(t\right)\parallel +\sum _{\mathcal{l}\in \mathcal{K}\setminus \left\{k\right\}}\sum _{i\in \mathcal{M}}{a}_{\mathcal{l}k,i}\parallel {\mathbf{C}}_{\mathcal{l}k,i}\parallel C_{\mathcal{l}k,i}\parallel {\mathit{e}}_{\mathcal{l}}(t-T_{\mathcal{l}k,i})\parallel .$$

(15)

This upper bound reveals that the impact of different modalities on system stability is inherently heterogeneous, governed by the interaction between the sensitivity coefficients $C_{\mathcal{l}k,i}$, the gain matrices ${\mathbf{C}}_{\mathcal{l}k,i}$, and the asynchronous delays $T_{\mathcal{l}k,i}$. Specifically, the error propagation is scaled by the product of semantic sensitivity and feedback gain, while the phase lag in coordination is dictated by $T_{\mathcal{l}k,i}$. To quantify this cumulative destabilizing effect across all active modalities, we define a stability perturbation metric ${\Delta }_k$:

$${\Delta }_k\triangleq f\left({\left\{\parallel {\mathbf{C}}_{\mathcal{l}k,i}\parallel ,C_{\mathcal{l}k,i},T_{\mathcal{l}k,i}\right\}}_{\mathcal{l},i}\right),$$

(16)

which encapsulates the total perturbation energy introduced by the multimodal semantic flow. The explicit analytical form of ${\Delta }_k$, derived from the Lyapunov-Krasovskii stability conditions, will be provided in Section 4.1.

3.3. Resource-Adaptive MIMO Optimization for Stability Guarantee

Section 3.1 and Section 3.2 establish complete distributed communication systems, their corresponding control systems, and stability boundaries ${\Delta }_k$ (as shown in (16)). Next, based on the above model, we will give the complete methods for solving C1 and C2 respectively. The hierarchical diagram of the whole method is shown in Figure 3.

Figure 3. A diagram illustrating the guidance hierarchy of the stability disturbance index ${\Delta }_k$ for the device communication and control layers.

3.3.1. Stability-Driven Modality Selection

The first stage of the adaptation mechanism focuses on identifying the optimal subset of semantic modalities to prevent communication congestion. To quantify the importance of each modality, we formulate a dynamic stability sensitivity coefficient $C_{\mathcal{l}k,m}\left(t\right)$ that bridges offline task knowledge with real-time system states:

$$C_{\mathcal{l}k,m}\left(t\right)={\varphi }_m·{\parallel {\mathbf{C}}_{\mathcal{l}k,m}\parallel }_2,$$

(17)

where ${\varphi }_m$ denotes the base contribution weight determined via offline supervised learning (will be detailed in Section 4.3), and $\parallel {\mathbf{C}}_{\mathcal{l}k,m}{\parallel }_2$ is the Jacobian norm derived from the linearized error dynamics in (14).

To avoid unstable delay growth caused by channel oversubscription, we introduce a binary selection vector ${\mathit{a}}_{\mathcal{l}k}={[a_{\mathcal{l}k,1},\dots ,a_{\mathcal{l}k,M}]}^T$, where $a_{\mathcal{l}k,i},i\in 1\dots K$ is the modality selection variable in (1). The selection process is modeled as a stability-maximization problem under a coarse capacity constraint:

$$\begin{array}{cc}\hfill \underset{\left\{a_{\mathcal{l}k,m}\right\}}{min}& \sum _{m\in \mathcal{M}}{a}_{\mathcal{l}k,m}{C}_{\mathcal{l}k,m}\left(t\right)T_{\mathcal{l}k,m}^{\mathrm{est}}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \sum _{m\in \mathcal{M}}{a}_{\mathcal{l}k,m}{d}_{\mathcal{l}k,m}\le R_{\mathcal{l}k}^{\mathrm{prev}},\hfill \\ \hfill & a_{\mathcal{l}k,m}\in \{0,1\},\forall m\in \mathcal{M}.\hfill \end{array}$$

(18)

where $T_{\mathcal{l}k,m}^{\mathrm{est}}$ and $R_{\mathcal{l}k}^{\mathrm{prev}}$ represent the estimated delay and the achievable rate from the previous iteration, respectively. This selection mechanism acts as a “semantic gate,” pruning redundant or low-sensitivity information to preserve the system’s stability margin before fine-grained resource allocation.

3.3.2. Stability-Aware Precoding Optimization

In the second stage, the system performs fine-grained resource adaptation by optimizing the precoding matrix ${\mathbf{W}}_{\mathcal{l}k}$ based on the active modality set. Unlike traditional capacity-maximization methods, our goal is to reshape the communication resource distribution to minimize the stability-related cost. By directly utilizing the rate formulation in (7), the transmission delay of each modality becomes an implicit function of the precoder’s spatial structure.

The resource adaptation is formulated as a stability-driven optimization problem:

$$\begin{array}{cc}\hfill \underset{{\mathbf{W}}_{\mathcal{l}k}}{min}& \sum _{m\in {\mathcal{M}}_{\mathrm{act}}}{C}_{\mathcal{l}k,m}\left(t\right)\left(T_{\mathcal{l}k,m}^{\mathrm{proc}}+{\displaystyle \frac{a_{\mathcal{l}k,m}{d}_{\mathcal{l}k,m}}{R_{\mathcal{l}k}\left({\mathbf{W}}_{\mathcal{l}k}\right)}}\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \mathrm{Tr}\left({\mathbf{W}}_{\mathcal{l}k}{\mathbf{W}}_{\mathcal{l}k}^{\mathrm{H}}\right)\le P_{\mathcal{l}}^{max}.\hfill \end{array}$$

(19)

In this framework, ${\mathbf{W}}_{\mathcal{l}k}$ serves as a spatial power allocator. The objective function in (19) assigns higher priority to modalities with larger stability sensitivity $C_{\mathcal{l}k,m}\left(t\right)$. To achieve the minimum of (19), the precoder adaptively steers its beamforming gain towards the spatial directions that offer the highest throughput-to-stability benefit. This creates a task-centric communication link where physical layer resources are non-uniformly distributed to shield the control loop from critical information loss or excessive latency.

Remark 2.

The computational overhead of the two-stage optimization in Section 3.3 is designed to meet the stringent latency requirements of 6G-enabled AIoT. The semantic selection in (18) is a discrete combinatorial problem; however, since the number of modalities M is typically small (e.g., $M\le 5$), the optimal configuration can be found with $O(MlogM)$ complexity via greedy sorting or $O\left(2^M\right)$ via exhaustive search, both of which are negligible. The precoding optimization in (19) exhibits a convex structure with respect to the power allocation, permitting efficient solution via closed-form water-filling or interior-point methods with $O\left(N_t^3\right)$ complexity. This modular complexity ensures that the entire adaptation process converges within the sub-millisecond coherence time of the wireless channel.

4. Stability-Aware System Architecture and Optimization Framework

The system model in Section 3 has fully characterized the multimodal wireless communication method and the ${\Delta }_k$ system stability boundary for distributed systems, and proposed solutions to the C1 and C2 challenges. However, the specific functional form of ${\Delta }_k$ in (16) and the concrete implementation of the system stability optimization scheme are still not given. Therefore, this section will give the specific formula of ${\Delta }_k$, reveal the fundamental trade-off between information richness and dynamic stability, and propose a stability-aware optimization framework that operates strictly within the allowable region.

4.1. Stability Conditions Under Heterogeneous Semantic Delays

As illustrated in (14), to characterize the collective stability of the distributed system, we introduce the Lyapunov function [44] based on the error state $e_k\left(t\right)$:

$$V\left(t\right)=\sum _k{e}_k^{\top }\left(t\right){\mathbf{P}}_k{e}_k\left(t\right),$$

(20)

where ${\mathbf{P}}_k\succ 0$ is the weighting matrix. Substituting the error dynamics (14) into the derivative of $V\left(t\right)$, we have:

$$\begin{array}{cc}\hfill \dot{V}\left(t\right)& =\sum _k{\displaystyle \frac{d}{dt}}\left(e_k^{\top }\left(t\right){\mathbf{P}}_k{e}_k\left(t\right)\right)\hfill \\ \hfill & =\sum _k\left(e_k^{\top }\left(t\right)({\mathbf{A}}_k^{\top }{\mathbf{P}}_k+{\mathbf{P}}_k{\mathbf{A}}_k)e_k\left(t\right)\right)\hfill \\ \hfill & +2\sum _k\sum _{\mathcal{l}\ne k}\sum _m{a}_{\mathcal{l}k,m}{e}_k^{\top }\left(t\right){\mathit{P}}_k{\mathbf{C}}_{\mathcal{l}k,m}\left({\mathcal{E}}_{\mathcal{l}k}^{\left(m\right)}\left(q_{\mathcal{l}}(t-T_{\mathcal{l}k}^{\left(m\right)})\right)-{\mathcal{E}}_{\mathcal{l}k}^{\left(m\right)}\left(q_{\mathcal{l}}^{\star }\right)\right).\hfill \end{array}$$

(21)

where, ${\mathbf{A}}_k$ and ${\mathbf{C}}_{\mathcal{l}k,m}$ are the Jacobian matrices derived in (14), and $a_{\mathcal{l}k,m}$ is the modality selection variable.

Applying the Cauchy–Schwarz inequality together with the Lipschitz bounds from (13), the derivative (21) is bounded as:

$$\begin{array}{cc}\hfill \dot{V}(t)& \le \sum _k{e}_k^{\top }\left(t\right)({\mathbf{A}}_k^{\top }{\mathbf{P}}_k+{\mathbf{P}}_k{\mathbf{A}}_k)e_k\left(t\right)\hfill \\ \hfill & +2\sum _k\sum _{\mathcal{l}\ne k}\sum _m{a}_{\mathcal{l}k,m}\parallel {\mathbf{P}}_k{\mathbf{C}}_{\mathcal{l}k,m}\parallel ·C_{\mathcal{l}k}^{\left(m\right)}\parallel e_k\left(t\right)\parallel \parallel e_{\mathcal{l}}(t-T_{\mathcal{l}k}^{\left(m\right)})\parallel .\hfill \end{array}$$

(22)

To resolve the delay dependence, we utilize the integral bound where the error energy accumulates over the latency interval $T_{\mathcal{l}k,m}$. By defining the stability sensitivity coefficient as $C_{\mathcal{l}k,m}\left(t\right)\triangleq 2\parallel {\mathit{P}}_k{\mathbf{C}}_{\mathcal{l}k,m}\parallel C_{\mathcal{l}k}^{\left(m\right)}$, the inequality (22) can be rearranged to isolate the impact of nodal dynamics and multimodal coupling:

$$\begin{array}{c}\hfill \dot{V}\left(t\right)\le -\sum _k\left({\lambda }_{min}\left({\mathit{Q}}_k\right)-\sum _{\mathcal{l}\ne k}\sum _m{a}_{\mathcal{l}k,m}{C}_{\mathcal{l}k,m}\left(t\right)T_{\mathcal{l}k,m}\right){\parallel e_k\left(t\right)\parallel }^2,\end{array}$$

(23)

where ${\mathit{Q}}_k=-({\mathbf{A}}_k^{\top }{\mathit{P}}_k+{\mathit{P}}_k{\mathbf{A}}_k)$.

Theorem 1.

Through the Lyapunov-Krasovskii stability analysis, the explicit analytical form of the stability perturbation metric ${\Delta }_k$ introduced in (16) is derived as:

$${\Delta }_k=\sum _{\mathcal{l}\ne k}\sum _{m\in \mathcal{M}}{a}_{\mathcal{l}k,m}{C}_{\mathcal{l}k,m}\left(t\right)T_{\mathcal{l}k,m}.$$

(24)

A sufficient condition for the distributed system to remain asymptotically stable is ${\Delta }_k<{\lambda }_{min}\left({\mathbf{Q}}_k\right)$ for all $k\in \mathcal{K}$, where ${\lambda }_{min}\left({\mathbf{Q}}_k\right)$ is the inherent convergence rate determined by the local dynamics ${\mathbf{A}}_k$ from Section 3.2.

Remark 3.

This derivation provides the formal mathematical proof for the intuitive coupling suggested in Section 3.2. It reveals that the perturbation energy ${\Delta }_k$ is not an arbitrary function, but a precise linear combination of modality-specific richness $C_{\mathcal{l}k,m}\left(t\right)$ and their corresponding asynchronous latencies $T_{\mathcal{l}k}^{\left(m\right)}$. This theoretical foundation directly justifies the stability-driven optimization framework (C1 and C2) in Section 3.3: the system must actively constrain the modality set and optimize precoding to ensure the aggregated impact ${\Delta }_k$ does not breach the stability margin ${\lambda }_{min}\left({\mathbf{Q}}_k\right)$.

4.2. Fundamental Trade-Off Between Semantic Richness and Dynamic Stability

As shown in Figure 4, the stability analysis in Section 4.1 reveals that the dynamic stability of a distributed multimodal communication system is fundamentally constrained by the asynchronous coupling induced by heterogeneous sensing and communication processes. Specifically, for each device k, the Lyapunov-based sufficient condition for stability requires that the aggregated asynchronous impact ${\Delta }_k$ remains bounded by the system’s inherent recovery capability. Based on Theorem 1, stability is guaranteed only if:

$$\sum _{\mathcal{l}\ne k}\sum _{m\in {\mathcal{M}}_{\mathcal{l}k}}{a}_{\mathcal{l}k,m}{C}_{\mathcal{l}k,m}\left(t\right)T_{\mathcal{l}k}^{\left(m\right)}<{\lambda }_{min}\left({\mathbf{Q}}_k\right),$$

(25)

where ${\lambda }_{min}\left({\mathbf{Q}}_k\right)>0$ characterizes the intrinsic stability margin of device k derived from its local dynamics ${\mathbf{A}}_k$ in (14), and $T_{\mathcal{l}k}^{\left(m\right)}$ denotes the end-to-end delay associated with modality m.

Figure 4. Illustration of the fundamental trade-off between semantic richness and dynamic stability in distributed multimodal communication systems.

According to Section 3.1, the end-to-end delay $T_{\mathcal{l}k}^{\left(m\right)}$ consists of both local processing latency $T_{\mathcal{l}k,\mathrm{proc}}^{\left(m\right)}$ and wireless transmission latency ${\displaystyle \frac{d_{\mathcal{l}k}^{\left(m\right)}}{R_{\mathcal{l}k}^{\left(m\right)}}}$. Substituting $T_{\mathcal{l}k}^{\left(m\right)}$ into the stability condition (25) yields:

$$\sum _{\mathcal{l}\ne k}\sum _{m\in {\mathcal{M}}_{\mathcal{l}k}}{a}_{\mathcal{l}k,m}{C}_{\mathcal{l}k,m}\left(t\right)\left(T_{\mathcal{l}k,\mathrm{proc}}^{\left(m\right)}+{\displaystyle \frac{d_{\mathcal{l}k}^{\left(m\right)}}{R_{\mathcal{l}k}^{\left(m\right)}}}\right)<{\lambda }_{min}\left({\mathbf{Q}}_k\right).$$

(26)

By rearranging (26), we can isolate the terms associated with semantic richness to reveal a fundamental trade-off:

$$\sum _{\mathcal{l}\ne k}\sum _{m\in {\mathcal{M}}_{\mathcal{l}k}}{a}_{\mathcal{l}k,m}{\displaystyle \frac{C_{\mathcal{l}k,m}\left(t\right)}{R_{\mathcal{l}k}^{\left(m\right)}}}{d}_{\mathcal{l}k}^{\left(m\right)}<{\lambda }_{min}\left({\mathbf{Q}}_k\right)-\sum _{\mathcal{l}\ne k}\sum _{m\in {\mathcal{M}}_{\mathcal{l}k}}{a}_{\mathcal{l}k,m}{C}_{\mathcal{l}k,m}\left(t\right)T_{\mathcal{l}k,\mathrm{proc}}^{\left(m\right)}.$$

(27)

The left-hand side of (27) represents the cumulative stability cost of transmitted semantic information, while the right-hand side represents the remaining stability margin after accounting for intrinsic processing overhead.

Equation (27) provides the formal theoretical foundation for the two-stage stability-aware design introduced in Section 3.3. By analyzing the mathematical structure of this inequality, the adaptation mechanism can be interpreted as an active management of the system’s stability “budget”:

• Mathematical Realization of C1 (Modality Selection): The objective in (18), $\sum a_{\mathcal{l}k,m}{C}_{\mathcal{l}k,m}\left(t\right)T_{\mathcal{l}k,m}^{\mathrm{est}}$, directly corresponds to the discrete minimization of the aggregate perturbation ${\Delta }_k$ in (25). Since the available stability margin in (27) is fundamentally limited by the local dynamics ${\mathbf{A}}_k$, C1 serves as a discrete feasibility regulator that selects a modality subset $\left\{a_{\mathcal{l}k,m}\right\}$ by pruning semantic streams with excessive stability cost per bit, i.e., large $C_{\mathcal{l}k,m}/R_{\mathcal{l}k}^{\mathrm{prev}}$.
• Mathematical Realization of C2 (Precoding Optimization): Given the active modalities determined by C1, the precoding design in (19) performs a continuous reshaping of the stability margin. The ratio $C_{\mathcal{l}k,m}\left(t\right)/R_{\mathcal{l}k}^{\left(m\right)}$ in (27) assigns task-dependent priorities to spatial resources. Accordingly, by optimizing the precoder ${\mathbf{W}}_{\mathcal{l}k}$ to increase $R_{\mathcal{l}k}^{\left(m\right)}$ for modalities with high Jacobian sensitivity $C_{\mathcal{l}k,m}\left(t\right)$, the most critical delay components are compressed, thereby enlarging the stability-safe region for semantic transmission.

In summary, increasing semantic richness, either by activating additional modalities or enlarging the dimension $d_{\mathcal{l}k}^{\left(m\right)}$, monotonically increases the asynchronous coupling. Once this cumulative load violates the threshold in (27), the system enters an unstable regime. This result confirms that more semantic information is not always better; instead, richness must be carefully regulated to balance task utility and dynamic stability. This insight transforms the empirical adaptation in Section 3.3 into a mathematically grounded control-communication co-design, where C1 and C2 serve as the discrete and continuous operators for stability preservation, respectively.

4.3. Stability-Aware Multimodal Communication Design

We have now fully mathematically represented the system’s optimization method. However, we haven’t yet explained it from the perspective of model design and the practical application of the system control link. Therefore, based on the above modeling, we will next implement this optimization method at the device level.

4.3.1. Stability-Aware SA-MSC Architecture with AI-Driven Sensitivity Estimation

To translate the theoretical stability criteria derived in Section 4.2 into a deployable online scheduling policy, we propose a stability-enhanced framework based on the Large AI Model-based Multimodal SC (LA-MSC) architecture [31]. As illustrated in Figure 5, this framework augments the original LA-MSC by incorporating a stability-monitoring closed-loop, enabling a cross-layer mapping from high-dimensional raw sensing features to system dynamical responses.

Figure 5. Optimized LA-MSC architecture end-to-end training diagram.

In the perception and encoding phase, the system first leverages the fine-tuned Multimodal Alignment (MMA) module to extract features from the raw sensing set $\left\{{\mathit{m}}_{k,i}\left(t\right)\right\}$. Distinguishing from conventional models focused solely on reconstruction fidelity, we integrate an auto-differentiation-based sensitivity estimation branch at the tail of the MMA encoder. By back-propagating the instantaneous task error $e_k\left(t\right)$ into the semantic latent space, this branch real-time computes the Jacobian norm $\parallel {\mathbf{C}}_{\mathcal{l}k,m}{\parallel }_2$ for each modality’s feature. Simultaneously, the LLM-based Knowledge Base (LKB) module is re-purposed as a task-prior evaluator. Based on off-line supervised learning on diverse control trajectories (e.g., obstacle avoidance or formation control), the LKB assigns a base contribution weight ${\varphi }_m$ to different modalities according to the current task context. Consequently, each device can autonomously construct the dynamic stability sensitivity coefficient:

$$C_{\mathcal{l}k,m}\left(t\right)={\varphi }_m·{\parallel {\mathbf{C}}_{\mathcal{l}k,m}\parallel }_2.$$

(28)

At the computational core, the stability evaluation unit aggregates the real-time transmission rates $R_{\mathcal{l}k}^{\left(m\right)}$ from the Conditional Generative Adversarial Networks-based Channel Estimation (CGE) module and the AI-derived sensitivity coefficients. According to (27), the scheduler assesses the cumulative asynchronous coupling strength ${\Delta }_k$. When ${\Delta }_k$ approaches or exceeds the intrinsic stability margin ${\lambda }_{min}\left({\mathit{Q}}_k\right)$ defined by the system dynamics, a dual-stage optimization feedback loop is triggered.

In the first stage (discrete scheduling), the scheduler functions as a “semantic gatekeeper” by ranking modalities based on their stability-cost-to-rate ratio $C_{\mathcal{l}k,m}\left(t\right)/R_{\mathcal{l}k}^{\left(m\right)}$. By manipulating the modality selection variables $a_{\mathcal{l}k,m}$ within the LA-MSC framework, the algorithm prunes modalities that impose the highest stability burden with minimal task utility. In the second stage (continuous resource reshaping), for the active modality set ${\mathcal{M}}_{\mathrm{act}}$, the scheduler utilizes the precise channel state information from CGE to coordinate the physical-layer precoding matrix ${\mathbf{W}}_{\mathcal{l}k}$. By prioritizing the transmission rates of high-sensitivity modalities, the system actively compresses delay-induced instability factors while preserving semantic richness $d_{\mathcal{l}k}^{\left(m\right)}$. This AI-assisted online scheduling ensures that the flow of multimodal semantic information remains strictly within the stability-feasible region of the distributed system.

4.3.2. Stability-Aware Feasible Control Law

Based on the AI-driven sensitivity parameters derived in (28), the controller monitors the system’s operational safety by defining the instantaneous residual stability margin ${ϵ}_k\left(t\right)$. According to the fundamental trade-off established in (27), the margin is formulated as:

$${ϵ}_k\left(t\right)\triangleq \left[{\lambda }_{min}\left({\mathit{Q}}_k\right)-{\Phi }_k\left(t\right)\right]-\sum _{\mathcal{l}\ne k}\sum _{m\in \mathcal{M}}{a}_{\mathcal{l}k,m}{\displaystyle \frac{C_{\mathcal{l}k,m}\left(t\right)}{R_{\mathcal{l}k}^{\left(m\right)}}}{d}_{\mathcal{l}k}^{\left(m\right)},$$

(29)

where ${\Phi }_k\left(t\right)=\sum a_{\mathcal{l}k,m}{C}_{\mathcal{l}k,m}\left(t\right)T_{\mathcal{l}k,\mathrm{proc}}^{\left(m\right)}$ represents the stability cost incurred by intrinsic processing latencies within the LA-MSC modules. The system is considered to be in a stability-admissible regime if ${ϵ}_k\left(t\right)\ge 0$. A transition to ${ϵ}_k\left(t\right)<0$ signifies that the information-induced asynchronous coupling has exceeded the recovery capability of the local controller, necessitating immediate resource adaptation.

To restore the stability margin with minimal impact on semantic richness, we introduce the Stability-Cost-per-Bit (SCB) index as the primary decision metric:

$${\gamma }_{\mathcal{l}k,m}\left(t\right)={\displaystyle \frac{C_{\mathcal{l}k,m}\left(t\right)}{R_{\mathcal{l}k}^{\left(m\right)}}}.$$

(30)

The SCB index ${\gamma }_{\mathcal{l}k,m}\left(t\right)$ quantifies the “marginal damage” a single unit of semantic richness $d_{\mathcal{l}k}^{\left(m\right)}$ inflicts on the system’s stability under current channel conditions. Within the LA-MSC framework, modalities with high SCB values are prioritized for suppression. Specifically, the feasible control law executes two types of adjustments:

• Discrete Pruning (C1): For modalities where ${\gamma }_{\mathcal{l}k,m}\left(t\right)$ exceeds a dynamic threshold, the controller sets $a_{\mathcal{l}k,m}=0$, effectively deactivating the corresponding MMA encoder branch to prevent catastrophic divergence.
• Continuous Compression (C2): For active modalities, the controller adaptively reduces the semantic dimension $d_{\mathcal{l}k}^{\left(m\right)}$ by adjusting the latent space compression ratio of the MMA, while simultaneously requesting the physical layer to re-allocate precoding power to increase $R_{\mathcal{l}k}^{\left(m\right)}$, thereby lowering the SCB index.

This control law ensures that the updated operating point $(\mathit{a},\mathit{d},\mathbf{W})$ is driven back into the feasible region defined by ${ϵ}_k\left(t\right)\ge 0$ in an optimal, task-centric manner.

4.3.3. Online Stability-Aware Multimodal Scheduling Algorithm

The complete online feasible control procedure is summarized in Algorithm 1. Unlike conventional heuristic scheduling, this framework operates as a state-triggered closed-loop; the execution is initiated whenever the tracking error $e_k\left(t\right)$ deviates from the equilibrium ${\mathit{q}}^{\star }$ or the estimated channel state ${\mathit{H}}_{\mathcal{l}k}$ undergoes a non-negligible fluctuation.

The algorithm starts by leveraging the MMA and LKB modules to perform real-time sensitivity analysis, while the CGE module provides the achievable rate $R_{\mathcal{l}k}^{\left(m\right)}$. The core of the scheduling logic lies in maintaining the stability margin ${ϵ}_k\left(t\right)\ge 0$ by dynamically adjusting the modality set and semantic dimensions based on the SCB index ${\gamma }_{\mathcal{l}k,m}\left(t\right)$. Specifically, the pruning process in Step 3 is governed by the SCB index ${\gamma }_{\mathcal{l}k,m}\left(t\right)$, providing a rigorous stopping criterion: the iteration terminates immediately upon satisfying ${ϵ}_k\left(t\right)\ge 0$ or exhausting the cardinality of the modality set $\left|\mathcal{M}\right|$. This ensures that the system not only optimizes communication efficiency but also strictly adheres to the dynamic stability envelope of the control task.

Algorithm 1 LA-MSC Based Online Stability-Aware Multimodal Control

Input:  Intrinsic margin ${\lambda }_{min}\left({\mathbf{Q}}_k\right)$, Task prior weights $\left\{{\varphi }_m\right\}$ from LKB, Initial semantic dimensions $\left\{d_{\mathcal{l}k}^{\left(m\right)}\right\}$. Output:  Stability-feasible configuration $\{a_{\mathcal{l}k,m},d_{\mathcal{l}k}^{\left(m\right)},{\mathbf{W}}_{\mathcal{l}k}\}$. 1: Step 1: AI-Driven Perception and Estimation 2: Extract Jacobian norm $\parallel {\mathbf{C}}_{\mathcal{l}k,m}{\parallel }_2$ via MMA auto-differentiation branch based on state error $e_k\left(t\right)$. 3: Obtain channel state information and estimate rates $R_{\mathcal{l}k}^{\left(m\right)}$ via CGE module. 4: Compute stability sensitivity $C_{\mathcal{l}k,m}\left(t\right)={\varphi }_m·{\parallel {\mathbf{C}}_{\mathcal{l}k,m}\parallel }_2$. 5: Step 2: Stability Assessment 6: Calculate aggregate perturbation ${\Delta }_k=\sum C_{\mathcal{l}k,m}\left(t\right)(T_{\mathrm{proc}}+d/R)$. 7: Compute residual margin ${ϵ}_k\left(t\right)=[{\lambda }_{min}\left({\mathit{Q}}_k\right)-{\Phi }_k\left(t\right)]-{\Delta }_k$. 8: Step 3: Joint Communication-Control Scheduling 9: if ${ϵ}_k\left(t\right)\ge 0$ then 10:    // Stable Regime: Optimize throughput 11:    Execute Precoder Optimization (C2) to maximize $R_{\mathcal{l}k}^{\left(m\right)}$ for all $a_{\mathcal{l}k,m}=1$. 12: else 13:    // Unstable Regime: Priority-based Adaptation 14:    Calculate SCB indices ${\gamma }_{\mathcal{l}k,m}\left(t\right)=C_{\mathcal{l}k,m}\left(t\right)/R_{\mathcal{l}k}^{\left(m\right)}$ for all active modalities. 15:    Sort modalities $m\in {\mathcal{M}}_{\mathrm{act}}$ in descending order of ${\gamma }_{\mathcal{l}k,m}\left(t\right)$. 16:    for each sorted modality m do 17:      Reduce $d_{\mathcal{l}k}^{\left(m\right)}$ via MMA latent compression or set $a_{\mathcal{l}k,m}=0$. 18:      Re-adjust ${\mathbf{W}}_{\mathcal{l}k}$ and re-compute ${ϵ}_k\left(t\right)$. 19:      if ${ϵ}_k\left(t\right)\ge 0$ then 20:         break 21:      end if 22:    end for 23: end if 24: return  $\{a_{\mathcal{l}k,m},d_{\mathcal{l}k}^{\left(m\right)},{\mathbf{W}}_{\mathcal{l}k}\}$ for the current transmission slot.

The proposed algorithm maintains the system within the theoretical stability region defined in (27). Computational overhead is dominated by modality sorting and scalar updates ($O(MlogM)$), ensuring sub-millisecond convergence given $M\ll 10$. Beyond efficiency, the framework is intrinsically robust to practical impairments: imperfect Channel State Information (CSI) is mitigated by incorporating safety margins in rate estimation, while packet loss is handled as a state-triggered perturbation. Any loss-induced error growth is instantly detected by the real-time monitoring of ${ϵ}_k\left(t\right)$, triggering immediate modality pruning to restore the stability margin. This synergy between low-latency execution and closed-loop resilience allows the LA-MSC to balance semantic richness and dynamic integrity in volatile 6G environments.

5. Numerical Results

The primary objective of the proposed framework is to achieve a stable and high-precision closed-loop control system by adaptively managing multimodal semantic information. Our numerical analysis is specifically designed to demonstrate how the proposed SA-MSC architecture effectively addresses C1 and C2 challenges illustrated in Section 3.1.

To validate the efficacy of the proposed solutions, the numerical evaluation is organized into the following three progressive subsections:

• Equivalent semantic load and the impact of SNR on control accuracy: This experiment characterizes the basic trade-off between semantic richness, signal-to-noise ratio and system stability, thereby verifying the theoretical stability boundary derived in Section 4.2 and the effectiveness of the proposed scheme.
• Trade-off between semantic richness and information timeliness: This evaluation explores the impact of modality-dependent latencies on task execution. It investigates how adaptive semantic pruning maintains the freshness of core features under constrained bandwidth, thereby preventing the system from entering an unstable state caused by asynchronous information staleness.
• Optimization Efficiency and Training Convergence: This evaluation confirms the AI modules’ ability to learn task-specific sensitivity accurately, ensuring that the semantic alignment is not only precise but also computationally feasible for real-time control.

5.1. Simulation Setup

To provide a realistic evaluation for multimodal semantic communication, we utilize a diverse set of benchmark datasets to simulate heterogeneous data streams:

• VOC2012 (Image) [45]: Comprising 17,125 RGB images for visual context modeling. In our framework, it serves as the foundational visual semantic source to evaluate the robustness of cross-modal alignment under diverse object categories.
• LibriSpeech (Audio) [46]: Containing approximately 1000 h of English speech for auditory semantic analysis.
• UCF101 (Video) [47]: Consisting of realistic action videos to simulate dynamic temporal observations in control tasks.
• KITTI (LiDAR) [48]: Incorporated to provide sparse spatial point clouds, which are essential for high-dimensional spatial perception and stability.

The numerical experiments are implemented using the PyTorch 2.4.0 framework on a Python 3.13 environment. All computations are executed on an Apple Silicon M4 Max platform, leveraging the Metal Performance Shaders (MPS) backend for hardware-level GPU acceleration.

Training is performed using the Adam optimizer with a learning rate of $5\times {10}^{-5}$ and a batch size of 64. This specific learning rate was selected based on a grid search within the range of $[{10}^{-6},{10}^{-3}]$, where $5\times {10}^{-5}$ was found to strike the optimal balance between rapid convergence and the prevention of catastrophic forgetting in the pre-trained LLM and semantic encoders. We employ a joint-training strategy for 100 epochs to ensure stable parameter optimization. The reported results are averaged over multiple runs to guarantee statistical significance, maintaining consistency across repeated trials.

5.2. Joint Stability Evaluation Under Semantic Load and Channel Degradation

In this subsection, we evaluate the control stability of the proposed framework under the joint impact of semantic overload and channel degradation. Rather than treating the equivalent semantic load (ESL) and the channel signal-to-noise ratio (SNR) as independent factors, we characterize their coupled effect on the control tracking error $\parallel e\parallel$ through a unified stability landscape. Four schemes are considered for comparison: Group 1 (Deep-JSCC) as a unimodal joint source–channel coding baseline that transmits only the most significant modality; Group 2 (LA-MSC w/o MMA), which incorporates the language-knowledge bridge (LKB) without multimodal alignment; Group 3 (Vanilla LAM-SC), which employs both LKB and MMA but lacks stability-aware control; and Group 4 (Proposed SA-MSC), which integrates the full stability-aware optimization mechanism.

To bridge communication parameters with control stability, ESL is defined based on the delay model in Section 4.3.2 as

$$\mathrm{ESL}=\sum _{m\in \mathcal{M}}{\omega }_{\mathcal{l}k,m}\left(T_{\mathcal{l}k,m}^{\mathrm{proc}}+{\displaystyle \frac{a_{\mathcal{l}k,m}{d}_{\mathcal{l}k,m}}{R_{\mathcal{l}k}\left({\mathit{W}}_{\mathcal{l}k}\right)}}\right)f_s,$$

(31)

which characterizes the cumulative temporal pressure imposed by heterogeneous multimodal semantics on the control loop.

As shown in Figure 6, when the semantic load is moderate and the channel quality is high, all schemes operate within the stability tolerance region, and the tracking error is bounded. However, with the increase of ESL or the decrease of SNR, the feasible region of schemes in groups 1–3 collapses rapidly due to the disordered accumulation of delay. Once the residual stability margin ${ϵ}_k\left(t\right)$ approaches zero, the error grows exponentially. In contrast, the proposed SA-MSC scheme actively adjusts the semantic dimension and modal activation based on the stability cost index ${\gamma }_{\mathcal{l}k,m}\left(t\right)$, thereby limiting the growth of effective delay and maintaining ${ϵ}_k\left(t\right)\ge 0$. This adaptive behavior significantly expands the joint stability region and ensures that the control error is bounded under high semantic pressure and adverse channel conditions.

Figure 6. Joint ESL–SNR stability landscape illustrating the control tracking error $\parallel e\parallel$ under varying semantic loads and channel conditions. The proposed SA-MSC reshapes the feasible stability region by actively bounding the effective asynchronous delay.

5.3. Task-Oriented Evaluation: The Richness-Latency Trade-Off in KITTI Dataset

To illustrate the core trade-off between semantic richness and dynamic stability, we use intelligent driving scenarios from the KITTI dataset as examples. The following numerical results are obtained by averaging over 10 independent trials to ensure statistical significance. As shown in Figure 7, while Group 3 provides high-fidelity reconstruction, it suffers from a severe “semantic staleness” problem, where bounding boxes lag behind their physical real-world locations due to excessive transmission latency. In contrast, our proposed SA-MSC (Group 4) actively prunes redundant background details to prioritize temporal alignment. Despite employing abstract feature representations, Group 4 still ensures the most accurate real-time synchronization between the semantic stream and the vehicle’s actual location. Although the third group of algorithms has a low initial error, its perturbation metric ${\Delta }_k$ exhibits a smooth upward trend as asynchronous latency accumulates, eventually approaching the stability boundary, leading to performance divergence in $\left|\right|e_k\left(t\right)\left|\right|$. Conversely, SA-MSC (Group 4) is able to smoothly maintain ${\Delta }_k$ within a safe region. These results confirm that in highly mobile distributed tasks, ensuring the timeliness of core semantic features through adaptive pruning is more robust than pursuing information richness.

Figure 7. Comparative analysis of semantic-aware transmission and control stability.

5.4. Analysis of Training Convergence and Learning Efficiency

The final evaluation focuses on the training dynamics of the proposed AI modules to verify their learnability and optimization stability. As shown in Figure 8, Group 4 achieves the lowest final training loss among all schemes. While the initial convergence rate of Group 4 is slightly lower than that of the simpler Group 1 (Deep-JSCC), which may caused by the higher model complexity and the inclusion of stability-related constraints, it eventually converges to a superior local optimum. This indicates that the stability-aware branches effectively guide the MMA modules to prioritize features that contribute most to the control gain.

Figure 8. Training dynamics of the four groups over 100 epochs: (a) convergence profiles of the training loss and (b) evolution of task accuracy for the multimodal semantic alignment.

The accuracy of 4 groups demonstrates the evolution of task accuracy, where the proposed framework reaches a peak accuracy of approximately $97\%$. Compared to Group 3, Group 4 exhibits higher learning efficiency in the mid-to-late training stages. This performance boost is attributed to the sensitivity-aware learning enabled by the SCB index ${\gamma }_{\mathcal{l}k,m}\left(t\right)$, which allows the network to distinguish between critical control semantics and redundant information. The results confirm that the proposed SA-MSC architecture not only ensures theoretical stability but also possesses the robust learning capability required for complex multimodal semantic communication tasks.

6. Conclusions

This paper studied the stability of distributed multimodal wireless systems under heterogeneous and asynchronous semantic communication. By modeling multimodal semantic streams as delayed feedback embedded in the task state evolution, we established a delay-coupled dynamic system that explicitly captures the impact of modality-dependent sensing and MIMO transmission. Using Lyapunov–Krasovskii analysis, we derived stability conditions and a stability perturbation metric, revealing a fundamental trade-off between semantic information richness and dynamic stability: incorporating additional modalities does not always improve system performance and may destabilize the system when asynchronous delays accumulate. To address this issue, a stability-aware optimization framework was introduced to regulate modality participation and communication design, ensuring stable system operation. While this work primarily focuses on linearized stability, future research will explore high-order nonlinear compensations and the impact of non-ideal channel state information, further extending the framework to resource-constrained and highly dynamic intelligent systems.