Adaptive Multimodal LSTM with Online Learning for Evolving IoT Data Streams ^†

Abstract

The Internet of Things (IoT) uses networked devices, dispersed sensors, and cameras to create huge, diverse data streams. Concept drift, in which the underlying data distribution shifts over time, is frequently caused by the non-stationary and multimodal character of these streams. Static machine learning models, based on fixed data distributions, reduce forecast accuracy and system reliability since they are unable to adapt to such changes. This paper proposes an Adaptive Multimodal Long Short-Term Memory (AM-LSTM) architecture to address these challenges by combining modality-specific temporal modelling, attention-based dynamic fusion, and drift-aware online learning. An attention mechanism adaptively weights informative streams to mitigate the impact of noisy or missing input, while specialist LSTM encoders capture the temporal correlations of each modality. Concept drift is detected using a sliding-window error monitoring technique, and adaptive learning rate adjustment and selective retraining are started when significant distributional changes occur. The proposed system is tested under synthetic drift conditions using the Edge-IoT and UNSW-NB15 benchmark datasets. Experimental results demonstrate that AM-LSTM achieves 88.7% accuracy and an F1-score of 0.85, adapting to drift within 620 samples while maintaining an average update latency of 47 ms per batch. Compared with static and existing adaptive baselines, the proposed approach provides improved robustness, faster drift adaptation, and computational efficiency suitable for real-time IoT environments.

1. Introduction

The rapid expansion of the Internet of Things (IoT) has led to the deployment of billions of interconnected devices that continuously generate large-scale data streams across domains such as smart homes, healthcare, industrial automation, and intelligent transportation systems [1]. Recent projections estimate that more than 16 billion IoT devices are actively connected worldwide, with this number expected to grow steadily in the coming years [2]. These devices produce heterogeneous and multimodal data, including sensor measurements, video feeds, network traffic logs, and environmental signals [3].

Unlike traditional static datasets, IoT data streams are inherently non-stationary. Their statistical properties evolve due to environmental variations, user behaviour changes, device ageing, network reconfiguration, or emerging attack patterns [4]. This phenomenon, which is often called concept drift, happens when the joint probability distribution between target labels and input information shifts. Consequently, models developed in the past might become outdated, resulting in worse prediction accuracy and decreased system dependability.

Recurrent neural networks (RNNs) have proven to be highly effective at modelling temporal dependencies in sequential data, especially Long Short-Term Memory (LSTM) networks. For IoT stream analytics, they are appealing because of their memory structure, which allows them to record long-range temporal correlations. Conventional LSTM-based models, on the other hand, are usually trained offline under the presumption of stationary data distribution. These static models are unable to adapt continuously in dynamic IoT contexts, making it difficult for them to continue operating under changing circumstances.

Previous research [5,6] has been presented on multimodal learning approaches to enhance resilience through the integration of data from several sources. Performance is further improved by attention-based fusion processes, which dynamically weight information modalities. However, drift detection and online adaptation algorithms are not explicitly included in many of the multimodal frameworks that are now in use because they are primarily intended for offline environments. Similarly, multimodal temporal modelling is not adequately handled by drift-aware frameworks, which frequently concentrate on single-modality streams.

Not many studies combine idea drift detection, online learning, and multimodal fusion into a single architecture that is appropriate for real-time IoT deployment, despite significant advancements in these areas. Models that are able to (i) capture temporal patterns specific to a modality, (ii) dynamically adjust to changing data distributions, and (iii) function under stringent computational and latency restrictions are necessary for practical IoT systems.

The study addresses these issues by proposing an Adaptive Multimodal Long Short-Term Memory (AM-LSTM) framework, which integrates drift-aware online adaptation, attention-based dynamic fusion, and modality-specific temporal encoding into a single architecture. The framework uses a sliding-window error monitoring strategy to detect concept drift and perform selective retraining, as well as dedicated LSTM encoders for each modality. Experimental evaluation on benchmark IoT datasets with synthetic drift situations suggests that the proposed approach is more robust, adapts more quickly, and is more computationally efficient than both static and existing adaptive baselines.

2. Related Work

Numerous methodological approaches have been used to further studies on changing IoT data streams. However, current studies generally only address a portion of the fundamental issues that arise in IoT environments, which include: (1) managing idea drift, (2) combining data from several modes, (3) online or federated learning paradigms, and (4) modelling temporal dependencies. Previous approaches are systematically reviewed in these four aspects with the aim to objectively evaluate the state of the art.

• Concept Drift Detection and Adaptation in IoT Streams

In streaming IoT analytics, where statistical characteristics of data distributions vary over time, concept drift continues to be a major problem. Drift adaptation techniques are divided into window-based, statistical, and ensemble-based tactics in early surveys and systematic reviews [2,7].

It has been suggested that lightweight sliding-window techniques allow for real-time adaptation in resource-constrained settings, such as edge and fog [4]. Despite their computing efficiency and suitability for low-latency settings, these techniques frequently rely on unimodal error monitoring and have the potential to produce false positives in noisy environments.

In order to increase response to sudden drift, ensemble-based and online boosting algorithms dynamically re-weight or replace base learners [8]. Although these techniques improve resilience, they are rarely made for heterogeneous multimodal streams and usually presume a fixed feature space. Moreover, a lot of drift-aware methods limit their capacity to express intricate temporal connections by concentrating on static feature representations or shallow classifiers.

In most drift-handling frameworks, the lack of integrated multimodal fusion methods is a significant drawback. Drift detection is commonly used for single-stream inputs, however heterogeneous sensor, visual, and network data is naturally generated by IoT deployments. Consequently, complementary modality information during adaptation is frequently overlooked by current drift-aware algorithms.

2.

Multimodal Learning in IoT Environments

Heterogeneous data sources are utilised in multimodal learning to enhance representation quality and predictability. The three main categories of fusion techniques are attention-based dynamic fusion, late fusion (decision-level integration), and early fusion (feature concatenation) [9,10].

The implementation of early fusion techniques is simple, but they require full and synchronised feature availability. They frequently deteriorate in noisy or incomplete data environments and are sensitive to missing modalities. Modularity is improved by late fusion techniques, however cross-modal interactions might not be adequately captured. Current developments place a strong emphasis on modality-specific encoders and attention-based architectures, especially LSTM-based models for temporal data [11]. These techniques lessen the effect of missing inputs and dynamically weight informative streams. Nevertheless, the majority of multimodal LSTM frameworks are based on stationary data distributions and were created using offline training assumptions. Multimodal approaches generally include explicit concept drift-detection procedures, despite the fact that they greatly enhance representation learning. Consequently, in dynamic IoT situations where distributions change, performance can suffer.

3.

Online and Federated Multimodal Learning for Distributed IoT Systems

Scalability, privacy, and dispersed training limitations in IoT systems are addressed by online and federated learning frameworks [12]. Using streaming data, online learning allows for incremental model updates, while federated learning disperses training among edge devices without centralising raw data [6,13]. Missing-modality scenarios and multimodal federated learning architectures in edge-cloud systems are examined in recent works [14,15]. These methods improve scalability and privacy protection, although they frequently prioritise data heterogeneity and communication effectiveness above drift-aware adaptation.

Furthermore, not all federated multimodal frameworks contain statistically supported drift-detection techniques or explicit temporal modelling mechanisms like LSTMs. Therefore, whereas federated systems handle distribution among devices, they may not be able to handle changes in distribution over time.

4.

Cross-Dimensional Analysis and Research Gap

Present methods for IoT data stream analysis are still disjointed, despite notable advancements in individual components. Multimodal attention models work offline and disregard concept drift, federated systems offer scalability but lack temporal adaptation, temporal LSTMs are static and rely on offline retraining, and drift-detection techniques are primarily unimodal and shallow. Few approaches combine explicit drift detection, dynamic attention-based fusion, modality-specific temporal modelling, and selective online retraining under real-time limitations all at once. Consequently, a cohesive architecture that unifies these dimensions to handle continuously evolving IoT streams remains largely unexplored, representing a critical gap in the literature. Table 1 presents the cross-dimensional classification of existing approaches for evolving IoT data streams.

Table 1. Cross-dimensional classification of existing approaches for evolving IoT data streams.

Study	Temporal Modelling	Multimodal Fusion	Drift Detection	Online Learning	Federated	Key Limitation
[4]	✗	✗ (Unimodal)	Sliding-window	✓	✗	No multimodal integration
[5]	Limited	Multimodal	✗	✓	✓	No explicit drift detection
[8]	✗	✗ (Unimodal)	Ensemble/Boosting	✓	✗	Fixed feature space; shallow models
[11]	✓ (LSTM)	Attention-based	✗	✗ (Offline)	✗	Assumes stationary distributions
[13]	✗	Multimodal	✗	✓	✓	Focus on communication efficiency
[16]	✗	Early / Late fusion	✗	✗	✗	Static multimodal learning
[17]	✓ (LSTM)	✗	Limited	✓	✗	Not designed for multimodal streams
Proposed AM-LSTM	✓ (Modality-specific LSTM)	Attention-based dynamic fusion	Sliding-window error monitoring	✓ (Selective retraining)	Extension planned	Unified integration of all dimensions

Note: ✗ indicates Absence and ✓ indicates Presences.

3. Methodology

The proposed study suggests the Adaptive Multimodal LSTM (AM-LSTM) framework to combine drift-aware online adaptation, attention-based fusion, and modality-specific temporal modelling to continuously learn from changing IoT streams.

3.1. Problem Formulation

Equation (1) denotes a multimodal IoT data instance at time (t), where ${\chi }_t^{\left(M\right) }$ represents the observation from modality (m). Each instance is associated with a label $(y_t$). The objective is to learn a predictive function $f_t : {\chi }_t \to y_t$ that minimizes prediction error under non-stationary data distributions while satisfying real-time latency constraints.

$${\chi }_t=\{{\chi }_t^{\left(1\right) }, {\chi }_t^{\left(2\right) }, ..., {\chi }_t^{\left(M\right) }\}$$

(1)

3.2. Temporal Alignment of Multimodal Streams

All modalities are synchronised using timestamp-based sliding windows of fixed length W = 20. Samples arriving within the same window are aligned to form a multimodal instance. Missing modalities are explicitly handled at the fusion stage via the masking mechanism defined in Equation (2); no separate imputation is required.

3.3. Modality Specific Encoding

Each modality is processed using a dedicated LSTM network to capture modality-specific temporal dependencies. We adopt the standard LSTM formulation as the base encoder; the novelty of our work lies not in the recurrent unit itself but in its integration with attention-based fusion and drift-aware online adaptation within a single framework tailored for evolving IoT streams. For completeness, the LSTM transition is presented in Equation (2):

$$h_t^{\left(m\right)},c_t^{\left(m\right)}=LSTM\left(x_t^{\left(m\right)},h_{t-1}^{\left(m\right)},c_{t-1}^{\left(m\right)},{\theta }^{\left(m\right)}\right)$$

(2)

where h_t^(m) and c_t^(m) are the hidden and cell states, and ϴ^(m) represents learnable parameters. Each LSTM comprises a single layer with 128 hidden units and a dropout rate of 0.2, a configuration kept consistent across all modalities and datasets.

3.4. Dynamic Multimodal Fusion with Missing-Modality Masking

To fuse modality-specific representations, we employ an attention-based mechanism. Critically, to handle missing modalities explicitly—a common occurrence in real-world IoT streams—we introduce a binary presence mask $p_t^{\left(m\right)}$∈{0,1} where $p_t^{\left(m\right)}$ = 1 indicates that modality m is available at time t. The attention weight for each modality is then computed only over the set of available modalities, and missing modalities receive zero weight.

Previous studies [9,10] on multimodal LSTM are fused using an attention-based mechanism in hidden representations as shown in Equation (3):

$${\alpha }_t^{\left(m\right)}= {\displaystyle \frac{p_t^{\left(m\right)}.exp\left(W_f{h}_t^{\left(m\right)}\right)}{\sum _{j=1 }^M{p}_t^{\left(j\right)} .exp\left(W_f{h}_t^{\left(j\right)}\right)}}, z_t={\sum }_{m=1}^M{\alpha }_t^{\left(m\right)}{h}_t^{\left(m\right)}$$

(3)

where $W_f$ is a learnable weight vector and ${\alpha }_t^{\left(m\right)}$ represents adaptive modality weights. This formulation ensures that missing modalities are explicitly masked out before the softmax normalisation, preventing them from influencing the fused representation and effectively addressing the missing-data challenge. The attention mechanism dynamically emphasises informative modalities while suppressing unreliable or absent ones.

3.5. Drift Detection and Online Adaptation

Concept drift was detected by continuously monitoring the prediction error of the AM-LSTM framework. For each incoming multimodal instance at time t, we computed the instantaneous prediction error

$$e_t=\mathcal{L}\left(y_t,y_t\right),$$

(4)

where $\mathcal{L}$ is the cross-entropy loss for classification tasks and the mean absolute error (MAE) for regression tasks. The most recent ${\rm K}= 100$ error values are stored in a sliding window $w_e=e_t-K+1,\dots ,e_t$.

Reference baseline. During an initial stable period (the first 500 samples after deployment, assumed to be drift-free), we collect a reference window of prediction errors and calculate its mean ${\mu }_{ref}$ and standard deviation ${ \sigma }_{ref}$. These statistics represent the expected error behaviour under stationary conditions.

Threshold definition. A drift event is triggered when the average error over the current window eˉ_we exceeds a statistically defined threshold

$${\tau =\mu }_{ref}+\mathcal{Z}.{ \sigma }_{ref}$$

(5)

where z is the critical value derived from a chosen significance level α. Assuming the errors are approximately normally distributed (validated by the central limit theorem for the windowed mean), we set

$$\mathcal{Z}={\mathsf{\Phi }}^{-1} (1-\alpha ),$$

(6)

with ${\mathsf{\Phi }}^{-1}$ as the inverse cumulative distribution function of the standard normal distribution. In all experiments, α = 0.005 (one-sided), yielding z = 2.58. This outline stabilises the sensitivity false-positive rate. The sensitivity analysis of z was performed to access its impact on the results.

Adaptation Mechanism

When a drift is detected, the framework determines which of the modality-specific LSTM encoders’ recent per-modality errors have increased the most. Then only these affected modules are retrained selectively with fresh B = 64 samples (a lightweight mini-batch). This focused update constrains the computational overhead to the drifting modalities, allowing fast adaptation, with an average latency of less than 50 ms per batch.

3.6. Computational Efficiency

The framework can perform the online updates in lightweight mini-batches and maintain an average update latency of less than 50 ms per batch. This keeps the AM-LSTM to be used in a leap-edge/fog scenario on a real-time basis, as it offers a tradeoff among accuracy, versality, and computational convenience.

4. Experimental Setup

4.1. Datasets and Preprocessing

Two datasets were used in the testing of the model: the UNSW-NB15 and Edge-IIoTset. Both sets contain sample data representative of intrusions that could occur under ‘real world’ conditions; while the UNSW-NB15 set provides examples of intrusion detection from an actual real-world scenario, the Edge-IIoTset is a set of multiple different telemetry data types collected from IoT-based sensors on edge-specific devices.

The One Hot Encoding of categorical features was included in the test.

The numerical feature standardization was conducted using the following samples:

x′ = (x − μ)/σ

(7)

Each modality was normalized independently, and samples were sorted chronologically.

4.2. Streaming Experimental Protocol

The data were ordered chronologically with the first thirty percent (30%) of data designated for offline initialisation and the remaining seventy percent (70%) designated for the online stream.

Temporal sequences were created using a sliding window of size W = 20:

X_t = {x_{t − W + 1}, …, x_t}

The offline training sessions were performed using the Adam Optimiser with a learning rate of 0.001, a batch size of 64, a maximum number of epochs of 20, and an early-stopping criterion of five epochs of patience.

4.3. Formal Definition of Synthetic Concept Drift

The following definition was used to define abrupt drift:

P_t(X,Y) = P_0(X,Y)   for t < t_d

P_t(X,Y) = P_1(X,Y)   for t ≥ t_d

The following equation was used for implementing perturbation of the features:

X′ = X + $𝒩$(0, δ²),   δ ∈ {0.1, 0.2, 0.3}

The following definition was used for defining gradual drift utilizing linear interpolation:

P_t = (1 − α_t)P_0 + α_tP_1

α_t = (t − t_d)/T_g,  T_g = 1000 samples

The following list represents the manner by which alternation between the P_0 and P_1 condition occurs during the predefined intervals.

4.4. Drift Detection Mechanism

The method used to detect drift used a sliding error window with a size of W_e = 620. The following average error formula was used:

ē_t = (1/W_e) Σ_{i = t − W_e+1}^{t} e_i

Drift detection met criteria when:

ē_t > μ_e + 2σ_e

4.5. Online Selective Adaptation

Only the modality encoder with the greatest error contribution underwent selective retraining.

Learning rate decay was specified as follows:

ηt = 0.001 × 0.95^k

The average study delay was 47 ms per batch.

4.6. Reproducible Baseline Configurations

Static Multimodal LSTM: 128 hidden layers or nodes (units) for each of the three inputs (streams); use of attention mechanism to combine inputs; MLP with three output nodes (128, 64) does not include drift detection. PacketLSTM: One LSTM (128 units) with MLP (128, 128); full retraining required due to detected drift. Early Fusion: All three input streams combined as one stream; only one LSTM (128 units); no attention mechanism is used.

4.7. Ablation Studies for Missing Modalities

The simulated dropout rates for each modality are 10%, 20%, and 30%. The double confirmation of the full model retraining to create 91 ms latency from an original 47 ms latency both verified improved computational efficiency when using selective updates and added the possibility to incorporate additional improvements in the use of selective update techniques during drift detection.

4.8. Evaluation Metrics and Statistical Validation

The metrics we used to evaluate performance consisted of the following: accuracy; F1-score; detection latency; recovery times; total adaptation latency; and update latencies. Statistical validation of significance was performed using a two-tailed t-test on paired means from five experimental runs at a significance level of p < 0.01.

4.9. Results

Detection Delay is defined as the number of samples between actual drift injection and the triggering of the drift detection mechanism. Recovery Time measures the number of samples required for model performance to return to at least 95% of pre-drift accuracy. Total Adaptation Latency is the sum of detection delay and recovery time, as presented in Table 2.

Table 2. Shows the results of the metrics used for evaluation.

Model	Accuracy (%)	F1-Score	Detection Delay (Samples)	Recovery Time (Samples)	Total Adaptation Latency (Samples)	Update Latency (ms/Batch)
Static Multimodal LSTM	72.3 ± 1.4	0.69 ± 0.02	–	–	>2000	35 ± 2
Packet LSTM [13]	80.1 ± 1.1	0.77 ± 0.01	540 ± 60	660 ± 70	1200 ± 85	48 ± 3
OASW Framework [4]	83.5 ± 0.9	0.81 ± 0.02	420 ± 45	530 ± 50	950 ± 60	42 ± 2
Proposed AM-LSTM	88.7 ± 0.8	0.85 ± 0.01	280 ± 30	340 ± 35	620 ± 45	47 ± 2

4.9.1. Statistical Significance Analysis

To verify that the observed improvement over the strongest baseline (OASW framework) was statistically significant, a paired t-test was conducted across five independent experimental runs. For each run, the accuracy of AM-LSTM was directly compared with the corresponding baseline accuracy under identical streaming and drift conditions. The test yielded a p-value below 0.01 at α = 0.05 (two-tailed), rejecting the null hypothesis of performance being equal. This demonstrates that the difference between 83.5% to 88.7% is statistically significant and not due to chance variation.

4.9.2. Parameter Analysis

Table 3 shows the parameter analysis of the AM-LSTM framework, which suggests valuable insights into active robustness in dynamic environments. The drift detection sensitivity was also set to the optimum 620 samples, a tradeoff between speed of adaptation and stability, which minimises the number of false positives. The attention dropout rate had a range of 0.1–0.5, which was set to 0.2 to prevent overfitting, but still ensured that the model could fuse multimodal information even with partial inputs. An online learning rate with a dynamically scheduled rate starting with 0.001 and decreasing by 0.95 per epoch was used to ensure the online learning converged to a steady state without overshooting. In the main time, a batch size of 64 gave an effective balance between gradient stability and real-time latency, and two fusion attention heads ensured that the model was able to learn relevant cross-model relationships without any competition cost. All of these tuned parameters allow AM-LSTM to be highly accurate, have low adaptation latency, and have stable performance with concept drift.

Table 3. Parameter analysis of the AM-LSTM framework.

Parameter	Tested Range	Observation	Optimal Value
Window Size	500–1000	Smaller windows (e.g., 500) detect drift faster but increase false positives.	620 Samples
Attention Dropout Rate	0.1–0.5	Higher dropouts improve generalisation but reduce fusion accuracy.	0.2
Learning Rate Schedule	0.0001–0.01 (dynamic)	Prevents overshooting and maintains stable convergence during retraining.	0.001 (decay 0.95/epoch)
Batch Size	32–128	Larger batches improve stability but increase update latency.	64

5. Conclusions

Our study combines drift detection, online adaptation, and multimodal fusion into a single adaptive multimodal LSTM (AM-LSTM) framework. Experimental results on two IoT benchmarks demonstrate that it can maintain higher accuracy and faster drift when compared to both static and adaptive alternatives. One of AM-LSTM’s key advantages is its dynamic fusion method, which makes the model robust to missing or incomplete datasets.

Additionally, it provides flexibility without incurring a significant computational burden due to its selective retraining strategy. We conducted the evaluation using the benchmarked datasets with artificial drift instead of real deployment. Also, evaluation was done on scalability to high-dimensional multimodal data (such as video + audio + sensors), which might require further optimization through quantitation or data pruning.

Our future works will include, as part of our research, the ability to extend AM-LSTM to federated and privacy-preserving environments. We will also be preserving the said environment where adaptation occurs across distributed devices without sharing data. We will also include the use of transformer-based techniques to enhance modality fusion and enable better long-range sequence modelling.

Finally, we observed that using the AM-LSTM framework is a step toward achieving a unified and adaptive multimodal learning for IoTs. The proposed framework learns continuously under drift and maintains computing efficiency at the same time. This has huge potential for smart homes as well as applications in healthcare and systems monitoring for industries that operate in dynamic environments.