G-CMTF Net: Spectro-Temporal Disentanglement and Reliability-Aware Gated Cross-Modal Temporal Fusion for Robust PSG Sleep Staging

Abstract

Automatic sleep staging from polysomnography is challenged by marked spectro-temporal heterogeneity and non-stationary cross-channel artifacts, which often undermine naïve multimodal fusion. To address this, a Gated Cross-Modal and Temporal Fusion Network (G-CMTF Net) is proposed as an end-to-end model operating on 30 s EEG epochs and auxiliary EOG and EMG signals, in which cross-modal contributions are regulated through reliability-aware gating. A spectro-temporal disentanglement frontend learns multi-scale temporal features while incorporating FFT-derived band-power embeddings to preserve physiologically meaningful oscillatory cues. At the epoch level, gated fusion suppresses artifact-prone auxiliary inputs, thereby limiting noise transfer into a shared latent space. Long-range sleep dynamics are modeled via a convolution-augmented self-attention encoder that captures both local morphology and transition structure. On Sleep-EDF-20 and Sleep-EDF-78, G-CMTF Net achieves Macro-F1/ACC of 81.3%/85.5% and 78.2%/83.4%, respectively, while maintaining high sensitivity and geometric-mean performance on transitional epochs, consistent with the function of reliability-aware gated fusion under non-stationary auxiliary artifacts. From a symmetry perspective, the proposed framework enforces a structured balance between heterogeneous modalities by promoting representational consistency while adaptively suppressing asymmetric noise contributions.

1. Introduction

Sleep is essential to normal neurophysiological function and constitutes a dynamically regulated biological state through which systemic homeostasis and neural–metabolic recovery are sustained [1]. Rather than a uniform condition, sleep is organized as a cyclic progression of distinct stages, whose coordinated alternation supports memory-related processing, metabolic equilibrium, and immune regulation [2]. For this reason, robust automatic staging is a prerequisite for deriving quantitative sleep biomarkers, thereby facilitating early screening of conditions such as chronic insomnia and obstructive sleep apnea. Conversely, insufficient or fragmented sleep is frequently accompanied by measurable impairments in executive function and emotional control [3,4]. From a mechanistic perspective, rapid eye movement (REM) sleep is closely associated with synaptic remodeling and affective processing, whereas non-rapid eye movement (NREM) sleep—particularly slow-wave activity—contributes to cerebral energy restoration and glymphatic clearance. Although differentiating these states is clinically important, the prevailing gold standard—manual PSG scoring—remains time-consuming and is subject to considerable inter-rater variability [5]. The burden and subjectivity of manual polysomnographic scoring have therefore accelerated interest in automated solutions that are both reproducible and scalable. In contrast to conventional machine-learning workflows that depend on empirically designed features, deep neural networks (DNNs) learn task-relevant representations directly from physiological waveforms in an end-to-end manner. By capturing temporal structure across multiple scales, such models have shown improved generalization to diverse subjects and recording settings [6,7]. However, practical deployment in real-world polysomnography remains challenged by pronounced signal heterogeneity and modality-specific artifacts, which can compromise the reliability of naive fusion strategies and motivate more structured modeling of cross-channel interactions.

In the context of the journal Symmetry, our design can be viewed as enforcing a structured balance across modalities: modality-specific encoders produce comparable token representations, while the gating node, as shown in Figure 1, selectively down-weights unreliable EOG/EMG tokens to prevent asymmetric artifact leakage into the fused sequence. This mechanism promotes representational consistency in the unified multimodal tensor used for subsequent temporal reasoning, thereby aligning the model’s robustness objective with a symmetry-related notion of balanced cross-modal integration. Consistent with this symmetry-motivated view, multimodal staging can be interpreted as seeking an approximately invariant shared representation across channels, while preventing asymmetric, time-varying artifacts from one modality from contaminating the others. This perspective is particularly relevant for PSG, where auxiliary channels may contribute complementary evidence when reliable, yet may also introduce strongly non-stationary disturbances when artifacts dominate. Accordingly, robustness hinges not only on extracting discriminative spectral–temporal features, but also on enforcing a balanced integration mechanism that preserves informative agreement and suppresses unreliable asymmetries.

Figure 1. Overall architecture of the proposed G-CMTF Net, illustrating the cross-modal fusion pipeline, spectrum-aware bio-temporal feature encoding, and global temporal modeling with Conformer Encoders.

Given the high annotation cost of PSG and the prevalence of distributional shifts across cohorts, a series of advanced learning paradigms has been introduced to improve both robustness and adaptability in sleep staging models. In this context, transfer learning has been widely adopted, in which representations learned from large-scale datasets are reused so that models can be adapted to smaller cohorts or population-specific recordings with reduced reliance on extensive retraining [8]. Closely related to this line of work, meta-learning approaches—exemplified by MetaSleepLearner—have been formulated to treat rapid subject-level adaptation as an explicit objective; this is typically achieved via model-agnostic optimization procedures that enable efficient updating when new subjects are encountered [9]. This emphasis on adaptability becomes particularly relevant when inter-subject variability dominates performance degradation in real-world deployments. In parallel, when computational and memory budgets constitute the primary bottleneck, knowledge distillation has been explored as a practical means of reducing model complexity, allowing compact student models to approximate the predictive behavior of larger teachers and thereby supporting deployment on wearable and embedded platforms without substantial loss of accuracy [10]. A growing body of recent work suggests that hybrid architectures combining multi-scale convolutional feature extraction with temporal attention mechanisms are effective at modeling the coupled spectral–temporal structure of sleep-related physiological signals [11,12,13]. These local–global modeling strategies have advanced benchmark performance; nonetheless, robustness in practical PSG settings remains limited when auxiliary channels become intermittently artifact-dominated and are fused in a uniform manner. This reliability bottleneck is further exacerbated under distribution shifts and inter-subject variability, where noisy cross-channel cues can be propagated into the shared representation and impair generalization.

These observations suggest that robust PSG-based staging requires representations that couple multi-scale temporal structure with physiologically grounded spectral cues, as well as fusion mechanisms that explicitly account for time-varying modality reliability so that artifact-prone auxiliary evidence does not corrupt shared features. Here, reliability is treated as an implicit, representation-driven quantity inferred during feature learning, rather than an externally defined signal-quality metric. Accordingly, G-CMTF Net is developed as an end-to-end framework that integrates EEG with auxiliary EOG/EMG through reliability-aware gating and models long-range sleep dynamics using convolution-augmented attention. Guided by the above considerations, G-CMTF Net is designed around three key components, each targeting a recurring source of degradation in practical PSG-based staging:

• Spectro-temporal heterogeneity (slow waves vs. spindles). A spectro-temporal disentanglement module is introduced, where multi-scale convolutions are combined with spectral prior injection (FFT-derived band-power embeddings) to preserve physiologically defined oscillatory cues across disparate time scales.
• Non-stationary artifacts in auxiliary modalities (EOG/EMG). To avoid indiscriminate fusion, a reliability-aware Gated Cross-Modal Fusion mechanism is employed to re-weight auxiliary streams at the epoch level, attenuating artifact-dominated segments while retaining informative cross-modal context.
• Long-range temporal dependencies. A convolution-augmented self-attention encoder (Conformer) is used to model both local morphology and global stage-transition structure, which are difficult to capture with shallow temporal models.

While the present study is instantiated on clinical PSG, the underlying failure mode targeted in this work—namely, time-varying degradation of auxiliary sensor reliability—also arises in ubiquitous sleep monitoring pipelines, where wearable biosensors and contactless sensors are frequently affected by motion, placement changes, and environment-dependent disturbances. Accordingly, the proposed reliability-aware fusion mechanism is positioned as a principled step toward robust multi-sensor sleep staging beyond laboratory-grade montages (see Section 2.3). The proposed framework is evaluated on Sleep-EDF-20 and Sleep-EDF-78 [14,15], yielding competitive performance while preserving sensitivity on transitional and minority stages under subject-wise evaluation.

To provide an intuitive view of the proposed workflow, Figure 2 presents a block-level overview of G-CMTF Net from multimodal PSG inputs to sleep-stage prediction; the full architectural specification is detailed in Figure 1.

Figure 2. High-level overview of the proposed G-CMTF Net for PSG-based sleep staging. The schematic highlights the end-to-end pipeline (spectro-temporal encoding, reliability-aware cross-modal fusion, and temporal modeling).

2. Related Work

2.1. Transformers and Cross-Modal Temporal Modeling for Sleep Staging

The evolution of automated sleep staging has historically mirrored advancements in sequence modeling. Early deep learning frameworks established the efficacy of hybrid architectures that couple Convolutional Neural Networks (CNNs) for invariant feature extraction with Recurrent Neural Networks (RNNs) for transition rule learning [16]. However, LSTM-based paradigms are inherently impeded by gradient saturation and high computational latency when processing whole-night recordings [17]. To circumvent these bottlenecks, U-Time proposed a fully convolutional encoder–decoder framework, demonstrating that analyzing physiological signals via multi-scale convolutional streams could achieve state-of-the-art temporal segmentation without the computational burden of recurrent layers [18]. TinySleepNet further reported that, when the hybrid design was explicitly optimized for compression, the parameter budget could be substantially reduced while sleep scoring performance was largely preserved [19].

As modeling requirements have evolved from short-range morphology characterization to long-range sleep-cycle dependency capture, a clear transition toward Transformer-based designs has been observed. Within this line of research, TransSleep leverages self-attention to target boundary epochs that are prone to misclassification; a transition-aware attention mechanism is introduced so that hierarchical semantic context can be encoded, which in turn reduces confusions that frequently persist in recurrent baselines during stage transitions [20]. A related yet distinct strategy has been adopted by hybrid architectures such as EEG-Conformer, where local temporal–spatial patterns are first extracted through a convolutional module and global dependencies are subsequently modeled by self-attention. By coupling these complementary inductive biases within a unified framework, competitive performance has been reported for electroencephalogram (EEG) decoding tasks [21].

Beyond single-channel pipelines, multi-view learning has been increasingly explored for sleep staging, as complementary information can be derived when signals are represented from distinct perspectives. XSleepNet exemplifies this direction by jointly encoding raw waveforms together with their time–frequency projections, thereby exploiting the complementarity between the two representations for improved stage discrimination [22]. Along similar lines, SalientSleepNet employs a U-shaped architecture to detect salient waveforms across heterogeneous modalities [23]. This trend, nevertheless, exposes a recurring limitation: multimodal performance is often contingent on heuristic fusion rules that implicitly presume consistent and non-conflicting contributions across modalities. For example, SalientSleepNet combines EEG and EOG representations using simple element-wise addition, which does not explicitly account for modality-specific corruption or cross-modal conflict. In clinical recordings, where EOG and EMG channels are frequently affected by non-stationary artifacts, the absence of a mechanism that can down-weight noise-dominant modalities on a per-epoch basis becomes a consequential gap in current representation learning practice. The above observation suggests that static arithmetic fusion should be replaced by context-aware integration strategies, in which signal reliability is estimated and fusion weights are adjusted accordingly.

2.2. Spectro-Temporal Representation and Multi-Scale Feature Dynamics

Sleep neurophysiology exhibits spectral activity that is distributed across markedly different time scales, ranging from the high-amplitude delta oscillations (0.5–4 Hz) that dominate N3 to the short-lived sleep spindles (11–16 Hz) that are characteristic of N2 [24]. Such heterogeneity imposes a practical constraint on conventional convolutional encoders, whose receptive fields are fixed once the kernel configuration is specified. Consequently, a single receptive-field scale is unlikely to represent long-duration slow waves and brief micro-events with comparable fidelity; when the field is enlarged, transient structures may be smoothed, whereas overly local fields tend to underrepresent slow-wave dynamics that unfold over extended intervals. This observation motivates multi-resolution feature extraction, as exemplified by early designs such as DeepSleepNet, which adopted a two-branch convolutional architecture with parallel pathways of different kernel sizes to separate temporal features at multiple resolutions [16]. Recent architectures such as MSTCN have adopted a genuinely multi-scale temporal convolutional design, in which parallel convolutional pathways with heterogeneous kernel sizes and dilation factors are deployed to systematically capture discriminative temporal structures spanning short- and long-range dynamics [25]. Furthermore, contemporary hybrid paradigms demonstrate that aggregating these multi-scale contexts—often via attention-based or CNN-Transformer fusion—ensures that both local morphological details and broad contextual dependencies are rigorously preserved [13,26].

Closely related to multi-scale temporal modeling, the explicit incorporation of spectral priors provides a principled means of aligning learned representations with established clinical scoring conventions. Although convolutional architectures are capable of implicitly approximating spectral decompositions through learned filters, representations derived purely from data-driven optimization are not guaranteed to respect the well-defined frequency boundaries prescribed by standard scoring criteria [27]. To address this deficit, recent paradigms have moved towards multi-view learning, treating different signal representations as complementary input streams [28]. For example, the MST-GCN framework introduces a multi-view spatial–temporal graph architecture in which multiple adjacency matrices articulate distinct inter-channel and inter-epoch relationships. This formulation enables the model to integrate complementary structural dependencies that govern stage-to-stage transitions with greater fidelity [29]. Building on this line of research, architectures such as MVFSleepNet further show that coupling multi-scale convolutional features derived from raw PSG signals with time–frequency representations, and integrating them through an adaptive multi-view attention module, yields a richer and more balanced characterization of spectral and temporal dynamics. This joint representation strategy has been empirically demonstrated to enhance sleep-stage classification performance across multiple public datasets [30]. Nevertheless, a prevailing bottleneck in these multi-view frameworks is the computational redundancy associated with processing high-dimensional spectrogram images alongside raw waveforms. Furthermore, existing fusion mechanisms often treat spectral and temporal features as static parallel streams, lacking the flexibility to dynamically recalibrate spectral priors based on the signal’s real-time quality. Consequently, establishing a unified, lightweight framework that explicitly disentangles spectro-temporal dynamics while maintaining computational efficiency remains an important yet insufficiently explored direction.

2.3. Wearable and Ubiquitous Sleep Monitoring

Driven by ubiquitous computing, sleep monitoring has increasingly been extended beyond sleep laboratories toward longitudinal, at-home assessment. In this setting, both wearable biosensors and contactless sensors have been explored to reduce acquisition burden, albeit at the cost of reduced physiological observability and increased susceptibility to non-stationary interference. While these paradigms represent an important direction for large-scale deployment, this work deliberately focuses on PSG as a controlled reference setting to isolate the effects of reliability-aware multimodal fusion under well-annotated conditions.

For wearable biosensing, wrist-worn photoplethysmography (PPG) and inertial signals constitute a common sensing configuration. Representative work has shown that stage estimation from PPG can be improved via transfer learning, leveraging large annotated datasets from related cardiac modalities to mitigate limited PPG-labeled data [31]. A parallel direction seeks to preserve richer neurophysiological information while retaining wearability. Ear-centered sensing provides a notable compromise: ear-EEG has been validated as a minimally obtrusive alternative capable of supporting automatic sleep staging [32]. More recently, EarSleep demonstrated that in-ear acoustic sensing can capture sleep activities and infer sleep stages by interpreting in-ear body sounds [33].

Beyond wearables, non-contact approaches such as radar-based sleep monitoring have also been validated against PSG in home-like environments [34]. Despite methodological diversity, a shared challenge across these ubiquitous sensing modalities is that signal quality may degrade intermittently and in a stage-dependent manner. This observation motivates reliability-aware integration mechanisms, and provides additional context for the gating-based fusion principle adopted in this work for PSG, where auxiliary channels likewise exhibit time-varying contamination. The proposed gating-based fusion principle is modality-agnostic and can be transferred to ubiquitous sensing scenarios where signal reliability typically varies more severely.

3. Motivation

Sleep staging from PSG is shaped by recurrent failure modes rooted in both physiology and acquisition artifacts. Figure 3 summarizes three such factors and the corresponding design choices in G-CMTF Net.

Figure 3. Motivation-driven overview of G-CMTF Net for PSG-based sleep staging. Key challenges are mapped to the corresponding modules and expected benefits. The orange and purple triangles mark transient events, while the gray shaded boxes indicate noisy segments that are suppressed by the gated fusion.

First, discriminative signatures span disparate time scales: slow-wave activity develops over extended intervals, whereas micro-events such as spindles and K-complexes are temporally localized. Because single-scale convolutions impose an effectively fixed receptive field, a unified representation may under-resolve either long-duration dynamics or short-lived transients. Moreover, even when multi-scale encoders are adopted, learned features are not guaranteed to respect clinically defined frequency bands. This observation motivates injecting explicit spectral priors via FFT-derived band-power cues to steer representation learning toward physiologically interpretable regimes.

Relatedly, auxiliary channels (EOG/EMG) provide complementary evidence but exhibit time-varying reliability in practice. Existing fusion pipelines often weight modalities to optimize average discrimination (e.g., channel-wise attention in MASleepNet [35]); nevertheless, modality contributions are rarely regulated explicitly under non-stationary channel quality, allowing artifact-dominated segments to contaminate the shared latent space [36]. G-CMTF Net therefore formulates fusion as a reliability-sensitive operation and employs epoch-level Gated Cross-Modal Fusion to suppress unreliable auxiliary evidence while preserving informative context.

Finally, accurate staging depends on long-range temporal structure, including stage persistence and constrained transitions. To couple local morphology with global dynamics, a convolution-augmented self-attention encoder is adopted to model fine-grained waveform patterns together with sequence-level transition regularities.

4. G-CMTF Net: Gated Cross-Modal and Temporal Fusion for Sleep Staging

To rigorously deconvolute the spectral–temporal heterogeneity of polysomnography while mitigating cross-channel artifacts, we formulate the G-CMTF Net, which integrates Gated Cross-Modal Fusion with hierarchical temporal modeling for sleep staging. This architecture is instantiated via a hierarchical pipeline that synergizes three coupled functional blocks: (1) a spectro-temporal disentanglement frontend, which augments multi-scale convolutional feature maps with explicit band-power priors to enforce frequency-domain fidelity; (2) a Gated Cross-Modal Fusion (GCMF) interface, designed to autonomously attenuate noise-corrupted modalities through learnable reliability weights before integration; and (3) a Conformer-based Temporal Encoder, which synthesizes local inductive biases with global self-attention mechanisms to reconstruct long-range sleep cycle transitions. The schematic workflow of the proposed framework is illustrated in Figure 1.

4.1. Overall Architecture

4.1.1. Problem Formulation

Formally, let the polysomnography recording for a single subject be represented as a sequence of N epochs, denoted as $\mathcal{X}=\{X_1,X_2,\dots ,X_N\}$. Each epoch $X_t\in {\mathbb{R}}^{C\times L}$ comprises multi-channel physiological signals, where C denotes the channel set $\{EEG, EOG, EMG\}$ and L represents the sampling length (typically $L=3000$ for a 30 s epoch at 100 Hz). The objective of automatic sleep staging is to learn a non-linear mapping function $\mathcal{F}:\mathcal{X}\to \mathcal{Y}$, where $\mathcal{Y}=\{y_1,y_2,\dots ,y_N\}$ corresponds to the sequence of discrete sleep stages $y_t\in \{W, N1, N2, N3, REM\}$ strictly adhering to AASM standards [37].

4.1.2. Architectural Overview

To approximate the mapping $\mathcal{F}$ under cross-channel noise and modality-dependent corruption, the Gated Cross-Modal and Temporal Fusion Network (G-CMTF Net) is developed. As shown in Figure 1, the input $X_t$ is processed through three hierarchically coupled stages:

• Spectro-Temporal Feature Extraction: For each modality, raw waveforms are encoded by a Multi-Scale Sleep Stage CNN, in which multiple receptive-field scales are employed so that local sleep-related morphologies can be represented without committing to a single temporal resolution. In parallel, spectral priors are incorporated explicitly by a dedicated band-power embedding module. This design is intended to preserve frequency-domain specificity and to ensure that the learned representations remain consistent with clinically defined oscillatory bands in neurophysiology.
• Gated Cross-Modal Fusion: Since the reliability of different channels may vary across epochs, the resulting embeddings are forwarded to a Gated Cross-Modal Fusion (GCMF) module, where auxiliary modalities are re-weighted according to their estimated reliability prior to fusion. Drawing inspiration from channel-wise excitation principles [38], this module computes learnable reliability gates to autonomously suppress artifact-corrupted modalities (e.g., noisy EMG) before feature integration.
• Global Temporal Context Modeling: The fused multimodal tokens are finally projected into a Conformer Encoder. By synthesizing local convolutional inductive biases with global self-attention mechanisms [39], this encoder reconstructs long-range sleep cycle transitions and inter-epoch dependencies with enhanced robustness.

The model is optimized end-to-end under a composite training objective that combines a focal loss term, introduced to mitigate the pronounced class imbalance typical of sleep-staging corpora, with auxiliary tasks tailored to waveform-level characteristics. This joint formulation encourages the network to remain sensitive to under-represented sleep stages while exploiting complementary supervisory signals.

4.2. Spectro-Temporal Feature Disentanglement

To resolve the scale-resolution uncertainty inherent in single-view modeling, the G-CMTF Net explicitly decouples feature extraction into two complementary streams: a spectral prior injection module and a multi-scale temporal encoder.

4.2.1. Explicit Spectral Prior Injection

Given an input epoch $X\in {\mathbb{R}}^{C\times L}$, we first transform the raw waveform into the frequency domain to capture steady-state oscillatory patterns. Let $x_c\left(t\right)$ denote the signal of the c-th channel. We apply Welch’s method to compute the power spectral density (PSD), $S_c\left(f\right)$, which provides a robust estimate of spectral characteristics by averaging periodograms over overlapping segments. To align with clinical scoring rules [37], we aggregate the spectral energy into K canonical frequency bands (e.g., $\delta ,\theta ,\alpha ,\sigma ,\beta ,\gamma$). The band-power vector ${\mathbf{p}}_c\in {\mathbb{R}}^K$ is derived as

$${\mathbf{p}}_{c,k}=log\left(1+{\int }_{f_{\mathrm{low}}^{\left(k\right)}}^{f_{\mathrm{high}}^{\left(k\right)}}{S}_c\left(f\right)df\right),k=1,\dots ,K$$

(1)

The logarithmic transformation mitigates the skewness of power distributions. These vectors are projected via a linear embedding to form the spectral token $E_{spec}$, serving as a global frequency anchor.

Physiologically, transitional stages such as N1 are typically characterized by mixed-frequency and low-amplitude activity, for which discriminative evidence is often manifested as subtle band-wise power shifts rather than salient morphology. By injecting band-power priors as an explicit global anchor, the learned representation is encouraged to preserve clinically grounded spectral contrasts (e.g., theta emergence and alpha attenuation), thereby complementing convolutional detection of transient events (e.g., spindles in the sigma band) and improving separability under inter-subject spectral variability.

4.2.2. Multi-Scale Temporal Convolution

Parallel to spectral analysis, we design a Multi-Scale Convolutional (MSC) encoder to disentangle morphological transients across varying temporal resolutions. The detailed architecture is illustrated in Figure 4. Formally, let $X\in {\mathbb{R}}^{C\times L}$ denote the standardized input epoch. The encoder ${\Phi }_{MSC}$ deploys three parallel branches $\mathcal{B}=\left\{\mathrm{low},\mathrm{mid},\mathrm{high}\right\}$ to extract hierarchical representations. The feature map produced by branch $b\in \mathcal{B}$ is defined as

$$H_b={\mathcal{F}}_b(X;{\theta }_b),\mathrm{with}{\mathcal{F}}_b\sim \mathrm{Conv}1\mathrm{D}(k_b,s_b).$$

(2)

In this setting, $k_b$ and $s_b$ are used to denote the kernel size and stride of branch b, respectively. Because sleep-related events are expressed over heterogeneous temporal spans, the branches are designed to impose complementary inductive biases rather than sharing an identical receptive-field configuration. The low-frequency branch is configured with a larger kernel ($k_{\mathrm{low}}=31$), by which slow-wave activity that persists over longer intervals can be represented within an extended temporal context. The high-frequency branch is implemented using a depthwise–separable Conv1D with $k_{\mathrm{high}}=40$ (depthwise convolution with $\mathrm{groups} =$ in_channels followed by a $1\times 1$ pointwise projection). An SE-style recalibration and a pooling operator are then applied to emphasize transient micro-events (e.g., spindle-/EMG-related bursts) while keeping the computational footprint bounded. The Mid-Frequency Branch is implemented with ConvNeXt blocks, where inverted bottlenecks are adopted to trade representational capacity against computational cost in a controlled manner [40].

Figure 4. Architecture of the proposed multi-scale spectro-temporal CNN frontend. Three parallel convolutional branches with complementary receptive fields are employed to capture sleep-related patterns across different temporal scales. The resulting features are temporally aligned, concatenated, and refined through projection and residual temporal modeling to form bio-temporal embeddings.

Owing to the use of heterogeneous strides, the feature maps $H_{\mathrm{low}},H_{\mathrm{mid}},H_{\mathrm{high}}$ are produced at different temporal resolutions. To enable coherent fusion across branches, a common target length is specified as $T^{\prime }={min}_b\left(\mathrm{len}\left(H_b\right)\right)$, and a linear interpolation operator $\mathcal{U}(·)$ is applied to align each feature map accordingly. After temporal alignment, the unified representation is formed by channel-wise concatenation followed by projection:

$$Z_{\mathrm{agg}}=\mathrm{BN}\left({\mathbf{W}}_p\ast \left[\mathcal{U}\left(H_{\mathrm{low}}\right)\Vert \mathcal{U}\left(H_{\mathrm{mid}}\right)\Vert \mathcal{U}\left(H_{\mathrm{high}}\right)\right]\right),$$

(3)

where ‖ denotes concatenation along the channel dimension and ${\mathbf{W}}_p$ denotes a $1\times 1$ convolution used to project the aggregated features into a compact embedding space.

After projection, the fused sequence is refined by a lightweight channel–temporal recalibration pipeline. First, a channel-wise SE recalibration is applied to re-weight feature responses:

$$Z_{\mathrm{ch}}=Z_{\mathrm{agg}}\otimes \sigma \left({\mathcal{A}}_{\mathrm{ch}}\left(Z_{\mathrm{agg}}\right)\right),$$

(4)

where ${\mathcal{A}}_{\mathrm{ch}}(·)$ denotes a squeeze–excitation operator producing channel-wise weights, which are broadcast along the temporal axis.

Next, temporal refinement is implemented by a dilated residual block followed by a temporal SE recalibration. For notational compactness, their composition is denoted as ${\mathcal{A}}_{\mathrm{temp}}(·)$:

$$E_{\mathrm{temp}}={\mathcal{A}}_{\mathrm{temp}}\left(Z_{\mathrm{ch}}\right),{\mathcal{A}}_{\mathrm{temp}}\left(Z\right)=\left(Z+{\mathcal{R}}_{\mathrm{dil}}\left(Z\right)\right)\otimes \sigma \left({\mathcal{A}}_{\mathrm{se}}\left(Z+{\mathcal{R}}_{\mathrm{dil}}\left(Z\right)\right)\right),$$

(5)

where ${\mathcal{R}}_{\mathrm{dil}}(·)$ is a dilated temporal residual block (dilation $= 2$ in our implementation) and ${\mathcal{A}}_{\mathrm{se}}(·)$ denotes the temporal SE recalibration. This design matches the implementation in which the projected sequence is refined by residual temporal modeling and then re-weighted by a temporal SE recalibration to suppress artifact-amplified fluctuations while preserving transition-relevant dynamics.

The channel-wise and temporal SE operators in the MSC encoder are introduced as intra-modality feature recalibration. Their role is to rescale feature responses within a single modality by leveraging global pooling statistics, thereby attenuating locally amplified fluctuations and emphasizing discriminative morphology. Importantly, these SE-style operators do not determine the relative contribution of EEG versus auxiliary streams during fusion; instead, they regularize the feature formation process prior to cross-modal interaction.

Through this joint channel–temporal recalibration, the resulting bio-temporal embedding $E_{\mathrm{temp}}$ is rendered less sensitive to artifactual perturbations while retaining discriminative sleep-related patterns [38].

The MSC encoder comprises three parallel branches whose outputs are aligned to a common temporal length before fusion. (1) The low-frequency branch applies a strided Conv1D ($k=31$, $s=2$, $p=15$) producing 16 channels, followed by MaxPool1D ($k=2$) and two residual Conv1D blocks with downsampling (16→32→128; each block uses stride 2). (2) The mid-frequency branch adopts a ConvNeXt-style stem (Conv1D $k=7$, $s=2$, $p=3$ + MaxPool1D $k=2$) followed by two residual blocks, yielding 64 channels. (3) The high-frequency branch uses a depthwise Conv1D ($k=40$, $s=1$, $p=20$; groups$=$in_channels) with a pointwise $1\times 1$ projection to 32 channels, together with SE-style recalibration and MaxPool1D ($k=2$). Let $T_{\mathrm{low}},T_{\mathrm{mid}},T_{\mathrm{high}}$ denote the resulting temporal lengths; the three feature maps are linearly interpolated to $T^{\prime }=min(T_{\mathrm{low}},T_{\mathrm{mid}},T_{\mathrm{high}})$, concatenated (224 channels), and projected to 128 dimensions by a $1\times 1$ Conv1D with BatchNorm and dropout. Finally, channel-wise recalibration and a dilated temporal residual refinement block are applied, followed by an SE-style recalibration, to produce the bio-temporal embeddings.

4.3. Gated Cross-Modal Fusion

In multi-channel sleep recordings, auxiliary signals such as EOG and EMG may alternately provide complementary physiological context or be dominated by artifacts, giving rise to a recurring noise–information trade-off. Unlike static fusion, GCMF operates as a reliability-aware filter. To explicitly account for this uncertainty during fusion, a Gated Cross-Modal Fusion (GCMF) mechanism is formulated, in which auxiliary information is integrated in a reliability-aware manner rather than through static aggregation.

Let $Z_{\mathrm{main}}\in {\mathbb{R}}^{T\times D}$ denote the EEG feature sequence (Query) and $Z_{\mathrm{aux}}\in {\mathbb{R}}^{T^{\prime }\times D}$ denote the auxiliary EOG/EMG features (Key/Value). The fusion process involves two sequential steps:

First, we introduce a cross-attention module, following the encoder–decoder attention mechanism originally formalized in the Transformer architecture [17]. This enables the auxiliary modality to attend selectively to temporally informative components of the primary EEG stream, thereby achieving consistent alignment between their dynamics:

$$\widehat{Z}\mathrm{aux}=\mathrm{Softmax}\left({\displaystyle \frac{Z_{\mathrm{main}}{W}_Q{\left(Z_{\mathrm{aux}}{W}_K\right)}^{\top }}{\sqrt{d_k}}}\right)\left(Z_{\mathrm{aux}}{W}_V\right).$$

(6)

In contrast to the above intra-modality recalibration, the gating vector g in GCMF is designed as a cross-modal reliability gate. It is computed from the aligned auxiliary representation and is used to modulate the auxiliary contribution before integration with the EEG stream. Under this formulation, g explicitly controls whether auxiliary evidence should be propagated into the shared latent space on a per-epoch basis, which directly targets non-stationary artifact contamination in EOG/EMG.

Second, to suppress noise, a learnable Gating Network (illustrated in the bottom panel of Figure 5) computes a channel-wise reliability score $g\in {[0,1]}^D$ based on the global context of the aligned features:

$$g=\sigma \left(\mathrm{Convex}\left(\delta \left(\mathrm{Convsq}\left(\mathrm{AvgPool}\left({\widehat{Z}}_{\mathrm{aux}}\right)\right)\right)\right)\right),$$

(7)

where $\sigma (·)$ is the Sigmoid function and $\delta (·)$ denotes the ReLU activation. While we do not employ a separate signal-quality regressor, we explicitly define the modal reliability as the dynamic importance weights learned through a squeeze–excitation mechanism. Here, $\mathrm{AvgPool}(·)$ denotes global average pooling over the temporal axis, yielding a D-dimensional summary that aggregates the overall activation statistics of ${\widehat{Z}}_{\mathrm{aux}}$ within the current epoch. ${\mathrm{Conv}}_{\mathrm{sq}}$ and ${\mathrm{Conv}}_{\mathrm{ex}}$ are implemented as two lightweight $1\times 1$ convolutions (equivalently, fully connected layers) with a reduction ratio r (we use $r=4$), forming a squeeze–excitation style bottleneck mapping. Instead of processing the auxiliary input in isolation, the gate is computed from the cross-attended intermediate representation. This allows the mechanism to dynamically recalibrate the fused features, emphasizing contextually relevant cross-modal patterns while suppressing noise introduced during the interaction. The fused representation is obtained by effectively filtering the auxiliary signal: $Z_{\mathrm{fused}}=\mathrm{Norm}(Z_{\mathrm{main}}+g\odot {\widehat{Z}}_{\mathrm{aux}})$. This ensures that only high-quality auxiliary information is integrated, while artifact-heavy segments are autonomously attenuated.

Figure 5. Gated Cross-Modal Fusion (GCMF) module. A channel-wise gating mechanism estimates modality reliability from global context and suppresses artifact-dominated auxiliary features prior to fusion with the EEG stream.

4.4. Global Temporal Context Modeling and Composite Optimization

4.4.1. Conformer-Based Sequence Modeling

Following the cross-modal fusion, the integrated feature sequence $Z_{\mathrm{fused}}\in {\mathbb{R}}^{T\times D}$ is passed through a Conformer Encoder to capture long-range temporal dependencies, such as the cyclic alternation between NREM and REM sleep stages. Unlike conventional Transformer architectures, the Conformer model integrates a convolution module within the self-attention block, as shown in Figure 6. This design merges the local inductive biases of convolutional neural networks (CNNs)—which are essential for capturing micro-events such as K-complexes—with the broader receptive field of Multi-Head Self-Attention (MHSA), facilitating robust modeling of both transient waveforms and prolonged sleep transitions across the night [39]. The output feature tokens are subsequently aggregated through global average pooling (GAP) and passed through a linear classifier to produce the conditional probability distribution $P\left(y\right|X)$.

Figure 6. Internal architecture of a Conformer Encoder block, in which global multi-head self-attention is integrated with a local convolutional module to jointly model long-range sleep-stage transitions and short-duration waveform events.

4.4.2. Composite Loss Function with Deep Supervision

To mitigate the class imbalance inherent in sleep staging, the primary objective $L_{\mathrm{main}}$ is instantiated as a focal loss regularized by label smoothing:

$$L_{\mathrm{main}}=-\sum _{c=1}^{N_{\mathrm{class}}}{(1-p_c)}^{\gamma }log\left(p_c^{\mathrm{LS}}\right),p_c^{\mathrm{LS}}=(1-ϵ)y_c+ϵ/N_{\mathrm{class}},$$

(8)

where $\gamma$ controls the focusing strength and $ϵ$ denotes the smoothing factor. Unless otherwise stated, $\gamma$ is set to $1.5$ and $ϵ$ is fixed to $0.1$ for both the main head and the auxiliary supervision head.

Deep supervision is imposed through an auxiliary classification head, whose loss $L_{\mathrm{aux}}$ is implemented as cross-entropy with label smoothing ($ϵ=0.1$). The overall objective is defined as

$$L_{\mathrm{total}}={\lambda }_1{L}_{\mathrm{main}}+{\lambda }_2\left(e\right)L_{\mathrm{aux}},$$

(9)

with ${\lambda }_1$ fixed to $1.0$. To avoid destabilizing early optimization, ${\lambda }_2\left(e\right)$ is linearly ramped from 0 to ${\lambda }_2^{max}$ during the first $E_{\mathrm{ramp}}$ epochs:

$${\lambda }_2\left(e\right)={\lambda }_2^{max}·min\left(1,{\displaystyle \frac{e}{E_{\mathrm{ramp}}}}\right),{\lambda }_2^{max}=0.3,E_{\mathrm{ramp}}=10.$$

(10)

To address class imbalance—most notably the scarcity of N1—a composite objective (Focal Loss with Label Smoothing) was adopted in this work. Since imbalance can also be handled through established, architecture-agnostic techniques, two widely used alternatives were included as baselines under an identical network configuration: (i) Class-Weighted Cross-Entropy (W-CE), where penalties are scaled by inverse class frequency; and (ii) balanced re-sampling, where minority classes are oversampled during training to approximate a balanced mini-batch distribution. A quantitative comparison is reported in Section 5.7.

5. Experiments and Results

5.1. Datasets

All experiments were conducted using the Sleep-EDF Database Expanded (Sleep-EDFX), a publicly accessible polysomnography repository hosted on PhysioNet [14,15]. In this study, analysis was restricted to the Sleep Cassette subset, which contains full-night PSG recordings acquired from healthy subjects and is commonly adopted for modeling baseline sleep architecture in the absence of pharmacological intervention.

Sleep stages in Sleep-EDFX were originally annotated by clinical experts according to the Rechtschaffen and Kales (R&K) criteria. Following standard practice, these annotations were mapped to the AASM convention by merging stages N3 and N4 into a single deep sleep stage (N3). Epochs labeled as Movement (M) or Unclassified (?) were excluded from subsequent analysis to ensure label consistency.

To facilitate both benchmarking and robustness evaluation, two experimental cohorts were derived from the Sleep Cassette subset:

• Sleep-EDF-20 which consists of 39 recordings from 20 subjects (SC4001–SC4192) and serves as a widely used reference benchmark for comparison with existing deep learning methods.
• Sleep-EDF-78 comprising 153 recordings from 78 subjects, which introduces substantially greater inter-subject variability and provides a more stringent test of model generalization.

The class distribution statistics for both cohorts are summarized in Table 1.

Table 1. Experimental settings and dataset statistics. The symbol # denotes the total number of 30-s epochs.

Dataset	No. of Subjects	Channels	Evaluation Scheme	Sampling Rate	Class Distribution
					Wake	N1	N2	N3	REM	# Total
Sleep-EDF-20	20	EEG, EOG EMG	5-fold subject-wise CV	100 Hz	8285 (19.6%)	2804 (6.6%)	17,799 (42.1%)	5703 (13.5%)	7717 (18.2%)	42,308
Sleep-EDF-78	78	EEG, EOG EMG	5-fold subject-wise CV	100 Hz	65,951 (33.7%)	21,522 (11.0%)	69,132 (35.4%)	13,039 (6.7%)	25,835 (13.2%)	195,479

CV: Cross Validation.

5.2. Data Preprocessing and Implementation Details

In accordance with the multimodal design of G-CMTF Net, three PSG channels were used throughout this study, namely EEG, horizontal EOG, and submental EMG, which, respectively, characterize cortical oscillations, ocular movements, and muscle tone. Raw signals were uniformly sampled at 100 Hz. To suppress low-frequency drift and high-frequency interference, zero-phase band-pass filtering was applied with cutoff frequencies of 0.2–40 Hz. The continuous recordings were then segmented into non-overlapping 30 s epochs, consistent with the AASM scoring convention. To reduce inter-subject amplitude variability, Z-score normalization was performed on a subject-wise basis for each channel, yielding zero-mean and unit-variance inputs.

The complete preprocessing workflow, including the parallel construction of time-domain waveforms and spectral priors, is summarized in Figure 7. Specifically, in addition to the normalized epoch sequences used for temporal encoding, a Welch-based power spectral density (PSD) estimate was computed per epoch to derive band-power vectors, which were subsequently embedded as explicit spectral cues. This design allows clinically interpretable oscillatory information to be retained during representation formation, rather than being inferred implicitly from raw waveforms alone.

Figure 7. Overview of the spectro-temporal input construction pipeline for G-CMTF Net. Time-domain sequences and explicit spectral representations are constructed in parallel and jointly provided as model inputs. The colored bars in the inset are illustrative and indicate a generic spectral distribution; the colors do not encode specific channels.

Experiments were conducted on two commonly adopted cohorts derived from the Sleep-EDFX Sleep Cassette subset. Sleep-EDF-20 comprises 39 recordings from 20 subjects (SC4001–SC4192) and is used as a standard benchmark for comparison with prior deep learning approaches. Sleep-EDF-78 extends the evaluation to 153 recordings from 78 subjects, introducing substantially greater inter-subject heterogeneity and therefore providing a more stringent test of robustness. For both cohorts, 5-fold cross-validation was employed to assess generalization. We strictly adopted subject-wise cross-validation, ensuring that recordings from the same subject never appeared in both training and testing sets. Each fold contained mutually exclusive subject sets, thereby preventing information leakage across folds. We report the mean and 95% confidence interval (CI) of Macro-F1 over 5-fold cross-validation. The CI is computed using the t-distribution with four degrees of freedom.

Model implementation and training were performed in PyTorch (v2.5.1+cu121) on a single NVIDIA RTX 4090 GPU. Parameters were optimized using AdamW with an initial learning rate of $5\times {10}^{-4}$ and a weight decay of $1\times {10}^{-2}$. A cosine-annealing schedule with warm restarts (CosineAnnealingWarmRestarts) was adopted with $T_0=10$, $T_{\mathrm{mult}}=1$, and ${\eta }_{min}={10}^{-6}$. Mini-batch training used a batch size of 16. For each cross-validation fold, training was run for at most 40 epochs and was terminated early if the validation macro F1-score failed to improve by more than ${10}^{-3}$ for 10 consecutive epochs. These hyperparameters were chosen based on preliminary validation within the training portion of each cross-validation split and were then kept fixed for all reported experiments.

To facilitate feasibility assessment in resource-constrained deployments, we report the number of trainable parameters, floating-point operations (FLOPs), and inference latency of G-CMTF Net. FLOPs are computed for a single 30 s epoch (100 Hz) with batch size 1, and latency is measured as the average wall-clock time over 100 repeated forward passes on an NVIDIA RTX 4090 (excluding data loading). The results are summarized in Table 2.

Table 2. Model complexity and inference latency of G-CMTF Net.

Model	Params (M)	FLOPs (G)	Latency (ms/epoch)
G-CMTF Net	2.86	0.810	15.8

5.3. Evaluation Metrics

To evaluate performance under class imbalance, we report Accuracy (ACC), Macro-F1 (MF1), Cohen’s $\kappa$, and Macro G-Mean (MGm). Let $T{P}_c,T{N}_c,F{P}_c,F{N}_c$ denote the confusion counts for class $c\in \{W,N1,N2,N3,REM\}$, and N be the total number of epochs.

(1)

Accuracy (ACC).

$$\mathrm{ACC}={\displaystyle \frac{{\sum }_{c=1}^{C}T{P}_c}{N}}.$$

(11)

(2)

Macro-F1 (MF1). We compute per-class precision and sensitivity (recall) and average class-wise F1 over C classes:

$${\mathrm{Pre}}_c={\displaystyle \frac{T{P}_c}{T{P}_c+F{P}_c}},{\mathrm{Sen}}_c={\displaystyle \frac{T{P}_c}{T{P}_c+F{N}_c}},$$

(12)

$$\mathrm{MF}1={\displaystyle \frac{1}{C}}\sum _{c=1}^C{\displaystyle \frac{2{\mathrm{Pre}}_c{\mathrm{Sen}}_c}{{\mathrm{Pre}}_c+{\mathrm{Sen}}_c}}.$$

(13)

(3)

Cohen’s $\kappa$. Agreement beyond chance is measured as

$$\kappa ={\displaystyle \frac{p_o-p_e}{1-p_e}},$$

(14)

where $p_o$ is the observed agreement (ACC) and $p_e$ is the chance agreement from class marginals.

(4)

Macro G-Mean (MGm). To reflect balanced sensitivity–specificity under imbalance, we compute

$${\mathrm{Spec}}_c={\displaystyle \frac{T{N}_c}{T{N}_c+F{P}_c}},\mathrm{MGm}=\sqrt{\mathrm{Macro}\left\{{\mathrm{Sen}}_c\right\}·\mathrm{Macro}\left\{{\mathrm{Spec}}_c\right\}},$$

(15)

where $\mathrm{Macro}\left\{x_c\right\}={\displaystyle \frac{1}{C}}{\sum }_{c=1}^C{x}_c$.

To explicitly evaluate performance around stage boundaries, we define a 30 s epoch index $t\in \{1,\dots ,T\}$ as transitional if it lies within a $\pm K$ neighborhood of any label-change boundary in the reference hypnogram. We note that the definition of transitional epochs is algorithmic rather than clinical, and is introduced solely as an evaluation proxy to quantify boundary-related uncertainty, rather than as a new physiological category. Let ${\left\{y_t\right\}}_{t=1}^T$ denote the ground-truth stage sequence. We define the set of boundary indices

$$\mathcal{C}=\{t\in \{1,\dots ,T-1\}\mid y_t\ne y_{t+1}\},$$

(16)

where each $t\in \mathcal{C}$ indicates a stage transition between epochs t and $t+1$. Then, the transitional set is defined as

$${\mathcal{T}}_K=\left\{i\in \{1,\dots ,T\}|\exists t\in \mathcal{C},t-K\le i\le (t+1)+K\right\},$$

(17)

and the complementary set ${\mathcal{N}}_K=\{1,\dots ,T\}\setminus {\mathcal{T}}_K$ is regarded as non-transitional epochs. We report macro Sensitivity (SEN) and macro geometric mean (MGm) on ${\mathcal{T}}_K$ and ${\mathcal{N}}_K$ for $K\in \{0,1,2\}$.

5.4. Performance Evaluation on Sleep-EDF Benchmarks

5.4.1. Overall Performance Analysis

The overall discriminative capability of G-CMTF Net was evaluated on Sleep-EDF-20 and Sleep-EDF-78 by benchmarking against representative baselines in Table 3.

Table 3. Overall performance (%) comparison on Sleep-EDF-20 and Sleep-EDF-78. Results are taken from the corresponding publications unless otherwise stated; “-” indicates not reported. Best results are highlighted in bold. Reported results may involve different preprocessing and evaluation protocols; we quote the numbers as reported in the publications.

Dataset	Methods	ACC	MF1	κ	SEN	SPE	MGm	Channels
Sleep-EDF-20	DeepSleepNet [16]	82.0	76.9	0.76	-	-	-	EEG
	SleepEEGNet [41]	84.3	79.7	0.79	-	-	-	EEG
	MultitaskCNN [11]	83.1	75.0	0.77	-	-	83.1	EEG
	TinySleepNet [42]	85.4	80.5	0.80	-	-	-	EEG
	XSleepNet1 [22]	85.2	79.8	0.798	79.0	95.9	87.0	EEG, EOG
	XSleepNet2 [22]	86.4	80.9	0.813	79.9	96.2	87.6	EEG, EOG
	Naïve Fusion [22]	83.4	77.8	0.773	77.1	95.5	85.8	EEG, EOG
	FCNN+RNN [22]	83.5	77.7	0.775	77.2	95.5	85.9	EEG, EOG
	MASleepNet [35]	84.5	78.9	0.785	78.4	95.7	86.6	Multi *
	MMASleepNet [36]	87.3	82.7	0.826	-	-	81.7	Multi *
	G-CMTF Net (Ours)	85.5 ± 2.0	81.3	0.802 ± 0.028	82.1	96.1	88.5	Multi *
Sleep-EDF-78	DeepSleepNet [42]	77.1	71.2	0.69	-	-	-	EEG
	SleepEEGNet [41]	80.0	73.6	0.73	-	-	-	EEG
	MultitaskCNN [11]	79.6	72.8	0.72	-	-	82.5	EEG
	TinySleepNet [42]	83.1	78.1	0.77	-	-	-	EEG
	XSleepNet1 [22]	84.0	78.4	0.777	77.1	95.6	85.9	EEG, EOG
	XSleepNet2 [22]	84.0	77.9	0.778	77.6	95.7	86.2	EEG, EOG
	Naïve Fusion [22]	82.5	76.9	0.757	75.8	95.3	85.0	EEG, EOG
	FCNN+RNN [22]	82.7	76.9	0.759	75.5	95.3	84.8	EEG, EOG
	MASleepNet [35]	82.6	76.1	0.75	75.9	95.2	85.0	Multi *
	MMASleepNet [36]	82.7	77.6	0.76	-	-	76.1	Multi *
	G-CMTF Net (Ours)	83.4 ± 1.4	78.2	0.771 ± 0.017	78.5	95.6	86.6	Multi *

* Multi: Indicates the use of the full core PSG channels for sleep staging (EEG + EOG + EMG).

On Sleep-EDF-20, G-CMTF Net attains a Macro-F1 of 81.3% with a Cohen’s $\kappa$ of 0.802, computed on pooled test predictions across the 5 folds. Across 5-fold subject-wise cross-validation, the fold-wise mean Macro-F1 is 81.29% (95% CI: 79.03–83.55), with a fold-wise mean $\kappa$ of 0.802 (95% CI: 0.765–0.835). When contrasted with EEG-only pipelines such as DeepSleepNet and SleepEEGNet, the gain is consistent with the additional physiological context provided by EOG/EMG, which is not accessible to single-stream models.

Evidence for the contribution of the proposed design is provided by two complementary analyses. First, the boundary-focused evaluation in Table 4 shows that SEN/MGm degrades substantially on the closest transition neighborhood (${\mathcal{T}}_0$) and recovers monotonically as the neighborhood expands to ${\mathcal{T}}_2$, indicating that a dominant fraction of residual errors concentrates around stage boundaries. Second, the ablation study in Table 5 demonstrates systematic degradations in MF1/$\kappa$ when reliability-aware gated fusion, band-power prior injection, or the multi-scale CNN frontend is removed. Taken together, these controlled comparisons support attributing the observed robustness gains to the proposed reliability-aware fusion and spectro-temporal representation constraints, rather than to qualitative interpretation alone.

Table 4. Macro SEN and MGm (%) on transitional and non-transitional epochs (mean ± std, 5-fold CV).

Subset	SEN (%)		MGm (%)
	Sleep-EDF-20	Sleep-EDF-78	Sleep-EDF-20	Sleep-EDF-78
All epochs ($\{1,\dots ,T\}$)	82.1 ± 2.2	78.5 ± 1.1	88.5 ± 1.5	86.6 ± 0.8
Transitional epochs (${\mathcal{T}}_0$)	61.5 ± 1.1	62.7 ± 1.5	73.7 ± 0.6	75.0 ± 1.1
Transitional epochs (${\mathcal{T}}_1$)	66.7 ± 0.9	66.5 ± 1.4	77.6 ± 0.5	77.7 ± 1.1
Transitional epochs (${\mathcal{T}}_2$)	69.6 ± 1.3	68.4 ± 1.3	79.6 ± 0.8	79.0 ± 1.0
Non-transitional epochs (${\mathcal{N}}_0$)	87.3 ± 2.5	82.0 ± 1.4	92.1 ± 1.6	89.1 ± 1.0
Non-transitional epochs (${\mathcal{N}}_1$)	89.0 ± 2.8	82.5 ± 1.5	93.2 ± 1.8	89.6 ± 1.0
Non-transitional epochs (${\mathcal{N}}_2$)	89.9 ± 3.1	82.7 ± 1.7	93.8 ± 1.9	89.9 ± 1.1

Note: Transitional epochs ${\mathcal{T}}_K$ are defined as epochs within a $\pm K$ neighborhood around each stage-boundary transition in the reference hypnogram; ${\mathcal{N}}_K$ is the complement.

Table 5. Ablation study on Sleep-EDF-20 and Sleep-EDF-78. Performance is reported as mean ± standard deviation across 5 folds. The p-value indicates the statistical significance of the difference in MF1 scores compared to the full G-CMTF Net model (paired t-test).

Model Variants	Sleep-EDF-20 (5-Fold)				Sleep-EDF-78 (5-Fold)
	ACC (%)	MF1 (%)	κ	p-Value	ACC (%)	MF1 (%)	κ	p-Value
G-CMTF Net	85.5 ± 2.0	81.3 ± 1.8	0.802 ± 0.03	–	83.4 ± 1.4	78.2 ± 1.2	0.771 ± 0.02	–
w/o Gating (Concat)	85.1 ± 2.5	80.8 ± 2.3	0.797 ± 0.03	0.31	83.1 ± 1.4	77.8 ± 1.3	0.767 ± 0.02	0.15
w/o Band-Power	85.5 ± 2.3	80.8 ± 2.5	0.800 ± 0.03	0.29	83.0 ± 1.3	77.2 ± 1.1	0.765 ± 0.017	0.005
w/o Multi-Scale	85.3 ± 2.4	80.6 ± 2.5	0.799 ± 0.03	0.23	83.1 ± 1.3	77.3 ± 0.9	0.766 ± 0.02	0.09
w/o EMG	85.3 ± 2.4	81.0 ± 2.4	0.800 ± 0.04	0.39	83.1 ± 1.2	77.2 ± 1.0	0.767 ± 0.01	0.02

A similar trend is observed on Sleep-EDF-78, where inter-subject heterogeneity is markedly increased. G-CMTF Net reaches 83.4% ACC and 78.2% MF1 (with $\kappa =0.771$), computed on pooled test predictions across the five folds. Collectively, the results suggest that combining (i) spectro-temporal priors for resolving scale-dependent ambiguity with (ii) reliability-aware gating for mitigating cross-channel artifacts improves robustness under non-stationary PSG noise, without sacrificing majority-stage specificity.

5.4.2. Class-Wise Performance Analysis

To elucidate where the proposed improvements arise, stage-wise F1 scores are reported in Table 6. Across existing baselines, N1 remains the most difficult stage, largely because its low-amplitude and mixed-frequency characteristics provide weak and easily perturbed cues, especially around transitions.

Table 6. Stage-wise F1 scores (%) on Sleep-EDF-20 and Sleep-EDF-78. Best results per stage are highlighted in bold. Only methods with publicly reported per-stage F1 scores are included.

Methods	W	N1	N2	N3	REM
Dataset: Sleep-EDF-20
DeepSleepNet [42]	84.7	46.6	85.9	84.8	82.4
TinySleepNet [42]	90.1	51.4	88.5	88.3	84.3
SeqSleepNet [42]	90.5	45.4	88.1	86.4	81.8
MultitaskCNN [11]	87.9	33.5	87.5	85.8	80.3
SleepEEGNet [41]	89.2	52.2	86.8	85.1	85.0
G-CMTF Net (Ours)	91.0	56.2	87.0	83.9	88.3
Dataset: Sleep-EDF-78
DeepSleepNet [42]	90.4	46.0	79.1	68.6	71.8
TinySleepNet [41]	92.8	51.0	85.3	81.1	80.3
SeqSleepNet [7]	92.2	47.8	84.9	77.2	79.9
MultitaskCNN [11]	90.9	39.7	83.2	76.6	73.5
SleepEEGNet [41]	91.7	44.1	82.5	73.5	76.1
G-CMTF Net (Ours)	93.2	51.4	84.7	76.6	85.0

On Sleep-EDF-20, G-CMTF Net attains an N1 F1 score of 56.2%, exceeding DeepSleepNet (46.6%) and MultitaskCNN (33.5%), and also outperforming other competitive EEG-based baselines (e.g., SleepEEGNet at 52.2%). Because N1 is characterized by low-amplitude mixed-frequency patterns and is frequently adjacent to stage boundaries, the interpretation of this gain is grounded in additional quantitative evidence beyond the per-class table. As indicated by the aggregated confusion matrices (Figure 8), the dominant error mode is concentrated on physiologically adjacent substitutions, particularly N1↔N2. Consistent with this structure, the boundary-focused evaluation (Table 4) shows that performance is lowest on ${\mathcal{T}}_0$ and increases as the boundary neighborhood expands to ${\mathcal{T}}_2$, implying that a substantial fraction of confusions arises in transition-adjacent regimes where N1 is overrepresented.

Figure 8. Aggregated confusion matrices over 5-fold cross-validation on Sleep-EDF-20 and Sleep-EDF-78. Values denote epoch counts. Most confusions occur between physiologically adjacent stages, notably N1↔N2 and REM↔N2. (a) Sleep-EDF-20, (b) Sleep-EDF-78.

Moreover, controlled ablations (Table 5) demonstrate that removing reliability-aware gating, band-power prior injection, or the multi-scale frontend yields systematic degradations in MF1/$\kappa$ across both cohorts. Accordingly, the improved N1 performance is interpreted as reflecting enhanced robustness under transition-prone and artifact-affected conditions, rather than a uniform gain across all stages.

In addition, the REM stage is recognized with high fidelity on this cohort (88.3%), suggesting that the injected spectral priors and multi-scale encoding contribute useful discriminative cues for stages with distinct oscillatory structure.

When the evaluation is extended to Sleep-EDF-78, where inter-subject heterogeneity is substantially higher, G-CMTF Net remains competitive on N1 (51.4%), ranking among the strongest reported results in the table. Meanwhile, the best scores for certain stages (e.g., N3 or REM) are achieved by other sequence models, indicating that stage-wise performance is not uniformly improved across all classes. Overall, the stage-wise profile suggests that the proposed design primarily benefits ambiguous and transition-prone epochs (particularly N1), while maintaining stable performance on the remaining stages under a more challenging cross-subject setting.

5.4.3. Stage-Wise Error Analysis via Aggregated Confusion Matrices

To further characterize stage-wise error patterns, the confusion matrices aggregated over 5-fold cross-validation are reported in Figure 8.

For both cohorts, predictions are largely concentrated on the diagonal, indicating that the dominant stages (Wake, N2, N3, and REM) are recognized with stable consistency under multimodal inputs. The residual errors are not uniformly distributed; instead, they are primarily localized to physiologically adjacent transitions. In particular, the most frequent confusions occur between N1 and N2 (e.g., N1→N2 and N2→N1), reflecting the well-known overlap of low-amplitude mixed-frequency activity at the onset of sleep. A second recurrent pattern is the interaction between N2 and N3 (notably N3→N2 and N2→N3), which is consistent with the gradual emergence of slow-wave activity and its boundary ambiguity under epoch-wise labeling. In addition, a non-negligible portion of REM epochs is mapped to N2, whereas Wake is occasionally mapped to N1, suggesting that short-lived transitions and artifact-affected segments remain challenging even with gated fusion. Overall, the confusion structure indicates that most errors are constrained to neighboring-stage substitutions rather than long-range jumps, which is aligned with the design goal of G-CMTF Net to suppress artifact-driven noise propagation while improving separability for transition-prone epochs.

To verify robustness around stage boundaries, we further evaluate the model on transitional subsets ${\mathcal{T}}_K$ and their complements ${\mathcal{N}}_K$ (defined in Section 5.3). As shown in Table 4, performance degrades near boundaries (e.g., ${\mathcal{T}}_0$) but consistently improves as the neighborhood expands from $K=0$ to $K=2$, indicating that most confusions concentrate in the closest vicinity of stage transitions.

5.5. Ablation Studies

To isolate the contribution of key components and to examine whether their effects are consistent across data scales, an ablation study was conducted on both Sleep-EDF-20 and Sleep-EDF-78 under an identical training protocol and subject-wise 5-fold split. The results are reported in Table 5.

Replacing the proposed Gated Cross-Modal Fusion with static concatenation (w/o Gating (Concat)) reduced mean MF1 by 0.5 pp on Sleep-EDF-20 (81.3% to 80.8%) and by 0.4 pp on Sleep-EDF-78 (78.2% to 77.8%), with $\kappa$ decreasing from 0.802 to 0.797 and from 0.771 to 0.767, respectively. These differences were not statistically significant under a paired two-sided t-test across the 5 folds (p = 0.31 and 0.15), but the consistent degradation across both cohorts suggests that reliability-aware re-weighting helps mitigate the impact of unreliable auxiliary cues during fusion.

Removing the band-power embedding branch (w/o Band-Power) caused a statistically significant degradation on Sleep-EDF-78 (p = 0.005) but not on Sleep-EDF-20 (p = 0.29). On Sleep-EDF-78, MF1 dropped by 1.0 pp (78.2% to 77.2%) with $\kappa$ decreasing from 0.771 to 0.765, whereas on Sleep-EDF-20 MF1 decreased by 0.5 pp (81.3% to 80.8%) with only a minor $\kappa$ reduction (0.802 to 0.800) and unchanged mean ACC (85.5%). This cohort-dependent behavior suggests that explicitly injected spectral cues become more beneficial when cross-subject variability is higher.

Simplifying the multi-scale CNN frontend to a single-scale configuration (w/o Multi-Scale) decreased MF1 by 0.7 pp on Sleep-EDF-20 (81.3% to 80.6%) and by 0.9 pp on Sleep-EDF-78 (78.2% to 77.3%), accompanied by $\kappa$ reductions from 0.802 to 0.799 and from 0.771 to 0.766, respectively. Although these reductions did not reach statistical significance (Sleep-EDF-20: p = 0.23; Sleep-EDF-78: p = 0.09), the consistent trend supports the premise that stage-defining PSG events span multiple temporal extents, for which a single receptive-field scale provides less complete coverage.

Excluding the EMG stream (w/o EMG) exhibited a cohort-dependent effect. On Sleep-EDF-78, removing EMG led to a significant MF1 drop of 1.0 pp (78.2% to 77.2%; p = 0.02), indicating that muscle-tone cues contribute to robustness under higher cross-subject heterogeneity. In contrast, on Sleep-EDF-20 the drop was smaller (0.3 pp; 81.3% to 81.0%) and non-significant (p = 0.39), suggesting EMG is less critical in smaller, more homogeneous cohorts.

5.6. Sensitivity to Spectral Band Definitions

A sensitivity analysis was performed to examine whether the FFT-based band-power priors injected by the spectro-temporal disentanglement module are sensitive to hand-chosen frequency cutoffs. Setting A uses a fine-grained $N=6$ partition: $\delta$ (0.5–4), $\theta$ (4–8), $\alpha$ (8–12), $\sigma$ (12–16), $\beta$ (16–30), and $\gamma$ (30–45) Hz; Setting B keeps $N=6$ but shifts all intermediate boundaries by +0.5 Hz; Setting C adopts a coarse clinical $N=4$ partition: $\delta$ (0.5–4), $\theta$ (4–8), $\alpha$ (8–13), and $\beta$ (13–30) Hz. Table 7 reports MF1 and Cohen’s $\kappa$, expressed as the mean ± standard deviation across 5-fold cross-validation, on Sleep-EDF-20 and Sleep-EDF-78.

Table 7. Sensitivity analysis of the model performance on Sleep-EDF-20 and Sleep-EDF-78 datasets under different frequency band definitions.

Configuration	Sleep-EDF-20		Sleep-EDF-78
	MF1 (%)	κ	MF1 (%)	κ
Setting A (Default)	81.3 ± 1.8	0.802 ± 0.03	78.2 ± 1.2	0.771 ± 0.02
Setting B (Shifted)	79.8 ± 2.8	0.795 ± 0.03	77.6 ± 1.2	0.767 ± 0.02
Setting C (Clinical)	80.2 ± 2.3	0.794 ± 0.03	76.8 ± 1.2	0.760 ± 0.02

Note: Setting A uses the proposed fine-grained bands. Setting B evaluates robustness to a +0.5 Hz boundary shift. Setting C adopts coarse clinical bands (Delta, Theta, Alpha, Beta).

Across both datasets, shifting the intermediate boundaries by +0.5 Hz (Setting B) decreased MF1 by 1.5 pp on Sleep-EDF-20 (81.3% to 79.8%) and by 0.6 pp on Sleep-EDF-78 (78.2% to 77.6%), accompanied by $\kappa$ reductions from 0.802 to 0.795 and from 0.771 to 0.767, respectively. Using coarse clinical bands (Setting C) yielded a larger degradation on Sleep-EDF-78, with MF1 decreasing by 1.4 pp (78.2% to 76.8%) and $\kappa$ dropping from 0.771 to 0.760; on Sleep-EDF-20, MF1 decreased by 1.1 pp (81.3% to 80.2%) with a $\kappa$ reduction from 0.802 to 0.794. Taken together, the proposed priors remain reasonably robust to changes in frequency boundaries and band definitions, while finer band partitioning provides a measurable advantage, particularly on Sleep-EDF-78.

5.7. Impact of Class Imbalance Strategies

Sleep staging corpora exhibit pronounced class imbalance, where minority stages (especially N1) occur far less frequently than dominant stages such as W and N2. When optimized with a uniform objective, the learning dynamics tend to be dominated by majority-class gradients, which can undermine macro-level discrimination and reduce agreement with reference annotations. To quantify the contribution of imbalance-aware optimization independently of architectural factors, three loss configurations were evaluated under an identical model and training protocol.

• Standard CE: conventional Cross-Entropy without class weighting or label smoothing.
• Weighted CE (W-CE): Cross-Entropy with class weight $w_c$ computed from the training split of each fold, where $w_c$ is inversely proportional to the empirical class frequency of class c.
• Proposed (Focal + LS): a focal objective with label smoothing, implemented with class-weighted $\alpha$ (shared with $w_c$), focusing parameter $\gamma =1.5$, and smoothing factor $ϵ=0.1$.

Table 8 reports Macro-F1 (MF1) and Cohen’s $\kappa$ on Sleep-EDF-20 and Sleep-EDF-78 as mean ± standard deviation over 5-fold cross-validation, thereby providing a direct comparison against alternative imbalance-aware objectives. Compared with Standard CE, inverse-frequency weighting offers a competitive baseline and improves the overall agreement on Sleep-EDF-78, whereas its effect on Sleep-EDF-20 is marginal. The proposed loss achieves the best overall trade-off on both datasets, reaching MF1 of $81.3\pm 1.8$ and $\kappa$ of $0.802\pm 0.03$ on Sleep-EDF-20, and MF1 of $78.2\pm 1.2$ with $\kappa$ of $0.771\pm 0.02$ on Sleep-EDF-78. These results indicate that the combination of hard-example emphasis (focal term) and confidence regularization (label smoothing), when coupled with class re-weighting, provides a more favorable macro-level balance than reweighting alone.

Table 8. Comparison of class-imbalance handling strategies on Sleep-EDF-20 and Sleep-EDF-78 (mean ± SD across 5-fold cross-validation).

Strategy	Sleep-EDF-20		Sleep-EDF-78
	MF1 (%)	κ	MF1 (%)	κ
Standard CE	79.7 ± 2.5	0.790 ± 0.03	76.3 ± 2.3	0.753 ± 0.02
Weighted CE (W-CE)	79.6 ± 2.6	0.790 ± 0.03	77.0 ± 1.4	0.763 ± 0.01
Proposed (Focal + LS)	81.3 ± 1.8	0.802 ± 0.03	78.2 ± 1.2	0.771 ± 0.02

Note: Standard CE denotes vanilla cross-entropy. Weighted CE applies inverse-frequency class weights $w_c$ estimated on the training split of each fold. The proposed method uses focal loss ($\gamma =1.5$) with label smoothing ($ϵ=0.1$) and class-weighted $\alpha$.

5.8. Parameter Sensitivity Analysis

To examine the role of architectural depth in temporal representation learning, a controlled sensitivity analysis was conducted by varying the number of Conformer Encoder layers, with $L\in \{1,2,4,6\}$. The corresponding Macro-F1 scores on Sleep-EDF-20 and Sleep-EDF-78 are reported in Table 9.

Table 9. Parameter sensitivity to Conformer depth L on Sleep-EDF-20 and Sleep-EDF-78 (5-fold). Macro-F1 scores are reported for different numbers of encoder layers.

Depth L	Sleep-EDF-20	Sleep-EDF-78
1	80.5	77.8
2	80.8	77.9
4	81.3	78.2
6	79.9	74.9

The degradation observed for deeper Conformer stacks is consistent with overfitting under limited subject diversity. In our training setup, weight decay, label smoothing, and early stopping already serve as primary regularization mechanisms. More aggressive regularization (e.g., increased dropout, stochastic depth, or stronger data augmentation) is expected to further stabilize deeper configurations, and will be explored in future work.

A clear trend emerges across both cohorts as model depth increases from a shallow to a moderately deep configuration. When the depth is expanded from $L=1$ to $L=4$, Macro-F1 improves from 80.5% to 81.3% on Sleep-EDF-20 and from 77.8% to 78.2% on Sleep-EDF-78. This progressive gain suggests that sleep-stage classification benefits from a minimum level of temporal modeling capacity, which is afforded by stacking multiple Conformer blocks and enables the encoding of long-range dependencies that are insufficiently captured by shallow architectures.

With further depth expansion, however, the performance trend reverses. At $L=6$, Macro-F1 decreases to 79.9% on Sleep-EDF-20 and to 74.9% on Sleep-EDF-78, indicating that the additional representational capacity no longer translates into improved generalization. Such degradation is indicative of over-parameterization under the current data regime and is consistent with increased optimization difficulty and overfitting tendencies in deeper temporal models.

In light of these observations, $L=4$ is adopted as the default depth throughout this study. This configuration provides sufficient expressive power for modeling long-range temporal structure while maintaining stable optimization behavior and avoiding unnecessary computational overhead.

6. Discussion

These results should not be interpreted as EMG being intrinsically uninformative; rather, its utility is cohort-dependent and hinges on signal reliability, and the proposed gating is designed to exploit stable muscle-tone cues while suppressing artifact-dominated segments. Under a symmetry-related interpretation, this selective attenuation helps preserve representational consistency across modalities by limiting asymmetric noise injection into the shared features. We acknowledge that the learned reliability gates reflect task-driven importance rather than an externally validated measure of signal quality, and future work may explore incorporating explicit signal-quality proxies.

The proposed fusion treats EEG as the primary stream and EOG/EMG as auxiliary cues, which introduces inherent modality asymmetry. While this choice is consistent with clinical practice, where EEG typically provides the most informative staging cues, it may become suboptimal when EEG quality degrades (e.g., due to electrode displacement or persistent artifacts). In such cases, a natural extension is to adopt symmetric reliability gating, i.e., estimating reliability for both EEG and auxiliary modalities and allowing fusion weights to adapt bidirectionally. We leave systematic evaluation under controlled EEG corruption and clinically challenging recordings to future work.

While the proposed model attains competitive performance under subject-wise evaluation, several limitations should be acknowledged. First, the framework is developed under the assumption of a complete polysomnography (PSG) configuration, incorporating EEG, EOG, and EMG signals. In practical home-monitoring settings, however, acquisition is often restricted to single-channel EEG, in which case the cross-modal fusion mechanism cannot be fully exploited. This motivates future investigations into modality-dropout or modality-agnostic training paradigms to enable stable inference under incomplete sensor availability. Second, the evaluation is conducted on Sleep-EDF cohorts that predominantly contain recordings from healthy subjects, which does not fully reflect clinically diverse sleep disorders. Further validation on pathological sleep datasets and disorder-specific cohorts is therefore required to assess robustness under clinically heterogeneous conditions.

In addition, the observed improvement in N1 performance is supported by transition-boundary-focused evaluation. As shown in Table 4, performance is notably lower near stage transition boundaries (${\mathcal{T}}_0$) and improves progressively as the neighborhood widens (i.e., ${\mathcal{T}}_1$, ${\mathcal{T}}_2$), suggesting that boundary-adjacent epochs are more challenging. These findings provide evidence that reliability-aware gating may contribute to the N1 gains, particularly around transitions where auxiliary signals are more likely to be artifact-prone. By attenuating unreliable auxiliary inputs near boundaries, the gating mechanism may stabilize shared representations and improve N1 discrimination.

7. Conclusions

In this work, the Gated Cross-Modal and Temporal Fusion Network (G-CMTF Net) is presented as an end-to-end solution for polysomnography-based sleep staging, where scale-dependent feature ambiguity and cross-channel artifact contamination remain persistent obstacles. A multi-scale spectro-temporal frontend is coupled with an adaptive gating mechanism so that physiologically meaningful sleep dynamics can be emphasized while modality-specific noise is attenuated. Along the temporal dimension, a Conformer Encoder is employed to represent long-range transition regularities, thereby supporting consistent modeling of sleep-cycle structure.

Experimental results on the Sleep-EDF-20 and Sleep-EDF-78 benchmarks indicate that G-CMTF Net performs competitively relative to state-of-the-art baselines. Of particular importance is the improved sensitivity obtained for N1 and REM, which are known to be prone to confusion under multimodal artifacts; this outcome suggests that reliability-aware fusion contributes to reducing modality-induced ambiguity. Evidence from experiments further shows that explicit spectral injection and multi-scale encoding are necessary to capture the heterogeneous oscillatory characteristics that define sleep stages.

Several practical limitations should also be acknowledged. The current framework assumes the availability of a full PSG montage (EEG, EOG, and EMG), which may not be feasible in home-monitoring deployments. Future studies will therefore explore lightweight variants tailored to single-channel devices, together with self-supervised pre-training strategies, in order to improve generalization across heterogeneous clinical populations.