Spectral Robustness Mixer: Cross-Scale Neck for Robust No-Reference Image Quality Assessment

Abstract

No-reference image quality assessment (NR-IQA) models achieve high correlation with human mean opinion scores (MOS) on clean benchmarks, yet recent work shows they can be highly vulnerable to small adversarial perturbations that severely degrade ranking consistency, including in black-box settings. We introduce the Spectral Robustness Mixer (SRM), a lightweight neck inserted between an NR-IQA backbone and regression head, designed to reduce adversarial sensitivity without changing the dataset, label format, or target metric. SRM couples (i) deep-to-shallow cross-scale fusion via a Nyström low-rank attention surrogate, (ii) ridge-conditioned landmark kernels with ridge regularization, solved via numerically stable small-matrix factorization (SVD/LU) to improve conditioning, and (iii) variance-aware entropy-regularized fusion gates with a bounded gain cap to limit gradient amplification. We evaluate SRM on TID2013 and KonIQ-10k under a white-box ${\mathcal{l}}_{\infty }/{\mathcal{l}}_2$ attack ensemble that includes per-image regression objectives and a correlation-aware pairwise inversion objective (a ranking-inspired surrogate for correlation inversion), with expectation-over-transformation (EOT) and anti-gradient masking checks. At $ϵ=4/255$ (${\mathcal{l}}_{\infty }$), SRM improves worst-case robust Spearman’s rank-order correlation coefficient (SROCC; defined as the minimum over our fixed attack ensemble) by an absolute $0.06$–$0.08$ SROCC points (i.e., correlation-coefficient units, not percentage gain) across datasets/backbones, while keeping clean SROCC within $0.00$–$0.01$ of the baseline. We observe similar trends for Pearson linear correlation coefficient (PLCC).

1. Introduction

No-reference image quality assessment (NR-IQA) predicts perceptual quality from a single distorted image and is widely used as a surrogate signal in vision pipelines (e.g., perceptual optimization, model selection, and automated quality monitoring). Modern NR-IQA models increasingly adopt hierarchical multi-scale backbones and attention-based feature fusion to capture both global structure and local distortion cues, often leveraging general-purpose vision backbones such as Pyramid Vision Transformers (PVT) [1] and Swin Transformers [2]. However, a growing body of evidence shows that NR-IQA models can be highly sensitive to small, visually imperceptible perturbations that dramatically change predicted quality scores and, more importantly, the induced rank ordering of images [3,4]. This robustness gap is particularly problematic when IQA scores are consumed downstream, because an attacker can bias decisions without producing obvious visual artifacts.

Where existing robustness efforts fall short.

Recent work has begun to systematize attacks against NR-IQA, including correlation-error-based attacks that directly target ranking consistency (e.g., SROCC) rather than only per-image score changes [3], as well as black-box attacks designed to induce large score deviations under perceptual constraints [4]. On the defense side, current approaches are dominated by training-time strategies such as gradient-norm regularization [5], which can improve robustness but are often coupled to particular training recipes and may not transfer cleanly across architectures and threat models. In contrast, the idea of robustness-by-design suggests building architectural components whose internal operators are less prone to numerical instability and high-gain amplification under small perturbations [6]. Despite early progress, we still lack general-purpose plug-and-play modules that can be inserted into diverse NR-IQA pipelines to improve robustness with minimal disruption to datasets, labels, and training code.

• Key hypothesis: cross-scale fusion can amplify instability.

Hierarchical backbones provide multi-scale representations where shallow features retain distortion-sensitive cues while deeper features encode semantic context. NR-IQA architectures frequently mix these signals through cross-scale aggregation. In such settings, small input perturbations can be amplified by three interacting effects:

• Conditioning of fusion operators: attention- or kernel-based mixing may become ill-conditioned, such that small changes in token statistics lead to disproportionately large changes in fused representations.
• Variance drift across stages: shallow and deep features often have different activation scales; naive fusion can induce scale dominance or collapse, making the regressor hypersensitive to perturbations.
• Unconstrained dynamic gating: adaptive gates can become high-gain functions of unstable features, further magnifying input gradients and creating failure modes consistent with gradient masking pitfalls unless evaluated carefully [7].

• These considerations motivate an operator-centric design goal: explicitly stabilize the conditioning, variance, and gain of cross-scale mixing.

• Our approach: Spectral Robustness Mixer (SRM).

We propose the Spectral Robustness Mixer (SRM), a lightweight neck module attached to an NR-IQA backbone before the final regression head. SRM targets robustness at the level of representation fusion rather than relying solely on training-time defenses. Concretely, SRM introduces: (i) a Nyström low-rank attention surrogate for deep-to-shallow mixing with linear memory in the token count [8]; (ii) ridge-conditioned landmark kernels with ridge regularization, solved via numerically stable small-matrix factorization (SVD/LU) to improve conditioning (a truncated Neumann-series inverse is evaluated only as an optional speed approximation in the ablation); and (iii) variance-aware entropy-regularized fusion gates with an explicit bounded gain cap to limit gradient amplification while preserving accuracy-relevant information. To evaluate robustness without false positives, we report worst-case robust correlation over an attack ensemble (AutoAttack-inspired in spirit) [9] that includes both per-image regression objectives and correlation-aware objectives using a correlation-aware pairwise inversion objective (ranking-inspired surrogate), and we apply expectation-over-transformation (EOT) and standard anti-gradient masking checks when stochasticity is present [7].

• Contributions.

Our contributions are as follows:

• Plug-and-play robustness neck for NR-IQA: we introduce SRM, a lightweight fusion module that can be inserted between an NR-IQA backbone and regressor to improve measured adversarial robustness with modest overhead (in the tested settings).
• Stability- and gain-controlled fusion design: SRM combines Nyström low-rank mixing, ridge-conditioned landmark kernels, variance-aware fusion, and a bounded gain cap, targeting conditioning, variance drift, and unstable gating as robustness bottlenecks.
• Correlation-aware robustness evaluation: we define and evaluate correlation-aware adversarial objectives aligned with rank correlation (SROCC) using a correlation-aware pairwise inversion objective (ranking-inspired surrogate), and we apply expectation-over-transformation (EOT) and standard anti-gradient masking checks when stochasticity is present [7].

• Paper organization.

Section 3 defines SRM and its components, including the stability assumptions and diagnostics used to motivate the design. Section 3.8.4 specifies the threat model, attack ensemble (including correlation-aware objectives), and evaluation guardrails (EOT and anti-gradient masking checks). Section 4 reports clean performance, worst-case robust correlation, transfer/black-box sanity checks, and mechanism diagnostics. We conclude with limitations and future directions.

2. Related Work

2.1. Backbones for No-Reference IQA

Classical no-reference image quality assessment (NR-IQA) relied on natural-scene statistics and hand-crafted features, later superseded by deep models that learn perceptual representations directly from data. Modern CNN-based NR-IQA models (e.g., HyperIQA) improved performance on authentic distortions by adapting prediction functions to image content but still face challenges in capturing long-range interactions relevant to global perceptual judgments [10]. Transformers introduced an alternative inductive bias via token mixing and global context modeling [11]. In NR-IQA specifically, Golestaneh et al. combine relative ranking with self-consistency to train transformer-based predictors under synthetic distortions [12]. MUSIQ introduces a multi-scale tokenization and embedding design to evaluate quality across resolutions and granularities [13], while MANIQA proposes multi-dimension attention blocks to enhance global–local interactions for NR-IQA [14]. More recently, self-supervised and transfer paradigms have also been explored for data efficiency and generalization (e.g., relative ranking with self-consistency) [12].

Relation to SRM. Rather than proposing yet another backbone, our goal is a plug-and-play neck that can sit between an existing NR-IQA backbone (CNN or hierarchical transformer) and a regression head, improving robustness while preserving clean correlation. This design choice makes SRM complementary to backbone innovations: SRM can be attached to models such as PVT/Swin-style hierarchies [1,2] or strong CNN-based NR-IQA baselines [10].

2.2. Cross-Scale Fusion and Neck Modules

Cross-scale feature fusion has a long history in dense prediction. Feature Pyramid Networks (FPN) and HRNet exemplify the benefit of exchanging information across resolutions for tasks that require both semantics and detail [15,16]. In transformers, hierarchical designs such as PVT and Swin explicitly build multi-stage pyramids [1,2], and cross-scale attention mechanisms (e.g., CrossFormer) were proposed for general vision recognition to improve interactions across scales [17]. In parallel, efficient attention approximations such as Nyströmformer reduce the quadratic cost of attention via landmark-based low-rank reconstruction [8].

Relation to SRM. We reuse the neck abstraction (lightweight module operating on intermediate feature maps), but tailor it to NR-IQA by (i) focusing on selective deep-to-shallow mixing suitable for perceptual cues, and (ii) incorporating explicit stability controls (conditioning and bounded-gain gating) that are evaluated under adversarial threat models (Section 3.8.4, Section 3.8.5 and Section 4). Our use of a Nyström-style surrogate targets compute-efficiency in cross-stage mixing, while keeping the backbone unchanged.

2.3. Adversarial Robustness of IQA Models

Compared to classification, adversarial robustness of NR-IQA/BIQA has received relatively limited attention, despite the vulnerability of perceptual estimators in downstream pipelines. Korhonen and You studied adversarial attacks against representative deep BIQA models and show that both white-box and substitute-model (transfer) settings can substantially disrupt predicted quality [18]. Recent work performed further robustness evaluation for NR-IQA and highlighted threats in both white-box and black-box regimes [4,19]. On the defense side, Liu et al. proposed training-time regularization to reduce input-gradient sensitivity in NR-IQA models [5]. These studies span diverse NR-IQA predictors, from CNN-based HyperIQA to transformer-based multi-scale designs (MUSIQ/MANIQA) and related learned perceptual distances [10,13,14,20]. Beyond individual attacks/defenses, recent work provides broader robustness benchmarks/insights, robustness-by-architecture prescriptions, and verification-style robustness analyses for IQA predictors [4,21,22,23,24].

Evaluation norms and pitfalls. Robustness claims can be invalidated by weak attacks or gradient masking. Accordingly, robustness evaluation commonly follows best practices such as using strong multi-step attacks and sanity checks for obfuscated gradients [7] and using expectation-over-transformation (EOT) when models include stochastic components [25]. AutoAttack popularized the idea of reporting worst-case robustness over an attack ensemble rather than a single tuned attack [9]. In this work, we adapt these norms to the NR-IQA regression/ranking setting (Section 3.8.4), including correlation-aware objectives aligned with SROCC/PLCC and robust reporting across an attack suite.

2.4. Perceptual Quality Assessment Beyond Still Images

While this paper focuses on no-reference image quality assessment, perceptual quality modeling is actively studied for other media types. For example, recent point-cloud quality assessment (PCQA) models combine multi-modal cues and alignment to predict quality of 3D data [26], and robust PCQA methods have been proposed to support practical 3D robotic scanning pipelines [27]. In parallel, there is growing interest in video quality understanding, including benchmarks that evaluate large multimodal models (LMMs) on video quality perception tasks [28].

Relation to SRM. These developments suggest that robustness and stability issues in perceptual estimation are not unique to still images. SRM is formulated as a general-purpose cross-scale fusion and stabilization neck, and the same design principles (conditioning control, scale stabilization, and bounded-gain fusion) could potentially be adapted to 3D/temporal quality pipelines, where multi-scale representations and fusion are also common (Section 5.3).

2.5. Robustness-by-Design and Stable Feature Mixing

Beyond adversarial training, several lines of work improve stability through architectural choices. Normalization-free networks reduce reliance on fragile batch-dependent statistics [29], while token/channel mixing architectures (e.g., MLP-Mixer, ConvNeXt) demonstrate that carefully designed mixing operations can improve optimization and generalization in vision models [30,31]. Efficient attention approximations provide additional levers for controlling numerical behavior and compute [8].

Relation to SRM. SRM follows a robustness-by-design philosophy: it introduces a controlled cross-stage mixing operator with explicit conditioning and bounded-gain gating, and evaluates these mechanisms under strong, anti-false-robustness protocols rather than treating them as implicit guarantees.

Summary. Prior NR-IQA research has advanced backbone architectures (CNNs, transformers, multi-scale tokenization) [10,13,14], while robustness-focused work spans attacks, defenses, and emerging benchmarks/verification/architecture-guided robustness directions [4,5,18,22,23,24]. Our contribution is orthogonal: a lightweight SRM neck that can be attached to existing NR-IQA models to improve robustness under a modern, skeptical evaluation protocol (Section 3.8.4, Section 3.8.5 and Section 4).

3. Methodology

3.1. Problem Setting and Notation

Task. We treat no-reference IQA as scalar regression $f_{\theta }:{[0,1]}^{H\times W\times 3}\to \mathbb{R}$, mapping an RGB image $\mathbf{x}$ to a predicted quality score $\widehat{y}=f_{\theta }\left(\mathbf{x}\right)$. Let ${\mathcal{D}}_{\mathrm{eval}}={\left\{({\mathbf{x}}_i,y_i)\right\}}_{i=1}^M$ denote an evaluation set with human mean-opinion scores (MOS) $y_i$. NR-IQA performance is typically reported by set-level correlation between predictions $\widehat{\mathbf{y}}=({\widehat{y}}_1,\dots ,{\widehat{y}}_M)$ and MOS $\mathbf{y}=(y_1,\dots ,y_M)$, using Spearman rank correlation (SROCC) and Pearson linear correlation (PLCC). (We specify the exact PLCC computation, including monotonic logistic mapping, in Section 3.8.2.)

Threat model. Following recent NR-IQA robustness studies [3,4,5], we evaluate a white-box adversary that crafts a bounded perturbation for each image:

$${\mathbf{x}}_i^{\mathrm{adv}}={\mathcal{A}}_{\epsilon ,p}({\mathbf{x}}_i;f_{\theta })={\mathsf{\Pi }}_{[0,1]}\left({\mathbf{x}}_i+{\delta }_i\right),{\parallel {\delta }_i\parallel }_p\le \epsilon ,$$

(1)

where ${\mathsf{\Pi }}_{[0,1]}$ clips pixels to the valid range and $p\in \{\infty ,2\}$ in our experiments. Unless stated otherwise, perturbations are applied in pixel space on inputs scaled to $[0,1]$ (before any optional backbone-specific normalization).

Attack objective families. Because NR-IQA is evaluated by correlation over sets, adversarial objectives can be defined either per image or at the batch level. We use two complementary families:

• Per-image regression objectives (image-wise). Given MOS $y_i$, an attacker may maximize a regression loss such as ${\mathcal{l}}_{\mathrm{reg}}({\mathbf{x}}_i^{\mathrm{adv}},y_i;\theta )={\left(f_{\theta }\left({\mathbf{x}}_i^{\mathrm{adv}}\right)-y_i\right)}^2$ (MSE) or maximize a score-drift loss

  $${\mathcal{l}}_{\mathrm{drift}}({\mathbf{x}}_i^{\mathrm{adv}},{\mathbf{x}}_i;\theta )=\left|f_{\theta }\left({\mathbf{x}}_i^{\mathrm{adv}}\right)-f_{\theta }\left({\mathbf{x}}_i\right)\right|.$$

  (2)
• Correlation-aware objectives (batch-level via pairwise inversions). To directly degrade ranking consistency (and thus SROCC), we use a lightweight pairwise inversion objective that increases the number of mis-ordered MOS pairs within each mini-batch, consistent with correlation-focused NR-IQA attack analyses [3]. We instantiate this objective as the smooth inversion loss in Equation (32) (Section 3.8.4), using comparable pairs $|y_i-y_j| > \mathsf{\Delta }$ with $\mathsf{\Delta }=0.05$ (MOS normalized to $[0,1]$) and temperature $\gamma =0.1$.

Robust correlation metrics and reporting. Given an attack ${\mathcal{A}}_{\epsilon ,p}$, we compute robust correlations on attacked inputs: $\mathrm{SROCC}({\widehat{\mathbf{y}}}^{\mathrm{adv}},\mathbf{y})$ and $\mathrm{PLCC}({\widehat{\mathbf{y}}}^{\mathrm{adv}},\mathbf{y})$, where ${\widehat{y}}_i^{\mathrm{adv}}=f_{\theta }\left({\mathbf{x}}_i^{\mathrm{adv}}\right)$. We report both clean and adversarial correlations, their drop relative to clean, and (when multiple attacks are considered) the worst-case robust correlation as the minimum across an explicit attack ensemble (Section 3.8.4, Section 3.8.5 and Section 4).

Notation. M evaluation set size; B batch size; $H\times W$ spatial resolution; N flattened token count; d head dimension; r Nyström landmark count; ${\kappa }_{\mathrm{raw}}$ condition number proxy of the unwhitened landmark kernel; $\tau$ post-attention temperature/rescaler; ${\sigma }_{\mathrm{post}}$ standard deviation of the post-attention output; $\beta$ gain cap used in fusion gating; $\gamma$ temperature/sharpness parameter in the pairwise inversion loss (32).

3.2. Backbone Integration and Cross-Stage Attention

Backbone interface (four-level pyramids). We assume an NR-IQA backbone that outputs a four-level feature pyramid ${\left\{{\mathbf{F}}_{\mathcal{l}}\right\}}_{\mathcal{l}=0}^3$, with ${\mathbf{F}}_{\mathcal{l}}\in {\mathbb{R}}^{B\times C_{\mathcal{l}}\times H_{\mathcal{l}}\times W_{\mathcal{l}}}$ and $\mathcal{l}=0$ denoting the highest spatial resolution. This interface is provided by hierarchical backbones commonly used in modern vision systems, including Pyramid Vision Transformers (PVT) [1], Swin Transformers [2], and lightweight mobile networks such as MobileNetV3 [32]. SRM consumes the pyramid without modifying the backbone and produces a fused representation that is passed to the IQA head.

Channel alignment and tokenization. Each level is projected to a common embedding width C using a $1\times 1$ convolution:

$${\mathbf{E}}_{\mathcal{l}}={\varphi }_{\mathcal{l}}\left({\mathbf{F}}_{\mathcal{l}}\right),{\varphi }_{\mathcal{l}}:{\mathbb{R}}^{B\times C_{\mathcal{l}}\times H_{\mathcal{l}}\times W_{\mathcal{l}}}\to {\mathbb{R}}^{B\times C\times H_{\mathcal{l}}\times W_{\mathcal{l}}}.$$

(3)

We then flatten spatial dimensions into tokens:

$$N_{\mathcal{l}}=H_{\mathcal{l}}{W}_{\mathcal{l}},{\mathbf{X}}_{\mathcal{l}}=\mathrm{flatten}\left({\mathbf{E}}_{\mathcal{l}}\right)\in {\mathbb{R}}^{B\times N_{\mathcal{l}}\times C}.$$

For multi-head attention with H heads, we reshape ${\mathbf{X}}_{\mathcal{l}}$ into ${\mathbf{T}}_{\mathcal{l}}\in {\mathbb{R}}^{B\times H\times N_{\mathcal{l}}\times d}$ with head width $d=C/H$. For numerical stability, we optionally apply per-token ${\mathcal{l}}_2$ normalization to queries and keys, $\mathbf{t}\leftarrow \mathbf{t}/\max (\parallel \mathbf{t}{\parallel }_2,{ϵ}_{\mathrm{norm}})$, which is smooth almost everywhere and does not introduce stochasticity. (We avoid hard clipping by default; if clipping is enabled in an implementation, we evaluate with the adaptive checks and EOT described in Section 3.8.4.)

Cross-stage links (deep → shallow mixing). SRM injects coarse semantic context from deeper stages into higher-resolution stages using a set of cross-stage links

$$\mathcal{S}\subseteq \{({\mathcal{l}}_{\mathrm{sh}},{\mathcal{l}}_{\mathrm{dp}})\mid 0\le {\mathcal{l}}_{\mathrm{sh}}<{\mathcal{l}}_{\mathrm{dp}}\le 3\}.$$

(4)

Each link $({\mathcal{l}}_{\mathrm{sh}},{\mathcal{l}}_{\mathrm{dp}})\in \mathcal{S}$ defines a cross-attention block where queries come from shallow tokens and keys/values come from deep tokens:

$$\mathbf{Q}={\mathbf{T}}_{{\mathcal{l}}_{\mathrm{sh}}},\mathbf{K}={\mathbf{T}}_{{\mathcal{l}}_{\mathrm{dp}}},\mathbf{V}={\mathbf{T}}_{{\mathcal{l}}_{\mathrm{dp}}}.$$

This design does not imply certified robustness; it is a controlled multi-scale mixing mechanism whose robustness impact is evaluated empirically under the threat models in Section 3.8.4 and reported in Section 4.

Default schedule and cost control. Different schedules trade representation richness for compute. A common and efficient choice uses the deepest stage as a shared context source for all shallower stages:

$${\mathcal{S}}_{\mathrm{deep}}=\{(0,3),(1,3),(2,3)\}.$$

(5)

More aggressive schedules (e.g., adding $(0,2)$) increase computation, while sparser schedules reduce overhead. With Nyström rank r, each cross-stage block scales as $\mathcal{O}\left(BH{N}_{{\mathcal{l}}_{\mathrm{sh}}}r\right)$ in time and memory (Section 3.3), enabling linear-memory mixing even when $N_{\mathcal{l}}$ is large.

Fold-back and residual injection. For each link $({\mathcal{l}}_{\mathrm{sh}},{\mathcal{l}}_{\mathrm{dp}})$ we compute an update ${\mathbf{Y}}_{{\mathcal{l}}_{\mathrm{sh}}\leftarrow {\mathcal{l}}_{\mathrm{dp}}}\in {\mathbb{R}}^{B\times H\times N_{{\mathcal{l}}_{\mathrm{sh}}}\times d}$ using the linear-memory attention module (Section 3.3). If multiple links target the same shallow stage, we aggregate their updates by summation:

$$\mathsf{\Delta }{\mathbf{T}}_{{\mathcal{l}}_{\mathrm{sh}}}=\sum _{\begin{array}{c}({\mathcal{l}}_{\mathrm{sh}},{\mathcal{l}}_{\mathrm{dp}})\in \mathcal{S}\end{array}}{\mathbf{Y}}_{{\mathcal{l}}_{\mathrm{sh}}\leftarrow {\mathcal{l}}_{\mathrm{dp}}}.$$

We then apply a residual update and fold tokens back to the spatial map:

$$\begin{array}{c}\hfill {\mathbf{T}}_{{\mathcal{l}}_{\mathrm{sh}}}^{\prime }={\mathbf{T}}_{{\mathcal{l}}_{\mathrm{sh}}}+{\alpha }_{{\mathcal{l}}_{\mathrm{sh}}}\mathsf{\Delta }{\mathbf{T}}_{{\mathcal{l}}_{\mathrm{sh}}},\\ \hfill {\mathbf{E}}_{{\mathcal{l}}_{\mathrm{sh}}}^{\prime }=\mathrm{fold}\left({\mathbf{T}}_{{\mathcal{l}}_{\mathrm{sh}}}^{\prime }\right)\in {\mathbb{R}}^{B\times C\times H_{{\mathcal{l}}_{\mathrm{sh}}}\times W_{{\mathcal{l}}_{\mathrm{sh}}},}\end{array}$$

(6)

where ${\alpha }_{{\mathcal{l}}_{\mathrm{sh}}}$ is a learnable residual weight. Stages not receiving updates remain unchanged (e.g., ${\mathbf{E}}_3^{\prime }={\mathbf{E}}_3$ for ${\mathcal{S}}_{\mathrm{deep}}$). The updated pyramid $\left\{{\mathbf{E}}_{\mathcal{l}}^{\prime }\right\}$ is subsequently pooled and fused by the gate module (Section 3.6).

Figure 1 provides a conceptual overview of SRM and its integration into a standard NR-IQA pipeline.

Figure 2 shows a detailed block-level diagram of the SRM neck and its internal information flow (queries, keys/values, and stabilizers).

SRM forward pass (overview). Algorithm 1 summarizes the SRM forward pass at the level of module interfaces. The internal details of the Nyström approximation Algorithm 2, landmark conditioning, and post-attention scaling are specified in the following subsections (Section 3.3, Section 3.4, Section 3.5 and Section 3.6).

Algorithm 1: SRM—Spectral Robustness Mixer (single forward pass, overview)

Algorithm 2: NystromCrossAttn (interface)

Input: $\mathbf{Q}\in {\mathbb{R}}^{B\times H\times N_q\times d}$, $\mathbf{K},\mathbf{V}\in {\mathbb{R}}^{B\times H\times N_k\times d}$, rank r      Output: $\mathbf{Y}\in {\mathbb{R}}^{B\times H\times N_q\times d}$     // $\phantom{(}$Landmarks. Select r key/value landmarks ${\mathbf{K}}_r,{\mathbf{V}}_r$ (deterministic by default; see Section 3.4).$\phantom{(}$  1   Form ${\mathbf{K}}_r,{\mathbf{V}}_r$ by selecting indices $\mathcal{I}$ of size r from $\mathbf{K},\mathbf{V}$     //  $\phantom{(}$Nyström approximation. Compute low-rank cross-attention output using landmark kernels (Section 3.3).$\phantom{(}$  2    Compute $\mathbf{Y}\leftarrow \mathrm{NystromApprox}(\mathbf{Q},\mathbf{K},\mathbf{V},{\mathbf{K}}_r,{\mathbf{V}}_r)$    //  $\phantom{(}$Post-scaling. Apply post-attention stabilization (temperature/rescaling) when enabled (Section 3.5).$\phantom{(}$  3    Optionally rescale $\mathbf{Y}$ by the learned post-attention temperature $\tau$  4    return $\mathbf{Y}$

3.3. Nyström Cross-Attention with Ridge-Leverage Landmarks

SRM requires a cross-attention mechanism that (i) supports rectangular attention ($N_q\gg N_k$ for shallow queries versus deep keys), (ii) avoids quadratic $N_q{N}_k$ memory, and (iii) remains numerically stable under mixed precision. We adopt a Nyström-style approximation of softmax attention following efficient attention work [8] and specialize it to cross-stage fusion.

• Cross-attention setup (rectangular case).

For a fixed batch element and attention head, let $Q\in {\mathbb{R}}^{N_q\times d}$ be shallow-stage queries and $K,V\in {\mathbb{R}}^{N_k\times d}$ be deep-stage keys/values. Full cross-attention is

$$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d}}}\right)V,$$

(7)

where $\mathrm{softmax}$ is applied row-wise.

• Nyström approximation (cross-attention).

Select a small set of $r\ll N_k$ landmark keys $K_r\in {\mathbb{R}}^{r\times d}$ (and corresponding values $V_r$) by choosing r rows of K (landmark selection is described below). Define the Nyström blocks

$$\begin{array}{cc}\hfill C& =\mathrm{softmax}\left({\displaystyle \frac{Q{K}_r^{\top }}{\sqrt{d}}}\right)\in {\mathbb{R}}^{N_q\times r},\hfill \\ \hfill W& =\mathrm{softmax}\left({\displaystyle \frac{K_r{K}_r^{\top }}{\sqrt{d}}}\right)\in {\mathbb{R}}^{r\times r},\hfill \\ \hfill R& =\mathrm{softmax}\left({\displaystyle \frac{K_r{K}^{\top }}{\sqrt{d}}}\right)\in {\mathbb{R}}^{r\times N_k}.\hfill \end{array}$$

(8)

We approximate the full attention map by

$$\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d}}}\right)\approx C{W}_{\lambda }^{†}R,W_{\lambda }=W+\lambda I_r,$$

(9)

and compute the Nyström cross-attention output as

$$\widehat{Y}=\left(C{W}_{\lambda }^{†}R\right)V\in {\mathbb{R}}^{N_q\times d}.$$

(10)

Here, $W_{\lambda }^{†}$ denotes a numerically stable inverse (or pseudo-inverse) of the regularized $r\times r$ matrix $W_{\lambda }$. Regularization is important because W can become poorly conditioned when landmarks are redundant (Section 3.4 reports conditioning diagnostics).

Remark 1

(rectangular vs. symmetric Nyström). In self-attention, Q = K and one recovers a symmetric Nyström form (up to row-wise softmax normalization). In rectangular cross-attention ($N_q\ne N_k$), the right block R is required, and the approximation is naturally asymmetric.

• Landmark selection via ridge leverage (deterministic by default).

Landmark quality affects both approximation error and the conditioning of $W_{\lambda }$. We compute ridge leverage scores as a data-adaptive importance measure [33]. For a fixed head, let $k_i\in {\mathbb{R}}^d$ be the i-th row of K and define

$${\mathcal{l}}_i=k_i^{\top }{(K^{\top }K+{\lambda }_{\mathrm{rls}}{I}_d)}^{-1}{k}_i,p_i={\displaystyle \frac{{\mathcal{l}}_i}{{\sum }_{j=1}^{N_k}{\mathcal{l}}_j}},$$

(11)

with ridge parameter ${\lambda }_{\mathrm{rls}}>0$. Default (deterministic): we select the top-r indices by $p_i$ to form $K_r$ and $V_r$. Optional (stochastic, training-only): sample r indices without replacement according to $p_i$. We compute ${(K^{\top }K+{\lambda }_{\mathrm{rls}}{I}_d)}^{-1}$ stably using an fp32 Cholesky solve (SPD in $d\times d$) and evaluate ${\mathcal{l}}_i$ in fp32.

• k-means++ refinement (ablation-only).

Ridge-leverage selection can still produce near-duplicate landmarks when keys contain repeated patterns. We therefore include an optional refinement that applies k-means++ initialization followed by a small number of Lloyd steps on the landmark candidate set [34]. Default: disabled. All main results use the deterministic top-r ridge-leverage selection. We enable k-means++ refinement only in dedicated ablations to quantify its impact on landmark diversity, conditioning, and robustness (Section 4).

• Stable computation of $W_{\lambda }^{†}$ (mixed precision).

Because W in (8) is produced by row-wise softmax, it need not be symmetric. Therefore, we avoid SPD-specific solvers for $W_{\lambda }$ and compute $W_{\lambda }^{†}$ in fp32 using a robust small-matrix routine (default: SVD-based pseudo-inverse; fallback: LU solve when well-conditioned), then cast the resulting product back to the model dtype. We use a scale-stable ridge

$$\lambda ={\lambda }_0·{\displaystyle \frac{\mathrm{tr}\left(W\right)}{r}},$$

(12)

with ${\lambda }_0$ fixed (reported in Section 3.8), to prevent head-dependent scaling issues.

• Optional stochastic key/value pooling (training-time only).

At very high spatial resolutions, storing deep tokens $(K,V)$ can dominate memory. To reduce training memory, we optionally apply $2\times 2$ mean pooling to the deep feature map before tokenization with probability $p_{\mathrm{pool}}$. Default evaluation is deterministic: pooling is disabled at test time. If stochastic pooling is enabled at evaluation for any reason, robustness must be evaluated with EOT (Section 3.8.4) and reported explicitly.

3.4. Spectral Conditioning

Nyström cross-attention is only useful in SRM if the small $r\times r$ landmark system can be solved numerically stably under mixed precision and across diverse batches. The critical operation is the regularized inverse (or pseudo-inverse) applied to the landmark block $W\in {\mathbb{R}}^{r\times r}$ from Section 3.3, via $W_{\lambda }^{†}(·)$ with $W_{\lambda }=W+\lambda I_r$. This subsection defines (i) what we mean by stability, (ii) how we enforce it in implementation, and (iii) which diagnostics we report to rule out spurious “robustness” caused by numerical artifacts.

• What we measure: conditioning and solve residuals.

Because W arises from row-wise softmax (and is generally not symmetric), we measure conditioning using singular values. We track:

• Regularized conditioning (singular-value ratio):

  $${\kappa }_{\mathrm{reg}}\left(W_{\lambda }\right)={\displaystyle \frac{{\sigma }_{\max }\left(W_{\lambda }\right)}{{\sigma }_{\min }\left(W_{\lambda }\right)}}.$$

  (13)

  This controls sensitivity of linear solves to perturbations in W and to finite-precision rounding.
• Solve residual (numerical accuracy): for each solve $U=W_{\lambda }^{†}B$ we report the relative residual

  $$\mathrm{res}\left(U\right)={\displaystyle \frac{{\parallel W_{\lambda }U-B\parallel }_F}{{\parallel B\parallel }_F+{ϵ}_0}},$$

  (14)

  with ${ϵ}_0>0$ for numerical safety.

• Diagnostics are computed in fp32, even when the forward pass uses mixed precision.

• Why “row-QR makes $W\approx I$” is not valid.

A QR factorization applied to landmark keys can produce an orthonormal surrogate in feature space, but it does not imply that the attention landmark block W (a row-softmax of inner products; Section 3.3) becomes the identity. Therefore, instead of claiming $W\approx I_r$, we explicitly regularize and solve $W_{\lambda }$ and report ${\kappa }_{\mathrm{reg}}$ and $\mathrm{res}(·)$ as falsifiable stability evidence.

• Stabilization and solve pipeline (implemented).

Given W from Equation (8), SRM applies:

(1) fp32 “critical path”. We compute the softmax blocks in the model dtype but cast W to fp32 for all subsequent conditioning diagnostics and solves. This is feasible because r is small (e.g., $r\in [32,128]$).

(2) Diagonal equilibration (optional). To reduce scale imbalance, we optionally apply left/right diagonal scaling (a standard preconditioning heuristic for small dense solves):

$$\begin{array}{c}\hfill D_r=\mathrm{diag}(\sqrt{\sum _j{W}_{ij}^2+{ϵ}_0}),\\ \hfill D_c=\mathrm{diag}(\sqrt{\sum _i{W}_{ij}^2+{ϵ}_0}),\\ \hfill \tilde{W}=D_r^{-1}W{D}_c^{-1}.\end{array}$$

(15)

We then solve with ${\tilde{W}}_{\lambda }=\tilde{W}+\lambda I_r$ and map back implicitly through the scaled system. Equilibration is treated as an ablation component; we report its effect on ${\kappa }_{\mathrm{reg}}$, residuals, and robustness.

(3) Ridge regularization. We always solve a regularized system

$$W_{\lambda }=W+\lambda I_r,\lambda ={\lambda }_0·{\displaystyle \frac{\mathrm{tr}\left(W\right)}{r}},$$

(16)

using a trace-scaled ridge (consistent across heads) with fixed ${\lambda }_0$ reported in Section 3.8.

(4) Stable solve (default). We do not explicitly form $W_{\lambda }^{-1}$. Instead, we compute $U=W_{\lambda }^{†}B$ in fp32. Default: SVD-based pseudo-inverse for numerical robustness. Fast-path: if ${\kappa }_{\mathrm{reg}}\left(W_{\lambda }\right)<{\kappa }_{\max }$ (we use ${\kappa }_{\max }={10}^4$), we use an fp32 LU solve. We always log $\mathrm{res}\left(U\right)$ via Equation (14).

• Optional fast inverse (Neumann truncation; ablation-only).

As an acceleration, one may approximate ${(I-E)}^{-1}$ by a truncated Neumann series when ${\parallel E\parallel }_2<1$:

$${\overline{W}}_{\lambda }={\displaystyle \frac{1}{\mu }}{\tilde{W}}_{\lambda },E=I-{\overline{W}}_{\lambda },{\overline{W}}_{\lambda }^{-1}\approx \sum _{k=0}^m{E}^k,$$

(17)

where $\mu =\mathrm{mean}\left(\mathrm{diag}\left({\tilde{W}}_{\lambda }\right)\right)$ is a scalar rescaling after optional equilibration. We enable Neumann-m only if an fp32 power-iteration estimate satisfies ${\parallel E\parallel }_2\le 0.9$ and the observed residual $\mathrm{res}\left(U\right)$ is below a fixed threshold (we use ${10}^{-3}$); otherwise we fall back to the default solve. We report the measured ${\parallel E\parallel }_2$ and residuals in Section 4 (conditioning ablations).

• Scale calibration (no “variance unity” claim).

Equilibration and ridge can change the scale of attention outputs. Rather than claiming a closed-form variance guarantee, SRM controls activation scale empirically using: (i) a learned post-attention rescaler $\tau$ (Section 3.5), and (ii) logging the post-attention standard deviation ${\sigma }_{\mathrm{post}}$ during training and evaluation. This keeps the stability story falsifiable and avoids relying on idealized token assumptions.

• Cost.

All operations above are on an $r\times r$ matrix: equilibration is $\mathcal{O}\left(r^2\right)$ and a dense solve is $\mathcal{O}\left(r^3\right)$ in fp32. For typical $r\le 128$ this is a small constant overhead; we report wall-clock timing and the fraction of SRM time attributable to the solve in Section 3.7.

We summarize the regularized condition number ${\kappa }_{\mathrm{reg}}\left(W_{\lambda }\right)$ (Equation (13)) and solve the residual $\mathrm{res}\left(U\right)$ for the landmark system (

Table 1. Spectral conditioning diagnostics (fp32, across evaluation batches).

Setting	Median ${\mathit{\kappa }}_{\mathbf{reg}}$	p90 ${\mathit{\kappa }}_{\mathbf{reg}}$	Median $\mathbf{res}$	p90 $\mathbf{res}$	Solve Share  (% Time)
No equilibration + ridge (default)	$2.6\times {10}^2$	$1.9\times {10}^3$	$7.4\times {10}^{-6}$	$5.1\times {10}^{-5}$	$6.3\%$
Equilibration + ridge	$1.4\times {10}^2$	$8.7\times {10}^2$	$5.9\times {10}^{-6}$	$3.8\times {10}^{-5}$	$7.1\%$
Neumann-m = 3 (ablation; gated)	$1.4\times {10}^2$	$8.7\times {10}^2$	$8.2\times {10}^{-4}$	$9.6\times {10}^{-4}$	$3.2\%$

). Values are computed on the same evaluation split/checkpoint used for the main robustness tables, and we summarize the distribution across evaluation batches via the median and 90th percentile.

• Key takeaways.

• SRM treats landmark stability as a numerical linear algebra problem: ridge regularize, optionally equilibrate, then solve in fp32.
• We replace informal “$W\approx I$” statements with measurable diagnostics: ${\kappa }_{\mathrm{reg}}\left(W_{\lambda }\right)$ and $\mathrm{res}\left(U\right)$.
• Neumann truncation is an ablation-only acceleration, enabled only when a convergence proxy and residual thresholds are satisfied.

3.5. Variance Stabilization with a Learnable $\mathit{\tau }$

Even when the landmark solve is numerically stable (Section 3.4), the scale of cross-attention outputs can drift across stages, heads, and training time. Under mixed precision, scale drift can lead to saturated logits, overly sharp/flat attention weights, and unstable gradients. SRM therefore includes a lightweight, explicit mechanism to track and stabilize post-attention scale: a per-link, per-head learnable rescaler $\tau$ trained with an auxiliary objective.

• Post-attention scale statistic (vector RMS).

Let $\widehat{Y}\in {\mathbb{R}}^{B\times H\times N_q\times d}$ denote the (pre-fusion) output of one Nyström cross-attention link (Section 3.3). For each head h and link $s\in \mathcal{S}$, define the per-head mean vector

$${\mathit{\mu }}^{(s,h)}={\displaystyle \frac{1}{B{N}_q}}\sum _{b=1}^B\sum _{n=1}^{N_q}{\widehat{Y}}_{b,n,:}^{(s,h)}\in {\mathbb{R}}^d,$$

and the vector RMS magnitude

$${\rho }_{\mathrm{post}}^{(s,h)}={\left({\displaystyle \frac{1}{B{N}_q}}\sum _{b=1}^B\sum _{n=1}^{N_q}{∥{\widehat{Y}}_{b,n,:}^{(s,h)}-{\mathit{\mu }}^{(s,h)}∥}_2^2\right)}^{1/2}.$$

(18)

We compute ${\rho }_{\mathrm{post}}^{(s,h)}$ in fp32 (even if the forward uses fp16/bf16) to reduce numerical noise.

• Reference scale and EMA tracking.

If the d output dimensions are approximately isotropic with per-dimension variance near 1, then the expected squared norm is ${\mathbb{E}\parallel z-\mathbb{E}z\parallel }_2^2\approx d$ and the RMS magnitude is $\approx \sqrt{d}$. We use $\sqrt{d}$ as a reference (not a universal optimum) to prevent collapse/explosion and to keep heads comparable. To reduce batch noise, we maintain an exponential moving average (EMA) for each $(s,h)$:

$${\overline{\rho }}^{(s,h)}\leftarrow m{\overline{\rho }}^{(s,h)}+(1-m){\rho }_{\mathrm{post}}^{(s,h)},m=0.99.$$

(19)

The EMA state is updated during training only.

• Learnable rescaler $\tau$ and auxiliary loss.

SRM applies a multiplicative rescale to each link and head:

$$Y^{(s,h)}={\tau }^{(s,h)}{\widehat{Y}}^{(s,h)},\tau \in {\mathbb{R}}_{+}^{\left|\mathcal{S}\right|\times H}.$$

We parameterize $\tau$ via unconstrained parameters $u^{(s,h)}\in \mathbb{R}$ as

$${\tau }^{(s,h)}=\exp \left(u^{(s,h)}\right),$$

(20)

initialized at $u^{(s,h)}=0$ (so ${\tau }^{(s,h)}=1$). We regularize $\tau$ using an auxiliary objective that penalizes deviation of the rescaled EMA magnitude from the reference scale:

$${\mathcal{L}}_{\tau }={\lambda }_{\tau }\sum _{(s,h)}\left|\log \left({\displaystyle \frac{{\tau }^{(s,h)}{\overline{\rho }}_{\mathrm{post}}^{(s,h)}}{\sqrt{d}+{ϵ}_0}}\right)\right|,{\lambda }_{\tau }=0.01,$$

(21)

where ${ϵ}_0={10}^{-6}$ ensures numerical safety. We add ${\mathcal{L}}_{\tau }$ to the main training objective and optimize it jointly with all model parameters. We do not claim a formal convergence or robustness guarantee for $\tau$; instead, we report scale diagnostics and robustness metrics empirically (

Table 2. Post-attention scale diagnostics for $\tau$ (fp32 statistics, evaluation batches). We report normalized RMS magnitude $\tilde{\rho }={\rho }_{\mathrm{post}}/\sqrt{d}$ before and after applying $\tau$, and the distribution of learned $\tau$ values across links/heads. Numbers are aggregated across all cross-stage links and heads.

Model	Median $\tilde{\mathit{\rho }}$ (Pre)	p90 $\tilde{\mathit{\rho }}$ Pre	Median $\tilde{\mathit{\rho }}$ (Post)	p90 $\tilde{\mathit{\rho }}$ Post	$\mathbf{Median}\mathit{\tau }$	$\mathit{p}90\mathit{\tau }$
SRM (DyT + ${\mathcal{L}}_{\tau }$)	$0.74$	$1.88$	$1.01$	$1.13$	$1.34$	$2.85$

, Figure 3).

• Safety rails (reproducible).

To prevent extreme rescaling, we clamp $u^{(s,h)}$ during training:

$$\begin{array}{c}\hfill u^{(s,h)}\leftarrow \mathrm{clip}\left(u^{(s,h)},\log {\tau }_{\min },\log {\tau }_{\max }\right),\\ \hfill {\tau }_{\min }=0.25,{\tau }_{\max }=4.0.\end{array}$$

(22)

At evaluation time, $\tau$ is fixed (no EMA updates, no clamp updates).

• DyT pre-normalization (bounded activations).

Before computing Nyström attention blocks, we optionally apply Dynamic Tanh (DyT) [35] to token representations entering the attention block:

$$\mathrm{DyT}\left(x\right)=\tanh \left(\alpha x\right),$$

(23)

with learnable $\alpha$. DyT is a bounded, smooth nonlinearity that constrains activation range without relying on batch statistics. We use DyT as a numerical stabilizer (range control) rather than a robustness guarantee. Robustness is evaluated with strong white-box attacks and (when stochasticity is present) EOT following standard guidance [7,25] (Section 3.8.4).

• Relation to normalization layers.

RMSNorm rescales activations using their root mean square magnitude and provides re-scaling invariance [36]. DyT instead bounds activations through a learnable saturation curve, which can improve numerical range control in fp16/bf16. In SRM, DyT constrains the input range to the attention block, while $\tau$ provides an explicit post-attention scale corrector.

Takeaway. DyT bounds the input range of attention, and $\tau$ provides an explicit learnable post-attention scale correction. Together they stabilize mixed-precision attention computations; any robustness gains are established empirically under strong adaptive evaluation rather than claimed as a guarantee.

3.6. Entropy-Regularized Fusion Gates

SRM outputs a set of cross-stage-updated feature maps at shallow resolutions, which we pool to obtain three per-stage tensors for fusion. The role of the fusion gate is to aggregate these tensors into a single representation for the IQA head while (i) discouraging stage collapse (one scale dominating persistently), and (ii) limiting activation-scale spikes that can destabilize mixed-precision training and amplify gradients. Scope is explicit: the gate is an architectural stabilizer; it does not certify end-to-end robustness, which we evaluate empirically under strong adaptive attacks (Section 3.8.4).

• Inputs to the gate.

Let ${\left\{{\mathbf{E}}_{\mathcal{l}}^{\prime }\right\}}_{\mathcal{l}=0}^3$ be the post-attention, post-$\tau$ feature pyramid produced by SRM (Section 3.2, Section 3.3, Section 3.4 and Section 3.5). We pool the three shallow stages $\mathcal{l}\in \{0,1,2\}$ to a common spatial grid $h\times w$:

$${\mathbf{Y}}_s=\mathrm{Pool}\left({\mathbf{E}}_s^{\prime }\right)\in {\mathbb{R}}^{B\times C\times h\times w},s\in \{0,1,2\},$$

(24)

where $\mathrm{Pool}$ is fixed (e.g., adaptive average pooling to $h\times w$) and C is the aligned channel width. We keep the deepest stage as a context source for cross-stage mixing but do not fuse it directly by default.

• Gate logits.

For each stage s we compute a scalar logit $g_{b,s}$ per image using a lightweight conv–MLP:

$$\begin{array}{cccc}\hfill {\mathbf{u}}_{b,s}& =\mathrm{GAP}\left({\psi }_s\left({\mathbf{Y}}_{b,s}\right)\right)\in {\mathbb{R}}^{C_g},\hfill & \hfill g_{b,s}& ={\mathbf{w}}_s^{\top }{\mathbf{u}}_{b,s}+b_s\in \mathbb{R},\hfill \end{array}$$

(25)

where ${\psi }_s$ is a two-layer $1\times 1$ conv block (optionally with DyT from Equation (23)), $\mathrm{GAP}$ is global average pooling over $(h,w)$, and $({\mathbf{w}}_s,b_s)$ is a linear readout. We stack logits as ${\mathbf{g}}_b=[g_{b,0},g_{b,1},g_{b,2}]\in {\mathbb{R}}^3$.

• Fusion weights.

Stage weights are obtained by a temperature-scaled softmax:

$${\mathit{\omega }}_b=\mathrm{softmax}\left({\mathbf{g}}_b/T\right)\in {\mathsf{\Delta }}^2,\sum _s{\omega }_{b,s}=1,{\omega }_{b,s}\ge 0,$$

(26)

with $T>0$. In all experiments we fix $T=1.0$ unless stated otherwise (Section 3.8).

• Entropy/balance regularization (discourages collapse).

To avoid consistently routing most mass to a single stage, we add an auxiliary regularizer that encourages distributed utilization. This is conceptually aligned with load-balancing losses used to prevent router collapse in mixture-of-experts models [37,38]. We use two complementary terms:

(1) Per-example entropy floor. Let $H\left({\mathit{\omega }}_b\right)$ denote Shannon entropy (natural logarithm): $H\left({\mathit{\omega }}_b\right)=-{\sum }_s{\omega }_{b,s}\log {\omega }_{b,s}.$ We penalize low-entropy gates:

$${\mathcal{L}}_{\mathrm{ent}}={\lambda }_H·{\displaystyle \frac{1}{B}}\sum _{b=1}^B{\left(\max \{0,H_{\mathrm{target}}-H\left({\mathit{\omega }}_b\right)\}\right)}^2,$$

(27)

where $H_{\mathrm{target}}=0.8$ corresponds to discouraging near-one-hot routing for 3 choices.

(2) Batch-level balance. Let $\overline{\mathit{\omega }}={\displaystyle \frac{1}{B}}{\sum }_b{\mathit{\omega }}_b$. We encourage the mean routing mass to remain close to uniform:

$${\mathcal{L}}_{\mathrm{bal}}={\lambda }_{\mathrm{bal}}\sum _{s=0}^2{\left({\overline{\omega }}_s-\frac{1}{3}\right)}^2.$$

(28)

The total gate regularizer is ${\mathcal{L}}_{\mathrm{gate}}={\mathcal{L}}_{\mathrm{ent}}+{\mathcal{L}}_{\mathrm{bal}}$ and is added to the training objective. We use ${\lambda }_H={\lambda }_{\mathrm{bal}}=0.02$ unless stated otherwise (Section 3.8).

• Gain-capped fusion (activation stability, not certification).

We fuse stage maps by a convex combination and apply a per-sample gain cap:

$$\begin{array}{c}\hfill {\mathbf{U}}_b=\sum _{s=0}^2{\omega }_{b,s}{\mathbf{Y}}_{b,s},{\beta }_b=\min \left(1,{\displaystyle \frac{{\beta }_{\max }}{\parallel {\mathbf{U}}_b{\parallel }_{\infty }+ {ϵ}_0}}\right),\\ \hfill {\mathbf{Z}}_b={\beta }_b{\mathbf{U}}_b,\end{array}$$

(29)

where ${\beta }_{\max }\in (0,1]$ is fixed and ${ϵ}_0$ prevents division by zero. This guarantees the output magnitude bound ${\parallel {\mathbf{Z}}_b\parallel }_{\infty }\le {\beta }_{\max }$ by construction. Importantly, this is not a certified global Lipschitz bound for the full mapping $\left\{{\mathbf{Y}}_{b,s}\right\}\mapsto {\mathbf{Z}}_b$ because the weights ${\mathit{\omega }}_b$ depend on the inputs; we treat gain capping as a practical stabilizer and validate robustness empirically with adaptive evaluation (Section 3.8.4).

• Implementation (forward pass).

Algorithm 3 summarizes the gate forward pass. Gate losses are computed during training and added to the total loss; no separate update loop is required.

• Gate diagnostics (reported).

To verify that the gate behaves as intended (Figure 4), we report: (i) the distribution of per-sample entropies $H\left(\mathit{\omega }\right)$ and the fraction of samples below $H_{\mathrm{target}}$, (ii) the batch-level routing mass $\overline{\mathit{\omega }}$, (iii) distributions of $\beta$ and ${\parallel \mathbf{U}\parallel }_{\infty }$, and (iv) the effect of removing ${\mathcal{L}}_{\mathrm{gate}}$ and/or gain capping on clean and robust correlation (Section 4).

Algorithm 3: GateFuse: entropy-regularized, gain-capped fusion

3.7. Complexity and Backbone Compatibility

Asymptotic cost (full vs. Nyström SRM). Consider one SRM cross-stage link with $N_q$ query tokens (shallow stage), $N_k$ key/value tokens (deep stage), B batch size, H heads, head width d, and Nyström rank $r\ll N_k$. Full cross-attention forms an $N_q\times N_k$ attention map, with compute $\mathcal{O}\left(BH{N}_q{N}_{k}d\right)$ and memory $\mathcal{O}\left(BH{N}_q{N}_k\right)$. In contrast, SRM uses Nyström cross-attention (Section 3.3) and replaces the quadratic map by low-rank blocks of size $N_q\times r$ and $r\times N_k$, yielding per-link complexity

$$\begin{array}{cc}\hfill \mathbf{Compute}:& \mathcal{O}\left(BH(N_{q}rd+r{N}_{k}d+r^{2}d)+BH{r}^3\right),\hfill \\ \hfill \mathbf{Memory}:& \mathcal{O}\left(BH(N_{q}r+r{N}_k+r^2)\right),\hfill \end{array}$$

(30)

where $r^3$ is the fp32 $r\times r$ solve (Section 3.4) and is a small constant for typical $r\le 128$. For a schedule $\mathcal{S}$ with $\left|\mathcal{S}\right|$ links, the SRM neck cost scales linearly in $\left|\mathcal{S}\right|$.

Windowed attention as a reference point. For window-based self-attention with window size $M\times M$ and N total tokens, compute scales as $\mathcal{O}\left(BHN{M}^{2}d\right)$ and is linear in image size when M is fixed. We use this reference when comparing SRM overhead to hierarchical transformer backbones.

Instantiated MACs (unambiguous reporting). When reporting SRM compute, we explicitly distinguish: (i) per head vs. summed over H heads, and (ii) per link vs. summed over the schedule $\left|\mathcal{S}\right|$. The dominant term in (30) is typically the matmul $Q{K}_r^{\top }$, which costs $N_{q}rd$ MACs per head per link. For the example used throughout the paper, $N_q=\mathrm{12,288}$, $r=96$, and $d=55$, the per-head-per-link cost is $N_{q}rd\approx 6.49\times {10}^7$ MACs. With $H=8$ heads and schedule ${\mathcal{S}}_{\mathrm{deep}}=\{(0,3),(1,3),(2,3)\}$ (Equation (5)), the $Q{K}_r^{\top }$ term contributes approximately $0.68$ GMACs per image (summing over links and heads) under the token sizes induced by a $384\times 512$ input. We report both (a) per-link MACs and (b) total neck MACs under the exact schedule used in experiments.

How MACs/params are measured (reproducibility). We report neck-only parameter counts and operation counts using a standard PyTorch counter (fvcore FlopCountAnalysis) on the exact tensor shapes induced by the test resolution. Because counting conventions differ (e.g., whether a fused multiply–add counts as one or two FLOPs), we report MACs/GMACs as the primary compute unit and state the tool/convention used. We complement static counts with operator-level profiling using torch.profiler.

Peak GPU memory (what is measured). We report peak allocated and peak reserved GPU memory for (i) a forward pass and (ii) training, resetting CUDA memory stats before measurement. All memory numbers are accompanied by batch size, precision (fp32/bf16/fp16), and input resolution.

Backbone compatibility (scoped; not “universal”). SRM is a plug-in neck that assumes a multi-scale backbone interface: a feature pyramid ${\left\{{\mathbf{F}}_{\mathcal{l}}\right\}}_{\mathcal{l}=0}^3$ with arbitrary channels and spatial sizes (Section 3.2). A $1\times 1$ projection aligns channels to a common width C, after which SRM operates on tokens. Thus SRM is compatible with both CNN-style multi-stage backbones (e.g., ConvNeXt, MobileNetV3) [31,32] and hierarchical transformer pyramids (e.g., PVT/Swin) [1,2], We avoid claiming unconditional universality: empirical backbone-agnosticism is established only by evaluating qualitatively different backbones under the same training and robustness protocol (Section 3.8.4, Section 3.8.5 and Section 4).

Neck efficiency table (reported separately from robustness). 

Table 3. Neck efficiency comparison (fixed backbone/input). Neck-only compute and memory are measured at $384\times 512$ with bf16 inference, batch size B = 1. Performance metrics (clean and robust correlation) are reported separately in the main results tables.

Neck	Params (M)	MACs (GMAC)	Peak Alloc (GB)
None (backbone→head)	0.00	0.00	0.00
Simple concat+MLP fuse	0.41	0.09	0.06
SRM (Nyström + $\tau$ + GateFuse)	0.78	0.82	0.24
SRM w/o conditioning (no fp32 solve)	0.78	0.81	0.20
SRM w/o gain cap ($\beta \equiv 1$)	0.78	0.82	0.24

reports neck-only efficiency overhead (parameters, MACs, and peak memory) under a fixed measurement protocol. Clean accuracy and robustness are reported in the main results tables (Section 4.1 and Tables 6–8) to avoid mixing metrics across datasets/threat models.

3.8. Implementation Details

This section consolidates all practical details needed to reproduce our results under the MDPI Technologies format: the backbone interfaces used by SRM, datasets and splits, preprocessing and metric computation, training hyperparameters, and the adversarial evaluation protocol (including threat models and attack objectives). For reader convenience, we structure these details as numbered subsubsections under this implementation section.

3.8.1. Backbones Under Study

Backbone interface. SRM is a plug-in neck that consumes a four-level feature pyramid ${\left\{{\mathbf{F}}_{\mathcal{l}}\right\}}_{\mathcal{l}=0}^3$ (Section 3.2). For each backbone, we extract four intermediate stages (stride $\{4,8,16,32\}$) and project channels to a common width C using 1 × 1 convolutions before tokenization.

Backbones. We evaluate SRM on four backbones spanning transformer and ConvNet families: (i) Pyramid Vision Transformer (PVT) [1], (ii) Swin Transformer [2], (iii) ConvNeXt [31], (iv) MobileNetV3 [32]. All backbones use ImageNet-1K pretrained weights and are fine-tuned end-to-end on IQA.

Pyramid extraction and channel alignment. 

Table 4. Extracted 4-level pyramids used by SRM. We extract stage features at strides $\{4,8,16,32\}$ and align to a common width C via 1 × 1 projections. Listed channel counts correspond to standard published configurations.

Backbone	$({\mathit{C}}_0,{\mathit{C}}_1,{\mathit{C}}_2,{\mathit{C}}_3)$	Strides	Pretrain	Notes
PVT-Small [1]	(64, 128, 320, 512)	(4, 8, 16, 32)	IN-1K	hierarchical tokens
Swin-T [2]	(96, 192, 384, 768)	(4, 8, 16, 32)	IN-1K	shifted windows
ConvNeXt-T [31]	(96, 192, 384, 768)	(4, 8, 16, 32)	IN-1K	ConvNet stages
MobileNetV3-L [32]	(24, 40, 112, 160)	(4, 8, 16, 32)	IN-1K	low-resource

lists the extracted stage channels. Unless otherwise stated, SRM uses aligned width $C=192$ for all backbones.

3.8.2. Datasets, Splits, and Metrics

Datasets. We report results on:

• TID2013 [39]: 25 reference images, 24 distortion types, 5 levels (3000 distorted images) with MOS.
• KonIQ-10k [40]: 10,073 in-the-wild images with MOS from crowdsourcing.

Splits (content-separated where applicable).

• TID2013: content-separated split by reference-image ID (no leakage of content). We use 15 reference images for training, 5 for validation, and 5 for testing (3000 images distributed accordingly across distortions). We repeat with three seeds (0/1/2) by re-sampling the reference-ID partition and report mean ± std.
• KonIQ-10k: we use the widely adopted train/val/test assignment distributed with common KonIQ loaders [40].

Preprocessing. Images are loaded in RGB and normalized by ImageNet mean/std for all backbones. We use a fixed evaluation resolution of $384\times 512$:

• Train: resize shorter side to 512, random crop $384\times 512$, random horizontal flip (p = 0.5).
• Eval: resize shorter side to 512, center crop $384\times 512$ (deterministic).

MOS values are linearly normalized to $[0,1]$ per dataset for regression stability; reported correlations are computed against the original MOS ordering.

Metrics (clean and robust). We report SROCC and PLCC on the test set for clean images and for adversarially perturbed images. To compute PLCC in the standard IQA manner, we apply a monotonic five-parameter logistic mapping fit on the validation set, following VQEG-style calibration [41]. We fit the mapping once per model using clean validation predictions and keep the parameters fixed when evaluating PLCC on the test set for both clean and adversarial images. Specifically, given raw predictions $\widehat{y}$ we fit parameters $\mathit{\pi }=({\pi }_1,\dots ,{\pi }_5)$ on val:

$$g(\widehat{y};\mathit{\pi })={\pi }_1\left(\frac{1}{2}-\frac{1}{1+\exp \left({\pi }_2(\widehat{y}-{\pi }_3)\right)}\right)+{\pi }_4\widehat{y}+{\pi }_5.$$

(31)

We then compute PLCC between $g(\widehat{\mathbf{y}};\mathit{\pi })$ and MOS on the test set. SROCC is computed on raw scores (no mapping).

3.8.3. Training Protocol

Objective. All models minimize MSE on normalized MOS: ${\mathcal{L}}_{\mathrm{reg}}={\displaystyle \frac{1}{B}}{\sum }_{i=1}^B{({\widehat{y}}_i-y_i)}^2.$ We add SRM auxiliary losses when enabled: ${\mathcal{L}}_{\tau }$ (Equation (21)) and ${\mathcal{L}}_{\mathrm{gate}}$ (Equations (27) and (28)).

Optimization and schedules. Unless otherwise stated, we use AdamW with betas $(0.9,0.999)$, $ϵ={10}^{-8}$, weight decay ${10}^{-4}$, cosine LR decay with 5-epoch warmup, and 120 epochs. We use the following differential learning rates: backbone LR $1\times {10}^{-5}$, neck/head LR $5\times {10}^{-5}$. Gradient clipping uses global norm 1.0.

Precision and determinism. Training uses bf16 when available; otherwise fp16 with dynamic loss scaling. We fix seeds (Python/NumPy/PyTorch), enable deterministic CuDNN where supported, and report mean ± std over three runs.

SRM hyperparameters (fixed unless ablated). 

Table 5. Fixed hyperparameters used across experiments (unless ablated).

Component	Setting
SRM links	schedule ${\mathcal{S}}_{\mathrm{deep}}=\{(0,3),(1,3),(2,3)\}$ (one-way deep→shallow)
Nyström	rank $r=96$, heads $H=8$, head dim $d=55$, aligned width $C=192$
Conditioning	ridge ${\lambda }_0={10}^{-3}$ (Equation (16)), solve in fp32; LU fast-path if ${\kappa }_{\mathrm{reg}}<{10}^4$
$\tau$ stabilization	EMA $m=0.99$ (Equation (19)), ${\lambda }_{\tau }=0.01$ (Equation (21)), clamp $\tau \in [0.25,4.0]$
DyT	enabled by default (Equation (23))
GateFuse	temperature $T=1.0$, $H_{\mathrm{target}}=0.8$, ${\lambda }_H={\lambda }_{\mathrm{bal}}=0.02$, gain cap   ${\beta }_{\max }=1.0$

lists all constants used in SRM, conditioning, $\tau$-stabilization, and GateFuse.

Throughput measurement (training). On an RTX 4090 (24 GB), batch size 4, $384\times 512$, bf16, 8 dataloader workers: we measure 0.26 s/iter (forward+backward+optimizer) and peak allocated memory ≈ 6.0 GB using CUDA events, after a 50-iteration warm-up and excluding dataloader time (Section 3.8.5).

3.8.4. Adversarial Evaluation

Threat models. We evaluate white-box robustness under ${\mathcal{l}}_{\infty }$ with $ϵ=4/255$ and ${\mathcal{l}}_2$ with $ϵ=0.5$, using projection onto the norm ball and pixel clipping (Equation (1)).

Attack objectives (per-image and correlation-aware). We evaluate three complementary objectives:

(1) Per-image regression: maximize MSE on MOS: ${\mathcal{l}}_{\mathrm{mse}}({\mathbf{x}}^{\mathrm{adv}},y)={(f_{\theta }\left({\mathbf{x}}^{\mathrm{adv}}\right)-y)}^2.$

(2) Score drift (label-free): maximize prediction change: ${\mathcal{l}}_{\mathrm{drift}}({\mathbf{x}}^{\mathrm{adv}},\mathbf{x})=|f_{\theta }\left({\mathbf{x}}^{\mathrm{adv}}\right)-f_{\theta }\left(\mathbf{x}\right)|.$

(3) Pairwise inversion (correlation-aware; batch-level). We fix the attack batch size to $B=B_{\mathrm{atk}}=8$ throughout and form batches by iterating over the test set in a deterministic order (no shuffle); the same batches are reused across methods to avoid batch-selection variance. For a batch ${\left\{({\mathbf{x}}_i,y_i)\right\}}_{i=1}^B$, define all comparable pairs $\mathcal{P}=\{(i,j):|y_i-y_j| >\mathsf{\Delta }\}$ with $\mathsf{\Delta }=0.05$ (MOS normalized to $[0,1]$). If $\left|\mathcal{P}\right|=0$ for a batch, we set ${\mathcal{l}}_{\mathrm{inv}}=0$ for that batch. Let $s_{ij}=\mathrm{sign}(y_i-y_j)$. We maximize a smooth inversion loss

$$\begin{array}{cc}\hfill {\mathcal{l}}_{\mathrm{inv}}\left(\left\{{\mathbf{x}}_i^{\mathrm{adv}}\right\}\right)& ={\displaystyle \frac{1}{\left|\mathcal{P}\right|}}\sum _{(i,j)\in \mathcal{P}}\log \left(1+\exp \left(\frac{s_{ij}}{\gamma }({\widehat{y}}_i^{\mathrm{adv}}-{\widehat{y}}_j^{\mathrm{adv}})\right)\right),\hfill \\ \hfill {\widehat{y}}_i^{\mathrm{adv}}& =f_{\theta }\left({\mathbf{x}}_i^{\mathrm{adv}}\right).\hfill \end{array}$$

(32)

with temperature $\gamma =0.1$. Maximizing (32) encourages violations of the MOS ordering within a batch and provides a lightweight, differentiable proxy for rank-correlation degradation. More general differentiable ranking/sorting relaxations can be used to construct direct surrogates for rank-based metrics (including differentiable Spearman-style objectives) but are computationally heavier and introduce additional approximation/temperature choices; we therefore use the pairwise logistic surrogate for stability and simplicity [42,43,44,45].

First-order attacks and hyperparameters. We use FGSM and multi-step PGD with random starts for both norms:

• FGSM-${\mathcal{l}}_{\infty }$:$ϵ=4/255$.
• PGD-${\mathcal{l}}_{\infty }$: steps $T=40$, step size $\alpha =1/255$, restarts $R=5$.
• PGD-${\mathcal{l}}_2$: steps $T=40$, step size $\alpha =0.02$, restarts $R=5$.

Attack-strength sweeps and monotonicity checks. To guard against false robustness, we run a sweep over PGD steps $T\in \{10,20,40,80\}$ (Figure 5) and verify monotonic degradation of robust correlation. We use $R=5$ random restarts throughout (Table 9) and report per-objective results as well as the worst-case over the attack ensemble, following anti-gradient masking guidance and an AutoAttack-inspired “worst-case over strong attacks” philosophy [7,9].

Transfer and black-box sanity checks. We include transfer PGD attacks crafted on a surrogate backbone (Swin-T→PVT and vice versa) and evaluated on the target model, reporting robust SROCC/PLCC under transfer to detect gradient masking artifacts [7].

Stochastic components and EOT. When evaluating SRM variants with stochastic key/value pooling, we attack with EOT: each PGD step estimates the gradient by averaging over $K=20$ stochastic forward passes. This avoids overstating robustness due to randomness and follows standard practice for randomized defenses [25].

AutoAttack-inspired ensemble for regression. AutoAttack advocates using a diverse, fixed ensemble of strong attacks as a minimal robustness test without per-model tuning [9]. Adapting this philosophy to NR-IQA (regression/ranking), we define a fixed attack suite consisting of: (i) PGD-MSE, (ii) PGD-drift, (iii) PGD-pairwise inversion (correlation-aware; batch-level), and (iv) transfer PGD (black-box sanity check). For each model and threat setting (norm, $ϵ$), we report worst-case robust correlation as the minimum SROCC/PLCC over this suite. All hyperparameters are fixed a priori (Table 9) and we include step sweeps, transfer tests, and EOT when stochastic, following standard anti–false-robustness guidance [7,25,46].

3.8.5. Hardware Footprint and Profiling

Compute (neck-only MACs) and parameters. We report neck-only parameters and MACs using fvcore FlopCountAnalysis on the exact input tensor shapes induced by $384\times 512$ evaluation resolution (Section 3.7). We report MACs/GMACs as the primary unit and specify the tool convention.

Timing. We measure GPU time with CUDA events with 50-iteration warm-up and synchronization. We report both inference (forward only) and training (forward + backward + optimizer) time per iteration, and state whether dataloader time is included (we exclude it by default).

Peak memory. We report peak allocated and peak reserved CUDA memory after resetting memory stats. All numbers are accompanied by batch size and precision (bf16/fp16/fp32).

Quadratic attention baseline (defined). For efficiency comparisons, the “quadratic baseline” replaces Nyström cross-attention with full cross-attention (softmax over $N_q\times N_k$) under the same schedule ${\mathcal{S}}_{\mathrm{deep}}$ and the same backbone. We report measured MACs, wall-clock time, and peak memory for this operator under identical conditions.

4. Results

4.1. Setup and Reporting Conventions

We evaluate SRM on two NR-IQA datasets (TID2013 and KonIQ-10k) and four backbones (PVT-S, Swin-T, ConvNeXt-T, MobileNetV3-L) using the implementation details in Section 3.8. All metrics are computed on the test set and reported as mean ± std over three seeds. Clean performance is reported as SROCC (raw scores) and PLCC after a five-parameter logistic mapping fit on the validation set (Equation (31)). Robust performance is reported under the threat models and fixed attack suite in Section 3.8.4, using worst-case robust correlation (minimum SROCC/PLCC over the full suite: per-image objectives, correlation-aware objective, and transfer PGD, for a fixed norm and $ϵ$). We additionally report attack-strength sweeps (monotonicity check) and transfer results as sanity checks to rule out false robustness/obfuscated gradients [7,9,46].

4.2. Clean-Set Performance

Goal. We test whether SRM improves or preserves standard NR-IQA accuracy on clean inputs, measured by SROCC/PLCC between predicted scores and MOS.

Results. 

Table 6. Clean-set performance (higher is better). mean ± std over three seeds. SROCC uses raw scores; PLCC uses logistic mapping fit on val (Equation (31)).

Model	TID2013 (SROCC/PLCC)	KonIQ-10k (SROCC/PLCC)
PVT-S baseline	$0.887\pm 0.004$ / $0.892\pm 0.005$	$0.904\pm 0.003$ / $0.909\pm 0.004$
PVT-S + SRM	$\mathbf{0.892}\pm \mathbf{0.003}$ / $\mathbf{0.897}\pm \mathbf{0.004}$	$\mathbf{0.910}\pm \mathbf{0.003}$ / $\mathbf{0.915}\pm \mathbf{0.004}$
Swin-T baseline	$0.880\pm 0.003$ / $0.886\pm 0.004$	$0.898\pm 0.003$ / $0.904\pm 0.003$
Swin-T + SRM	$\mathbf{0.886}\pm \mathbf{0.003}$ / $\mathbf{0.891}\pm \mathbf{0.003}$	$\mathbf{0.906}\pm \mathbf{0.002}$ / $\mathbf{0.911}\pm \mathbf{0.003}$
ConvNeXt-T baseline	$0.868\pm 0.004$ / $0.874\pm 0.005$	$0.889\pm 0.004$ / $0.896\pm 0.004$
ConvNeXt-T + SRM	$\mathbf{0.875}\pm \mathbf{0.003}$ / $\mathbf{0.881}\pm \mathbf{0.004}$	$\mathbf{0.899}\pm \mathbf{0.003}$ / $\mathbf{0.905}\pm \mathbf{0.004}$
MobileNetV3-L baseline	$0.842\pm 0.005$ / $0.851\pm 0.006$	$0.865\pm 0.004$ / $0.873\pm 0.005$
MobileNetV3-L + SRM	$\mathbf{0.850}\pm \mathbf{0.004}$ / $\mathbf{0.858}\pm \mathbf{0.005}$	$\mathbf{0.875}\pm \mathbf{0.004}$ / $\mathbf{0.883}\pm \mathbf{0.005}$

shows that SRM consistently improves clean correlation across backbones by a small but repeatable margin (typically $+0.005$ to $+0.010$ SROCC), indicating that cross-scale mixing and gated fusion do not harm standard IQA quality prediction.

Interpretation. Clean gains are intentionally modest: SRM is not designed to outperform specialized IQA heads on clean benchmarks, but to improve adversarial correlation preservation while maintaining competitive clean correlation.

4.3. Robustness Under White-Box Attacks

Goal. Robustness in NR-IQA is measured by correlation preservation under adversarial perturbations: we compute SROCC/PLCC on perturbed test inputs under the threat models and objectives defined in Section 3.8.4.

Why per-image attacks are insufficient. Because NR-IQA is evaluated by set-level ranking/linear correlation, robustness must be tested with both: (i) per-image regression attacks (which degrade pointwise accuracy), and (ii) correlation-aware attacks that directly induce ordering inversions in a batch, as emphasized by recent NR-IQA attack analyses [22]. SROCC is the correlation statistic applied to ranked variables; therefore, attacks that increase rank inversions directly target SROCC degradation. This also connects to Kendall-style rank agreement measures that are explicitly defined via concordant/discordant pairs. We therefore include the pairwise inversion loss in Equation (32), and we discuss stronger (but heavier) differentiable ranking relaxations in Section 3.8.4.

Per-attack breakdown 

Table 7. White-box robustness breakdown on TID2013 (PVT-S). Robust SROCC/PLCC at ${\mathcal{l}}_{\infty }$$ϵ=4/255$ with PGD (40 steps, 5 restarts) under each objective family. Overall worst-case robustness (minimum over the full fixed suite, including transfer) is summarized in Table 8.

Model	PGD MSE	PGD Drift	PGD pairInv	Transfer PGD
PVT-S baseline	$0.620/0.630$	$0.580/0.592$	$0.540/0.552$	$0.570/0.581$
PVT-S + SRM	$\mathbf{0.710}/\mathbf{0.720}$	$\mathbf{0.660}/\mathbf{0.672}$	$\mathbf{0.620}/\mathbf{0.632}$	$\mathbf{0.650}/\mathbf{0.662}$

reports robust correlations for PVT-S on TID2013 under each objective family. Pairwise inversion is consistently the strongest objective for degrading SROCC, while MSE can be strongest for PLCC depending on the dataset. We report the worst-case across the ensemble as the primary robust score.

Worst-case robustness across backbones and datasets. 

Table 8. Worst-case robustness (min over attack ensemble). Robust SROCC/PLCC under ${\mathcal{l}}_{\infty }$$ϵ=4/255$ and ${\mathcal{l}}_2$$ϵ=0.5$. Worst-case takes the minimum over {PGD-MSE, PGD-drift, PGD-pairInv, Transfer-PGD} under the same norm and $ϵ$. PGD uses 40 steps and 5 restarts; EOT is used only when stochastic pooling is enabled (Section 3.8.4).

Model	TID ${\mathit{\ell }}_{\mathit{\infty }}$ (SROCC/PLCC)	TID ${\mathit{\ell }}_2$ (SROCC/PLCC)	KonIQ ${\mathit{\ell }}_{\mathit{\infty }}$ (SROCC/PLCC)	KonIQ ${\mathit{\ell }}_2$ (SROCC/PLCC)
PVT-S baseline	$0.540/0.552$	$0.600/0.612$	$0.580/0.592$	$0.630/0.644$
PVT-S + SRM	$\mathbf{0.620}/\mathbf{0.632}$	$\mathbf{0.680}/\mathbf{0.692}$	$\mathbf{0.660}/\mathbf{0.674}$	$\mathbf{0.710}/\mathbf{0.725}$
Swin-T baseline	$0.520/0.533$	$0.580/0.593$	$0.560/0.573$	$0.610/0.625$
Swin-T + SRM	$\mathbf{0.590}/\mathbf{0.603}$	$\mathbf{0.650}/\mathbf{0.666}$	$\mathbf{0.630}/\mathbf{0.645}$	$\mathbf{0.680}/\mathbf{0.696}$
ConvNeXt-T baseline	$0.490/0.504$	$0.550/0.565$	$0.530/0.545$	$0.580/0.596$
ConvNeXt-T + SRM	$\mathbf{0.560}/\mathbf{0.575}$	$\mathbf{0.620}/\mathbf{0.637}$	$\mathbf{0.600}/\mathbf{0.616}$	$\mathbf{0.650}/\mathbf{0.668}$
MobileNetV3-L baseline	$0.440/0.455$	$0.500/0.517$	$0.480/0.496$	$0.530/0.549$
MobileNetV3-L + SRM	$\mathbf{0.500}/\mathbf{0.516}$	$\mathbf{0.560}/\mathbf{0.579}$	$\mathbf{0.540}/\mathbf{0.558}$	$\mathbf{0.590}/\mathbf{0.611}$

summarizes worst-case robust correlation across backbones for both norms. SRM improves worst-case robust SROCC by an absolute $+0.06$ to $+0.08$ (SROCC points) depending on dataset and backbone, while preserving clean performance (Table 6). The gains are largest for low-resource backbones (MobileNetV3-L), consistent with SRM acting as a stabilizing fusion module.

Attack configuration table. 

Table 9. Attack configurations used throughout. PGD uses random starts; all gradients are taken through the full model (including SRM) with no BPDA approximations. For the batch-level pairwise inversion objective, we fix the attack batch size to $B_{\mathrm{atk}}$ = 8 and use deterministic test-set batching (Section 3.8.4). EOT with K = 20 samples per PGD step is used only for stochastic pooling variants; otherwise models are deterministic at test time.

Attack	Objective	Norm	$\mathit{ϵ}$	Steps	Step Size $\mathit{\alpha }$	Restarts / EOT K
FGSM	MSE	${\mathcal{l}}_{\infty }$	$4/255$	1	$4/255$	–
PGD	MSE	${\mathcal{l}}_{\infty }$	$4/255$	40	$1/255$	$R=5$
PGD	drift	${\mathcal{l}}_{\infty }$	$4/255$	40	$1/255$	$R=5$
PGD	pairInv (Equation (32))	${\mathcal{l}}_{\infty }$	$4/255$	40	$1/255$	$R=5$
PGD	MSE	${\mathcal{l}}_2$	$0.5$	40	$0.02$	$R=5$
PGD	drift	${\mathcal{l}}_2$	$0.5$	40	$0.02$	$R=5$
PGD	pairInv (Equation (32))	${\mathcal{l}}_2$	$0.5$	40	$0.02$	$R=5$
EOT-PGD	any above	${\mathcal{l}}_{\infty }/{\mathcal{l}}_2$	same	40	same	$R=5,K=20$

lists all attack hyperparameters. We follow anti-gradient masking guidance by using multiple restarts, step sweeps, transfer sanity checks, and (when stochasticity is present) EOT [7,9,25].

Attack-strength sweeps (anti-masking check). We run a sweep over steps (Section 3.8.4) and verify monotonic degradation. Figure 5 shows SROCC vs. PGD steps for PVT-S on TID2013 under ${\mathcal{l}}_{\infty }$$ϵ=4/255$.

4.4. Attack Transfer (Black-Box Sanity Check)

Motivation. Transfer-based black-box attacks are a standard sanity check (

Table 10. Transfer robustness (black-box sanity check). SROCC/PLCC on adversarial examples crafted on the surrogate model and evaluated on the target. Results shown for ${\mathcal{l}}_{\infty }$$ϵ=4/255$.

Transfer Setting	Target Robust SROCC/PLCC	Interpretation
Swin-T baseline → PVT-S baseline (craft on surrogate, evaluate on target)	$0.650/0.662$	cross-backbone transfer is non-trivial
PVT-S baseline → Swin-T baseline	$0.635/0.648$	consistent with a standard transfer gap
PVT-S baseline → PVT-S SRM	$0.695/0.707$	SRM remains attackable under transfer (anti-masking)
PVT-S SRM → PVT-S baseline	$0.640/0.653$	baseline remains vulnerable

): if a model appears robust only to direct white-box gradients but remains vulnerable to transfer, this can indicate gradient masking or evaluation artifacts [7].

Protocol. We craft adversarial examples on a surrogate backbone and evaluate on a target backbone under the same norm and $ϵ$, using PGD with MSE objective (40 steps, 5 restarts). We report both directions: baseline→SRM and SRM→baseline for the same backbone family.

4.5. Diagnostics (Stability and Mechanism Evidence)

Goal. SRM includes numerical-stability mechanisms (conditioning, $\tau$ scale stabilization, gating with gain cap). To support the mechanism story without implying certification, we report diagnostics that directly measure: (i) landmark solve stability, (ii) post-attention scale behavior, and (iii) gate behavior.

What we report (concrete artifacts).

• Conditioning and solve accuracy: Table 1 reports quantiles of ${\kappa }_{\mathrm{reg}}\left(W_{\lambda }\right)$ and solve residuals (Section 3.4).
• Scale stabilization: Table 2 reports normalized RMS magnitudes before/after $\tau$; Figure 3 visualizes training trajectories (Section 3.5).
• Gate behavior: Figure 4 reports entropy, routing mass, and gain-cap distributions (Section 3.6).

Design links (interpretation, not proof). Across backbones, robust gains correlate with (i) improved landmark conditioning (lower p90 ${\kappa }_{\mathrm{reg}}$ and residuals), (ii) stabilized post-attention scale (post-$\tau$$\tilde{\rho }$ concentrated near 1), and (iii) non-collapsed routing (entropy above $H_{\mathrm{target}}$ for most samples) with bounded fused activation magnitude. These diagnostics strengthen the causal narrative, while the primary evidence remains robust correlation under the full attack suite (Table 7 and Table 8).

4.6. Ablation Study

Goal. We ablate SRM components to identify which design choices matter for (i) clean NR-IQA correlation and (ii) robust correlation preservation under a fixed white-box threat model.

Setup (fixed protocol). Unless otherwise stated, all ablations use PVT-S trained on TID2013 with the training protocol in Section 3.8.3. We report mean ± std over three seeds. Clean uses test-set SROCC on unperturbed inputs. Robust uses test-set SROCC under ${\mathcal{l}}_{\infty }$ attacks with $ϵ=4/255$ and PGD (40 steps, 5 restarts), reporting the worst-case over the full attack ensemble from Section 3.8.4 (PGD-MSE, PGD-drift, PGD-pairInv, and transfer PGD). (All SRM variants are deterministic at test time; EOT is not used in this ablation table).

• Landmark quality and conditioning.

Replacing ridge-leverage landmarks with uniform landmark selection yields the largest robustness drop ($-0.034$ SROCC), supporting that landmark diversity and the resulting conditioning of the landmark system are important for robust correlation. This aligns with the conditioning diagnostics reported in Table 1: uniform landmarks increase the p90 ${\kappa }_{\mathrm{reg}}\left(W_{\lambda }\right)$ and solve residuals (Section 3.4), which in turn increases sensitivity of the cross-attention fusion to perturbations.

• Scale stabilization.

Disabling $\tau$ stabilization ($\tau \equiv 1$) reduces clean and robust SROCC ($-0.006$/$-0.018$), consistent with $\tau$ acting as a numerical scale corrector that prevents attention-output drift (Section 3.5, Table 2). We interpret this as mechanism evidence (diagnostics shift) rather than a proof-backed robustness guarantee.

• Gating and gain capping.

Turning off gate regularization or removing the gain cap mainly impacts robustness ($-0.015$ and $-0.020$) while leaving clean SROCC nearly unchanged, consistent with GateFuse acting as a robustness-oriented stabilizer (Section 3.6). We do not interpret gain capping as a certified global Lipschitz bound; it bounds only the fused activation magnitude by construction (Equation (29)).

• Approximation budget and inverse routine.

Reducing Nyström rank ($r=48$) decreases robust SROCC by $-0.021$, indicating that approximation capacity matters. Replacing the default fp32 SVD/LU routine by the gated Neumann-m = 3 approximation yields a smaller robustness drop ($-0.008$), suggesting that, in this configuration, rank is a more critical knob than the specific stable-solve routine.

Key takeaways (supported by

Table 11. One-out ablations on TID2013 (PVT-S). SROCC (higher is better). Robust is worst-case over the attack ensemble at ${\mathcal{l}}_{\infty }$$ϵ=4/255$ (Section 3.8.4). $\mathsf{\Delta }$ values are relative to Full SRM. Boldface in the table body highlights the full SRM configuration and the best values in each column.

Variant	Clean	Robust	${\mathsf{\Delta }}_{\mathit{C}}$	${\mathsf{\Delta }}_{\mathit{R}}$
No neck (PVT-S baseline)	$0.887\pm 0.004$	$0.540\pm 0.010$	$-0.005$	$-0.080$
Full SRM	$\mathbf{0.892}\pm \mathbf{0.003}$	$\mathbf{0.620}\pm \mathbf{0.010}$	—	—
SRM without spectral landmarks (uniform)	$0.891\pm 0.003$	$0.555\pm 0.010$	$-0.001$	$-0.065$
SRM without regularized solve ($\lambda$ = 0)	$0.891\pm 0.004$	$0.565\pm 0.010$	$-0.001$	$-0.055$
SRM without $\tau$ stabilization ($\tau \equiv 1$)	$0.892\pm 0.004$	$0.585\pm 0.010$	$+0.000$	$-0.035$
SRM without entropy gating (uniform fusion)	$0.891\pm 0.003$	$0.575\pm 0.010$	$-0.001$	$-0.045$
SRM with fixed Nyström rank (r = 32)	$0.889\pm 0.004$	$0.590\pm 0.010$	$-0.003$	$-0.030$
SRM without stage gating (no cross-scale routing)	$0.890\pm 0.003$	$0.580\pm 0.010$	$-0.002$	$-0.040$

).

 I1. 

Landmark selection and conditioning matter for robustness: uniform landmarks lead to the largest robust drop.

 I2. 

Scale stabilization and gating are complementary: $\tau$ affects both clean and robust performance, while gate regularization and gain capping primarily affect robustness.

 I3. 

Approximation budget matters: reducing Nyström rank harms robustness more than changing the inverse routine.

5. Discussion, Limitations, and Future Work

5.1. Mechanism Evidence: What We Measure (Not What We “Guarantee”)

SRM is motivated by the hypothesis that numerical stability (in low-rank attention solves and activation scaling) and controlled cross-scale fusion can improve adversarial correlation preservation in NR-IQA without external defense wrappers. We therefore support the design with direct measurements tied to each component, while avoiding theorem-level guarantees.

• Conditioning/solve stability (Nyström landmarks). We report quantiles of the regularized condition number ${\kappa }_{\mathrm{reg}}\left(W_{\lambda }\right)$ and solve residuals for the $r\times r$ landmark system (Table 1; Section 3.4). These diagnostics make the “stable landmark solve” claim falsifiable and help interpret robustness differences between landmark/conditioning ablations (Table 11).
• Post-attention scale stabilization. We report normalized post-attention RMS magnitudes before/after $\tau$ (Table 2) and training-time trajectories (Figure 3; Section 3.5). We treat $\tau$ (and DyT) as numerical stabilizers that reduce scale drift; robustness is established empirically under the attack suite.
• Gate behavior and activation spikes. We report entropy and routing-mass diagnostics for fusion weights, along with gain-cap statistics (Figure 4; Section 3.6). These measurements support that the fusion does not collapse to a single stage and that fused activations are bounded in magnitude. We do not interpret gain capping as a certified Lipschitz guarantee for the full network because gate weights are input-dependent (Equation (29)).

Primary evidence remains robust correlation under strong evaluation. The core empirical claim is supported by worst-case robust SROCC/PLCC under an explicit attack ensemble (Table 7 and Table 8), including correlation-aware objectives (Equation (32)), attack-strength sweeps (Figure 5), transfer sanity checks (Table 10), and EOT for stochastic variants when enabled (Section 3.8.4) [7,9,25].

5.2. Limitations

 L1. 

Dataset coverage is still limited. We evaluate on two datasets (TID2013 and KonIQ-10k), which cover synthetic distortions and in-the-wild conditions, respectively. However, broader generality (e.g., LIVE/CSIQ/CLIVE/KoNViD for video) is not established in this work.

 L2. 

Attack coverage remains primarily first-order. Our evaluation focuses on strong first-order, multi-step attacks with restarts, sweeps, transfer checks, and EOT for stochasticity (Section 3.8.4; Figure 5) [7,25]. We do not claim robustness against all possible adaptive strategies (e.g., query-based black-box attacks at large budgets, or attacks optimizing non-differentiable dataset-level correlation directly).

 L3. 

Correlation-aware objectives are proxies. The pairwise inversion loss (Equation (32)) is a lightweight differentiable surrogate that induces ranking inversions within a batch and empirically reduces SROCC. It is not identical to directly optimizing Spearman correlation over the full dataset, and different surrogates could lead to different worst-case outcomes.

 L4. 

No certified robustness claims. Although SRM includes explicit magnitude caps and numerical stabilizers, we do not provide certified robustness bounds for the full end-to-end mapping from pixels to quality score. All robustness improvements are empirical under the specified threat model and attack ensemble.

5.3. Future Work

• Regression-/ranking-specific “AutoAttack-style” suites. AutoAttack is a strong parameter-free baseline for classification evaluation [9]. A natural extension for NR-IQA is a standardized ensemble that mixes regression and ranking-aligned objectives (including pairwise and listwise surrogates), with fixed hyperparameters, monotonicity checks, transfer tests, and EOT requirements when stochasticity is present.
• Stronger black-box and physically plausible perturbations. Beyond transfer-based checks, future work should evaluate SRM under query-based black-box attacks (score- and decision-based) and under perceptual/physical constraints (e.g., content-preserving, camera/codec distortions) to complement the norm-bounded threat model studied here.
• Certified deviation bounds for scalar regression. A concrete direction is to combine SRM’s bounded-gain components with certification techniques for regression, e.g., Interval Bound Propagation (IBP) [47] or randomized smoothing adapted to scalar outputs, to bound output deviation ${\sup }_{\parallel x-x^{\prime }\parallel \le ϵ}|f_{\theta }\left(x\right)-f_{\theta }\left(x^{\prime }\right)|$ for IQA regressors under well-defined norms.
• Extensions to video and 3D quality pipelines. The cross-scale stabilization principles in SRM are potentially applicable to temporal and 3D settings where fusion and stability are also central: (i) video quality assessment (temporal fusion + long-range dependencies), and (ii) point-cloud quality assessment (PCQA) pipelines that must remain stable under acquisition noise and adversarial perturbations [26,27]. Recent benchmarks for video quality understanding in LMMs further motivate robust evaluation beyond still images [28].
• Alternative backbone families and operators. Future work could attach SRM-style fusion to non-attention sequence backbones such as selective state-space models (e.g., Mamba) [48] and diffusion-based perceptual decoders [49] and study whether the same conditioning/scale/gating diagnostics predict robustness improvements.

Summary. SRM provides empirical evidence that robustness-by-design components (conditioning, scale stabilization, gated fusion) can improve adversarial correlation preservation in NR-IQA. The remaining gaps are broader dataset coverage, stronger black-box and perceptual threat models, and (if desired) certified deviation bounds.

6. Conclusions

We presented Spectral Robustness Mixer (SRM), a lightweight plug-in neck for no-reference image quality assessment that injects cross-stage feature mixing with explicit numerical-stability and fusion-stability mechanisms. SRM combines (i) Nyström low-rank cross-attention to avoid quadratic attention maps, (ii) an fp32-stabilized landmark solve with conditioning diagnostics, (iii) post-attention scale stabilization via DyT and a learnable $\tau$ rescaler, and (iv) entropy-regularized fusion with gain capping to discourage stage collapse and limit activation spikes.

Across two datasets (TID2013 and KonIQ-10k) and four backbone families, SRM improves worst-case robust correlation under a diverse white-box attack ensemble that includes regression objectives, score drift, and a correlation-aware pairwise inversion loss. Robust gains are achieved while preserving (and modestly improving) clean correlation. We emphasize that SRM does not provide certified robustness guarantees; robustness is established empirically under the specified threat model, with anti–false-robustness checks including sweeps, transfer, and EOT when stochasticity is present.