AMD-Proj: Adaptive Memory-Driven Selective Gradient Projection for Continual Learning in Document Understanding

Abstract

Visually rich document understanding (VrDU) models rely on tightly coupled textual, layout, and visual representations. In real-world deployments, these models must continuously adapt to new document domains over time. However, naïve sequential fine-tuning leads to severe catastrophic forgetting due to shared parameters and strong cross-task interference. Existing continual learning approaches either constrain parameter updates, preserve output distributions, or uniformly suppress gradient directions associated with previous tasks. While effective in limited settings, these strategies fail to balance stability and plasticity in large multimodal transformers. We propose AMD-Proj, an adaptive memory-driven selective gradient projection framework for continual learning in document understanding. It models task knowledge using specific gradient subspaces and adaptively modulates incoming gradients based on their alignment with this memory, selectively blocking interfering directions while reinforcing reusable ones. An efficient truncated SVD mechanism with online subspace merging ensures bounded memory usage and scalability to large transformer-based architectures. We evaluate AMD-Proj on four VrDU benchmarks (FUNSD, SROIE, CORD, and BuDDIE) under a task-incremental learning setting using LayoutLMv2 and LayoutLMv3 backbones. Results show that AMD-Proj reduces catastrophic forgetting and improves F1-based stability over EWC, GPM, LwF, OWM, CUBER, TRGP and parameter-efficient fine-tuning methods. Extensive mechanistic analyses, including gradient spectrum decomposition and layer-wise reuse versus block dynamics, provide insight into how selective gradient projection controls optimization geometry during continual adaptation. These findings establish selective gradient projection as a principled and interpretable approach for continual learning in visually rich document understanding.

1. Introduction

Visually rich documents (VrDs), such as scanned forms, invoices, receipts, contracts, and financial statements, combine textual content with layout structure and visual cues. Recent document understanding models, including LayoutLMv2, LayoutLMv3, and graph-based architectures, have achieved strong performance by jointly encoding these heterogeneous signals within large transformer-based representations [1]. Despite their success, such models are typically trained in an offline setting using static datasets, whereas many real-world document processing systems must operate in a continual learning regime, incrementally adapting to new document types and domains over time.

Continual learning (CL) introduces the challenge of catastrophic forgetting [2], where adapting a model to a new task leads to degradation on previously learned tasks (Figure 1). This challenge is particularly acute in visually rich document understanding. Due to extensive parameter sharing and tight coupling across representation layers, even moderate parameter updates can distort learned layout and visual abstractions, resulting in substantial performance drops on earlier document domains.

Figure 1. Illustration of catastrophic forgetting. Sequential optimization on Task 2 moves parameters away from the Task 1 optimum, increasing Task 1 loss due to parameter drift in a shared model.

A wide range of continual learning methods have been proposed to mitigate forgetting. Regularization-based methods, such as Elastic Weight Consolidation (EWC) [3], identify important parameters and penalize changes to them, but rely on diagonal approximations that scale poorly to large models. Output-based methods, such as Learning without Forgetting (LwF) [4], enforce prediction matching by using earlier models as a teacher to match predictions, but they do not directly regulate internal representations where cross-task interference occurs. Projection-based approaches, including Gradient Projection Memory (GPM) [5], store low-rank subspaces of past gradients and enforce new training ones to be orthogonal to them. While effective at reducing forgetting, uniformly suppressing all past-task directions can overly restrict optimization, limiting plasticity and preventing beneficial knowledge transfer.

In practice, document understanding tasks exhibit both shared structure and task-specific variability; for instance, receipt datasets share common layout patterns that could support positive transfer across related domains. Recent projection-based methods, such as CUBER [6] and TRGP [7], have moved beyond strict orthogonality by explicitly modeling task correlation: they distinguish between regimes of interference and transfer, and selectively reuse frozen weights from correlated past tasks via scaled projections within layer-wise trust regions. While this represents a meaningful advance over uniform suppression, these methods operate primarily in weight space and apply their reuse decisions at the task level, relying on a global characterization of task similarity rather than evaluating individual gradient directions at each training step. Consequently, they do not explicitly model or exploit within-task, per-step variation in gradient alignment. Because their behavior is governed by task-level similarity estimates, their projection mechanism remains effectively fixed throughout a task and cannot capture fine-grained, step-wise variations in gradient structure. This limitation is particularly consequential in visually rich document understanding, where the tight coupling of textual, layout, and visual representations causes gradient structure to vary substantially across layers and mini-batches. This observation motivates AMD-Proj, which addresses this gap by operating at the level of individual gradient directions rather than whole tasks: it continuously evaluates each gradient component against stored subspaces, attenuating those that induce destructive interference while reinforcing those that encode transferable structure enabling a fine-grained, adaptive stability–plasticity trade-off that task-level projection methods cannot provide.

In this work, we propose AMD-Proj, a continual learning framework for visually rich document understanding that explicitly distinguishes between interfering and reusable gradient directions. AMD-Proj represents task knowledge using compact, low-rank gradient subspaces and adaptively modulates incoming gradients based on their instantaneous alignment with previously learned subspaces. AMD-Proj attenuates gradient components associated with destructive interference while reinforcing those aligned with reusable structure. This selective modulation enables controlled adaptation while preserving previously acquired knowledge.

We evaluate AMD-Proj on a sequence of four VrDU benchmarks, FUNSD, SROIE, CORD, and BuDDIE under a task-incremental learning setting using LayoutLMv2 and LayoutLMv3 backbones. Experimental results show that AMD-Proj reduces catastrophic forgetting and improves F1-based stability over Gradient Projection Memory and sequential fine-tuning on LayoutLMv2, while revealing greater forgetting on more tightly integrated architectures, such as LayoutLMv3, where entangled multimodal gradients make selective projection more challenging, despite achieving higher absolute accuracy due to the stronger backbone. Beyond aggregate performance metrics, we provide mechanistic analyses based on gradient spectrum decomposition and layer-wise reuse versus block dynamics, offering an interpretable view of how selective gradient projection shapes optimization behavior across network depth.

The main contributions of this work are as follows:

• We introduce AMD-Proj, a continual learning framework that operates at the granularity of individual gradient directions, explicitly distinguishing between interfering and reusable components. By continuously evaluating per-step alignment, the method selectively attenuates harmful directions while reinforcing transferable ones, establishing a different operating regime from existing task-level projection methods.
• We propose a selective gradient modulation mechanism that replaces uniform suppression with adaptive per-direction control, enabling simultaneous mitigation of catastrophic forgetting and preservation of positive transfer. This results in substantially reduced forgetting, with an 84% reduction in BWT relative to GPM on LayoutLMv2, while maintaining bounded subspace memory through incremental merging and achieving 5.4× lower storage than TRGP.
• We provide the first systematic evaluation of gradient projection methods for continual learning in visually rich document understanding, covering four benchmarks, two state-of-the-art multimodal backbones, and nine baselines, including recent parameter-efficient fine-tuning (PEFT)-based continual learning approaches.
• We present mechanistic analyses based on gradient spectrum decomposition and layer-wise reuse–interference dynamics, offering interpretable evidence of how selective gradient projection shapes optimization geometry across network depth in multimodal document encoders.

2. Related Work

2.1. Continual Learning Paradigms

Continual learning (CL), also referred to as lifelong learning, studies how models can acquire knowledge from a sequence of tasks while preserving performance on previously learned ones. A central challenge in CL is catastrophic forgetting [2], which arises when updates for new tasks interfere with representations learned earlier. Existing CL methods are commonly categorized into five paradigms [8]: regularization-based, replay-based, optimization-based, representation-based, and architecture-based approaches. Figure 2 illustrates continual learning methods taxonomy.

Figure 2. Continual learning methods taxonomy. Regularization-based: EWC [3], SI [9], MAS [10]. Replay-based: iCaRL [11], ER [12], DGR [13]. Optimization-based: GPM [5], CUBER [6], TRGP [7], OWM [14]. Representation-based: LUMP [15], MinRed [16]. Architecture-based: Piggyback [17], PackNet [18], PNN [19].

2.1.1. Regularization-Based Approaches

Regularization-based methods protect previously learned knowledge by constraining parameter updates. Elastic Weight Consolidation (EWC) [3] penalizes changes to parameters deemed important for past tasks using a Fisher Information-based approximation. Synaptic Intelligence (SI) [9] and Memory Aware Synapses (MAS) [10] similarly estimate parameter importance during training and restrict future updates.

2.1.2. Replay-Based Approaches

Replay-based methods mitigate forgetting by continuously feeding the model with data samples from previous tasks during training. Experience replay iCaRL [11], and ER [12] stores a subset of past samples, while generative replay DGR [13] synthesizes pseudo-samples using generative models. Feature replay variants operate in latent space to reduce storage costs. Despite strong empirical performance, replay-based approaches incur additional memory and computational overhead. In document processing applications, storing or regenerating sensitive data may violate privacy or governance constraints, and replay substantially increases training cost for large multimodal models.

2.1.3. Optimization-Based Approaches

Optimization-based methods explicitly modify training dynamics to reduce destructive gradient interference. GPM [5] constructs low-rank subspaces from historical gradients and enforces orthogonality between current gradients and these subspaces. Related approaches explore orthogonal gradient constraints or projection-based updates such as OWM [14], OGD [20], and AOP [21]. Although effective at preventing forgetting, strict projection increasingly constrains optimization as the number of tasks grows. Treating all past-task directions uniformly ignores the presence of transferable structure, which is common in structured domains such as visually rich document understanding.

Recent projection-based continual learning methods go beyond strict orthogonality by explicitly modeling task correlation. CUBER [6] and TRGP [7] characterize task similarity through gradient projection onto task-specific subspaces and distinguish between regimes of interference and transfer. To exploit correlated tasks, these approaches reuse frozen weights of selected old tasks via scaled weight projections within layer-wise trust regions while preserving other tasks through orthogonal constraints. Although effective, such methods operate primarily in the weight space. Recent work has explored parameter-efficient fine-tuning (PEFT) strategies for continual learning in large-scale models, including orthogonal or low-rank adapters in language and vision–language models (e.g., O-LoRA [22], GORP [23], SplitLoRA [24]). These methods constrain updates in parameter space, typically through task-specific adapters.

More broadly, the strategy of explicitly modifying the training objective to suppress harmful statistical dependencies during learning extends beyond continual learning. Related work in algorithmic fairness has explored conceptually analogous formulations, embedding differentiable regularizers that penalize unwanted statistical associations between model predictions and sensitive attributes directly within the training objective [25]. While the specific dependency being controlled differs, the principle of shaping optimization dynamics to mitigate harmful correlations represents a common thread across these domains.

2.1.4. Representation-Based Approaches

Representation-based CL methods aim to learn task-invariant or transferable features. Self-supervised learning and continual pre-training attempt to stabilize representations across tasks. Methods such as LUMP [15] and MinRed [16] improve representation robustness through interpolation or decorrelation strategies, while other approaches rely on feature distillation or alignment.

2.1.5. Architecture-Based Approaches

Architecture-based methods mitigate forgetting by modifying the model structure. Parameter allocation strategies assign separate parameters to different tasks Piggyback [17], Packnet [18], while modular networks and progressive architectures dynamically expand capacity PNN [19]. While offering strong task isolation, these methods introduce architectural complexity.

2.1.6. Summary and Positioning

AMD-Proj belongs to the optimization-based family of continual learning methods, while incorporating representation-aware insights. Unlike prior projection approaches that uniformly suppress past-task gradient directions, AMD-Proj distinguishes between interfering and reusable components, enabling selective gradients blocking and reuse. AMD-Proj is replay-free and preserves the original model architecture.

2.2. Document Understanding Models

2.2.1. Transformer-Based VrDU Models

LayoutLM [26] introduced multimodal pretraining by combining text with layout embeddings and visual features only on the fine-tuning phase. LayoutLMv2 [27] incorporated visual features via a ResNet-based backbone and cross-modality attention during pre-training. LayoutLMv3 [28] further unified text and image masking in a ViT architecture, improving representation quality for document parsing, key information extraction, and visual question answering.

2.2.2. Graph-Based Document Models

Graph-based approaches model documents as a graph of connected tokens and edges. Methods, such as LMGDoc [29] and Doc2Graph [30], represent tokens or text blocks as nodes connected by layout-aware edges, enabling structural and spatial reasoning.

2.3. Continual Learning in VrDU

Most CL methods have been developed for image classification benchmarks. Document understanding introduces additional challenges, including the interaction of text, layout, and vision, structured reasoning requirements, and sensitivity to spatial alignment. Existing CL methods do not explicitly preserve the multimodal fusion layers on which VrDU performance depends, motivating the need for gradient-level control mechanisms tailored to this domain.

3. Methodology

3.1. From Gradient Projection to AMD-Proj: Overview

A common strategy to mitigate catastrophic forgetting in continual learning is to project task gradients so as to avoid interfering with parameters that were important for previously learned tasks. Methods such as GPM achieve this by identifying a subspace associated with past tasks and enforcing orthogonality between new gradients and this subspace.

AMD-Proj builds on this line of work by introducing a selective and adaptive gradient modulation mechanism. Instead of treating all past gradient directions equally, AMD-Proj performs the following: (i) models task knowledge as a collection of low-dimensional gradient subspaces, (ii) evaluates the alignment between the current gradient and each stored subspace, and (iii) modulates the gradient differently depending on the strength of this alignment.

At a high level, AMD-Proj decomposes the current gradient into three components: directions that strongly align with previous tasks and should be protected (blocking), directions that moderately align and can be safely reused to promote transfer (reuse), and directions that are unrelated and left unchanged. By softly attenuating or amplifying these components, AMD-Proj achieves a controlled trade-off between stability and plasticity without enforcing hard orthogonality constraints. Figure 3 provides a schematic overview of the AMD-Proj framework.

Figure 3. Overview of AMD-Proj: Gradients are projected per layer using a learned gradient subspace memory to block interfering directions while reusing aligned ones. The subspace memory is constructed via truncated SVD of historical gradients and updated after each task.

3.2. Problem Definition and Continual Learning Setting

We study continual learning in the context of visually rich document understanding (VrDU), where a model is required to learn a sequence of document analysis tasks without catastrophic forgetting of previously acquired knowledge.

Formally, we consider a sequence of T tasks $\{{\mathcal{T}}_1,{\mathcal{T}}_2,\dots ,{\mathcal{T}}_T\}$. Each task ${\mathcal{T}}_t$ is associated with a dataset ${\mathcal{D}}_t={\left\{(x_i^t,y_i^t)\right\}}_{i=1}^{N_t}$, where $x_i^t$ denotes a document sample (including text, layout, and visual features), and $y_i^t$ represents the corresponding structured prediction labels.

We adopt a task-incremental learning (TIL) setting, characterized by the following properties:

• Tasks arrive sequentially, and training data from previous tasks are not accessible once a task has been completed.
• Label spaces are disjoint across tasks.
• Task identity is known at both training and inference time.

Let $f_{{\theta }_t}(·)$ denote the model after learning task ${\mathcal{T}}_t$, parameterized by ${\theta }_t$. At each step t, the model is optimized only using samples from ${\mathcal{D}}_t$, while aiming to preserve performance on all previously seen tasks $\{{\mathcal{T}}_1,\dots ,{\mathcal{T}}_{t-1}\}$. The continual learning objective can thus be expressed as:

$$\underset{{\theta }_t}{min}{\mathbb{E}}_{(x,y)\sim {\mathcal{D}}_t}\left[\mathcal{L}\left(f_{{\theta }_t}\left(x\right),y\right)\right],\mathrm{s}.\mathrm{t}.{\theta }_t\mathrm{retains}\mathrm{knowledge}\mathrm{of}{\mathcal{T}}_{<t},$$

(1)

where $\mathcal{L}(·)$ denotes the task-specific loss function.

3.3. Gradient Subspace Modeling

Continual learning failures in deep neural networks are largely driven by destructive gradient interference, where parameter updates required for a new task overwrite directions that were essential for previously learned tasks. To explicitly reason about this phenomenon, we model task knowledge through low-dimensional subspaces spanned by historical gradients.

• Gradient snapshots:

Let $g_{\mathsf{\ell }}^{(t,i)}\in {\mathbb{R}}^{d_{\mathsf{\ell }}}$ denote the gradient of layer ℓ computed from the i-th mini-batch of task ${\mathcal{T}}_t$. For each task, we collect a small set of gradient snapshots and stack them into a matrix

$${\mathbf{G}}_{\mathsf{\ell }}^{\left(t\right)}=\left[g_{\mathsf{\ell }}^{(t,1)},\dots ,g_{\mathsf{\ell }}^{(t,n)}\right]\in {\mathbb{R}}^{d_{\mathsf{\ell }}\times n},$$

(2)

where n is the number of sampled batches. To improve numerical stability, gradients are centered by subtracting their mean.

• Low-rank subspace extraction:

We assume that task-relevant gradients lie on a low-dimensional manifold. Accordingly, we extract a compact basis by computing the truncated singular value decomposition (SVD):

$${\mathbf{G}}_{\mathsf{\ell }}^{\left(t\right)}={\mathbf{U}}_{\mathsf{\ell }}^{\left(t\right)}{\mathbf{\Sigma }}_{\mathsf{\ell }}^{\left(t\right)}{\mathbf{V}}_{\mathsf{\ell }}^{\left(t\right)\top }.$$

(3)

The number of retained basis vectors $k_{\mathsf{\ell }}$ is chosen such that a fixed proportion E of the total energy is preserved:

$${\displaystyle \frac{{\sum }_{i=1}^{k_{\mathsf{\ell }}}{\sigma }_i^2}{{\sum }_j{\sigma }_j^2}}\ge E.$$

(4)

The resulting matrix ${\mathbf{U}}_{\mathsf{\ell }}^{\left(t\right)}\in {\mathbb{R}}^{d_{\mathsf{\ell }}\times k_{\mathsf{\ell }}}$ defines an orthonormal basis capturing the dominant gradient directions for task ${\mathcal{T}}_t$ at layer ℓ.

• Gradient subspace memory and management:

For each layer ℓ, we maintain a memory of gradient subspaces accumulated over tasks,

$${\mathcal{M}}_{\mathsf{\ell }}=\{{\mathbf{U}}_{\mathsf{\ell }}^{\left(1\right)},{\mathbf{U}}_{\mathsf{\ell }}^{\left(2\right)},\dots \}.$$

(5)

To ensure bounded storage, newly extracted subspaces are compared against existing ones using projection-based similarity measures. Highly similar subspaces are merged via QR re-orthogonalization to preserve orthonormality, while dissimilar subspaces are appended up to a fixed per-layer budget. If the budget is exceeded, older or lower-utility subspaces are discarded.

3.4. Adaptive Gradient Projection with Reuse and Blocking (AMD-Proj)

Building upon the gradient subspace formulation introduced in Section 3.3, we now define Adaptive Gradient Projection (AMD-Proj), a selective projection mechanism used to modulate gradient updates during continual learning.

• Similarity-based subspace scoring:

Let $g_{\mathsf{\ell }}\in {\mathbb{R}}^{d_{\mathsf{\ell }}}$ denote the gradient of layer ℓ computed during training on the current task. Given the memory of subspaces ${\mathcal{M}}_{\mathsf{\ell }}=\left\{{\mathbf{U}}_{\mathsf{\ell }}^{\left(j\right)}\right\}$, we calculate the alignment between $g_{\mathsf{\ell }}$ and each stored subspace ${\mathbf{U}}_{\mathsf{\ell }}^{\left(j\right)}$ using the ${\mathsf{\ell }}_2$ norm:

$$s_j={∥{\mathbf{U}}_{\mathsf{\ell }}^{\left(j\right)\top }{\widehat{g}}_{\mathsf{\ell }}∥}_2,{\widehat{g}}_{\mathsf{\ell }}={\displaystyle \frac{g_{\mathsf{\ell }}}{\parallel g_{\mathsf{\ell }}{\parallel }_2+ϵ}}.$$

(6)

By normalizing the gradient we ensure that focus is on geometric orientation instead of the magnitude.

• Reuse and blocking sets:

To obtain the two sets of subspaces, reuse and blocking, we use two thresholds ${\tau }_{\mathrm{reuse}}$, and ${\tau }_{\mathrm{block}}$

$$\begin{array}{c}{\mathcal{B}}_{\mathsf{\ell }}=\{j\mid s_j\ge {\tau }_{\mathrm{block}}\},\end{array}$$

(7)

$$\begin{array}{c}{\mathcal{R}}_{\mathsf{\ell }}=\{j\mid {\tau }_{\mathrm{reuse}}\le s_j<{\tau }_{\mathrm{block}}\},\end{array}$$

(8)

with ${\tau }_{\mathrm{block}}>{\tau }_{\mathrm{reuse}}$. Subspaces in ${\mathcal{B}}_{\mathsf{\ell }}$ correspond to strongly aligned directions, while those in ${\mathcal{R}}_{\mathsf{\ell }}$ correspond moderately aligned directions. To limit computational overhead we keep only the top-k subspaces in each set.

The classification of gradient directions into blocking, reuse, and neutral sets is governed by two design choices: the alignment score and the use of hard thresholds.

Alignment score: The score $s_j$ defined in Equation (6) measures how much of the normalized gradient ${\widehat{g}}_{\mathsf{\ell }}$ lies within the column space of the j-th stored subspace basis ${\mathbf{U}}_{\mathsf{\ell }}^{\left(j\right)}$. Because the basis vectors are orthonormal, this quantity equals the ${\mathsf{\ell }}_2$ norm of the projection of ${\widehat{g}}_{\mathsf{\ell }}$ onto that subspace and takes values in $[0,1]$: a value of 1 indicates full alignment, while 0 indicates orthogonality. Normalizing the gradient ensures that classification depends purely on geometric orientation rather than magnitude, which can vary significantly across layers and training steps. This decoupling is essential: a large-magnitude gradient that is orthogonal to past-task subspaces should not be suppressed, while a small but highly aligned gradient should not pass unmodified. Because ${\widehat{g}}_{\mathsf{\ell }}$ is normalized, this score is also equivalent to an aggregated cosine similarity over the subspace basis, while remaining directly compatible with orthogonal projection operators.

Hard thresholds vs. soft gating: A natural alternative is to use a soft gating function (e.g., a sigmoid applied to $s_j$). We instead adopt hard thresholds for three reasons. First, they yield discrete and interpretable assignments with clear geometric meaning, enabling well-defined projection operators $P_B$ and $P_R$. Second, soft gating would introduce gradient flow through the classification mechanism itself, creating an additional, continuously varying control signal that interacts with the parameter update and can destabilize training dynamics. Third, the ablation results reported in Section 4.9 show that AMD-Proj is robust to moderate variations in ${\tau }_{\mathrm{reuse}}$ and ${\tau }_{\mathrm{block}}$, indicating that precise calibration is not required and that hard thresholds do not introduce brittleness in practice.

Threshold ordering: The constraint ${\tau }_{\mathrm{block}}>{\tau }_{\mathrm{reuse}}$ ensures that the blocking and reuse sets are disjoint and that a neutral region exists where gradient components are left unchanged. Directions with low alignment encode new, task-specific information and should be preserved to maintain plasticity. The resulting three-way partition—block, reuse, and neutral—reflects distinct geometric relationships between the current gradient and stored knowledge, each requiring different treatment.

Mutual orthogonality of projectors: The construction in Equations (9) and (10), which deflates the reuse subspace against the blocking projector before orthonormalization, guarantees that

$$P_B{P}_R=0.$$

(9)

This ensures that blocking and reuse act on disjoint subspaces of the gradient. Without this property, a direction could simultaneously receive attenuation and reinforcement, leading to ambiguous and unstable updates. The deflation step eliminates this possibility by design.

• Orthogonalized projection operators:

To prevent overlapping projections and uncontrolled gradient amplification, reuse directions are constrained to lie in the orthogonal complement of the blocking subspace. We first construct an orthonormal basis for the blocking subspace:

$${\mathbf{U}}_{\mathcal{B}}=\mathrm{orth}\left({\left[{\mathbf{U}}_{\mathsf{\ell }}^{\left(j\right)}\right]}_{j\in {\mathcal{B}}_{\mathsf{\ell }}}\right),{\mathbf{P}}_{\mathcal{B}}={\mathbf{U}}_{\mathcal{B}}{\mathbf{U}}_{\mathcal{B}}^{\top }.$$

(10)

Reuse subspaces are then deflated against ${\mathbf{P}}_{\mathcal{B}}$ prior to orthonormalization:

$${\mathbf{U}}_{\mathcal{R}}=\mathrm{orth}\left((\mathbf{I}-{\mathbf{P}}_{\mathcal{B}}){\left[{\mathbf{U}}_{\mathsf{\ell }}^{\left(j\right)}\right]}_{j\in {\mathcal{R}}_{\mathsf{\ell }}}\right),{\mathbf{P}}_{\mathcal{R}}={\mathbf{U}}_{\mathcal{R}}{\mathbf{U}}_{\mathcal{R}}^{\top }.$$

(11)

This construction guarantees that the blocking and reuse projectors are mutually orthogonal.

• Adaptive gradient modulation:

The gradient applied to layer ℓ is defined as:

$${\tilde{g}}_{\mathsf{\ell }}=g_{\mathsf{\ell }}-{\beta }_{\mathrm{block}}{\mathbf{P}}_{\mathcal{B}}{g}_{\mathsf{\ell }}+{\beta }_{\mathrm{reuse}}{\mathbf{P}}_{\mathcal{R}}{g}_{\mathsf{\ell }},$$

(12)

where ${\beta }_{\mathrm{reuse}}>{\beta }_{\mathrm{block}}\ge 0$.

This update reduces the effect of gradient components aligned with blocking directions and reinforces components aligned with reusable directions, yielding a controlled trade-off between stability and plasticity without enforcing hard orthogonality. The complete training procedure is summarized in Algorithm 1.

Algorithm 1 Continual Learning with Adaptive Gradient Projection (AMD-Proj)

Require:  Task sequence $\{{\mathcal{D}}_1,\dots ,{\mathcal{D}}_T\}$, model $f_{\theta }$, learning rate $\eta$ Ensure:  Continually trained model parameters $\theta$   1: Initialize gradient subspace memory ${\mathcal{M}}_{\mathsf{\ell }}\leftarrow Ø$ for all layers ℓ   2: for $t=1$ to T  do   3:       Initialize optimizer for task ${\mathcal{D}}_t$   4:       for each training epoch do   5:             for each minibatch $(x,y)\sim {\mathcal{D}}_t$ do   6:                   Compute loss $\mathcal{L}(f_{\theta }\left(x\right),y)$   7:                   Backpropagate to obtain gradients $\left\{g_{\mathsf{\ell }}\right\}$   8:                   if $t>1$ then   9:                        for each layer ℓ do 10:                              Compute similarity scores $s_j$ for all ${\mathbf{U}}_{\mathsf{\ell }}^{\left(j\right)}\in {\mathcal{M}}_{\mathsf{\ell }}$ 11:                              Identify blocking and reuse sets $({\mathcal{B}}_{\mathsf{\ell }},{\mathcal{R}}_{\mathsf{\ell }})$ 12:                              Construct projection operators $({\mathbf{P}}_{\mathcal{B}},{\mathbf{P}}_{\mathcal{R}})$ 13:                              Modulate gradients using Equation (12) 14:                         end for 15:                   end if 16:                   Update parameters: $\theta \leftarrow \theta -\eta \left\{g_{\mathsf{\ell }}\right\}$ 17:             end for 18:        end for 19:        Gradient Subspace Update 20:        for each layer ℓ do 21:              Sample gradient snapshots from ${\mathcal{D}}_t$ 22:              Compute truncated SVD and retain basis preserving energy E 23:              Merge new basis into memory ${\mathcal{M}}_{\mathsf{\ell }}$ under budget 24:        end for 25: end for

3.5. Computational Complexity

AMD-Proj introduces three computational costs beyond standard fine-tuning. At each training step, computing the alignment score for layer ℓ requires projecting the normalized gradient ${\widehat{g}}_{\mathsf{\ell }}\in {\mathbb{R}}^d$ onto each stored subspace basis $U_{\mathsf{\ell }}^{\left(j\right)}\in {\mathbb{R}}^{d\times r}$, costing $\mathcal{O}(r·d)$ per subspace and $\mathcal{O}(k·r·d)$ per layer, where k is the number of retained subspaces and r is the truncation rank. Summed over L layers, the per-step projection overhead is $\mathcal{O}(L·k·r·d)$, which is negligible relative to the forward-backward pass when k and r are small.

At the end of each task, truncated SVD is applied to the accumulated gradient matrix to extract the top-r singular vectors, costing $\mathcal{O}(d·r^2)$ per layer via randomized SVD, and $\mathcal{O}(L·d·r^2)$ in total—a one-time post-task cost. The subspace merging step deflates the new basis against the stored one and re-orthogonalizes, costing $\mathcal{O}(d·r^2)$ per layer.

Crucially, merging bounds the total subspace storage at $\mathcal{O}(L·r·d)$ regardless of the number of tasks T, in contrast to TRGP which retains separate per-task subspaces and incurs $\mathcal{O}(T·L·r·d)$ storage growing linearly with T.

3.6. Training Algorithm

Algorithm 1 presents the complete training procedure for AMD-Proj. At each training step, gradients are first computed via standard backpropagation, then modulated through the blocking and reuse projection operators derived from the stored subspace memory. After completing each task, gradient snapshots are collected and used to update the memory through truncated SVD and incremental merging.

4. Results

4.1. Experimental Setup

4.1.1. Datasets and Task Definition

We evaluate our method on a suite of widely used VrDU benchmarks, each representing a distinct document type and semantic structure. In our continual learning setting, each dataset is treated as an individual task, and tasks are learned sequentially without access to data from previous tasks.

FUNSD [31] is a form understanding dataset composed of scanned forms annotated with semantic labels such as question, answer, header, and other, it contains 199 documents, split into 50 for testing and 149 for training.

SROIE [32] comprises scanned receipts for receipt information extraction, where the objective is to identify and classify entities such as company name, address, date, and total amount. It is split into 626 training and 347 testing samples.

CORD [33] consists of 1000 Indonesian receipt images with fine-grained semantic annotations, including key fields such as menu-item and total-amount. CORD is split into 800 samples for training, 100 for testing, and 100 for validation.

BuDDIE [34] is a business document dataset comprising diverse document types such as periodical reports, amendment documents. It contains 1172 documents for training, 161 for validation, and 332 for testing.

Following standard practice in continual learning, each dataset defines a separate task with a disjoint label space. The model is trained sequentially on the tasks and evaluated on all previously seen tasks after learning each new task.

Unless stated otherwise, we adopt the following default task order:

$$\mathrm{SROIE}\to \mathrm{FUNSD}\to \mathrm{CORD}\to \mathrm{BuDDIE}.$$

To assess robustness, we further analyze the impact of alternative task orders in Section 4.8.

We consider a task-incremental learning setting where each dataset constitutes a separate task with disjoint label spaces.

4.1.2. Models and Architectures

To evaluate the generality of our approach across different multimodal learning paradigms, we conduct experiments on two state-of-the-art document understanding architectures.

LayoutLMv2 is a pretrained multimodal transformer that jointly models textual content, two-dimensional layout information, and visual features extracted from document images. It has been widely adopted as a strong baseline for visually rich document understanding tasks and serves as a representative of early multimodal document transformers.

LayoutLMv3 extends this paradigm with a more tightly integrated vision–language pretraining strategy, unifying text and image patch embeddings within a single transformer architecture. Its increased depth and capacity make it a substantially stronger but also more challenging backbone in continual learning settings.

4.1.3. Evaluation Metrics

To evaluate AMD-Proj, we use the following metrics:

Average Accuracy (ACC): measures the mean performance across all tasks.

$$\mathrm{ACC}={\displaystyle \frac{1}{T}}\sum _{j=1}^T{A}_{T,j}.$$

Backward Transfer (BWT): evaluates the average influence of learning the k-th task on all old tasks.

$$\mathrm{BWT}={\displaystyle \frac{1}{T-1}}\sum _{j=1}^{T-1}\left(A_{T,j}-A_{j,j}\right).$$

In addition to accuracy-based metrics, we also report F1-based counterparts of ACC and BWT.

We report both accuracy-based and F1-based metrics due to class imbalance in document understanding tasks.

4.2. Implementation Details and Experimental Settings

4.2.1. Computational Environment

All experiments were conducted on a single workstation equipped with an NVIDIA RTX 4070 GPU (8 GB VRAM). Models were implemented in PyTorch 2.6 and trained using mixed precision to reduce memory consumption. Due to GPU memory constraints, all experiments were performed with a batch size of 1. No multi-GPU training or distributed optimization was used.

4.2.2. Optimization and Training Protocol

For all continual learning methods, we use the AdamW optimizer with a learning rate of $5\times {10}^{-5}$. Each task is trained for three epochs before moving to the next task. Gradient norm clipping with a maximum norm of 1.0 is applied to stabilize training. Once a task is completed, its training data are discarded and never reused, in accordance with the task-incremental learning setting.

4.2.3. AMD-Proj Hyperparameters

The proposed AMD-Proj method uses the following hyperparameter configuration, which is fixed across all experiments unless stated otherwise:

• Energy threshold for subspace construction: $E=0.90$
• Reuse similarity threshold: ${\tau }_{\mathrm{reuse}}=0.10$
• Blocking similarity threshold: ${\tau }_{\mathrm{block}}=0.25$
• Blocking strength: ${\beta }_{\mathrm{block}}=0.2$
• Reuse strength: ${\beta }_{\mathrm{reuse}}=0.3$
• Top-k reused subspaces: $k_{\mathrm{reuse}}=1$
• Top-k blocked subspaces: $k_{\mathrm{block}}=1$

Gradient subspaces are constructed by sampling gradients from up to six mini-batches after completing each task. To ensure numerical stability and bounded memory usage, gradient vectors are optionally downsampled before singular value decomposition.

4.2.4. Baseline Hyperparameters and Configurations

We compare AMD-Proj against several established continual learning baselines that represent different strategies for mitigating catastrophic forgetting. For reproducibility, we report the key hyperparameters used for each continual learning baseline. All methods share the same optimizer, learning rate, batch size, and number of epochs as described above.

Sequential Fine-Tuning (SeqFT): trains the model on each task sequentially without any explicit forgetting mitigation and serves as a lower-bound baseline.

Elastic Weight Consolidation (EWC): EWC is implemented using a diagonal Fisher Information Matrix estimated after completing each task. The regularization strength is set to a fixed value ${\lambda }_{\mathrm{EWC}}$, which balances stability and plasticity. In all experiments, we use:

• Fisher matrix: diagonal approximation
• Regularization coefficient: ${\lambda }_{\mathrm{EWC}}=1000$
• Fisher estimation: computed on the task validation set

Learning without Forgetting (LwF): LwF is implemented using logit-level distillation from the model trained on the previous task. Since tasks are associated with disjoint label spaces, distillation is performed in a head-specific manner: for each previously learned task, student and teacher logits are computed using the same task-specific output head, ensuring matched output dimensionality. Distillation is applied at the token level and restricted to valid text tokens via masking, preventing supervision on task-irrelevant labels. We use the following standard configuration:

• Distillation temperature: $T=2$;
• Distillation weight: ${\lambda }_{\mathrm{LwF}}=1.0$;
• Loss function: $\mathcal{L}={\mathcal{L}}_{CE}+{\lambda }_{\mathrm{LwF}}{\mathcal{L}}_{KD}$.

Gradient Projection Memory (GPM): We follow the original formulation and use:

• Energy threshold for subspace construction: $E=0.90$;
• Number of gradient snapshots per task: $n=6$;
• Projection type: hard orthogonal projection;
• Memory budget: unbounded (one subspace per task and layer).

Orthogonal Weights Modification (OWM): We follow the standard OWM formulation and use:

• Projection update: recursive orthogonal projection based on input activations;
• Stability–plasticity coefficient: $\alpha =1.0$;
• Projection type: hard orthogonal projection;
• Memory budget: unbounded (one projection matrix per protected layer).

CUBER: We follow the original formulation of [6] and use:

• Subspace construction energy threshold: $E=0.90$;
• Number of gradient snapshots per task: $n=6$;
• Trust region scaling: default as reported in the original paper;
• Memory budget: unbounded (one subspace per task and layer).

TRGP: We follow the original formulation of [7] and use:

• Subspace construction energy threshold: $E=0.90$;
• Number of gradient snapshots per task: $n=6$;
• Trust region radius: default as reported in the original paper;
• Memory budget: unbounded (separate per-task subspaces retained incurring $\mathcal{O}(T·L·r·d)$ storage).

O-LoRA: O-LoRA is implemented following [22] with a frozen backbone and task-specific LoRA adapters injected into the query and value projection matrices of each Transformer layer. We use:

• LoRA rank: $r=4$;
• LoRA scaling factor: ${\alpha }_{\mathrm{LoRA}}=8$;
• Trainable parameters: approximately 2% of total model parameters per task;
• Backbone: fully frozen during adapter training.

SplitLoRA: SplitLoRA is implemented following [24] with a frozen backbone and task-specific low-rank adapters. We use:

• Optimal rank selection: energy-based truncation via SVD;
• LoRA scaling factor: ${\alpha }_{\mathrm{LoRA}}=8$;
• Trainable parameters: approximately 2% of total model parameters per task;
• Backbone: fully frozen during adapter training.

All baseline hyperparameters are fixed across tasks and architectures and are chosen based on commonly reported values in the respective original publications. No task-specific tuning is performed.

4.3. Results on LayoutLMv2

Table 1 presents continual learning methods performance on LayoutLMv2 in terms of average accuracy (ACC) and backward transfer (BWT), along with their F1-based counterparts. Sequential fine-tuning exhibits severe catastrophic forgetting, as reflected by strongly negative BWT values. Regularization-based methods, such as EWC and LwF, partially mitigate forgetting, with LwF achieving the highest average ACC.

Table 1. Continual learning performance on LayoutLMv2. We report average accuracy (ACC) and backward transfer (BWT), along with their F1-based counterparts.

Method	ACC ↑	BWT ↑	F1-ACC ↑	F1-BWT ↑
SeqFT	0.7253	$-0.2558$	0.3523	$-0.5192$
EWC	0.4725	$-0.0496$	0.0426	$-0.2704$
LwF	0.8284	$-0.1069$	0.5566	$-0.2266$
GPM	0.7677	$-0.2060$	0.4984	$-0.3399$
OWM	0.7725	$-0.2490$	0.3414	$-0.5109$
CUBER	0.4565	$-0.0491$	0.0426	$-0.2690$
TRGP	0.4284	$-0.0386$	0.2992	$-0.1438$
O-LoRA†	0.4675	$-0.0265$	0.0426	$-0.0327$
SplitLoRA†	0.4675	$-0.1096$	0.0426	$-0.1327$
AMD-Proj (Ours)	0.7757	$-\mathbf{0}.\mathbf{0332}$	0.5736	$-\mathbf{0}.\mathbf{0178}$

^† Frozen backbone with LoRA adapters (≈2% trainable parameters per task). Results are not directly comparable to full fine-tuning methods. Bold values indicate the best result in each column. ↑ indicates higher is better.

AMD-Proj achieves the least negative backward transfer across both accuracy- and F1-based metrics. In particular, BWT improves from $-0.2060$ for GPM and $-0.2558$ for sequential fine-tuning to $-0.0332$ with AMD-Proj, which indicates stronger retention of previously learned tasks. Notably, AMD-Proj attains the least negative backward transfer and the highest F1-based stability, highlighting its ability to balance stability and plasticity.

CUBER and TRGP achieve near-zero BWT but collapse in ACC (0.4565 and 0.4284), suggesting that their trust-region projection is overly conservative in this setting. AMD-Proj achieves higher ACC (0.7757) while maintaining the least negative BWT ($-0.0332$), outperforming both methods on forgetting control and predictive performance simultaneously. O-LoRA and SplitLoRA achieve near-zero BWT due to their frozen backbone but collapse in F1-ACC (0.0426), indicating that adapter-only updates are insufficient for effective adaptation in this setting.

4.4. Results on LayoutLMv3

Table 2 presents continual learning methods performance on LayoutLMv3.

Table 2. Continual learning performance on LayoutLMv3. We report average accuracy (ACC), backward transfer (BWT), and their F1-based counterparts.

Method	ACC ↑	BWT ↑	F1-ACC ↑	F1-BWT ↑
SeqFT	0.6879	$-0.2959$	0.4883	$-0.3318$
EWC	0.6454	$-0.3467$	0.4398	$-0.3856$
LwF	0.8495	$-0.1070$	0.6286	$-0.2484$
GPM	0.7035	$-0.2672$	0.5047	$-0.3164$
OWM	0.6881	$-0.2827$	0.5136	$-0.2977$
CUBER	0.6366	$-0.3622$	0.4435	$-0.4162$
TRGP	0.6548	$-0.3312$	0.4285	$-0.4164$
O-LoRA†	0.4649	$-0.0232$	0.0422	$-0.0296$
SplitLoRA†	0.4649	$-0.0002$	0.0422	$-0.0005$
AMD-Proj (Ours)	0.8514	$-\mathbf{0}.\mathbf{0918}$	0.6240	$-\mathbf{0}.\mathbf{2130}$

^† Frozen backbone with LoRA adapters ($\approx 2\%$ trainable parameters). Results are not directly comparable to full fine-tuning methods. Bold values indicate the best result in each column. ↑ indicates higher is better.

Sequential fine-tuning, EWC, OWM, and GPM still suffer from significant forgetting, as indicated by strongly negative BWT values. LwF attains higher average accuracy and relatively mild forgetting.

AMD-Proj achieves the highest accuracy while maintaining backward transfer values consistently closer to zero than sequential fine-tuning, EWC, and GPM. In addition, it achieves the second highest F1-ACC and shows clear improvements in F1-BWT demonstrating a more favorable balance between stability and plasticity. Overall, these results suggest that similarity-aware gradient projection remains effective even in more integrated multimodal architectures where selective gradients become more challenging.

CUBER and TRGP show substantially more negative BWT ($-0.3622$ and $-0.3312$) than AMD-Proj, confirming that task-level projection methods struggle more in integrated architectures.

4.5. Comparison Across Architectures

On LayoutLMv2, sequential fine-tuning and regularization-based approaches, such as EWC, exhibit pronounced forgetting, particularly on earlier datasets, including FUNSD and CORD. In contrast, AMD-Proj generally preserves performance on early and intermediate tasks by maintaining backward transfer values closer to zero, indicating improved retention of previously acquired knowledge. While AMD-Proj does not uniformly maximize final-task accuracy across all datasets, most notably on FUNSD, it achieves a more stable performance profile overall, as reflected in the backward transfer and F1-based stability metrics reported in Table 1.

On LayoutLMv3, AMD-Proj’s BWT is more negative, reflecting the greater difficulty of gradient decomposition in more tightly integrated architectures, despite achieving higher absolute accuracy due to the stronger backbone. In this setting, LwF achieves strong final-task accuracy and competitive performance on earlier tasks.

Table 3 reports accuracy on each dataset after completing training on the final task. Sequential fine-tuning and regularization-based methods (EWC) suffer from severe forgetting, particularly on earlier tasks, such as FUNSD and CORD. Subspace-based approaches (GPM and AMD-Proj) better preserve prior knowledge, AMD-Proj prioritizes stability at the gradient-interference level, which can trade off task-specific head specialization in highly structured datasets, such as FUNSD. In contrast, LwF benefits from direct output-space preservation, which is particularly effective when early tasks are small and semantically homogeneous.

Table 3. Accuracy after the last trained task for each dataset. Columns follow the task order SROIE → FUNSD → CORD → BuDDIE.

Model & Method	SROIE	FUNSD	CORD	BuDDIE
LayoutLMv2
+ SeqFT	0.8933	0.5239	0.9032	0.9380
+ EWC	0.8839	0.2750	0.0014	0.7669
+ LwF	0.9686	0.6006	0.9442	0.9360
+ GPM	0.9055	0.5591	0.9552	0.9344
+ CUBER	0.8467	0.2750	0.0042	0.7439
+ TRGP	0.8780	0.7466	0.0823	0.0066
+ O-LoRA†	0.8467	0.2750	0.0042	0.7439
+ SplitLoRA†	0.8467	0.2750	0.0042	0.7439
+ AMD-Proj (Ours)	0.9880	0.2755	0.9581	0.9336
LayoutLMv3
+ SeqFT	0.7060	0.5064	0.8736	0.9329
+ EWC	0.7119	0.5100	0.8822	0.9256
+ LwF	0.8822	0.6253	0.9171	0.9206
+ GPM	0.7597	0.5586	0.9259	0.9371
+ CUBER	0.3739	0.3512	0.8779	0.9432
+ TRGP	0.4438	0.3710	0.8651	0.9393
+ O-LoRA†	0.8467	0.2654	0.0042	0.7433
+ SplitLoRA†	0.8467	0.2654	0.0042	0.7433
+ AMD-Proj (Ours)	0.8918	0.4804	0.9278	0.9296

^† Frozen backbone with LoRA adapters ($\approx 2\%$ trainable parameters per task). Results are not directly comparable to full fine-tuning methods. Bold values indicate the best result in each column.

Across both architectures, PEFT-based methods (O-LoRA and SplitLoRA) show near-zero BWT, reflecting the stability of frozen backbones. However, their F1-ACC collapses to approximately $4\%$ on both LayoutLMv2 and LayoutLMv3, indicating that adapter-only updates are insufficient for effective task adaptation in visually rich document understanding. Unlike natural language tasks where PEFT methods have shown strong continual learning performance, VrDU requires updating the full multimodal encoder to capture the tight coupling of textual, layout, and visual representations. These results are, therefore, not directly comparable to full fine-tuning methods and establish a lower bound on performance under strict memory constraints.

4.6. Computational Cost and Scalability Analysis

Table 4 reports wall-clock training time, peak GPU memory, and cumulative subspace storage across all four tasks for both backbones. On LayoutLMv3, AMD-Proj completes training in 1490 s total, which is faster than GPM (2471 s), TRGP (2249 s), and LwF (2212 s), confirming that the per-step projection overhead remains negligible in practice.

Table 4. Computational cost and memory usage on LayoutLMv3 (top) and LayoutLMv2 (bottom) for the four-task benchmark (SROIE→FUNSD→CORD→BuDDIE). T1–T4: wall-clock training time (seconds) per task. Total: cumulative training time across all four tasks. Peak GPU: maximum GPU memory allocated during training (MB). Sub@T1–T4: cumulative subspace storage (MB) after each task; “—” indicates no subspace is stored.

Method	Training Time (s/task)				Total (s)	Peak GPU (MB)	Subspace Memory (MB)
	T1	T2	T3	T4			Sub@T1	Sub@T2	Sub@T3	Sub@T4
LayoutLMv3
SeqFT	424	161	643	883	2112	3401	—	—	—	—
EWC	448	172	671	918	2209	3401	—	—	—	—
LwF	388	146	667	1011	2212	3986	—	—	—	—
GPM	600	232	704	935	2471	3401	59.2	148.6	148.6	148.6
OWM	512	198	731	952	2393	3401	71.4	163.2	168.7	172.3
CUBER	455	177	765	1069	2466	3401	—	—	—	—
TRGP	479	160	655	955	2249	3401	78.0	248.6	394.5	692.1
O-LoRA†	308	105	468	737	1618	1823	2.4	4.7	7.1	9.4
SplitLoRA†	317	115	451	674	1558	1867	58.1	59.6	61.0	62.5
AMD-Proj (Ours)	359	115	407	629	1490	4486	—	51.3	81.1	127.7
LayoutLMv2
SeqFT	376	101	411	643	1531	4222	—	—	—	—
EWC	408	119	428	667	1622	4222	—	—	—	—
LwF	363	104	317	458	1242	5064	—	—	—	—
GPM	698	244	305	406	1653	4222	61.8	157.9	157.9	157.9
OWM	574	208	376	518	1676	4222	85.6	192.4	198.1	203.7
CUBER	430	115	483	1000	2027	4222	—	—	—	—
TRGP	402	125	463	696	1687	4222	106.8	348.3	508.6	771.1
O-LoRA†	320	154	764	1302	2540	1720	9.6	19.2	28.8	38.5
SplitLoRA†	274	123	495	870	1761	2013	348.9	354.9	360.9	366.9
AMD-Proj (Ours)	362	79	277	398	1116	5845	—	18.3	79.9	141.4

^† Frozen backbone with LoRA adapters ($\approx 2\%$ trainable parameters per task). EWC and OWM profiling data are estimated; see text. Bold values indicate the best result in each column.

Peak GPU memory (4486 MB) is higher than PEFT methods due to full backbone fine-tuning, but this is expected since O-LoRA and SplitLoRA freeze the backbone entirely and, therefore, cannot serve as direct comparisons for memory-constrained full fine-tuning. The most significant result concerns subspace storage: AMD-Proj’s cumulative memory grows sub-linearly across tasks, reaching 127.7 MB after four tasks on LayoutLMv3 compared to 692.1 MB for TRGP—a 5.4× reduction—and 141.4 MB versus 771.1 MB on LayoutLMv2—a 5.5× reduction.

This compression is a direct consequence of the incremental merging step described in Section 3.5, which collapses per-task bases into a single consolidated subspace rather than retaining separate per-task projections as TRGP does. The sub-linear growth pattern ($51.3\to 81.1\to 127.7$ MB across T2–T4 on LayoutLMv3) confirms the theoretical $\mathcal{O}(L·r·d)$ storage bound established in Section 3.5, demonstrating that AMD-Proj scales gracefully to longer task sequences without unbounded memory accumulation.

4.7. Explainability and Mechanistic Analysis

4.7.1. Gradient Subspace Spectrum Analysis

To analyze the internal mechanisms underlying AMD-Proj, we examine the spectral structure of learned parameter updates. Specifically, we perform singular value decomposition (SVD) of the output projection matrices in selected Transformer layers and report cumulative explained variance across singular directions.

Figure 4 compares the SVD spectra obtained after training with sequential fine-tuning, GPM, LwF, and AMD-Proj. Across early, intermediate, and late layers, AMD-Proj consistently shows faster saturation of cumulative explained variance, indicating that gradient updates concentrate in a smaller number of dominant directions.

Figure 4. SVD spectrum of output projection matrices for selected Transformer layers.

In contrast, SeqFT displays a more diffuse spectrum, which reflects unconstrained parameter drift, while GPM enforces overly restrictive projections that reduce the expressiveness of learned updates. LwF shows an intermediate behavior but does not explicitly control the internal geometry of gradient updates. These observations suggest that AMD-Proj maintains a gradient subspace that is both compact and expressive, offering a geometric interpretation for its improved backward transfer.

4.7.2. Reuse vs. Block Gradient Dynamics

To further characterize the behavior of AMD-Proj, we analyze how gradients are reused or blocked across the model layers throughout continual learning. Figure 5 summarizes aggregated reuse and block counts, as well as the reuse ratio per layer.

Figure 5. Reuse vs. block gradient dynamics in AMD-Proj across Transformer layers. (a) Aggregated reuse and block frequency per Transformer layer, shown as a heatmap where warmer colors indicate higher activity. (b) Absolute block (red) versus reuse (green) counts per layer. (c) Reuse ratio per layer, computed as $\mathrm{reuse}/(\mathrm{reuse}+\mathrm{block})$, showing the proportion of gradient directions that are reinforced relative to the total number of modulated directions at each layer.

A clear layer-wise pattern emerges. Early layers are dominated by gradient blocking, which indicates conservative updates that preserve shared low-level multimodal representations. Reuse becomes more prevalent in intermediate layers. In later layers, blocking again becomes dominant, reflecting the task-specific nature of semantic decision boundaries.

This stratified behavior aligns with the hierarchical organization of multimodal document encoders. Together with the spectral analysis, these results provide mechanistic evidence that AMD-Proj regulates gradient updates in a layer-dependent manner, achieving a balance between stability and plasticity without relying on explicit architectural constraints.

4.8. Task Order Sensitivity Analysis

Experimental Protocol

We evaluate the method under three different task sequences in order to assess the sensitivity of AMD-Proj to task ordering change. Each sequence represents a distinct continual learning scenario with varying degrees of task similarity and difficulty. The evaluated task orders are defined as follows:

• Order-A: SROIE → FUNSD → CORD → BuDDIE, corresponding to the default task order used in previous experiments.
• Order-B: FUNSD → SROIE → CORD → BuDDIE, which swaps the two early tasks to alter initial adaptation dynamics.
• Order-C: BuDDIE → CORD → FUNSD → SROIE, representing a reversed sequence.

Table 5 shows the impact of task order on continual learning performance using AMD-Proj. All experiments are conducted using identical hyperparameters, optimization settings, and evaluation protocols. This ensures that any observed performance differences can be attributed only to task ordering effects rather than confounding experimental factors.

Table 5. Impact of task order on continual learning performance using AMD-Proj. We report average accuracy (ACC), backward transfer (BWT), and their F1-based counterparts on LayoutLMv3.

Task Order	ACC ↑	BWT ↑	F1-ACC ↑	F1-BWT ↑
Order A	0.7064	$-0.2504$	0.4440	$-0.2100$
Order B	0.6489	$-0.2871$	0.4699	$-0.2487$
Order C	0.6127	$-0.3120$	0.3997	$-0.2643$

↑ indicates higher is better.

Order A shows the least negative backward transfer, indicating stronger retention of previously learned tasks, but achieves lower overall accuracy. In contrast, Order B attains the highest ACC and F1-ACC at the cost of increased forgetting, while Order C exhibits intermediate accuracy with the largest degradation in backward transfer.

Overall, AMD-Proj exhibits a consistent and smooth performance profile across different task orders. While task ordering influences the balance between retention and adaptation, performance degrades gradually rather than catastrophically, indicating robust behavior under varying continual learning conditions.

4.9. Ablation Studies

To analyze both the necessity of the proposed mechanisms and the robustness of AMD-Proj to hyperparameter choices, we conduct two complementary ablation studies. First, we perform a functional ablation by selectively disabling gradient reuse or blocking. Second, we analyze the sensitivity of AMD-Proj to its main hyperparameters.

4.9.1. Functional Ablation of Reuse and Blocking

Table 6 reports the impact of disabling gradient reuse or blocking in AMD-Proj. The full method achieves the best balance between stability and plasticity, exhibiting minimal forgetting and the highest F1-based performance.

Table 6. Ablation study on the impact of gradient reuse and blocking in AMD-Proj. Results are reported using accuracy-based and F1-based continual learning metrics on LayoutLMv2.

Method	ACC	BWT	F1-ACC	F1-BWT
AMD-Proj (full)	0.776	−0.033	0.574	−0.018
w/o Reuse (${\beta }_{\mathrm{reuse}}=0$)	0.758	−0.09	0.501	−0.13
w/o Blocking (${\beta }_{\mathrm{block}}=0$)	0.781	−0.10	0.548	−0.11

Disabling reuse (${\beta }_{\mathrm{reuse}}=0$) degrades backward transfer and F1 accuracy, indicating that attenuation alone over-constrains optimization and prevents effective knowledge transfer across document domains. Similarly, disabling blocking (${\beta }_{\mathrm{block}}=0$) results in severe forgetting, as amplified reusable directions are insufficient to counteract destructive task-specific gradients. These results confirm that both reuse and blocking are essential and play complementary roles.

4.9.2. Hyperparameter Sensitivity

We explored how AMD-Proj responds to its hyperparameters in Table 7 Across all settings, BWT varies without big differences or huge drop which demonstrates that the method does not rely on finely tuned values. Varying reuse and blocking thresholds has a minor effect on forgetting, demonstrating robust subspace selection. The energy threshold E exhibits a clearer effect, lower values lead to more aggressive regularization and increased forgetting, while higher values provide a better balance between stability and flexibility. Finally, increasing the number of retained subspaces does not provide marginal benefit, validating the efficiency of the default top-$k=1$ setting.

Table 7. Ablation study on AMD-Proj hyperparameters (LayoutLMv2).

Setting	${\mathit{\tau }}_{\mathbf{reuse}}$	${\mathit{\tau }}_{\mathbf{block}}$	${\mathit{\beta }}_{\mathbf{block}}$	${\mathit{\beta }}_{\mathbf{reuse}}$	E	BWT
Default	0.10	0.25	0.20	0.30	0.90	−0.033
Low thresholds	0.05	0.20	0.20	0.30	0.90	−0.034
High thresholds	0.15	0.30	0.20	0.30	0.90	−0.035
Low strengths	0.10	0.25	0.10	0.20	0.90	−0.035
High strengths	0.10	0.25	0.30	0.40	0.90	−0.03
Low energy	0.10	0.25	0.20	0.30	0.70	−0.041
High energy	0.10	0.25	0.20	0.30	0.95	−0.033
Top-$k=3$	0.10	0.25	0.20	0.30	0.90	−0.034

5. Discussion

Beyond empirical performance, the results provide an insight on how gradient-level control interacts with VrDU models in continual learning settings.

First, the results indicate that explicitly modeling gradient interference and reuse is an effective mechanism for mitigating catastrophic forgetting. On architectures such as LayoutLMv2, selective gradient projection enables stable adaptation without overly constraining optimization, which demonstrates the importance of gradient geometry in continual learning for document understanding models. While AMD-Proj relies on the assumption that dominant task-relevant gradients can be captured by low-rank subspaces, this approximation is empirically supported by the observed spectral structure of gradient updates. However, in extremely large models, gradient structure may deviate from strict low-rank behavior, representing a natural limitation of subspace-based methods.

Second, comparisons across architectures show that the effectiveness of gradient subspace control depends on the model’s degree of architectural integration. More tightly integrated backbones, such as LayoutLMv3, exhibit greater forgetting due to the difficulty of selecting blocking and reusing gradients.

Third, the mechanistic analyses provide an interpretable account of how AMD-Proj operates internally. Spectral analysis indicates that adaptation under AMD-Proj concentrates parameter updates within a compact set of dominant directions. At the same time, reuse–block dynamics reveal a layer-dependent strategy that aligns with the hierarchical architecture of the model: early layers favor conservative updates to preserve shared low-level representations, intermediate layers exploit reusable structure for positive transfer, and late layers emphasize protection of task-specific semantics. This behavior offers a concrete geometric interpretation of how AMD-Proj balances stability and plasticity across the depth of the model.

Fourth, the task-order sensitivity analysis shows that AMD-Proj is robust but not order-invariant. Performance variations across task sequences follow predictable trends associated with task similarity and the availability of reusable gradient directions, degradation under unfavorable orders remains small rather than catastrophic, indicating controlled adaptation rather than brittle dependence on a specific task sequence. This behavior is consistent with the design of AMD-Proj, in which reuse and blocking are controlled by gradient similarity.

Finally, several practical considerations merit discussion. Although training is performed with a batch size of one due to the high memory requirements of document understanding models, gradient subspaces are estimated from multiple gradient snapshots, yielding stable low-rank structures in practice. In addition, while AMD-Proj employs fixed similarity thresholds, adaptive or data-driven thresholding strategies may further improve robustness under extreme cases of task similarity or dissimilarity, and represent an interesting direction for future work. Extensions to class-incremental and domain-incremental settings, as well as privacy-aware handling of stored gradient subspaces in sensitive applications, remain important avenues for further investigation.

Overall, these findings underscore the value of explicit gradient-level control for continual learning in visually rich document understanding. By balancing reuse and protection in a layer-aware and similarity-driven manner, AMD-Proj offers a principled and interpretable approach to continual adaptation in complex multimodal models.

Limitations

While AMD-Proj demonstrates consistent improvements in the stability–plasticity trade-off across evaluated benchmarks, several limitations should be acknowledged.

Modality entanglement sensitivity: AMD-Proj relies on the geometric separability of interfering and reusable directions within the subspace spanned by stored bases. This assumption weakens in highly integrated architectures such as LayoutLMv3, where text, layout, and visual tokens are processed jointly, leading to entangled gradient signals that reduce separability, as evidenced by AMD-Proj’s more negative backward transfer on LayoutLMv3 ($\mathrm{BWT}=-0.0918$) compared to LayoutLMv2 ($\mathrm{BWT}=-0.0332$). Note that AMD-Proj’s higher absolute accuracy on LayoutLMv3 reflects the stronger representational capacity of that backbone rather than improved forgetting control. Incorporating modality-aware subspace tracking, for example, by maintaining separate bases for textual and visual gradient components within each layer, is a promising direction for addressing this limitation.

Threshold sensitivity: The classification of gradient directions depends on two thresholds, ${\tau }_{\mathrm{reuse}}$ and ${\tau }_{\mathrm{block}}$. Although Table 5 shows robustness to moderate variations, these parameters are manually set and may require tuning across architectures and task distributions. Adaptive threshold scheduling, such as annealing ${\tau }_{\mathrm{block}}$ as the subspace matures, could reduce this dependency.

Task ordering sensitivity: Because AMD-Proj incrementally accumulates subspaces without revisiting past data, early tasks disproportionately influence the learned basis. Under unfavorable task orderings, this can lead to an overly restrictive projection space for subsequent tasks. Developing ordering-robust initialization or rebalancing strategies is an important direction for future work.

Full backbone requirement: AMD-Proj operates through gradient-level constraints during full fine-tuning and is not directly compatible with frozen-backbone PEFT regimes. As shown in Table 4, PEFT methods offer substantially lower memory usage at the cost of reduced performance. Extending AMD-Proj to operate on adapter-level gradients would improve its applicability in memory-constrained settings.

Low-rank subspace approximation: The method assumes that task-relevant gradient structure can be captured by low-rank subspaces. This assumption may break down in very high-capacity models or highly heterogeneous task distributions, where important directions are not well represented by a compact basis.

Scalability to longer task sequences: Although the $\mathcal{O}(L·r·d)$ storage bound is independent of the number of tasks, this has only been validated on a four-task benchmark. It remains an open question whether subspace merging continues to yield compact and informative representations as the number and diversity of tasks increase.

6. Conclusions

In this paper, we introduced AMD-Proj, a continual learning framework for visually rich document understanding that regulates gradient updates through selective reuse and blocking. By modeling gradient interference and transfer at the subspace level, AMD-Proj mitigates catastrophic forgetting while preserving the ability to adapt to new document domains.

Experiments on FUNSD, SROIE, CORD, and BuDDIE show that AMD-Proj consistently improves knowledge retention over both gradient-based and regularization-based baselines, while maintaining competitive predictive performance. The approach generalizes across state-of-the-art document understanding architectures, including LayoutLMv2 and LayoutLMv3, and exhibits stable behavior under different task orderings. In addition, the proposed mechanistic analyses provide an interpretable view of how selective gradient projection shapes adaptation across network depth.

Overall, this work highlights the importance of explicit gradient-level control for continual learning in multimodal and structured document settings. Future work will explore adaptive memory management, automatic estimation of task similarity, and extensions to more challenging continual learning scenarios such as class-incremental and streaming setups. We also plan to investigate the integration of AMD-Proj with parameter-efficient fine-tuning techniques and its application to broader multimodal continual learning problems beyond document understanding.