T-CASP: Time-Aware Continual Aspect Semantic-Driven Incremental Pretraining

Abstract

With the rapid advancement of medical foundation models, their deployment in clinical practice is increasingly required. However, privacy constraints of hospital-specific data make large-scale retraining infeasible, limiting model adaptability. To address this issue, we propose a Continual Aspect Semantic-driven Incremental Pretraining (CASP) framework, which enables efficient adaptation of foundation models to private data, and the pre-trained models can be effectively applied to downstream tasks. In this paper, we focus on fundus fluorescein angiography (FFA) in ophthalmology as a representative application scenario to validate the proposed approach. FFA is a critical imaging modality for retinal disease diagnosis, as it is able to capture dynamic vascular changes across multiple angiographic phases. However, most existing learning-based methods treat FFA images statically and independently, failing to exploit the rich temporal and phase-specific semantics that are essential for accurate diagnosis. In this paper, a Time-aware Continual Aspect Semantic-driven incremental Pretraining (T-CASP) framework is proposed for FFA sequences. To compensate for limited temporal descriptions in clinical reports, large language models are first used to construct a temporal disease knowledge dictionary with phase-specific semantic descriptions. Based on this dictionary, a disease correlation matrix is injected into contrastive learning to guide fine-grained image–text alignment. A multi-layer gated residual adapter is further designed to capture phase-level semantics and enable knowledge transfer across learning stages through phase-wise continual pretraining. Extensive experiments demonstrate that T-CASP effectively models dynamic temporal semantics in FFA sequences, yielding consistent performance improvements over time-unaware and static baselines in retinal disease recognition. By explicitly integrating phase-wise temporal knowledge and continual semantic refinement, T-CASP provides a clinically consistent and effective solution for temporal FFA analysis, enhancing robustness and diagnostic accuracy in ophthalmic multimodal learning.

1. Introduction

With the rapid advancement of medical foundation models, their deployment in real-world clinical environments has become increasingly important. However, due to the privacy and sensitivity of hospital-specific data, retraining large-scale medical pre-trained models on local datasets is often infeasible, which limits their adaptability to specific clinical scenarios. To address this challenge, we propose a Continual Aspect Semantic-driven Incremental Pretraining (CASP) framework. The CASP framework is designed to further adapt pre-trained medical foundation models through incremental pretraining, enabling them to effectively support various downstream tasks, including disease diagnosis report generation and medical question answering. The core idea lies in introducing a multi-layer gated residual adapter to achieve parameter-efficient model adaptation. Specifically, CASP leverages a medical knowledge dictionary constructed by a Large Language Model to model aspect-level semantics, and fits these aspect-specific adapters using private clinical data to capture fine-grained semantic representations. On top of this, a class-level adapter is employed with supervised learning to model relationships among disease categories. Through this hierarchical design, CASP constructs specialized pre-trained models tailored to specific data distributions, thereby improving their effectiveness in downstream clinical applications.

The eye is one of the most important and disease-prone organs in the human body. Effective fundus imaging plays a critical role in the early diagnosis of ocular diseases. Fundus fluorescein angiography (FFA) dynamically records the circulation of fluorescein within the retinal and choroidal vasculature. It provides highly valuable information on blood flow and vascular permeability. As a result, FFA has become one of the indispensable imaging tools in ophthalmic clinical practice [1,2].

During FFA examination, imaging appearances of different diseases evolve dynamically as the angiographic process progresses. These temporal changes not only reflect the physiological stages of normal blood circulation but also reveal pathological features such as blood flow impairment, vascular leakage, and perfusion defects. Such dynamic characteristics are of great importance for differential diagnosis.

Across different angiographic phases, FFA reveals progressively discriminative disease characteristics. In the arterial phase, arteries are clearly visualized, while early perfusion delay may appear in multiple diseases, leading to overlapping imaging features and limited diagnostic specificity. As FFA enters the capillary phase, the capillary network and perfusion status become more evident, allowing clearer assessment of microvascular abnormalities and focal leakage. In the venous and late phases, disease-specific patterns such as venous dilation, abnormal drainage, fluorescein leakage, and extravascular retention become more prominent, providing more reliable cues for accurate disease differentiation [3].

The above observations indicate that, in FFA-based analysis of fundus diseases, different diseases do not consistently exhibit independent and static imaging features. Instead, they may present overlapping or confusing appearances at certain angiographic phases [4], while showing distinctive characteristics at other phases. Therefore, ophthalmologists typically anlyze FFA images by integrating dynamic fluorescence changes across temporal phases to distinguish diseases with similar early manifestations but different underlying pathological mechanisms.

Based on this clinical understanding, integrating the phase-wise dynamic relationships of FFA as aspect-specific representations into the CASP framework is both natural and well-justified. Compared with using a single disease label, refining disease representations into phase-specific imaging descriptions enables the model to capture key diagnostic features and their temporal dependencies more effectively.

By constructing a temporal disease knowledge dictionary covering the arterial phase, capillary phase, venous phase, and late phase, the model can preserve reasonable semantic overlap among diseases while more accurately extracting dynamic imaging features with clinical diagnostic relevance. This strategy provides clinically consistent supervision signals for subsequent knowledge injection and temporal modeling, thereby improving model performance in dynamic temporal feature analysis.

In addition, considering that each phase in FFA is composed of multiple angiographic images, a single image is insufficient to fully capture the semantic characteristics of that phase. Building upon the CASP framework, we further propose a model for Time-aware Continual Aspect Semantic-driven incremental Pretraining (T-CASP). In this model, the LSTM mechanism [5,6] is introduced in the image encoding process to effectively capture temporal dependencies within FFA sequences, enabling the model to leverage intra-phase dynamic information and explicitly incorporate phase-level knowledge into contrastive pre-training. The overall framework of T-CASP is illustrated in Figure 1.

Figure 1. Time-aware continual aspect semantic-driven incremental pretraining framework (T-CASP) uses LLM to construct temporal knowledge dictionary (A). Based on the dictionary and the pretrained textual tower, the disease correlation matrix (B) is constructed for fine-grained image–text alignment in the subsequent stage. The fine-grained image–text alignment (C) is conducted via time-aware phase-level contrastive learning (C1) and semantic-aware multi-phase aggregation (C2). The image–text alignment employs multi-stage incremental pre-training (D).

Due to the limited temporal information in clinical diagnostic reports, we first leverage large language models (LLMs) to construct a temporal knowledge dictionary [7]. This dictionary enriches the temporal information of the FFA dataset by providing phase-level temporal descriptions of disease, as shown in Figure 1A. Based on this knowledge dictionary, we construct a disease correlation knowledge matrix and apply it in the contrastive learning process by knowledge injection, as shown in Figure 1B.

To fully utilize information from each angiographic phase, this paper proposes a new model architecture for fine-grained image–text alignment. A multi-layer gated residual adapter [8,9] is introduced to capture phase-level semantic knowledge and facilitate knowledge transmission across learning stages. As shown in Figure 1C, each fine-grained image–text alignment is conducted in two sequential steps. First, time-aware phase-level contrastive learning is performed, where the adapters learn phase-specific representations, as shown in Figure 1(C1). Next, semantic-aware multi-phase aggregation integrates information across all phases to generate more discriminative disease representations, as shown in Figure 1(C2).

Since the temporal knowledge dictionary generated by large language models and the disease correlation matrix derived from the text encoder may not fully capture the semantic knowledge of the target dataset, we adopt incremental pre-training [10]. As training progresses, the disease correlation knowledge matrix incorporates more semantic information related to the dataset, enabling knowledge update and being used for fine-grained image–text alignment in the subsequent stage [11], as shown in Figure 1D.

2. Materials and Methods

We first uses LLM to construct temporal knowledge dictionary (Section 2.1). Subsequently, we construct disease correlation matrix using the dictionary and apply this matrix to image–text alignment training process through the knowledge injection mechanism (Section 2.2). And thirdly, Section 2.3 introduces our multi-layer gated residual adapter. Finally, Section 2.4 and Section 2.5 respectively introduce the time-aware phase-level contrastive learning and the semantic-aware multi-phase aggregation for fine-grained image-text alignment.

2.1. Construction of Temporal Knowledge Dictionary

To systematically characterize disease-specific dynamic imaging features along the temporal dimension, we conducted in-depth discussions with experienced ophthalmologists and reviewed relevant professional literature [12]. Based on the standard FFA interpretation workflow, the complete angiographic process was divided into four key stages, which serve as temporal phase-level aspects:

• Arterial Phase: This phase begins when fluorescein first enters and fills the retinal arterial circulation. It is characterized by early arterial filling and any vascular occlusion or delayed arterial filling, which are considered abnormal findings in this phase.
• Capillary Phase: The retinal capillary bed becomes fully visualized and perfusion status is most clearly delineated. This phase often reveals microvascular leakage, areas of non-perfusion, and other microstructural changes associated with retinal vascular pathology.
• Venous Phase: The venous system progressively fills and the circulation completes. This phase often reveals microvascular leakage, areas of non-perfusion, and other microstructural changes associated with retinal vascular pathology.
• Late Phase: Fluorescein diffusion, leakage, or retention outside the vessels gradually emerges. This phase is marked by tissue staining, accumulated leakage, and other definitive pathological evidence that may not be apparent in earlier phases.

These four phase-level aspects comprehensively reflect the full evolution of disease from early hemodynamic abnormalities to late pathological changes, providing a structured semantic basis for temporal modeling. After defining the phase-level aspects, we designed a set of prompt engineering strategies to guide large language models [13] in generating corresponding English descriptions for each disease at each anchor, and used the generated outputs to construct a structured temporal knowledge dictionary. The disease set and the aspect set are denoted respectively as $D=\{d_1,d_2,\dots ,d_N\}$ and $C=\{c_1,c_2,\dots ,c_M\}$, where each $d_n$ represents a specific disease, and each $c_m$ represents a semantic aspect. In addition, the disease set D is constructed from a private dataset and contains 67 disease categories, while the aspect set C corresponds to phase-level aspects and consists of 4 semantic aspects. For generality and theoretical completeness, we use N and M to denote the number of diseases and aspects, respectively.

Then, use the large language model to generate the semantic description of disease $d_n$ under aspect $c_m$:

$$l_n^{\left(m\right)}=\mathrm{LLM}(d_n,c_m),d_n\in D,c_m\in C,$$

(1)

where $d_n$ denotes the name of the n-th disease in the disease set D, e.g., NPDR and $c_m$ denotes the m-th aspect in the aspect set C, e.g., Capillary Phase. $\mathrm{LLM}(·,·)$ indicates the Large Language Model, and $l_n^{\left(m\right)}$ represents the corresponding disease descriptive information, e.g., the characteristic manifestation of NPDR in the Capillary Phase. Then, we organize $l_n^{\left(m\right)}$ into a structured semantic knowledge dictionary $dic{t}_N^M$ to comprehensively represent phase-level disease characteristics:

$$dic{t}_N^M=\left\{l_n^{\left(m\right)}\right|n=1,\cdots ,N;m=1,\cdots ,M\},$$

(2)

where N denotes the number of diseases, M denotes the number of aspects, and $dic{t}_N^M$ denotes the resulting knowledge dictionary, as shown in Figure 1A.

2.2. Knowledge Injection Mechanism

The core objective of knowledge injection mechanism is to effectively integrate the dictionary into the image–text alignment, enabling the model to understand and exploit the relationships among diseases across multiple phases.

First, the semantic description $l_n^{\left(m\right)}$ in the dictionary $dic{t}_N^M$ is encoded using a text encoder, resulting in textual features.

$$\widehat{t_n^{\left(m\right)}}=TextEncoder\left(l_n^{\left(m\right)}\right),$$

(3)

where $\widehat{t_n^{\left(m\right)}}$ denotes the textual feature of disease $d_n$ under semantic aspect $c_m$, generated by the text encoder $TextEncoder$. The $TextEncoder$ consists of a textual tower and multi-layer gated residual adapters. During training, the adapter parameters are continuously updated based on the optimization objective, enabling ${\widehat{t}}_n^{\left(m\right)}$ to be updated accordingly. Hence, ${\widehat{t}}_n^{\left(m\right)}$ acquires increasingly rich medical semantic information that better align with the target dataset.

After obtaining textual features of all diseases across multiple aspects, we compute pairwise disease-to-disease relationships using cosine similarity [14]. The resulting similarities are used to construct a disease correlation matrix:

$${\mathbf{U}}_{N^2}^M=\left\{\mathrm{sim}\left(\widehat{t_n^{\left(m\right)}},\widehat{t_{n^{\prime }}^{\left(m^{\prime }\right)}}\right)\right\}\in {\mathbb{R}}^{M\times N\times N},$$

(4)

where $\mathrm{sim}(·,·)$ denotes the cosine similarity function, ${\mathbf{U}}_{N^2}^M$ represents the disease correlation matrix with dimensionality $M\times N\times N$, and each slice of ${\mathbf{U}}_{N^2}^M$, i.e., ${\mathbf{U}}_{N^2}^{\left(m\right)}$ captures the pairwise relationships between diseases under one of the M aspects. This matrix reflects the same characteristics of diseases and latent differences between different phase-level aspects.

As shown in Figure 2, extending this computation to all diseases yields a complete $N\times N$ disease similarity matrix. Since the similarity is calculated separately for each of the M aspects, the final representation forms a three-dimensional tensor of size $M\times N\times N$, capturing disease relationships under different aspect-specific conditions.

Figure 2. Construct the disease correlation matrix. In the figure, the pairwise similarity between the first disease and all other diseases is computed, and the resulting values are assigned to the first row of the disease correlation matrix.

Subsequently, the model utilizes the knowledge injection mechanism to incorporate this matrix into the subsequent training process. For a batch $\mathbf{B}$ containing K images, a corresponding multi-hot encoding matrix is constructed:

$${\mathbf{H}}_{\mathbf{B}}=Multi\text{-}hot(b_1,b_2,\cdots ,b_K)\in {\mathbb{R}}^{K\times N},$$

(5)

where $b_k$ denotes the disease label set of the k-th sample in batch $\mathbf{B}$, and $Multi\text{-}hot(·)$ represents the multi-label encoding operation that maps disease categories to binary row vectors, aggregating them into the batch-level multi-label matrix ${\mathbf{H}}_{\mathbf{B}}$.

The core mechanism of knowledge injection is to transfer the disease correlation matrix, which includes disease-level prior knowledge, to the batch level. As shown in Figure 3, for the first sample, the activated entries in $H_B$ indicate the presence of multiple diseases, which in turn select the corresponding disease correlation vectors from $U_{N^2}^{\left(m\right)}$. Then, two disease-related vectors are activated and these selected vectors are then averaged to obtain the correlation representation between the current sample and all N diseases. It is worth noting that the disease correlation matrix $\mathbf{U}$ contains M aspect-specific components. Therefore, the above process is performed independently across all M aspects, resulting in an $M\times N$ representation that captures the correlations between the current sample and all diseases under different aspects. Finally, by extending this operation to all K samples in the batch, we obtain a tensor of size $K\times M\times N$, which represents the correlations between each sample and all diseases across the M aspects.

Figure 3. Model the relationships between samples and diseases at the batch level. The multi-hot encoding matrix ${\mathbf{H}}_{\mathbf{B}}\in {\mathbb{R}}^{K\times N}$ serves as input, where K denotes the batch size and N denotes the number of diseases. In the figure, for a given sample, its multi-label encoding vector is used to interact with the disease correlation matrix ${\mathbf{U}}_{N^2}^{\left(m\right)}$.

The above process can be formally expressed using matrix operations. For clarity, we consider a single aspect $c_m$ and focus on the corresponding disease correlation matrix ${\mathbf{U}}_{N^2}^{\left(m\right)}\in {\mathbb{R}}^{N\times N}$ extracted from ${\mathbf{U}}_{N^2}^M$. Based on this, the formulation is defined as follows:

$${\mathbf{T}}_{\mathbf{B}}^{\left(m\right)}={\mathbf{H}}_{\mathbf{B}}·{\mathbf{U}}_{N^2}^{\left(m\right)}\odot {\displaystyle \frac{1}{{\mathbf{H}}_{\mathbf{B}}·{\mathbf{1}}^{(N\times N)}}}\in {\mathbb{R}}^{K\times N},$$

(6)

where ${\mathbf{U}}_{N^2}^{\left(m\right)}\in {\mathbb{R}}^{N\times N}$ represents the corresponding disease–disease correlation matrix of $c_m$, ${\mathbf{1}}^{(N\times N)}$ denotes an all-ones matrix of size $N\times N$; ⊙ denotes the Hadamard product, i.e., element-wise multiplication between two matrices of the same dimension. Specifically, the unnormalized sample–disease correlation matrix under the m aspect is first computed via the matrix multiplication ${\mathbf{H}}_{\mathbf{B}}·{\mathbf{U}}_{N^2}^{\left(m\right)}$. Then, the number of disease labels associated with each sample is obtained by ${\mathbf{H}}_{\mathbf{B}}·{\mathbf{1}}^{(N\times N)}$, which is further expanded to match the dimensionality of the correlation matrix and used as a normalization factor. Based on this, element-wise reciprocal is applied to the normalization factor, followed by Hadamard multiplication with the correlation matrix where each entry is multiplied independently with its counterpart to achieve label-cardinality-aware normalization. Essentially, this process can be interpreted as an averaging aggregation over the correlation vectors corresponding to the activated disease labels, thereby effectively mitigating the scale bias introduced by variations in the number of labels across samples.

Finally, the aspect-level semantic similarity matrix ${\mathbf{T}}_{\mathbf{B}}^{\left(m\right)}$ is combined with the multi-hot encoding matrix ${\mathbf{H}}_{\mathbf{B}}$, and a decay factor $\gamma$ is introduced to control the strength of knowledge injection, yielding the final multi-aspect contrastive matrix:

$${\mathbf{G}}_{\mathbf{B}}^{\left(m\right)}=\gamma \times {\mathbf{T}}_{\mathbf{B}}^{\left(m\right)}+{\mathbf{H}}_{\mathbf{B}}\in {\mathbb{R}}^{K\times N},$$

(7)

where $\gamma$ is a knowledge injection coefficient that dynamically balances the learning objective during training. In the early stages, a relatively small value of $\gamma$ (=0.1) is adopted to encourage the model to primarily rely on data-driven alignment. As training progresses, $\gamma$ is linearly increased to an upper bound (=0.7), thereby strengthening the influence of the aspect-level semantic similarity matrix. This mechanism guides the model to increasingly learn a multi-aspect contrastive structure that is gradually consistent with medical knowledge, enabling a smooth transition from data-driven learning to knowledge-guided optimization.

2.3. Multi-Layer Gated Residual Adapter

A multi-phase update strategy is employed for the transition of knowledge injection from data-driven learning to knowledge-guided optimization. Meanwhile, the fine-grained image–text alignment process is conducted in sequential phases, starting with the phase-level contrastive learning, followed by the multi-aspect aggregation. These multi-phase characteristics naturally motivate an incremental learning paradigm that supports gradual and structured knowledge accumulation [10,15,16,17].

However, incremental learning faces the well-known issue of catastrophic forgetting [17,18,19], where previously learned knowledge is overwritten by specific updates for the new task. To mitigate these challenges, we introduce an adapter based on a multi-layer gated residual architecture [8,20], as shown in Figure 4.

Figure 4. Framework of multi-layer gated residual adapter.The adapter consists of a bottleneck-structured multilayer perceptron (MLP). Multiple MLP blocks are stacked to form a deep adapter, and the contribution of each layer is precisely controlled by its corresponding learnable gating parameter.

The multi-layer gated residual adapter incorporates learnable gating parameters [21] to regulate the degree of adaptation at each layer. When additional task-specific adjustments are deemed unnecessary, these gating parameters naturally converge to zero. Due to the residual formulation, the adapter is able to degenerate into an identity mapping, allowing the original feature representations produced by the backbone encoder to remain unchanged [9,22]. This design effectively preserves the performance of the pretrained model when newly introduced task-specific information fails to yield meaningful performance gains. The computation is defined as follows:

$$h^{\left(l\right)}=h^{(l-1)}+{\alpha }^{\left(l\right)}{u}^{\left(l\right)},l=1,\dots ,L,$$

(8)

$$u^{\left(l\right)}={\mathrm{LN}}_l\left({\mathrm{MLP}}_l\left(h^{(l-1)}\right)\right),l=1,\dots ,L,$$

(9)

where $h^{\left(l\right)}$ denotes the feature vector, i.e., $h^{\left(0\right)}$ denotes the input feature representation, $L{N}_l$ is the normalization at the l-th layer, $ML{P}_l$ denotes the multilayer perceptron at layer l, $u^{\left(l\right)}$ represents the intermediate output, ${\alpha }^{\left(l\right)}$ is a learnable gating parameter which controls the strength of adaptation at layer l, and L denotes the number of MLP modules within the adapter, which can be understood as the depth of the adapter. The ${\alpha }^{\left(l\right)}$ is initialized to 0, corresponding to an identity mapping, and is progressively updated during training to enable adaptive feature modulation.

By viewing the stacked layers as a whole, the multi-layer gated residual adapter can be compactly expressed as:

$$h^{\left(L\right)}=\mathrm{Adapter}\left(h^{\left(0\right)}\right).$$

(10)

2.4. Time-Aware Phase-Level Contrastive Learning

To fully exploit the pronounced temporal phase dependency of disease representations in FFA, a time-aware phase-level contrastive learning framework is proposed in this paper. As shown in Figure 5, phase-specific image and textual adapters are introduced to enable fine-grained phase-level modeling, while the parameters of the pre-trained visual and textual towers remain frozen, and only the adapters are optimized. The aspect-level semantic similarity matrix ${\mathbf{G}}_{\mathbf{B}}^{\left(m\right)}$ is used as the supervising signal for contrastive learning, enhancing both the phase-level and image–text alignment.

Figure 5. Time-aware phase-level contrastive learning framework.

It is worth noting that, there are multiple FFA images within a single phase and the phase division is strictly determined by acquisition time. Using only a single image from each phase may lead to insufficient high-level semantic representation, and it also requires manual selection of representative images from multiple candidates.

To address this issue, we employ a Long Short-Term Memory (LSTM) network to model temporal dependencies within each phase. LSTM is a type of recurrent neural network designed to capture long-term dependencies in sequential data by leveraging the hidden state from previous time steps to inform the current representation. Specifically, the hidden state at each time step encodes both the current input and the accumulated information from previous observations [5,6].

By applying LSTM to a sequence of images within each phase, we aggregate semantic information from multiple images. The last hidden state before transitioning to the next phase is selected as the feature representation of the current phase. In this way, the representation fully captures the temporal information within the phase while avoiding interference from subsequent phases, which is consistent with the imaging mechanism of FFA, where the injected fluorescein dye progressively propagates through the retinal vasculature over time [23].

Meanwhile, temporal dependencies across FFA image sequences are learned by the LSTM machine, from which a representative feature is extracted for each phase and used for subsequent cross-modal alignment and aggregation.

In addition, we adopt a phase-wise knowledge injection and refinement strategy. As shown in Figure 1(C1), contrastive learning [24] in each phase is guided by a disease correlation matrix generated from the previous phase. This iterative process alternates across phases, progressively constructing the disease correlation matrix that integrates refined semantic information.

Specifically, on the textual side, multi-phase semantic descriptions $l_n^{\left(m\right)}$ are encoded by the frozen textual tower $f_{\mathrm{text}}$ to generate base embeddings:

$$t_n^{\left(m\right)}=f_{\mathrm{text}}\left(l_n^{\left(m\right)}\right).$$

(11)

Subsequently, an independent textual adapter ${\mathrm{Adapter}}_{\mathrm{text}}^{\left(m\right)}$ is assigned to each aspect to obtain phase-level fine-grained textual features:

$$\widehat{t_n^{\left(m\right)}}={\mathrm{Adapter}}_{\mathrm{text}}^{\left(m\right)}\left(t_n^{\left(m\right)}\right),$$

(12)

on the visual side, each sample $X_s$ contains an FFA image sequence, i.e., $X_s=\{x_s^{\left(1\right)},x_s^{\left(2\right)},\dots ,x_s^{\left(T\right)}\}$, where $x_s^{\left(t\right)}$ denotes the FFA image at the t-th angiographic phase, T denotes the total number of FFA images in the sequence. Subsequently, $X_s$ are encoded by the frozen imageual tower $f_{\mathrm{image}}$ to generate base embeddings:

$$v_s=f_{\mathrm{image}}\left(X_s\right),$$

(13)

where $v_s$ denotes the visual feature extracted from the FFA image. To capture the temporal dependencies across different angiographic phases, the visual feature sequence $v_s$ is fed into the LSTM network for sequential modeling:

$$h_s^{\left(t\right)}=\mathrm{LSTM}(v_s^{\left(t\right)},h_s^{(t-1)}),t=1,\cdots ,T,$$

(14)

where $h_s^{\left(t\right)}$ represents the temporal feature embedding that integrates the visual information of the current angiographic phase with its historical context. This phase-level temporal representation is subsequently treated as the visual representation of the t-th angiographic phase.

Furthermore, in accordance with the clinical FFA acquisition protocol, semantically meaningful angiographic phases are selected along the temporal dimension. Specifically, for each phase, the last frame of each phase is chosen as the representative key frame, according to the chronological angiographic progression:

$$h_s^{\left(\mathcal{P}\right)}=\left(h_s^{\left(t_{art}\right)},h_s^{\left(t_{cap}\right)},h_s^{\left(t_{ven}\right)},h_s^{\left(t_{late}\right)}\right),t_{art}<t_{cap}<t_{ven}<t_{late},$$

(15)

where $t_{art}$, $t_{cap}$, $t_{ven}$, and $t_{late}$ correspond to the terminal time points of the arterial phase, capillary phase, venous phase, and late phase, respectively; $h_s^{\left(\mathcal{P}\right)}$ denotes the time-aware phase-level visual representation of sample s, which is formed by aggregating the image features.

Subsequently, for each image feature $h_s^{\left(t\right)}$, an independent visual adapter ${\mathrm{Adapter}}_{\mathrm{image}}^{\left(m\right)}$ is assigned to obtain phase-level fine-grained visual features:

$$\widehat{h^{\left(m\right)}}={\mathrm{Adapter}}_{\mathrm{image}}^{\left(m\right)}\left(h_s^{\left(\mathcal{P}\right)}\right).$$

(16)

Subsequently, bidirectional contrastive learning is performed using the multi-aspect contrastive matrix ${\mathbf{G}}_{\mathbf{B}}^{\left(m\right)}$. Sepcifically, it is formulated using the InfoNCE loss as follows:

$${\mathcal{L}}_{\mathrm{phase}}^{\left(m\right)}=\mathrm{CE}({\mathbf{G}}_{\mathbf{B}}^{\left(m\right)},{\mathbf{S}}_{\mathbf{v}\mathbf{2}\mathbf{t}}^{\left(m\right)})+\mathrm{CE}[{\left({\mathbf{G}}_{\mathbf{B}}^{\left(m\right)}\right)}^T,{\mathbf{S}}_{\mathbf{t}\mathbf{2}\mathbf{v}}^{\left(m\right)}],$$

(17)

where CE represents the InfoNCE loss [25], m indexes different fine-grained phase-level aspects, ${\mathbf{S}}_{\mathbf{v}\mathbf{2}\mathbf{t}}^{\left(m\right)}=\tau (\widehat{h^{\left(m\right)}},\widehat{t_n^{\left(m\right)}})$ and ${\mathbf{S}}_{\mathbf{t}\mathbf{2}\mathbf{v}}^{\left(m\right)}=\tau (\widehat{t_n^{\left(m\right)}},\widehat{h^{\left(m\right)}})$ represent the phase-level similarity matrices from image to text and from text to image, respectively, $\tau$ is a learnable temperature parameter [26].

Additionally, in contrastive learning, the temperature parameter controls the concentration of the similarity distribution, thereby affecting the separation between positive and negative pairs. In practice, the use of temperature scaling—especially learnable temperature parameters—is a commonly adopted and well-established strategy in contrastive learning to improve optimization behavior. Following this practice, we introduce learnable temperature parameters within each aspect-specific adapter, allowing adaptive adjustment of similarity scaling for different semantic aspects. This design facilitates a smoother training process and more stable convergence, without requiring manual tuning of temperature across different aspects.

The loss of time-aware phase-level contrastive learning is defined as follows:

$${\mathcal{L}}_{\mathrm{phase}}={\displaystyle \frac{1}{2M}}\sum _{m=1}^M{\mathcal{L}}_{\mathrm{phase}}^{\left(m\right)}.$$

(18)

2.5. Semantic-Aware Multi-Phase Aggregation

After completing the time-aware phase-level contrastive learning, we introduce the semantic-aware multi-phase aggregation phase, which enforces disease-level semantic consistency by applying class-level contrastive learning.

As shown in Figure 1(C2), the initial visual features are extracted using the phase-level visual adapters from the previous stage. To preserve the learned time-aware representations and avoid catastrophic forgetting, the parameters of the phase-level adapters are frozen, and only the final gating parameters are optimized to adaptively re-weight the contributions of different phases.

Specifically, on the visual side, the outputs of the phase-level visual adapters ${\widehat{h}}^{\left(m\right)}$ are concatenated:

$$\widehat{h^{\mathrm{phase}}}=\mathrm{Concat}(\widehat{h^{\left(1\right)}},\cdots ,\widehat{h^{\left(m\right)}}).$$

(19)

Subsequently, the class-level visual adapter ${\mathrm{Adapter}}_{\mathrm{image}}^{\mathrm{class}}$ encodes the joint representation:

$$\widehat{h^{\mathrm{class}}}={\mathrm{Adapter}}_{\mathrm{image}}^{\mathrm{class}}(\widehat{h^{\mathrm{phase}}}).$$

(20)

On the textual side, the class-level textual adapter encodes the overall class-level disease descriptions $l_n^{\mathrm{class}}$:

$$\widehat{t^{\mathrm{class}}}={\mathrm{Adapter}}_{\mathrm{text}}^{\mathrm{class}}\left(f_{\mathrm{text}}\left(l_n^{\mathrm{class}}\right)\right).$$

(21)

where $\widehat{t^{\mathrm{class}}}$ represents the class-level textual feature, $f_{\mathrm{text}}$ represents the pre-trained and frozen textual tower, ${\mathrm{Adapter}}_{\mathrm{text}}^{\mathrm{class}}$ represents the class-level and unfrozen textual adapter.

The loss of semantic-aware multi-phase aggregation is defined as follows:

$${\mathcal{L}}_{\mathrm{class}}={\displaystyle \frac{1}{2}}\left[\mathrm{CE}({\mathbf{H}}_{\mathbf{B}},{\mathbf{S}}_{\mathbf{v}\mathbf{2}\mathbf{t}})+\mathrm{CE}[{\left({\mathbf{H}}_{\mathbf{B}}\right)}^T,{\mathbf{S}}_{\mathbf{t}\mathbf{2}\mathbf{v}}]\right],$$

(22)

where ${\mathbf{S}}_{\mathbf{v}\mathbf{2}\mathbf{t}}=(\widehat{h^{\mathrm{class}}},\widehat{t^{\mathrm{class}}})$ and ${\mathbf{S}}_{\mathbf{t}\mathbf{2}\mathbf{v}}=(\widehat{t^{\mathrm{class}}},\widehat{h^{\mathrm{class}}})$ represent the class-level similarity matrices from image to text and from text to image, respectively. Through class-level contrastive learning, the model emphasizes global class-level consistency rather than fine-grained phase-level differences.

The total loss for T-CASP as follows:

$${\mathcal{L}}_{\mathrm{casp}}={\mathcal{L}}_{\mathrm{concept}}+{\mathcal{L}}_{\mathrm{class}}.$$

(23)

3. Results

Experimental Settings. All experiments were conducted on a workstation equipped with an NVIDIA RTX 4090 GPU and 64 GB of system memory. The implementation was based on the PyTorch (version 1.13.1) framework. This hardware configuration provides sufficient computational capacity for model training and evaluation, and supports the reproducibility of our experiments.

Data Preprocessing. The private dataset is derived from clinical diagnostic reports in PDF format, where the angiographic analysis section contains disease-level descriptions based on FFA observations. These descriptions are manually written by different clinicians, leading to variability in expression and terminology, and thus exhibit unstructured textual characteristics. During preprocessing, the angiographic descriptions are extracted and standardized via rule-based methods (e.g., regular expression matching) to obtain structured multi-label annotations. Initially, 149 fine-grained disease categories are identified. However, such a large label space introduces challenges for contrastive learning due to the quadratic scaling of the disease correlation matrix, and further aggravates class imbalance. To address this, we design a clinically guided disease merging strategy to consolidate related subcategories. The final label space contains 67 disease categories, with sample sizes ranging from 100 to 800 per class, resulting in a relatively balanced distribution for training.

Pre-training Data. All experiments were conducted on a private clinical FFA dataset collected and annotated by experienced ophthalmologists during routine diagnosis. Each sample corresponds to a single examined eye and comprises a temporally ordered sequence of FFA, a angiographic analysis report, and multi-label disease annotations. The dataset contains 14,870 eyes in total, with 10,387 for training, 2258 for validation, and 2225 for testing, which were split at the patient level. In this study, the disease set consists of 67 disease categories, corresponding to the multi-label classification targets. In addition, the aspect set is defined based on phase-level characteristics of FFA and includes 4 semantic aspects.

Optimizer and Training Strategy. We use the AdamW optimizer [27] with a cosine annealing learning rate schedule [28]. Linear warm-up [29], periodic decay, and a final cooling stage are applied to stabilize optimization. To better optimize different trainable components, we adopt grouped learning rates, assigning separate learning rates to different parameter groups.

Image Preprocessing. To ensure consistency across samples and improve model robustness, all input images are processed through a unified preprocessing pipeline. Each raw FFA image is first loaded and converted into a floating-point representation, followed by intensity normalization to the range $[0,1]$, which standardizes the dynamic range and stabilizes model optimization [30]. In addition, all images are then converted into a three-channel format. Given the original image resolution of $2588\times 1958$, all images are resized to a fixed resolution of $512\times 512$ to ensure a consistent input size. To avoid geometric distortion, a canvas-based resizing strategy is adopted, where each image is isotropically scaled according to its aspect ratio and subsequently zero-padded to the target resolution. During training, data augmentation is further applied to enhance generalization. The augmentation pipeline includes random horizontal flipping, affine transformations with small rotations and scaling, as well as color jittering to simulate variations in illumination and imaging conditions.

Time-Aware Phase-Level Contrastive Learning. RetCop [10] is adopted as the visual tower due to its strong FFA representation capability, while BioClinicalBERT [31,32] is used as the textual tower to model medical semantics in angiographic reports. The parameters of visual tower and textual tower are frozen throughout training. The phase-level adapters adopt the multi-layer gated residual adapter. Only the phase-level visual adapters, phase-level textual adapters, and the temperature parameter in contrastive learning are optimized. Phase-level knowledge injection is progressively introduced using a scaling factor $\gamma$, initialized at 0.2 and increased by 0.2 after each training cycle for a total of three cycles. The experimental results are shown in Table 1.

Table 1. Performance comparison of different adapter configurations under time-aware phase-level contrastive learning in FFA modality. r denotes the bottleneck reduction ratio and n denotes the number of adapter layers. AUC indicates results without time-aware phase-level contrastive learning, while AUC* indicates results with the proposed time-aware phase-level contrastive learning.

Method	Number of Parameters	AUC (%)	AUC* (%)
$r=16$, $n=1$	75,265	66.884	89.225
$r=16$, $n=3$	222,723	64.986	91.478
$r=16$, $n=6$	443,907	66.830	92.784
$r=32$, $n=1$	38,401	66.841	87.954
$r=32$, $n=3$	112,131	65.966	91.594
$r=32$, $n=6$	222,723	66.986	92.673
$r=64$, $n=1$	19,969	69.816	89.231
$r=64$, $n=3$	56,835	68.086	91.219
$r=64$, $n=6$	112,131	68.287	91.952

Semantic-Aware Multi-Phase Aggregation. At this stage, we select the phase-level visual adapter that achieves the best performance in the previous stage (AUC = 92.784%) as the knowledge transmission. The parameters of the phase-level visual adapters are frozen, except for the gating parameters in their final layers. The class-level adapter also adopts the multi-layer gated residual adapter. Both the class-level visual and textual adapters are optimized to through class-level contrastive learning. The experimental results are shown in Table 2.

Table 2. Performance comparison of different class-level adapter configurations for semantic-aware multi-phase aggregation in FFA modality. r denotes the bottleneck reduction ratio and n denotes the number of adapter layers.

Method	ACC (%)	AUC (%)
$r=8,n=1$	85.147	93.184
$r=8,n=3$	86.452	94.732
$r=16,n=1$	85.234	93.734
$r=16,n=3$	86.114	94.164

Methods for Comparison. For a comprehensive comparison, we evaluate multiple representative models under a unified experimental setting. The baselines include single-modality ResNet [22] and ViT [33] models trained for multi-label disease classification, as well as models fine-tuned with RetCoP and LoRA [34,35] to incorporate domain-specific FFA representations.

Temporal Modeling Settings. To investigate the impact of temporal modeling in FFA analysis, we conduct a comparative study under both time-unaware and time-aware settings. In the time-unaware setting, all FFA images from a patient are treated as independent samples and assigned the same disease label, without considering the temporal progression across angiographic phases. In contrast, the time-aware setting explicitly incorporates sequential information from FFA image series. For baseline methods, temporal dependencies are modeled by encoding the FFA sequence into a unified feature representation using the LSTM mechanism described in Formula (14). Specifically, the image sequence is processed in chronological order, and the hidden state at each time step captures the accumulated information of the sequence. We adopt the hidden state at the final time step as the representative feature vector of the entire FFA sequence, which summarizes the temporal dynamics and is used for subsequent training. The experimental results are shown in Table 3.

Table 3. Performance comparison of different methods under time-unaware and time-aware settings in FFA modality.

Method	Without Time-Aware		With Time-Aware
	ACC (%)	AUC (%)	ACC (%)	AUC (%)
ResNet-50	75.794	76.538	81.52	89.475
ViT-Base/16	72.164	74.371	85.203	92.161
RetCoP + LoRA	69.231	71.375	86.283	91.465
T-CASP(Ours)	68.157	72.249	86.452	94.732

Evaluation Metrics. To provide a comprehensive evaluation of model performance, we report accuracy (ACC) and area under the ROC curve (AUC) as primary metrics. ACC measures the overall proportion of correctly classified samples, while AUC evaluates the model’s discriminative ability across different decision thresholds. In addition, we include Kappa and specificity as complementary metrics. Kappa measures the agreement between predicted and ground-truth labels beyond chance, offering a more reliable assessment under class imbalance [36]. Compared with ACC, which can be biased toward majority classes in imbalanced datasets, Kappa accounts for the agreement expected by chance and thus provides a more balanced evaluation [37]. Specificity quantifies the true negative rate, which is particularly important in medical diagnosis for evaluating the model’s ability to correctly identify non-diseased cases and reduce false positives [38]. The experimental results are shown in Table 4.

Table 4. Performance comparison of different methods under time-aware settings in FFA modality.

Method	ACC (%)	AUC (%)	Specificity (%)	Kappa
ResNet-50	81.5	89.4	65.2	0.48
ViT-Base/16	85.2	92.1	68.7	0.51
RetCoP + LoRA	86.3	91.5	73.5	0.71
T-CASP(Ours)	86.5	94.7	78.8	0.74

Visualization of Disease Correlation Matrix. To further analyze the effectiveness of the proposed method and highlight the role of the disease correlation matrix during training, we visualize the learned matrix to provide intuitive insights into inter-disease relationships. We further conduct a case study on two clinically related diseases, Diabetic Retinopathy (DR) and Severe Non-Proliferative Diabetic Retinopathy (Severe NPDR), which exhibit high similarity. These diseases share similar pathological features in the arterial phase, while showing significant differences in the venous phase. Based on this observation, as shown in Figure 6, we visualize the correlation matrices under arterial and venous phases. For clarity, we zoom into the regions corresponding to these two diseases and annotate their similarity scores.

Figure 6. Visualization of disease correlation matrices under different phases: (a) disease correlation matrix under the arterial phase; (b) disease correlation matrix under the venous phase. Zoomed-in regions highlight the similarity between Diabetic Retinopathy (DR) and Severe Non-Proliferative Diabetic Retinopathy (Severe NPDR).

4. Discussion

As shown in Table 1, compared with the phase-agnostic setting, introducing temporal modeling with LSTM leads to significant AUC gains, with improvements of up to +26.4%. Moreover, time-aware phase-level contrastive learning consistently improves performance across all adapter configurations, highlighting the importance of explicitly modeling inter-phase temporal dependencies in FFA.

As shown in Table 2, introducing class-level aggregation further improves performance, demonstrating that semantic-aware multi-phase aggregation is beneficial for FFA-based disease recognition. Compared with the best phase-level result (AUC = 92.784%), class-level aggregation consistently yields additional gains, achieving a maximum AUC of 94.732%. Notably, deeper class-level adapters tend to produce more pronounced improvements, suggesting that richer semantic integration across phases contributes to more discriminative disease representations.

As shown in Table 3, when temporal information is not considered, all methods exhibit relatively limited performance, underscoring the inherent difficulty of FFA-based disease recognition without modeling angiographic dynamics. In this time-unaware setting, single-modality models trained from scratch achieve comparatively competitive results, as they are not constrained by cross-phase alignment. However, once temporal modeling is introduced, all methods benefit from sequential information, with the proposed T-CASP achieving the most significant performance gains.

Compared to single-modality baselines, T-CASP attains superior performance while introducing substantially fewer additional parameters, demonstrating a more parameter-efficient utilization of temporal information. Moreover, T-CASP outperforms models fine-tuned with RetCoP and LoRA, indicating that explicitly modeling fine-grained, phase-level temporal dependencies is more effective than generic adaptation strategies. These results confirm that phase-level fine-grained temporal modeling plays a crucial role in capturing disease-specific angiographic evolution and leads to more robust and discriminative representations.

As shown in Table 4, although single-modality models (ResNet-50 and ViT-Base/16) achieve competitive performance in terms of ACC and AUC, their specificity remains relatively low. This indicates that a considerable number of normal samples are misclassified as diseased cases, reflecting limited reliability in distinguishing healthy from pathological conditions. Such a phenomenon can be attributed to the intrinsic temporal characteristics of FFA sequences, where disease manifestations do not consistently appear at every time point but are often only evident in specific angiographic phases. Consequently, models that ignore phase-wise dynamics tend to overfit to partial or ambiguous features, leading to increased false positives. This observation further highlights the importance of phase-wise modeling in the proposed T-CASP framework, which enables fine-grained learning across different temporal stages. From the perspective of Kappa, a similar trend can be observed. Single-modality models yield relatively low Kappa values, indicating limited agreement with ground-truth labels beyond chance, particularly under class-imbalanced conditions. In contrast, T-CASP achieves a significantly higher Kappa score, demonstrating improved robustness and reliability. It is worth noting that our dataset exhibits severe class imbalance, where common diseases, e.g., NPDR contain hundreds of samples, while rare conditions may only include a few dozen cases.

The improved Kappa performance of T-CASP can be attributed to two key factors. First, the use of pretrained models provides strong prior knowledge learned from large-scale data, which helps alleviate the long-tail effect. Second, T-CASP does not rely solely on image-label supervision but incorporates a disease knowledge dictionary constructed by large language models. This knowledge-driven mechanism provides structured and phase-specific descriptions for each disease, ensuring more balanced semantic representation even for rare categories, thereby mitigating data imbalance and improving agreement.

Furthermore, we compare T-CASP with parameter-efficient fine-tuning methods such as LoRA. Both approaches are built upon the same RetCoP pretrained backbone, ensuring a fair comparison. While the two methods achieve comparable performance in terms of ACC and Kappa, T-CASP demonstrates clear advantages in AUC and specificity. This suggests that although pretrained models already enhance recognition performance, simply adapting them through low-rank updates is insufficient for capturing fine-grained disease characteristics. In contrast, T-CASP explicitly models aspect-level semantic information and performs fine-grained decomposition and aggregation across phases, enabling the model to better capture discriminative features. As a result, it achieves superior performance in distinguishing normal and pathological cases, particularly in terms of reducing false positives and improving clinical reliability.

As shown in Figure 6, further analysis is conducted on the visualization of the disease correlation matrices. In the arterial phase, the similarity between DR and Severe NPDR is relatively high (0.72), whereas in the venous phase, it decreases significantly to 0.2. This observation indicates that the two diseases are difficult to distinguish in the arterial phase, where their imaging features are highly similar. In contrast, during the venous phase, as fluorescein dye progressively fills and diffuses through the retinal vasculature, more discriminative features emerge, such as vascular leakage, capillary non-perfusion, and microvascular abnormalities, enabling clearer differentiation between the two diseases. From the perspective of Kappa, a similar trend can be observed. Single-modality models yield relatively low Kappa values, indicating limited agreement with ground-truth labels beyond chance, particularly under class-imbalanced conditions. In contrast, T-CASP achieves a significantly higher Kappa score, demonstrating improved robustness and reliability. It is worth noting that our dataset exhibits severe class imbalance, where common diseases such as NPDR contain hundreds of samples, while rare conditions may only include a few dozen cases.

This phenomenon further highlights the importance of phase-wise modeling. In the arterial phase, DR and Severe NPDR exhibit highly similar imaging characteristics, such as microaneurysms and early leakage patterns. Under such conditions, treating the task as a conventional multi-label classification problem based on a single phase forces the model to assign different disease labels to samples with nearly identical visual features, leading to semantic conflicts during training. Specifically, similar feature representations are simultaneously mapped to different disease categories, which increases the difficulty of optimization and may result in unstable decision boundaries and degraded generalization performance. From a clinical perspective, such a formulation also fails to reflect the progressive nature of FFA imaging, where disease characteristics evolve over time. In contrast, by modeling different phases as distinct semantic aspects, the proposed framework allows the model to first capture shared features in the arterial phase and then progressively learn discriminative features in later phases, such as the venous phase. This phase-aware design effectively alleviates feature-label conflicts and enables more fine-grained and clinically consistent disease representation learning.

Future research will further extend the proposed T-CASP framework along several complementary directions.

First, we extend the CASP framework to the CFP modality and further investigate its effectiveness in fine-grained ophthalmic tasks. In this setting, spatial structural information is modeled as multiple semantic aspects, enabling the framework to capture fine-grained lesion characteristics and validate the generality of CASP beyond phase-based representations. Moreover, we explore the integration of modality-specific adapters from CFP and FFA into a unified model for cross-modal learning. Preliminary results indicate that the proposed adapter-based architecture effectively mitigates catastrophic forgetting during incremental learning, allowing the model to retain previously learned knowledge while incorporating new modality-specific information.

Second, we plan to train domain-specific adapters across a broader range of medical domains and imaging tasks and to integrate them into a federated learning paradigm for medical AI [39,40]. In this setting, we will further explore the feasibility of applying CASP under a globally shared pretrained backbone, while multi-layer gated residual adapters are trained locally on private institutional data. During federated aggregation, only lightweight adapter representations are exchanged and adaptively gated, enabling collaborative learning without exposing sensitive patient data.

In addition, although LLM acquire extensive static and encyclopedic knowledge through pre-training, they lack dynamic, task-oriented working memory and struggle to continually assimilate new task-specific semantics. In future work, we plan to leverage the T-CASP framework’s knowledge injection and continual learning to achieve targeted knowledge memorization, whereby semantic information is selectively distilled from pre-trained models [41] and adaptively injected into adapters, and the continual learning architecture enables progressive updating and refinement of knowledge across sequential tasks.

Finally, the proposed knowledge injection strategy still has certain limitations. In this paper, the coefficient $\gamma$ is manually designed to increase linearly from 0.1 to 0.7 during training, which serves as a heuristic mechanism to gradually shift the learning paradigm from data-driven alignment to knowledge-guided optimization. Although this strategy empirically stabilizes training and alleviates potential mismatches introduced by the LLM-generated disease correlation matrix, its scheduling scheme remains manually defined and may not be optimal for all datasets or training scenarios. In future work, more adaptive strategies could be explored to improve this process.

5. Conclusions

This study investigates the utilization of temporal information in FFA and presents a time-aware learning framework (T-CASP) tailored to angiographic sequences. To address the limited disease descriptions in initial diagnostic reports, we first employ large language models to construct the temporal knowledge dictionary that provides structured, phase-specific semantic descriptions. Subsequently, we propose the knowledge injection mechanism that builds a disease correlation matrix and transfer the disease correlation matrix, which includes disease-level prior knowledge, to the batch-level contrastive learning matrices.

To mitigate the mismatch between the initial knowledge and dataset-specific disease relationships, we adopt a phase-wise continual pretraining strategy that updates the disease correlation matrix progressively during training. Within this framework, fine-grained image–text alignment is iteratively performed, consisting of time-aware phase-level contrastive learning at each angiographic phase and semantic-aware multi-phase aggregation for class-level image–text alignment. To support robust and incremental knowledge transfer, we design the multi-layer gated residual adapter for both visual and textual encoders.

Experimental results demonstrate that the proposed approach effectively leverages dynamic temporal semantics in FFA sequences, leading to improved accuracy and robustness in retinal disease recognition.