TDA-Phys: Temporal Difference Adaptation of Video Foundation Model for Remote Photoplethysmography

Featured Application

This work demonstrates the potential of general-purpose video foundation models for non-contact physiological monitoring, which could support future applications in telehealth, smart environments, and human–computer interaction under unconstrained conditions.

Abstract

Remote photoplethysmography (rPPG) enables noncontact estimation of vital signs, particularly heart rate, by analyzing subtle periodic skin color variations in facial videos. While deep learning has advanced rPPG signal extraction, existing methods rely on carefully designed task-specific architectures that are costly to develop and generalize poorly. In this work, we demonstrate that the general video foundation model VideoMAE v2 can be effectively adapted to the rPPG signal regression task by introducing only a lightweight adapter, without modifying its pretrained backbone. We freeze the entire VideoMAE v2 encoder and introduce a Temporal Difference Convolutional Adapter to capture the subtle interframe intensity differences. To address the mismatch between VideoMAE v2′s short input window (16 frames) and the long temporal context typically required for robust rPPG extraction (e.g., 160 frames), we adopt an overlapping sliding window strategy for segmented inference and reconstruct the full signal through weighted temporal aggregation. On the COHFACE and UBFC-rPPG datasets, our method achieves mean absolute errors (MAEs) of 0.90 and 1.55, reducing the error by more than 55% and 42%, respectively, compared to PhysFormer (2.00 and 2.70). Furthermore, on challenging real-world datasets such as BUAA-MIHR, which features strong illumination variations, and VIPL-HR, which involves significant head movements, our approach achieves MAEs of 6.68 and 8.23, respectively, despite incorporating no task-specific robustness modules. These results demonstrate stable rPPG signal recovery and validate the feasibility of leveraging general video foundation models for physiological signal perception.

1. Introduction

Non-contact physiological parameter monitoring, as a crucial sensing technology, holds broad application prospects in fields such as telemedicine, intelligent human–computer interaction, and daily health monitoring [1]. Among these, remote photoplethysmography (rPPG) analyzes subtle changes in facial skin color caused by blood volume pulsations to enable contactless, unobtrusive estimation of key physiological indicators such as heart rate and respiratory rate, showcasing unique advantages [2,3,4]. However, rPPG signals are extremely weak and highly susceptible to interference from factors such as environmental lighting variations, head movements, skin tone differences, and low signal-to-noise ratios. This often leads to degraded signal quality, resulting in unstable estimations of physiological parameters. Enhancing its robustness and reliability remains a core challenge in current research [5,6].

In recent years, deep learning methods have brought breakthrough advancements to rPPG—a non-contact technique that estimates physiological signals by analyzing subtle, periodic skin color variations in facial videos caused by blood volume pulsations. Current rPPG systems typically employ end-to-end neural networks to directly regress rPPG waveforms from cropped and aligned face video sequences. Deep learning approaches such as PhysNet [2], PhysFormer [3], and DeepPhys [4] have significantly improved the accuracy and stability of signal estimation, making them the mainstream solutions today. However, these methods typically rely on carefully designed neural network architectures to explicitly model physiological priors (such as heart rate range, temporal periodicity, etc.) and require extensive training from scratch on large amounts of labeled rPPG data, leading to high model development costs and limited generalization capabilities.

With the rise of vision foundation models, the paradigm of large-scale self-supervised pre-training has shown strong potential for learning general spatiotemporal representations. Video foundation models are large neural networks pre-trained on extensive unlabeled video data, learning to understand both motion and appearance without task-specific labels. For example, VideoMAE v2 [7] uses a masked reconstruction strategy to train on massive video collections. It learns strong spatiotemporal features that generalize well across different downstream tasks—such as action recognition and temporal action detection. By leveraging the powerful spatiotemporal feature extraction capabilities learned during the pre-training of these models, and by introducing only lightweight adapter modules followed by fine-tuning on the target task, they can be effectively adapted to various downstream tasks—such as rPPG signal extraction. Building on this, our work explores the feasibility of applying pre-trained video foundation models to rPPG, aiming to advance non-contact physiological monitoring toward greater practicality and robustness.

This paper proposes a video foundation model-based adapter fine-tuning framework, TDA-Phys, to adapt to downstream rPPG signal extraction tasks. We select VideoMAE v2, pre-trained via self-supervision on large-scale video data, as the base model. While keeping the parameters of its backbone network frozen, we introduce a Temporal Difference Convolutional Adapter to capture the fine-grained temporal dynamics required for rPPG tasks. The model is fine-tuned end-to-end using a composite time-domain and frequency-domain loss, balancing signal waveform fidelity and physiological spectral characteristics. Furthermore, considering that VideoMAE v2 only accepts inputs of 16 frames while real-world facial video samples often contain hundreds of frames (e.g., 160 frames), we employ a sliding window strategy to partition a 160-frame long-sequence facial video into multiple overlapping 16-frame short clips for segment-wise inference. The final continuous and stable rPPG signal is reconstructed through a temporal aggregation mechanism.

The main contributions of this paper are summarized as follows:

• We show that a general video foundation model (VideoMAE v2) can be effectively adapted to rPPG signal extraction with only a lightweight adapter and no modification to its pretrained backbone. To the best of our knowledge, this is the first successful application of a generic video foundation model to end-to-end rPPG regression.
• We introduce a Temporal Difference Convolutional Adapter. By freezing the backbone parameters of the VideoMAE v2 foundation model and fine-tuning only the parameters within this adapter, our method effectively adapts to the downstream task of rPPG signal extraction.
• To address the limitation that VideoMAE v2 only processes short 16-frame input sequences, we employ a sliding window strategy. This strategy divides long video sequences into overlapping short clips for segment-wise inference, followed by a temporal aggregation mechanism to reconstruct continuous and stable rPPG signals.
• We conduct comprehensive validation experiments on the COHFACE, UBFC-rPPG, BUAA-MIHR, and VIPL-HR datasets. The results demonstrate that the proposed fine-tuning architecture for video foundation models can effectively adapt to the rPPG signal extraction task and exhibits promising generalization ability. This work robustly validates the potential of large-scale video foundation models for physiological signal perception tasks.

The rest of this paper is organized as follows: Section 2 reviews deep learning-based rPPG methods as well as related work on foundation models and adapter fine-tuning. Section 3 elaborates on the overall framework of the TDA-Phys method, including the sliding window strategy and the spatiotemporal difference convolutional adapter. Section 4 presents experimental results, comparative analysis, and ablation studies on multiple datasets. Finally, Section 5 summarizes the work of this paper and discusses future research directions.

2. Related Work

2.1. Deep Learning-Based rPPG Methods

Early rPPG methods primarily relied on traditional signal processing techniques, which can be broadly categorized into blind source separation (BSS)-based approaches [8,9,10] and model-based methods [11,12,13,14]. In recent years, deep learning has significantly advanced the field of remote photoplethysmography. Spetlík et al. [15] proposed a two-stage CNN architecture consisting of a feature extractor and an HR estimator. The feature extractor is trained to maximize the signal-to-noise ratio for extracting rPPG signals from video sequences, which are then fed into the HR estimator to obtain heart rate. This marked the first deep learning approach for rPPG. Inspired by HR-CNN, Chen et al. [4] introduced DeepPhys, which employs a dual-branch 2D CNN architecture to separately model appearance and motion information. The motion branch explicitly captures subtle motions and color variations caused by blood flow by using normalized differences between adjacent frames as input, while the appearance branch incorporates an attention mechanism that adaptively focuses on facial regions with stronger physiological signals through a learnable soft attention mask. To address the lack of temporal information in 2D CNNs, Liu et al. [16] proposed MTTS-CAN, which builds upon DeepPhys by introducing a temporal shift module to capture temporal dependencies. Niu et al. [17] introduced SynRhythm based on the concept of spatiotemporal maps, emphasizing HR information while suppressing irrelevant signals. They employed transfer learning to train the HR estimator and transferred the pre-trained model to real-world HR estimation tasks. Song et al. [18] explored the idea of constructing spatiotemporal maps based on rPPG signals. They first used the CHROM method to extract preliminary rPPG signals from facial ROIs and then generated more structured and motion-robust spatiotemporal maps based on these signals. These maps were subsequently used as input for CNN-based learning, thereby explicitly incorporating signal priors into the graph-structured modeling process.

These 2D CNN methods struggle to effectively model the temporal contextual dependencies between facial video frames when handling rPPG tasks. Although constructing spatiotemporal maps based on rPPG priors introduces some temporal information to a certain extent, their modeling of dynamic physiological signals remains insufficient. In contrast, 3D CNNs can simultaneously capture spatial structure and temporal features, aligning with the spatiotemporal characteristics of rPPG signals, thus demonstrating stronger feature extraction capabilities in rPPG signal extraction tasks. Bousefsaf et al. [19] proposed the first 3D CNN-based rPPG method, marking an initial exploration of temporal modeling in non-contact physiological signal extraction. This approach treats raw videos as continuous frame sequences and directly feeds them into a 3D CNN backbone network without relying on preprocessing steps. Through end-to-end learning, the 3D CNN jointly learns spatiotemporal features from unprocessed video streams and feeds the extracted high-level representations into a Multilayer Perceptron (MLP) for heart rate regression, showcasing the potential of 3D convolutions in capturing spatiotemporal features for rPPG tasks. TransRPPG [20] first introduced Vision Transformer to rPPG-related tasks by extracting rPPG features from preprocessed signal maps for 3D facial mask attack detection, successfully validating the effectiveness of the Transformer architecture in modeling rPPG signals and preliminarily demonstrating its potential in this field. Leveraging the advantages of Transformers, Yu et al. [3] proposed a novel rPPG method for heart rate measurement called Physformer, whose core lies in its designed temporal difference Transformer module. By modeling inter-frame difference signals, this module effectively explores long-range spatiotemporal dependencies. The architecture not only supports fine-grained modeling of local temporal dynamics but also enables joint modeling of global spatiotemporal contexts. The global attention mechanism enhances the extraction of periodic rPPG features and refines local spatiotemporal representations in the presence of motion or illumination interference, thereby achieving robust and globally consistent local rPPG features.

2.2. Foundation Models and Adapter-Based Transfer Learning

Unlike task-specific models, vision and video foundation models, such as Vision Transformer (ViT) [21], Masked Autoencoders [22], and their video extension VideoMAE [23], have demonstrated powerful general spatiotemporal representation capabilities through self-supervised pre-training on massive unlabeled data, achieving excellent performance in tasks such as action recognition and video classification. The core advantage of such models lies in their “pre-train then fine-tune” paradigm: via the transfer of general-purpose representations, they can effectively adapt to various downstream tasks.

To efficiently adapt these large-scale models to downstream tasks while avoiding the substantial costs associated with full-parameter fine-tuning, Parameter-Efficient Fine-Tuning (PEFT) [24] techniques have emerged. Among them, Adapter-based fine-tuning strategies have been successfully applied in both natural language processing and computer vision. Methods such as TS-SAM [25], for instance, freeze the backbone parameters of the large model and insert only lightweight adapters, training a minimal set of additional parameters. This enables the foundation model to rapidly adapt to new downstream tasks, significantly reducing computational and storage overhead while preserving the feature extraction capabilities of the pre-trained model.

However, existing research has yet to explore the transfer potential of such general-purpose video foundation models and their fine-tuning strategies to physiological signal extraction tasks like rPPG, which are highly sensitive to subtle spatiotemporal variations. To address this, we introduce a general video foundation model (VideoMAE v2) along with a parameter-efficient adapter-based fine-tuning mechanism into the rPPG domain. Our aim is to leverage its powerful spatiotemporal feature extraction capabilities, learned through large-scale pre-training, to tackle the fundamental challenge of limited generalization in current task-specific models.

3. Materials and Methods

3.1. Overall Network Framework

This paper proposes Temporal Difference Adaptation of Video Foundation Model for Remote Photoplethysmography (TDA-Phys), a parameter-efficient framework for rPPG signal extraction. Our approach is built upon the video foundation model VideoMAE v2, whose pretrained backbone is kept entirely frozen. For a 160-frame input video, we employ a sliding window strategy to generate 19 overlapping segments of 16 frames each.

Each segment is processed by the frozen VideoMAE v2 encoder. Specifically, after the initial patch embedding and linear projection, the token sequence is fed through the Vision Transformer Block (ViT Block). The TDC Adapter is integrated in parallel with selected ViT Blocks. At each integration point, the output from the preceding ViT Block is first fused with the refined feature from the previous TDC Adapter (or used directly for the first adapter) to form a combined representation. This combined representation is directly passed as input to the next ViT Block, preserving the main information flow of the frozen backbone. Simultaneously, this same combined representation is fed into the current TDC Adapter, which processes it to explicitly model rPPG-relevant temporal dynamics. The adapter then produces a compressed feature that serves as the input to the next TDC Adapter. Through this cascaded, co-propagating design, the TDC Adapters progressively refine the features along the temporal dimension, enhancing sensitivity to subtle, periodic skin color variations while preserving the general spatiotemporal knowledge encoded in the frozen backbone.

Only the parameters of the TDC Adapters and the final regression head are updated during training. The model outputs a 16-frame rPPG prediction for each segment, and the full 160-frame signal is reconstructed through weighted averaging of all overlapping predictions. The complete architecture is illustrated in Figure 1.

3.2. Sliding Window Strategy

Video foundation models are typically designed to process short video clips, with standard input sequences being relatively brief in the temporal dimension. For instance, VideoMAE v2 takes a 16-frame sequence as input. However, reliably extracting rPPG signals and estimating physiological parameters such as heart rate from facial videos often requires analyzing longer sequences spanning hundreds of frames (e.g., 160 frames). Specifically, physiological signals like heart rate operate within a specific frequency range, and a sufficiently long time window is necessary to achieve effective resolution in their frequency-domain representation. Shorter sequences can lead to increased errors in frequency estimation. Moreover, rPPG signals are inherently low-frequency, weak, and periodic. Short sequences struggle to stably capture meaningful periodic patterns. A 16-frame short sequence (at 30 fps, lasting only about 0.53 s) provides insufficient resolution in the frequency domain to accurately discern subtle variations within the normal heart rate range and may even miss a complete cardiac cycle entirely. Therefore, directly extracting rPPG signals from a 16-frame facial video sequence faces challenges due to inadequate temporal context and incomplete capture of signal cycles.

To address this issue, as illustrated in Figure 1a, for a given 160-frame facial video sequence with dimensions (1, 160, 3, 224, 224), we employ a sliding-window strategy. The input sequence length is 160 frames, with a window size of 16 frames and a stride of 8 frames, resulting in 19 windows. Adjacent windows overlap by 8 frames. This sliding window configuration covers the entire sequence, with the last window ending precisely at the 160th frame, without requiring padding or truncation. These segmented short sequences are then fed sequentially into the model for segment-wise inference, producing an output sequence of 19 short rPPG signal segments, each of length 16.

The window size was set to 16 frames to align with the input length of VideoMAE v2, thereby fully leveraging its pre-trained weights. The total sequence length was set to 160 frames following common practice in the rPPG field, ensuring sufficient duration at typical video frame rates to achieve adequate frequency-domain resolution for reliable heart rate range estimation. As for the stride size, a smaller stride helps generate smoother output signals during aggregation and reduces boundary artifacts caused by segmented inference, while a larger stride reduces the number of segments to be processed, thereby lowering computational cost. We chose a stride of 8 as a balanced trade-off between temporal continuity and computational efficiency.

To reconstruct the complete 160-frame long signal, we adopt an overlap-average aggregation method. As shown in Algorithm 1, each 16-frame segment prediction is placed back into its corresponding position in the full-length sequence. Since adjacent windows overlap (with a stride of 8 frames), most time points are covered by multiple predictions. We accumulate all overlapping predictions and then normalize each time point by the number of segments that contributed to it. This yields a smooth, continuous rPPG waveform over the entire 160-frame sequence.

Algorithm 1. Pseudocode for temporal aggregation

rPPG_full = zeros(160)            # Accumulated signal weight_count = zeros(160)       # Coverage counter for i in range(19):                # i = 0 to 18     start = i * 8                        # Start index in original sequence     end = start + 16                  # End index (exclusive)     pred = model_output [i]       # Predicted rPPG segment (length 16)     rPPG_full [start:end] += pred     weight_count [start:end] += 1.0 rPPG_final = rPPG_full/weight_count  # Element-wise division

3.3. Temporal Difference Convolutional Adapter

This paper proposes a Temporal Difference Convolutional Adapter (TDC Adapter), which is inserted in parallel into the backbone network to guide the model in focusing on subtle differences between consecutive frames and emphasizing temporal features of periodic variations.

As illustrated in Figure 1c, the TDC Adapter consists of two Temporal Difference Convolution modules (TDConv). Each module is composed of a Central Difference Convolution, BatchNorm, and a ReLU activation function. The first TDConv module has an input channel size of 128 and an output channel size of 1408, aligning with VideoMAE v2. Its convolutional kernel size is 3 × 3 × 3, with a stride of 1 across all dimensions and padding of 1. The second TDConv module has an input channel size of 1408 and an output channel size of 128, with its kernel size, stride, and padding configurations identical to those of the first TDConv module. The first convolutional module expands the compressed features output by the TDC Adapter to match the feature dimension of the ViT Block output, which is then merged with the output features of the ViT Block to serve as the input for the next ViT Block layer. The second convolutional module compresses the merged features back to the input dimension of the TDC Adapter, which then becomes the input for the next TDC Adapter layer. Specifically, given the output of the (i − 1)-th ViT Block, denoted as $X_{Vit}^{i-1}$, and the output of the (i − 1)-th TDC Adapter, denoted as $X_{TDC}^{i-1}$:

$$X_{Vit}^i=X_{Vit}^{i-1}+TDConv\left(X_{TDC}^{i-1}\right)$$

(1)

$$X_{TDC}^i=TDConv\left(X_{Vit}^i\right)$$

(2)

where $X_{Vit}^i$ denotes the input to the i-th ViT Block, $X_{TDC}^i$ denotes the input to the i-th TDC Adapter, and TDConv represents the Temporal Difference Convolutional module within the adapter.

Vision Transformer networks typically exhibit the characteristic of encoding local details in shallow layers and global semantics in deeper layers. Considering that the later-stage ViT Blocks have already aggregated rich spatiotemporal contextual information, we inserted the adapters into the latter part of the backbone network to guide the model in focusing on subtle inter-frame differences.

The first adapter is positioned after the linear projection layer, mapping the original input to the adapter’s input dimension, thereby facilitating the subsequent parallel insertion of adapters into the backbone network. The next eight adapters are inserted after the 32nd to 39th ViT Blocks, respectively. The final adapter is placed after the 40th ViT Block, just before the signal average aggregation, serving to compress and optimize the features, ensuring that the output channel count aligns with the target signal dimensions. During fine-tuning, the parameters of the foundation model’s backbone ViT Blocks are frozen, and only the parameters of all adapter layers are updated.

4. Results and Discussion

4.1. Model Performance Evaluation Experiment

This paper evaluates model performance on the public datasets COHFACE [26], UBFC-rPPG [27], BUAA-MIHR [28], and VIPL-HR [29]. We adopt the three most commonly used evaluation metrics in the field of rPPG estimation. The Pearson correlation coefficient (r) measures the correlation between the predicted rPPG waveform and the ground-truth waveform, where a value closer to 1 indicates a stronger correlation. The Mean Absolute Error (MAE) quantifies the difference between the predicted heart rate and the true heart rate by calculating the average of absolute heart rate errors—a lower value indicates smaller errors. The Root Mean Square Error (RMSE) serves as a necessary complement to MAE and is more sensitive to outliers. A lower value suggests fewer outliers in heart rate estimation.

$$MAE={\displaystyle \frac{1}{n}}\sum _{i=1}^n\left|X_{HR}-X_{Truth}\right|$$

(3)

$$RMSE=\sqrt{{\displaystyle \frac{1}{n}}\sum _{i=1}^n{\left(X_{HR}-X_{Truth}\right)}^2}$$

(4)

The COHFACE dataset includes facial RGB videos and synchronized Blood Volume Pulse (BVP) signals from 40 subjects. The videos were recorded under stable conditions (with natural lighting and subjects at rest) at a resolution of 640 × 480 and a frame rate of 20 Hz. The UBFC-rPPG dataset contains 42 RGB videos recorded in stable scenarios. These videos have a frame rate of 30 frames per second and a resolution of 640 × 480. Ground truth BVP signals were recorded using a CMS50E pulse oximeter. The BUAA-MIHR dataset consists of 165 videos from 15 participants recorded under varying illumination conditions (100.0~102.0 lux), at a resolution of 640 × 480 and a frame rate of 30 fps. Synchronized PPG signals were measured using a CMS50E device. The VIPL-HR dataset contains a total of 2378 facial videos and synchronized BVP waveforms from 107 subjects, recorded across 9 scenarios (v1–v9). Model performance is evaluated on the v2 scenario, which requires subjects to perform substantial head movements.

For the videos in the dataset, all frames are first extracted. The retinaface algorithm is then used to detect and crop the facial regions in each frame. Subsequently, all facial images are uniformly resized to 224 × 224 pixels. This size is chosen because the VideoMAE v2 base model was pre-trained on large-scale video data with an input resolution of 224 × 224. Using this size ensures compatibility with the pre-trained weights, allowing full utilization of its pre-trained representational capacity and avoiding feature map distortion due to size mismatch. If no face is detected in a certain frame, it is replaced by the most recent valid frame to maintain temporal continuity. During the data loading stage, pixel-level normalization is applied to each facial region, mapping pixel values to the range of [−1, 1].

The proposed method is implemented using PyTorch on an NVIDIA A800 GPU with CUDA version 12.5. The training process employs the Adam optimizer with a learning rate of 1 × 10⁻⁴ and a weight decay of 5 × 10⁻⁵. The batch size is set to 1. After applying the sliding window partitioning, the input size becomes (1 × 19, 3, 16, 224, 224). A random seed of 42 is set, and the model is trained for 50 epochs. During training, the VideoMAE v2 backbone network is frozen, and only the adapter modules are fine-tuned. Horizontal mirror flipping is applied to the input sequence with a probability of 0.5. Training utilizes the same composite loss as PhysFormer, which is defined as follows:

$${\mathcal{L}}_{loss}=a·{\mathcal{L}}_{NegPearson}+b·\left({\mathcal{L}}_{freq}+{\mathcal{L}}_{KL}\right)$$

(5)

where ${\mathcal{L}}_{NegPearson}$ is the negative Pearson correlation coefficient, ${\mathcal{L}}_{freq}$ is the cross-entropy loss, and ${\mathcal{L}}_{KL}$ is the Kullback–Leibler divergence loss. For the first 30 epochs, the weight a is set to 1 and b to 0. From epoch 30 to epoch 50, both a and b are set to 0.5.

The total number of model parameters is 1.1 B, with trainable parameters amounting to 89.32 M, accounting for only 8.11% of the total. The total FLOPs per window is 2.733 TFLOPs, and the inference latency is 392 ms (16FP). During fine-tuning training, the memory usage is 54 GB, allowing the training to be completed on a single NVIDIA A800 GPU.

To validate the effectiveness of the proposed method in the downstream task of rPPG signal extraction, we first conducted evaluation experiments on two stable-scenario datasets, COHFACE and UBFC-rPPG, and compared it with typical traditional methods and deep learning approaches.

As shown in

Table 1. Results on Stable Scenario Dataset.

Dataset	Method	r	MAE	RMSE
COHFACE	ICA [9]	-	8.89	14.55
	CHROM [30]	-	7.80	12.45
	POS [14]	-	13.43	17.05
	HR-CNN [15]	0.52	8.10	10.80
	PhysNet [2]	0.54	8.63	9.36
	DeepPhys [4]	0.62	6.60	10.79
	Cpulse [31]	0.99	1.99	3.83
	PhysFormer [3]	0.99	2.00	2.00
	TDA-Phys(ours)	0.99	0.90 ± 0.03 *	1.22 ± 0.02 *
UBFC-rPPG	ICA [9]	-	6.43	11.43
	CHROM [30]	-	2.98	3.80
	POS [14]	-	3.99	6.81
	HR-CNN [15]	0.64	4.90	5.89
	DeepPhys [4]	0.65	2.90	3.63
	PhysNet [2]	0.97	2.38	3.19
	AND-rPPG [32]	0.99	2.67	4.07
	PhysFormer [3]	0.99	2.70	4.38
	TDA-Phys(ours)	0.99	1.55 ± 0.02 *	3.35 ± 0.05 *

* Values are mean ± std. Paired t-test vs. PhysFormer: p < 0.05 on both datasets.

, our method achieves state-of-the-art performance on both stable-scenario datasets. On COHFACE, TDA-Phys attains an MAE of 0.90 ± 0.03 and an RMSE of 1.22 ± 0.02, substantially outperforming recent deep learning approaches: it reduces the MAE by 54.8% compared to Cpulse (1.99) and by 55.0% compared to PhysFormer (2.00). The high correlation coefficient (r = 0.99) and low error metrics confirm its ability to recover high-fidelity rPPG waveforms under controlled conditions.

On UBFC-rPPG, our method achieves an MAE of 1.55 ± 0.02 and an RMSE of 3.35 ± 0.05. This represents a 34.9% reduction in MAE relative to PhysNet (2.38) and is also lower than PhysFormer (2.70). Notably, while the MAE improvement is significant, the RMSE (3.35) remains comparable to PhysNet (3.19), suggesting that our method consistently reduces average prediction bias (reflected in MAE) while maintaining similar sensitivity to larger outliers (as captured by RMSE).

These results demonstrate that TDA-Phys not only achieves competitive accuracy but also exhibits strong stability, as evidenced by the small standard deviations across runs. The consistent r = 0.99 across all top-performing methods on both datasets indicates that waveform shape similarity is now saturated under stable conditions; the key differentiator lies in absolute error magnitude, where our approach shows clear advantages.

Furthermore, given that the pre-trained video foundation model VideoMAE v2 possesses robust general spatiotemporal feature extraction capabilities and has demonstrated excellent generalization performance across multiple downstream tasks, we anticipate that after adapting it to the downstream task of rPPG signal extraction via adapter fine-tuning, it will still be able to generalize effectively and stably extract rPPG signals under challenging conditions such as varying illumination and head movements, even without incorporating specifically designed robustness modules.

We further evaluate our method on two challenging real-world datasets: BUAA-MIHR (strong illumination variations) and VIPL-HR (significant head movements). As shown in

Table 2. Results on datasets with illumination variations and head movements.

Dataset	Method	r	MAE	RMSE
BUAA-MIHR	ICA [9]	-	11.27	16.68
	CHROM [30]	-	12.99	17.80
	POS [14]	-	11.75	17.31
	Green [8]	-	12.76	17.89
	iBVPNet [33]	0.34	10.18	12.50
	PhysNet [2]	0.24	15.23	18.98
	TDA-Phys(ours)	0.62	6.68	12.75
	PhysFormer [3]	0.84	4.48	8.03
	HLFF [5]	0.89	3.19	6.50
VIPL-HR	POS [14]	-	11.50	17.20
	DeepPhys [4]	0.11	11.00	13.80
	PhysNet [2]	0.20	10.80	14.80
	TDA-Phys(ours)	0.54	8.23	16.02
	AutoHR [34]	0.71	5.68	8.68
	RhythmNet [35]	0.74	5.30	8.14
	PhysFormer [3]	0.78	4.97	7.79

, TDA-Phys achieves an MAE of 6.68 and RMSE of 12.75 on BUAA-MIHR, and an MAE of 8.23 and RMSE of 16.02 on VIPL-HR. These results consistently outperform traditional signal processing methods (e.g., CHROM, POS, Green) and early deep learning models such as DeepPhys, PhysNet, and iBVPNet by a large margin—reducing MAE by 34–56% compared to these baselines.

However, our method still lags behind the current state-of-the-art approaches: on BUAA-MIHR, HLFF achieves a substantially lower MAE of 3.19 compared to our 6.68, and on VIPL-HR, PhysFormer reports an MAE of 4.97 versus our 8.23. This performance gap is primarily attributed to the inherent limitations of our current design: TDA-Phys relies solely on a frozen general-purpose video foundation model and lightweight adapters, without incorporating any task-specific mechanisms to explicitly handle real-world challenges such as severe illumination changes or large head motions. In contrast, state-of-the-art methods such as HLFF and PhysFormer are explicitly designed with task-specific robustness mechanisms to handle real-world disturbances like illumination changes and motion artifacts. The performance gap observed in complex scenarios is largely due to the absence of such specialized components in our framework, which relies solely on a frozen general-purpose backbone and lightweight adapters. This limitation highlights a clear direction for future work.

To intuitively demonstrate the method’s performance under both simple and complex interference scenarios, we present comparative figures of the time-domain waveforms and power spectral density (PSD). These figures compare the ground-truth and predicted signals in representative stable and low-light scenarios.

As shown in Figure 2 and Figure 3, the model exhibits excellent performance under stable conditions: the predicted signal closely tracks the fluctuation trends of the ground-truth signal, demonstrating a high degree of agreement between the two. In contrast, under low-light conditions, the predicted signal not only displays multiple abnormal peaks but also completely deviates from the variation trends of the ground-truth signal over extended intervals, while its fluctuation amplitude is significantly larger than that of the ground-truth signal. This reflects the current method’s limitations in adapting to complex lighting conditions.

As shown in Figure 4, the comparative analysis of Power Spectral Density (PSD) indicates that the model demonstrates relatively good performance under stable conditions (a). The predicted signal accurately captures the primary frequency location and maintains a peak morphology similar to that of the ground-truth signal. In contrast, the model’s performance degrades significantly in the low-light scenario (b). The predicted signal exhibits a shift in the primary frequency and a noticeable attenuation in amplitude, reflecting that non-physiological noise induced by lighting variations severely interferes with the model’s spectral representation of the heart rate signal.

4.2. Ablation Study

4.2.1. Impact of the Sliding Window Strategy

Considering the mismatch between the sensitivity of rPPG signal extraction to temporal length and the inherent 16-frame input limitation of VideoMAE v2, we introduce a sliding-window strategy to reconstruct long physiological signals. To evaluate the effectiveness of this strategy, we compare its impact on model performance with that of a non-overlapping concatenation method. Specifically, the 160-frame facial video sequence is divided into 10 non-overlapping 16-frame segments, which are fed into the model for segment-wise inference. The resulting ten short rPPG signals, each of length 16, are then directly concatenated to form the complete rPPG signal.

Figure 5 and Figure 6 show the training dynamics of our model when using a non-overlapping concatenation strategy. The Pearson correlation coefficient, plotted in Figure 5, remains consistently low throughout all 30 epochs, fluctuating around 0.05 with no sign of improvement. Similarly, the MAE shown in Figure 6 drops rapidly during the initial epochs but quickly stabilizes at approximately 15 and exhibits only minor fluctuations thereafter. This convergence behavior indicates that the model fails to learn meaningful physiological patterns from the input.

The underlying reason is the temporal fragmentation caused by non-overlapping segmentation. Each 16-frame segment is processed independently, which deprives the model of the long-range temporal context required to capture the low-frequency periodicity of rPPG signals (typically around 1 Hz). As a result, the optimization converges to a trivial solution, such as predicting a near-constant value or uncorrelated noise. This leads to both the near-zero correlation and high absolute error observed in the metrics. In contrast, the overlapping sliding window strategy used in our main experiments provides consistent temporal overlap across segments, enabling the model to learn coherent physiological dynamics and achieve stable, high-performance convergence as reported in Table 1.

4.2.2. Impact of the Temporal Difference Convolutional Adapter

To evaluate the contribution of the proposed Temporal Difference Convolutional Adapter to the overall model performance, we compare the results of inserting different numbers of adapters.

As shown in Figure 7, without any adapter, fine-tuning only the output head yields relatively high MAEs (6.86 on COHFACE, 8.32 on UBFC-rPPG). This baseline confirms that while the frozen VideoMAE v2 backbone provides useful general spatiotemporal features, a direct linear projection is insufficient for the precise regression of rPPG signals, highlighting the need for a dedicated adaptation mechanism.

Introducing 4 adapters substantially reduces the MAE, demonstrating the effectiveness of the proposed TDC modules in modulating the foundation model’s features for the target task. Expanding to 8 adapters leads to a further significant performance gain, suggesting that distributing temporal modeling capacity across more stages of the transformer backbone allows for a finer-grained capture of the periodic physiological dynamics throughout the feature hierarchy.

The optimal performance is achieved with 10 adapters, strategically inserted from the middle to the final layers of the VideoMAE v2 encoder (as detailed in Section 3.3). This configuration appears to strike a balance: it provides sufficient task-specific modulation without overwhelming or distorting the pre-trained representations. The observation that adding adapters beyond 10 yields only marginal improvements suggests a point of saturation, where the representational capacity for this specific task is adequately addressed, and additional parameters may not contribute to further disentangling the rPPG signal from noise.

5. Conclusions

This work adapts the general-purpose video foundation model VideoMAE v2 to the task of remote photoplethysmography (rPPG) signal regression. To reconcile the model’s native 16-frame input window with the 160-frame sequences typically used for rPPG, we employ an overlapping sliding window strategy for segmented inference and reconstruct the full signal through weighted temporal aggregation. A Temporal Difference Convolutional Adapter is inserted into the frozen VideoMAE v2 encoder, and only this adapter along with a regression head are fine-tuned.

On the COHFACE and UBFC-rPPG datasets, this approach achieves MAEs of 0.90 and 1.55, matching recent specialized deep learning models. On the challenging real-world datasets BUAA-MIHR, which features strong illumination variations, and VIPL-HR, which involves significant head movements, it obtains MAEs of 6.68 and 8.23, respectively, despite incorporating no task-specific robustness modules.

To the best of our knowledge, this is the first successful application of a generic video foundation model to end-to-end rPPG signal regression. The proposed framework provides a reference approach for rPPG research.

In addition, VideoMAE v2 is trained via self-supervised pre-training on large-scale unlabeled videos, where its learning objective focuses on high-level semantics and salient visual changes. In contrast, the rPPG task requires extracting periodic physiological signals from extremely subtle color variations. Under strong lighting or intense motion interference, these weak signals are easily overwhelmed. The current adapter is designed to guide the model in capturing temporal differences, but its relatively simple structure lacks sufficiently sophisticated mechanisms to dynamically suppress such interference. In future work, the robustness of the model could be improved from the following aspects:

• Introducing task-aware adapter designs. For example, embedding illumination-invariant modules or motion compensation modules into shallow ViT Blocks to suppress major interference at an early stage.
• Exploring multi-scale spatiotemporal adaptation mechanisms. Integrating local and global contexts through multi-branch adapters to separately model high-frequency noise and low-frequency physiological signals.
• Constructing an rPPG dataset with controlled motion and lighting variations to conduct targeted pre-training for the adapter or the entire model, thereby enhancing its generalization capability in real-world complex scenarios.