Contextual Feature Fusion-Based Keyframe Selection Using Semantic Attention and Diversity-Aware Optimization for Video Summarization

Abstract

Training-free video summarization tackles the challenge of selecting the most informative keyframes from a video without relying on costly training or complex deep models. This work introduces C2FVS-DPP (Contextual Feature Fusion Video Summarization with Determinantal Point Process), a lightweight framework that generates concise video summaries by jointly modeling semantic importance, visual diversity, temporal structure, and {symmetry. The design centers on a symmetry-aware fusion strategy, where appearance, motion, and semantic cues are aligned in a unified embedding space, and on a reward-guided optimization logic that balances representativeness and diversity. Specifically, appearance features from ResNet-50, motion cues from optical flow, and semantic representations from BERT-encoded BLIP captions are fused into a contextual embedding. A Transformer encoder assigns importance scores, followed by shot boundary detection and K-Medoids clustering to identify candidate keyframes. These candidates are refined through a reward-based re-ranking mechanism that integrates semantic relevance, representativeness, and visual uniqueness, while a Determinantal Point Process (DPP) enforces globally diverse selection under a keyframe budget. To enable reliable evaluation, enhanced versions of the SumMe and TVSum50 datasets were curated to reduce redundancy and increase semantic density. On these curated benchmarks, C2FVS-DPP achieves F1-scores of 0.22 and 0.43 and fidelity scores of 0.16 and 0.40 on SumMe and TVSum50, respectively, surpassing existing models on both metrics. In terms of compression ratio, the framework records 0.9959 on SumMe and 0.9940 on TVSum50, remaining highly competitive with the best-reported values of 0.9981 and 0.9983. These results highlight the strength of C2FVS-DPP as an inference-driven, symmetry-aware, and resource-efficient solution for video summarization.

1. Introduction

Video summarization aims to generate concise yet informative representations of lengthy videos by preserving their most essential content, thereby facilitating efficient video understanding and retrieval. Video summarization systems have become increasingly important in a variety of domains and are typically grouped into two major categories: visual summarization and semantic summarization. Visual summarization involves the extraction of keyframes or segments to create condensed previews, such as movie trailers [1,2] or summaries for surveillance footage [3], supporting fast navigation and content indexing. In contrast, semantic summarization seeks to convey the core meaning of a video using natural language, as exemplified in video captioning [4] and narrative generation [5].

Despite significant progress, prior works in unsupervised video summarization have not sufficiently addressed the problem of redundancy. Many methods retain repetitive or uninformative frames, leading to higher computational overhead and summaries that are less concise and informative [6]. At the same time, widely used benchmark datasets such as SumMe [7] and TVSum50 [8] themselves contain redundant frames and uneven ground-truth score distributions, which can further dilute evaluation reliability. These issues contribute to a misalignment between automatically generated summaries and human preferences for informativeness, diversity, and brevity, underscoring the need for frameworks and datasets that jointly emphasize conciseness and semantic density. Furthermore, while some studies argue for supervised learning to leverage annotated datasets, others emphasize unsupervised or training-free approaches to avoid expensive model training, highlighting diverging hypotheses within the field.

Keyframe extraction from a video clip is typically approached using either static or dynamic methods [9]. Static keyframe extraction techniques rely on low-level visual descriptors—such as color histograms, motion vectors, and texture features [9]—to identify representative frames. These methods involve segmenting the video into frames, extracting handcrafted features, and applying threshold-based filtering or clustering heuristics based on inter-frame similarity. While effective in reducing redundancy and retaining visual diversity, static methods often struggle with capturing semantic content and are sensitive to noise and parameter tuning, thereby limiting their accuracy and interpretability in complex scenarios [10].

In contrast, dynamic approaches leverage deep learning models to derive abstract, high-level semantic feature representations from video data. Convolutional Neural Networks (CNNs) [11], Long Short-Term Memory (LSTM) networks [12], and Transformer-based architectures [13] have been successfully employed to learn temporal dependencies and semantic patterns, enabling more precise and automated keyframe selection. Nevertheless, despite their progress, dynamic methods still struggle with high computational demands and inference latency, limiting their effectiveness in real-time scenarios. Moreover, deep models often lack interpretability, making it difficult to understand or justify their decisions [14]. Addressing key challenges such as computational efficiency, accuracy, interpretability, and symmetry-aware design therefore remains a critical direction for future research in keyframe extraction.

The aim of this work is to address these challenges by proposing a training-free framework for generating diverse, representative, symmetry-aware, and semantically meaningful video summaries while reducing redundancy. The proposed framework adopts a hybrid strategy for keyframe selection: feature representations are extracted using pre-trained deep learning models in conjunction with a deep probing mechanism; clustering is performed based on attention-derived semantic scores; and representative frames are ultimately selected using the Determinantal Point Process (DPP) [7], as illustrated in Figure 1.

The contributions and main highlights of this research can be summarized as follows:

• We propose C2FVS-DPP, a training-free framework that generates diverse, representative, symmetry-aware, and visually unique video summaries. The novelty lies in two key aspects: (i) a contextual multimodal fusion strategy that projects appearance (ResNet-50), motion (optical flow), and semantic (BLIP→BERT) cues into a shared temporally encoded space for cross-modal comparability, and (ii) a reward-guided optimization logic that couples per-shot clustering with global DPP-based selection, ensuring semantic relevance, representativeness, and diversity under a keyframe budget.
• We curate a new benchmark dataset, derived from the SumMe [7] and TVSum50 [8] datasets using a statistical filtering method that retains frames with high ground-truth informativeness. This curated dataset reduces redundancy, increases semantic density, and provides a stronger testbed for future video summarization research.
• We introduce a novel evaluation metric, Hybrid Frame Similarity Assessment, that integrates semantic features from ResNet-50 and ViT with perceptual similarity from LPIPS. This adaptive metric captures both semantic fidelity and perceptual closeness, offering a more reliable comparison across models. Experimental results on the curated SumMe and TVSum50 datasets demonstrate that C2FVS-DPP consistently outperforms existing approaches in keyframe selection and overall summarization quality.

2. Related Work

Among existing techniques in video understanding, frame selection and summarization have attracted significant attention as key tasks for reducing redundancy and highlighting salient content. Early approaches primarily relied on low-level visual cues, whereas more recent works incorporate deep learning and attention mechanisms to capture semantic and temporal dependencies. Across these methods, a common trade-off emerges: while models aim to balance informativeness, diversity, and efficiency, most approaches face challenges in terms of scalability, interpretability, or generalization. Within this broader landscape, we review four main categories of summarization methods: clustering-based, feature-based, deep learning-based, and attention-based approaches.

2.1. Clustering-Based Approaches

Clustering methods group visually or semantically similar frames and select representatives to construct concise summaries. Early works emphasized handcrafted features: Khalid et al. [9] fused SURF and HOG descriptors, reduced dimensionality with covariance analysis, and applied Fuzzy C-Means clustering to preserve story continuity. Similarly, Tonge and Thepade [15] employed static shot segmentation and k-means clustering, which reduced storage demands but required manual cluster number selection. To improve automation, Wang et al. [16] introduced a hierarchical clustering method guided by silhouette optimization, while Gangwani and Ramteke [17] combined handcrafted SIFT features with VGG16 embeddings to enhance contextual awareness. Graph-based approaches have also been explored:  Gharbi et al. [18] used SIFT and LBP to build repeatability graphs for modularity clustering. More recently,  Kaur et al. [19] proposed hybrid feature fusion with Fuzzy C-Means and meta-heuristic optimization. These methods highlight the value of clustering but often remain sensitive to feature design choices or introduce additional algorithmic complexity.

2.2. Feature-Based Approaches

Feature-based keyframe extraction relies on low-level descriptors to identify informative frames. Rodriguez et al. [20] used histogram analysis with adaptive thresholding to detect significant variations, offering lightweight summarization. Nandinia et al. [21] employed Edge-LBP descriptors combined with shot detection to retain both textural and structural details, though preprocessing requirements increased complexity. Motion-driven summarization has also been investigated: Zhang et al. [22] introduced a spatiotemporal slice analysis framework that correlates motion with perceptual importance, well suited for surveillance but less effective in static scenarios. Beyond visual cues, Song et al. [8] proposed the TVSum dataset and demonstrated how video titles can guide semantic frame selection, though this reliance on metadata limits applicability to unlabeled content.

2.3. Deep Learning-Based Approaches

Deep learning models have enabled the learning of high-level semantics and temporal dependencies for video summarization. Yang et al. [23] proposed KFRFIF, which integrates intra- and inter-frame attributes through multi-head attention to capture fine-grained cues. Li et al. [24] employed sparse coding with dictionary learning for robust shot segmentation and keyframe identification. Alharbi et al. [25] introduced a channel-attention encoder–decoder framework with InceptionV3 features to improve multiscale representation. Zhang et al. [26] developed DGC-FNet, a dynamic graph-aware model combining Bi-LSTMs, GCNs, and contrastive learning to address long-range dependencies and noisy graph structures. In domain-specific contexts,  Mehta et al. [27] proposed DF Sampler, a self-supervised method tailored for industrial human activity recognition. While effective in their respective scenarios, these approaches often rely on substantial computation or extensive annotations, which can limit their general-purpose applicability.

Building on unimodal methods, Singh et al. [28] propose MVSAT, a Transformer-based framework for multimodal video summarization, integrating visual and textual cues through cross-attention mechanisms. Frame-level features are extracted using GoogLeNet, while captions provide complementary semantic information. The Transformer aligns and fuses multimodal inputs, with self-attention layers modeling temporal dependencies. Segment importance is estimated through semantic relevance, spatial location, and center-ness, followed by temporal segmentation techniques for keyshot selection. By combining these components, MVSAT achieves improved semantic coherence and diversity, establishing a strong baseline for training-free and unsupervised summarization.

Extending this line of work, Guo et al. [29] introduce CFSum, which adopts a coarse-to-fine fusion strategy to integrate visual, textual, and optionally audio features. Coarse fusion captures global semantic alignment across modalities, while fine fusion refines local cross-modal interactions through attention mechanisms. A multi-layer Transformer encoder models long-range dependencies, and a regression head predicts segment importance before temporal segmentation. Compared to multimodal baselines such as MVSAT and MM-Transformer, CFSum demonstrates stronger summary coherence and modality synergy, providing a benchmark for evaluating multimodal summarization strategies.

2.4. Attention-Based Approaches

Attention mechanisms have been employed to improve semantic modeling and capture long-range dependencies. Ji et al. [30] proposed Attentive Encoder–Decoder Networks (AVS), combining BiLSTMs with attention variants and Kernel Temporal Segmentation to generate importance-weighted summaries. Hsu et al. [31] extended this with a Spatiotemporal Vision Transformer (STVT), which integrates inter-frame and intra-frame attention modules to jointly model temporal and spatial cues, significantly enhancing summary coherence. Beyond summarization, Guo et al. [32] explored Reinforced Multichoice Attention for multi-turn video question generation, illustrating how attention can support deeper semantic grounding in related tasks. Together, these methods underscore the role of attention in advancing contextual and structural modeling for video summarization.

3. Dataset

3.1. Benchmark Datasets

Two widely adopted benchmark datasets are used: SumMe [7] and TVSum50 [8]. SumMe consists of 25 user-generated videos in .mp4 format, with durations ranging from 1 to 6 min. Each video is annotated by 15 to 18 users, who provide frame-level importance scores reflecting the semantic contribution of each frame to the overall summary.

TVSum50 includes 50 YouTube videos grouped into 10 categories (five videos per category), with durations ranging from 2 to 10 min. Each video is annotated by 20 users, who rate short temporal segments on a scale from 1 to 5, indicating segment-level relevance. This section outlines the dataset curation process and the strategy employed for frame selection, followed by quantitative and qualitative evaluations of the curated dataset. A comparison with existing video summarization benchmarks is also presented, highlighting improvements in summary compactness and semantic richness.

3.2. Dataset Curation and Selection Strategy

Although the SumMe and TVSum50 datasets provide rich annotations, they also contain a significant number of redundant or low-importance frames, which can dilute the learning signal and hinder framework performance. To mitigate this limitation, a statistical threshold-and-average-based filtering strategy is proposed to curate a compact and semantically meaningful subset of frames. The primary objective is to enhance supervision clarity and improve semantic precision for tasks such as video summarization and captioning.

To ensure fair evaluation, the ground-truth (GT) importance scores for each video in the SumMe dataset are normalized to the range $[0,1]$, following the protocol of the original authors. The same normalization is applied to the TVSum50 dataset, ensuring a consistent scale across both datasets while preserving the relative importance values assigned by the annotators. The complete dataset curation and frame selection procedure is detailed in Algorithm 1. The curated frame selection scores for each video clip, including ground-truth and selected frames, are provided in the Supplementary Materials (Supplementary Section).

Algorithm 1 Statistical selection of informative frames based on GT scores.

Require:  Sequence of frames $\mathcal{F}=\{f_1,f_2,\dots ,f_n\}$ with ground-truth scores $s_i$ Ensure:  Selected representative frames ${\mathcal{F}}_{\mathrm{selected}}$ 1: ${\mathcal{F}}_{\mathrm{valid}}\leftarrow \{f_i\in \mathcal{F}\mid s_i\ge \tau \},\tau =0.1$ 2: Compute average score: $\overline{s}\leftarrow {\displaystyle \frac{1}{|{\mathcal{F}}_{\mathrm{valid}}|}}{\sum }_{f_i\in {\mathcal{F}}_{\mathrm{valid}}}{s}_i$ 3: for all  $f_i\in {\mathcal{F}}_{\mathrm{valid}}$  do 4:     if $s_i>\overline{s}$ then 5:         $\mathrm{frame}\_\mathrm{status}(f_i)\leftarrow 1$ 6:     else 7:         $\mathrm{frame}\_\mathrm{status}(f_i)\leftarrow 0$ 8: Group consecutive frames with $\mathrm{frame}\_\mathrm{status}=1$ into sequences ${\mathrm{sequence}}_k$ 9: for all  ${\mathrm{sequence}}_k$  do 10:     Let ${\mathrm{sequence}}_k=\{f_{i_k},f_{i_k+1},\dots ,f_{j_k}\}$ 11:     $m\leftarrow i_k+⌊{\displaystyle \frac{j_k-i_k}{2}}⌋$ 12:     Add $f_m$ to ${\mathcal{F}}_{\mathrm{selected}}$ 13: return  ${\mathcal{F}}_{\mathrm{selected}}$

• $\mathcal{F}=\{f_1,f_2,\dots ,f_n\}$: Sequence of video frames.
• $s_i=\mathrm{gt}\_\mathrm{score}\left(f_i\right)$: Ground-truth importance score for frame $f_i$.
• $\tau$: Threshold for filtering low-score frames ($\tau =0.1$).
• $\overline{s}$: Average ground-truth score of valid frames.
• $\mathrm{frame}\_\mathrm{status}(f_i)$: Binary indicator (1 = important, 0 = not important).
• ${\mathrm{sequence}}_k$: A group of consecutive frames with frame_status = 1.
• $f_m$: Middle frame of a valid sequence, selected as representative.
• ${\mathcal{F}}_{\mathrm{selected}}$: Final set of representative frames.

3.3. Statistical Analysis of the Dataset

Figure 2 illustrates the statistical properties of the curated SumMe and TVSum50 datasets. Additional insights are provided in Figure 2a–d, which shows the distributions resulting from threshold-based frame selection. Frames with average ground-truth (GT) scores equal to or above the threshold values of 0.3 (for SumMe) and 2.03 (for TVSum50) are selected. Figure 2 further visualizes the distribution of all frame-level scores in the original datasets versus the selected frames. The illustrated examples are taken from the videos Airforce (SumMe) and -xMr-HKMfVA (TVSum50), respectively.

Quantitative Metrics.

We define two quantitative indicators to assess the quality of the curated dataset:

• GT Score Improvement (Factor): This measures how much more representative the curated frames are compared to the original summary. It is computed as follows:

  $$\mathrm{GT}\mathrm{Score}\mathrm{Improvement}\left(\mathrm{Factor}\right)={\displaystyle \frac{\mathrm{Avg}.\mathrm{GT}{\mathrm{Score}}_{\mathrm{curated}}}{\mathrm{Avg}.\mathrm{GT}{\mathrm{Score}}_{\mathrm{original}}}}$$
• Frame Reduction (%): This indicates the percentage of frames removed from the original dataset to produce a more compact summary. It is calculated as follows:

  $$\mathrm{Frame}\mathrm{Reduction}(\%)=\left(1-{\displaystyle \frac{N_{\mathrm{curated}}}{N_{\mathrm{original}}}}\right)\times 100$$

  where $N_{\mathrm{original}}$ and $N_{\mathrm{curated}}$ denote the number of frames in the original and curated datasets, respectively.

Furthermore,

Table 1. Comparison of original and curated video summarization datasets.

Dataset	Type	Avg. Frames	Summary Length (%)	Avg. GT Score	GT Score Improvement (Factor)	Frame Reduction (%)
SumMe	Original	4393	5–15 (avg. ∼10)	0.13	–	–
	Curated	55	1.25	0.47	3.62	98.75
TVSum50	Original	7047	5–15 (avg. ∼10)	0.35	–	–
	Curated	53	0.75	0.58	1.66	99.25

The bold is used to distinguish between the original dataset values and the curated dataset values, highlighting the improvement and significance to the reader.

presents a comparative analysis between the original and curated versions of the SumMe [7] and TVSum50 [8] datasets used for video summarization. The curated datasets significantly reduce the number of frames—by 98.75% for SumMe and 99.25% for TVSum50—while still preserving essential content. This results in much shorter summaries: 1.25% of the original length for SumMe and 0.75% for TVSum50, compared to the typical 5–15% in the original datasets. Additionally, the average ground-truth (GT) scores are markedly higher in the curated datasets (0.47 for SumMe and 0.58 for TVSum50), reflecting improved semantic representativeness. Notably, the curated summaries achieve GT scores that are 3.62 and 1.66 times higher than the original summaries for SumMe [7] and TVSum50 [8], respectively. These findings confirm and demonstrate the effectiveness of the curation strategy in producing summaries that are both concise and semantically informative.

• Qualitative Results:

Qualitative comparison results for two exemplar videos from the original SumMe dataset and the curated SumMe dataset are shown in Figure 3. The overlaid line and interval plot depict frame-level importance scores (green line) along with the selected summary intervals (orange-shaded regions). The visualization demonstrates how the proposed method effectively identifies and retains high-importance segments while removing redundant and low-importance frames, including those with zero scores.

3.4. Comparison of Performance Metrics: Proposed vs. Existing Datasets

The curated datasets are evaluated using standard performance metrics: Precision [33], Recall [33], and F1-Score [34], which are defined as follows:

$$\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$$

(1)

$$\mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$$

(2)

$$\mathrm{F}1-\mathrm{Score}=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(3)

Here, TP (true positives) represents the number of correctly selected important frames, FP (false positives) denotes the number of incorrectly selected frames, and FN (false negatives) corresponds to the number of important frames that were not selected.

As shown in

Table 2. Comparison of evaluation metrics (F1-Score, Precision, Recall) for original and curated datasets.

Dataset	Type	F1-Score	Precision	Recall
SumMe	Original	0.3100	–	–
	Curated	0.2023	1.0000	0.1137
TVSum50	Original	0.5400	–	–
	Curated	0.1776	1.0000	0.0976

The bold is used to distinguish between the original dataset values and the curated dataset values, highlighting the improvement and significance to the reader.

, although the F1-score is lower for the curated datasets (0.2023 vs. 0.3100 for SumMe and 0.1776 vs. 0.5400 for TVSum50), the Precision improves significantly, reaching a perfect score of 1.0000.

This trade-off reflects the design goal of our curated dataset: to prioritize the selection of highly informative and non-redundant frames, even if it comes at the cost of reduced coverage. Such characteristics are particularly beneficial in downstream applications like detailed video captioning and highlight localization, where semantic clarity and precision are more critical than broad recall coverage.

4. The Proposed Methodology

The problem of video summarization is formulated as selecting a representative subset of keyframes ${\mathcal{V}}_{\mathrm{rep}}\subset \mathcal{V}$ from an input video $\mathcal{V}={\left\{v_i\right\}}_{i=1}^T$, where T is the total number of frames and $v_i$ is the i-th frame. The objective is to maximize a multi-objective score function $\mathcal{J}\left(\mathcal{S}\right)$ that captures semantic importance, visual diversity, and temporal coverage, subject to the constraint $|{\mathcal{V}}_{\mathrm{rep}}|\le K$, where K is the desired number of keyframes. Formally,

$${\mathcal{V}}_{\mathrm{rep}}=\underset{\mathcal{S}\subset \mathcal{V},\left|\mathcal{S}\right|\le K}{argmax}\mathcal{J}\left(\mathcal{S}\right),$$

(4)

where $\mathcal{S}$ denotes a candidate subset of frames from $\mathcal{V}$, and ${\mathcal{V}}_{\mathrm{rep}}$ is the optimal subset maximizing the objective function.

Our methodology is a hierarchical, training-free pipeline leveraging pre-trained models.

4.1. Frame Sampling and Preprocessing

To reduce computational complexity, frames from the video $\mathcal{V}=\{F_1,F_2,\dots ,F_T\}$ are sampled at a fixed interval $S=5$. This yields the following sampled frame set:

$$\mathcal{F}=\{F_t\mid t=kS,k\in \mathbb{N},t\le T\}.$$

(5)

Each sampled frame $F_t$ is resized to ${\mathbf{x}}_t\in {\mathbb{R}}^{3\times 224\times 224}$ and normalized by subtracting the mean RGB values and scaling to $[0,1]$, ensuring compatibility with pre-trained models and robustness to varying frame rates.

4.2. Contextual Feature Fusion

Each frame is represented by three modality-specific feature vectors:

• ${\mathbf{v}}_t^{\mathrm{res}}\in {\mathbb{R}}^{2048}$: Visual features extracted using ResNet-50 [35].
• ${\mathbf{m}}_t^{\mathrm{opt}}\in {\mathbb{R}}^{1024}$: Motion features computed from dense optical flow via Farneback’s method [36].
• ${\mathbf{s}}_t^{\mathrm{text}}\in {\mathbb{R}}^{768}$: Semantic features obtained by encoding BLIP-generated captions [37] with BERT [38].

These heterogeneous features are concatenated:

$${\mathbf{z}}_t=[{\mathbf{v}}_t^{\mathrm{res}},{\mathbf{m}}_t^{\mathrm{opt}},{\mathbf{s}}_t^{\mathrm{text}}]\in {\mathbb{R}}^{3840}.$$

(6)

To unify the feature dimensions, a projection matrix ${\mathbf{W}}_{\mathrm{proj}}\in {\mathbb{R}}^{512\times 3840}$ maps the concatenated features into a 512-dimensional embedding:

$${\mathbf{e}}_t={\mathbf{W}}_{\mathrm{proj}}·{\mathbf{z}}_t.$$

(7)

To incorporate temporal ordering, sinusoidal positional encodings ${\mathbf{p}}_t\in {\mathbb{R}}^{512}$ [39] are added:

$${\widehat{\mathbf{e}}}_t={\mathbf{e}}_t+{\mathbf{p}}_t,$$

(8)

with

$${\mathbf{p}}_t={[sin(t/{\mathrm{10,000}}^{2k/512}),cos(t/{\mathrm{10,000}}^{2k/512})]}_{k=0}^{255},$$

(9)

where the notation ${[·]}_{k=0}^{255}$ means concatenating the sine and cosine terms for each embedding dimension k, producing a 512-dimensional positional encoding.

4.3. Context-Aware Semantic Scoring via Transformer

Given the embedding set $\widehat{\mathcal{E}}=\{{\widehat{\mathbf{e}}}_1,\dots ,{\widehat{\mathbf{e}}}_N\}\in {\mathbb{R}}^{N\times 512}$, where $N=\left|\mathcal{F}\right|$, a pre-trained Transformer encoder ${\mathcal{T}}_{\theta }$ with six layers and eight attention heads [39] captures contextualized frame representations:

$${\mathbf{h}}_t={\mathcal{T}}_{\theta }\left({\widehat{\mathbf{e}}}_t\right).$$

(10)

A linear layer followed by a sigmoid computes each frame’s semantic score:

$$s_t=\sigma ({\mathbf{w}}^{\top }{\mathbf{h}}_t+b),$$

(11)

where $\mathbf{w}\in {\mathbb{R}}^{512}$, $b\in \mathbb{R}$, and $s_t\in [0,1]$ indicates semantic relevance.

4.4. Temporal Segmentation (Shot Detection)

The sampled sequence $\mathcal{F}$ is partitioned into shots $\mathcal{S}=\{S_1,S_2,\dots ,S_K\}$, where each shot $S_k=\{F_{a_k},\dots ,F_{b_k}\}$. PySceneDetect’s ContentDetector [40] computes HSV color differences between consecutive frames, applying adaptive thresholds to detect shot boundaries.

4.5. Clustering with K-Medoids

For each shot $S_k$, K-Medoids clustering [41] is applied to embeddings $\{{\widehat{\mathbf{e}}}_i\mid i\in S_k\}$. The optimal cluster number $C_k^{*}$ is chosen by maximizing the silhouette score:

$$C_k^{*}=arg\underset{C}{max}\mathrm{SilhouetteScore}(\left\{{\widehat{\mathbf{e}}}_i\right\},C),$$

(12)

where silhouette is computed using cosine distances. The medoid of each cluster (the frame minimizing distances to others) is selected as a candidate keyframe.

4.6. Reward-Based Frame Selection

Each candidate frame is re-ranked with a composite reward:

$$R\left(i\right)=s_i+{\lambda }_{1}D\left(i\right)+{\lambda }_2{R}_{\mathrm{rep}}\left(i\right)+{\lambda }_{3}V\left(i\right),$$

(13)

where $s_i$ is semantic importance, and ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ are hyperparameters balancing the following terms:

• Diversity Reward ($D\left(i\right)$): This promotes temporal diversity by penalizing similarity with previously chosen frames.
• Representativeness Reward ($R_{\mathrm{rep}}\left(i\right)$): This measures closeness to the cluster centroid.
• Visual Diversity Reward ($V\left(i\right)$): This encourages visual distinctiveness using ResNet features.

For each cluster, the frame with the highest reward is retained in the candidate set $\mathcal{C}$.

4.7. DPP-Based Diversity-Aware Selection

From the candidate set $\mathcal{C}$, a subset $\mathcal{Y}\subseteq \mathcal{C}$ of size $k=min(\lfloor K/|\mathcal{S}|\rfloor ,|\mathcal{C}\left|\right)$ is chosen with a Determinantal Point Process (DPP). The kernel is as follows:

$$L_{ij}=s_i{s}_j·cos({\widehat{\mathbf{e}}}_i,{\widehat{\mathbf{e}}}_j),$$

(14)

where cosine similarity ensures consistent scaling across embeddings. With regularization $\mathbf{L}\leftarrow max(\mathbf{L},0)+{10}^{-6}\mathbf{I}$, subsets maximizing both diversity and quality are selected [42]:

$${\mathbb{P}}_L\left(\mathcal{Y}\right)\propto det\left({\mathbf{L}}_{\mathcal{Y}}\right).$$

(15)

The complete procedure is summarized in Algorithm 2 (C2FVS-DPP) and illustrated in Figure 4.

Algorithm 2 C2FVS-DPP: Contextual feature fusion-based video summarization with DPP-based diversity selection

Require:  Input video $\mathcal{V}={\left\{v_i\right\}}_{i=1}^T$, keyframe budget K Ensure:  Selected summary frames ${\mathcal{V}}_{\mathrm{rep}}$ 1: Frame Sampling: Sample frames $\mathcal{F}=\{F_t\mid t=kS,k\in \mathbb{N},t\le T\}$ with interval S; preprocess each to ${\mathbf{x}}_t\in {\mathbb{R}}^{3\times 224\times 224}$ 2: Feature Fusion: For each frame $F_t$: Extract visual (${\mathbf{v}}_t^{\mathrm{res}}$), motion (${\mathbf{m}}_t^{\mathrm{opt}}$), and semantic (${\mathbf{s}}_t^{\mathrm{text}}$) features Fuse: ${\mathbf{z}}_t=[{\mathbf{v}}_t^{\mathrm{res}},{\mathbf{m}}_t^{\mathrm{opt}},{\mathbf{s}}_t^{\mathrm{text}}]$ Project and add positional encoding: ${\widehat{\mathbf{e}}}_t={\mathbf{W}}_{\mathrm{proj}}{\mathbf{z}}_t+{\mathbf{p}}_t$ 3: Semantic Scoring: $$s_t=\sigma ({\mathbf{w}}^{\top }{\mathcal{T}}_{\theta }\left({\widehat{\mathbf{e}}}_t\right)+b),s_t\in [0,1]$$ 4: Shot Detection: Segment $\mathcal{F}$ into $\mathcal{S}=\{S_1,\dots ,S_K\}$ using PySceneDetect. 5: Clustering: For each shot $S_k$, apply K-Medoids on $\{{\widehat{\mathbf{e}}}_i\mid i\in S_k\}$; select medoids ${\mathcal{C}}_k$. Let $\mathcal{C}={\bigcup }_k{\mathcal{C}}_k$. 6: Reward Re-Ranking: For each candidate $i\in \mathcal{C}$: $$R\left(i\right)=s_i+{\lambda }_{1}D\left(i\right)+{\lambda }_2{R}_{\mathrm{rep}}\left(i\right)+{\lambda }_{3}V\left(i\right)$$ Retain highest-reward frame(s) per cluster. 7: DPP Selection: Construct kernel $$L_{ij}=s_i{s}_j·\langle {\widehat{\mathbf{e}}}_i,{\widehat{\mathbf{e}}}_j\rangle$$ and select $\mathcal{Y}\subseteq \mathcal{C}$ via greedy MAP inference under budget K. 8: return  ${\mathcal{V}}_{\mathrm{rep}}=\mathcal{Y}$

The list of variables and their corresponding notations used in the proposed C2FVS-DPP framework are summarized in

Table 3. Summary of notations used in the proposed C2FVS-DPP framework.

Notation	Description
$\mathcal{V}={\left\{v_i\right\}}_{i=1}^T$	Input video with T frames
K	Keyframe budget (maximum number of summary frames)
S	Frame sampling interval (set to 5)
$\mathcal{F}$	Set of sampled frames from $\mathcal{V}$
${\mathbf{x}}_t\in {\mathbb{R}}^{3\times 224\times 224}$	Preprocessed image tensor of frame $F_t$
${\mathbf{v}}_t^{\mathrm{res}}\in {\mathbb{R}}^{2048}$	Visual features from ResNet-50
${\mathbf{m}}_t^{\mathrm{opt}}\in {\mathbb{R}}^{1024}$	Motion features from optical flow
${\mathbf{s}}_t^{\mathrm{text}}\in {\mathbb{R}}^{768}$	Semantic features from BLIP captions encoded by BERT
${\mathbf{z}}_t$	Concatenated contextual feature vector
${\mathbf{e}}_t\in {\mathbb{R}}^{512}$	Projected feature vector via ${\mathbf{W}}_{\mathrm{proj}}$
${\mathbf{p}}_t\in {\mathbb{R}}^{512}$	Positional encoding for frame t
${\widehat{\mathbf{e}}}_t$	Final embedding (projected + positional encoding)
${\mathcal{T}}_{\theta }$	Transformer encoder with parameters $\theta$
${\mathbf{h}}_t$	Hidden representation of frame t from ${\mathcal{T}}_{\theta }$
$s_t\in [0,1]$	Semantic importance score of frame t
$\mathcal{S}=\{S_1,\dots ,S_K\}$	Set of shots obtained via segmentation
$C_k^{*}$	Optimal number of clusters for shot $S_k$
${\mathcal{C}}_k$	Candidate keyframes from clusters in shot $S_k$
$\mathcal{C}$	Union of all candidate keyframes, $\mathcal{C}={\bigcup }_k{\mathcal{C}}_k$
$R\left(i\right)$	Reward score of candidate frame i
$D\left(i\right)$	Diversity reward of frame i
$R_{\mathrm{rep}}\left(i\right)$	Representativeness reward of frame i
$V\left(i\right)$	Visual uniqueness reward of frame i
${\lambda }_1,{\lambda }_2,{\lambda }_3$	Weighting coefficients for reward terms
$L_{ij}$	DPP kernel entry for frames i and j
$\langle {\widehat{\mathbf{e}}}_i,{\widehat{\mathbf{e}}}_j\rangle$	Cosine similarity between embeddings
$\mathcal{Y}$	Subset of selected frames by DPP inference
${\mathcal{V}}_{\mathrm{rep}}$	Final summary set of keyframes

.

5. Experiment

This section details the experimental design and evaluation strategy adopted for the curated dataset. The baseline methods used for comparison are introduced first, followed by a description of the evaluation metrics. An ablation study is then conducted to assess the contribution of individual components. Finally, a comprehensive performance analysis of the baseline methods is presented to demonstrate the effectiveness of the proposed approach.

5.1. Baseline Methods for Evaluation

To evaluate the effectiveness of the proposed method, comparisons are made against a set of baseline methods and techniques commonly employed in the domain of video summarization and frame selection. Fuzzy C-Means (FCM) [9] clustering is employed as a soft clustering baseline that assigns membership probabilities to each frame, allowing flexible grouping based on feature similarities. We also implement a Dynamic Frame Sampler (DF Sampler) [27], which adaptively selects frames by analyzing motion cues and temporal differences, serving as a baseline for dynamic selection approaches. For feature representation, the Edge-Local Binary Pattern (Edge-LBP) [21] is adopted as a traditional texture descriptor, capturing spatial edge features across video frames. In addition, Feature Intercross and Fusion (FIF) [23] is applied to combine appearance and motion features into a unified representation, providing a simple yet effective baseline technique. Furthermore, Unsupervised Learning Methods (ULMs) Gangwani and Ramteke [17], such as K-Means and FCM, are incorporated to assess the ability of clustering-based selection without reliance on labeled data. Histogram-Based Representations (HBRs) [20], including color and edge histograms, are used to measure frame similarity and serve as low-level feature baselines. Generative adversarial networks (GANs) [43] are employed to model frame importance and diversity in an unsupervised manner, serving as a robust baseline for generative approaches. The graph neural network (GNN) [44] formulates frame selection as a binary node classification task over a sparse temporal graph, providing a scalable and memory-efficient baseline compared to dense graph-based methods.

5.2. Experimental Setup and Implementation Details

Preprocessing: Video frames are extracted from each clip by sampling every 5th frame, corresponding to approximately 6 frames per second for a 30 fps video, in order to balance temporal coverage and computational efficiency. Each frame is resized to $224\times 224$ pixels using OpenCV’s default bilinear interpolation, converted from BGR to RGB format, and normalized to the range [0, 1]. Optical flow is computed between consecutive grayscale frames using the Farneback method with the following parameters: pyr_scale = 0.5, levels = 3, winsize = 15, iterations = 3, poly_n = 5, and poly_sigma = 1.2. The resulting motion vectors are flattened into 1024-dimensional representations and stored as 16-bit floats for memory efficiency.

Model Architecture: The proposed framework, C2FVS-DPP, performs contextual feature fusion for representative frame selection. Appearance features are extracted using the shallow layers of ResNet-50 [35], yielding 2048-dimensional vectors. Motion features are captured via the Farneback optical flow algorithm [36], producing 1024-dimensional vectors. Semantic visual features are obtained from CLIP (ViT-B/32) [45], while semantic text features are derived from BLIP-generated captions [37] and encoded with BERT [38], resulting in 768-dimensional vectors. The extracted features are concatenated into a 3840-dimensional fused representation and linearly projected into a 512-dimensional embedding space with positional encodings. A cross-modal attention module with 8 heads aligns 256-dimensional visual and textual embeddings derived from the fused representation, generating a 256-dimensional joint representation. This is further refined using a Transformer-based semantic scorer comprising 4 layers with 8 attention heads, which computes the final frame-level importance scores.

Within each video shot, K-Medoids clustering with cosine distance groups semantically similar frames, where the optimal cluster number is determined via silhouette analysis. Frame selection is further guided by a reinforcement learning-inspired reward function that balances semantic relevance, intra-cluster diversity, representativeness, and visual uniqueness. The influence of each component is controlled by tunable hyperparameters ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$. Finally, a Determinantal Point Process (DPP) [7] is applied to sample a compact subset of frames from each shot by constructing a kernel matrix from fused embeddings and scoring outputs, thereby ensuring diversity and comprehensive content coverage in the final summary.

Weight Initialization: Pre-trained weights from ImageNet (ResNet50), CLIP (ViT-B/32), BERT (base-uncased), and BLIP (image-captioning-base) were used to initialize feature extraction modules, leveraging robust representations without fine-tuning [35,37,38,45]. The CrossModalAttention and TransformerScorer modules employed Xavier initialization for linear layers to ensure stable gradients [46]. The projection layer ($\mathbf{W}\in {\mathbb{R}}^{512\times 3840}$) was implemented using PyTorch v2.3.1+cu121.

default initialization (Kaiming) [47]. Positional encodings were deterministically computed using sinusoidal functions (Equation (9)), requiring no random initialization.

Optimization: Due to limited computational resources, the proposed framework operates in inference mode, utilizing pre-trained models (ResNet50, CLIP, BERT, BLIP) without fine-tuning. Feature extraction is performed using batch processing with a batch size of 4 and mixed-precision inference via PyTorch’s autocast to reduce memory overhead. All features are stored in 16-bit float precision to minimize GPU memory usage. The K-Medoids algorithm is employed for clustering, with cosine distance optimized using $ϵ={10}^{-8}$. The number of clusters is determined by maximizing the silhouette score. Frame selection is guided by a reward function combining semantic score, diversity, representativeness, and visual diversity, with the corresponding weights ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ as tunable hyperparameters. A Determinantal Point Process (DPP) further refines selection by maximizing kernel diversity. All processes are designed for efficient single-pass inference on a Quadro P5000 (NVIDIA Corporation, Santa Clara, CA, USA) (16 GB VRAM) without training.

5.3. Evaluation

The proposed Hybrid Frame Similarity Assessment metric is employed to measure the alignment between frames generated by the Frame Selection Framework (C2FVS-DPP) and those obtained from baseline methods. The evaluation relies on a weighted combination of semantic embeddings extracted from ResNet-50 and ViT, together with perceptual similarity computed using LPIPS.

The weights for hybrid similarity scoring are heuristically assigned as $w_R=0.4$, $w_V=0.4$, and $w_L=0.2$, balancing semantic contributions from ResNet-50 and ViT [48] with perceptual similarity from LPIPS [49]. These weights were later dynamically adjusted based on the variance of similarity scores in the respective feature spaces, allowing the model to emphasize more discriminative modalities depending on the frame content. The threshold T for frame matching was computed as the maximum between Otsu’s threshold and the 75th percentile of all similarity scores, enabling adaptive and robust filtering. The full procedure is detailed in Algorithm 3.

Algorithm 3 Proposed hybrid frame similarity assessment metric leveraging ResNet-50, ViT, and LPIPS.

Require:  Model-generated frames ${\mathcal{F}}_M$, ground-truth frames ${\mathcal{F}}_G$ Ensure:  CSV files and histogram plots 1: Load pre-trained ResNet50, ViT, LPIPS models 2: Set weights $w_R=0.4$, $w_V=0.4$, $w_L=0.2$, threshold $T=0.70$ 3: for each video folder v do 4:     Load model frames ${\mathcal{F}}_M^{\left(v\right)}$, ground-truth frames ${\mathcal{F}}_G^{\left(v\right)}$ 5:     for each $f_m^i\in {\mathcal{F}}_M^{\left(v\right)}$ do 6:         for each $f_g^j\in {\mathcal{F}}_G^{\left(v\right)}$ do 7:            Extract ResNet and ViT features for $f_m^i$ and $f_g^j$ 8:            Compute $S_R^{i,j}=\cos \mathrm{ine}({\varphi }_R^i,{\varphi }_R^j)$ 9:            Compute $S_V^{i,j}=\cos \mathrm{ine}({\varphi }_V^i,{\varphi }_V^j)$ 10:            Compute $S_L^{i,j}=1/(1+\mathrm{LPIPS}(f_m^i,f_g^j))$ 11:     Compute variances ${\sigma }_R^2,{\sigma }_V^2,{\sigma }_L^2$ 12:     if ${\sigma }_R^2+{\sigma }_V^2+{\sigma }_L^2\ne 0$ then 13:         Update weights: $$w_R={\displaystyle \frac{{\sigma }_R^2}{\Sigma }},w_V={\displaystyle \frac{{\sigma }_V^2}{\Sigma }},w_L={\displaystyle \frac{{\sigma }_L^2}{\Sigma }}$$ 14:     for each pair $(i,j)$ do 18:         $S_{\mathrm{final}}^{i,j}=w_R{S}_R^{i,j}+w_V{S}_V^{i,j}+w_L{S}_L^{i,j}$ 16:     Compute threshold $T=max(\mathrm{Otsu}\left(S_{\mathrm{final}}\right),{\mathrm{Percentile}}_{75}\left(S_{\mathrm{final}}\right))$ 17:     for each $f_m^i$ do 18:         Find $j^{*}=arg{max}_j{S}_{\mathrm{final}}^{i,j}$ 19:         if $S_{\mathrm{final}}^{i,j^{*}}\ge T$ then 20:            Mark $f_m^i$ as matched to $f_g^{j^{*}}$

• ${\mathcal{F}}_M$: Set of model-generated video frames.
• ${\mathcal{F}}_G$: Set of ground-truth video frames.
• $f_m^i$: i-th frame from model-generated frames.
• $f_g^j$: j-th frame from ground-truth frames.
• ${\varphi }_R^i,{\varphi }_R^j$: ResNet50 features for $f_m^i$ and $f_g^j$.
• ${\varphi }_V^i,{\varphi }_V^j$: ViT features for $f_m^i$ and $f_g^j$.
• $S_R^{i,j}$: ResNet cosine similarity between $f_m^i$ and $f_g^j$.
• $S_V^{i,j}$: ViT cosine similarity between $f_m^i$ and $f_g^j$.
• $S_L^{i,j}$: LPIPS-based perceptual similarity between $f_m^i$ and $f_g^j$.
• $S_{\mathrm{final}}^{i,j}$: Final hybrid similarity score.
• $w_R,w_V,w_L$: Weights for ResNet, ViT, and LPIPS similarity, respectively.
• ${\sigma }_R^2,{\sigma }_V^2,{\sigma }_L^2$: Variances of respective similarity scores.
• T: Adaptive similarity threshold.

5.4. Results on Curated Dataset: SumMe and TVSum50

Several baseline methods are evaluated using the curated versions of the widely adopted SumMe and TVSum50 datasets. The baseline methods include FCM [9], FIF [23], DF Sampler [27], ULM [17], Edge-LBP [21], HBR [20], GAN [43], and GNN [44]. Each of these methods targets keyframe extraction and redundancy reduction through diverse strategies, including clustering, multimodal feature fusion, histogram analysis, edge-based descriptors, generative adversarial networks, and graph neural networks. Specifically, GAN models frame importance and diversity in an unsupervised manner, while GNN formulates frame selection as a binary node classification task over a sparse temporal graph, together providing robust and scalable baselines. Collectively, these approaches offer a comprehensive state-of-the-art comparison for evaluating the effectiveness of our training-free approach.

Following previous works [50,51,52,53] in video summarization evaluation, the F1-score, fidelity score, and compression ratio (CR) are adopted as key evaluation metrics.

Table 4. Performance comparison on curated SumMe datasets.

Model	F1-Score (%)	Fidelity Score (%)	CR (%)
FCM [9]	0.1708	0.1044	0.9975
FIF [23]	0.1774	0.1198	0.9951
DF Sampler [27]	0.2012	0.1382	0.9961
ULM [17]	0.2124	0.1384	0.9960
Edge-LBP [21]	0.1256	0.0750	0.9981
HBR [20]	0.1778	0.1083	0.9973
GAN [43]	0.1567	0.0906	0.9978
GNN [44]	0.1983	0.1297	0.9962
Proposed (C2FVS-DPP)	0.2211	0.1603	0.9959

The bold is used to highlight the improvement and significance of the proposed framework, showing that the results are better compared to existing methods.

and 

Table 5. Performance comparison on curated TVSum50 dataset.

Model	F1-Score (%)	Fidelity Score (%)	CR (%)
FCM [9]	0.2338	0.1650	0.9980
FIF [23]	0.3103	0.2991	0.9934
DF Sampler [27]	0.4134	0.3527	0.9951
ULM [17]	0.4218	0.3606	0.9957
Edge-LBP [21]	0.2700	0.2177	0.9969
HBR [20]	0.2364	0.1551	0.9983
GAN [43]	0.1928	0.1154	0.9989
GNN [44]	0.2516	0.1923	0.9975
Proposed (C2FVS-DPP)	0.4344	0.4071	0.9940

The bold is used to highlight the improvement and significance of the proposed framework, showing that the results are better compared to existing methods.

provides a comparative analysis of all evaluated methods. The proposed approach consistently outperforms baseline models across both datasets. Notably, the incorporation of multi-descriptor similarity—leveraging ResNet50, ViT, and LPIPS—substantially enhances keyframe representativeness and visual fidelity. Our model achieves the highest average F1-score, reflecting superior alignment with human-annotated summaries. Furthermore, it records elevated fidelity scores and comparison ratios, underscoring the effectiveness of our method in generating compact summaries while preserving semantic content.

Furthermore, Figure 5 and Figure 6 illustrate the F1-score and fidelity score comparisons between the proposed model and baseline methods on the curated SumMe and TVSum50 datasets, respectively. In both cases, the proposed approach achieves the highest scores, demonstrating its effectiveness in generating accurate and semantically rich video summaries.

5.5. Ablation Study

To evaluate the contributions of key components in the proposed video summarization model, an ablation study was conducted on the curated SumMe and TVSum50 datasets. This study investigates the impact of various feature types—ResNet50 visual features, optical flow motion features, and BERT textual features—as well as reward components used within the extract_features module. Evaluation metrics include the F1-score, fidelity score, and compression ratio (CR).

For reward ablations on the curated SumMe dataset, as shown in

Table 6. Ablation on visual and representativeness rewards (diversity reward active) on curated SumMe dataset.

Reward		Evaluation Metrics
Visual+Rep	Diversity	F1-Score	Fidelity Score	CR
✗	✗	0.1453	0.0832	0.9980
✗	✓	0.1258	0.0709	0.9983
✓	✗	0.1256	0.0708	0.9983
✓	✓	0.2212	0.1603	0.9959

The symbols “✗” and “✓” provides better visualization to indicate which features were included or skipped during the ablation study, clarifying their meaning for the reader.

, removing the diversity reward (${\lambda }_1=0$) led to a 43.1% F1-score decrease to 0.1258. Excluding the representativeness reward (${\lambda }_2=0$) resulted in a 43.2% reduction to 0.1256. Omitting the visual diversity reward (${\lambda }_3=0$) caused a 34.3% decrease, reducing the F1-score to 0.1453. All reward ablations maintained high CRs (≥0.998), indicating comprehensive content coverage. Nevertheless, the full model consistently achieved an optimal balance between coverage and semantic relevance.

Table 7. Feature ablation on curated SumMe dataset.

Model	F1-Score	Fidelity Score	CR
w/o ResNet Features	0.1326	0.0754	0.9980
w/o Motion Features	0.1144	0.0636	0.9986
w/o BERT Features	0.1258	0.0709	0.9983
Full Model	0.2212	0.1603	0.9959

presents the feature ablation study on the curated SumMe dataset. The full model achieved the highest performance, with an F1-score of 0.2212 and a fidelity score of 0.1603. Removing ResNet features resulted in a 40.0% drop in F1-score to 0.1326, while excluding motion features led to a 48.3% decline to 0.1144, underscoring the critical role of temporal structure. BERT-based textual information also proved significant—excluding it caused the F1-score to fall by 43.1% to 0.1258. Although CR remained consistently high (≥0.998), the F1 and fidelity metrics reflect the semantic degradation from each exclusion, highlighting the importance of integrating visual, motion, and textual modalities.

Table 8. Ablation on visual and representativeness rewards (diversity reward active) on curated TVSum50 dataset.

Reward		Evaluation Metrics
Visual+Rep	Diversity	F1-Score	Fidelity Score	CR
✗	✗	0.1908	0.1188	0.9986
✗	✓	0.1716	0.1057	0.9987
✓	✗	0.1730	0.1055	0.9987
✓	✓	0.4344	0.4071	0.9940

The symbols “✗” and “✓” provides better visualization to indicate which features were included or skipped during the ablation study, clarifying their meaning for the reader.

shows the reward ablation on the curated TVSum50 dataset. The full model, with all three reward components—diversity, visual, and representativeness—achieved the highest F1-score of 0.4344 and a fidelity score of 0.4071. When both visual and representativeness rewards were removed, the F1-score dropped by 56.1% to 0.1908, with a fidelity score of 0.1188. Removing only the diversity reward resulted in a 60.2% F1-score reduction to 0.1730. Likewise, removing only visual and representativeness components (keeping diversity active) dropped the F1-score to 0.1716. These results confirm that the synergy of all three rewards is critical for optimal summarization performance on TVSum50, despite high CR values (>0.998) across all variants.

Table 9. Feature ablation on curated TVSum50 dataset.

Model	F1-Score	Fidelity Score	CR
w/o ResNet Features	0.1778	0.1099	0.9987
w/o Motion Features	0.1724	0.1101	0.9984
w/o BERT Features	0.1658	0.1010	0.9987
Full Model	0.4344	0.4071	0.9940

presents the feature ablation on the curated TVSum50 dataset. The full model again attained the best performance with an F1-score of 0.4344 and a fidelity score of 0.4071. Removing ResNet features led to a 59.1% drop in F1-score to 0.1778, while excluding motion features dropped it to 0.1724 (a 60.3% decline), indicating the essential role of temporal structure. BERT-based textual information also proved significant—excluding it caused the F1-score to fall by 61.8% to 0.1658. Although CR remained consistently high (>0.998), the F1 and fidelity metrics reflect the semantic degradation from each exclusion, underlining the importance of integrating visual, motion, and textual modalities.

5.6. Complexity and Efficiency Analysis

We report per-frame compute and runtime for the proposed C2FVS-DPP under the same settings used in our experiments ($224\times 224$ input, $S=5$; batch size = 4 unless otherwise specified). All major components are included in the analysis, while CPU-bound steps such as optical flow, K-Medoids, and DPP are reported in terms of wall-time.

Module-wise complexity.

Table 10. Module-wise computational cost of C2FVS-DPP (per frame at 224×224). GFLOPs are forward pass only, measured as 2 × MACs. Notes: FLOPs for the Transformer scorer estimated analytically; others measured via THOP. “Total (edge-mode)” amortizes BLIP across shots (example ∼50 frames/shot ⇒ BLIP $\approx 0.675$ GFLOPs/frame).

Module	GFLOPs	Notes
ResNet-50 (appearance features)	8.27	measured (THOP)
CLIP ViT-B/32 (vision features)	8.73	measured (THOP)
BLIP (caption-based semantic features)	33.73	measured (THOP)
Projection (3840→512), $N=32$	0.126	measured (THOP)
Transformer scorer (4 layers, $d=512$, $d_{\mathrm{ff}}=2048$, $N=32$)	0.81	analytic estimate
Optical flow (Farnebäck, CPU)	N/A	wall-time reported
K-Medoids clustering (per shot)	N/A	wall-time reported
DPP selection (diversity-aware)	N/A	wall-time reported
Total (full, per-frame BLIP)	51.67	dominated by BLIP/CLIP
Total (edge-mode, BLIP amort.)	18.61	amortized BLIP across shots

The bold is used to highlight the key value (0.81 GFLOPs) to draw the reader’s attention to the computational cost of the Transformer scorer compared to other components.

presents the per-module GFLOPs (forward pass only). The total cost in full mode is $51.67$ GFLOPs/frame, dominated by the feature extraction modules (BLIP: $33.73$, CLIP: $8.73$, ResNet-50: $8.27$). In contrast, the projection layer ($0.126$ GFLOPs) and Transformer scorer ($0.81$ GFLOPs) are lightweight. In edge-mode, amortizing BLIP across shots reduces the total cost to $18.61$ GFLOPs/frame, indicating that feature extraction is the primary computational bottleneck.

End-to-end efficiency.

Table 11. End-to-end efficiency of C2FVS-DPP under standardized settings ($224\times 224$ input, $S=5$, batch $=4$ unless specified). Latency/FPS are mean ± std over 5 runs; video decoding excluded. Notes: Mixed precision when applicable; CUDA events for GPU timing and perf_counter for CPU timing. Peak GPU via NVML; host RAM via RSS.

Platform/Mode	Latency (ms/Frame)	Throughput (FPS)	Peak GPU (GB)	Peak Host (GB)
Desktop GPU (CUDA) — full	$1075.94\pm 177.93$	$0.95\pm 0.14$	4.38	30.38
Desktop GPU (CUDA) — edge-mode	$927.35\pm 89.46$	$1.09\pm 0.10$	2.72	28.45
CPU-only (8-core) — full	$1193.28\pm 58.58$	$0.84\pm 0.04$	N/A	30.88
Desktop GPU (CUDA), batch $=1$ — edge-mode (simulated edge)	$3867.29\pm 780.63$	$0.27\pm 0.05$	2.72	28.24

reports latency, throughput, and memory usage. On a desktop GPU, full mode achieves $1075.94\pm 177.93$ ms/frame ($0.95\pm 0.14$ FPS) with $4.38$ GB peak GPU memory. In edge-mode, the runtime improves to $927.35\pm 89.46$ ms/frame ($1.09\pm 0.10$ FPS) with $2.72$ GB peak GPU memory and reduced host RAM usage. A batch $=1$ simulated-edge configuration records $3867.29\pm 780.63$ ms/frame ($0.27\pm 0.05$ FPS) while keeping GPU memory below 3 GB.

Backbone and Embedding Variants. The results in

Table 12. Backbone and feature extractor choices within the proposed C2FVS-DPP framework: accuracy vs. efficiency trade-offs. Accuracy reported on curated SumMe and TVSum datasets. GFLOPs derived from Table 10; latency and peak GPU memory from Table 11. Notes: “Ours (no-CLIP)” removes the CLIP branch, reducing 8.73 GFLOPs; “Ours (edge-mode)” amortizes BLIP cost across shots. The compression ratio (CR) remained stable (≈0.995–0.998) across all variants and is omitted for brevity. Dashes (—) indicate measurements not performed.

Configuration	Curated SumMe F1/Fidelity	Curated TVSum F1/Fidelity	GFLOPs	Latency (ms)	GPU (GB)
Ours (full) = ResNet-50 + CLIP + BLIP→BERT	0.22/0.16	0.43/0.41	51.67	$1075.94\pm 177.93$	4.38
Ours (no-CLIP) = ResNet-50 + BLIP→BERT	0.22/0.16	0.43/0.41	42.94	—	—
Ours (edge-mode) = ResNet-50 + CLIP + BLIP→BERT (BLIP amort.)	0.22/0.16	0.43/0.41	18.61	$927.35\pm 89.46$	2.72
ViT-B/16 + BLIP→BERT (swap ResNet-50)	0.21/0.15	0.41/0.39	—	—	—
CLIP-only (joint embedding; drop BLIP/BERT)	0.20/0.14	0.40/0.38	≥8.73	—	—

The bold is used to highlight our method in comparison to other approaches, drawing the reader’s attention to the reported efficiency metrics.

confirm that the combination of ResNet-50 + BLIP→BERT achieves the best accuracy–efficiency balance, matching or surpassing ViT/CLIP variants while requiring substantially fewer GFLOPs and lower GPU memory. While ViT and CLIP models are competitive in accuracy, they incur significantly higher computational cost without consistent improvements in summarization quality. Moreover, BLIP→BERT provides interpretable textual evidence (captions) that can be inspected, whereas joint CLIP embeddings tightly couple vision and language, sometimes biasing selection toward “captionable” content while neglecting visually distinct but semantically sparse frames. The edge-mode variant further reduces per-frame cost (51.67 → 18.61 GFLOPs) and peak GPU usage (4.38 → 2.72 GB) with negligible accuracy loss, highlighting the deployability of the proposed framework in resource-constrained settings.

5.7. Application to Surveillance Video Summarization

To evaluate the generalizability of the proposed C2FVS-DPP framework, we used the publicly available UCF-Crime Annotation (UCA) dataset [54,55], which provides temporally localized event annotations for real-world surveillance videos. We follow the UCA convention: frames inside an anomalous event interval are labeled with score $=1$ (anomaly), and frames outside are labeled with score $=0$ (normal). For demonstration, two videos—Arson011_x264 and Robbery137_x264—were selected. The annotated ground-truth importantframe intervals were $(150,420)$ and $(680,1267)$ for Arson011_x264, and $(135,1950)$ for Robbery137_x264.

Figure 7 comprises four panels: (a–b) “Arson011” and (c–d) “Robbery137”. In each pair, the upper panel corresponds to the baseline ULM [17] and the lower panel to the proposed C2FVS-DPP. Gray bars denote the annotated anomalous event intervals (score $=1$), white regions denote normal frames (score $=0$), and orange bars indicate the segments selected by each method.

These results indicate that the proposed method generates coherent, event-aware, and resource-conscious video summaries—properties that are particularly desirable for surveillance workflows where accurate event localization is critical. In our experiments, ResNet-50 and BERT are used for visual and textual feature extraction, respectively, due to their robustness and efficiency in training-free settings.

6. Discussion

The results of the C2FVS-DPP framework mark a significant step forward in training-free video summarization, offering a lightweight, inference-driven approach that integrates multimodal features to produce concise and semantically rich video summaries. This section discusses the evaluation outcomes, compares them with previous studies, addresses the framework’s implications, and highlights limitations and future research directions.

6.1. Evaluation and Comparison with Prior Work

The C2FVS-DPP framework achieves F1-scores of 0.22 and 0.43, as well as fidelity scores of 0.16 and 0.40, on the curated SumMe and TVSum50 datasets, respectively, as shown in Table 4 and Table 5. In terms of compression ratio, the framework attains 0.9959 on SumMe and 0.9940 on TVSum50, values that are close to the best-reported results of 0.9981 and 0.9983, respectively. These results outperform baseline methods such as FCM [9], FIF [23], DF Sampler [27], ULM [17], Edge-LBP [21], and HBR [20]. The superior performance validates the hypothesis that fusing appearance (ResNet-50), motion (optical flow), and semantic (BERT-encoded BLIP captions) features, combined with symmetry-aware DPP-based selection, enhances keyframe selection by balancing semantic relevance, visual diversity, and temporal coherence.

Compared to clustering-based methods such as Khalid et al. [9] and Tonge and Thepade [15], which rely on handcrafted or static features, C2FVS-DPP leverages pre-trained deep models to capture richer semantic and temporal information, resulting in higher F1 and fidelity scores. Unlike deep learning-based approaches such as Yang et al. [23], which incur high computational costs due to complex feature interactions, our framework operates efficiently in inference mode, making it suitable for resource-constrained environments. The incorporation of semantic features via BLIP captions extends the work of Song et al. [8], which used video titles for semantic guidance, by enabling applicability to videos without predefined metadata. Additionally, the DPP-based selection aligns with Gygli et al. [7] but improves scalability through greedy MAP inference, addressing redundancy more effectively than traditional clustering methods [16].

The ablation studies (Table 7 and Table 9) underscore the importance of each feature type, with motion features showing the largest impact (48.3% and 60.3% F1-score reductions when removed on SumMe and TVSum50, respectively), consistent with Zhang et al. [22]’s emphasis on temporal dynamics in video summarization. The reward-based re-ranking mechanism, balancing diversity, representativeness, and visual uniqueness (Table 6 and Table 8), addresses a persistent limitation in prior methods—the tendency to select redundant frames [15]. The Hybrid Frame Similarity Assessment metric (Algorithm 3), combining ResNet-50, ViT, and LPIPS features, provides a robust evaluation framework, improving upon static metrics used in Narasimhan et al. [50] by incorporating dynamic weight adjustments for adaptive similarity scoring.

6.2. Dataset Curation and Implications

The curated SumMe and TVSum50 datasets, reducing frames by approximately 98.8% and 99.3% while achieving GT score improvements of 3.62× and 1.66×, respectively (Table 1), address a critical limitation of the original datasets: redundancy and uneven score distributions. The statistical selection strategy (Algorithm 1) enhances semantic density, resulting in summaries with near-perfect compression ratios (0.9959 for SumMe and 0.9940 for TVSum50) close to the best-reported values. This trade-off prioritizes concise, high-quality summaries, aligning with the needs of applications like video captioning and highlight localization, where semantic clarity is paramount. Compared to Gygli et al. [7] and Song et al. [8], the curated datasets provide a more robust benchmark for evaluating summarization frameworks, as evidenced by the improved GT scores and reduced summary lengths (1.25% for SumMe and 0.75% for TVSum50).

The application to the UCF-Crime Annotation dataset (Figure 7) demonstrates C2FVS-DPP’s generalizability to surveillance contexts, where it aligns more closely with annotated event intervals than the unsupervised baseline [17]. This suggests practical utility in domains like security monitoring, where precise event localization is critical, offering an advantage over computationally intensive methods such as that by Yuan et al. [55].

6.3. Limitations and Future Work

Despite its strengths, C2FVS-DPP has certain limitations. The reliance on pre-trained models (ResNet-50, CLIP, BERT, BLIP) assumes that their representations generalize well across diverse video domains, which may not hold for highly specialized content such as medical or industrial videos. In line with the reviewer’s observation, we have expanded this discussion to explicitly acknowledge this limitation. While our framework mitigates some risks through motion features and diversity-aware selection, pre-trained encoders may not transfer perfectly. Future work may explore domain-adapted encoders or advanced vision–language models (e.g., CLIP, FLAVA, BLIP-2) under higher compute budgets. We also plan to investigate lightweight adaptation strategies such as parameter-efficient fine-tuning (e.g., LoRA, adapters) and few-shot domain-specific training to improve adaptability without compromising efficiency.

The fixed sampling interval ($S=5$) may miss rapid scene changes in high-frame-rate videos, potentially reducing temporal granularity. To address this, we introduce a preliminary adaptive sampling strategy in which the interval is dynamically adjusted based on optical flow intensity—using smaller intervals in high-motion regions and larger ones in static regions. This motion-driven scheme aims to preserve temporal granularity while reducing redundancy, thereby improving both efficiency and summary quality without altering the training-free nature of C2FVS-DPP.

We also recognize that several hyperparameters (e.g., clustering settings, reward weights, similarity thresholds) were chosen heuristically. Although preliminary experiments indicated stable performance trends, a more systematic sensitivity analysis could yield deeper insights. We plan to perform such an analysis to further validate parameter robustness.

To enhance robustness against geometric attacks and distortions, we plan to integrate distortion-invariant feature descriptors (e.g., SIFT, ORB) and lightweight alignment techniques inspired by anti-geometric attack strategies in the watermarking literature Wang et al. [56], Wen et al. [57].

Furthermore, adopting multi-level feature hierarchies, as demonstrated in robust human parsing frameworks, may strengthen the framework’s ability to handle complex and cluttered scenes. In particular, we aim to incorporate multi-level feature learning inspired by From Simple to Complex Scenes: Learning Robust Feature Representations for Accurate Human Parsing [58], where hierarchical visual features from ResNet-50’s shallow, intermediate, and deep layers will be aligned with motion and semantic features via lightweight attention. This integration is expected to enhance robustness in complex scenarios and complement our reward-guided and DPP-based selection strategies, improving both semantic and temporal coherence.

In addition, although the current framework focuses on visual and semantic modalities, extending it toward multimodal audio–visual–textual fusion represents a natural next step. Future work will explore how textual cues (e.g., video titles or transcripts) and audio signals can complement visual features to enhance summarization quality in multimodal scenarios.

Finally, while our experiments demonstrate strong average performance across benchmarks, detailed per-category breakdowns (e.g., sports, news, surveillance) remain constrained by dataset size and available annotations. We recognize this as an important direction and plan to conduct category-level evaluations once more balanced multimodal datasets become available.

In summary, future extensions of C2FVS-DPP will explore adaptive frame sampling, systematic parameter studies, distortion-resilient alignment, multi-level feature learning, lightweight fine-tuning, multimodal fusion with audio, and category-level evaluations. These directions will further strengthen the framework’s robustness, generalization, and semantic–temporal coherence.

6.4. Broader Implications

In the broader context, C2FVS-DPP bridges the gap between computational efficiency and semantic richness, offering a scalable, training-free solution that rivals deep learning-based methods while maintaining interpretability. Its modular design allows integration with emerging feature extractors or clustering algorithms, facilitating advancements in real-time video summarization. The framework’s high compression ratios and effectiveness in surveillance contexts (Figure 7) suggest potential for real-world applications, such as content indexing, video browsing, and automated surveillance analysis. By addressing redundancy and enhancing semantic density through dataset curation, C2FVS-DPP sets a new standard for evaluating video summarization frameworks, with implications for both research and practical deployment in resource-constrained environments.

7. Conclusions

The proposed C2FVS-DPP framework offers a robust, training-free solution for video summarization, effectively addressing the challenge of selecting semantically rich, diverse, and representative keyframes without the need for costly model training. By integrating appearance features from ResNet-50, motion cues from optical flow, and semantic representations from BERT-encoded BLIP captions, the framework achieves superior performance on the curated SumMe and TVSum50 datasets, with F1-scores of 0.221 and 0.434 and fidelity scores of 0.160 and 0.407, respectively. The combination of K-Medoids clustering, reward-guided re-ranking, and Determinantal Point Process (DPP) selection ensures summaries that balance semantic importance, visual diversity, temporal coherence, and symmetry.

Beyond methodological advances, this work contributes curated benchmark datasets that reduce frames by 98.75% (SumMe) and 99.25% (TVSum50), while improving ground-truth scores by factors of 3.62 and 1.66, respectively. Moreover, the introduction of a Hybrid Frame Similarity Assessment metric offers a robust evaluation tool that better aligns with human perceptual judgments. The framework’s generalizability is further confirmed on the UCF-Crime Annotation dataset, where it demonstrates strong performance in event localization for surveillance applications.

Despite its advantages, the proposed framework has certain limitations, including reliance on general-purpose pre-trained models, heuristic hyperparameter choices, and the use of a fixed sampling interval. These limitations point to several promising directions for future research: adaptive frame sampling strategies guided by optical flow, systematic threshold analysis, lightweight parameter-efficient fine-tuning (e.g., LoRA, adapters), integration of advanced vision–language models (e.g., CLIP, FLAVA, BLIP-2), incorporation of multi-level feature hierarchies, enhanced resilience against geometric distortions, multimodal audio–visual–textual fusion, and comprehensive category-level evaluations. Such extensions would further strengthen the framework’s robustness, adaptability, and semantic–temporal coherence while maintaining the efficiency of its training-free design.

In summary, the C2FVS-DPP framework establishes a new benchmark for efficient, interpretable, and scalable video summarization. By leveraging symmetry in both multimodal feature fusion and the trade-off between representativeness and diversity, the framework produces compact yet semantically meaningful summaries that preserve balance across visual and semantic dimensions. This symmetry-aware design has broad implications for applications in content indexing, video browsing, and automated surveillance, advancing resource-efficient and robust multimedia analysis.