Cross-Modal Semantic Communication for Text-to-Video Retrieval in Internet of Vehicles

Abstract

Text-to-video retrieval offers an intelligent solution for Internet of Vehicles (IoV) users to access desired content on demand. However, the constrained communication channels in IoV, characterized by low signal-to-noise ratios (SNR), pose significant obstacles to retrieval performance. To tackle these issues, this study presents SemTVR, a semantic communication framework dedicated to achieving superior robustness in text-to-video retrieval tasks in low-SNR IoV environments. By integrating the semantic communication paradigm with edge–cloud collaboration, our architecture leverages roadside unit (RSU) features and cloud resources to enable collaborative retrieval. We introduce a multi-semantic interactive reliable transmission mechanism that utilizes historical search records to enhance semantic recovery accuracy under adverse channel conditions. Furthermore, we devise a cross-modal fine-grained matching strategy assigning differentiated weights to video content and query sentences. Experimental results conducted on authoritative datasets demonstrate that SemTVR significantly outperforms baseline methods in terms of search accuracy, particularly in low SNR scenarios, validating its effectiveness for future IoV applications.

1. Introduction

With the advancements in information and communication technologies, the Internet of Vehicles (IoV) integrates vehicles, passengers, and smart devices into the Internet ecosystem, facilitating information exchange and data sharing among intelligent agents like vehicles, thereby providing intelligent services to passengers [1,2]. Traditional in-vehicle systems primarily present driving information, whereas modern intelligent in-vehicle systems equipped with advanced human–computer interaction interfaces provide a broader range of functionalities [3], including navigation and video playback, significantly enriching the passenger experience in the IoV [4]. As vehicle display technology, multimedia capabilities, and in-vehicle operating systems evolve [5], the demand for diverse video content among passengers is rapidly increasing, posing a substantial challenge in searching for and locating videos of interest from vast content repositories [6].

Video retrieval technology addresses this challenge by enabling IoV passengers to perform keyword-based searches [7,8]. However, conventional keyword-based retrieval systems rely on keywords and preset video tags, limiting its intelligence and efficiency. Text-to-video retrieval (TVR) technology enhances this process by matching user query keywords with video content semantics to retrieve relevant videos from extensive media libraries [9], thereby elevating the intelligence level and improving the search experience for passengers. With advancements in large-scale contrastive language-image pre-training technologies like CLIP [10], text–video retrieval has achieved notable improvements. Methods like CLIP4Clip [11] embed images and texts into a shared semantic space, significantly enhancing retrieval performance. Nevertheless, these methods typically treat entire sentences and images/videos as holistic entities, lacking fine-grained interaction between text and video content, which limits search accuracy. To overcome this limitation, researchers [12,13] have proposed fine-grained contrastive frameworks that capture detailed semantic relationships by comparing each word in the query sentence with each frame in the video. Despite these advancements, discrepancies persist in the treatment of semantic information importance between query sentence and video content. For instance, when searching for videos related to “pedestrian dancing causing traffic jam,” frames depicting “dancing” and “traffic jam” are crucial, whereas frames presenting “pedestrian” may be less critical. Current methods often overlook this fine-grained semantic relevance, potentially affecting retrieval accuracy by introducing irrelevant semantic information.

In the IoV, passengers input search requests through in-vehicle interaction systems, transmit via V2X (Vehicle-to-Everything) wireless communication technology to roadside units, and subsequently forward to cloud service providers for matching target video content [14]. However, IoV networks face challenges like poor coverage, weak signals, and huge demands for IoV multimedia services, posing significant obstacles to text-to-video retrieval [15,16]. In particular, in future smart cockpit applications, the quality and latency of text-to-video retrieval are crucial, directly determining the immediacy of the interactive system and the user’s immersive experience. Passengers expect to quickly and accurately access entertainment, navigation, or driver assistance-related video content via natural language commands. However, the inherently low signal-to-noise ratio of the IoV communication environment severely distorts retrieval requests and increases transmission latency, thus becoming a major bottleneck in improving the smart cockpit experience. Therefore, designing a retrieval system that can withstand channel interference, ensure accurate semantic recovery, and efficiently complete matching is of key significance for achieving high-quality, low-latency smart cockpit services. Semantic communication, a key technology of 6G, introduces an intelligent communication paradigm focused on task-oriented communication [17,18], garnering attentions in recent years. Semantic communication selectively extracts and transmits only relevant semantic information for user tasks, optimizing data transmission efficiency in complex communication scenarios like IoV, where low SNR channels prevail [19,20]. Despite its potential, research on semantic communication based text-to-video retrieval in the IoV remains unexplored.

To address this gap, this paper first proposes a text-to-video retrieval method for IoV based on semantic communication, introducing a semantic communication paradigm. This study pioneers the integration of semantic communication technology into IoV text-to-video retrieval. The main contributions of this paper are as follows:

• A semantic communication-based Text-to-Video Retrieval architecture is designed. Traditional video retrieval architectures overlook real-world communication characteristics, performing search request matching solely in the cloud. This results in low search matching accuracy and significant cloud processing loads in low SNR IoV communication scenarios. The proposed architecture introduces the semantic communication paradigm and the edge–cloud collaborative concept, which adapts the semantic information extraction of query texts in the vehicle to suit low SNR channel environment and integrates the characteristics of roadside units and cloud resources, enabling efficient edge–cloud collaborative retrieval.
• A multi-semantic interactive reliable transmission method is developed. Existing methods fail to account for distortions in the transmission of search requests caused by channel noise and ignore the rich semantic information contained in the user’s historical search, limiting the semantic recovery accuracy of search requests. To ensure robustness, the system leverages the inherent features of the communication channel and combines historical query texts with current one for multi-semantic interaction, enhancing the semantic content of query texts and improving the semantic recovery accuracy of search requests in low SNR IoV scenarios.
• A cross-modal fine-grained semantic matching method is proposed. Traditional matching methods lack sufficient interaction between the semantic relevance of search request keywords and video contents, thereby limiting search matching accuracy. The proposed method performs layered cross-modal interactions between different video frames and query sentences, as well as between word-level and visual representations. This approach assigns differentiated weights to terms in both candidate video contents and query sentences, improving retrieval accuracy and enhancing the video retrieval experience for passengers.

The subsequent sections are arranged as follows. Section 2 lists related work. Section 3 presents the semantic communication based text-to-video retrieval system. Section 4 proposes a multi-semantic interactive reliable transmission method. Section 5 devises a cross-modal fine-grained matching method. Section 6 conducts simulation experiments. Finally, Section 7 summarizes this paper.

2. Related Work

2.1. Text-to-Video Retrieval

Text-to-video retrieval evaluates the similarity between a given request text and a candidate video, providing search results with high similarities based on this evaluation. Prior studies [21,22,23] have improved text-to-video retrieval through visual-language pre-training models. The attention mechanism mimics the attention characteristics of the human visual system, enabling the model to focus on the most relevant parts when processing complex inputs. This mechanism solves the problem that traditional models struggle to capture long-distance dependencies, greatly improving model performance and interpretability [24], and is often used in typical applications such as semantic segmentation [25,26]. The CLIP model, pre-trained on large-scale text and image data, exhibits strong zero-shot learning capabilities, offering rich semantic understanding and cross-modal alignment for text-to-video retrieval. ClipBERT [27] applied CLIP in text-to-video retrieval, proposing an efficient end-to-end pre-training scheme. CLIP4clip [11] introduced CLIP as the backbone network, significantly enhancing retrieval performance.

Global matching is a common technique in cross-modal contrastive learning. Studies [28,29] leveraged the semantic information of language or video for global matching, aligning text and video from an overall perspective before performing text-to-video retrieval tasks. However, global matching methods focus only on overall semantics, struggling to accurately match details in complex queries and video content. Models based on fine-grained alignment decompose text and video into multiple levels, emphasizing token-level alignment between both modalities. Despite advancements, the existing literature lacks sufficient fine-grained interaction between text and video. Wray et al. [30] decomposed action phrases into verbs and nouns for fine-grained video retrieval but did not differentiate the important semantic information in different video frames. Wang et al. [31] learned the semantic correlation between request text and videos by jointly modeling local and global alignment. X-Pool [32] utilized scaled dot-product attention to focus on semantically similar frames in the text but ignored the semantic importance of individual words. However, these works fail to consider the characteristics of IoV scenarios. The high mobility of vehicles, strong interference in communication channels, and weak network coverage undoubtedly distort search requests submitted by IoV passengers during transmission, which would further seriously impact the retrieval accuracy.

2.2. Task-Oriented Semantic Communication

Unlike traditional communication, semantic communication transmits only task-relevant semantic information by utilizing a shared background knowledge base between the sender and receiver for semantic compression and message understanding [19,33,34]. Leveraging deep learning technology, semantic communication can effectively extract and restore semantic information of the original modality [35,36]. Existing work in semantic communication is mainly divided into two categories: data reconstruction and task-oriented communication. In data reconstruction, semantic communication systems extract global semantic information from source data. DeepSC [37] employed joint source-channel encoding to extract and encode semantic information for transmission, considering channel characteristics. Another study [38] proposed a semantic-preserving compression method for text transmission, reducing the number of bits required for message representation. In task-oriented applications, only the semantic information relevant to the specific task is extracted and encoded by the sender [39,40]. A semantic communication system for UAV image retrieval was developed in [41], transmitting only the semantic information of images to address personalized retrieval needs based on different user interests. In Visual Question Answering (VQA) [42], MU-DeepSC captured the semantic information of multimodal data, including textual questions and query images, enabling multimodal semantic communication. Farshbafan et al. [43] presented a task-oriented unified semantic communication framework that adapted to transmit different semantic information according to the specific transmission target.

Research shows that additional semantic information can effectively improve the accuracy of semantic recovery at the receiver. The literature [44] enhanced the text transmission robustness by utilizing semantic noise such as typos or ambiguous expressions. Other studies [45,46] applied contextual information to semantic extraction and recovery in fields like text and image processing. The literature [47] introduced a memory module to store contextual semantic information for guiding text transmission. However, increased text representation can also lead to the accumulation of redundant semantic noise unrelated to the search task and lack semantic mining of users’ historical search interests, thus limiting search performance. Semantic communication can significantly enhance the efficiency of text-to-video retrieval in low SNR IoV scenarios. However, research results on text-to-video retrieval based on semantic communication in IoV have not yet emerged.

3. Semantic Communication Based Text-to-Video Retrieval System

3.1. Edge–Cloud Collaborative Retrieval Architecture

Traditional text-to-video retrieval methods focus only on improving retrieval accuracy, often overlook the characteristics of communication networks and low-latency requirements in practical applications. In the IoV, communication channels with low SNR characteristics can cause distortion of search request texts during transmission. Additionally, cloud-centric architecture will lead to extremely heavy communication and computation overhead to the cloud server. Addressing these limitations, we propose a text-to-video retrieval architecture for the IoV based on edge–cloud collaboration and semantic communication technology, optimizing the retrieval efficacy in complex IoV communication scenarios. The proposed architecture, as illustrated in Figure 1, consists of three layers: vehicle terminal layer, edge server layer, and cloud server layer.

Figure 1. Edge–cloud collaborative text-to-video retrieval architecture.

Vehicle Terminal Layer: It consists of vehicle terminals in the IoV. Passengers submit search request texts via the in-vehicle interaction system. The vehicle terminal extracts the semantic information of the search request text and sends it to the roadside unit (RSU).

Edge Server Layer: This layer comprises RSUs responsible for responding to search requests, recovering query texts, matching semantic features of text and video, and providing feedback on search results from the cloud.

Cloud Server Layer: This one contains massive cloud servers owned by video content providers, which is responsible for offline training of models, storing vast amounts of video contents, extracting semantic features, and providing video retrieval services to passengers.

The execution process of the designed retrieval system is as follows: If passengers initiate a video search, they first input the video search request text through the in-vehicle interaction system. The vehicle uses a multi-semantic interaction encoder trained in the cloud (as described in Section 4) to extract the query’s semantic information (Step ①), which is then encoded by channel encoder and transmitted to the RSU via a wireless channel. The RSU decodes the transmitted bit stream and employs a multi-semantic interaction decoder (as proposed in Section 4) to recover the query text (Step ②). The RSU then extracts features from the recovered query text adopting a cross-modal fine-grained matching model (Step ③) (as presented in Section 5). Simultaneously, the cloud server uses the same cross-modal fine-grained matching model to extract features from its video database, which are then transmitted to the RSU (Step ④). The RSU matches the semantic features of both the query text and video (Step ⑤) to identify the most relevant video. The matching results are reported to the cloud server, which finally sends the matched video back to the in-vehicle passengers (Step ⑥), completing the text-to-video retrieval task.

Traditional video retrieval architectures deploy video storage, search response, and content matching all in the cloud, bring significant communication and computation loads to the cloud, causing queuing delays in retrieval tasks, and severely impacting the service experience of users. The proposed architecture introduces the concept of edge computing, enabling RSUs to respond to search requests in real-time and store the semantic features of video contents cached in the cloud for semantic relevance calculation between the query text and candidate videos, thereby achieving a significant reduction in both computational complexity and bandwidth consumption on the cloud. Moreover, the cloud server’s powerful computing capabilities are utilized for offline training of the multi-semantic interaction encoder-decoder and the cross-modal fine-grained matching model, which are then deployed to RSUs and vehicle terminals to improve the semantic extraction accuracy for search matching.

3.2. Semantic Communication Model

This paper establishes a semantic communication-based system model for IoV text-to-video retrieval. Section 3.1 describes the overall architecture of edge–cloud collaborative retrieval from the perspective of system composition and interaction. The efficient operation of this architecture relies on its underlying core communication and computing paradigm. Therefore, this section formally defines the semantic communication system model supporting this architecture, corresponding to the wireless semantic transmission link from the vehicle terminal to the RSU in Figure 1, and the subsequent semantic recovery and matching process. This model clarifies the complete process of semantic information encoding, channel transmission, decoding, and recovery, and forms the basis for the specific methods (MSI and FGM) proposed in Section 4 and Section 5.

As shown in Figure 2, the semantic communication system model established in this paper mainly consists of two parts: Multi-semantic interaction reliable transmission (Section 4), which is responsible for enhancing and restoring the semantics of the query text under low SNR channels, and Cross-modal fine-grained matching (Section 5), which is designed for using the restored semantics to perform accurate video retrieval.

Figure 2. Semantic communication based IoV text-to-video retrieval.

Multi-Semantic Interactive Transmission (Section 4): This module delves into the effectiveness of semantic transmission in real-world IoV scenarios for passengers. For the sender, semantic encoding and channel encoding are performed, while at the receiver, channel decoding and semantic decoding are conducted to recover the query text. Due to the poor signal quality experienced in the IoV, transmitted search request texts are prone to substantial interference. To accurately restore the original query text at the receiver, this module introduces the Multi-Semantic Interactive (MSI) transmission method, which designs a semantic encoder and decoder for search texts and integrates cross-modal attention interaction with historical query texts. By leveraging historical query texts, the enhanced background knowledge is provided, enabling the receiver to restore the query text with more semantic information reflecting the passengers’ search intentions.

Cross-Modal Fine-Grained Matching (Section 5): This module is dedicated to enhancing the accuracy of the text-to-video retrieval model. It involves feature extraction for the recovered query text and videos, followed by cross-modal fine-grained matching to refine the model’s retrieval accuracy. This module introduces a cross-modal fine-grained matching (FGM) method, leveraging the rich historical semantic context embedded in the reconstructed query text at the receiver, and leverages a contrastive learning paradigm aimed at maximizing the affinity scores of positive samples while suppressing the correlations among negative counterparts. To facilitate the model in grasping the correlations between text and video modalities, cross-modal interactions between different video frames and query sentences, as well as among distinct words and the entire video, are introduced. This enables the model to learn semantic information from both modalities, paying more attention to frames and words with higher weights. Consequently, the performance of the text-to-video retrieval task is improved, the video retrieval experience for passengers is largely enhanced.

4. Multi-Semantic Interactive Reliable Transmission Method

In the intricate channel environment of the IoV, strong channel noise significantly interferes with the transmission of search request text submitted by passengers. Additionally, subjective cognitive biases and variations in background knowledge among passengers will definitely introduce semantic noise, which seriously affect the restoration of query texts at the receiver. To address these issues, this section propose a multi-semantic interactive reliable semantic transmission (MSI) method, which considers both channel noise and semantic noise. Figure 3 illustrates the semantic encoder–decoder of the semantic communication system.

Figure 3. Multi-semantic interaction encoder–decoder module.

4.1. Multi-Semantic Interactive Encoder

To improve the robustness of query text transmission in low SNR channels, the encoder not only extracts the semantics of the current query but also incorporates the user’s historical query sentences as context for semantic enhancement. The core idea is to leverage the relevance between historical and current queries, using a cross-attention mechanism to filter and strengthen semantic features consistent with the user’s search intent, thereby providing a richer and more accurate semantic representation for subsequent transmission. Assuming the input of MSI is the passenger’s query sentence $s=\{s_1,s_2\dots \dots s_N\}$, to obtain an enhanced text representation, historical query sentence $h=\{h_1,h_2\dots \dots h_N\}$ similar to the current query text are introduced. Here, $s_i$ and $h_i$, respectively, represent the i-th word in the current query sentence and historical queries. The historical query sentence h is retrieved and selected by calculating the cosine similarity between its semantic feature vectors and those of the current query. The most relevant historical query record is determined for semantic fusion with the current query. The system maintains a fixed-capacity local cache (e.g., 100 records), managed using a first-in, first-out (FIFO) strategy. This naturally creates a sliding window that focuses on recent queries, ensuring the timeliness of the historical context used. First, both the query sentence s and the historical query sentences h are semantically encoded to extract their semantic information as follows:

$$z_s=\mathcal{E}\left(s;\phi \right),z_h=\mathcal{E}\left(h;\phi \right)$$

(1)

where $\mathcal{E}\left(·;\phi \right)$ represents the text semantic encoder and $\phi$ is a learnable parameter, including multiple Transformer encoder layers, which is designed to align the latent representations of queries with those derived from historical context.

Next, a cross-attention mechanism is applied to extract the semantic information of the query and historical texts, using the semantic information of the historical text as the query Q, and the query text as the key K and value V, to obtain an enhanced text representation $z_e$ based on the historical text’s semantic information.

$$\begin{array}{c}{z}_e=CrossAtten(q,k,v)\hfill \\ =CrossAtten(z_h,z_s,z_s)\hfill \\ =soft\max \left({\displaystyle \frac{\left(W^Q{z}_h\right){\left(W^K{z}_s\right)}^T}{\sqrt{d_h}}}\right)W^V{z}_s\hfill \end{array}$$

(2)

where $d_h$ is the dimension size of the attention heads, and $W^Q$, $W^K$ is the linear projection matrices for the query, key, and value, respectively. After passing through the normalization layer and the feed-forward layer, the final text representation is obtained.

Regarding the availability and selection of historical data, it is assumed that users’ historical search records (such as recent or high-frequency queries) can be obtained through local access to the in-vehicle system or privacy-protected cloud records. During encoding, the system retrieves several historical queries that are semantically most similar to the current query for enhancement. In terms of relevance handling, the proposed cross-attention mechanism inherently possesses a certain weight allocation capability, automatically focusing on historical semantic information more relevant to the current query while assigning lower weights to relatively irrelevant parts. Regarding robustness, the method exhibits inherent resilience to certain noise or sparsity in historical data. Through the attention mechanism, the model primarily relies on strongly correlated historical context. When historical data is completely missing or extremely sparse, the model can degenerate into semantic encoding primarily relying on the current query, yet its performance lower bound is still better than benchmark methods that do not utilize any context.

4.2. Wireless Channel Transmission

This section employs two classic channel models, AWGN and Rayleigh fading, to simulate noise and multipath fading effects in vehicular networks. The practical significance lies in the fact that AWGN represents the ubiquitous additive noise, while Rayleigh fading characterizes the random fluctuations in signal caused by vehicle movement and environmental scattering. Combining these two models allows for a comprehensive evaluation of the anti-interference capability of semantic transmission schemes in real-world wireless environments. Let $C_{\alpha }(·)$ denote the channel encoder with parameter set $\alpha$, which adopts dense layers with different units to transform the semantic information into transmittable symbols. The enhanced text semantic information after cross-attention is encoded into a symbol stream x through channel encoding as

$$x=C_{\alpha }\left(z_e\right)$$

(3)

Let y represent the signal received at the receiver, which can be expressed as

$$y=hx+n$$

(4)

Consider additive white Gaussian noise (AWGN) and Rayleigh fading channels, where h is the Rayleigh fading coefficient and n signifies the channel noise, all subject to a circular symmetric complex Gaussian distribution, i.e., $h\sim CN(0,{\sigma }_h^2)$, $n\sim CN(0,{\sigma }_n^2)$, where ${\sigma }_h^2$ and ${\sigma }_n^2$ represent the variance of the distribution.

At the receiver, let $C_{\delta }^{-1}(·)$ denote the channel decoder with parameter set $\delta$, which decodes and restores the semantic information represented as ${\widehat{z}}_e$

$${\widehat{z}}_e=C_{\delta }^{-1}\left(y\right)$$

(5)

4.3. Multi-Semantic Interactive Decoder

The core task of the decoder is to gradually reconstruct the original query sentence using the received semantic features. It focuses on the internal relationships of the generated word sequence through self-attention, while interacting with the semantic features from the encoder through cross-attention. This allows it to fuse key semantic information from before noise interference when predicting the next word at each step. This process ensures that even under poor channel conditions, the user’s search intent can be restored to the greatest extent possible. The semantic decoder includes multiple Transformer decoder layers, with the initial input identifier $E_{cls}$. The embedding representation of the first generated word is $E\left(x_1\right)$, and the input for generating the n-th word is the embedding sequence of the previous $n-1$ words, denoted as

$$E\left(\widehat{x}\right)=[E_{cls},E\left(x_1\right),E\left(x_2\right),\dots ,E\left(x_{n-1}\right)]$$

(6)

Self-attention computation on the current input $E\left(\widehat{x}\right)$ is represented as $S_t$

$$S_t=SelfAtten\left(E\left(\widehat{x}\right)\right)$$

(7)

The self-attention representation $S_t$ and the feature representation ${\widehat{z}}_e$ of the channel encoder output generate the cross-attention representation $C_t$

$$C_t=\mathrm{CrossAtten}(S_t,{\widehat{z}}_e,{\widehat{z}}_e)$$

(8)

Splice self-attention output $S_t$ and cross-attention output $C_t$, calculate the probability distribution of each word through the linear layer and Softmax layer, finally restore the original query sentence represented by $\widehat{s}$, set the semantic decoder represented by $S_{\chi }^{-1}(·)$, and the semantic information obtained after the above operation represented by $\tilde{z}$, then the final recovered sentence can be defined as

$$\widehat{s}=S_{\chi }^{-1}\left(\widehat{z}\right)$$

(9)

The whole semantic communication process is trained by the cross-entropy (CE) loss function to minimize the difference between the transmitted query text s and the reconstructed text $\widehat{s}$, which can be expressed as

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{CE}}(s,\widehat{s};\phi ,\alpha ,\delta ,\chi )& =\sum _{i=1}(q\left(s_i\right)-1)log\left(1-p\left(s_i\right)\right)\hfill \\ \hfill & -\sum _{i=1}q\left(s_{\mathrm{i}}\right)log\left(p\left(s_i\right)\right)\hfill \end{array}$$

(10)

where ${\mathcal{L}}_{CE}$ is the cross-entropy loss of sentence similarity, $q\left(s_i\right)$ denotes the actual probability that the word $s_i$ appears in the transmitted sentence s, and $p\left(s_i\right)$ means the predicted probability that $s_i$ appears in the estimated sentence $\widehat{s}$. By minimizing the loss function, the model can effectively leverage historical query information, guarantee the semantic accuracy of text transmission, and provide comprehensive query semantic details for following cross-modal fine-grained matching.

Through the aforementioned MSI mechanism, the system can recover high-quality and robust query text representations even in low SNR channels. This recovered semantic representation carries the user’s core search intent, laying a crucial foundation for downstream cross-modal video retrieval tasks. However, the final performance of the retrieval system depends not only on the quality of query semantic recovery but also on how to utilize this semantic information for accurate matching within the video library.

5. Cross-Modal Fine-Grained Matching Method

The core contribution of the MSI module lies in providing a robust recovered semantic representation of the query for downstream tasks. This section will elaborate on the FGM module that works in conjunction with it. The design goal of the FGM module is to accurately and efficiently measure and match the similarity between the query semantics output by the preceding MSI module and the video semantics in the candidate video set, thereby completing the retrieval from text to video. Together, they constitute the end-to-end semantic communication and retrieval shown in Figure 1.

Existing methods (such as X-Pool) utilize attention to frames semantically similar to the text, but they typically ignore the differences in importance of words within the query statement, often applying uniform pooling to words. One of the core innovations of our proposed method lies in its bidirectional fine-grained interaction. It not only performs interaction between video frames and the query statement but also specifically designs interaction between video and query terms. This allows for the assignment of differentiated weights to different words in the query, more accurately capturing the key semantics of the user’s search intent. Furthermore, the designed FGM module does not operate in isolation, the text input it processes is a query statement recovered by the preceding MSI module using historical query enhancement. Therefore, its fine-grained matching is performed on a semantically richer and more robust query representation, which is significantly different from models that match on the original or simply recovered query. This section proposes a cross-modal FGM method, as shown in Figure 4, to match video frame-query sentence and video-query words using attention mechanism to derive word representations with different weights, thereby deepening the level of cross-modal fine-grained interaction between visual and textual representations.

Figure 4. Cross−modal fine-grained semantic matching module.

5.1. Text and Video Feature Representation

This section aims to provide a high-quality feature foundation for subsequent cross-modal fine-grained matching. The extracted video frame features and text word features will serve as inputs to the subsequent ‘frame-to-sentence’ and ‘video-to-word’ interaction modules, respectively, and their representation quality directly determines the accuracy of fine-grained semantic alignment. Given the necessity of performing cross-modal matching between the recovered query text and video at the receiver to ensure accurate matching between text and video, it is imperative for multimodal semantic encoder to adapt to specific characteristics of text and video modalities, and extract semantic features accordingly. The extraction of semantic features of the two modalities is outlined as follows.

Let $t_i$ and $v_j$ represent the recovered query text and video data inputted by the FGM module, respectively. Semantic features extracted from the query text recovered at the receiver are denoted as $t_u$, and semantic features extracted from the video are represented as $v_f$. For querying text $t_i=\{t_i^1,t_i^2,\dots ,t_i^m\}\in T$, the semantic feature $t_u$ is defined as $t_u=\{t_u^0,t_u^1,t_u^2,t_u^{m-1}\}$, and the text semantic encoder in CLIP is applied to generate the text feature representation. In order to obtain video feature representation, frames are first extracted from a video clip and then encoded by a video semantic encoder. In this paper, CLIP (ViT-B/32) is introduced as a video encoder to convert video representation into image representation. The representation generated for the input frame sequence $v_j=\{v_{\mathrm{j}}^1,v_j^2\dots \dots v_j^n\}\in V$ of the video is $v_f=\{v_f^0,v_f^1,v_f^2,v_f^{n-1}\}$. Therefore, the above operations can be expressed as

$$t_u=\mathcal{E}\left(t_i;{\phi }_i\right),v_f=\mathcal{E}\left(v_j;{\phi }_j\right)$$

(11)

Here, $\mathcal{E}(;{\phi }_i)$ and $\mathcal{E}(;{\phi }_j)$ is the text semantic encoder with learnable parameter ${\phi }_i$ and the video semantic encoder with learnable parameter ${\phi }_j$, respectively.

5.2. Video Frame-Query Statement Interaction

Since continuous frames contain content displacements that reflect actual actions, this paper employs the transform difference embedded in frames between adjacent timestamps in Transfer-CLIP [48] to describe motion changes, so as to enhance the interactivity between adjacent frames. By obtaining the differentially enhanced frame video representation for cross-modal fine-grained matching with query statements, some unimportant frames can be learned and filtered out, so that the model is able to prioritize and comprehend the query intents.

The semantic transformations of the actual actions within the video content are reflected in the consecutive time frames. Differential transformations are performed between the feature representations of these adjacent time frames to derive an enhanced visual representation

$$v_d=2\zeta \left(\delta (\{v_f^1-v_f^0,v_f^2-v_f^1,\dots ,v_f^{n-1}-v_f^{n-2}\}+\mathbf{P})\right)-1$$

(12)

where, $\mathbf{P}$ is positional embedding, $v_f^{n-1}$ and $v_f^{n-2}$ are feature embedding of two adjacent frames, $\zeta$ means sigmoid function, $\delta$ signifies 1-layer self-attention, and $v_d$ represents differential enhancement marker. Differences that have long-term relationships between all segments are encoded and normalized to represent global temporal representations and capture subtle changes. Then, a differential enhancement token is inserted between each adjacent frame to obtain the final visual representation:

$$v_f=\{v_f^0,v_d^1,v_f^1,v_d^2,v_f^2,\dots ,v_d^{n-1},v_f^{n-1}\}+\mathbf{P}+\mathbf{T}$$

(13)

For query text semantic features, $t_u=\{t_u^0,t_u^1,t_u^2,t_u^{m-1}\}$ is denoted as $z_t\in {\mathbb{R}}^{1\times M}$. The text features are projected as $Q_{t_u}\in {\mathbb{R}}^{1\times L_p}$, and the video frame features $v_f\in {\mathbb{R}}^{(2n-1)\times L}$ are defined as $K_{v_f}\in {\mathbb{R}}^{n\times L_p}$ and $V_{v_f}\in {\mathbb{R}}^{n\times L_p}$, where L and M are the size of the model dimension and $L_p$ denotes the size of the projection dimension. Take the query text features as Q and the visual features as K and V, and embed them into the same dimension using the normalized layer LN and the projection matrix as

$$Q_{z_t}=\mathrm{LN}\left(z_t^T\right)W_Q{Q}_{t_u}$$

(14)

$$K_{v_f}=\mathrm{LN}\left(v_f\right)W_K$$

(15)

$$V_{v_f}=\mathrm{LN}\left(v_f\right)W_V$$

(16)

$W_Q$, $W_K$, and $W_V$ are the projection matrix $L\times L_p$ in R. Cross-modal interaction with visual feature $V_f$ is

$$\mathrm{Attention}(Q_{z_t},K_{v_f},V_{v_f})=\mathrm{softmax}\left({\displaystyle \frac{Q_{z_t}{K}_{v_f}^T}{\sqrt{L_p}}}\right)V_{v_f}$$

(17)

According to the attention values between different video frames, the weight $W_O\in {\mathbb{R}}^{L_p\times L}$ is applied to acquire a video representation based on the query text:

$$z_{v|t}=LN\left(\mathrm{Attention}(Q_{z_t},K_{v_f},V_{v_f})W_O\right)$$

(18)

Output the final feature representation via FC and LN:

$$\stackrel{\wedge }{z_{v|t}}=\mathrm{LN}\left(\mathrm{FC}\left(z_{v|t}\right)\right)+z_{v|t}{)}^T$$

(19)

5.3. Video-Query Word Interaction

Due to the varying importance of different words in the search text, this paper calculates the important weights of different words, and assigns higher scores to crucial words. By pooling the frame feature representation $v_f$ with time-difference sequence enhancement into $z_v\in {\mathbb{R}}^{1\times L}$, the visual feature $z_v$ is taken as Q, and the text feature $t_u$ is considered as K and V, the above operations are carried out as

$$\mathrm{Attention}(Q_{z_v},K_{t_u},V_{t_u})=\mathrm{softmax}\left({\displaystyle \frac{Q_{z_v}{K_{t_u}}^T}{\sqrt{L_p}}}\right)V_{t_u}$$

(20)

Applying weight $W_O\in {\mathbb{R}}^{L_p\times L}$ yields a representation of the word features based on the query video

$$z_{t|v}=LN\left(\mathrm{Attention}(Q_{z_v},K_{t_u},V_{t_u})W_O\right)$$

(21)

Through FC and LN, the final feature representation is obtained

$$\stackrel{\wedge }{z_{t|v}}=\mathrm{LN}\left(\mathrm{FC}\left(z_{t|v}\right)\right)+z_{t|v}{)}^T$$

(22)

Define the similarity function between query text and different frames as

$$s_{q-f}={\displaystyle \frac{{\mathbf{z}}_t·{\mathbf{z}}_{v|t}^{\wedge }}{\Vert {\mathbf{z}}_t\Vert \Vert {\mathbf{z}}_{v|t}^{\wedge }\Vert }}$$

(23)

The similarity between the video and different words is

$$s_{v-w}={\displaystyle \frac{{\mathbf{z}}_v·{\mathbf{z}}_{t|v}^{\wedge }}{\Vert {\mathbf{z}}_v\Vert \Vert {\mathbf{z}}_{t|v}^{\wedge }\Vert }}$$

(24)

Using the mean to calculate the correlation between the visual representation and the query representation, the final similarity function is

$$s(v_i,t_j)=(s_{q-f}+s_{v-w})/2$$

(25)

5.4. Objective Function

In this paper, a symmetric cross-entropy loss is applied to maximize the similarity between the query and the matched video, and minimize the similarity between other negative pairs. Given B pairs of video-query text pair, the model needs to generate and optimize B × B similarities using symmetric cross-entropy loss on these similarity scores to train the model parameters.

$${\mathcal{L}}_{v2t}=-{\displaystyle \frac{1}{B}}\sum _{i=1}^{B}log{\displaystyle \frac{exp\left(s(v_i,t_i)\right)}{{\displaystyle \sum _{j=1}^B}exp\left(s(v_i,t_j)\right)}}$$

(26)

$${\mathcal{L}}_{t2v}=-{\displaystyle \frac{1}{B}}\sum _{i=1}^{B}log{\displaystyle \frac{exp\left(s(v_i,t_i)\right)}{{\displaystyle \sum _{j=1}^B}exp\left(s(v_j,t_i)\right)}}$$

(27)

$$\mathcal{L}={\mathcal{L}}_{v2t}+{\mathcal{L}}_{t2v}$$

(28)

Loss $\mathcal{L}$ is the sum of video to text loss ${\mathcal{L}}_{v2t}$ and text-to-video loss ${\mathcal{L}}_{t2v}$. By training the above objective function to complete the matching module, the target video content can be matched according to the search request text.

The MSI module serves as a pre-processor and enabler for the FGM module. The restored query semantics output by MSI are the input for FGM’s similarity calculation. The restoration accuracy of MSI directly determines the semantic quality of the input FGM, thus setting the potential upper limit for the overall retrieval system’s performance. The two modules are connected through a shared semantic vector space. During encoding and restoration, MSI strives to align the query semantic vector with the original query intent within this space. FGM utilizes vector representations in the same space to calculate fine-grained similarity between text and video features. This shared representation space ensures consistency in the optimization objectives of both stages. MSI provides robust semantic input, while FGM achieves precise matching, enabling the overall framework to effectively address the challenges of low SNR channels.

6. Experiment

This section aims to comprehensively evaluate the robustness and performance advantages of the proposed SemTVR framework for text-to-video retrieval in low SNR IoV environments. The experimental platform was configured with an Ubuntu 18.04 LTS 64-bit operating system, an Intel(R) Core(TM) i9-10900K 3.70 GHz processor, an NVIDIA GeForce RTX 3090 graphics card, and 64 GB of RAM. The development tools were PyCharm 2020, PyTorch-1.13.1, and CUDA-11.6. The authoritative MSRVTT [49] and MSVD [50] datasets are used as benchmarks in the experiments. In the simulation settings, the parameters of the semantic transmission module and the wireless communication physical layer are configured in detail, and a real-world channel test environment based on the USRP hardware platform is constructed. Performance evaluation primarily uses recall (R@K) and sentence semantic similarity as core metrics. Experimental results verify the superiority of the SemTVR framework by systematically comparing the performance of the proposed method with various baseline schemes (including traditional communication schemes and existing semantic communication schemes) under different channel models (AWGN, Rayleigh, and real-world channels). The simulation analysis covers retrieval accuracy, semantic recovery quality, the contribution of each core module, and the visualization of the verification results. All validation results reported in this paper (such as R@1 and semantic similarity) are averages calculated after 100 independent repeated experiments for each SNR condition.

6.1. Dataset

We conduct validations using the authoritative datasets MSRVTT [49] and MSVD [50], which are well-established standard datasets in the field of text-to-video retrieval, consisting of rich pairs of text and video data. MSR-VTT: It comprises 10,000 video clips, each annotated with 20 descriptive sentences generated by human annotators via Amazon Mechanical Turk (AMT). The content spans a broad spectrum of daily activities and public scenes, including traffic-related actions (e.g., driving, crossing), which are highly relevant to vehicular perception tasks. Our experiments are performed on the MSRVTT1k-A split, which includes 9000 videos for training and 1000 videos for testing.

MSVD: It consists of 1970 short video clips, ranging from 1 to 62 s in duration. Each clip is associated with approximately 40 English subtitles, capturing detailed descriptions of dynamic actions and static objects (such as pedestrian behaviors and natural landscapes). In our experiments, we use 1200 videos for training, 100 videos for validation, and 670 videos for testing.

6.2. Simulation Settings

In the initial phase of the semantic transmission task for IoV retrieval, the transmitter is equipped with three typical Transformer encoders and one Cross Transformer encoder, while the receiver is set up with a corresponding number of decoders. The channel encoder and decoder are densely configured with 16 and 128 units, respectively. The specific parameters are detailed in Table 1.

Table 1. Parameter settings of semantic transmission.

	Layer Name	Units	Activation
Transmitter	3 × Typical Transformer Encoder	128 (8 heads)	Linear
	1 × Cross Transformer Encoder	128 (8 heads)	Linear
	Dense	128	Sigmod
	Dense	256	ReLU
	Dense	16	ReLU
Channel	AWGN/Rayleigh	None	None
Receiver	Dense	256	ReLU
	Dense	128	ReLU
	3 × Cross Transformer Decoder	128 (8 heads)	Linear
	Prediction Layer	Dictionary Size	Softmax

In the second part of the text-to-video matching task, the text is encoded using ViT-B/32 in CLIP as the default video encoder and the default text encoder in CLIP. The pre-training weight of CLIP is applied to initialize all encoder parameters. The dimension of query, key and value projection is set to Dp = 512, the input resolution of video frame is set to 224 × 224, maximum sampling frame m is 12, and maximum text length is 32. Set the clip initialization weight learning rate to 1 × 10⁻⁶, use the Adam optimizer to train 5 epochs with a batch size of 128, and set the weight decay to 0.2.

In order to verify the performance of the proposed method in real-world wireless transmission scenarios, this paper builds a USRP-2974 based semantic communication platform, and the communication system suite is LabVIEW Communications System Design Suite 2.0, which is implemented based on LTE protocol. Two USRP devices are used for wireless transmission experiment, one for transmission and the other for reception. Firstly, a trained encoder network is introduced to encode the query text, and the encoded symbols are sent to the USRP through the LAN of UDP protocol, and the received symbols are transmitted wirelessly by the USRP. The receiver decodes the signal and inputs subsequent text into the video matching network for video retrieval. The USRP2974 technical layer parameter settings are given in Table 2.

Table 2. USRP2974 technical layer parameter settings.

Experimental Parameter	Parameter Value
Antenna distance/m	1.5
Transceiver frequency/GHz	2
Modulation mode	64QAM
Bit rate	0.43
Bandwidth/MHz	40

6.3. Experimental Result Analysis

In this paper, to comprehensively and fairly evaluate the performance of the proposed SemTVR framework, it is compared with the methods using traditional communication and DeepSC [37] based semantic communication, and the text-to-video retrieval baseline model Clip4 is introduced. The experimental design generally adopted a combination of a communication module and a retrieval module (e.g., Huffman + CC + CLIP4). This design aims to achieve modular comparison. By fixing the retrieval module, the capabilities of different communication schemes (traditional coding, DeepSC, MSI) in combating channel interference and recovering query semantics can be evaluated separately. By fixing the communication module, the advantages and disadvantages of different retrieval models (e.g., CLIP4, FGM) in cross-modal matching accuracy can be evaluated separately. This combined comparison clearly reveals the contributions and synergistic effects of the two core innovative modules in the SemTVR framework, MSI and FGM. “+” is used to connect the two components.

Traditional communication methods include Huffman + CC + CLIP4 [11] and Huffman + CC + FGM, where CC is convolutional code, CLIP4 adopts seqTransf structure, and FGM is the cross-modal fine-grained matching method adopted in this paper.

Methods based on semantic communication: DeepSC + CLIP4 and DeepSC + FGM.

Multi-semantic transmission method proposed in this paper: MSI + CLIP4.

6.3.1. Comparison of R@1 Under MSR-VTT Dataset

Figure 5 shows the results on the MSR-VTT dataset under different SNR conditions. We compare SemTVR with the baseline method under different channel conditions: AWGN, Rayleigh channel, and real channel (the real channel is the experiment conducted under the USRP physical device).

Figure 5. R@1 values on the MSR−VTT dataset under different SNR conditions.

When comparing the performance of different semantic communication models under AWGN channels, it can be seen that the recall rate of query text transmission using traditional communication methods is low under low SNR conditions. Although there is a significant improvement in low SNR scenarios when replacing traditional communication methods with DeepSC for semantic transmission, the recall rate is lower than that of SemTVR because the historical query semantic information is not used during the transmission process. The proposed multi-semantic interaction reliable transmission method (MSI) makes full use of the user’s historical query information, making the semantic transmission more robust. Compared with DeepSC for text transmission, the recall rate is improved by 5.5%.

When comparing the performance of different models for text-to-video retrieval, although the CLIP4 model shows good performance, due to the lack of cross-modal fine-grained matching, the interaction between the query text and the video is not enough, resulting in the inadequate utilization of important frame and word information. Therefore, the cross-modal fine-grained matching method (FGM) adopted in this paper is 1.9% higher than the CLIP4 model.

In addition, the proposed method also shows superior performance under Rayleigh channels and real channels using USRP transmission. This highlights the effectiveness of SemTVR in various challenging communication environments.

6.3.2. Comparison of R@1 Under MSVD Dataset

Figure 6 shows the results on the MSVD dataset under three different channel environments. The results are basically consistent with those on the MSR-VTT dataset, which proves the robustness and versatility of our model and its ability to maintain stable performance under different channel conditions. Under AWGN, Rayleigh, and real channels, the proposed SemTVR improves the recall rate by 2.5%, 0.5%, and 0.7%, respectively, compared with the CLIP4 model when the proposed multi-semantic interactive transmission method (MSI) is used in the transmission stage, proving the superiority of our cross-modal fine-grained matching module (FGM).

Figure 6. R@1 values on the MSVD dataset under different SNR conditions.

When the SNR is 6 dB, the performance of DeepSC combined with the proposed cross-modal fine-grained matching method in the transmission stage is not as good as the traditional method Huffman coding combined with it. This shows that if DeepSC cannot fully recover the sentence semantics, its text-to-video retrieval performance is not as good as the traditional method that accurately recovers the text. However, using the proposed multi-semantic interactive transmission method, there is a significant improvement, indicating that it is effective and necessary to effectively use historical query information as an enhanced text representation.

In real channel environments, at an SNR of 18 dB, traditional coding methods that accurately recover the query text show excellent performance. However, the proposed method, by integrating more semantic information at the receiver and conducting fine-grained interactions with the video, enables the model to learn the semantic information of important frames and words, so that it can achieve competitive performance.

In terms of ablation study, firstly, with the retrieval module fixed, different transmission schemes were compared. Specifically, the performance difference between SemTVR (MSI + FGM) and DeepSC + FGM under the same SNR conditions was compared. This comparison directly measures the value of introducing historical queries for multi-semantic interactive transmission compared to basic semantic communication (DeepSC) in improving the robustness of semantic recovery. Secondly, with the transmission module fixed, different retrieval schemes were compared. Specifically, the performance difference between SemTVR (MSI + FGM) and MSI + CLIP4 was compared. This comparison directly measures the value of the proposed cross-modal fine-grained matching method compared to classic global matching methods (such as CLIP4) in improving matching accuracy. As shown in Figure 5 and Figure 6, the MSI and FGM modules of the proposed method achieve significant performance gains compared to existing methods under various channel models and SNR conditions, fully demonstrating the independent contribution of the designed modules.

These results validate that the proposed SemTVR method can effectively ensure search performance across various text-to-video retrieval datasets under different SNR scenarios.

6.3.3. Sentence Similarity Comparison

Figure 7 demonstrates the relationship between the similarity score of recovered sentences and different SNRs under AWGN and Rayleigh channels. The results indicate the superiority of the proposed semantic transmission model over DeepSC in recovering semantic information. At −3 dB, it exceeds by 0.1% in the AWGN channel and 0.06% in the Rayleigh channel indicating that the proposed semantic transmission module in this paper has an advantage in extracting and recovering text semantic features.

Figure 7. Comparison of semantic similarity in MSVD dataset under different SNR conditions.

Although the designed semantic transmission model may not perform as well as traditional encoding methods in recovering sentences at high SNRs, this is not a significant drawback for subsequent text-to-video retrieval tasks. The proposed decoder module, which integrates historical semantic information, provides richer semantic content in the recovered sentences. This enriched semantic representation enhances the subsequent text-to-video retrieval process, even if the original text is not perfectly restored.

6.3.4. Performance Comparison of Fine-Grained Matching

In this section, we compare the performance metrics of existing state-of-the-art methods in the field of text-video retrieval with those of the MSRVTT-1k-A and MSVD datasets, respectively, using R@1, R@5 and R@10 performance metrics.

Table 3 presents the performance results on the MSRVTT-1k-A dataset. It can be observed that utilizing the CLIP pre-trained model for retrieval surpasses the performance of most fine-tuning methods. Although the use of CLIP does not fully exploit fine-grained interactions between video and text, the strong spatial representation in the videos compensates for the loss of critical semantic information. The proposed model demonstrates significant performance improvements over all baseline models. This indicates that further fine-grained matching, based on enhanced frame-level representations through temporal differential blocks, yields substantial benefits.

Table 3. Comparison of text-to-video retrieval methods in dataset MSRVTT 1k-A.

Method	R@1	R@5	R@10
MMT-Pretrained [51]	26.6	57.1	69.6
SUPPORT-SET [52]	27.4	56.3	67.7
T2VLAD [53]	29.5	59.0	70.1
FROZEN [54]	31.0	59.5	70.5
CLIPforward [55]	31.2	53.7	64.2
HIT-pretrained [56]	30.7	60.9	73.2
MDMMT [57]	38.9	69.0	79.7
CLIP4Clip-meanP [11]	43.1	70.4	80.8
CLIP4Clip-seqTransf [11]	44.5	71.4	81.6
Transfer-Clip [48]	45.6	72.6	81.7
ours	45.9	73.8	82.5

Table 4 shows the results of MSVD compared with X-pool, which achieves the best performance, this paper has improved the retrieval performance in various indicators. Once again, it proves the necessity of fine-grained interaction with different frame-sentences and video-query words.

Table 4. Comparison of text-to-video retrieval methods in dataset MSVD R@k.

Method	R@1	R@5	R@10
VSE [58]	12.3	30.1	42.3
CE [59]	19.8	49.0	63.8
SSML [60]	20.3	49.0	63.3
SUPPORT-SET [52]	28.4	60.0	72.9
FORZEN [54]	33.7	64.7	76.3
CLIPforward [55]	37.0	64.1	73.8
CLIP4Clip-seqTransf [11]	46.2	76.1	84.6
CLIP4Clip-meanP [11]	45.2	75.5	84.3
Transfer-Clip [48]	47.1	76.9	85.9
X-pool [32]	47.2	77.4	86.0
ours	47.8	77.9	86.3

6.3.5. Visual Analysis of Results

Figure 8 provides a visual analysis of the receiver’s recovery of query sentences and retrieval videos using MSI and FGM. In this figure, words with higher weights assigned by the FGM model are highlighted in orange. Under low SNR conditions (3 dB), it is evident that traditional encoding methods (Huffman + CC) can only recover the semantics of individual words. DeepSC tends to restore words unrelated to the query text, leading to deviations in retrieval results. In contrast, the proposed model effectively restores the semantics of the query text by incorporating historical query texts under low SNR conditions. The FGM model focuses on important words in the query statement, enabling the identification of relevant frame information in the video library and accurately retrieving the video.

Figure 8. Analysis of visual results in an AWGN channel with an SNR of 3 dB.

Figure 9 illustrates the retrieval error of the proposed method, which occurs due to the excessive use of semantic information from historical queries when recovering text. This results in the recovered sentence containing more irrelevant semantic information, leading to deviations in retrieval results.

Figure 9. Analysis of visual results of errors.

6.3.6. End-to-End Latency

To evaluate the real-time performance of the edge–cloud collaborative architecture, we simulated end-to-end latency on the Rayleigh channel, simulating a scenario of 10 vehicles making concurrent requests. Communication parameters were set according to the USRP2974 platform. Experiments compared the performance of cloud-based (Huffman + CC + FGM), edge–cloud collaboration (DeepSC + FGM), and our proposed edge–cloud collaboration (SemTVR) on the MSR-VTT and MSVD datasets, as shown in Figure 10 and Figure 11.

Figure 10. End−to−end latency on the MSR-VTT dataset under different SNR conditions.

Figure 11. End−to−end latency on the MSVD dataset under different SNR conditions.

The results show that the edge–cloud collaborative architecture significantly reduces latency by offloading semantic recovery and matching computation to the RSU, maintaining latency in the hundreds of milliseconds range even at SNRs as low as −15 dB, outperforming traditional solutions. Specifically, SemTVR consistently exhibits the lowest latency due to its enhanced transmission robustness and reduced retransmissions via the MSI mechanism, coupled with efficient matching by the FGM module. Furthermore, the latency curve of the MSR-VTT dataset fluctuates between −9 dB and −3 dB, possibly due to uneven computational load caused by its complex content. The MSVD dataset, on the other hand, exhibits a smoother decreasing trend, reflecting the regularity of its content. Simulation results confirm the potential of the proposed architecture to meet real-time interaction requirements in low SNR and multi-user concurrent scenarios.

6.3.7. Communication Overhead

To evaluate the semantic preservation capabilities of different transmission schemes under limited bandwidth, we compared the performance of Huffman + CC, DeepSC, and SemTVR (MSI) on AWGN and Rayleigh channels under a fixed SNR (3 dB) by controlling the average number of symbols transmitted per word. Semantic recovery accuracy was measured using the BLEU score (1-g), reflecting the ability to preserve core vocabulary, as shown in Figure 12.

Figure 12. BLEU scores under different communication overhead.

The results show that under the same symbol count constraint, SemTVR consistently achieves the highest BLEU score, especially at low symbol counts (e.g., 1 symbol/word), demonstrating its multi-semantic interaction mechanism can more efficiently compress and recover task-related semantics. The traditional scheme (Huffman + CC) suffers from severely limited performance at low bit rates, exhibiting weak growth and significant fluctuations, highlighting its inherent limitations of sensitivity to channel noise and insufficient semantic preservation capabilities. The curves of the two semantic communication schemes are flatter and more stable, reflecting the superiority of the semantically-oriented transmission paradigm in bandwidth utilization.

Furthermore, all schemes perform worse on Rayleigh channels than on AWGN channels, but the performance ranking remains unchanged. SemTVR exhibits relatively less performance degradation under Rayleigh channels, indicating that its design, which utilizes historical context to enhance semantic representation, improves robustness against channel fading and is more adaptable to the dynamic environment of vehicular networks.

6.3.8. Cache Strategy Verification

To evaluate the impact of historical query cache management strategies, we systematically compared three strategies, First-In-First-Out (FIFO), Last-In-First-Out (LIFO), and Random replacement, as shown in Table 5 and Table 6.

Table 5. R@1 under different cache replacement strategies on the MSR-VTT dataset.

No. of Historical Query	Channel Model	SemTVR + FIFO	SemTVR + LIFO	SemTVR + Random
20	AWGN	41.0	39.8	40.1
	Rayleigh	36.5	35.0	34.2
	Real	39.2	37.5	37.9
40	AWGN	42.8	41.5	40.0
	Rayleigh	38.0	38.4	36.0
	Real	40.0	39.2	38.5
60	AWGN	43.7	42.0	41.3
	Rayleigh	38.9	37.8	36.8
	Real	40.5	39.8	39.0
80	AWGN	44.2	42.8	43.1
	Rayleigh	39.3	38.5	37.5
	Real	40.7	40.0	39.4
100	AWGN	44.8	43.2	43.0
	Rayleigh	39.8	39.0	37.8
	Real	40.9	40.3	39.6

Table 6. R@1 under different cache replacement strategies on the MSVD dataset.

No. of Historical Query	Channel Model	SemTVR + FIFO	SemTVR + LIFO	SemTVR + Random
20	AWGN	43.3	42.5	42.0
	Rayleigh	40.3	40.4	38.8
	Real	40.5	39.9	39.2
40	AWGN	44.5	43.6	43.1
	Rayleigh	40.9	40.5	39.5
	Real	41.5	41.0	40.3
60	AWGN	45.0	43.9	44.0
	Rayleigh	41.2	40.8	39.8
	Real	41.8	41.4	40.5
80	AWGN	45.1	44.0	44.2
	Rayleigh	41.3	41.0	40.0
	Real	41.9	41.6	40.7
100	AWGN	45.2	44.1	44.3
	Rayleigh	41.3	41.1	40.1
	Real	41.9	41.7	40.8

The results show that FIFO performs best in most scenarios, especially when the channel is good or the cache capacity is large. Its mechanism of maintaining the continuity of user interests through a sliding window is the most robust, making it a reliable foundational strategy for engineering deployments. The LIFO strategy exhibits strong scenario dependence: when user interests change abruptly, discarding the latest record may inadvertently preserve more valuable earlier context, allowing it to approach FIFO performance under certain conditions. However, when interests evolve steadily, this strategy loses key recent information, leading to a general performance degradation. The Random strategy is the most unstable, with highly random performance and a lack of optimization direction.

6.3.9. R@1 Under Extreme SNR Conditions

To verify the retrieval performance of the proposed method under extreme SNR conditions, we verified R@1 in extreme channel environments with SNR ranging from −3 dB to −15 dB, thereby exploring the performance boundaries of traditional methods and semantic communication-based methods.

As presented in Figure 13 and Figure 14, all methods exhibit a clear exponential decay trend, meaning that the performance degradation rate is initially rapid and then slows down as the SNR decreases. However, the proposed SemTVR (MSI + FGM) method maintains the highest R@1 value at all tested SNR points, and its performance degradation curve is the flattest. This indicates that the complete framework integrating multi-semantic interactive transmission and cross-modal fine-grained matching possesses the strongest anti-interference capability and semantic information preservation capability in the face of deep channel fading. We also note that when the SNR drops to −15 dB, the R@1 values of almost all methods approach 0, indicating that existing methods need further optimization in the complex channel environment of vehicular networks.

Figure 13. R@1 of MSR-VTT under extreme SNR conditions.

Figure 14. R@1 of MSVD under extreme SNR conditions.

However, the in-vehicle text–video retrieval service is primarily positioned to provide passengers with infotainment or navigation assistance content, rather than a mission-critical application directly related to vehicle control or driving safety. Therefore, even if a temporary performance degradation or even interruption of the retrieval service occurs in extreme scenarios such as tunnels or urban canyons, its direct impact on vehicle safety and passengers is relatively limited and controllable.

7. Conclusions

Existing methods often neglect the low SNR characteristics of IoV and lack fine-grained interaction between search requests and video contents, which limits the performance of text-to-video retrieval systems. This paper proposes a text-to-video retrieval method for IoV based on semantic communication, addressing the complex channel characteristics in IoV scenarios. For the first time, the semantic communication paradigm is introduced, and a text-to-video retrieval architecture based on semantic communication is designed. A reliable multi-semantic interactive transmission method and a cross-modal fine-grained semantic matching method are proposed to improve the accuracy of text-to-video retrieval under low SNR conditions in IoV. Experimental results demonstrate the superior performance of the proposed method. While some general information security standards exist, text–video retrieval in the connected vehicle environment involves multimodal semantic communication and edge–cloud collaborative architecture. These standards may not be able to directly address specific privacy risks such as leakage of query semantics during transmission and malicious use of users’ historical search records. Therefore, it is necessary to design privacy protection methods specifically for such scenarios.

In the future, we will explore how to achieve privacy protection within a semantic communication framework. For example, we will research distributed model training schemes based on federated learning, enabling models to be optimized without centralized raw query data, or introduce differential privacy techniques to add controllable noise during the semantic encoding stage to protect sensitive user information in the transmitted content. Currently, our research focuses on single-user retrieval tasks. Future work can explore collaborative retrieval scenarios involving multiple passengers and multiple points of interest within the IoV, designing efficient request scheduling and resource allocation mechanisms, and studying the interaction and knowledge sharing of interest models among users. Furthermore, in practical deployments, state consistency presents a core engineering challenge in distributed environments. In scenarios involving high-speed vehicle movement and intermittent network connectivity, it is difficult for vehicles, RSUs, and the cloud to maintain real-time updates of data and models. Ensuring efficient synchronization and eventual consistency of RSU cached feature libraries, model parameters, and historical states between the edge and the cloud is a key area requiring further research. Future research needs to investigate how to ensure efficient synchronization of models, feature libraries, and states, and seamless service continuity in extreme scenarios with high-speed vehicle movement, frequent network topology changes, and stronger connectivity intermittency. For example, we can explore distributed caching and predictive preloading strategies assisted by vehicle-to-vehicle communication.