LDMP-FEC: A Real-Time Low-Latency Scheduling Algorithm for Video Transmission in Heterogeneous Networks

Abstract

With the rapid development of mobile networks and devices, real-time video transmission has become increasingly important worldwide. Constrained by the bandwidth limitations of single networks, extensive research has shifted towards video transmission in multi-network environments. However, differences in bandwidth and latency in heterogeneous networks (such as LTE and Wi-Fi) lead to high latency and packet loss issues, severely affecting video quality and user experience. This paper proposes a Forward Error Correction (FEC)-based Low-Delay Multipath Scheduling algorithm (LDMP-FEC). This algorithm combines the Gilbert model with a continuous Markov chain to adaptively adjust FEC redundancy, thereby enhancing data integrity. Through the FEC Recovery Priority Scheduling (FEC-RPS) algorithm, it dynamically optimizes the transmission order of data packets, reducing the number of out-of-order packets (OFO-packets) and end-to-end latency. Experimental results show that LDMP-FEC significantly reduces the number of out-of-order packets in heterogeneous network environments, improving performance by 50% compared to the round-robin and MinRtt algorithms, while maintaining end-to-end latency within 150 ms. Under various packet loss conditions, LDMP-FEC sustains a playable frame rate (PFR) above 90% and a Peak Signal-to-Noise Ratio (PSNR) exceeding 35 dB, providing an efficient and reliable solution for real-time video and other low-latency applications.

1. Introduction

With the widespread adoption of mobile networks and mobile devices, live video streaming has become increasingly popular worldwide. Ensuring stable and reliable video streaming services has become a critical challenge. The primary constraints for real-time video transmission are network bandwidth and packet loss. The limited bandwidth provided by a single network is often insufficient to effectively support the transmission of high-bitrate videos. Thanks to advancements in mobile devices, modern smartphones and computers can simultaneously utilize 4G networks and Wi-Fi, providing a foundation for transmitting higher-quality videos. However, due to significant differences in network characteristics such as latency and bandwidth between 4G and Wi-Fi, these heterogeneous networks struggle to deliver low-latency, high-quality video services. As a result, in recent years, multi-stream video transmission using multiple access links has been extensively researched. For instance, EVOLUTE implements a full-IP network infrastructure aimed at providing seamless multimedia services for roaming users [1]. It attempts to address the ubiquitous service provision challenges in heterogeneous environments [2]. Additionally, efforts have been made to apply the high-bandwidth characteristics of heterogeneous networks to drones, leading to the development of the ETF algorithm, which explores fast and resource-efficient drone transitions to enable aerial video group communication while maintaining high group communication performance. MPDASH [3,4,5] introduced a deadline-sensitive multipath scheduler [5] within Dynamic Adaptive Streaming HTTP (DASH) to optimize video transmission. However, it is only suitable for stable network environments, and its performance significantly degrades in mobile networks. Some researchers have explored the use of deep reinforcement learning (DRL) to predict network coding, as seen in studies [6,7,8,9,10], leading to several DRL-based solutions such as Meta-DAMS [11] and DRL360 [12]. However, these methods focus on solving packet scheduling issues and fail to address packet loss during actual transmission. For example, GaDam [13] and Peekaboo [14] leverage reinforcement learning to control the scheduling of data streams across different subflows. Yet, when packet loss occurs, these methods lack effective mechanisms for data recovery, which severely impacts the video reconstruction quality at the receiver. Heterogeneous networks typically consist of a combination of 4G and Wi-Fi, as illustrated in Figure 1. Due to the wireless environment, partial data loss frequently occurs, and significant network jitter greatly affects video transmission quality. The aforementioned methods fail to effectively address these challenges.

In addition to the aforementioned applications of heterogeneous networks for transmission, some researchers have explored the use of Forward Error Correction (FEC) to protect data transmission, thereby addressing the issue of packet loss in channels [4,15,16,17]. However, most of these studies focus on integrating FEC encoding with video bitrates, and their scope is largely confined to homogeneous networks with minor variations. In highly heterogeneous networks, these solutions often emphasize maintaining video quality rather than ensuring transmission latency, overlooking the significant number of out-of-order packets (OFO-packets) generated during transmission. Some researchers have begun to utilize Multipath TCP (MPTCP) to enhance mobile video quality. However, directly employing MPTCP does not always guarantee satisfactory transmission performance, especially in heterogeneous networks. This is primarily because MPTCP struggles to provide low-latency transmission services for latency-sensitive applications in such environments, as it cannot effectively address the issue of OFO-packets [15].

To enable MPTCP to deliver low-latency transmissions, numerous scheduling methods have been proposed to reduce OFO-packets caused by network path asymmetry. The fundamental approach of these methods involves estimating the arrival time of packets based on current network conditions and subsequently scheduling packet transmissions based on these estimated arrival times rather than their inherent sequence numbers [18,19,20]. However, the issue of OFO-packets persists in practical applications because existing schedulers may not perform well in mobile networks characterized by significant jitter. In some cases, even well-designed schedulers perform worse than simple round-robin schedulers [21].

Furthermore, since real-time video transmission demands stringent real-time performance, and MPTCP is fundamentally based on TCP, alternative approaches should consider using UDP in combination with FEC for data transmission to better meet real-time performance requirements.

To address the aforementioned issues of packet loss and OFO-packets, we propose a Forward Error Correction (FEC)-based Low-Delay Multipath Scheduling algorithm (LDMP-FEC). The core of this method involves using FEC encoding to protect data integrity based on the current network environment and adaptively adjusting FEC redundancy through the Gilbert model and continuous Markov chains. Subsequently, for the generated encoding groups, the proposed FEC Recovery Priority Scheduling (FEC-RPS) algorithm is employed to schedule data packet transmissions, with the primary objective of recovering FEC encoding groups in the shortest possible time.

The fundamental idea is to calculate the estimated arrival delay intervals for different subflows. When these intervals overlap, the packet sending order is adjusted from the minimum Round-Trip Time (MinRTT) method to round-robin scheduling, ensuring that packets sent within adjacent time frames arrive simultaneously. This approach allows FEC decoding to recover lost or delayed packets at the fastest possible speed.

To evaluate the proposed method, we constructed a UDP-based transmission platform and compared LDMP-FEC with existing algorithms. The results indicate that the LDMP-FEC algorithm significantly reduces the number of out-of-order packets in heterogeneous network environments, improving performance by 50% compared to the round-robin and MinRTT algorithms. Additionally, it enhances video transmission quality by ensuring that end-to-end latency remains within 150ms and maintaining a playable frame rate (PFR) above 90% and a Peak Signal-to-Noise Ratio (PSNR) exceeding 35dB across various packet loss scenarios. This provides an efficient and reliable solution for real-time video and other low-latency applications:

• Proposing the LDMP-FEC algorithm to ensure video transmission quality in heterogeneous networks: This is achieved by using FEC encoding to protect transmitted data and adaptively deriving FEC redundancy based on the Gilbert model and continuous Markov chains to counteract packet loss in the channel.
• Modeling and deriving the expected arrival time intervals of data packets on each subflow based on current channel conditions: When the arrival time intervals of subflows do not overlap, data are sent through the shortest subflow. When intervals overlap, the scheduling algorithm switches to round-robin scheduling to ensure that data packets can arrive at the receiver end simultaneously within the shortest possible time for FEC decoding. This reduces waiting times and lowers the end-to-end latency of the video transmission.

2. Causes of OFO-Packets and the FEC Mechanism

In real-time video transmission, the original data packets are encoded in groups, and the receiver must receive the entire group of packets to perform complete decoding. However, due to significant latency differences in heterogeneous networks, the order in which packets arrive via multiple paths often does not match the order in which they were sent. To accommodate OFO-packets, the receiver requires a large buffer to hold the incoming data, which inevitably introduces greater end-to-end latency.

As shown in Figure 2, after grouping the original video data packets, they are scheduled to two different subflows. $Subflow\#1$ has a lower RTT, allowing packets to arrive quickly. In contrast, $Subflow\#2$ experiences a higher RTT during transmission and is accompanied by a certain degree of packet loss. In this scenario, the receiver must wait for the packets from $Subflow\#2$ to arrive before delivering the data to the upper-layer application.

However, due to the partial packet loss in $Subflow\#2$, it is impossible for the receiver to obtain a complete set of data. When the waiting time exceeds the maximum threshold, incomplete data are forcibly delivered to the upper-layer decoder, resulting in errors in video frame decoding. This leads to visual artifacts such as screen tearing, increased latency, and stuttering in video playback. To address this issue, the current mainstream solution is to employ FEC to reduce latency and protect data packets. The mechanism of FEC operates as follows.

As shown in Figure 3, the receiver has already received data packets numbered 1 to 4 and only needs to receive two additional packets to perform decoding. In the figure, data packet 5 experiences delays due to channel congestion, and data packet 6 is lost. Without FEC protection, it would be impossible to receive the complete set of data. However, because two redundant packets, 7 and 8, were generated using FEC before transmission, when these two repair packets arrive, the receiver can perform FEC decoding to directly recover lost data packet 6 without waiting for data packet 5. This demonstrates that using FEC to protect real-time video transmission is highly effective.

3. Low Delay Multipath FEC (LDMP-FEC) Scheme

3.1. System Overview

To address the aforementioned issues, we propose a multipath video transmission model based on FEC, which utilizes FEC technology to recover lost data during transmission. Additionally, we introduce a multipath data scheduling algorithm, FEC-RPS, that prioritizes FEC recovery at the receiver end. This ensures that video frames are recovered at the receiver in the shortest possible time, thereby reducing video transmission latency.

Figure 4 illustrates the data transmission model of the entire video transmission system. At the sender side, the video application captures raw uncompressed video streams through a camera and compresses them using an H.264 encoder. The compressed video is then segmented into fixed-size data packets, which are placed into the video buffer. The encoding unit takes a fixed number of data packets each time for FEC encoding, with the FEC redundancy allocated by the parameter control unit. The appropriate redundancy level is selected based on the set target recovery rate. The FEC-encoded packets, along with the original data packets, are then moved to the sending buffer. The path scheduler dynamically selects suitable subpaths for transmission based on the current network conditions, ensuring that the data packets reach the receiver in the shortest possible time.

At the receiver side, each subflow independently receives its respective data packets, which are then grouped and merged before being placed into the receiving buffer awaiting FEC decoding. Simultaneously, the receiver sends feedback information to the sender to update the FEC encoding parameters. The original video data, after FEC decoding, is placed into the video decoding queue buffer, waiting to be decoded and played back.

3.2. FEC Encoding

In our model, we adopt RaptorQ codes as the FEC encoding scheme. RaptorQ [22] is a type of fountain code, and compared to traditional FEC schemes, the most important characteristic of fountain codes is that they are rateless—the number of generated redundant packets is not fixed. In other words, the sender can dynamically adjust the number of encoded data packets based on network conditions, providing a better recovery rate and better adapting to fluctuating network environments. As shown in Figure 5, for the generated video sequence in our work, we employ GoP-level FEC encoding to protect the data. Specifically, during each FEC encoding process, data frames within a video GoP are segmented into data packets of equal size S. For the last data packet, zero padding is applied if necessary to ensure its size reaches S.

The segmented data packets are then placed into the sending buffer. The FEC encoding module extracts a fixed number K of original data packets and encodes them to generate N data packets, where $(N-K)$ represents redundant packets, denoted as $FEC(N,K)$. The sender executes the scheduling algorithm to transmit all N data packets across different subflows. After transmission through lossy channels, the receiver’s subflows receive and merge the data. For groups with some lost data packets, as long as the number of received packets is greater than or equal to K, the receiver can directly recover the missing data, ensuring data integrity. In static networks, FEC schemes typically use a fixed redundancy level for encoding. However, in heterogeneous networks, multiple paths exist with varying packet loss rates and latencies on each path, making fixed redundancy unsuitable. Therefore, we propose an adaptive FEC redundancy algorithm tailored for multipath environments.

3.3. FEC Adaptive Redundancy Algorithm

In the process of real-time video transmission, two types of packet loss are primarily considered: one is the transmission packet loss rate ${\mu }_p^t$, which occurs during data transmission; the other is the active out-of-time packet loss rate ${\mu }_p^o$, which applies to data packets that exceed their permissible time limit to ensure smooth video streaming at the receiving end. Therefore, for a specific subflow, its actual packet loss rate ${\mu }_p$ can be expressed as follows:

$${\mu }_p={\mu }_p^t+(1-{\mu }_p^t)·{\mu }_p^o$$

(1)

We define the actual packet loss rate of the entire transmission process as P. It only needs to satisfy $P\le {\displaystyle \frac{N-K}{N}}$, where the receiving end can recover all lost data. At this time, the actual packet loss rate can be directly regarded as 0. Otherwise, the actual packet loss rate P can be considered as follows:

$${\displaystyle \frac{{\sum }_{p=1}^J{R}_p·{\mu }_p}{{\sum }_{p=1}^J{R}_p}}$$

(2)

Therefore, the actual packet loss rate P for the entire transmission process can be expressed as follows:

$$P=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}{\displaystyle \frac{{\sum }_{p=1}^J{R}_p·{\mu }_p}{{\sum }_{p=1}^J{R}_p}}\le {\displaystyle \frac{N-K}{N}},\hfill \\ {\displaystyle \frac{{\sum }_{p=1}^J{R}_p·{\mu }_p}{{\sum }_{p=1}^J{R}_p}},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(3)

Here, $R_p$ represents the data transmission rate on this path, and J represents the number of paths. The selected transmission model for the channel is the Gilbert model, where the transmission process satisfies the continuous Markov chain. Now, we use the following equation to represent the transmission packet loss rate ${\mu }_p^t$ of a path:

$${\mu }_p^t={\displaystyle \frac{1}{n_p}}\sum _{c_p\in {\mathcal{V}}_{c_p}}\mathcal{L}\left(c_p\right)·\mathsf{\Pi }\left(c_p\right)$$

(4)

Here, $c_p$ represents a specific $n_p$-tuple configuration during transmission, indicating any possible transmission result, where $n_p$ is the total number of transmitted packets. $\mathcal{L}\left(c_p\right)$ represents the number of FEC data packets lost in the given $c_p$. The specific expression of $\mathcal{L}\left(c_p\right)$ is as follows:

$$\mathcal{L}\left(c_p\right)=\sum _{i=1}^{n_p}\mathbf{1}\{c_p^i=B\}$$

(5)

Let $\mathsf{\Pi }\left(c_p\right)$ represent the failure probability of the $n_p$-tuple configuration. Based on the continuous-time Gilbert loss model, $\mathsf{\Pi }\left(c_p\right)$ is derived using the product of state transition probabilities. We use the symbol $F_p^{\langle i,j\rangle }\left({\theta }_p\right)$ to represent the transition probability of path p from state i to state j over a time interval ${\theta }_p$, which is expressed as follows:

$$F_p^{\langle i,j\rangle }\left({\theta }_p\right)=\mathsf{\Pi }[X_p\left({\theta }_p\right)=j\mid X_p\left(0\right)=i]$$

(6)

Based on the properties of the continuous-time Markov chain, we can obtain the following state transition matrix:

$$\left[\begin{array}{cc}{F}_p^{\langle G,G\rangle }\left({\theta }_p\right)& F_p^{\langle G,B\rangle }\left({\theta }_p\right)\\ F_p^{\langle B,G\rangle }\left({\theta }_p\right)& F_p^{\langle B,B\rangle }\left({\theta }_p\right)\end{array}\right]=\left[\begin{array}{cc}{\mu }_p^G+{\mu }_p^B·k_p& {\mu }_p^B-{\mu }_p^B·k_p\\ {\mu }_p^G-{\mu }_p^G·k_p& {\mu }_p^B+{\mu }_p^G·k_p\end{array}\right]$$

(7)

Here, $k_p=e^{-({\xi }_p^B+{\xi }_p^G)·{\theta }_p}$. Now, assuming that the time intervals of data packets are the same, we can calculate $\mathsf{\Pi }\left(c_p\right)$. For $n_p=3$, taking $c_p^3=B\mid c_p^2=B\mid c_p^1=G$ as an example, we have the following:

$$\mathsf{\Pi }[c_p^3=B\mid c_p^2=B\mid c_p^1=G]={\mu }_p^G·F_p^{\langle G,B\rangle }\left({\theta }_p^1\right)·F_p^{\langle B,B\rangle }\left({\theta }_p^2\right)$$

(8)

Here, ${\theta }_p^i$ represents the time interval of a single data packet. By induction, $\mathsf{\Pi }\left(c_p\right)$ can be expressed as follows:

$$\mathsf{\Pi }\left(c_p\right)={\mu }_p^{c_p^1}·\prod _{i=1}^{n_p-1}{F}_p^{\langle c_p^i,c_p^{i+1}\rangle }\left({\theta }_p\right)$$

(9)

Finally, the transmission delay can be expressed as follows: The transmission packet loss rate ${\mu }_p^t$ can be expressed as follows:

$${\mu }_p^t={\displaystyle \frac{1}{n_p}}\sum _{\forall c_p}\mathcal{L}\left(c_p\right)·\mathsf{\Pi }\left(c_p\right)={\displaystyle \frac{1}{n_p}}\sum _{\forall c_p}\left\{\mathcal{L}\left(c_p\right)·\prod _{i=1}^{n_p-1}{F}_p^{\langle c_p^i,c_p^{i+1}\rangle }\left({\theta }_p\right)\right\}$$

(10)

Next, to derive the out-of-time packet loss, based on the statistical results from [23,24], the network end-to-end packet delay can be modeled using an exponential distribution:

$${\mu }_p^o=P[E\left(d_p^e\right)>T]\approx exp\left[-{\displaystyle \frac{T}{E\left(d_p^e\right)}}\right]$$

(11)

Here, $E\left(d_p^e\right)$ represents the expected end-to-end delay along the path, and $d_p^e$ can be expressed as the sum of propagation delay $d_p^{po}$ and transmission delay $d_p^{tr}$, i.e.,

$$d_p^e=d_p^{tr}+d_p^{po}$$

(12)

In cases where the RTT of the path is asymmetric, the propagation delay $d_p^{po}$ can be approximated as ${\displaystyle \frac{{\mathrm{RTT}}_p}{2}}$. The transmission delay can be represented as the ratio of the number of bytes to be sent $L_p$ to the bandwidth $B_p$:

$$d_p^{tr}={\displaystyle \frac{L_p}{B_p}}$$

(13)

Finally, the out-of-time packet loss rate for path p can be expressed as follows:

$${\mu }_p^o\approx exp\left[-{\displaystyle \frac{T}{E\left(d_p^{tr}\right)+E\left(d_p^{po}\right)}}\right]$$

(14)

Therefore, we can calculate the actual packet loss rate ${\mu }_p$ for all paths, and from that, the actual packet loss rate P for the entire transmission process can be calculated. As long as FEC redundancy $\alpha$ satisfies $P\le \alpha$, an appropriate FEC redundancy can be obtained.

3.4. FEC Recovery Priority Scheduling (FEC-RPS) Algorithm

Using FEC encoding can effectively resist packet loss during transmission. However, when the delay jitter is too large, a severe OFO-packet issue may occur. How to reasonably schedule the encoded packets is also a crucial consideration. To mitigate problems such as the delay caused by the disorder of retransmitted packets, we propose a scheduling algorithm that ensures prioritized FEC recovery, called FEC Recovery Prioritized Scheduling (FEC-RPS).

Simply put, the FEC-RPS algorithm considers the FEC recovery characteristics, ensuring that FEC packets are decoded at the receiver in the shortest possible time to reduce the waiting time on the receiving end.

To reduce OFO-packets, the FEC-RPS algorithm is designed to prioritize the subflow with the shortest RTT for transmitting packets. Based on the RTT, FEC-RPS calculates the expected arrival time of packets on each subflow. This is a common method in other scheduling algorithms. In addition, when calculating the expected arrival time, it also takes into account network jitter. Instead of predicting the arrival time of every subflow using a precise value, which is impractical due to the real-time changing nature of networks, the expected arrival time is calculated as an interval T. This interval’s minimum value should be the smallest RTT among all subflows, and the maximum value considers network jitter, representing the largest expected RTT at the receiver.

For subflows i and j, their respective time intervals are $T_i$ and $T_j$. When the upper bound of $T_i$ is less than the lower bound of $T_j$, that is, $T_i\cap T_j=⌀$, the subflow with the shortest delay should be i. However, when $T_i$ and $T_j$ overlap, that is, $T_i\cap T_j\ne ⌀$, it becomes more challenging to select an appropriate subflow for transmission.

When the time intervals of two subflows overlap, to ensure that the FEC blocks can meet decoding conditions in the shortest time, FEC-RPS will transmit data in a round-robin manner among the overlapping subflows. Overlapping arrival times imply that multiple subflows are expected to reach the receiver at approximately the same time. At this point, the scheduling method switches to round-robin scheduling until the time intervals of the two subflows no longer overlap. This method not only leverages the high throughput of multiple paths but also ensures that the packets from each subflow arrive at the receiver at similar times, reducing waiting times at the receiving end. By prioritizing FEC decoding, data integrity is ensured.

To ensure that the encoded packets arrive at the receiver at approximately the same time, the most important step is to estimate the arrival time of each packet and then schedule them on the path with the shortest arrival time. Define $T_p$ as the arrival time on path p, which mainly consists of three components: the transmission delay $d_p^{tr}$, propagation delay $d_p^{po}$, and queuing delay $d_p^{qu}$. The formula can be expressed as follows:

$$T_p=d_p^{tr}+d_p^{po}+d_p^{qu}$$

(15)

Previously, we have provided the formulas for transmission delay $d_p^{tr}$ and propagation delay $d_p^{po}$. The queuing delay $d_p^{qu}$ is the cumulative delay of the first $i-1$ packets, which can be expressed as follows:

$$d_p^{qu}=\sum _{i=1}^{k_q}\left({\displaystyle \frac{S·i}{B_p^i}}+{\displaystyle \frac{{\mathrm{RTT}}_p}{2}}\right)$$

(16)

$k_q$ represents the total number of packets still waiting in the transmission buffer, and S represents the size of each packet. In real-world networks, network jitter often exists. For jitter evaluation on a subpath, we can use the median instantaneous RTT, denoted as $\Delta RT{T}_p$, to determine it. The arrival time at the receiver $T_{\mathrm{arr}}$ can then be expressed as an interval:

$$T_{\mathrm{arr}}=\left[T_p,T_p+\Delta RT{T}_p\right]$$

(17)

With the above theoretical guidance, our FEC-RPS scheduling algorithm operates as follows Algorithm 1.

Algorithm 1 FEC-RPS scheduling algorithm.

Input: Subflow set S; Output: Selected subflows set;  1: for each subflow j in S do  2:       Calculate arrival time interval;  3:       $T_j=[T_p^i,T_p^i+\Delta RT{T}_p]$;  4: end for  5: Find subflows with minimal arrival time $T_{\min }$;  6: $min\_subflows\leftarrow$ subflows with $T_{\min }$;  7: $select\_paths\leftarrow select\_paths+min\_subflows$;  8: for each subflow j in S and $j\ne min\_subflows$ do  9:       if $T_j\cap T_{\min }\ne \varnothing$ then 10:            $select\_paths\leftarrow select\_paths+subflo{w}_j$; 11:     end if 12: end for 13: return $select\_paths$;

The algorithm starts by scheduling packets on the subflow with the earliest $T_{\mathrm{arr}}$. If $T_i\cap T_j=⌀$, the subflow continues to transmit packets during the scheduling process, and its queuing delay $d_p^{qu}$ keeps increasing. When $T_{\mathrm{arr}}$ overlaps with other subflows, the scheduling method switches to round-robin scheduling. At this point, packets are transmitted alternately across the subflows.

Since all transmitted packets undergo FEC encoding, even if packets from a certain subflow experience delays or loss, other subflows can still ensure that the same group of packets arrive at the receiver at approximately the same time.

When the receiver meets the FEC decoding conditions, it can directly decode and recover lost or delayed packets without waiting for retransmission. The detailed scheduling process is shown in Figure 6.

3.5. System Overview

In this section, we introduce the complete process of LDMP-FEC:

The comprehensive process of LDMP-FEC is presented in Algorithm 2. At the beginning of the program, the K original data packets are retrieved from the buffer that has already been segmented into data packets. These data packets are then encoded using an adaptive redundancy algorithm, which determines the appropriate level of redundancy, resulting in the generation of N data packets. All generated packets are subsequently placed into the sending buffer.

Algorithm 2 LDMP-FEC algorithm.

Input: videoDataBuffer, subflow set S Output: Selected subflow to send packets  1: while True do  2:       Update $RT{T}_p$, $B_p$ for each subflow through ACKs;  3:       Retrieve K packets from videoDataBuffer;  4:       Calculate the redundancy $\alpha$ based on Equations (1)–(14);  5:       $fec\_datas\leftarrow FEC(N,K)$ where $N=(1+\alpha )·K$;  6:       $select\_paths\_set\leftarrow FEC\_RPS$ (Algorithm 1);  7:       for each packet in $fec\_datas$ do  8:             if $\mathrm{len}(select\_paths\_set)$== 1 then  9:                $choose\_path\leftarrow select\_paths\_set\left[0\right]$; 10:             else 11:                $choose\_path\leftarrow$ Select a subflow from $select\_paths\_set$              that has not been chosen before; 12:             end if 13:             Send_data($choose\_path$, packet); 14:     end for 15: end while

The scheduling algorithm calculates the estimated arrival delay intervals for each subflow corresponding to every packet awaiting transmission. It then assesses whether the selected subflow’s arrival delay interval overlaps with those of other subflows. If there is no overlap, the subflow is designated as the shortest RTT subflow for sending the packet. However, if an overlap is detected, the scheduling algorithm switches to a round-robin scheduling approach, distributing packets sequentially across the available subflows.

Throughout the transmission process, the algorithm continuously monitors the number of bytes sent on each subflow and receives channel feedback from the receiver. This feedback includes metrics such as RTT, packet loss rate, and available bandwidth. These metrics are used to prepare and optimize the transmission of subsequent datasets, ensuring efficient and reliable video streaming.

4. Experiments and Results

4.1. Experimental Setup

To simulate the process of multipath transmission in heterogeneous networks, we constructed the topology illustrated in Figure 7, deploying the sender and receiver on two separate Ubuntu 18.04 servers. The experiment simulated two transmission paths: one using a 4G network and the other a Wi-Fi network. The receiver was equipped with two 100 Mbps wired network interfaces to simultaneously receive data from multiple paths. The network environment was emulated using the HoloWan network emulator. For video transmission, we utilized the FFmpeg tool to encode the same video sequence, “Highway”, using H.264 encoding, thereby simulating video streaming. For comparative analysis, we set up three scenarios:

• MinRtt-FEC: Combines FEC for data protection with the MinRtt scheduling algorithm, with FEC redundancy ($\alpha$) set to 10%.
• MinRtt: Employs the MinRtt scheduling algorithm without FEC encoding.
• Round-robin-FEC: Combines FEC for data protection with the round-robin scheduling algorithm, with FEC redundancy ($\alpha$) set to 10%.
• Round-robin: Employs the round-robin scheduling algorithm without FEC encoding.
• FEC-RPS: Utilizes the FEC-RPS algorithm independently to evaluate its performance without FEC data protection.

Additionally, to explore the performance of the FEC-RPS algorithm when FEC is not used for data protection, we included FEC-RPS in the comparative experiments. In all subsequent experiments, the server uses FFmpeg to provide a video stream with a bitrate of 8 Mbps. The bandwidth of each subflow is set to 5 Mbps, indicating that neither subflow alone can meet the transmission requirements of the video stream. Aggregating both subflows is necessary to provide sufficient bandwidth for data transmission, highlighting the necessity of multipath utilization.

4.2. The Operation Status of Each Subflow

To illustrate the details of the LDMP-FEC algorithm, we first validated its actual operational performance. We configured $Path\#1$ with an RTT of 20 ms to simulate a relatively stable channel in a real-world environment. $Path\#2$ was used to simulate a less stable network environment with packet loss and latency, setting its RTT to 100 ms, accompanied by network jitter with a maximum value of 10%. The instantaneous throughput of each subflow was recorded.

As shown in the Figure 8, at the beginning, the RTT difference between $Path\#1$ and $Path\#2$ was significant, and LDMP-FEC primarily scheduled data on the lower-latency $Path\#1$. At the 30 ms mark, network jitter occurred, and LDMP-FEC calculated that the arrival times for both paths at the receiver were similar. At this point, the algorithm switched to a round-robin approach, scheduling data packets equally across both paths. Data packets were transmitted evenly over both subflows.

After 20 s of jitter, the network conditions began to stabilize, and LDMP-FEC switched back to the MinRTT scheduling method, prioritizing $Path\#1$ for data scheduling. This describes the operational behavior of LDMP-FEC in a real-world scenario.

4.3. Out-of-Order Queue Size

In this experiment, we explore the effectiveness of LDMP-FEC in reducing OFO-FEC. For the RTT settings, $Path\#1$’s RTT is configured to 20 ms. $Path\#2$ is used to simulate a poorer network environment, with a packet loss rate of 2%, and its RTT is varied from 20 ms to 160 ms to fully demonstrate LDMP-FEC’s performance under heterogeneous networks with different latency gaps.

As shown in Figure 9, due to packet loss in the channel, the FEC-RPS, MinRTT, and round-robin algorithms all perform suboptimally. However, the OFO-packet size caused by FEC-RPS is generally smaller than that of the round-robin and MinRTT algorithms. This is because the round-robin algorithm evenly distributes packets across both paths, resulting in significantly more packets being allocated to $Path\#2$ compared to FEC-RPS. On the other hand, the MinRTT algorithm continuously transmits data through the path with the shortest RTT, even if the subflow is congested, continuously scheduling data to that subflow. In contrast, the FEC-RPS algorithm prioritizes scheduling data through queues with lower latency and available capacity, significantly reducing the packet loss rate and the size of OFO-packets.

When FEC is used for data protection, it can be observed that FEC significantly reduces the number of OFO-packets. This is because the receiver can directly decode the packets to recover lost or delayed data without indefinitely waiting for delayed packets to arrive. Regarding redundancy, LDMP-FEC uses an adaptive redundancy algorithm, while round-robin-FEC and MinRTT use fixed redundancy of 10%. Although the redundancy levels of the latter two are sufficient to counter channel packet loss, LDMP-FEC benefits from the scheduling approach of FEC-RPS, which prioritizes high-quality channels for transmission. This allows the receiver to quickly decode the received data and recover lost or delayed packets, significantly reducing the size of OFO-packets. Therefore, compared to the other schemes, the LDMP-FEC approach demonstrates remarkable effectiveness in reducing OFO-packets size.

4.4. End-to-End Delay

End-to-end delay represents the total time consumed for a video frame to travel from the sender to the receiver. Higher end-to-end delay indicates longer transmission times for the video frame, causing a greater delay in displaying the frame at the receiver, which results in noticeable visual latency. Therefore, end-to-end delay should be minimized as much as possible within the allowable range.

To explore the performance of the LDMP-FEC scheme in terms of end-to-end delay, we configured the RTT of $Path\#1$ to 20 ms and the RTT of $Path\#2$ to 100 ms. The bandwidth settings were the same as in the previous experiment. The end-to-end delay of video frames was recorded, and Figure 9 shows the CDF (Cumulative Distribution Function) of end-to-end delay for different schemes.

As observed in Figure 10, 90% of the video frames in the LDMP-FEC algorithm achieve an end-to-end delay within 150 ms, while 90% of the end-to-end delays in the round-robin-FEC and MinRTT-FEC algorithms exceed 150 ms. For schemes that do not utilize FEC, such as round-robin, MinRTT, and FEC-RPS, their inability to handle packet loss leads to prolonged waiting for undelivered packets at the receiver. Moreover, since decoding at the receiver occurs sequentially, an undecoded frame blocks the decoding of subsequent frames, even if the subsequent frame has already been fully received. This results in 90% of the video frames experiencing an end-to-end delay exceeding 200 ms.

The turning point at 210 ms in the figure is due to prolonged blocking and waiting for certain packets, which triggers the receiver’s timeout packet loss mechanism. This mechanism discards packets that have been delayed for too long and cannot be decoded, thereby reducing the pressure on the receiver’s buffer.

In summary, the results demonstrate that the LDMP-FEC scheme significantly reduces the end-to-end delay of video transmissions in heterogeneous networks.

4.5. Playable Frame Rate

An important metric for evaluating video transmission quality is the playable frame rate (PFR), which represents the number of video frames that can be decoded and played by the receiver. The more frames the receiver can decode, the smoother the video playback and the better the user experience. However, due to the characteristics of H.264 encoding, when the decoding of a reference frame fails, the entire Group of Pictures (GoP) fails to decode, resulting in a large number of unplayable video frames and severely impacting the PFR.

To investigate the performance of LDMP-FEC under different packet loss scenarios, we configured the RTT of $Path\#1$ to 20 ms and $Path\#2$ to 100 ms, while varying the packet loss rate of $Path\#2$ from 0.02 to 0.1. The results are shown in Figure 11.

As observed in Figure 11, without FEC protection, the PFR of the round-robin, MinRTT, and FEC-RPS algorithms drops significantly. When the packet loss rate reaches 0.1, the PFR is only 50%, making the video almost unwatchable. This is due to extensive reference frame losses at the receiver, causing decoding failures. Without any means of data recovery, video decoding cannot proceed, and a large number of frames are discarded.

In contrast, the round-robin-FEC and MinRTT-FEC algorithms with FEC protection maintain a PFR of over 80%, while LDMP-FEC demonstrates even better performance, consistently achieving a PFR of over 90%. This indicates that LDMP-FEC outperforms round-robin in ensuring effective FEC decoding.

4.6. Peak Signal-to-Noise Ratio

PSNR (Peak Signal-to-Noise Ratio) is an important metric for evaluating video quality. It represents the ratio of the maximum possible signal power to the power of corrupting noise. A higher PSNR indicates better video quality and a better user experience. Furthermore, PSNR can also reflect the degree to which video communication is affected by channel packet loss.

In this experiment, $Path\#1$’s RTT was set to 20 ms, $Path\#2$’s RTT was set to 100 ms, and the packet loss rate was 2%. To assess overall performance, we compared the LDMP-FEC algorithm, round-robin-FEC algorithm, and MinRTT-FEC algorithm. Figure 11 shows the instantaneous PSNR for frames 1000 to 1350 across the three algorithms.

As illustrated in Figure 12, the overall PSNR of LDMP-FEC consistently remains above 35, with minimal fluctuations and relatively stable performance. In contrast, the round-robin-FEC algorithm with a redundancy level of 10% exhibits more significant fluctuations. This is because round-robin scheduling causes slower FEC decoding, which leads to frame timeout and triggers the timeout packet loss mechanism. The MinRTT-FEC algorithm performs better than the round-robin-FEC algorithm as it prioritizes paths with lower latencies for transmission. However, when the shortest RTT path becomes congested, it continues to transmit data along that path, resulting in a large amount of data waiting to be sent, which negatively impacts video transmission quality. Consequently, its overall PSNR is inferior to that of the LDMP-FEC algorithm.

In summary, in terms of PSNR performance, the LDMP-FEC algorithm demonstrates superior results.

5. Conclusions

When transmitting video over heterogeneous networks using multiple paths, significant disparities between the networks can easily result in a large number of out-of-order packets (OFO-packets) at the receiver, leading to excessive video transmission delays. Additionally, due to channel packet loss and timeout packet loss at the receiver, substantial video data loss causes numerous instances of screen tearing and stuttering at the receiver.

To address these issues, we propose the LDMP-FEC scheme, which provides low-latency, stable, reliable, and high-quality video transmission for real-time video applications and other latency-sensitive applications. To mitigate channel packet loss, LDMP-FEC integrates FEC encoding protection and offers an adaptive FEC redundancy scheme based on channel variations. For the scheduling and transmission of the encoded data, LDMP-FEC sends packets based on the estimated arrival time intervals of each packet rather than striving for precise estimation. When the arrival delays of different subflows are similar, LDMP-FEC switches the scheduling scheme to round-robin, ensuring that data packets arrive simultaneously at the receiver to enable rapid FEC decoding.

Extensive experiments demonstrate that the proposed LDMP-FEC scheme effectively meets the objectives of real-time video transmission over heterogeneous networks. However, there are still limitations. For instance, the current work does not consider highly dynamic network environments such as those in drone flight control or vehicular video communication scenarios. Future work will focus on these areas, incorporating reinforcement learning into the scheme and exploring its deployment in practical, highly dynamic application scenarios.