# Time-Sensitive Network Simulation for In-Vehicle Ethernet Using SARSA Algorithm

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. TSN Data Queue Forwarding Latency Modeling

#### 2.1. Analysis of Mathematical Models

#### 2.1.1. Mathematical Modeling of FIFO Delay

#### 2.1.2. Mathematical Model of CBS Delay

_{1}) and a Class A frame (A

_{1}) are simultaneously added to the queue [29]. Therefore, the CBS delay can be calculated as follows:

#### 2.1.3. Mathematical Modeling of TAS Delay

#### 2.2. Delay Modeling

#### 2.2.1. Class A Data Delay Modeling

#### 2.2.2. Class B Data Delay Modeling

#### 2.2.3. Modeling of Latency for BE Class Data

## 3. Forwarding Strategy Model Optimization

#### 3.1. SARSA Reinforcement Learning Theory

- (1)
- Initialize the Q-value table.
- (2)
- Repeat the following steps: In states, use the greedy algorithm to select action a based on the Q-value table. Obtain the reward r and transition to the next states. Update the Q-value table using the iterative equation.
- (3)
- Output the optimal policy.

#### 3.2. Q-Table Determination

#### 3.3. Calculation of Time Delays

## 4. Simulation and Result Analysis

## 5. Test Validation

## 6. Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Conflicts of Interest

## References

- Hu, B.; Qin, G.; Liu, Y.; He, Y.; Wu, X. Next-generation automotive networks: Status and development of in-vehicle Ethernet technology. Comput. Eng. Appl.
**2016**, 52, 29–36. [Google Scholar] - Li, W.; Zhang, L.; Wang, Y. In-vehicle Ethernet technology and standardization. Telecommun. Netw. Technol.
**2016**, 6, 1–5. [Google Scholar] - Farkas, J.; Bello, L.L.; Gunther, C. Time-Sensitive Networking Standards. IEEE Commun. Stand. Mag.
**2018**, 2, 20–21. [Google Scholar] [CrossRef] - IEEE Std 802. 1QatTM-2010; IEEE Standard for Local and Metropolitan Area Networks Virtual Bridged Local Area Networks Amendment 14: Stream Reservation Proticol (SRP). IEEE: Piscataway, NJ, USA, 2010.
- IEEE Std 802. 1ASTM-2011; IEEE Standard for Local and Metropolitan Area Networks—Timing and Synchronization for Time-Sensitive Applications in Bridged Local Area Networks. IEEE: Piscataway, NJ, USA, 2011.
- IEEE 802. 1Qav-2009; Standard for Local and Metropolitan Area Networks-Virtual Bridged Local Area Networks Amendment 12: Forwarding and Queuing Enhancements for Time-Sensitive Streams. IEEE: Piscataway, NJ, USA, 2009.
- IEEE 802. 1Qbv-2015; Standard for Local and Metropolitan Area Networks-bridges and Bridged Networks-Amendment 25: Enhancements for Scheduled Traffic. IEEE: Piscataway, NJ, USA, 2016.
- IEEE 802. 1Qbu-2016; Standard for Local and Metropolitan Area Networks-Bridges and Bridged Networks-Amendment 26: Frame Preemption. IEEE: Piscataway, NJ, USA, 2016.
- IEEE P802.1Qch/D2.0.; Draft Standard for Local and Metropolitan Area Networks-Bridges and Bridged Networks-Amendment: Cyclic Queuing and Forwarding. IEEE: Piscataway, NJ, USA, 2016.
- IEEE 1722-2016; IEEE Computer Society: Transport Protocol for Time-Sensitive Applications in Bridged Local Area Networks. IEEE: New York, NY, USA, 2016.
- IEEE 802. 3br-2016; Standard for Ethernet Amendment 5: Specification and Management Parameters for Interspersing Express Traffic. IEEE: Piscataway, NJ, USA, 2016.
- Kirsten, M. Automotive Ethernet; Mechanical Industry Press: Beijing, China, 2019. [Google Scholar]
- Hank, P.; Mü, S. Automotive Ethernet: In-vehicle Networking and Smart Mobility. In Proceedings of the Design, Automation & Test in Europe Conference & Exhibition; IEEE: Manhattan, NY, USA, 2013. [Google Scholar]
- Fu, S.; Zhang, H.; Chen, J. Time Sensitive Networking Technology Overview and Performance Analysis. ZTE Commun.
**2018**, 16, 57–64. [Google Scholar] - Bruckner, D.; Stănică, M.P.; Blair, R.; Schriegel, S.; Kehrer, S.; Seewald, M.; Sauter, T. An Introduction to OPC UA TSN for Industrial Communication Systems. Proc. IEEE
**2019**, 99, 1121–1131. [Google Scholar] [CrossRef] - Bello, L.L.; Steiner, W. A Perspective on IEEE Time-sensitive Networking for Industrial Communication and Automation Systems. Proc. IEEE
**2019**, 107, 1094–1120. [Google Scholar] [CrossRef] - Cao, Z. Research on Data Delay Modeling for In-Vehicle TSN Reservation Class Data; Jiangsu University: Zhenjiang, China, 2022. [Google Scholar]
- Imtiaz, J.; Jasperneite, J.; Han, L. A Performance Study of Ethernet Audio Video Bridging (AVB) for Industrial Real-time Communication. In Proceedings of the IEEE Conference on Emerging Technologies & Factory Automation; IEEE: Manhattan, NY, USA, 1946. [Google Scholar]
- Maxim, D.; Song, Y.Q. Delay Analysis of AVB traffic in Time-sensitive networks (TSN). In Proceedings of the RTNS 2017—International Conference on Real-Time Networks and Systems, Grenoble, France, 4–6 October 2017. [Google Scholar]
- Thangamuthu, S.; Concer, N.; Cuijpers, P.J.; Lukkien, J.J. Analysis of Ethernet-Switch Traffic Shapers for In-Vehicle Networking Applications. In Proceedings of the Design, Automation & Test in Europe Conference & Exhibition; IEEE: Manhattan, NY, USA, 2015. [Google Scholar]
- Thiele, D.; Ernst, R.; Diemer, J. Formal Worst-case Timing Analysis of Ethernet TSN’s Time-aware and Peristaltic shapers. In Proceedings of the 2015 IEEE Vehicular Networking Conference (VNC), Kyoto, Japan, 16–18 December 2016; IEEE: Manhattan, NY, USA, 2016. [Google Scholar]
- Zhao, L.; Pop, P.; Zheng, Z.; Li, Q. Timing Analysis of AVB Traffic in TSN Networks Using Network Calculus. In Proceedings of the 2018 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), Porto, Portugal, 11–13 April 2018. [Google Scholar]
- Mohammadpour, E.; Stai, E.; Mohiuddin, M.; Le Boudec, J.Y. End-to-End Latency and Backlog Bounds in Time-sensitive Networking with Credit Based Shapers and Asynchronous Traffic Shaping. In Proceedings of the 2018 30th International Teletraffic Congress (ITC 30), Vienna, Austria, 3–7 September 2018; IEEE: Manhattan, NY, USA, 2018. [Google Scholar]
- Wang, Y.; Huang, F.; Li, Y.; Pan, B.; Wu, Y. Hierarchical Scheduling and Real-time Analysis for vehicular Time-sensitive Network. In Proceedings of the 2019 12th International Symposium on Computational Intelligence and Design (ISCID), Hangzhou, China, 14–15 December 2019; IEEE: Manhattan, NY, USA, 2019. [Google Scholar]
- Xu, H.-L.; Gao, H.; Wang, P.; Zhang, H.; Cao, W. A hybrid probabilistic model for frame preemption delay in time-sensitive networks. J. North China Electr. Power Univ. (Nat. Sci. Ed.)
**2023**, 50, 100–108. [Google Scholar] - Hellmanns, D.; Glavackij, A.; Falk, J.; Hummen, R.; Kehrer, S.; Dürr, F. Scaling TSN Scheduling for Factory Automation Networks. In Proceedings of the 16th IEEE International Conference on Factory Communication Systems, Porto, Portugal, 27–29 April 2020; IEEE: Manhattan, NY, USA, 2020. [Google Scholar]
- Craciunas, S.S.; Oliver, R.S.; Chmelík, M.; Steiner, W. Scheduling Real-Time Communication in IEEE 802.1Qbv Time Sensitive Networks. In Proceedings of the International Conference on Real-Time Networks & Systems; ACM: New York, NY, USA, 2016. [Google Scholar]
- Ciarletta, L.; Fejoz, L.; Guenard, A.; Navet, N. Development of a safe CPS component: The hybrid parachute, a remote termination add-on improving safety of UAS. In Proceedings of the Embedded Real-Time Software and Systems (ERTS 2016), Toulouse, France, 27–29 January 2016. [Google Scholar]
- Navet, N.; Fejoz, L.; Havet, L.; Sebastian, A. Lean Model-Driven Development through Model-Interpretation: The CPAL design flow. In Proceedings of the Embedded Real-Time Software and Systems (ERTS 2016), Toulouse, France, 27–29 January 2016. [Google Scholar]
- Altmeyer, S.; Navet, N.; Fejoz, L. Using CPAL to model and validate the timing behaviour of embedded systems. In Proceedings of the 6th International Workshop on Analysis Tools and Methodologies for Embedded and Real-time Systems (WATERS), Lund, Sweden, 7 July 2015. [Google Scholar]
- Seyler, J.; Navet, N.; Fejoz, L. Insights on the Configuration and Performances of SOME/IP Service Discovery. In Proceedings of the SAE World Congress, Detroit, MI, USA, 21–23 April 2015. [Google Scholar]
- Peng, Q. Research on TSN Scheduling Algorithm for Industrial Internet of Things; Zhejiang University of Technology: Hangzhou, China, 2019. [Google Scholar]
- Wang, Y. Research on Online Scheduling Algorithm for Real-Time Streams in Time-Sensitive Networks; Dalian University of Technology: Dalian, China, 2021. [Google Scholar]
- Zhou, X.; Qiu, Z.; Zhu, J.; Zhang, H.; Zhao, C. A GCL adaptive adjustment algorithm for time-sensitive networks. J. Beijing Univ. Posts Telecommun.
**2022**, 45, 26–32. [Google Scholar] [CrossRef] - Qin, Z.; Li, N.; Liu, X.; Liu, X.; Tong, Q. A review of model-free reinforcement learning. Computer Science
**2021**, 48, 180–187. [Google Scholar] - Tesauro, G. Temporal Difference Learning and TD-Gammon. Commun. ACM
**1995**, 38, 58–68. [Google Scholar] [CrossRef] - Hong, J. Mathematical Modeling of Delay Characteristics of AVB Data Transmission over Vehicular Ethernet; Jiangsu University: Zhenjiang, China, 2020. [Google Scholar]

**Figure 5.**Comparison of AVB and TSN network delay averages. (

**a**) Delay averages of Class A data frames in AVB network. (

**b**) Delay averages of Class A data frames in TSN network.

**Figure 6.**Graph of average delay statistics for traffic flow. (

**a**) Class A data latency. (

**b**) Class B data latency.

**Figure 7.**Delay statistics for Class A frames with different BE loads. (

**a**) BE is loaded with 15 Mbit/s. (

**b**) BE is loaded with 20 Mbit/s. (

**c**) BE is loaded with 25 Mbit/s.

**Figure 10.**Comparison of theoretical and measured delay for Class A data frames. (

**a**) BE is loaded with 10 Mbit/s. (

**b**) BE is loaded with 15 Mbit/s. (

**c**) BE is loaded with 20 Mbit/s. (

**d**) BE is loaded with 25 Mbit/s.

**Figure 11.**Comparison of theoretical and measured latency for Class B data. (

**a**) BE is loaded with 10 Mbit/s. (

**b**) BE is loaded with 15 Mbit/s. (

**c**) BE is loaded with 20 Mbit/s. (

**d**) BE is loaded with 25 Mbit/s.

s | AP | AA | CA | AB | CB | ABE | a |
---|---|---|---|---|---|---|---|

1 | ≠0 | random | random | random | random | random | 2 |

2 | =0 | ≠0 | ≥0 | ≠0 | ≥0 | ≠0 | 3 |

3 | =0 | ≠0 | ≥0 | random | <0 | ≠0 | 3 |

6 | =0 | ≠0 | ≥0 | ≠0 | ≥0 | =0 | 3 |

7 | =0 | ≠0 | ≥0 | =0 | ≥0 | =0 | 3 |

8 | =0 | ≠0 | <0 | ≠0 | ≥0 | ≠0 | 4 |

25 | =0 | =0 | <0 | ≠0 | ≥0 | =0 | 4 |

26 | =0 | =0 | <0 | random | <0 | =0 | 1 |

27 | =0 | =0 | <0 | =0 | ≥0 | =0 | 1 |

TP | TA | TB | TBE | ||||

28 | 0 | 0 | 0 | 1 | 6 | ||

29 | 0 | 0 | 1 | 0 | 6 | ||

30 | 0 | 1 | 0 | 0 | 6 | ||

31 | 1 | 0 | 0 | 0 | 6 |

Class A (Mbps) | Class B (Mbps) | Class BE (Mbps) | Theoretical Delay (μs) | Measured Delay (μs) | Deviation (%) | |
---|---|---|---|---|---|---|

1 | 50 | 25 | 10 | 110.93 | 109.35 | −1.44 |

2 | 50 | 25 | 15 | 114.47 | 115.91 | 1.24 |

3 | 50 | 25 | 20 | 118.60 | 122.76 | 3.39 |

4 | 50 | 25 | 25 | 122.07 | 128.43 | 4.95 |

Class A (Mbps) | Class B (Mbps) | Class BE (Mbps) | Theoretical Delay (μs) | Measured Delay (μs) | Deviation (%) | |
---|---|---|---|---|---|---|

1 | 50 | 25 | 10 | 181.35 | 183.15 | 0.98 |

2 | 50 | 25 | 15 | 185.81 | 187.97 | 1.15 |

3 | 50 | 25 | 20 | 189.94 | 193.61 | 1.90 |

4 | 50 | 25 | 25 | 194.03 | 198.38 | 2.19 |

Class A (Mbps) | Class B (Mbps) | Class BE (Mbps) | Theoretical Delay (μs) | Measured Delay (μs) | Deviation (%) | |
---|---|---|---|---|---|---|

1 | 50 | 25 | 10 | 177.84 | 181.36 | 1.94 |

2 | 50 | 25 | 15 | 179.48 | 183.95 | 2.43 |

3 | 50 | 25 | 20 | 183.40 | 189.34 | 3.14 |

4 | 50 | 25 | 25 | 187.64 | 196.26 | 4.39 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Huang, C.; Wang, Y.; Zhang, Y.
Time-Sensitive Network Simulation for In-Vehicle Ethernet Using SARSA Algorithm. *World Electr. Veh. J.* **2024**, *15*, 21.
https://doi.org/10.3390/wevj15010021

**AMA Style**

Huang C, Wang Y, Zhang Y.
Time-Sensitive Network Simulation for In-Vehicle Ethernet Using SARSA Algorithm. *World Electric Vehicle Journal*. 2024; 15(1):21.
https://doi.org/10.3390/wevj15010021

**Chicago/Turabian Style**

Huang, Chen, Yiqi Wang, and Yuxin Zhang.
2024. "Time-Sensitive Network Simulation for In-Vehicle Ethernet Using SARSA Algorithm" *World Electric Vehicle Journal* 15, no. 1: 21.
https://doi.org/10.3390/wevj15010021