High-Precision Time Synchronization Based on Timestamp Mapping in Datacenter Networks

Abstract

In datacenter networks, it is necessary to determine whether the path is congested according to the one-way delay of packets. The accurate measurement of one-way delay depends on the high-precision time synchronization of the source device and destination device. We have proposed a time synchronization method based on timestamp mapping, combined with in-band network telemetry technology to obtain the packet send timestamp and receive timestamp on devices. The results show that the maximum synchronization error is 19 ns, and the standard deviation is 7.8 ns with a 100 ms time synchronization period and offset adjustment strategy. The proposed time synchronization method achieves outstanding synchronization accuracy and stability.

1. Introduction

In some critical application scenarios, it is essential for systems to coordinate consistently in time to ensure data consistency, integrity, and timeliness. This is particularly important in distributed systems and high-performance computing networks, where time synchronization not only influences data processes and storage but also significantly impacts network performance, reliability, and security. As the speed of datacenter network links continue to improve, various distributed applications and network scheduling demand higher-precision time synchronization [1]. High-precision time synchronization enables researchers to obtain an accurate one-way delay for network monitoring and administration. Additionally, fine-grained packet-level scheduling in a datacenter can relieve congestion and enhance network performance, which requires effective time synchronization among network devices [2].

In modern datacenters, network performance and flexibility are critical for satisfying large-scale computing, storage, and communication demands. The rapid advancement of technologies such as cloud computing, artificial intelligence, machine learning, and virtualization have led to increasingly complex network requirements in datacenters. There are a large number of network communication demands in datacenters, especially in the case of high concurrency, low latency, and high throughput. Standard protocols may bring large protocol overhead and performance bottlenecks. Custom protocols can be optimized for the specific needs of the datacenter; they also can be adjusted flexibly according to different service scenarios and technology trends to better support the changing requirements of datacenters. To facilitate the application of custom protocols, P4-based programmable switching [3] and Smart Network Interface Cards (SmartNICs) [4] are widely deployed in datacenters. Traditional switches manage network traffic through fixed functionalities; in contrast, programmable switches enable the customization and updating of network process logic to accommodate evolving requirements [3,5]. Datacenter networks typically involve extensive high-speed packet process tasks, including packet routing, filtering, load balancing, traffic monitoring, and various other operations [6]. Field-Programmable Gate Arrays (FPGAs) have gained widespread adoption in datacenter network acceleration (such as FPGA-based SmartNIC) due to their high-performance parallel computing capabilities and inherent flexibility, particularly in using custom network protocols scenarios [7].

Network devices depend on path state changes between the local and the next-hop devices to adjust the local traffic sending rate. Congestion control implemented on the FPGA requires the measurement of the one-way delay of packets between the local device and the next-hop device to determine whether the path experiences congestion [8]. Devices maintain a timestamp to record the occurrence of events. The calculation of the one-way delay is based on the packet’s send time on the source device and the receive time on the destination device [9,10]. If the timestamps of the source device and destination device are not synchronized, it is impossible to accurately calculate the one-way delay. Consequently, it is essential to synchronize the timestamps of devices before measurement.

There are several technical challenges associated with implementing time synchronization between the FPGA and the programmable switch. Firstly, due to hardware design and external environmental factors, the actual operating frequency of the FPGA may not match the configured frequency, therefore it is hard to determine exactly how long a clock cycle is. Secondly, modification to the existing data-plane clock of the switch is forbidden for the following reasons: (1) rewriting the internal data-plane clock is an expensive operation since it involves consistency checks and (2) other applications such as in-band network telemetry (INT) may be using the data-plane clock in parallel [11]. These limitations increase the complexity and challenge of time synchronization.

INT has received a lot of attention from industry and academia in recent years. Unlike traditional network measurement techniques, INT combines packet forwarding with network measurement; the internal state of the network is collected and inserted into telemetry packets by the forwards’ nodes (such as programmable switches and SmartNICs), so the INT has the advantages of real-time, fine measurement granularity and rich measurement states.

In this paper, we address the time synchronization challenge associated with an FPGA-based SmartNIC and programmable switch. We propose a time synchronization method based on timestamp mapping, which combines INT technology to obtain the packet send and receive timestamps of the FPGA and switch. The contributions of this paper are outlined as follows:

• We present an innovative time synchronization method based on timestamp mapping, elaborate the principle and steps of the method, and make a detailed description of the implementation on an FPGA and programmable switch;
• We introduce the evaluation method and build a test platform to evaluate the feasibility and effectiveness of the method. Firstly, we investigate the effect of the synchronization period on the time synchronization performance, and secondly, we optimize the synchronization method using an offset adjustment strategy. Using the 100 ms time synchronization period and the offset adjustment strategy, the maximum synchronization error of the proposed method is found to be 19 ns, and the standard deviation is 7.8 ns. These results indicate a high level of synchronization accuracy and stability.

2. Related Work

Global Positioning System (GPS) time synchronization [12,13] is a technology that synchronizes ground equipment clocks using a high-precision time source provided by a satellite’s internal atomic clock. It is widely used in various fields that require accurate time synchronization. With the wireless signal transmitted by GPS, ground receiving devices can recover time and frequency information from the signal, achieving a synchronization accuracy of less than 20 ns. This method exhibits robust resistance to interference and maintains exceptionally high accuracy. However, the system necessitates the installation of dedicated GPS timing units in each terrestrial device, which can be prohibitively expensive for widespread deployment. Additionally, the system typically requires signals from at least four satellites at the same time to achieve synchronization, limiting its effectiveness in shielded environments.

The Network Time Protocol (NTP) [14] and Precision Time Protocol (PTP) [15,16] are widely used time synchronization protocols, which transmit time information from a master clock through Ethernet packets. The synchronization principle of the NTP and PTP involves recording timestamps between the master clock and the slave clock, exchanging timestamps, and calculating the time offset between the slave clock and the master clock using four timestamps, thereby correcting the slave clock to achieve time synchronization between the two devices. However, the NTP completely relies on a software timestamp; network fluctuations and delay during transmission greatly affect the synchronization accuracy, resulting in a synchronization accuracy of only milliseconds. In contrast, the PTP uses special hardware to assist the realization of physical-layer timestamp marking, avoiding the uncertainty of network time; its synchronization accuracy can reach the sub-microsecond or even nanosecond level.

Synchronous Ethernet (SyncE) [17,18] is a time synchronization technology based on a link-layer data stream, which realizes time synchronization among nodes by embedding timestamp information into the data stream. This technology enables the recovery of the transmitter’s clock from the serial data stream through Ethernet physical layer chips, thereby achieving network clock synchronization. Since the clock extraction process occurs within the underlying hardware, it necessitates support from this hardware. Furthermore, SyncE is limited to frequency synchronization; however, it does not synchronize time information. The Datacenter Time Protocol (DTP) [19] is a time synchronization method used in datacenter networks; it leverages the Ethernet PHY layer of network devices instead of packets to implement a clock synchronization protocol. It does neutralize the uncertainty in the network but requires special hardware at every PHY in the network.

The White Rabbit protocol (WR) [20] is a new clock synchronization protocol proposed by the European Organization for Nuclear Research (CERN), which is an extension of the IEEE-1588. WR uses SyncE technology, PTP, and digital phase discrimination technology to achieve sub-nanosecond accuracy [21]. It automatically compensates for transmission delays in the fiber links, which are in the range of 10 km in length. WR has excellent performance in large-scale physics experiments and particle accelerator timing devices.

Regarding the hardware implementation of the PTP timestamp mechanism, different solutions have been proposed. There is extensive research on implementing PTP mechanisms with FPGA due to its inherent flexibility and hardware reusability. Pharos [22] is a performance monitoring tool for multi-FPGA systems, capable of measuring one-way latency among multiple network-connected FPGAs. The time synchronization protocol employed in Pharos, referred to as pPTP, is an FPGA-based implementation of the PTP protocol. It uses a synchronization interval of 10 ms for the system result in synchronizations that are accurate within one clock cycle. To eliminate the delay jitter which is caused by the network protocol stack and improve the synchronization accuracy, Yin et al. [23] proposed a method to capture the timestamps based on an FPGA between the physical layer and MAC layer and designed the IEEE 1588 message detection module and frequency compensation clock to detect the IEEE 1588 message and record the timestamps, respectively. The synchronization deviation is located within ± 40 ns. Eleftherios et al. [24] proposed a hardware architecture that combines hardware-based timestamping with a rate-adjustable clock design, and it was evaluated on an experimental setup composed of two FPGA boards communicating through a commercial off-the-shelf switch, which achieved sub-microsecond clock synchronization with a worst-case offset of 138 ns. MAC-based timestamping can be found implemented in modern commercial micro-processors. Kexin et al. [15] introduced a physical-layer PTP clock synchronization system based on a STM32 microcontroller and a DP83640 chip, achieving a synchronization time error of less than 1 microsecond.

3. Proposed Solution

We propose a method called timestamp mapping for time synchronization, which involves the relationship of timestamps on different devices. We utilize an exchange message approach similar to PTP for obtaining timestamps on different devices. The telemetry message and ACK message are used to substitute the Sync message and Delay_req message of the PTP synchronization method. This section explains the fundamental principle of the PTP and the inspiration behind the proposed method.

3.1. Precision Time Protocol Overview

The PTP is a protocol for Ethernet that enables time synchronization with sub-microsecond accuracy in a local area network (LAN); it operates based on a master and slave hierarchy. Depending on periodically exchanging messages and calculating offsets, the slave clock is able to adjust its time consistently with the master clock.

The fundamental principle of the PTP synchronization method is shown in the Figure 1. When using the PTP one-step mode for time synchronization in the network, first of all, the master sends a Sync message which includes a timestamp $t_1$ indicating the time when the Sync message is sent. The slave records the receive time of the Sync message $t_2$ and extracts the previously noted timestamp $t_1$. Subsequently, the slave then sends a Delay_Req message and records the timestamp $t_3$ when it is sent. When the master receives the Delay_Req message, it records the receive timestamp $t_4$ and sends a Delay_Resp message to the slave. The Delay_Resp message carries the timestamp $t_4$ which will be extracted by the slave. After the exchange of three messages between the master and slave, the slave has obtained four timestamps, $t_1$~$t_4$. Assuming the time offset between the master clock and slave clock is $T_{offset}$, and the transmission delay of the Sync message and the Delay_Req message are $T_{m-s}$ and $T_{s-m}$, the synchronization process can be derived in accordance with the PTP protocol.

According to the above process, we can obtain the relationships between the timestamps:

$$t_1+T_{offset}+T_{m-s}=t_2$$

(1)

$$t_3-T_{offset}+T_{s-m}=t_4$$

(2)

If $T_{m-s}$ is equal to $T_{s-m}$, the formula for calculating the time offset between the master clock and slave clock is

$$T_{offset}={\displaystyle \frac{{(t}_2-t_1)-(t_4-t_3)}{2}}$$

(3)

3.2. The Method Proposed in This Paper

The FPGA interacts with the switch using a telemetry message and ACK message to obtain a set of information (including four timestamps). There are multiple interactions in each synchronization period, resulting in multiple sets of information. A mapping relationship (related parameters are stored in the FPGA registers) between the FPGA’s timestamp and the switch’s timestamp can be calculated through multiple sets of information. With the help of a mapping relationship, obtaining a timestamp on one device can result in obtaining the timestamp on another device at the same reference time, thus achieving time synchronization.

Referring to Figure 2, the subsequent section outlines the procedure for the exchange of packets.

Firstly, the FPGA sends a telemetry message to the switch, which includes an FPGA send timestamp $t_1$ indicating the time when it will be sent.

Secondly, the switch receives the telemetry message from the FPGA, records the switch recv timestamp $t_2$, and extracts the timestamp $t_1$ contained within the message. The switch generates a response message called the ACK, and the message carries the switch send timestamp $t_3$ recorded at the time of sending the ACK message as well as the timestamps $t_1$ and $t_2$.

Finally, the FPGA receives the ACK message from the switch and records the FPGA recv timestamp $t_4$ upon receipt of the message.

The frequency of the FPGA timestamp and the switch timestamp may not be the same. Moreover, there exists an offset between the two timestamps when they are not synchronized. There is a mapping relationship between the FPGA timestamp and the switch timestamp. After knowing the timestamp of one device, we can estimate the other device’s timestamp at the same time by using this relationship and can match the two timestamps to achieve time synchronization. Suppose that the frequency ratio and offset of the switch timestamp and FPGA timestamp is denoted as $A$ and $offset$. At the Coordinated Universal Time (UTC) $t_{UTC}$, the timestamp on the FPGA is $t_F$, and the timestamp on the switch is $t_S$; their relationship can be expressed as follows:

$$t_S=A\times t_F+offset$$

(4)

$$t_F=1/A\times (t_S-offset)$$

(5)

In this formula, $A$ and $offset$ are important parameters to characterize the mapping relationship. By obtaining these two parameters, we can obtain the FPGA (or switch) timestamps at the same UTC corresponding to the switch (or FPGA) timestamps, which is called the mapped FPGA (or switch) timestamp. After obtaining the message, like in the PTP, the transmission delay can be expressed in the following formulas:

$$Dela{y}_{F-S}=t_2-A\times t_1-offset$$

(6)

$$Dela{y}_{S-F}=t_4\times A-t_3+offset$$

(7)

When $Dela{y}_{F-S}$ is equal to $Dela{y}_{S-F}$, we can conclude that

$$t_2+t_3=A\times (t_4+t_1)+2\times offset$$

(8)

However, the $Dela{y}_{F-S}$ is hardly equal to $Dela{y}_{S-F}$ in the practical network environment. A compensation value C is set to counteract the asymmetry delay part according to the actual situation, then

$$t_2+t_3+\mathrm{C}=A\times (t_4+t_1)+2\times offset$$

(9)

In the above formula, if we obtain multiple sets of timestamps, the mapping of parameters $A$ and $offset$ can be estimated using the least square method. During each time synchronization period, the two devices to be synchronized exchange packets repeatedly to obtain N sets of timestamps $t_1$, $t_2$, $t_3$, and $t_4$. We define $t_4+t_1$ as X and $t_2+t_3+C$ as Y; the least square method is used to estimate the two mapping parameters $A$ and $offset$. The estimation is deduced as

$$\widehat{A}={\displaystyle \frac{\sum _{i=0}^{N-1}{X}_i{Y}_i-N\overline{X}\overline{Y}}{\sum _{i=0}^{N-1}{X}_i^2-N{\overline{X}}^2}}$$

(10)

$$\widehat{offset}={\displaystyle \frac{\overline{Y}-\overline{X}\times A}{2}}$$

(11)

These mapping parameters are subsequently stored on the FPGA after their calculation is completed.

At the same UTC, the relationship between the mapped FPGA timestamp $M_F\left(t\right)$ on the FPGA and the real switch timestamp $t_1$ on the switch is

$$M_F\left(t_1\right)=1/\widehat{A}\times (t_1-\widehat{offset)}$$

(12)

And the relationship between the mapped switch timestamp $M_S\left(t\right)$ on the FPGA and the real FPGA timestamp $t_2$ on the FPGA is

$$M_S\left(t_2\right)=\widehat{A}\times (t_2+\widehat{offset)}$$

(13)

3.3. Problems of Clock Synchronization

The clock of the device is constructed using a crystal oscillator, which is sensitive to environmental temperature fluctuation, aging, and various other factors. These influences can result in frequency drift of the oscillator. Over time, the accumulated drift tends to increase, leading to a greater offset between the master clock and slave clock, which significantly affects synchronization accuracy. To address this issue, we employ an offset adjustment strategy. The specific procedure is as follows: during the interval between two synchronization events, the adjustment for the offset is determined by the offset value variation between the current synchronization and the previous synchronization.

4. Design and Implementation

4.1. Implementation on an FPGA

To acquire the mapped timestamps from other devices on the FPGA, we have developed a hardware architecture on the FPGA that is shown in Figure 3. The design is implemented on a XILINX Zynq UltraScale+ ZU19EG FPGA.

The hardware architecture comprises a timestamp generator, a telemetry tagging module, a transmission (TX) timestamp insertion module, a reception (RX) timestamp insertion module, a calculation module located in the host, and a storage module.

During time synchronization, packets from other pipelines on the FPGA NIC firstly pass through the telemetry tagging module, where the packets are marked as telemetry packets. Then, the packets are sent to the TX timestamp insertion module and can be added with timestamp $t_1$. After that, they are transmitted to the TX Ethernet port. The ACK packets coming from the RX Ethernet port are first added with timestamp $t_4$ in the RX timestamp insertion module; then, they are sent to the calculation module for timestamps parsing and the mapping parameters’ calculation.

The timestamp generator is used to generate the local FPGA timestamp. Its input is a 167 MHz pulse which is generated by PLL, and the output timestamp is derived from a 64-bit counter that counts the ticks since the initialization of the FPGA. The timestamp increments by one at the rising edge of each pulse, signifying that 6 nanoseconds of actual time has elapsed. The timestamp generator is connected to the RX timestamp insertion module and the TX timestamp insertion module.

The telemetry tagging module is user-configured to process data packets originating from the pipeline. This module selects data packets and tags them with telemetry information based on the configured synchronization period and the amount of packets of time synchronization in each period. A packet with added telemetry information becomes a telemetry packet. The telemetry information contains a tag field and a timestamp field, which are inserted into these selected packets. The packets are subsequently transmitted to the TX timestamp insertion module after being processed.

The TX timestamp insertion module adds the local timestamp into the designated positions in the telemetry packets according to the predetermined offset. Conversely, the RX timestamp insertion module adds the local timestamp into the ACK packets received from the switch and then sends them to the host.

The calculation module is situated on the host. It is responsible for receiving the ACK packets from the switch. It parses the timestamps ($t_1$~$t_4$) of each packet and then calculates the parameters $A$ and $offset$ of the timestamp mapping relationship. After the calculation is completed, it distributes the results to the storage module on the FPGA via the PCIe bus. The storage module, when necessary, provides the mapping parameters to other modules for one-way delay calculation. This module also incorporates an offset adjustment, which includes an internal timer that can be configured by the host. This configuration allows the host to decide how many clock cycles can adjust the offset by one.

We use Vivado 2020.2 for the work’s synthesis and implementation. The simulation tools used to run test benches are iverilog and cocotb. The maximal frequency for the FPGA circuit is 167 MHz after the place and route stage. The FPGA resource utilization and power consumption for the time synchronization modules are listed in

Table 1. FPGA resource utilization for time synchronization modules.

Modules	LUT	Registers	BRAM	CARRY8
Telemetry tagging	2217	5759	8.5	7
TX timestamp insertion	1790	1254	0	288
RX timestamp insertion	1897	1240	0	336
Timestamp generator	2012	64	0	0
Storage	185	100	0	0
Others (mux/demux et al.)	1369	4018	1	4
Total	9470	12435	9.5	635

and

Table 2. Power consumption for time synchronization modules.

Modules	Power Consumption (W)
Telemetry tagging	0.101
TX timestamp insertion	0.037
RX timestamp insertion	0.028
Timestamp generator	0.013
Storage	0.003
Others (mux/demux et al.)	0.097
Total	0.279

.

4.2. Implementation on Programmable Switch

The process of telemetry messages is conducted on the Intel Tofino 1 platform [25]. Tofino is a pipelined chip that features data plane programmability. The Tofino 1 chip operates with a global clock frequency of 1.22 GHz. There is a Global Time Counter that consists of a 48-bit nanosecond component and a 28-bit fractional component. The Global Time Counter experiences a rollover approximately every three days. It disseminates time information to hardware blocks such as the MAC and Parser every 1 ns, which can be used to record the timestamps of packets in different process stages [26].

Tofino processes numerous packets that require being forwarded to their respective destinations. Upon receiving a packet containing telemetry information, the switch records the receive timestamp of packet $t_2$ in the data plane and resolves the timestamp $t_1$ from the original packet. Subsequently, it generates an ACK packet (a small size packet) that carries timestamps $t_1$~$t_3$ and transmits it to the source FPGA.

Specifically, as shown in Figure 4, when receiving a telemetry packet, Tofino records the packet’s arrival time $t_2$ and parses the timestamp $t_1$ contained within the packet in the Ingress pipeline process. Both timestamps are stored in the metadata of the data plane. Subsequently, it triggers the mirroring function to generate a mirror packet that serves as an ACK packet, using $t_1~t_2$ as the mirror header. After the mirror packet is generated, the Tofino populates the ACK packet, exchanges the source and destination IP addresses, and transmits the ACK packet to the loopback interface to reroute the routing pipeline. In the Egress pipeline, the send time $t_3$ is then appended in ACK packet.

5. Evaluation

To evaluate the proposed time synchronization method, we use a Network Tester IXIA-XGSHS to measure the synchronization error. IXIA-XGSHS is capable of generating multiple types of packets and recording the send and receive times of each packet. It has a more accurate and stable hardware timestamp than the FPGA and switch, so we considered the hardware timestamp of the Network Tester as the reference timestamp. Under the assumption that the synchronization approach is entirely accurate, the mapped switch timestamp calculated using the mapping parameters should correspond precisely to the actual timestamp on the switch at the same reference time. Consequently, we can interpret the discrepancy between the mapped timestamp and the actual timestamp as the synchronization error.

When the Network Tester transmits messages to the FPGA and the switch simultaneously, the messages will arrive at the FPGA and the switch at the same reference time after they traverse an equal length of optical fiber; thus, we can use the receive timestamps from the FPGA and the switch to calculate the synchronization error.

The Network Tester is interfaced with the FPGA and the switch via distinct ports. It periodically exchanges messages with two devices. The messages contain timestamps that indicate the send time and receive time as they traverse the three devices. Ultimately, these messages are received and captured by the Network Tester. Upon capturing the messages, the Network Tester analyzes the timestamps for further evaluation. Specifically, the accuracy and stability of the mapping relationship are assessed. The mapped switch time is computed using the mapping parameters and the FPGA timestamp included in the message from the FPGA. This computed mapped time is then compared to the actual switch timestamp present in the message from the switch, and the result difference is regarded as the synchronization error.

The three devices are interconnected as illustrated in Figure 5. The FPGA and switch communicate with messages transmitted over an optic fiber for the purpose of time synchronization. Concurrently, the Network Tester transmits measurement packets to both the FPGA and the switch. Following being processed by either the FPGA or the switch, the measurement packets are returned to the Network Tester. The block diagram depicting the FPGA’s process for the measurement packets is presented in Figure 6. This diagram incorporates shadow components into the original time synchronization architecture. The demultiplexer (demux) module on the FPGA will categorize the received packets into two distinct types: those used for time synchronization (telemetry packets and ACK packets) and those used for evaluation purposes (measurement packets). These packets will subsequently be directed to different modules for further process. The measurement packets are sent to the forwarding module. The multiplexer (mux) module aggregates two types of packets together and sends them to the TX timestamp insertion module. The function of the forwarding module is to relay the measurement packets from the receive pipeline to the send pipeline, which entails swapping the source and destination addresses, as well as appending two parameters, A and $offset$, in the storge module to specific locations within the packets. Additionally, measurement packets are added with a local timestamp at the RX timestamp insertion module upon their arrival at the FPGA. When the switch receives the measurement packets, it records the time at which the packets enter the pipeline. The formats of these messages are shown in Figure 7. The Network Tester captures the measurement packets returned from either the FPGA or the switch and extracts the local send timestamps contained within the packets, along with the timestamps received by the FPGA or switch. Furthermore, two mapping parameters are also extracted if the packets are from the FPGA.

6. Results

6.1. Clock Drift Measurement

Due to the issue of frequency drift associated with the device’s clock, we initially conducted measurements to assess the clock drift of both the FPGA and the switch. We employed a Network Tester to measure the degree of drift between the FPGA clock and the switch clock. The results are shown in Figure 8. The initial measurement indicated a clock drift of 0. The results demonstrated a linear relationship between clock drift and measurement time, revealing a drift of 5520 ns for the FPGA and 4300 ns for the switch over a duration of 100 ms, which corresponds to a clock drift rate of 55 PPM and 43 PPM.

6.2. Time Synchronization Measurement

We employed a Network Tester to evaluate the accuracy and stability of time synchronization between the FPGA and the switch. It is assumed that the FPGA synchronizes its time with the switch over a period denoted as T, utilizing a total of 100 packets for each synchronization event. The Network Tester transmitted measurement packets to both the FPGA and the switch at intervals of 1 ms. The measurement time lasted for 1 s, resulting in a total of one thousand packets sent. The offset compensation C is set to 140 ns. Upon capturing the packets forwarded by the time synchronization devices, we analyzed the send timestamps from the Network Tester and the receive timestamps from the time synchronization devices, in addition to the mapping $offset$ and mapping $A$ in the packets from the FPGA. We calculated the discrepancy between the switch time derived from the mapping parameters and the actual switch timestamp. We conducted an analysis of the mean value, standard deviation, and the maximum values of the results obtained. We performed multiple measurements with synchronization period T from 1~100 ms; the results are shown in

Table 3. Results for various synchronization periods.

Synchronization Period (ms)	Mean (ns)	Std (ns)	Max_Error (ns)
1	3.25	7.88	21
2	3.76	7.91	21
5	5.99	8.08	24
10	9.18	8.68	31
20	15.61	10.93	45
30	21.50	14.17	57
40	25.82	17.12	69
50	36.17	20.76	81
60	37.67	22.56	93
70	40.91	27.62	106
80	47.37	33.23	123
90	56.94	37.45	135
100	68.96	39.36	143

and Figure 9 and Figure 10.

The results indicated that reducing the clock synchronization period from 100 ms to 1 ms leads to a decrease in the mean synchronization error from 69 ns to 3 ns, accompanied by a reduction in the standard deviation from 39 ns to 7.8 ns. Additionally, the maximum offset error decreased from 143 ns to 21 ns. This variation is due to the clock drift of the device’s crystal oscillator, whereby the synchronization error reaches its minimum at the moment when the mapping $offset$ and mapping $A$ are calculated and updated. Due to clock drift, the timestamp offset between two devices gradually increases while the mapping $offset$ in the FPGA register remains unchanged in synchronization updating intervals, resulting in a large synchronization error until the next period register update. By shortening the synchronization period, the interval between two adjacent register updates is shortened, resulting in a smaller synchronization error and reduced jitter.

The distribution of the synchronization error also varies with different synchronization periods, which represents different degrees of stability. From the different results for T = 1 ms and T = 100 ms shown in Figure 10, it was evident that the concentration degree of the calculation results decreased with a longer synchronization period. When the synchronization period was 100 ms, the distribution of synchronization error values was scattered, ranging from -20 ns to 140 ns. In contrast, when the synchronization period was 1 ms, the calculation results were very accurate, and the distribution of error values was concentrated, with the error values mainly distributed between −20 ns and 20 ns. Therefore, it can be concluded that the stability increases with the shortening of the synchronization period.

6.3. Offset Adjustment

In Figure 11a, we observed the change in the mapping parameter $offset$ on the FPGA at every time synchronization when the synchronization period was 1 ms and found that it showed a linear growth trend in a short time (at least 100 ms).

Figure 11. (a) Mapping parameter $offset$ variation when synchronization period T = 1 ms; (b) synchronization results for period = 100 ms with offset adjustment.

Figure 11. (a) Mapping parameter $offset$ variation when synchronization period T = 1 ms; (b) synchronization results for period = 100 ms with offset adjustment.

According to the aforementioned conclusions, if the mapping register is updated regularly without any intervention, it is necessary to synchronize time at shorter intervals to achieve higher time synchronization accuracy. However, the frequent transmission of time synchronization messages will lead to increased network overhead. To address the trade-off between network overhead or message process capability and synchronization accuracy, we propose an offset adjustment method. This method allows for the adjustment of the offset between the adjacent update intervals, thereby achieving a smaller synchronization error while extending the synchronization period. Based on the linear change in the offset in Figure 11a, we can make a prediction for the mapping parameter $offset$ so as to adjust it. The procedure for predicting the offset is as follows: when the offset register is updated at the Nth time, the change between the current update value and the value from the previous update (N-1)th is calculated. This change is then evenly distributed across the two adjacent synchronization events’ time period, and an adjustment strategy is applied to the offset register of the FPGA. Specifically, the offset is increased by one for every specified number of clock cycles. The number of clock cycles is calculated by the software and delivered to the timer of the storage module. The results depicted in Figure 11b validated this method using a time synchronization period of 100 ms, resulting in synchronization performance with a maximum error of 19 ns and a standard deviation of 7.8 ns, which represented an 80% reduction in the standard deviation of the synchronization error with respect to the system without an offset adjustment. Furthermore, the maximum synchronization error was reduced by 86%.

In the above evaluation experiment, we achieved an accuracy of 19 ns with a synchronization period of only 100 ms.

Table 4. Performance comparison against the state-of-the-art.

Approach	Platform	Synchronization Period	Accuracy (ns)
Yuan et al. [15]	STM32 MCU	/	1 μs
Kong et al. [27]	FPGA	/	200 ns
Eleftherios et al. [24]	FPGA	0.5 ms	138 ns
Yin et al. [23]	FPGA	/	40 ns
Pharos [22]	FPGA	10 ms	40 ns
Proposed Approach	FPGA	100 ms	19 ns

gives a comparative analysis of the existing approaches and our proposed method. Comparing the performance of the method proposed in this paper with existing studies, the proposed method achieves higher accuracy using a longer synchronization period. It proves that we have obvious advantages, which indicated that the synchronization accuracy is 2~50 times higher than that of previous studies.

7. Conclusions

To solve the problem of high-precision time synchronization between devices in a datacenter, we proposed an innovative time synchronization method based on the timestamp mapping, which combines in-band telemetry to obtain the packet send and receive timestamps of the FPGA and switch. This paper introduced the principles and steps of the method and described the implementation of the method on an FPGA and programmable switch in detail. We also introduced the evaluation method and set up a test platform to evaluate the feasibility and effectiveness of the method. In the experiment, we studied the effect of the synchronization period on time synchronization performance and optimized the synchronization method by using an offset adjustment strategy. Finally, with a 100 ms time synchronization period and the offset adjustment strategy, the maximum synchronization error of this method is 19 ns, and the standard deviation is 7.8 ns; the synchronization accuracy is 2~50 times higher than that of previous studies. The proposed method achieves good time synchronization accuracy and stability and provides a new solution for the measurement of one-way delay in datacenters.