White Rabbit Expansion Board: Design, Architecture, and Signal Integrity Simulations

Abstract

The White Rabbit protocol allows synchronization and communication via an optical link in an integrated, modular, and scalable manner. It provides a solution to those applications that have very demanding requirements in terms of synchronization. Field-programmable gate arrays are used to implement the protocol; additionally, special hardware is needed to provide the necessary clock signals used by the dual-mixer time difference for precise phase measurement. In the present work, an expansion board that allows for White Rabbit functionality is presented. The expansion board contains the oscillators required by the White Rabbit protocol, one running at 125 MHz and another at 124.922 MHZ. The architecture of this board includes two oscillator systems for tests and comparison. One is based on VCOs and another on crystal oscillators running at the desired frequencies. In addition, it incorporates a temperature sensor, from where the medium access control address is extracted, an electrically erasable programmable read-only memory, a pulse-per-second output, and a USB UART to access the White Rabbit IP core at the field-programmable gate array. Finally, to ensure the quality of the layout design and guarantee the level of synchronization desired, the results of the power and signal integrity simulations are also presented.

1. Introduction

Many applications have very stringent requirements in terms of synchronization. In telecommunications, the performance of 5G New Radio and next-generation technologies rely on phase synchronization of Radio Access Network (RAN) nodes [1,2,3]. In data centers, time synchronization is needed for data consistency, task scheduling, and resource sharing [4]. Particle physics experiments, where the trajectory of relativistic particles has to be determined with high precision, require a high level of synchronization. For instance, in neutrino telescopes [5,6,7,8,9] the synchronization requirements are at the level of the nanosecond. Similar requirements are also needed in astrophysics detectors such as Auger [10], SKA [11], or CTA [12]. Therefore, it is mandatory for these applications to develop systems that incorporate a high-level synchronization facility. The White Rabbit (WR) protocol allows for subnanosecond synchronization and deterministic data transfer via optical fibers, making it an ideal solution. However, the implementation of the WR protocol requires specific hardware. An IP core, the White Rabbit PTP core (WRPC) [13], which is available for several field-programmable gate array (FPGA) architectures (Virtex6, Artix7, Kintex7, Virtex7, Ultrascale, Ultraescale+, Arria II, and V) [14], provides the synchronization functionality. Additionally, two clocks running at frequencies of 125 MHz and 124.992 MHz are needed by the digital dual-mixer time difference (DDMTD) running at the FPGA for precise phase measurement. This work presents an expansion board that includes all the necessary hardware to operate the WR protocol, facilitating the use of this protocol in an already existing FPGA board. The WR protocol is discussed in Section 2, while the design of the expansion board is presented in Section 3. The pre-layout signal integrity analysis performed on the design and the results of the virtual prototyping analysis are presented in Section 4. Finally, some conclusions and future plans are presented in Section 5.

2. White Rabbit Protocol

WR is an open-source project developed by an international collaboration [15] and stored in the Open Hardware Repository at the European Organization for Nuclear Research (CERN), the main promoting institution. Data transmission and synchronization are provided via Ethernet [13,16], ensuring subnanosecond synchronization and deterministic data transfer by using PTP [17,18] and Sync-E [19]. It has been designed to provide precise synchronization of clocks over packet-based networks, and recently it has been adopted by the latest PTP standard [20], IEEE 1588 HA with a guarantee that the accuracy of the resulting synchronization is better than one nanosecond [21]. Its goal is to provide nanosecond-level synchronization accuracy over wide area networks, and can be used to synchronize distributed systems, such as high-energy physics experiments [22,23,24], power distribution grids [25], control or measurement systems, with applications such as short-circuit fault location [26]. The protocol is based on an end-to-end concept, meaning that a source node broadcasts its timestamp, and the receiving node uses this timestamp to synchronize its local clock. WR uses a distributed master-slave architecture, which allows for high accuracy and scalability. In a WR master-slave network, the master hierarchically controls several slaves, distributing the synchronization down the network. The topmost WR master in the hierarchy, synchronized with Global Positioning System [27,28], provides time to lower layers. Synchronization is achieved by accurately measuring the delay from master to slave. The slave clock is synchronized with the master, and the phase shift is accurately measured by the key element of WR, the DDMTD phase detector [29]. The operation of the DMDT requires two very precise clock signals of 125 MHz and 124.992 MHz. These two signals can be generated by voltage-controlled oscillators (VCOs) or by oscillators oscillating at that precise frequency. A special hardware, that needs to be available to any WR device, provides these two clock signals. The phase noise of the two clocks is directly affecting the level of synchronization obtained. Therefore, careful implementation of the DDMTD hardware is required. The basic synchronization operation of WR includes measuring the round trip time, adjusting for the hardware delays in the master and slave (${\Delta }_{txm}$, ${\Delta }_{rxm}$, ${\Delta }_{txs}$ and ${\Delta }_{rxs}$), and the asymmetry of the fiber interconnecting the slave and the master ($\alpha$). The WR model is shown in Figure 1. The asymmetry of the fiber is caused by different transmission speeds for the uplink and the downlink, which can be computed using the refractive indices of the respective wavelengths ($n_{\lambda 1}$ and $n_{\lambda 2}$). Experimentally, $\alpha$ can be determined by measuring the up-link and down-link transmission time (${\delta }_{sm}$ and ${\delta }_{ms}$) in the lab:

$$\alpha =\frac{n_{\lambda 1}}{n_{\lambda 2}}-1=\frac{{\delta }_{ms}}{{\delta }_{sm}}-1$$

(1)

A PTP package is used to measure the time of emission and reception of data between the master and the slave. The round trip time is computed using the values measured at the master ($t_1$,$t_4$) and the slave ($t_2$,$t_3$). Additional values such as the fixed hardware delays ($\Delta ={\Delta }_{txm}+{\Delta }_{txs}+{\Delta }_{rxm}+{\Delta }_{rxs}$) and the fiber asymmetry ($\alpha$) are also taken into consideration when calculating the delay from the master to the slave ($dela{y}_{ms}$). This is achieved using the following equation,

$$dela{y}_{ms}=\frac{1+\alpha }{2+\alpha }\times (dela{y}_{mm}-\Delta )+{\Delta }_{txm}+{\Delta }_{rxs}$$

(2)

Thanks to this, the clock in the slave can be adjusted and both devices can be synchronized with a precision lower than one nanosecond.

3. Architecture of the Expansion Board

The designed board includes all the WR hardware needed to successfully operate the protocol, with the exception of the FPGA. It is composed of six subsystems, which are the power supply subsystem; the one-wire temperature sensor; an electrically erasable programmable read-only memory (EEPROM); a pulse-per-second (PPS) adapter; a USB UART; and the oscillator subsystems. The architecture of the expansion board is displayed in Figure 2. Details about each subsystem follow:

• The power supply subsystem: This subsystem generates all the internal voltages needed. The expansion board is supplied at 5 V. The 5 V is cleaned from high-frequency noise by means of a ferrite bead and supplies the linear regulators where the power rail needed is generated. A 3.3 V voltage is generated, which is used for supplying a USB UART. The 3.3 V output feeds a linear and low-dropout (LDO) where the 2.5 V rail to supply the EEPROM is generated. In addition, the cleaned 5 V feeds another LDO to generate a 4.1 V rail, which is used, in turn, to supply four low-noise LDOs to generate the 3.3 V needed by the oscillator systems. The architecture of the power supply is shown in Figure 3. The low noise 3.3 V outputs are used as a reference by the oscillators, to supply the digital-to-analog converter (DAC), and to supply the secondary clock system. Each of the previous functions is achieved by means of independent rails.
• One-wire temperature sensor, the medium access controller address provider: A one-wire [30] temperature sensor is included in the board. In addition, to provide temperature measurement, the identification number of this sensor is used to generate a unique medium access controller (MAC) [31] address for the protocol. The sensor is accessed with a one-wire interface.
• Electrically erasable programmable read-only memory: An EEPROM allows for the storage of several configuration parameters of the protocol. It can be accessed via an I²C interface and can store the device’s configuration data, such as the MAC address.
• Pulse-per-second adapter: An adapter of the PPS signal generated by the WR PTP Core is included in the expansion board. The PPS signal is an input of the expansion board, which adapts the signal for a SMA connector.
• USB UART: A USB UART with a mini-USB connector has been included to provide access to the WR FPGA through the expansion board. This USB UART is particularly useful for outputting debug messages from the WRPC. The FPGA implementation of a regular UART is relatively simple and it can be connected to any modern computer through the bridge.
• Oscillator subsystems: These two subsystems generate the clock needed by the WRPC DDMTD running at the FPGA for a precise measurement of the phase. The frequencies generated are 125 MHz and 124.992 MHz.

Figure 3. The power subsystem generates the necessary voltages from the 5 V input. The 5 V input is filtered and then fed to two LDOs, which generate the 3.3 V and 4.1 V voltages. The 3.3 V voltage is then fed to another LDO to obtain the 2.5 V voltage. The 4.1 V voltage is used to derive the 3.3 V outputs for operating the oscillator system, using four low-noise LDOs. To ensure a clean power supply and prevent interference, the 3.3 V low-noise rails have been isolated.

Figure 3. The power subsystem generates the necessary voltages from the 5 V input. The 5 V input is filtered and then fed to two LDOs, which generate the 3.3 V and 4.1 V voltages. The 3.3 V voltage is then fed to another LDO to obtain the 2.5 V voltage. The 4.1 V voltage is used to derive the 3.3 V outputs for operating the oscillator system, using four low-noise LDOs. To ensure a clean power supply and prevent interference, the 3.3 V low-noise rails have been isolated.

3.1. Oscillator Subsystems

The main subsystem of the board is the clock generator. The WR protocol needs two clocks, one with a frequency of 125 MHz, and a second one with 124.992 MHz, which is the source of the offset frequency for DDMTD. These two clocks are generated by the expansion board in two different ways. The first one uses oscillators of that frequency, and the second one uses clock generators fed with oscillators of 25 MHz. The crystals that will generate the 125 MHz and 124.992 MHz frequencies are from Abracon. The noise phase for these oscillators is shown in Figure 4 and its rise and fall time are, as a maximum value, 3 ns. The second subsystem uses two crystals of 25 MHz, which supply two CDCM61002 clock generators. The phase jitter for these oscillators is 0.4 ps for an integration bandwidth from 12 kHz to 20 MHz. One creates the 125 MHz, while the other creates the 124.992 MHz clock signal. The crystal oscillators in both systems are supplied at 3.3 V. Two DAC provides the voltage control of the crystals. The output voltage of the DACs is controlled from the WRPC via an I²C bus, in order to lock the oscillators to the PLL (phase locked loop) of White Rabbit. The inclusion of the two subsystems will allow for their evaluation and comparison in real operation. The schematics of the two subsystems for the 125 MHz generation are shown in Figure 5. Prior to testing, it is expected that the subsystem using the quartz at the desired frequencies will have lower phase noise and achieve higher precision in synchronization, while the system using the 25 MHz quartz will be more cost-effective and simpler to manufacture. Future tests will determine the differences in terms of phase noise and synchronization.

3.2. Printed Circuit Board Layout

The layout of the board has been made using four layers, two for signals, placed on the top and bottom layers, one for power planes, and one for ground. The layout of the board is shown in Figure 6, while the stackup is shown in Figure 7. The bottom of the board has no components. The oscillator signals are routed in the external layer as the dielectric constant of the air is lower than any of the dielectric materials in the PCB. In these layers, the signal propagation speed is higher, and, therefore, the attenuation is lower [32]. An important characteristic to obtain precise synchronization is the phase noise of the clocks. The transmission of clock signals across a printed circuit board (PCB) degrades by a variety of factors, such as channel losses, cross-talk noise, power noise, or reflection noise [33]. The frequency-dependent losses of the transmission line have a significant impact on signal quality and timing, resulting in an increase in jitter [34]. Additionally, the stability of supply voltage, or the layout of the tracks, will affect the figure of noise. To try to reduce these effects, an isolated ground area dedicated solely to the components of the clock system has been created. To limit insertion losses for both package and board, routing length has been kept as low as possible. Separated power distribution networks (PDN) have been used for the reference clocks, the DDMTD clocks, and the DACs. The PDN has been designed to minimize its impedance, as it directly impacts noise generation.

4. Power and Signal Integrity Analysis

4.1. Previous Developments and Justification of Virtual Prototyping

A daughter board was designed to provide White Rabbit capabilities to the Pixie-Net XL board [35]. The expansion board designed included two VCOs, and achieved an overall jitter of 300 ps. A redesign of the clock circuitry was made in order to improve the jitter but it was unsuccessful.

Virtual prototyping [36,37,38,39] can be used to verify the behavior of the PCB before its production, being able to identify possible design flaws, signal integrity issues, and electrical performance problems in an early stage and introduce the necessary modifications before the manufacturing of the first prototypes avoiding expensive physical iterations [40,41,42]. The use of advanced tools to simulate the PCB layout has the advantage of reducing the design cycle since it is not necessary to have a real prototype.

In our case, the track width of the oscillator signal and the via architecture were optimized before the layout was implemented. Once the layout was completed, the PDN of the oscillator system and the signal attenuation of the clock signals were analyzed to validate the design through simulations before starting its manufacturing.

4.2. Pre-layout Signal Integrity Simulations

The most critical signals on the board in terms of signal integrity are the oscillator system signals since they determine synchronization. The clock signals are transmitted using LVDS (low-voltage differential signals). A track geometry has been designed to maintain a characteristic impedance of 100 Ω. For the non-differential signals, geometry has been designed to maintain a characteristic impedance of 50 Ω. These geometries have been simulated using PathWave Advance Design System (ADS) 2020 from Keysight [43,44]. For the differential lines, a separation of 0.25 mm has been set in external tracks. Furthermore, a simulation has been performed to determine the optimal track width to maintain the characteristic impedance for both differential and single tracks. An optimal width of 0.27 mm results from the simulations (See Figure 8). The connections between tracks of different layers through pathways can generate negative effects that affect the integrity of the signal [45]. When using a via, all layers of the electronic board are perforated. In the layers where it connects with the tracks, a copper zone is generated around the hole called pad and in those layers where there is no connection, a copper-free zone is generated that surrounds the perforation. This copper-free zone is called antipad. This structure carries implicit parasitic capacitances and inductances that generate discontinuities affecting the integrity of the signal. The model equivalent of the via is shown in Figure 9. In this model, the parasitic capacitance and the inductance are defined by the following equations:

$$C\left(pF\right)={\epsilon }_r\sqrt[]{2}\frac{D_{pad}\times l\upsilon }{D_{antipad}-Dpad}$$

(3)

$$L\left(nH\right)=5.08\times l\upsilon \left(ln\left(\frac{4\times l\upsilon }{d\upsilon }\right)+1\right)$$

(4)

where ${\epsilon }_r$ is the dielectric constant, $D_{pad}$ and $D_{antipad}$ are the diameters of the pads and antipads, $l\upsilon$ is the length of the via, and $d\upsilon$ is the diameter of the via hole, all expressed in inches. This model creates a low-pass filter, where the cutoff frequency is determined by the values of L and C. To mitigate the effects caused by vias, efforts will be made to reduce parasitic capacitance by increasing the diameter of the antipad. To minimize the impact of parasitic inductance, return vias can be used, connected to the ground planes.

A simulation has been carried out modeling the differential vias. The model created is shown in Figure 10. For this model, the diameter of the antipad has been varied, from 0.7 mm to 1.5 mm. It is observed how the smallest impedance discontinuity is obtained for an antipad diameter of 1.5 mm.

4.3. Post-layout Power and Signal Integrity Simulations

It is important to keep PDN impedance on the PCB as low as possible to minimize the simultaneous switching noise (SSN). This is critical in the case of the oscillator system, as the noise in the PDN will result in a jitter in the clock system. Any transient current in the PDN causes SSN [46]. The level of the SSN generated is directly related to the impedance of the PDN. The noise is induced by the power rail and contributes to the appearance of jitter in the signals generated by the elements supplied for that PDN. The peak-to-peak voltage of the SSN is given by Equation (5) [47]:

$$V_{SSN-pp}=I\times Z_{PDN}$$

(5)

where $V_{SSN-pp}$ is the SSN on a particular power rail; I is the transient current circulating in the PDN; and $Z_{PDN}$ is the impedance of the PDN.

The impedance of the PDNs has been simulated using ADS, where the capacitors employed to reduce the impedance have been optimized. The desired impedance value has been set to 1 Ω at 100 MHz. The bandwidth of a step current change, corresponding to a specific rise time, is determined by the equation:

$$\mathrm{BW}=\frac{0.35}{\mathrm{rise}\mathrm{time}}$$

(6)

For the current scenario, with a rise time of 3 ns, the resulting bandwidth (BW) is calculated as 116.7 MHz. The oscillators operate within the voltage range of 3.135 to 3.465 V. The maximum ripple achieved while adhering to the target impedance is approximately 0.035 V, equivalent to around 1% of the amplitude, comfortably below the oscillators’ operational range. The same conclusion can be reached assuming a very conservative ripple of the current of 20 mA. At the most unfavorable case (Vi = 3.135 V) and with a voltage ripple of 1%, the impedance obtained using Equation (5) is 1.57 Ω. The result of the simulation of the PDN for the power rail of the DDMTD clock generator is shown in Figure 11.

The lines that are most critical to signal integrity are those related to the oscillator system. There are various factors that can cause signal quality degradation, including connectors, dielectric materials, capacitors, encapsulated components, and board and trace design and geometry. One method to analyze signal integrity is through the use of scattering parameters or S-parameters [48]. This technique models interconnections as two-port networks in the frequency domain, representing the relationships between incoming and outgoing waves. Figure 12 depicts a representation of these networks, where ports 1 and 4 represent the incident waves, and ports 2 and 3 represent the outgoing waves. Based on this representation and nomenclature, the S-parameters are defined according to the relationship:

$$s_{kj}=\frac{Outputwaveatportk\left(V\right)}{Inputwaveatportj\left(V\right)}$$

(7)

Therefore, parameter $S_{11}$ refers to the relationship between an incoming wave through port 1 and the wave reflected back from the same port, while parameter $S_{21}$ refers to the relationship between an incoming wave through port 1 and the wave exiting from port 2. Figure 13 illustrates both of these relationships. The parameter $S_{11}$ in an interconnection evaluates the amount of signal that will be reflected by that interconnection. This reflection occurs due to impedance mismatch in the interconnection and can be quantified by the reflection coefficient, as indicated in the following expression:

$$S_{11}\left(dB\right)=20\times log\frac{Z_L-Z_o}{Z_L+Z_o}$$

(8)

where the term $Z_L$ refers to the load impedance, and the term $Z_o$ refers to the characteristic impedance. Parameter $S_{21}$ on the other hand pertains to insertion loss. It measures the amount of signal loss experienced along the path from the input port to the output port. These losses can also arise from impedance mismatch. The longer the mismatch region, the greater the losses will be. In the case of good matching, the dominant term for insertion losses would be those caused by the dielectric. Without accounting for losses due to coupling to adjacent traces, conductor losses, or losses from radiated emissions, the conservation of energy establishes a relationship between the $S_{11}$ and $S_{21}$ parameters as shown below:

$$S_{11}^2+S_{21}^2=1\Rightarrow S_{21}=\sqrt[]{1-S_{11}^2}$$

(9)

The same relationship of S-parameters can also be applied to differential lines, where the ports themselves are differential ports. The use of differential ports gives rise to signal conversion between differential and common modes, leading to the creation of the so-called mixed S-parameter matrix.

Table 1. Mixed S-parameter matrix for differential ports.

		Differential Signal		Common Signal
		Port 1	Port 2	Port 1	Port 2
Differential Signal	Port 1	SDD${}_{11}$	SDD${}_{12}$	SDC${}_{11}$	SDC${}_{12}$
	Port 2	SDD${}_{21}$	SDD${}_{22}$	SDC${}_{21}$	SDC${}_{22}$
Common Signal	Port 1	SCD${}_{11}$	SCD${}_{12}$	SCC${}_{11}$	SCC${}_{12}$
	Port 2	SCD${}_{21}$	SCD${}_{22}$	SCC${}_{21}$	SCC${}_{22}$

illustrates the representation of the mixed matrix for two differential ports. The SDD parameters refer to ports where the signal enters and exits in differential mode, SCC refers to the signal entering and exiting the ports in common mode, and the terms SDC and SCD refer to signals entering through one port in common mode and exiting through the other in differential mode, and vice versa.

Following the same nomenclature as in Equation (7), the parameter $SC{D}_{21}$ refers to the ratio of differential signal that enters through port 1 and exits through port 2 in common mode, whereas the term $SD{D}_{21}$ pertains to the ratio of signal that enters through port 1 and exits through port 2 in differential mode.

Under these parameters, the insertion and return losses of the clock lines were simulated with the rise (and fall) time set to 1 ns. Figure 14 displays the insertion losses in the clock differential lines, where low insertion losses guarantee low jitter and high synchronization. Another potential source of noise that can affect signal integrity is coupling, where parasitic capacitances between traces or tracks create induced voltages and produce noise in adjacent tracks. These tracks, referred to as victims, will have two noise components: one at their opposite end, known as FEXT (far-end crosstalk), and another at their near end, called NEXT (near-end crosstalk) [49]. These effects have been simulated in the traces of two differential pairs corresponding to the lines of the clocks, as they are the closest to each other. All the clock lines are routed with the same separation, so simulating two of them is sufficient to validate the design. Figure 15 and Figure 16 show the results of simulations of induced noise NEXT and FEXT in differential mode to differential mode (parameter SDD) and in differential mode to common mode (parameter SCD) between two differential clock tracks. These figures show that the coupled noise, around −40 dB, is very low and will not affect the integrity of the signals. In addition, a simulation of the characteristic impedances of the differential pairs of the clock lines has been performed. All of them were designed with an appropriate topology to maintain a characteristic impedance of 100 Ω. The differential pairs of the greatest length were simulated. Figure 17 shows the simulation results, indicating that a variation greater than ±6% is not exceeded, which is an acceptable variation. The pronounced peaks appearing in the simulation are capacitive effects due to the perforations or vias at the ends of the connection. The simulation results align with the pre-layout analysis, the results of which are presented in Figure 11.

To verify that clock line fluctuations will not introduce additional jitter, an eye diagram simulation has been performed. Figure 18 shows the results of the eye diagrams obtained for clock differential lines. Based on the diagram, a detailed study of fluctuations or jitter has been conducted. The total jitter observed is caused by two main components: random and deterministic [50,51]. Random fluctuations (random jitter—RJ) are produced by noise sources such as transmitter clock phase noise or intrinsic noise such as thermal noise. They are not related to any signal in the system and follow a Gaussian distribution. As it is a Gaussian distribution, the most appropriate way to express it is in terms of standard deviation or root mean square (RMS). Deterministic fluctuations occur when working with components that are not ideal. They are repetitive fluctuations that are predictable and can be classified into several types:

• Inter symbol interference (ISI), produced by the interference of one bit with the next or due to reflections.
• Duty cycle distortion (DCD), produced by alterations in the symmetry of the rising and falling edges, generating a distortion figure with two peaks.
• Periodic jitter (PJ), generated by coupling, noise on power rails, noise on substrates, or instabilities in feedback loops.

Table 2. Measurement of the fluctuations produced in the clock lines.

Contribution	Abracom
	125 MHz	124.992 MHz
Random Jitter RMS	0.11 ps	0.12 ps
Inter Symbols (ISI)	0.09 ps	0.14 ps
Duty Cycle distortion (DCD)	0.01 ps	0.01 ps
Periodic (PJ) RMS	0.26 ps	0.16 ps
Total	1.56 ps	1.74 ps
Eye aperture (width)	94.6%	95.3%
Eye aperture (height)	97.1%	83.4%

shows the contributions to the observed fluctuations in the 125 MHz and 124.992 MHz lines of the first oscillator system. The low jitter values obtained, along with the opening of the eye diagram, ensure the required level of synchronization.

5. Conclusions and Future Work

The White Rabbit protocol requires special hardware to operate. In this work, the design and architecture of an expansion board that includes the hardware required by the protocol, which is not usually available on FPGA boards, has been presented. The expansion board adds flexibility by including two different oscillator systems. Additionally, the signal integrity analysis performed on the expansion board is presented. The signal integrity results predict that the phase noise of the oscillator will be acceptable, and therefore the obtained synchronization will be of the subnanosecond order. The next steps will include manufacturing the board and testing both oscillator systems in a White Rabbit network.