Guaranteeing Isochronous Control of Networked Motion Control Systems Using Phase Offset Adjustment

Guaranteeing isochronous transfer of control commands is an essential function for networked motion control systems. The adoption of real-time Ethernet (RTE) technologies may be profitable in guaranteeing deterministic transfer of control messages. However, unpredictable behavior of software in the motion controller often results in unexpectedly large deviation in control message transmission intervals, and thus leads to imprecise motion. This paper presents a simple and efficient heuristic to guarantee the end-to-end isochronous control with very small jitter. The key idea of our approach is to adjust the phase offset of control message transmission time in the motion controller by investigating the behavior of motion control task. In realizing the idea, we performed a pre-runtime analysis to determine a safe and reliable phase offset and applied the phase offset to the runtime code of motion controller by customizing an open-source based integrated development environment (IDE). We also constructed an EtherCAT-based motion control system testbed and performed extensive experiments on the testbed to verify the effectiveness of our approach. The experimental results show that our heuristic is highly effective even for low-end embedded controller implemented in open-source software components under various configurations of control period and the number of motor drives.


Introduction
Cyber Physical Systems (CPS) are physical and engineered systems whose operations are monitored, coordinated, controlled and integrated by a computing and communication core [1]. Motion control systems (MCS), one of typical CPS applications, have been widely used in various industrial fields such as packaging, semiconductor manufacturing, and production machinery [2][3][4], and are becoming important driving components for the next industrial revolution. A typical MCS consists of computational units such as a motion controller and motor drives, and physical components such as actuators and sensors, that are tightly coupled and collaborate with each other in synchronized manner. In a recent decade, MCS are facing a steady but fundamental change, i.e., the introduction of real-time Ethernet (RTE) as the communication core, replacing conventional fieldbuses [5].
Networked MCS have stringent real-time constraints. Two of the most important are the bounded end-to-end actuation delay and its deviation. The end-to-end actuation delay refers to the time interval from the dispatch of control commands at the controller to the corresponding actuation at a motor drive. Actually, the maximum value of this delay limits the achievable minimum cycle time (MCT) at the controller. In general, the shorter the end-to-end actuation delay is, the higher the precision of single-axis motion becomes. As the term cycle implies, the transfer of actuation control message should also be isochronous while keeping the jitters of successive control message delivery intervals as small as possible. The deviation in the actuation delay is the time difference between the earliest and the latest actuation at different motor drives in the same control cycle. Similarly, the smaller the actuation deviation is, the higher the synchronicity of multi-axis coordinated motion becomes.
With an RTE communication network, previous works [6][7][8] show that careful message scheduling and hardware-based frame switching can come up with the real-time constraints mentioned above at the communication level. However, as pointed out in [9,10], such high precision and synchronicity should be validated by a thorough analysis of the networked process delay, which includes the time for in-controller processing, message delivery, and local handling by each motor drive. Although the clock-driven synchronization method such as distributed clock (DC) in EtherCAT [11] can hide the adverse effect on isochronous control caused by the deviation of the end-to-end actuation delay, its performance also relies on the time-deterministic end-to-end message delivery. Employing hard real-time commercial OS or dedicated hardware optimization may also be alternative solution for this problem. However, it would be a high-cost, time-consuming and error-prone job for engineers who have only limited domain-knowledge. Hence, in this paper, we propose a simple and efficient heuristic to guarantee isochronous control message delivery for higher precision and synchronicity of networked motion control in a systematic manner.
In this paper, we first formulate the isochronous control property required by target motion control systems. The property may not hold due to non-deterministic behavior of software in the motion controller in spite of the determinism provided by the underlying real-time communication technologies. To come up with the problem, we propose a simple and efficient phase offset adjustment heuristic to provide deterministic end-to-end isochronous control. In realizing the proposed heuristic, we customized an open-source, integrated development environment (IDE) to determine the proper phase offset in pre-runtime and applied the phase offset value to the runtime code of the motion controller in a systematic manner. By using a pre-runtime analysis, we eliminate possible interferences with real-time, deterministic message transfer at runtime. On an EtherCAT-based motion control system test-bed, we evaluated the jitters of the intervals between successive control frames observed at the motor drives for varying number of motor drives and control cycle. The experimental results show that the proposed heuristic can greatly reduce the deviation of intervals between successive control frame arrivals for various system configurations, even for the low-performance embedded controller. It is noteworthy that simultaneous actuation among different motor drives using clock-driven synchronization also relies on deterministic control message arrival time at the drive. Hence, our heuristic can also be applied profitably in determining the safe delay value of global clock event at each motor drive.
The rest of this paper is organized as follows. In Section 2, we review the background to networked motion control systems and present related works. Section 3 formulates the problem addressed in this paper and describes our heuristic to come up with the problem. We evaluate the effectiveness of the proposed approach in Section 4. Finally, Section 5 concludes this paper.

Real-Time Requirements for Modern Motion Control System
In a modern MCS considered in this paper, a motion controller and a number of motor drives, which are interconnected through industrial communication links, cooperate with each other in a synchronized manner. Figure 1 illustrates an example of such a configuration, a 6 degrees-of-freedom (DOF) industrial robot, of which components are interconnected through an EtherCAT network in line topology. In the robot system, the motion controller periodically generates control messages containing the commands of target position or velocity and transmits them to the motor drives. On receiving the control information, each motor drive operates its control loop and actuates the corresponding axis. The motor drives are also responsible for reporting status, i.e., current position and velocity, acquired from the associated sensors. By using the feedback information from the motor drives, the motion controller computes control command. In doing so, the operation of motor drives including actuation and sensing should also be carefully coordinated or synchronized with others to make the tool point accurately follow the desired motion trajectory. From the industrial robot example shown in Figure 1, we can derive real-time constraints generally imposed by the modern MCS.
First, the motion system must complete its operation regarding control command computation and data exchange between the controller and all the drives within a pre-defined cycle time for every control cycle. Although the desired cycle time depends on the application requirements, it is known that the cycle time less than 1 ms is needed for the general motion control applications [12]. For the accurate motion conforming to the designated motion profile, it is also crucial to guarantee that successive actuation and sensing operations performed by a motor drive should be performed at fixed intervals, i.e., isochronous. Furthermore, the achievable MCT in the system should be as small as possible to achieve higher-level of precision in the motion in many cases. As mentioned in Section 1, although the introduction of RTE communication network has been greatly contributed to achieve short and deterministic message relay, a holistic analysis and optimization of other processing overhead including in-controller processing delay and drive-local delays should also be considered to guarantee bounded end-to-end control delay and isochronous operations.  Second, for the higher accuracy of coordinated motion, each drive must perform actuation and sensing being accurately synchronized with other drives. A common approach to cope with the requirement is to use a global clock across the networked control system. Assuming that the local clocks at different drives are properly synchronized to a global reference clock, an actuation and sensing task in drives can be invoked by the autonomous interrupt that is generated synchronous to the global clock. Thus, the actuation deviation among different drives can be bounded to negligibly small value. Generally, the clock events are generated with a predefined shift time from the beginning of the control cycle. However, it is often regarded trivial or even ignored that, to properly handle the fresh commands from the controller, the clock events must occur after the control packets from the controller have been received. For this reason, it is very important to estimate the end-to-end delay observed at the drives to determine a safe clock event shift time. From a practical viewpoint, the deviation of the end-to-end delay of control command transmission primarily depends on the fluctuation in controller processing delay because other delay factors in link-level communication latency and drive-local operations are rather deterministic as presented in previous works [8][9][10]. Hence, this paper focuses on guaranteeing isochronous control by keeping the jitters of successive control message transfer as small as possible at the controller level.

EtherCAT Synchronization Methods
Among the RTE networks, we adopt EtherCAT [13] as the communication core of our target system due to its desirable features such as deterministic communication delay, high-speed message relaying, efficient synchronization feature based on global clock, flexible topology, and cost-effectiveness [5]. In a typical motion control system configuration as shown in Figure 1, the master, i.e., the motion controller, periodically generates and transmits motion control commands in the form of EtherCAT frame to the slaves, i.e., motor drives. Along the forwarding path, every slave in an EtherCAT network relays control frames between the input and output ports on-the-fly using a dedicated switching hardware. Once an EtherCAT frame arrives at the end of the network, it returns to the controller along the returning path updating the contents acquired from the associated sensors on-the-fly.
EtherCAT also provides two kinds of synchronization events, i.e., frame events and clock events, for the coordinated operations of multiple slaves [9]. As shown in Figure 2, a local slave task, which is responsible for actuation or sensing, can be synchronized with these events. The frame-driven control process may often experience relatively large deviation due to the variation in in-controller processing time. When a slave task requires a higher-degree of cycle precision with low deviation, a globally synchronized clock should be used. The slave task may take actions based on the autonomous interrupt that is generated with the global clock, referred as the Distributed Clock (DC), and therefore, the deviation of control process can be reduced by up to a few nanoseconds [11].

Related Works
Jitter compensation to avoid performance degradation and instability of control systems has already been studied in traditional real-time control literature. Marti et al. categorized three types of jitters in a control loop, which consists of sampling, control computation and actuation, and proposed a control parameter adjustment to compensate for those jitters [14]. Buttazo and Cervin presented three different approaches for reducing the jitter in control systems and evaluated their performances by simulation [15]. These researches have limitations in that they only considered a single, self-contained control system, which is far from modern control system architecture with distributed control nodes connected through communication links, and the efficiency of proposed methodologies has just been proved by simulation.
One of the major performance indicator for networked control systems has been the achievable MCT. Early works in [6][7][8] formulated the network-level delay components of EtherCAT network to find an MCT according to the control frame size. However, from practical viewpoints, a holistic end-to-end delay and jitter model considering other factors such as in-controller delay and drive-internal operations is required to find a viable MCT. Recent studies to guarantee deterministic operation of motion controller have been presented in two folds. Studies in [16,17] proposed highly customized solutions for the applications which require very short cycle time, i.e., under 1 ms. With the customized hardware and the use of commercial real-time OS, they optimized the operation of EtherCAT protocol stack and presented the achievable MCT under 100 µs including controller internal delay. On the other hand, with the increasing use of open source software (OSS), there are approaches to adopt OSS solutions in realizing real-time motion control systems [9,[18][19][20]. In common, they constructed a real-time motion controller with real-time Linux and open source EtherCAT protocol stack on top of commercial-off-the-shelf (COTS) hardware. The performance evaluation results of them show that OSS-based control systems can also provide very short control cycle with small jitter.
Recent studies have also begun to address a holistic end-to-end delay model of entire motion control system considering in-controller delay or drive-local operations [9,10,21]. Kim et al. considered the internal operation of motor drives, but they have considered the message release interval of controller as constant value [10]. Sung et al. measured the in-controller processing delay in order to carry out the end-to-end delay analysis where EtherCAT synchronization methods are used [9]. Lee et al. proposed a simulation framework to find an optimal phase of distributed nodes with respect to the end-to-end delay and the actuation jitters [21]. However, these works lack any consideration of actual workload [9,21] or do not consider the message release jitter caused by the motion controller [9,10,21].

System Model and Problem Statement
In this paper, we consider an MCS which consists of a motion controller and N (≥1) homogenous motor drives interconnected through EtherCAT communication link in line topology. Generally, a control loop on the controller is assumed to be a periodic activity, of which parts consisting of sampling, computation, and actuation. The control loop can be implemented as a single task or a set of subtasks where each subtask performs one or more parts of a control loop [14]. For simplicity, this paper assumes a single task implementation where the task performs the parts sequentially. We also assume that the control task is an application task with the highest priority among the application-level tasks. Therefore, the execution of the control task can be delayed or preempted only by kernel-level system maintenance services and I/O interrupt handling. Since we assumed homogenous motor drives, the delay caused by the drive-local operations for each drive can also be assumed to be identical.
As mentioned above, the control task, called τ , performs periodically a control sequence, which consists of retrieve, computation and publish operations. Note that the terms retrieve and publish are used instead of sampling and actuation, respectively, to emphasize that the operations are performed for motor drives distributed over the industrial communication link. In the retrieve phase, the control task reads the status information including current position and velocity reported by the motor drives during the previous control cycle. Using the information acquired in the retrieve phase, the control task checks if the motors are being moved properly according to the planned motion trajectory and, if not, generates correction commands in the computation phase. Finally, in the publish phase, the controller transmits motion commands to the motor drives. Figure 3 depicts the timing parameters deal with in our model. We now consider the timing properties of the control task τ . The control task is released periodically at fixed intervals, called T cycle , and the i-th instance of τ is denoted by τ i . In traditional real-time literature, the task instances are assumed to be released strictly on time, and thus the release time of τ i is defined as r i = (i − 1) · T cycle . However, task instances may often experience arbitrary release jitter, denoted by J i , due to timer resolution and interfering kernel-level operations. Then, the actual release time of τ i is given by ∀i ≥ 1, r i = r i−1 + T cycle + J i , where r 0 = 0 and J 0 = 0. As shown in Figure 3, the release jitter J i can be other than 0, which means that the task instance was released earlier, J i < 0, or later , J i > 0, than the designated release time. The response time of τ i , R i is the difference of its finishing time and its release time. The control task τ completes its execution when the publish phase is finished. Then, if we denote the finishing time of τ i as M con i , M con i is given by: For the holistic approach to the problem in this paper, we now consider the timing properties observed at the motor drives. Let M drv(k) i is the time when the i-th control message arrives at the k-th motor drive where k (1 ≤ k ≤ N ) means that the drive is k-th nearest from the controller. Then, we can express M drv(k) i as: where M con i and D comm (k) refer to the time when i-th motion message is released at the controller and the message communication delay to k-th drive, respectively. D comm (k) is computed by . As suggested in [8], the link transmission delay D link can be easily computed when the size of message payload and the link capacity are known a priori. Again, D relay and D prop refer to the message frame relaying delay at each motor drive and the cable propagation delay between consecutive drives, respectively. According to previous works [8,9], the sum of D relay and D prop can be treated as a small constant, e.g., 1 µs, thanks to the on-the-fly frame relay feature of EtherCAT. Therefore, D comm (k) is also treated as constant value for a fixed k in this paper.
To guarantee isochronous actuation, it is required that the time differences between two consecutive motion message arrivals observed at the k-th motor drive should be equal to T cycle as expressed in Equation (3).
Since we treat D comm (k) as constant, Equation (3) is reduced to: From Equations (1) and (4), it follows that the condition to guarantee isochronous control is alternatively given by:

Proposed Heuristic
As mentioned above, the task release jitter J i and the differences of response time of consecutive task instances (R i − R i−1 ) should be zero to guarantee isochronous control. However, it is very difficult to keep these values zero in practice even when we use commercial hard real-time OS and highly customized hardware. Instead, we aim to keep the condition expressed in Equation (4) be approximately met, that is, to keep the maximum variation between two consecutive message release time, (M con i+1 − M con i ), as small as possible. Strictly speaking, M con i should be the time when network interface controller (NIC) of the motion controller starts to transmit the control command packet, which can hardly be controlled. Instead, we can define M con i as the time when the i-th command packet is copied to the FIFO buffer in the NIC. However, it requires cumbersome porting job when the underlying OS and/or NIC hardware is changed. Hence, for higher portability of our heuristic implementation, we propose to control the starts times of the publish phase in the control task instances. Note that, hereinafter, M con i refers to the start time of the publish phase in τ i unless otherwise specified. It follows that the computation of M con i is rewritten as: where R comp i represents, for the i-th task instance, the time difference of its computation phase finishing time and its release time.
In a straightforward implementation of the control task, the publish phase starts to execute right after the completion of the computation phase. However, in our heuristic, the publish phase of task instances is forced to be executed with a proper offset from their designated release time. Suppose that a proper offset Φ p is known a priori, the first task instance τ 0 starts its publish phase with the phase offset Φ p from its release time, and the publish phases of successive task instances start with a fixed interval T cycle . Then, the adjusted start time of publish phase M con ′ i is give by: We now explain the way how to determine the proper phase offset Φ p . In determining the phase offset, we should consider the following constraints: • (1) For all the control cycles, the condition (M con ′ i > M con i ) should be met in order not for the controller to try to transmit out-of-date control command because the computation phase may not be completed at the time the controller begins to transmit the control command. The phase offset value Φ p satisfying this constraint becomes the lower bound of Φ p , denoted by Φ − p . • (2) Any two consecutive control cycles should not overlap with each other. If this constraint is violated, the next control cycle should wait for the arrival of fresh feedback information from the drives. Such a situation may occur when the adjusted phase offset becomes too large. Then, the phase offset value Φ p satisfying this constraint is the upper bound of Φ p , denoted by Φ + p . • (3) For safety, the condition (Φ − p < Φ + p ) must be satisfied. Otherwise, it means that there may not be enough time for all the required operations to complete in a control cycle.
Since it is very difficult to determine Φ − p satisfying Constraint (1) theoretically for randomly distributed J i and R comp i , we estimate Φ − p with the J i and R comp i values obtained empirically through a pre-runtime analysis. In determining Φ + p satisfying Constraint (2), a certain amount of time should be reserved between the control message release time and the release time of next control period. The reserved time amounts to the time taken to deliver the control commands to all the drives and convey the feedback information from the drives under the worst-case release jitter scenario. The worst-case release jitter scenario means that the next task instance is released earlier with maximum task release jitter max(|J − i |). Let D rtt denote the round-trip time taken for the communication, then we can compute D rtt by D rtt = (2N − 1) · D relay + 2N · D prop + D link . In summary, the safe bound of phase offset Φ p satisfying the constraints above is expressed as: where max(|J − i |) represents the maximum release jitter for the task instance among the task instances released earlier than the designated release time. Figure 4 outlines the discussion in this section.

Implementation of Our Heuristic
As described earlier, our goal is to minimize actuation jitters observed at the motor drives by keeping the time intervals of successive control message transfer by the controller at fixed intervals. For this purpose, we devised a heuristic to delay control message transfer time, i.e., message publish time, based on the estimation of the worst-case start time of the publish phase. In realizing our heuristic, we aim to avoid modifying kernel-level code or using customized hardware in order to save engineering efforts of engineers who have domain-knowledge only. Hence, we used a pre-runtime analysis integrated with an open-source IDE for motion applications, called Beremiz [20], to estimate the safe phase offset of the publish phase, Φ p . The runtime executable image generated by Beremiz is composed of motion application codes, OS-dependent task management stubs, and EtherCAT communication codes.
In order to obtain the information for determining the phase offset Φ p during the pre-run cycles, we first modified the task main body generation part for the pre-run mode to insert time measurement code using timer API supported by target OS, e.g., rt_timer_read API in Xenomai. Figure 5 depicts the execution flow of our pre-runtime analysis. In the pre-run cycles, the motion runtime code executes for the time intervals, which is configurable by users, and the information such as release jitter J i and response time of computation phase R comp i is collected. Then, the collected information is analyzed and displayed by the pre-runtime analyzer. As shown in Figure 6, the analyzer suggests the range of safe phase offset values computed by Equation (8). Note that the phase offset value represented as the percentile ratio normalized with respect to the control cycle, denoted by δ p , i.e., Φ p = δ p · T cycle . In addition, finally, once the user choose the phase offset value, the offset is applied to the final runtime code for the control task. Listing 1 describes the pseudo code for the control task body generated by the Beremiz IDE.

Performance Evaluation
In this section, we present experimental results to verify the effectiveness of our proposed heuristic. We performed a series of experiments in terms of the frame inter-arrival times observed at motor drives, , for various configurations, i.e., different types of controller, the control period T cycle , and the number of motor drives N . A safe and reliable phase offset factor δ p was obtained by pre-runtime analysis on a testbed.

Experimental Setup
The performance evaluation was conducted on an EtherCAT-based MCS testbed which is compromised of a motion controller, multiple homogeneous motor drives, and external measurement equipments as shown in Figure 7a. For the comparison of different implementations of motion controller, we used three different types of motion controller: a high-end industrial PC (IPC), a low-end embedded single board computer (SBC) board, and a commercial EtherCAT master controller. The IPC and embedded SBC were constructed solely from the open-source components such as Xenomai real-time patch for Linux [22] and IgH EtherCAT master stack [23]. Table 1 summarizes the details of our testbed. As the test motion application, we used a point-to-point (PTP) movement application in which each motor drive repeats to move from base position to target position and vice versa. Figure 7b depicts the state transition of the test application.  In order to reduce potential interferences caused by kernel-level tasks, open-source based controllers, e.g., IPC and embedded SBC, were configured to start with single-user mode and operate without X-windows desktop manager. In addition, the modified network device driver and NEON floating point unit (FPU) compiler option proposed in our previous work [24] are used for embedded SBC as the platform optimization. For the motor drive platform, we chose TI industrial communication engine (ICE) board and configured the drives to use identical process data object (PDO) data set for cyclic synchronous position (CSP) mode operation. The frame arrival time observed at the motor drives, M drv(k) i , was measured by logging GPIO output signals generated at the time when an EtherCAT frame arrives. The GPIO signals were collected by using an external data acquisition (DAQ) board with ns resolution timestamp. The statistical analysis on the measured samples was conducted using a LabVIEW application.

Analysis of the Behavior of Motion Control Task
As described in Section 3.2, we need to obtain pre-runtime analysis results to determine a safe phase offset Φ p for the publish phase. Recall that the lower bound of Φ p , Φ − p , is determined by the worst-case (J i + R comp i ) and the upper bound of Φ p , Φ + p , is determined by the terms max(|J − i |) and D rtt , respectively. Thus, we measured the J i and R comp i values of the control task τ and analyzed them on open-source based controllers. Table 2 shows the measurement results of J i , R comp i , and M con i as we vary the length of control cycle T cycle and the number of motor drives N . The M con i values, which are the start times of publish phase without heuristic, were also measured and analyzed for the comparison with the results of our heuristic later in Section 4.3. For each experiment, we conducted the measurement for 5 min, of which collected data amounts to 1,200,000 samples with T cycle = 250 µs for instance. In the results, ∆ denotes the absolute difference between the maximum and minimum value for each term.    According to Table 2, when we use high-end IPC controller, the task release jitter J i values are quite stable for varying T cycle and N . By contrast, J i for low-end embedded SBC tends to increase as we increase the control cycle T cycle . It is considered that the cache pollution due to other background tasks and kernel activities is enlarging the release jitter of control task as pointed out in previous works [9,19]. As for R comp i , its average and deviation depend on both T cycle and N when we use embedded SBC. Regardless of the controller used, the average of R comp i increases with respect to N . This is because the feedback data size is linearly increased with N and the motion computation is performed for each axis, i.e., drive, and thus the execution time for the retrieve phase and computation phase is monotonically increased. On the other hand, the deviation of R comp i using embedded SBC decreases significantly for shorter T cycle with a fixed N . It is considered that, with shorter T cycle , there are less probabilities for the control task to be exposed to the interferences from kernel-level activities and cache miss handling overhead will also be reduced. As for M con i , IPC and embedded SBC show different characteristics. With IPC controller, the deviation and absolute difference values of M con i are quite stable for all the experimental configurations. However, with embedded SBC, we observe that the values increase with respect to T cycle and N , which indicates the deviation of M con i in embedded SBC is affected by both J i and R comp i . Figure 8 illustrates the cumulative distribution function (CDF) of measured M con i values. As mentioned earlier, the sum of J i and R comp i determine M con i and the absolute difference ∆ of J i and R comp i primarily contributes to the deviation of M con i . This characteristic is more apparent for embedded SBC as shown in Figure 8. In summary, we expect from the pre-runtime analysis results that our heuristic will be more beneficial to low-end embedded SBC with highly fluctuating message release interval.

Frame Inter-Arrival Time Observed at Motor Drives
The objective of our heuristic is to keep the differences between frame inter-arrival times observed at the motor drives, T drv(k) i , and the control cycle T cycle as small as possible for deterministic isochronous control. To verify the effectiveness of our approach, we conducted a set of experiments to measure T drv(k) i for different controller implementations and compared the results. In each experiment, the measurement was performed for 30 min, for instance, of which collected data size amounts to 3,600,000 samples with T cycle = 500 µs. Since we used homogeneous motor drives and the switching and propagation delay of frames is very small and bounded, we can regard the values observed at the first motor drive, T drv(1) i , as the representative case without loss of generality. For performance evaluation, we define additional performance indicators, ϵ 1% and ϵ 10% . The parameters ϵ 1% and ϵ 10% account for the number of intervals between successive frame arrivals, of which length exceeding 1% and 10% with respect to a given T cycle , respectively.
We first measured and compared the frame inter-arrival times for various testbed configurations. The configurations were varied according to T cycle , N , controller hardware types, and whether to use our heuristic or not. For the experiments with our heuristic, we chose the δ max p values shown in Table 3. The performance evaluation results are summarized in Table 4 and Figure 9. The boxes in the Figure 9 represent the range in which 99% of measured samples are located. The lower and upper whiskers in the plots refer to the low 0.5% and low 99.5% value shown in Table 4. Table 4. Comparison of frame inter-arrival times for different controller configurations.

Cont.
Embedded SBC IPC T cycle 1000 µs 500 µs 1000 µs 500 µs      Table 4, IPC greatly outperforms embedded SBC in terms of the deviation of frame inter-arrival time jitters without our heuristic. For all T cycle and N configurations used, IPC controller has zero ϵ 10% and much smaller ϵ 1% than embedded SBC. However, after applying our heuristic, we can observe that embedded SBC controller shows performances comparable to IPC for all T cycle and N . For instance, ϵ 10% is decreased to zero and ϵ 1% is reduced by up to 89% compared with the case without our heuristic where T cycle = 500 µs and N = 8.  We conducted another set of experiments to observe the performances according to δ p shown in Table 3. Figure 10 illustrates the distributions of frame inter-arrival time for δ min p , δ med p , and δ max p . Compared with the results in Figure 9, our heuristic can greatly reduce the deviation of frame inter-arrival time irrespective of δ p used where the 99% of frame inter-arrival times have jitters less than ±8 µs for all N . However, we can also notice that the advantage of our heuristic is much more outstanding for δ max p compared with δ min p . This is because δ max p is computed with relatively deterministic values D rtt and J − i . Hence, we can state that it is recommended to use δ max p for best performance with our heuristic.  We also compared the performance of embedded SBC and that of a commercial EtherCAT master controller, which has similar hardware specification. All experimental configurations are the same as previous experiments, except for the implementation of motion control task. We used TwinCAT-2 NC PTP runtime to implement a motion application for the commercial controller. Figure 11 shows the performance evaluation results. The results demonstrate that embedded SBC with our heuristic outperforms the commercial controller for all the test configurations used. For T cycle = 1000 µs, the commercial controller has good performance in average because the 99% of measured samples are located in 24.9 µs, 29.4 µs, and 35.7 µs, which amount to 2%-3% with respect to T cycle for varying N . However, for a smaller control cycle T cycle = 500 µs, the performance degradation of the commercial controller is noticeable with increasing N in terms of absolute difference ∆. By contrast, the evaluation results of our approach show quite stable distribution for all the test configurations. Although the commercial controller used in the experiment may not be fully optimized, the results imply that our approach can achieve performance comparable to a commercial solution.  We now consider the usefulness of our approach for guaranteeing the synchronicity among multiple motor drives. For the purpose, we measured the frame inter-arrival times observed at different motor drives by connecting GPIO pins of corresponding motor drives to the probes of oscilloscope. In the experiment, we used embedded SBC for the motion controller. Figure 12 demonstrates the measurement results of T drv(1) i and T drv (8) i with T cycle = 500 µs and N = 8. The observations from the measurement results are as follows. First, although the measurements were conducted for the first drive and the last drive in the control network, the characteristics of frame inter-arrival times observed at those drives are similar to each other. This means that the communication delays and frame switching overheads are highly deterministic throughout the control network regardless of the relative position of a drive. Second, the inter-arrival times with our heuristic performs similar to the case without heuristic in average. However, the absolute jitter of frame inter-arrival times has been dramatically reduced after applying our heuristic. This characteristic makes it easy to determine a safe clock shift time for accurate clock-based synchronization described in Section 2.2, and thus contributes to higher synchronicity of coordinated motion.

Conclusions
In modern networked motion control systems, guaranteeing isochronous control is very crucial for higher precision and synchronicity of target motion application. This paper proposed a simple and efficient heuristic approach to guarantee isochronous control by adjusting message transfer phase offset based on pre-runtime analysis. We also made it easy to apply phase offset adjustment code to the auto-generated motion application runtime code by integrating the pre-runtime analysis stubs and user-controlled phase offset with an open-source motion application IDE. Our proposed heuristic can also be effectively used to determine a safe shift time of global clock for accurately coordinated motion of multiple motor drives.
For in-depth and practical evaluation, we conducted extensive measurements for various configurations on a realistic testbed that was constructed from open-source software and COTS hardware platforms. The performance evaluation results show that our heuristic can bound successive actuation jitters within 1%-2% of the control cycle time even on a low-end embedded processor without any kernel-level modification and customized hardware. We also observed that our approach can provide performances comparable to or even better than that of a commercial controller.
In our future research, we will study how to extend our heuristic to support lightweight on-line adjustment. We also plan to extend our model for multiple control tasks with heterogeneous periods to provide higher precision and synchronicity of complex control under heavy interfering workloads such as monitoring and logging.