Next Article in Journal
Novel Spectrum Sensing Algorithms for OFDM Cognitive Radio Networks
Next Article in Special Issue
On-Board Event-Based State Estimation for Trajectory Approaching and Tracking of a Vehicle
Previous Article in Journal
Human-Automation Interaction Design for Adaptive Cruise Control Systems of Ground Vehicles
Previous Article in Special Issue
Solving Energy-Aware Real-Time Tasks Scheduling Problem with Shuffled Frog Leaping Algorithm on Heterogeneous Platforms
Article Menu

Export Article

Sensors 2015, 15(6), 13945-13965;

Guaranteeing Isochronous Control of Networked Motion Control Systems Using Phase Offset Adjustment
Department of Mechanical and Information Engineering, University of Seoul, 163 Seoulsiripdae-ro, Dongdaemun-gu, Seoul 130-743, Korea
Author to whom correspondence should be addressed.
Academic Editor: Albert M. K. Cheng
Received: 22 April 2015 / Accepted: 5 June 2015 / Published: 12 June 2015


: Guaranteeing isochronous transfer of control commands is an essential function for networked motion control systems. The adoption of real-time Ethernet (RTE) technologies may be profitable in guaranteeing deterministic transfer of control messages. However, unpredictable behavior of software in the motion controller often results in unexpectedly large deviation in control message transmission intervals, and thus leads to imprecise motion. This paper presents a simple and efficient heuristic to guarantee the end-to-end isochronous control with very small jitter. The key idea of our approach is to adjust the phase offset of control message transmission time in the motion controller by investigating the behavior of motion control task. In realizing the idea, we performed a pre-runtime analysis to determine a safe and reliable phase offset and applied the phase offset to the runtime code of motion controller by customizing an open-source based integrated development environment (IDE). We also constructed an EtherCAT-based motion control system testbed and performed extensive experiments on the testbed to verify the effectiveness of our approach. The experimental results show that our heuristic is highly effective even for low-end embedded controller implemented in open-source software components under various configurations of control period and the number of motor drives.
networked motion control systems; isochronous control; phase offset adjustment; EtherCAT; open source software

1. Introduction

Cyber Physical Systems (CPS) are physical and engineered systems whose operations are monitored, coordinated, controlled and integrated by a computing and communication core [1]. Motion control systems (MCS), one of typical CPS applications, have been widely used in various industrial fields such as packaging, semiconductor manufacturing, and production machinery [24], and are becoming important driving components for the next industrial revolution. A typical MCS consists of computational units such as a motion controller and motor drives, and physical components such as actuators and sensors, that are tightly coupled and collaborate with each other in synchronized manner. In a recent decade, MCS are facing a steady but fundamental change, i.e., the introduction of real-time Ethernet (RTE) as the communication core, replacing conventional fieldbuses [5].

Networked MCS have stringent real-time constraints. Two of the most important are the bounded end-to-end actuation delay and its deviation. The end-to-end actuation delay refers to the time interval from the dispatch of control commands at the controller to the corresponding actuation at a motor drive. Actually, the maximum value of this delay limits the achievable minimum cycle time (MCT) at the controller. In general, the shorter the end-to-end actuation delay is, the higher the precision of single-axis motion becomes. As the term cycle implies, the transfer of actuation control message should also be isochronous while keeping the jitters of successive control message delivery intervals as small as possible. The deviation in the actuation delay is the time difference between the earliest and the latest actuation at different motor drives in the same control cycle. Similarly, the smaller the actuation deviation is, the higher the synchronicity of multi-axis coordinated motion becomes.

With an RTE communication network, previous works [68] show that careful message scheduling and hardware-based frame switching can come up with the real-time constraints mentioned above at the communication level. However, as pointed out in [9,10], such high precision and synchronicity should be validated by a thorough analysis of the networked process delay, which includes the time for in-controller processing, message delivery, and local handling by each motor drive. Although the clock-driven synchronization method such as distributed clock (DC) in EtherCAT [11] can hide the adverse effect on isochronous control caused by the deviation of the end-to-end actuation delay, its performance also relies on the time-deterministic end-to-end message delivery. Employing hard real-time commercial OS or dedicated hardware optimization may also be alternative solution for this problem. However, it would be a high-cost, time-consuming and error-prone job for engineers who have only limited domain-knowledge. Hence, in this paper, we propose a simple and efficient heuristic to guarantee isochronous control message delivery for higher precision and synchronicity of networked motion control in a systematic manner.

In this paper, we first formulate the isochronous control property required by target motion control systems. The property may not hold due to non-deterministic behavior of software in the motion controller in spite of the determinism provided by the underlying real-time communication technologies. To come up with the problem, we propose a simple and efficient phase offset adjustment heuristic to provide deterministic end-to-end isochronous control. In realizing the proposed heuristic, we customized an open-source, integrated development environment (IDE) to determine the proper phase offset in pre-runtime and applied the phase offset value to the runtime code of the motion controller in a systematic manner. By using a pre-runtime analysis, we eliminate possible interferences with real-time, deterministic message transfer at runtime. On an EtherCAT-based motion control system test-bed, we evaluated the jitters of the intervals between successive control frames observed at the motor drives for varying number of motor drives and control cycle. The experimental results show that the proposed heuristic can greatly reduce the deviation of intervals between successive control frame arrivals for various system configurations, even for the low-performance embedded controller. It is noteworthy that simultaneous actuation among different motor drives using clock-driven synchronization also relies on deterministic control message arrival time at the drive. Hence, our heuristic can also be applied profitably in determining the safe delay value of global clock event at each motor drive.

The rest of this paper is organized as follows. In Section 2, we review the background to networked motion control systems and present related works. Section 3 formulates the problem addressed in this paper and describes our heuristic to come up with the problem. We evaluate the effectiveness of the proposed approach in Section 4. Finally, Section 5 concludes this paper.

2. Background and Related Works

2.1. Real-Time Requirements for Modern Motion Control System

In a modern MCS considered in this paper, a motion controller and a number of motor drives, which are interconnected through industrial communication links, cooperate with each other in a synchronized manner. Figure 1 illustrates an example of such a configuration, a 6 degrees-of-freedom (DOF) industrial robot, of which components are interconnected through an EtherCAT network in line topology. In the robot system, the motion controller periodically generates control messages containing the commands of target position or velocity and transmits them to the motor drives. On receiving the control information, each motor drive operates its control loop and actuates the corresponding axis. The motor drives are also responsible for reporting status, i.e., current position and velocity, acquired from the associated sensors. By using the feedback information from the motor drives, the motion controller computes control command. In doing so, the operation of motor drives including actuation and sensing should also be carefully coordinated or synchronized with others to make the tool point accurately follow the desired motion trajectory. From the industrial robot example shown in Figure 1, we can derive real-time constraints generally imposed by the modern MCS.

First, the motion system must complete its operation regarding control command computation and data exchange between the controller and all the drives within a pre-defined cycle time for every control cycle. Although the desired cycle time depends on the application requirements, it is known that the cycle time less than 1 ms is needed for the general motion control applications [12]. For the accurate motion conforming to the designated motion profile, it is also crucial to guarantee that successive actuation and sensing operations performed by a motor drive should be performed at fixed intervals, i.e., isochronous. Furthermore, the achievable MCT in the system should be as small as possible to achieve higher-level of precision in the motion in many cases. As mentioned in Section 1, although the introduction of RTE communication network has been greatly contributed to achieve short and deterministic message relay, a holistic analysis and optimization of other processing overhead including in-controller processing delay and drive-local delays should also be considered to guarantee bounded end-to-end control delay and isochronous operations.

Second, for the higher accuracy of coordinated motion, each drive must perform actuation and sensing being accurately synchronized with other drives. A common approach to cope with the requirement is to use a global clock across the networked control system. Assuming that the local clocks at different drives are properly synchronized to a global reference clock, an actuation and sensing task in drives can be invoked by the autonomous interrupt that is generated synchronous to the global clock. Thus, the actuation deviation among different drives can be bounded to negligibly small value. Generally, the clock events are generated with a predefined shift time from the beginning of the control cycle. However, it is often regarded trivial or even ignored that, to properly handle the fresh commands from the controller, the clock events must occur after the control packets from the controller have been received. For this reason, it is very important to estimate the end-to-end delay observed at the drives to determine a safe clock event shift time. From a practical viewpoint, the deviation of the end-to-end delay of control command transmission primarily depends on the fluctuation in controller processing delay because other delay factors in link-level communication latency and drive-local operations are rather deterministic as presented in previous works [810]. Hence, this paper focuses on guaranteeing isochronous control by keeping the jitters of successive control message transfer as small as possible at the controller level.

2.2. EtherCAT Synchronization Methods

Among the RTE networks, we adopt EtherCAT [13] as the communication core of our target system due to its desirable features such as deterministic communication delay, high-speed message relaying, efficient synchronization feature based on global clock, flexible topology, and cost-effectiveness [5]. In a typical motion control system configuration as shown in Figure 1, the master, i.e., the motion controller, periodically generates and transmits motion control commands in the form of EtherCAT frame to the slaves, i.e., motor drives. Along the forwarding path, every slave in an EtherCAT network relays control frames between the input and output ports on-the-fly using a dedicated switching hardware. Once an EtherCAT frame arrives at the end of the network, it returns to the controller along the returning path updating the contents acquired from the associated sensors on-the-fly.

EtherCAT also provides two kinds of synchronization events, i.e., frame events and clock events, for the coordinated operations of multiple slaves [9]. As shown in Figure 2, a local slave task, which is responsible for actuation or sensing, can be synchronized with these events. The frame-driven control process may often experience relatively large deviation due to the variation in in-controller processing time. When a slave task requires a higher-degree of cycle precision with low deviation, a globally synchronized clock should be used. The slave task may take actions based on the autonomous interrupt that is generated with the global clock, referred as the Distributed Clock (DC), and therefore, the deviation of control process can be reduced by up to a few nanoseconds [11].

2.3. Related Works

Jitter compensation to avoid performance degradation and instability of control systems has already been studied in traditional real-time control literature. Marti et al. categorized three types of jitters in a control loop, which consists of sampling, control computation and actuation, and proposed a control parameter adjustment to compensate for those jitters [14]. Buttazo and Cervin presented three different approaches for reducing the jitter in control systems and evaluated their performances by simulation [15]. These researches have limitations in that they only considered a single, self-contained control system, which is far from modern control system architecture with distributed control nodes connected through communication links, and the efficiency of proposed methodologies has just been proved by simulation.

One of the major performance indicator for networked control systems has been the achievable MCT. Early works in [68] formulated the network-level delay components of EtherCAT network to find an MCT according to the control frame size. However, from practical viewpoints, a holistic end-to-end delay and jitter model considering other factors such as in-controller delay and drive-internal operations is required to find a viable MCT. Recent studies to guarantee deterministic operation of motion controller have been presented in two folds. Studies in [16,17] proposed highly customized solutions for the applications which require very short cycle time, i.e., under 1 ms. With the customized hardware and the use of commercial real-time OS, they optimized the operation of EtherCAT protocol stack and presented the achievable MCT under 100 μs including controller internal delay. On the other hand, with the increasing use of open source software (OSS), there are approaches to adopt OSS solutions in realizing real-time motion control systems [9,1820]. In common, they constructed a real-time motion controller with real-time Linux and open source EtherCAT protocol stack on top of commercial-off-the-shelf (COTS) hardware. The performance evaluation results of them show that OSS-based control systems can also provide very short control cycle with small jitter.

Recent studies have also begun to address a holistic end-to-end delay model of entire motion control system considering in-controller delay or drive-local operations [9,10,21]. Kim et al. considered the internal operation of motor drives, but they have considered the message release interval of controller as constant value [10]. Sung et al. measured the in-controller processing delay in order to carry out the end-to-end delay analysis where EtherCAT synchronization methods are used [9]. Lee et al. proposed a simulation framework to find an optimal phase of distributed nodes with respect to the end-to-end delay and the actuation jitters [21]. However, these works lack any consideration of actual workload [9,21] or do not consider the message release jitter caused by the motion controller [9,10,21].

3. Phase Offset Adjustment for Isochronous Control: A Tool-based Heuristic Approach

3.1. System Model and Problem Statement

In this paper, we consider an MCS which consists of a motion controller and N (≥1) homogenous motor drives interconnected through EtherCAT communication link in line topology. Generally, a control loop on the controller is assumed to be a periodic activity, of which parts consisting of sampling, computation, and actuation. The control loop can be implemented as a single task or a set of subtasks where each subtask performs one or more parts of a control loop [14]. For simplicity, this paper assumes a single task implementation where the task performs the parts sequentially. We also assume that the control task is an application task with the highest priority among the application-level tasks. Therefore, the execution of the control task can be delayed or preempted only by kernel-level system maintenance services and I/O interrupt handling. Since we assumed homogenous motor drives, the delay caused by the drive-local operations for each drive can also be assumed to be identical.

As mentioned above, the control task, called τ, performs periodically a control sequence, which consists of retrieve, computation and publish operations. Note that the terms retrieve and publish are used instead of sampling and actuation, respectively, to emphasize that the operations are performed for motor drives distributed over the industrial communication link. In the retrieve phase, the control task reads the status information including current position and velocity reported by the motor drives during the previous control cycle. Using the information acquired in the retrieve phase, the control task checks if the motors are being moved properly according to the planned motion trajectory and, if not, generates correction commands in the computation phase. Finally, in the publish phase, the controller transmits motion commands to the motor drives. Figure 3 depicts the timing parameters deal with in our model.

We now consider the timing properties of the control task τ. The control task is released periodically at fixed intervals, called Tcycle, and the i-th instance of τ is denoted by τi. In traditional real-time literature, the task instances are assumed to be released strictly on time, and thus the release time of τi is defined as ri = (i − 1) · Tcycle. However, task instances may often experience arbitrary release jitter, denoted by Ji, due to timer resolution and interfering kernel-level operations. Then, the actual release time of τi is given by ∀i ≥ 1, ri = ri−1 + Tcycle + Ji, where r0 = 0 and J0 = 0. As shown in Figure 3, the release jitter Ji can be other than 0, which means that the task instance was released earlier, Ji < 0, or later , Ji > 0, than the designated release time. The response time of τi, Ri is the difference of its finishing time and its release time. The control task τ completes its execution when the publish phase is finished. Then, if we denote the finishing time of τi as M i con, M i con is given by:

M 0 con = r 0 + R 0 = R 0 , ( r 0 = 0 , J 0 = 0 ) M 1 con = r 1 + R 1 = ( r 0 + T cycle + J 1 ) + R 1 = ( T cycle + J 1 ) + R 1 M 2 con = r 2 + R 2 = ( r 1 + T cycle + J 2 ) + R 2 = ( 2 T cycle + J 1 + J 2 ) + R 2 M i con = r i + R i = i T cycle + n = 1 i J n + R i

For the holistic approach to the problem in this paper, we now consider the timing properties observed at the motor drives. Let M i drv ( k ) is the time when the i-th control message arrives at the k-th motor drive where k (1 ≤ kN) means that the drive is k-th nearest from the controller. Then, we can express M i drv ( k ) as:

M i drv ( k ) = M i con + D comm ( k )
where M i con and Dcomm(k) refer to the time when i-th motion message is released at the controller and the message communication delay to k-th drive, respectively. Dcomm(k) is computed by Dcomm(k) = Dlink + k · (Drelay + Dprop). As suggested in [8], the link transmission delay Dlink can be easily computed when the size of message payload and the link capacity are known a priori. Again, Drelay and Dprop refer to the message frame relaying delay at each motor drive and the cable propagation delay between consecutive drives, respectively According to previous works [8,9], the sum of Drelay and Dprop can be treated as a small constant, e.g., 1 μs, thanks to the on-the-fly frame relay feature of EtherCAT. Therefore, Dcomm(k) is also treated as constant value for a fixed k in this paper.

To guarantee isochronous actuation, it is required that the time differences between two consecutive motion message arrivals observed at the k-th motor drive should be equal to Tcycle as expressed in Equation (3).

M i + 1 drv ( k ) = M i drv ( k ) + T cycle

Since we treat Dcomm(k) as constant, Equation (3) is reduced to:

( M i + 1 con M i con ) = T cycle

From Equations (1) and (4), it follows that the condition to guarantee isochronous control is alternatively given by:

i 1 , J i + ( R i R i 1 ) = 0

3.2. Proposed Heuristic

As mentioned above, the task release jitter Ji and the differences of response time of consecutive task instances (RiRi−1) should be zero to guarantee isochronous control. However, it is very difficult to keep these values zero in practice even when we use commercial hard real-time OS and highly customized hardware. Instead, we aim to keep the condition expressed in Equation (4) be approximately met, that is, to keep the maximum variation between two consecutive message release time, ( M i + 1 con M i con ), as small as possible.

Strictly speaking, M i con should be the time when network interface controller (NIC) of the motion controller starts to transmit the control command packet, which can hardly be controlled. Instead, we can define M i con as the time when the i-th command packet is copied to the FIFO buffer in the NIC. However, it requires cumbersome porting job when the underlying OS and/or NIC hardware is changed. Hence, for higher portability of our heuristic implementation, we propose to control the starts times of the publish phase in the control task instances. Note that, hereinafter, M i con refers to the start time of the publish phase in τi unless otherwise specified. It follows that the computation of M i con is rewritten as:

M i con = i T cycle + n = 1 i J n + R i comp
where R i comp represents, for the i-th task instance, the time difference of its computation phase finishing time and its release time.

In a straightforward implementation of the control task, the publish phase starts to execute right after the completion of the computation phase. However, in our heuristic, the publish phase of task instances is forced to be executed with a proper offset from their designated release time. Suppose that a proper offset Φp is known a priori, the first task instance τ0 starts its publish phase with the phase offset Φp from its release time, and the publish phases of successive task instances start with a fixed interval Tcycle. Then, the adjusted start time of publish phase M i con is give by:

i 1 , M i con = i T cycle + M 0 con
where M 0 con = r 0 + Φ p.

We now explain the way how to determine the proper phase offset Φp. In determining the phase offset, we should consider the following constraints:

  • (1) For all the control cycles, the condition ( M i con > M i con ) should be met in order not for the controller to try to transmit out-of-date control command because the computation phase may not be completed at the time the controller begins to transmit the control command. The phase offset value Φp satisfying this constraint becomes the lower bound of Φp, denoted by Φ p .

  • (2) Any two consecutive control cycles should not overlap with each other. If this constraint is violated, the next control cycle should wait for the arrival of fresh feedback information from the drives. Such a situation may occur when the adjusted phase offset becomes too large. Then, the phase offset value Φp satisfying this constraint is the upper bound of Φp, denoted by Φ p +.

  • (3) For safety, the condition ( Φ p < Φ p + ) must be satisfied. Otherwise, it means that there may not be enough time for all the required operations to complete in a control cycle.

Since it is very difficult to determine Φ p satisfying Constraint (1) theoretically for randomly distributed Ji and R i comp, we estimate Φ p with the Ji and R i comp values obtained empirically through a pre-runtime analysis. In determining Φ p + satisfying Constraint (2), a certain amount of time should be reserved between the control message release time and the release time of next control period. The reserved time amounts to the time taken to deliver the control commands to all the drives and convey the feedback information from the drives under the worst-case release jitter scenario. The worst-case release jitter scenario means that the next task instance is released earlier with maximum task release jitter max ( | J i | ). Let Drtt denote the round-trip time taken for the communication, then we can compute Drtt by Drtt = (2N − 1) · Drelay + 2N · Dprop + Dlink. In summary, the safe bound of phase offset Φp satisfying the constraints above is expressed as:

Φ p = [ Φ p , Φ p + ] = [ max ( J i + R i comp ) , T cycle ( D rtt + max ( | J i | ) ) ] , i 1
where max ( | J i | ) represents the maximum release jitter for the task instance among the task instances released earlier than the designated release time. Figure 4 outlines the discussion in this section.

3.3. Implementation of Our Heuristic

As described earlier, our goal is to minimize actuation jitters observed at the motor drives by keeping the time intervals of successive control message transfer by the controller at fixed intervals. For this purpose, we devised a heuristic to delay control message transfer time, i.e., message publish time, based on the estimation of the worst-case start time of the publish phase. In realizing our heuristic, we aim to avoid modifying kernel-level code or using customized hardware in order to save engineering efforts of engineers who have domain-knowledge only. Hence, we used a pre-runtime analysis integrated with an open-source IDE for motion applications, called Beremiz [20], to estimate the safe phase offset of the publish phase, Φp. The runtime executable image generated by Beremiz is composed of motion application codes, OS-dependent task management stubs, and EtherCAT communication codes.

In order to obtain the information for determining the phase offset Φp during the pre-run cycles, we first modified the task main body generation part for the pre-run mode to insert time measurement code using timer API supported by target OS, e.g., rt_timer_read API in Xenomai. Figure 5 depicts the execution flow of our pre-runtime analysis. In the pre-run cycles, the motion runtime code executes for the time intervals, which is configurable by users, and the information such as release jitter Ji and response time of computation phase R i comp is collected. Then, the collected information is analyzed and displayed by the pre-runtime analyzer. As shown in Figure 6, the analyzer suggests the range of safe phase offset values computed by Equation (8). Note that the phase offset value represented as the percentile ratio normalized with respect to the control cycle, denoted by δp, i.e., Φp = δp · Tcycle. In addition, finally, once the user choose the phase offset value, the offset is applied to the final runtime code for the control task. Listing 1 describes the pseudo code for the control task body generated by the Beremiz IDE.

Listing 1. Pseudo code of control task with proposed heuristic.

while(! PLC_shutdown)
retrieving sensed values;
computing motion command;
  /* Delay transmission time at first task instance */
   time2transmit = first_release_time + $\delta$ * TSK_PERIOD;
  /* Next transmission time is determined by
   sum of previous transmission time and task period */
   time2transmit += TSK_PERIOD;
now = get current system time;
/* Waiting for the time to transmit */
while(now < time2transmit)
   now = get current system time;
time2transmit = now;
publishing motion command;
wait until next task activation;

4. Performance Evaluation

In this section, we present experimental results to verify the effectiveness of our proposed heuristic. We performed a series of experiments in terms of the frame inter-arrival times observed at motor drives, T i drv ( k ) = M i drv ( k ) M i 1 drv ( k ), for various configurations, i.e., different types of controller, the control period Tcycle, and the number of motor drives N. A safe and reliable phase offset factor δp was obtained by pre-runtime analysis on a testbed.

4.1. Experimental Setup

The performance evaluation was conducted on an EtherCAT-based MCS testbed which is compromised of a motion controller, multiple homogeneous motor drives, and external measurement equipments as shown in Figure 7a. For the comparison of different implementations of motion controller, we used three different types of motion controller: a high-end industrial PC (IPC), a low-end embedded single board computer (SBC) board, and a commercial EtherCAT master controller. The IPC and embedded SBC were constructed solely from the open-source components such as Xenomai real-time patch for Linux [22] and IgH EtherCAT master stack [23]. Table 1 summarizes the details of our testbed. As the test motion application, we used a point-to-point (PTP) movement application in which each motor drive repeats to move from base position to target position and vice versa. Figure 7b depicts the state transition of the test application.

In order to reduce potential interferences caused by kernel-level tasks, open-source based controllers, e.g., IPC and embedded SBC, were configured to start with single-user mode and operate without X-windows desktop manager. In addition, the modified network device driver and NEON floating point unit (FPU) compiler option proposed in our previous work [24] are used for embedded SBC as the platform optimization. For the motor drive platform, we chose TI industrial communication engine (ICE) board and configured the drives to use identical process data object (PDO) data set for cyclic synchronous position (CSP) mode operation. The frame arrival time observed at the motor drives, M i drv ( k ), was measured by logging GPIO output signals generated at the time when an EtherCAT frame arrives. The GPIO signals were collected by using an external data acquisition (DAQ) board with ns resolution timestamp. The statistical analysis on the measured samples was conducted using a LabVIEW application.

4.2. Analysis of the Behavior of Motion Control Task

As described in Section 3.2, we need to obtain pre-runtime analysis results to determine a safe phase offset Φp for the publish phase. Recall that the lower bound of Φp, Φ p , is determined by the worst-case ( J i + R i comp ) and the upper bound of Φp, Φ p +, is determined by the terms max ( | J i | ) and Drtt, respectively. Thus, we measured the Ji and R i comp values of the control task τ and analyzed them on open-source based controllers.

Table 2 shows the measurement results of Ji, R i comp, and M i con as we vary the length of control cycle Tcycle and the number of motor drives N. The M i con values, which are the start times of publish phase without heuristic, were also measured and analyzed for the comparison with the results of our heuristic later in Section 4.3. For each experiment, we conducted the measurement for 5 min, of which collected data amounts to 1,200,000 samples with Tcycle = 250 μs for instance. In the results, Δ denotes the absolute difference between the maximum and minimum value for each term.

According to Table 2, when we use high-end IPC controller, the task release jitter Ji values are quite stable for varying Tcycle and N. By contrast, Ji for low-end embedded SBC tends to increase as we increase the control cycle Tcycle. It is considered that the cache pollution due to other background tasks and kernel activities is enlarging the release jitter of control task as pointed out in previous works [9,19]. As for R i comp, its average and deviation depend on both Tcycle and N when we use embedded SBC. Regardless of the controller used, the average of R i comp increases with respect to N. This is because the feedback data size is linearly increased with N and the motion computation is performed for each axis, i.e., drive, and thus the execution time for the retrieve phase and computation phase is monotonically increased. On the other hand, the deviation of R i comp using embedded SBC decreases significantly for shorter Tcycle with a fixed N. It is considered that, with shorter Tcycle, there are less probabilities for the control task to be exposed to the interferences from kernel-level activities and cache miss handling overhead will also be reduced. As for M i con, IPC and embedded SBC show different characteristics. With IPC controller, the deviation and absolute difference values of M i con are quite stable for all the experimental configurations. However, with embedded SBC, we observe that the values increase with respect to Tcycle and N, which indicates the deviation of M i con in embedded SBC is affected by both Ji and R i comp.

Figure 8 illustrates the cumulative distribution function (CDF) of measured M i con values. As mentioned earlier, the sum of Ji and R i comp determine M i con and the absolute difference Δ of Ji and R i comp primarily contributes to the deviation of M i con. This characteristic is more apparent for embedded SBC as shown in Figure 8. In summary, we expect from the pre-runtime analysis results that our heuristic will be more beneficial to low-end embedded SBC with highly fluctuating message release interval.

From the measurement results given in Table 2, we can obtain the range of safe phase offset factor δp as given in Equation (8). Since Drtt is the function of N and other parameters are known a priori, it can also be easily computed. Table 3 summarizes the phase offset factors normalized with respect to Tcycle when we vary the number of motor drives from 1 to 8 and the control cycle Tcycle from 250 μs to 1000 μs, respectively. It is noticeable that we cannot determine the safe phase offset factor for embedded SBC with Tcycle = 250 μs. This is because the estimated Φ p and Φ p + for the case were 94% and 84%, respectively, which violates Constraint (3) in Section 3.2. It is noteworthy that our pre-runtime analysis can also be used to estimate the achievable MCT for a given configuration.

4.3. Frame Inter-Arrival Time Observed at Motor Drives

The objective of our heuristic is to keep the differences between frame inter-arrival times observed at the motor drives, T i drv ( k ), and the control cycle Tcycle as small as possible for deterministic isochronous control. To verify the effectiveness of our approach, we conducted a set of experiments to measure T i drv ( k ) for different controller implementations and compared the results. In each experiment, the measurement was performed for 30 min, for instance, of which collected data size amounts to 3,600,000 samples with Tcycle = 500 μs. Since we used homogeneous motor drives and the switching and propagation delay of frames is very small and bounded, we can regard the values observed at the first motor drive, T i drv ( k ), as the representative case without loss of generality. For performance evaluation, we define additional performance indicators, ϵ1% and ϵ10%. The parameters ϵ1% and ϵ10% account for the number of intervals between successive frame arrivals, of which length exceeding 1% and 10% with respect to a given Tcycle, respectively.

We first measured and compared the frame inter-arrival times for various testbed configurations. The configurations were varied according to Tcycle, N, controller hardware types, and whether to use our heuristic or not. For the experiments with our heuristic, we chose the δ p max values shown in Table 3. The performance evaluation results are summarized in Table 4 and Figure 9. The boxes in the Figure 9 represent the range in which 99% of measured samples are located. The lower and upper whiskers in the plots refer to the low 0.5% and low 99.5% value shown in Table 4.

As shown in Table 4, IPC greatly outperforms embedded SBC in terms of the deviation of frame inter-arrival time jitters without our heuristic. For all Tcycle and N configurations used, IPC controller has zero ϵ10% and much smaller ϵ1% than embedded SBC. However, after applying our heuristic, we can observe that embedded SBC controller shows performances comparable to IPC for all Tcycle and N. For instance, ϵ10% is decreased to zero and ϵ1% is reduced by up to 89% compared with the case without our heuristic where Tcycle = 500 μs and N = 8.

We conducted another set of experiments to observe the performances according to δp shown in Table 3. Figure 10 illustrates the distributions of frame inter-arrival time for δ p min, δ p med, and δ p max. Compared with the results in Figure 9, our heuristic can greatly reduce the deviation of frame inter-arrival time irrespective of δp used where the 99% of frame inter-arrival times have jitters less than ±8 μs for all N. However, we can also notice that the advantage of our heuristic is much more outstanding for δ p max compared with δ p min. This is because δ p max is computed with relatively deterministic values Drtt and J i . Hence, we can state that it is recommended to use δ p max for best performance with our heuristic.

We also compared the performance of embedded SBC and that of a commercial EtherCAT master controller, which has similar hardware specification. All experimental configurations are the same as previous experiments, except for the implementation of motion control task. We used TwinCAT-2 NC PTP runtime to implement a motion application for the commercial controller. Figure 11 shows the performance evaluation results. The results demonstrate that embedded SBC with our heuristic outperforms the commercial controller for all the test configurations used. For Tcycle = 1000 μs, the commercial controller has good performance in average because the 99% of measured samples are located in 24.9 μs, 29.4 μs, and 35.7 μs, which amount to 2%–3% with respect to Tcycle for varying N. However, for a smaller control cycle Tcycle = 500 μs, the performance degradation of the commercial controller is noticeable with increasing N in terms of absolute difference Δ. By contrast, the evaluation results of our approach show quite stable distribution for all the test configurations. Although the commercial controller used in the experiment may not be fully optimized, the results imply that our approach can achieve performance comparable to a commercial solution.

We now consider the usefulness of our approach for guaranteeing the synchronicity among multiple motor drives. For the purpose, we measured the frame inter-arrival times observed at different motor drives by connecting GPIO pins of corresponding motor drives to the probes of oscilloscope. In the experiment, we used embedded SBC for the motion controller. Figure 12 demonstrates the measurement results of T i drv ( 1 ) and T i drv ( 8 ) with Tcycle = 500 μs and N = 8. The observations from the measurement results are as follows. First, although the measurements were conducted for the first drive and the last drive in the control network, the characteristics of frame inter-arrival times observed at those drives are similar to each other. This means that the communication delays and frame switching overheads are highly deterministic throughout the control network regardless of the relative position of a drive. Second, the inter-arrival times with our heuristic performs similar to the case without heuristic in average. However, the absolute jitter of frame inter-arrival times has been dramatically reduced after applying our heuristic. This characteristic makes it easy to determine a safe clock shift time for accurate clock-based synchronization described in Section 2.2, and thus contributes to higher synchronicity of coordinated motion.

5. Conclusions

In modern networked motion control systems, guaranteeing isochronous control is very crucial for higher precision and synchronicity of target motion application. This paper proposed a simple and efficient heuristic approach to guarantee isochronous control by adjusting message transfer phase offset based on pre-runtime analysis. We also made it easy to apply phase offset adjustment code to the auto-generated motion application runtime code by integrating the pre-runtime analysis stubs and user-controlled phase offset with an open-source motion application IDE. Our proposed heuristic can also be effectively used to determine a safe shift time of global clock for accurately coordinated motion of multiple motor drives.

For in-depth and practical evaluation, we conducted extensive measurements for various configurations on a realistic testbed that was constructed from open-source software and COTS hardware platforms. The performance evaluation results show that our heuristic can bound successive actuation jitters within 1%–2% of the control cycle time even on a low-end embedded processor without any kernel-level modification and customized hardware. We also observed that our approach can provide performances comparable to or even better than that of a commercial controller.

In our future research, we will study how to extend our heuristic to support lightweight on-line adjustment. We also plan to extend our model for multiple control tasks with heterogeneous periods to provide higher precision and synchronicity of complex control under heavy interfering workloads such as monitoring and logging.


This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2013R1A1A2004984).

Author Contributions

Ikhwan Kim and Taehyoun Kim conceived the idea and wrote the paper. Taehyoun Kim elaborated the problem formulation. Ikhwan Kim implemented the experimental testbed and performed the performance evaluation.

Conflicts of Interest

The authors declare no conflict of interest.


  1. Rajkumar, R.; Lee, I.; Sha, L.; Stankovic, J. Cyber-Physical systems: The Next Computing Revolution. Proceedings of the ACM/IEEE Design Automation Conference (DAC), Anaheim, CA, USA, 13–18 June 2010; pp. 731–736.
  2. Mahalik, N.P.; Nambiar, A.N. Trends in food packaging and manufacturing systems and technology. Trends Food Sci. Technol. 2010, 21, 117–128. [Google Scholar]
  3. Giam, T.S.; Tan, K.K.; Huang, S. Precision coordinated control of multi-axis gantry stages. ISA Trans. 2007, 46, 399–409. [Google Scholar]
  4. Ramesh, R.; Jyothirmai, S.; Lavanya, K. Intelligent automation of design and manufacturing in machine tools using an open architecture motion controller. J. Manuf. Syst. 2013, 32, 248–259. [Google Scholar]
  5. Danielis, P.; Skodzik, J.; Altmann, V.; Schweissguth, E.B.; Golatowski, F.; Timmermann, D.; Schacht, J. Survey on Real-Time Communication Via Ethernet in Industrial Automation Environments. Proceedings of the IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Barcelona, Spain, 16–19 September 2014; pp. 1–8.
  6. Jasperneite, J.; Schumacher, M.; Weber, K. Limits of Increasing the Performance of Industrial Ethernet Protocols. Proceedings of the IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Patras, Greece, 25–28 September 2007; pp. 17–24.
  7. Vitturi, S.; Peretti, L.; Seno, L.; Zigliotto, M.; Zunino, C. Real-time Ethernet networks for motion control. Comput. Stand. Interface 2011, 33, 465–476. [Google Scholar]
  8. Robert, J.; Georges, J.-P.; Rondeau, E.; Divoux, T. Minimum Cycle Time Analysis of Ethernet-Based Real-Time Protocols. Int. J. Comput. Commun. 2012, 7, 743–757. [Google Scholar]
  9. Sung, M.; Kim, I.; Kim, T. Toward a Holistic Delay Analysis of EtherCAT Synchronized Control Process. Int. J. Comput. Commun. 2013, 8, 608–621. [Google Scholar]
  10. Kim, K.; Sung, M.; Jin, H. Design and Implementation of a Delay-Guaranteed Motor Drive for Precision Motion Control. IEEE Trans. Industr. Inform. 2012, 8, 351–365. [Google Scholar]
  11. Cena, G.; Bertolotti, I.C.; Scanzio, S.; Valenzano, A.; Zunino, C. Evaluation of EtherCAT Distributed Clock Performance. IEEE Trans. Industr. Inform. 2012, 8, 20–29. [Google Scholar]
  12. Felser, M. Real-Time Ethernet-Industry Prospective. IEEE Proc. 2005, 93, 1118–1129. [Google Scholar]
  13. Jansen, D.; Buttner, H. Real-time Ethernet—The EtherCAT solution. Comput. Control Eng. 2004, 15, 16–21. [Google Scholar]
  14. Marti, P.; Fuertes, J.M.; Fohler, G.; Ramamritham, K. Jitter Compensation for Real-Time Control Systems. Proceedings of the IEEE International Real-Time Systems Symposium (RTSS), London, UK, 3–6 December 2001; pp. 39–48.
  15. Buttazzo, G.; Cervin, A. Comparative Assessment and Evaluation of Jitter Control Methods. Proceedings of the International Conference on Real-Time and Network Systems (RTNS 2007), Nancy, France, 29–30 March 2007; pp. 163–172.
  16. Maruyama, T.; Yamada, T. Hardware Acceleration Architecture for EtherCAT Master Controller. Proceedings of the IEEE International Workshop on Factory Communication Systems (WFCS), Lemgo, Germany, 21–24 May 2012; pp. 223–232.
  17. Orfanus, D.; Indergaard, R.; Prytz, G.; Wien, T. EtherCAT-based Platform for Distributed Control in High-Performance Industrial Applications. Proceedings of the IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Cagliari, Italy, 10–13 September 2013; pp. 1–8.
  18. Cereia, M.; Bertolotti, I.C.; Scanzio, S. Performance of a Real-Time EtherCAT Master Under Linux. IEEE Trans. Indstr. Inform. 2011, 7, 679–687. [Google Scholar]
  19. Cereia, M.; Scanzio, S. A User Space EtherCAT Master Architecture for Hard Real-Time Control Systems. Proceedings of the IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Krakow, Poland, 17–21 September 2012; pp. 1–8.
  20. Kim, I.; Kim, T.; Sung, M.; Tisserant, E.; Bessard, L.; Choi, C. An Open-source Development Environment for Industrial Automation with EtherCAT and PLCopen Motion Control. Proceedings of the IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Cagliari, Italy, 10–13 September 2013; pp. 1–4.
  21. Lee, S.; Jin, H.; Kim, K. A Simulation Tool for Optimal Phasing of Nodes Distributed over Industrial Real-Time Networks. Proceedings of 4th International Workshop on Analysis Tools and Methodologies for Embedded and Real-time Systems (WATERS 2013), Paris, France, 9 July 2013; pp. 13–18.
  22. Xenomai | Real-Time Framework for Linux. Available online: (accessed on 15 April 2015).
  23. IgH EtherCAT Master for Linux. Available online: (accessed on 15 April 2015).
  24. Kim, C.; Kim, I.; Kim, T. Xenomai-based Embedded Controller for High-Precision, Synchronized Motion Applications. Korea Inst. Inf. Sci. (KIISE) Trans. Comput. Pract. 2015, 21, pp. 173–182. Available online: (accessed on 22 April 2015). [Google Scholar]
Figure 1. An example of EtherCAT-based motion system with 6-DOF robot application.
Figure 1. An example of EtherCAT-based motion system with 6-DOF robot application.
Sensors 15 13945f1 1024
Figure 2. Two synchronization modes in the EtherCAT slave.
Figure 2. Two synchronization modes in the EtherCAT slave.
Sensors 15 13945f2 1024
Figure 3. System model.
Figure 3. System model.
Sensors 15 13945f3 1024
Figure 4. Our proposed heuristic. (a) Determine phase offset Φp for the publish phase; (b) Execution of motion control task with phase offset Φp.
Figure 4. Our proposed heuristic. (a) Determine phase offset Φp for the publish phase; (b) Execution of motion control task with phase offset Φp.
Sensors 15 13945f4 1024
Figure 5. Execution flow of pre-runtime analysis.
Figure 5. Execution flow of pre-runtime analysis.
Sensors 15 13945f5 1024
Figure 6. User interface of pre-runtime analyzer.
Figure 6. User interface of pre-runtime analyzer.
Sensors 15 13945f6 1024
Figure 7. Experimental setup. (a) Motion control system testbed; (b) State transition of PTP motion application.
Figure 7. Experimental setup. (a) Motion control system testbed; (b) State transition of PTP motion application.
Sensors 15 13945f7 1024
Figure 8. The CDF distribution of start time of publish phase in the Tcycle. (a) N = 8, Tcycle: 1000μs; (b) N = 8, Tcycle: 500μs.
Figure 8. The CDF distribution of start time of publish phase in the Tcycle. (a) N = 8, Tcycle: 1000μs; (b) N = 8, Tcycle: 500μs.
Sensors 15 13945f8 1024
Figure 9. Distribution of frame inter-arrival times for different controller configurations.
Figure 9. Distribution of frame inter-arrival times for different controller configurations.
Sensors 15 13945f9 1024
Figure 10. Frame inter-arrival times for different phase offset (δp).
Figure 10. Frame inter-arrival times for different phase offset (δp).
Sensors 15 13945f10 1024
Figure 11. Performance comparison between embedded SBC and commercial controller.
Figure 11. Performance comparison between embedded SBC and commercial controller.
Sensors 15 13945f11 1024
Figure 12. Frame inter-arrival times for different motor drives (Drive 1 and Drive 8) where Tcycle = 500 μs, N = 8.
Figure 12. Frame inter-arrival times for different motor drives (Drive 1 and Drive 8) where Tcycle = 500 μs, N = 8.
Sensors 15 13945f12 1024
Table 1. Specifications of testbed.
Table 1. Specifications of testbed.
Industrial PCIntel i7-620M 2.66 GHz (Single Core enabled.) 2 GB DDR3 Memory, 128 GB SSD Realtek RTL 8139D 100 Mbps EthernetOperating Systems Automation SoftwareLinux 3.8.13, Xenomai 2.6.3 Beremiz 1.1, IgH EtherCAT Master 1.5.2

Embedded SBCTI AM3358 ARM Cortex-A8 1 GHz 512 MB DDR3 Memory, 2 GB eMMC TI cpsw 100 Mbps Ethernet

Commercial EPCCortex-A8 1 GHz 1 GB DDR3 Memory, 512 MB microSD 100 Mbps EthernetOperating Systems Automation SoftwareMS Windows Embedded Compact 7 TwinCAT 2 NC PTP runtime

EtherCAT DrivesTI AM 3359 ICE board 8 E.A 22 bytes for each drive (RXPDO:11 bytes, TXPDO:11 bytes)FirmwareTI sysbios sdk 1.0.9

DAQ moduleNational Instrument USB-6356, 1.25 MS/s/ch, 32 MS memoryN / AN/A

OscilloscopeTektronix DPO 3014, 4 ch, 2.5 GS/sN/AN/A
Table 2. Measurement results of Ji, R i comp, and M i con for different controllers.
Table 2. Measurement results of Ji, R i comp, and M i con for different controllers.
ControllerEmbedded SBCIPC
Tcycle1000 μs500 μs250 μs1000 μs500 μs250 μs
Ji (μs)




st.d (σ)

diff. (Δ)41.754.450.649.638.234.336.031.523.

R i comp (μs)




st.d (σ)

diff. (Δ)101.7102.8134.3200.068.866.191.2105.640.436.543.731.88.010.512.310.78.09.710.611.38.810.312.611.3

M i con (μs)




st.d (σ)

diff. (Δ)113.4134.9166.9222.193.988.3104.2122.549.847.254.932.716.818.725.621.420.321.821.519.619.121.820.226.1
Table 3. Normalized phase offset factor δp for Embedded SBC and IPC.
Table 3. Normalized phase offset factor δp for Embedded SBC and IPC.

δ p (%) δ p min δ p med δ p max δ p min δ p med δ p max δ p min δ p med δ p max δ p min δ p med δ p max
Embedded SBCTcycle = 1000 μs165797205896286296436994

Tcycle = 500 μs276195316394446892657789

Tcycle = 250 μs386592476991687888N/AN/AN/A

IPCTcycle = 1000 μs552986529895396145595

Tcycle = 500 μs95396125495165593275990

Tcycle = 250 μs175593235892326088536780
Table 4. Comparison of frame inter-arrival times for different controller configurations.
Table 4. Comparison of frame inter-arrival times for different controller configurations.
Cont.Embedded SBCIPC

Tcycle1000μs500 μs1000 μs500 μs

Frame inter-arrival time, without heuristic




Low 0.5%983.6981.0975.4960.2486.3483.2481.0467.5993.7993.0993.3991.4494.4493.1493.9492.2

Low 99.5%1018.31021.61028.11048.8515.9519.3523.2542.41006.51006.91006.81008.4506.4506.5506.4507.5

st.d (σ)

diff. (Δ)165.1211.1255.5320.0139.2140.9163.9207.922.026.325.929.127.624.126.328.9

ϵ 1%74,33492,133233,6501,112,9191,406,0081,588,2741,794,8652,376,029522341563,21199,161185,027131,175298,757

ϵ 10%031712758168341371340300000000

Frame inter-arrival time, with heuristic applying δ max




Low 0.5%995.2994.9993.8991.9495.9495.8493.9493.4993.3993.3993.4991.1493.2493.3493.4492.6

Low 99.5%1004.71004.91006.21007.9504.4504.6505.5506.81006.41006.11006.21008.5506.5506.1506.1507.2

st.d (σ)

diff. (Δ)17.329.638.341.816.629.035.342.522.620.422.237.525.021.523.433.7

ϵ 1%0131028415,76510,725127,923260,307135233,813176,025147,529164,685234,918

ϵ 10%0000000000000000
Sensors EISSN 1424-8220 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top