We consider an asynchronous distributed system consisting of n processes, ∏ = {p₁, p₂, …, p_n}. Because the failure detector is running as a basic component in the node, one simple topology is considered, and we assume that each pair of processes is connected by a communication channel that can be used to send and receive messages. The type of failure is crash and channels are fair-lossy channels. No synchronized clock is assumed.

Heartbeat is a common method to implement failure detectors. The detection modules detect each other's status by sending heartbeat messages periodically at duration Δt_i. According to the different modes of implementation, there are two monitoring approaches: PUSH and PULL. For two processes p and q in system, where q is monitoring p, the two basic approaches are described in Figure 1.

Both of the approaches detect each other's status by sending out heartbeat messages periodically at duration Δt_i. The difference is, in PUSH, the monitored process p initiatively sends a periodical message “I am alive” to process q, informing q that p is still alive; while in PULL, process q sends a probing message “Are you alive?” to the monitored process p periodically. After receiving the query message, the monitored process p passively replies an “I am alive!” message to indicate its status. For traditional failure detectors based on timeout mechanism, an appropriate time-out value Δt_o needs to be set. If no response message is received after Δt_o, the monitored process will be suspected as a failure. Obviously, the PULL approach needs twice the number of messages to achieve the same performance, but this does not affect its scalability. However, PULL is an initiative detection method which launches detection only when needed, and it does not need the assumption of a global synchronization clock. This is very important for current complicate large-scale distributed applications. Therefore, PULL employed as the basic detection strategy in this paper.

One of the key factors that affect the performance of an accrual failure detector is the calculation method for sl(t). Whether the value of sl(t) can give an accurate description about the actual failure status of a process determines the detector's detection accuracy and delay, etc. In current implementations of the accrual failure detector, in order to improve the calculation precision for sl(t), we usually have to rely on the prediction of the arrival time of detection messages. An accurate prediction model will greatly increase the detector performance. Some examples of the estimation methods which are used most frequently are: estimating the arrival time of detection messages using the distribution probability of message delay, predicting possible transmission delay by a linear process based on learning, etc. These methods not only cause heavy computing and storage overhead but also are limited to specific distributed systems. For example, Avinash's prediction method based on exponential distribution is proposed according to the particular characteristics of the Facebook system. In order to find a prediction method with less overhead and better adaptability, we have observed transmission delays under two typical network conditions. The detection processes used in the experiment are located in Harbin, and the monitored processes are located in Beijing (China) and Pittsburgh (PA, USA) respectively. These two sets of experiments correspond to good (dataset 1 with an average delay of 82.1 ms) and poor (dataset 2 with an average delay of 1,297.8 ms) network conditions, respectively. We have observed for 24 h, respectively, and the results are shown in the figure below.

From Figure 2, we can see that in the two different network environments, transmission delay shows a continuity (in Figure 2(a), data is centralized on 50, 80 and 100 ms, and in Figure 2(b), data is centralized on 1,200 and 1,400 ms). Only a very small number of detection messages have a large deviated transmission delay due to network congestion or message loss, etc. Furthermore, from the statistical data in Figure 2(a), we can get: (1) P 0 [ ( delay i - delay i - 1 ) ≤ 0 ] = 73.1 % , i > 1

Even in Figure 2(b) for a poor network environment, has also reached 56.3%. Therefore, the transmission time delay_i for most detection messages is less than or close to the transmission time of previous message delay_i-1. delay_i-1 can be used as the predicted value for delay_i to support failure detection, which means the predicted value of the i-th detection message is prek_i = delay_i-1. This method does not cause overhead for modeling and recording a large amount of historical data, and it's adaptive to different network environments. However, we can see from P₀ that the accuracy of this method is not high, especially for the case of a poor network environment. Therefore, we refer to the evaluation method proposed by Jacobson [14] and add consideration of a safety margin to the predicted value: (2) margin i = margin i - 1 + α ( | prek i - 1 - delay i - 1 | - margin i - 1 ) , i > 1

Let α = 0.25, for data in Figure 2(a), we have P_m[delay_i ≤ delay_i-1 + margin_i-1] = 98.9%. For Figure 2(b), P_m has also reached 98.4%. Therefore, this new prediction method has greatly improved the prediction accuracy and met the needs for most failure detections. Based on this method, we have proposed the LA-FD failure detector.

LA-FD employs the PULL approach as the basic failure detection strategy. To simply the description, suppose the system consists of only two processes p and q, where q is monitoring p. The detection algorithm is shown in Figure 3.

Figure 3 shows that the LA-FD failure detector consists of a detection module and a query module. The detection module located on process q sends probing message mq_i to the monitored process q at interval Δt_i. After receiving the probing message mq_i, process p immediately replies an acknowledge message ma_i to indicate its status. Everytime after receiving the acknowledge message ma_i, process q needs to calculate the margin for the next detection message and records the transmission time of current detection message. When an upper application queries the detector, it will reply with a value of &rho:_qp. Then the upper application will set a threshold value P according to its own requirement for detection accuracy. When &rho:_qp > P, process p is suspected as failed.

The accuracy of accrual failure detector is usually affected by two main factors. One is detection delay, and lower detection delay will reduce the accuracy of detection results; the other is the threshold set by upper applications for the suspicion level sl, and higher threshold leads to higher accuracy. However, in comparative experiments, different implementations of accrual failure detector use different approaches to calculate sl and its threshold. For the same detection data, we have collected all the detection results and related data from different detectors under multiple sets of thresholds. The relationship between the average mistake rate (λ_M) and detection delay has been explored and the results are shown in Figure 4, where (a) and (b) represent the detection results for dataset 1 and dataset 2, respectively.

It's obviously from Figure 4 that LA-FD has demonstrated higher detection accuracy in both of the different network environments. Under the same accuracy requirement (mistake rate λ_M in Y-axis), LA-FD has lower detection delay. This point is more obvious under poor network conditions (Figure 4(b)). I-ϕ-FD based on exponential distribution has the worst detection performance in both of the network environments. so, the assumption of exponential distribution is only suitable for the specific P2P systems and normal distribution can describe the message transmission delay more accurately.

Since accrual failure detector needs to calculate detector parameters and maintain the history window of detection messages for each detection period, these two factors are the main reason for different system overhead in accrual detectors. A large history window will improve the prediction accuracy for model parameters and has a certain impact on the calculation accuracy for detector parameters. Meanwhile, the maintenance of history window will cause more overhead. For the experiments in this section, we have selected different historical window size settings and have made a detailed comparison of CPU utilization.

It can be seen from the Figure 5 that the CPU overhead is the heaviest in the ϕ-detector based on normal distribution and it grows the fastest as the window size changes. This is because the workload for calculating parameters of the normal distribution model is the most, and every time it needs the statistical data from the entire window. The overhead of LA-FD is the least (about 0.08%), and it isn't affected by window size. Each process in the experiment shown in Figure 5 only maintains five connections. In large-scale P2P systems, in order to maintain a high locating efficiency, each process is generally required to maintain logN (N is the number of processes in the system) connections. Therefore, the fact that LA-FD can reduce CPU overhead is more significant in real systems.