Next Article in Journal
Convolution of Barker and Golay Codes for Low Voltage Ultrasonic Testing
Previous Article in Journal
A Cold-Pressing Method Combining Axial and Shear Flow of Powder Compaction to Produce High-Density Iron Parts
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Control Limits for an Adaptive Self-Starting Distribution-Free CUSUM Based on Sequential Ranks

Graduate School of Excellence Computational Engineering, Technische Universität Darmstadt, Dolivostraße 15, 64293 Darmstadt, Germany
Technologies 2019, 7(4), 71; https://doi.org/10.3390/technologies7040071
Submission received: 7 August 2019 / Revised: 23 September 2019 / Accepted: 28 September 2019 / Published: 1 October 2019

Abstract

:
Since their introduction in 1954, cumulative sum (CUSUM) control charts have seen a widespread use beyond the conventional realm of statistical process control (SPC). While off-the-shelf implementations aimed at practitioners are available, their successful use is often hampered by inherent limitations which make them not easily reconcilable with real-world scenarios. Challenges commonly arise regarding a lack of robustness due to underlying parametric assumptions or requiring the availability of large representative training datasets. We evaluate an adaptive distribution-free CUSUM based on sequential ranks which is self-starting and provide detailed pseudo-code of a simple, yet effective calibration algorithm. The main contribution of this paper is in providing a set of ready-to-use tables of control limits suitable to a wide variety of applications where a departure from the underlying sampling distribution to a stochastically larger distribution is of interest. Performance of the proposed tabularized control limits is assessed and compared to competing approaches through extensive simulation experiments. The proposed control limits are shown to yield significantly increased agility (reduced detection delay) while maintaining good overall robustness.

1. Introduction

From a historical perspective, the advent of modern statistical process control (SPC) arose out of the post industrial revolution realization that to yield goods of acceptable quality a manufacturing process ought to operate within prespecified margins of error (in other words it ought to be stable or in control) [1]. In oversimplified terms, control charts are central to SPC and serve to continuously monitor a process to assess whether the observed deviations from the nominal process are due to mere chance (in control) or not (out-of-control) (see generally [1,2,3]).
Control charts were first introduced by W. A. Shewhart in 1924 and gained widespread popularity following the publication of Shewhart’s seminal monograph [4] in 1931. The following decades witnessed a substantial research interest and output resulting in important SPC developments including, but not limited to, cumulative sum (CUSUM) [5] and exponentially weighted moving average (EWMA) [6] control charts as well as Bayesian approaches [7,8,9]. Only the former will be considered here; the interested reader is referred to [1,2,3,10] for an exhaustive treatment of the subject matter and to [11,12,13] for a more concise overview.
Note that, as has been pointed out throughout the years by several prominent scholars [11,14,15], to this date and despite considerable advances in nonparametric approaches, most control charts remain based on the normality assumption. Despite its appeal, the normal distribution clearly is rarely an appropriate model for real-world applications. According to Stoumbos et al. [11] there exists a fundamental disconnect between practitioners and researchers as well as a gap between applied and theoretical research: “The existence of these gaps is disturbing, because it means that most practitioners have received little of the potential benefit from the technical advances made in SPC over the last half-century.” (see [11] at 993).
The present work aims to shrink the above-mentioned gap by providing ready-to-use tables of control limits for an adaptive self-starting distribution-free CUSUM suitable to a wide variety of applications where a process is monitored for a departure from the underlying sampling distribution to a stochastically larger distribution. While this procedure has previously briefly been outlined and used by this author in [16,17], respectively, it is first thoroughly proposed and assessed in the current work.
Following a review of some pertinent fundamentals in Section 2 we proceed by reviewing the adaptive distribution-free CUSUM, providing a simple, yet effective calibration algorithm and obtaining a set of control limits suitable for a wide variety of scenarios. The performance of the control limits obtained as outlined in Section 3 is then assessed through extensive simulation experiments, whose results are outlined and discussed in Section 4; it will be shown that the proposed control limits yield a significantly reduced detection delay while maintaining good overall robustness. Finally, our concluding remarks set out in Section 5 complete this work.

2. Parametric and Nonparametric Univariate CUSUM Control Charts

The following subsections concisely restate the parametric (normal) univariate CUSUM and McDonald’s sequential ranks CUSUM (SRC) [18]. All considerations will be limited to the most basic task of detecting a positive shift in the mean of a sequentially observed process using one-sided control charts.

2.1. Conventional Parametric CUSUM

Let F and G denote normal distributions given as F N ( μ 0 , 1 ) and G N ( μ 0 + δ , 1 ) . Furthermore, for the sake of simplicity, let μ 0 = 0 and δ = 1 . Consider observing a sequence of independent random variables { x n , n 1 } such that { x 1 , , x τ 1 } F and { x τ , x τ + 1 , } G , i.e., a distributional shift F G occurs at time instance τ . Assuming perfect knowledge of all parameters describing F and G (i.e., μ 0 and δ ) Page’s CUSUM [5] represents the gold standard change detection technique and can be computed sequentially as
C 0 = 0 , C n = max { 0 , C n 1 + x n k C } , n 1 .
The CUSUM signals, thereby declaring a distributional shift to have occurred, if
C n > h C ,
with h C and k C being the prespecified control limit and reference constant, respectively.
The CUSUM’s in-control average run length (ARL) is defined as the expected time until a change is signaled under F, i.e.,
A R L = E F inf { n > 0 : C n > h C }
.
Note that this is akin to a nominal type-I-error level in the realm of hypothesis testing and that hence the closeness of the actual in-control A R L to A R L 0 is commonly regarded as an indicator of the control chart’s robustness [15,19]. Accordingly, h C and k C are chosen such that A R L 0 is (at least approximately) attained when the observed process is in-control (see, e.g., [3,20]). It is well known that choosing k C = δ / 2 is optimal [21] (see also [22,23]).

2.2. Sequential Ranks CUSUM (SRC)

Consider again the sequence { x n , n 1 } ; the sequential rank of x n is defined as
R n = 1 + r = 1 n 1 x n x r + ,
where x + is 1 for x > 0 and 0 otherwise. The SRC is then
C SRC n = max { 0 , C SRC n 1 + R n n + 1 k SRC } , n 1 ,
with C SRC 0 = 0 and k SRC some reference constant. Akin to Equation (2), the SRC signals if C SRC n > h SRC .
A crucial advantage of the SRC stems from the fact that, given the observed process is in-control, it can be shown (see [18] and references therein) that the quantities R n n + 1 are independent and discrete uniform on { 1 n + 1 , 2 n + 1 , , n n + 1 } . Hence, in addition to the approach followed in [18], h SRC for a fixed k SRC can be obtained through a straightforward Monte Carlo procedure (see, e.g., [24] at 12) without requiring any historical training data.

3. Adaptive Control Limit SRC (AC-SRC)

As will be shown in detail in Section 4, the actual applicability of the SRC is often hampered due to its virtually unacceptable performance in certain scenarios. More specifically, the SRC suffers from a lack of agility, i.e., given a distributional shift actually occurred the SRC may require an undue amount of time to signal (i.e., it exhibits a large detection delay); as will be shown, this is especially pronounced if a change occurs soon after monitoring commenced. In such case, the amount of data gathered by the SRC may be grossly insufficient, thus resulting in a prolonged time to signal. It should be noted that, as McDonald correctly points out (see [18], pg. 628–629), the above mentioned lack of agility (compared to an optimal parametric approach) and the poor detection of changes occurring after a relatively small number of observations is to some degree inherent to all nonparametric procedures.
The idea behind the adaptive control limit SRC (AC-SRC) proposed by this author [16,17] is to mitigate the SRC’s drawbacks while maintaining its ease-of-use, robustness, and the ability to obtain generally valid control limits ahead of time. This is facilitated by the AC-SRC being inspired by and incorporating large parts of a distribution-free bootstrap based CUSUM proposed by Chatterjee and Qiu [19]. Said authors in 2009 proposed an elegant procedure where the conventional fixed control limit is swapped for a sequence of control limits obtained from the conditional distribution of the test statistic (i.e., the CUSUM) given the last time it was zero. Chatterjee and Qiu estimate these conditional distributions by means of bootstrapping; note that among other things this implies the need of a large amount of representative training data as well as a high computational burden. However, transferring the key idea of the approach by Chatterjee and Qiu to the SRC results in the AC-SRC described and analyzed in the following.
Akin to the SRC described in Section 2.2 let R n and C AC-SRC n denote the sequential rank of x n and the respective SRC as provided by Equations (4) and (5), respectively. Furthermore, let Y AC-SRC j be a random variable following the conditional distribution
Y AC-SRC j C AC-SRC n | T AC-SRC n = j ,
where T AC-SRC n , also referred to as sprint length, denotes the time elapsed since C AC-SRC n was last zero, i.e.,
T AC-SRC n = 0 if C AC-SRC n = 0 T AC-SRC n = j if C AC-SRC n 0 , , C AC-SRC n j + 1 0 , C AC-SRC n j = 0 ; j = 1 , , n .
Central to the method by Chatterjee and Qiu is the fact that the conditional distributions in Equation (6) depend only on j and F but not on n [19]. Then for any positive integer j max n the (unconditional) distribution of C AC-SRC n can be approximated by means of the conditional distributions in Equation (6) as
C AC-SRC n j = 1 j max Y AC-SRC j I T AC-SRC n = j + Y * I T AC-SRC n > j max ,
with I being the common indicator function and Y * C AC-SRC n | T AC-SRC n > j max . Since the AC-SRC is based on sequential ranks which, given the process is in control, are independent and discrete uniform on { 1 n + 1 , 2 n + 1 , , n n + 1 } (see Section 2.2) the sequence of control limits { h j } can be determined (ahead of time) without the need for training data by means of Monte Carlo simulations as outlined in Algorithm 1.
The AC-SRC then signals if T AC-SRC n = j and C AC-SRC n > h j for 1 j j max or if T AC-SRC n > j max and C AC-SRC n > h j max . Note that following recommendations by Chatterjee and Qiu the h j are only calculated up to a reasonably small j max after which, if the test statistic does not bounce back to zero, they are kept fixed at h j max . Furthermore, k AC-SRC is linked to j max such that a desired sprint length t E T n , which is set to be proportional to j max (see Section 3.1), e.g., as t E T n = 3 j max 4 , is approximately attained by the average sprint length. That is, k AC-SRC : T ¯ AC-SRC n t E T n = 3 j max 4 .
Algorithm 1: Adaptive Control Limit SRC (AC-SRC)
Technologies 07 00071 i001

3.1. Remarks on and Suggestions for the Selection of AC-SRC Parameters

The aim of this Section is twofold: first, to complete the description of the proposed procedure by justifying some seemingly completely arbitrary design choices of Section 3 , and second, to provide guidance to practitioners in order to facilitate the applicability of our method.
Average in-control and out-of-control run lengths, which are commonly referred to as A R L 0 and A R L 1 in the SPC literature, play a crucial role in the design and use of control charts. The A R L 0 characterizes the chart’s propensity to false alarms in terms of the average number of samples they are separated by, whereas A R L 1 describes what we referred to as the control chart’s agility, i.e., the average delay between the occurrence of an actual change and its detection. Clearly, then, there exists an inherent trade off between the objectives of low false alarm rates (large A R L 0 ) and small detection delays (small A R L 1 ). In this paper, we assume that the common approach of choosing an acceptable A R L 0 followed by attempts to minimize A R L 1 is pursued. The suitability of an A R L 0 highly depends on the particular problem at hand and is influenced, among other things, by crucial aspects such as weighting the aforementioned conflicting objectives to ensure compliance with requirements as well as detailed knowledge of the specific application. Accordingly, we find further discussions pertaining A R L 0 to be beyond the scope of this paper and, again, refer the interested reader to selected representatives of the established SPC literature [1,2,3].
Recall that, given a fixed and pre-determined j max , Algorithm 1 starts out by calibrating k AC-SRC such that the average sprint length T ¯ AC-SRC n equals the desired sprint length t E T n within a reasonable margin of error. Following Chatterjee and Qiu [19] ,we fix the desired sprint length t E T n as a in theory arbitrary ratio of j max ; throughout this work t E T n = 3 j max 4 is used. Note that, although the rationale for linking k AC-SRC and j max is compelling, doing so is not required.
The behavior of the AC-SRC’s test statistic C AC-SRC n is crucially influenced by the specific choice of k AC-SRC in that the propensity of C AC-SRC n bouncing back to zero decreases for smaller k AC-SRC (and vice versa for larger values of the reference constant). In other words, the average sprint length T ¯ AC-SRC n increases with a reduction of k AC-SRC , whereas increasing the reference value results in smaller sprint lengths. Furthermore, the sensible constraint of choosing t E T n j max restricts the computational burden of Algorithm 1 and reasonably ensures its algorithmic stability. In fact, in the absence of constraints on j max and k AC-SRC , `inappropriate’ combinations such as, e.g., (very) large k AC-SRC and j max could easily result in the inability to evaluate Equation (6) which, in turn, is required in Part II of Algorithm 1 . While we find the aforementioned to establish sufficient and convincing justification for the choice of calibrating k AC-SRC such that T ¯ AC-SRC n t E T n = 3 j max 4 holds, it is arbitrary in that other reasonable but not necessarily superior design choices are readily discernible (see [19]).
As expected, and consistent with the considerations expressed by Chatterjee and Qiu [19] pertaining to their bootstrap-based method, we observed diminishing returns with increasing the length j max of the sequence of adaptive control limits { h j } j = 1 j max .
While we are unable to provide specific guidelines pertaining to the selection of AC-SRC’s pertinent tuning parameters and further research in this area is required, we advocate the use of rather short sequences { h j } j = 1 j max with 6 j max 30 . Based on our current understanding and evidence, we recommend to set up the AC-SRC as discussed above and to adjust it to the requirements of the specific scenario by means of choosing either smaller or larger j max .
As will be corroborated by simulation results in Section 4.2, a reasonably consistent degree of fine-tuning is attainable with smaller j max allowing for good agility, whereas using slightly larger values for j max yields improved robustness at the expense of an increased detection delay.

4. Results and Discussion

4.1. Control Limits and Reference Values for the AC-SRC

Ready-to-use sets of reference constants k AC-SRC and respective sequences of control limits { h j } j = 1 j max for combinations of A R L 0 and j max have been determined following the calibration procedure described in Algorithm 1. Again we emphasize that the main contribution of this work is in providing practitioners with a wide choice of predetermined control limits to be used out-of-the-box without requiring any further adjustments.
We used values for k AC-SRC calibrated such that T ¯ AC-SRC n t E T n = 3 j max 4 and N AC-SRC = 5000 , B = 5 · 10 4 , B 1 = 5000 , Δ = 1 200 . All result were further averaged over 200 Monte Carlo runs.
To improve readability the tabularized sets of control limits { h j } j = 1 j max and reference values k AC-SRC for A R L 0 = { 100 , 200 , 300 , 370 , 400 , 500 , 600 , 700 , 800 , 900 , 1000 } and j max = { 6 , 8 , 10 , 12 , 14 , 16 , 18 } are deferred to Table A1, Table A2, Table A3, Table A4, Table A5, Table A6, Table A7, Table A8, Table A9, Table A10 and Table A11 in Appendix A. A R L 0 = 370 was included due to its popularity among practitioners, which stems from Shewhart x ¯ control charts using three-sigma limits having an in-control ARL of 370 (see generally [1]).

4.2. Performance Evaluation of the Proposed AC-SRC

To obtain an accurate representation of the proposed AC-SRC’s performance and put it into perspective we conducted simulation experiments to ascertain a control chart’s detection delay (DD), in-control ARL, and false alarm rate (FAR). A shift in the process distribution from F N 0 , 1 to G N 1 , 1 occurring at various time instances τ was simulated. FAR in this context refers to instances in which a particular control chart signaled although the actual shift at time instance τ had not occurred yet. Results were obtained for τ = { 10 , 20 , 30 , 40 , 50 } , j max = { 6 , 8 , 10 , 12 , 14 , 16 , 18 } , A R L 0 = { 100 , 500 , 1000 } and compared with optimal values for the parametric CUSUM (as provided in [3]) and the conventional SRC (as provided in [18]) for the respective A R L 0 as illustrated in Table 1. All results were averaged over 2 · 10 5 Monte Carlo runs.
Furthermore, the robustness of all three control charts to deviations from the normal distribution was assessed by simulating an impulsive noise environment through the use of a two component Gaussian mixture model, as is often done in related work (see [25], pg. 176; see also [26,27]). Accordingly, instead of F N 0 , 1 , the in-control are modeled as
F 1 η N 0 , 1 + η N 0 , κ
with 0 η 1 expressing the probability that contamination with the heavy-tailed component modeled using κ 1 occurs. Thus, again, at time instance τ a shift in distribution from F 1 η N 0 , 1 + η N 0 , κ to G 1 η N 1 , 1 + η N 1 , κ occurs. All reported results claiming impulsive noise contamination were obtained using η = 0.1 and κ = 100 .

4.2.1. Performance under Normality

Table 2, Table 3 and Table 4 show results of simulation experiments as outlined in Section 4.2 for the normal use case, i.e., a shift in distribution from F N 0 , 1 G N 1 , 1 occurs at time instance τ .
Clearly the parametric CUSUM’s exceptional performance comes as no surprise considering its optimality if, as is the case here, the monitored process is actually Gaussian. Questions of greater interest concern whether or not a substantial performance difference between the SRC and the proposed AC-SRC can be observed.
Our qualitative assessment of performance differences will focus on differences among the examined control charts pertaining to:
  • Detection delay (DD)
    -
    One of if not the major objective in practical applications is to detect a change as quickly as possible; hence, DD should be small (see also Section 3.1).
  • Average run length (ARL)
    -
    Recall that the ARL describes the average time or run length until the control chart signals under in-control conditions, i.e., without a change having occurred. The ARL is, loosely speaking, akin to the type-I error level in hypothesis testing. Rather than setting a false alarm rate control charts are typically designed by choosing a desired A R L 0 . The actual in-control ARL determined in our simulation experiment should be reasonably close to the nominal A R L 0 and we interpret this closeness as indicating the control chart’s robustness.
  • False alarm rate (FAR)
    -
    Moreover, recall that even if the monitored process is in-control any CUSUM chart will eventually signal. Clearly there is a relation between FAR and ARL; however, since said relation and false alarm properties of CUSUMs in general are neither well explored nor straightforward, especially for rather small ARLs, a discussion is deemed beyond the scope of this work. The interested reader is referred to, e.g., [28]. Coming back to the issue at hand, as fas as our performance assessment is concerned FAR values should be as small as possible (ideally zero).
Examining the entries of Table 2, Table 3 and Table 4 it can generally be observed that the proposed AC-SRC performs well, especially keeping in mind that the error margins allowed for in Algorithm 1 (namely up to 5 % deviation from A R L 0 ) are fairly relaxed and could easily be tightened at the expense of an increased computational burden. Still, the proposed AC-SRC more often than not outperforms the conventional SRC in all aspects. More specifically, it is offers substantial benefits especially for larger ARLs and small to medium τ .
However, it ought also be pointed out that the AC-SRC does struggle to outperform the conventional SRC for A R L 0 = 100 as shown in Table 2. Its overall performance however still appears acceptable. A likely cause stems from the fact that the sequential ranks approximation by an independent uniformly distributed random variable strictly speaking only holds asymptotically and convergence appears to be somewhat slow. Note that despite a deviation of up to 5 % was allowed in the determination of the AC-SRC control limits the AC-SRC’s actual ARL is remarkably close to A R L 0 and the FARs are consistently lower than both C and SRC. Finally, focusing on Table 3 and Table 4 it is evident that the AC-SRC indeed results in an increased agility, as evidenced by substantially reduced detection delays.

4.2.2. Performance under Impulsive Noise Contamination

The second part of our performance analysis focused on assessing the performance of C, SRC, and AC-SRC when subjected to impulsive noise contamination (as described in Section 4.2). Recall that at time instance τ the distributional shift now occurs from F 1 η N 0 , 1 + η N 0 , κ to G 1 η N 1 , 1 + η N 1 , κ with η = 0.1 and κ = 100 .
The breakdown of the parametric CUSUM is hardly surprising and not worthy of further discussion; rather the interest lies in whether or not the benefits shown by the SRC in the uncontaminated use case persist if the underlying process is heavier tailed. Examining Table 5, Table 6 and Table 7 we answer in the affirmative. More specifically, while a slight increase in both DD and ARL deviation is observed, all material arguments raised in Section 4.2.1 apply mutatis mutandis.
In conclusion, we would like to re-emphasize the reasonably consistent degree of fine-tuning attainable by means of sensibly choosing j max , wherein smaller j max yield reduced detection delays at the expense of an increased type-I error rate, whereas larger j max result in improved robustness and decreased agility.

5. Conclusions

In the present work we evaluated an adaptive self-starting distribution-free CUSUM based on sequential ranks and for the first time provided detailed pseudo-code of a simple, yet effective calibration algorithm. The main original contribution of this work, however, is in providing precomputed control limits and reference values for a wide variety of AC-SRC configurations, thus allowing practitioners to apply the procedure off-the-shelf without further adjustments and irrespective of the data generating model underlying their specific use case. Performance and robustness of the proposed tabularized control limits were assessed and compared to both parametric CUSUM and conventional SRC through extensive simulation experiments. While far from optimal, we were able to show that the proposed control limits result in a substantially decreased detection delay, while maintaining good overall robustness properties and allowing for easy and intuitive fine-tuning.

Acknowledgments

The work of M.L. was supported by the ’Excellence Initiative’ of the German Federal and State Governments and the Graduate School of Excellence Computational Engineering at Technische Universität Darmstadt. The views expressed in this article are solely those of the author in his private capacity and do not necessarily reflect the views of Technische Universität Darmstadt or any other organization.

Conflicts of Interest

The author declares no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:
AC-SRCAdaptive Control Limit SRC
ARLAverage Run Length
CUSUMCumulative Sum Control Chart
DDDetection Delay
EWMAExponentially Weighted Moving Average Control Chart
FARFalse Alarm Rate
SPCStatistical Process Control
SRCSequential Ranks CUSUM

Appendix A. Tabularized AC-SRC Control Limits and Reference Values

Table A1. Adaptive control limit SRC (AC-SRC) for A R L 0 = 100 .
Table A1. Adaptive control limit SRC (AC-SRC) for A R L 0 = 100 .
ARL 0 100100100100100100100
j max 681012141618
k AC-SRC 0.54860.53180.52670.52090.51800.51420.5131
h 1 0.41680.42740.42500.42470.42210.41990.4165
h 2 0.84870.84100.83310.83080.82510.82090.8144
h 3 1.20131.20801.20121.19101.17751.16971.1598
h 4 1.49611.50561.48851.47651.46271.45101.4378
h 5 1.74701.76051.73951.72831.71101.69841.6826
h 6 1.96641.98251.96521.95421.93751.92331.9053
h 7 2.18592.16752.15582.13882.12232.1042
h 8 2.37412.35202.34372.32442.30892.2895
h 9 2.52702.51672.49702.48182.4627
h 10 2.68862.68072.66322.64672.6268
h 11 2.83592.81572.80162.7772
h 12 2.98032.96382.94732.9238
h 13 3.10353.08653.0624
h 14 3.23513.22163.1945
h 15 3.34973.3252
h 16 3.47183.4474
h 17 3.5652
h 18 3.6794
Table A2. AC-SRC for A R L 0 = 200 .
Table A2. AC-SRC for A R L 0 = 200 .
ARL 0 200200200200200200200
j max 681012141618
k AC-SRC 0.54860.53180.52690.52070.51800.51450.5130
h 1 0.44090.45570.45520.45860.45700.45700.4549
h 2 0.89640.91730.90950.91120.90710.90700.9020
h 3 1.28751.30831.30541.30491.29421.29251.2848
h 4 1.61391.63661.62571.62651.61551.61051.6011
h 5 1.89111.92011.90761.91361.90151.89811.8865
h 6 2.13452.17152.16162.16572.15532.14882.1392
h 7 2.39882.38782.39472.38372.37912.3673
h 8 2.60872.59632.60872.59302.59042.5795
h 9 2.79192.80332.79032.78842.7725
h 10 2.97162.98822.97662.97252.9578
h 11 3.16403.14683.14873.1324
h 12 3.32813.31233.31233.2966
h 13 3.46913.47093.4557
h 14 3.62053.62273.6075
h 15 3.77013.7512
h 16 3.90813.8927
h 17 4.0295
h 18 4.1614
Table A3. AC-SRC for A R L 0 = 300 .
Table A3. AC-SRC for A R L 0 = 300 .
ARL 0 300300300300300300300
j max 681012141618
k AC-SRC 0.54900.53180.52660.52050.51800.51430.5129
h 1 0.45340.46780.46660.47060.46960.47130.4696
h 2 0.93740.94820.94450.95190.94860.95260.9496
h 3 1.34561.37141.36111.36051.35081.35531.3496
h 4 1.69101.72551.71381.71481.70491.70731.7001
h 5 1.98802.03072.01572.02192.00842.01032.0058
h 6 2.24702.29752.28222.29052.27762.28362.2728
h 7 2.53982.52602.53602.52302.52902.5178
h 8 2.76412.74922.75982.74762.75392.7459
h 9 2.95592.96892.95862.96452.9548
h 10 3.14763.16763.15023.16383.1522
h 11 3.35053.33673.34943.3405
h 12 3.52723.51363.52923.5158
h 13 3.68293.69863.6892
h 14 3.83983.86023.8491
h 15 4.01774.0039
h 16 4.16624.1569
h 17 4.2997
h 18 4.4396
Table A4. AC-SRC for A R L 0 = 370 .
Table A4. AC-SRC for A R L 0 = 370 .
ARL 0 370370370370370370370
j max 681012141618
k AC-SRC 0.54890.53160.52670.52080.51780.51440.5130
h 1 0.48220.50020.47470.47730.47640.47850.4744
h 2 0.98301.00850.95220.95390.95050.95450.9527
h 3 1.42091.45161.37581.37101.36001.36161.3602
h 4 1.78701.82261.72241.72511.71611.72101.7170
h 5 2.10022.14902.03372.03572.02462.03022.0252
h 6 2.37892.43352.30322.30662.29512.30162.2936
h 7 2.69232.54942.55542.54232.55122.5462
h 8 2.93092.77652.78092.76902.77882.7749
h 9 2.98732.99482.98192.99362.9876
h 10 3.18383.19343.17993.19363.1881
h 11 3.38063.36853.38373.3782
h 12 3.55653.54723.56003.5547
h 13 3.71323.73533.7295
h 14 3.87743.89953.8938
h 15 4.06024.0521
h 16 4.20784.1982
h 17 4.3509
h 18 4.4898
Table A5. AC-SRC for A R L 0 = 400 .
Table A5. AC-SRC for A R L 0 = 400 .
ARL 0 400400400400400400400
j max 681012141618
k AC-SRC 0.54860.53160.52670.52060.51800.51420.5129
h 1 0.49350.51210.48170.48260.47850.48090.4794
h 2 1.01091.03460.97180.97060.96070.96470.9619
h 3 1.46321.48701.39731.39111.37221.37791.3723
h 4 1.83881.87351.75361.74911.72971.73481.7287
h 5 2.16092.21052.07282.06672.04362.04752.0394
h 6 2.44702.50562.34802.34292.31702.32262.3141
h 7 2.77292.60012.59292.56642.57442.5657
h 8 3.01452.83072.82722.79922.80612.7965
h 9 3.04493.04243.00843.02143.0094
h 10 3.24453.24253.21143.22613.2125
h 11 3.43143.39863.41693.4044
h 12 3.61143.58313.60123.5891
h 13 3.75423.77283.7589
h 14 3.91803.93743.9242
h 15 4.09844.0863
h 16 4.25274.2403
h 17 4.3870
h 18 4.5315
Table A6. AC-SRC for A R L 0 = 500 .
Table A6. AC-SRC for A R L 0 = 500 .
ARL 0 500500500500500500500
j max 681012141618
k AC-SRC 0.54850.53140.52650.52080.51820.51420.5131
h 1 0.52080.54400.51220.50590.49000.49090.4846
h 2 1.07881.10471.03721.02410.99140.99330.9804
h 3 1.55731.59531.49671.47201.42301.42511.4062
h 4 1.96572.01811.88981.85641.79221.79191.7678
h 5 2.31542.38032.23432.19822.12222.12182.0951
h 6 2.62252.70342.53562.49452.41042.41192.3772
h 7 2.99262.80892.76672.67182.66982.6350
h 8 3.26113.05923.01232.91192.91292.8743
h 9 3.29053.24403.13903.13763.1000
h 10 3.50813.45863.35193.34833.3045
h 11 3.66073.54193.54623.5036
h 12 3.85163.73023.73893.6848
h 13 3.91123.92273.8699
h 14 4.08414.09284.0387
h 15 4.25564.2060
h 16 4.42434.3662
h 17 4.5165
h 18 4.6654
Table A7. AC-SRC for A R L 0 = 600 .
Table A7. AC-SRC for A R L 0 = 600 .
ARL 0 600600600600600600600
j max 681012141618
k AC-SRC 0.54860.53200.52680.52050.51810.51420.5132
h 1 0.54820.56800.53790.53780.51440.51500.4981
h 2 1.12951.16261.09531.09121.04111.04181.0075
h 3 1.63411.68431.58751.57111.49771.49691.4460
h 4 2.06442.13032.00311.99061.89631.89421.8294
h 5 2.43852.51022.36212.35012.24442.23882.1667
h 6 2.76232.84892.68822.67412.55302.55142.4636
h 7 3.15702.97892.96772.83232.82652.7325
h 8 3.43693.24533.23383.08983.08402.9809
h 9 3.49383.48083.32573.32653.2161
h 10 3.72113.71203.54703.54463.4283
h 11 3.93353.75953.76153.6345
h 12 4.14293.96413.96243.8278
h 13 4.14804.15124.0177
h 14 4.33124.33434.1938
h 15 4.51494.3631
h 16 4.68284.5280
h 17 4.6861
h 18 4.8414
Table A8. AC-SRC for A R L 0 = 700 .
Table A8. AC-SRC for A R L 0 = 700 .
ARL 0 700700700700700700700
j max 681012141618
k AC-SRC 0.54850.53180.52690.52080.51850.51460.5129
h 1 0.57290.59810.56550.56540.53990.54080.5253
h 2 1.17721.22011.14951.14591.09421.09561.0609
h 3 1.70631.76441.66411.65211.56921.57071.5224
h 4 2.15572.23362.10342.08971.99041.98901.9284
h 5 2.54592.63852.48552.47052.35382.35012.2775
h 6 2.88702.99662.82562.81172.67762.67682.5981
h 7 3.32493.13133.11672.97222.97052.8835
h 8 3.61613.41513.40053.24273.24363.1445
h 9 3.67343.66243.49193.49263.3935
h 10 3.91413.90493.72633.73233.6206
h 11 4.13813.94653.95073.8318
h 12 4.35654.15834.16544.0471
h 13 4.36334.36714.2385
h 14 4.54954.56404.4302
h 15 4.74624.6082
h 16 4.92264.7855
h 17 4.9518
h 18 5.1086
Table A9. AC-SRC for A R L 0 = 800 .
Table A9. AC-SRC for A R L 0 = 800 .
ARL 0 800800800800800800800
j max 681012141618
k AC-SRC 0.54870.53160.52690.52060.51810.51440.5134
h 1 0.59000.62510.58840.59350.57030.57020.5459
h 2 1.21471.26661.19241.19971.15241.15231.1024
h 3 1.76311.83621.72741.73171.66181.65801.5869
h 4 2.23182.32832.18742.19182.10452.09842.0080
h 5 2.63722.75692.58942.59572.48602.48042.3743
h 6 2.99283.13502.94222.95392.83342.82732.7040
h 7 3.47453.26973.27733.14063.14033.0023
h 8 3.78463.55753.57463.42953.42313.2787
h 9 3.83273.85213.69823.69073.5340
h 10 4.09094.11123.94513.94183.7774
h 11 4.35544.18094.17733.9958
h 12 4.58314.40284.40084.2094
h 13 4.61604.61654.4223
h 14 4.81544.82224.6167
h 15 5.01254.8086
h 16 5.20574.9797
h 17 5.1611
h 18 5.3314
Table A10. AC-SRC for A R L 0 = 900 .
Table A10. AC-SRC for A R L 0 = 900 .
ARL 0 900900900900900900900
j max 681012141618
k AC-SRC 0.54910.53180.52670.52040.51810.51480.5131
h 1 0.60160.64330.60970.61730.59090.58920.5726
h 2 1.24151.30661.23471.24721.19461.19041.1571
h 3 1.80511.89731.79271.80511.72561.71731.6715
h 4 2.28862.40692.27142.28702.18742.17662.1170
h 5 2.70382.85232.69292.71082.59062.57762.5060
h 6 3.07403.24593.06603.08662.94942.93732.8504
h 7 3.59733.39823.43143.27563.26173.1673
h 8 3.91843.70213.73713.57523.55873.4569
h 9 3.99494.03333.85523.83543.7322
h 10 4.26264.30114.11544.09523.9796
h 11 4.56014.36484.34394.2216
h 12 4.79924.59414.57104.4496
h 13 4.81694.79584.6658
h 14 5.02925.01404.8737
h 15 5.21385.0763
h 16 5.41205.2603
h 17 5.4477
h 18 5.6263
Table A11. AC-SRC for A R L 0 = 1000 .
Table A11. AC-SRC for A R L 0 = 1000 .
ARL 0 1000100010001000100010001000
j max 681012141618
k AC-SRC 0.54880.53150.52670.52040.51800.51450.5131
h 1 0.62100.66270.62880.63650.61230.61330.5929
h 2 1.28301.35211.27731.28971.24011.24151.2002
h 3 1.86601.96501.85651.86681.79461.79421.7343
h 4 2.36732.48392.35382.37002.27722.27542.2008
h 5 2.79702.95812.79362.81362.70322.69982.6098
h 6 3.18393.36393.18193.20123.07713.07842.9711
h 7 3.73123.53283.55463.41823.41493.3030
h 8 4.06953.85443.87933.72913.72793.6101
h 9 4.14544.18504.02494.02713.8824
h 10 4.42044.46944.29754.29914.1504
h 11 4.73594.55864.55214.4033
h 12 4.99084.79354.80174.6367
h 13 5.02535.03804.8617
h 14 5.24405.25585.0762
h 15 5.47595.2897
h 16 5.67745.4903
h 17 5.6778
h 18 5.8687

References

  1. Montgomery, D.G. Statistical Quality Control: A Modern Introduction, 7th ed.; John Wiley & Sons: Hoboken, NJ, USA, 2012; ISBN 978-1-118-32257-4. [Google Scholar]
  2. Qiu, P. Introduction to Statistical Process Control, 1st ed.; Chapman and Hall/CRC: Boca Raton, FL, USA, 2013; ISBN 978-1-439-84799-2. [Google Scholar]
  3. Hawkins, D.M.; Olwell, D.H. Cumulative Sum Charts and Charting for Quality Improvement, 1st ed.; Springer-Verlag New York: New York, NY, USA, 1998; ISBN 978-1-461-27245-8. [Google Scholar]
  4. Shewhart, W.A. Economic Control of Quality of Manufactured Product, 1st ed.; D. Van Nostrand Company, Inc.: New York, NY, USA, 1931; ISBN 978-0-87389-076-2. [Google Scholar]
  5. Page, E.S. Continuous Inspection Schemes. Biometrika 1954, 41, 100–115. [Google Scholar] [CrossRef]
  6. Roberts, S.W. Control Chart Tests Based on Geometric Moving Averages. Technometrics 1959, 1, 239–250. [Google Scholar] [CrossRef]
  7. Roberts, S.W. A Comparison of Some Control Chart Procedures. Technometrics 1966, 8, 411–430. [Google Scholar] [CrossRef]
  8. Shiryaev, A.N. On Optimum Methods in Quickest Detection Problems. Theory Probab. Appl. 1963, 8, 22–46. [Google Scholar] [CrossRef]
  9. Pollak, M. Optimal Detection of a Change in Distribution. Ann. Stat. 1985, 13, 206–227. [Google Scholar] [CrossRef]
  10. Basseville, M.; Nikiforov, I. Detection of Abrupt Changes: Theory and Application, 1st ed.; Prentice Hall: Englewood Cliffs, NJ, USA, 1993; ISBN 978-0-131-26780-0. [Google Scholar]
  11. Stoumbos, Z.G.; Reynolds, M.R.; Ryan, T.P.; Woodall, W.H. The State of Statistical Process Control as We Proceed into the 21st Century. J. Am. Stat. Assoc. 2000, 95, 992–998. [Google Scholar] [CrossRef]
  12. Frisén, M. Statistical Surveillance. Optimality and Methods. Int. Stat. Rev. 2003, 71, 403–434. [Google Scholar] [CrossRef] [Green Version]
  13. Bakir, S.T. Distribution-Free (Nonparametric) Statistical Quality Control Charts: A Concise Summary Part I (1920’s–2000), 1st ed.; CreateSpace Independent Publishing Platform: Scotts Valley, CA, USA, 2011; ISBN 978-1-46118-743-1. [Google Scholar]
  14. Qiu, P. Optimal Some perspectives on nonparametric statistical process control. J. Qual. Tech. 2018, 50, 49–65. [Google Scholar] [CrossRef]
  15. Chakraborti, S.; van der Laan, P.; Bakir, S.T. Nonparametric Control Charts: An Overview and Some Results. J. Qual. Tech. 2001, 33, 304–315. [Google Scholar] [CrossRef]
  16. Lang, M.; Zoubir, A.M. A nonparametric cumulative sum scheme based on sequential ranks and adaptive control limits. In Proceedings of the 23rd European Signal Processing Conference (EUSIPCO), Nice, France, 31 August–4 September 2015; pp. 1984–1988. [Google Scholar] [CrossRef]
  17. Lang, M. Automatic Near Real-Time Outlier Detection and Correction in Cardiac Interbeat Interval Series for Heart Rate Variability Analysis: Singular Spectrum Analysis-Based Approach. JMIR Biomed. Eng. 2019, 4, e10740. [Google Scholar] [CrossRef]
  18. McDonald, D.R. A Cusum Procedure Based on Sequential Ranks. Laboratory for Research in Statistics and Probability; Carleton University: Ottawa, ON, Canada, 1985. [Google Scholar]
  19. Chatterjee, S.; Qiu, P. Distribution-free cumulative sum control charts using bootstrap-based control limits. Ann. Appl. Stat. 2009, 3, 349–369. [Google Scholar] [CrossRef]
  20. Lucas, J.M. The Design and Use of Cumulative Sum Control Schemes. Technometrics 1976, 14, 51–59. [Google Scholar]
  21. Reynolds, M.R. Approximations to the average run length in cumulative sum control charts. Technometrics 1975, 17, 65–71. [Google Scholar] [CrossRef]
  22. Lorden, G. Procedures for Reacting to a Change in Distribution. Ann. Math. Stat. 1971, 42, 1897–1908. [Google Scholar] [CrossRef]
  23. Moustakides, G.V. Optimal Stopping Times for Detecting Changes in Distributions. Ann. Stat. 1986, 14, 1379–1387. [Google Scholar] [CrossRef]
  24. Lang, M. A Low-Complexity Model-Free Approach for Real-Time Cardiac Anomaly Detection Based on Singular Spectrum Analysis and Nonparametric Control Charts. Technologies 2018, 6, 26. [Google Scholar] [CrossRef]
  25. Wang, X.; Poor, H.V. Wireless Communication Systems: Advanced Techniques for Signal Reception; Prentice Hall: Englewood Cliffs, NJ, USA, 2003; ISBN 978-0-13702-080-5. [Google Scholar]
  26. Middleton, D. Statistical-physical models of electromagnetic interference. IEEE Trans. Electromagn. Compat. 1977, EMC-19, 106–127. [Google Scholar] [CrossRef]
  27. Middleton, D. Non-Gaussian noise models in signal processing for telecommunications: New methods and results for class A and class B noise models. IEEE Trans. Inf. Theory 1999, 45, 1129–1149. [Google Scholar] [CrossRef]
  28. Nishina, K.; Nishiyuki, S. False alarm probability function of CUSUM charts. Econ. Qual. Control 2003, 18, 101–112. [Google Scholar] [CrossRef]
Table 1. Optimal control limits h and reference values k for the parametric cumulative sum (CUSUM) (C) and the conventional sequential ranks CUSUM (SRC) for A R L 0 = { 100 , 500 , 1000 } (ARL = average run length).
Table 1. Optimal control limits h and reference values k for the parametric cumulative sum (CUSUM) (C) and the conventional sequential ranks CUSUM (SRC) for A R L 0 = { 100 , 500 , 1000 } (ARL = average run length).
ARL 0 = 100 ARL 0 = 500 ARL 0 = 1000
CSRCCSRCCSRC
k 0.50.64280.50.64250.50.6428
h 2.84970.7984.38911.20315.07081.382
Table 2. A R L 0 = 100 , 0% contamination.
Table 2. A R L 0 = 100 , 0% contamination.
j max AC-SRC
τ CSRC681012141618
10DD5.579833.197028.901629.556534.075736.793540.112142.063645.7209
ARL100.1217118.745699.976699.976499.645499.867799.968799.293199.6043
FAR0.06550.00560000.00010.00010.00020.0003
20DD4.582912.403812.343112.781413.435414.758316.136216.963618.2698
ARL100.1221118.766699.8431100.186499.5062100.0187100.010899.496299.5444
FAR0.15660.06070.01480.00840.00640.02800.05180.06420.0779
30DD4.57898.27888.70009.391310.468511.234912.070712.455713.2802
ARL100.0206118.856799.7885100.202999.4369100.151499.715699.451999.5012
FAR0.23900.13060.07020.08750.10680.11640.13140.14030.1587
40DD4.57496.98147.94738.53549.32789.814110.477910.933111.5477
ARL100.0611118.816799.9674100.301189.7641100.030389.840699.363799.3303
FAR0.31410.20130.17990.18180.18780.20290.21860.22680.2405
50DD4.58026.37527.61817.90708.66499.20049.647510.057810.5800
ARL100.1531118.828599.8177100.251399.382799.997799.728799.481799.6277
FAR0.38100.26950.26680.26200.28230.28890.29510.30720.3224
Table 3. A R L 0 = 500 , 0% contamination.
Table 3. A R L 0 = 500 , 0% contamination.
j max AC-SRC
τ CSRC681012141618
10DD7.4845268.0231127.1618115.8420114.1003113.0386118.7457122.5208134.0196
ARL500.3259532.0541489.4462487.0666483.9462486.4092500.5751504.5539527.5008
FAR0.00880.00010000000
20DD7.477489.378026.322736.446627.166228.611630.561332.501835.2079
ARL499.8330531.6094484.7291484.6379484.0727486.1253500.1508504.9982526.9485
FAR0.02800.00670.00120.00030.00010000
30DD7.476236.642515.435416.965717.771119.115120.277921.823223.2656
ARL500.0630531.3130489.2917484.4814484.2848486.0503499.7263505.1260526.9345
FAR0.04730.01900.00940.00440.00300.00160.00100.00060.0003
40DD7.469120.286213.010814.511715.244416.311017.189318.426619.5790
ARL500.2418531.3190487.2419486.2570484.3922486.3174500.3466505.4857527.5350
FAR0.06610.03360.02330.01440.01170.00780.00560.00380.0025
50DD7.480414.525712.048813.450714.035614.943015.697616.734617.6902
ARL499.8426531.3030487.4462484.9996483.3978486.6009500.5158504.9296527.0213
FAR0.08490.04930.04100.02850.02520.01870.01470.01090.0077
Table 4. A R L 0 = 1000 , 0% contamination.
Table 4. A R L 0 = 1000 , 0% contamination.
j max AC-SRC
τ CSRC681012141618
10DD8.8165648.1897362.1763324.2384318.3756317.0301318.4782316.5780316.4286
ARL998.80421045.71431003.2996993.8997991.7992.7993.7
FAR0.003000000000
20DD8.7880254.244560.937254.499254.186957.041859.454962.694964.8553
ARL1003.31045.57141003.71001.1993.1996.5992992.8993.8
FAR0.01300.00230.0001000000
30DD8.7839102.896524.490825.467827.128829.971331.424734.155335.4113
ARL999.41044.6429104.8998.3993.4993.7997.4994.1992.9
FAR0.02290.00800.00210.00030.00020000
40DD8.787147.567917.559120.357221.786024.297425.479627.664428.7028
ARL1000.45711045.31431002.9998.6993.2996.4993.9991.9993.4
FAR0.03280.01480.00610.00220.00140.00060.00030.00010.0001
50DD8.787826.762915.838818.389119.639121.798222.817124.703525.5985
ARL1000.51045.74291007.6997.3994.2996.9995.7993993.3
FAR0.04260.02220.01300.00650.00430.00230.00160.00090.0006
Table 5. A R L 0 = 100 , 10% contamination.
Table 5. A R L 0 = 100 , 10% contamination.
j max AC-SRC
τ CSRC681012141618
10DD3.891052.287046.368247.292752.337955.530559.814762.232266.5670
ARL24.7372118.846899.6766100.508899.6951100.287799.739799.421199.5998
FAR0.28340.005700000.00010.00020.0004
20DD3.889426.852523.594923.286924.871126.793229.351830.800533.2495
ARL24.7353118.8606100.0215100.411499.8287100.134999.733699.291299.6710
FAR0.53690.06100.01470.00890.00660.02770.05290.06400.0778
30DD3.883118.465115.838016.201418.170519.256520.641721.325022.9193
ARL24.7227118.906599.9930100.240399.769799.779199.865199.460599.5673
FAR0.70070.13060.06960.08820.10880.11610.13340.13770.1585
40DD3.905714.852613.820314.088315.197315.952417.117517.910618.9459
ARL24.7271118.837299.9986100.397199.7328100.173299.642799.336199.6082
FAR0.80710.20140.18210.18110.18900.20120.21890.22860.2415
50DD3.880013.019812.693012.725713.910514.528815.338915.988316.9118
ARL24.7468118.885699.9125100.502699.7612100.007499.656099.581499.4769
FAR0.87560.26970.26650.26490.28430.28760.29370.30870.3216
Table 6. A R L 0 = 500 , 10% contamination.
Table 6. A R L 0 = 500 , 10% contamination.
j max AC-SRC
τ CSRC681012141618
10DD6.3947353.9665225.9275209.4729205.9059204.0480213.5904214.6237233.6764
ARL65.5479531.4316488.0331484.4087488.4807488.2045499.9793506.4905526.8804
FAR0.09310.00010000000
20DD6.3823201.678381.350971.755870.501570.399472.924375.040882.1264
ARL65.5938531.5086488.6755483.7614485.5939487.7366501.0609506.4544527.0465
FAR0.22730.00680.00150.00030.00010000
30DD6.3845123.044840.402437.053136.841537.707339.437641.281644.6164
ARL65.6212531.4936488.4589482.7787485.7063487.1630488.9567507.7213526.9891
FAR0.34230.01920.01000.00430.00290.00170.00100.00050.0004
40DD6.384781.473627.347126.523227.046128.085429.364830.986033.1040
ARL65.5715531.2075488.4647485.5648485.1457487.8131500.0233506.9858527.3921
FAR0.44010.03380.02450.01520.01140.00800.00560.00360.0023
50DD6.389157.734722.386922.506423.167424.150525.183626.683028.3272
ARL65.5934531.5276488.1271484.9405485.9766487.5572498.3469506.6913526.9864
FAR0.52330.04930.04170.02970.02500.01910.01490.01080.0076
Table 7. A R L 0 = 1000 , 10% contamination.
Table 7. A R L 0 = 1000 , 10% contamination.
j max AC-SRC
τ CSRC681012141618
10DD7.5806794.075571.8691533.4575521.0822520.3675519.0203508.9829512.4341
ARL97.31431044.28571007.3996.5995.5998.7997.2991.8992.1
FAR0.055400000000
20DD7.5627509.1507223.2311187.2751180.3638179.9559175.3016177.2089178.9137
ARL97.32861044.37141008.3997.5994.8999.6997.5995.3997.6
FAR0.15110.00230.0001000000
30DD7.5561331.299993.748176.790175.541575.997276.408678.691580.2621
ARL97.35711044.97141006.5996.5998.61002.3997.9990.1997.2
FAR0.23750.00790.00180.00040.00020000
40DD7.5618221.163950.466344.473444.934747.232748.309951.187452.2784
ARL97.31044.48571006.6996.6994.7998.5997.2994.3994.9
FAR0.31520.01490.00630.00250.00150.00060.00030.00020.0001
50DD7.5575152.886435.121133.710334.839037.271338.412141.010642.1451
ARL97.34291044.65711006.7996.2994.1996.6996.8992.8995.1
FAR0.38480.02240.01310.00640.00430.00230.00160.00090.0007

Share and Cite

MDPI and ACS Style

Lang, M. Control Limits for an Adaptive Self-Starting Distribution-Free CUSUM Based on Sequential Ranks. Technologies 2019, 7, 71. https://doi.org/10.3390/technologies7040071

AMA Style

Lang M. Control Limits for an Adaptive Self-Starting Distribution-Free CUSUM Based on Sequential Ranks. Technologies. 2019; 7(4):71. https://doi.org/10.3390/technologies7040071

Chicago/Turabian Style

Lang, Michael. 2019. "Control Limits for an Adaptive Self-Starting Distribution-Free CUSUM Based on Sequential Ranks" Technologies 7, no. 4: 71. https://doi.org/10.3390/technologies7040071

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop