Next Article in Journal
Disposable and Low-Cost Colorimetric Sensors for Environmental Analysis
Next Article in Special Issue
Variability Matters
Previous Article in Journal
Long-Term Effects of a Cognitive Behavioral Conference Call Intervention on Depression in Non-Professional Caregivers
Open AccessArticle

Nonparametric Limits of Agreement in Method Comparison Studies: A Simulation Study on Extreme Quantile Estimation

by 1,2
1
Department of Nuclear Medicine, Odense University Hospital, 5000 Odense C, Denmark
2
Department of Clinical Research, University of Southern Denmark, 5000 Odense C, Denmark
Int. J. Environ. Res. Public Health 2020, 17(22), 8330; https://doi.org/10.3390/ijerph17228330
Received: 28 September 2020 / Revised: 6 November 2020 / Accepted: 9 November 2020 / Published: 11 November 2020
(This article belongs to the Special Issue Statistical and Epidemiological Methods in Public Health)

Abstract

Bland–Altman limits of agreement and the underlying plot are a well-established means in method comparison studies on quantitative outcomes. Normally distributed paired differences, a constant bias, and variance homogeneity across the measurement range are implicit assumptions to this end. Whenever these assumptions are not fully met and cannot be remedied by an appropriate transformation of the data or the application of a regression approach, the 2.5% and 97.5% quantiles of the differences have to be estimated nonparametrically. Earlier, a simple Sample Quantile (SQ) estimator (a weighted average of the observations closest to the target quantile), the Harrell–Davis estimator (HD), and estimators of the Sfakianakis–Verginis type (SV) outperformed 10 other quantile estimators in terms of mean coverage for the next observation in a simulation study, based on sample sizes between 30 and 150. Here, we investigate the variability of the coverage probability of these three and another three promising nonparametric quantile estimators with n=50(50)200,250(250)1000. The SQ estimator outperformed the HD and SV estimators for n=50 and was slightly better for n=100, whereas the SQ, HD, and SV estimators performed identically well for n150. The similarity of the boxplots for the SQ estimator across both distributions and sample sizes was striking.
Keywords: agreement; Bland–Altman analysis; coverage; limits of agreement; method comparison; quantile estimation; repeatability; reproducibility agreement; Bland–Altman analysis; coverage; limits of agreement; method comparison; quantile estimation; repeatability; reproducibility

1. Introduction

When comparing methods for the measurement of a continuous outcome, the Bland–Altman Limits of Agreement (BA LoAs) are a well-known and well-established means to this end [1,2,3]. Mean values are plotted against the paired differences in a scatter plot that is supplemented by an estimate of the bias (i.e., the mean of the paired differences) and the so-called LoAs (bias estimate +/− 1.96 standard deviations of the paired differences), including the respective 95% confidence intervals [4,5,6,7]. Under the assumptions of normally distributed paired differences, the BA LoAs represent estimates for the boundaries between which 95% of all population differences are supposed to lie. In the case of the variance heterogeneity of the paired differences, a normalizing transformation (like the natural logarithm) may render the usual analysis (on the log-scale) possible, whereas a bias that is non-constant across the measurement range might require a regression approach [3,8]. If neither a transformation of the data, nor a regression approach are applicable, the LoAs are often estimated by simple empirical 2.5% and 97.5% quantiles. However, various nonparametric quantile estimators have been proposed in the literature during the recent decades, which likewise may serve in the nonparametric estimation of 2.5% and 97.5% quantiles [9,10,11,12]. In an earlier endeavor, we performed a literature search on nonparametric quantile estimators from which we compared 15 in a simulation study [13]. We assessed the performance of these estimators by the average coverage probability for one newly generated observation from 20,000 fictive trials, and we assumed six different distributions and sample sizes between 30 and 150 to this end. We found that a simple sample quantile estimator based on two rank statistics [9], the Harrell–Davis estimator [14], and the estimators of the Sfakianakis–Verginis type [15] performed, on average, closely to the nominal coverage probability of 95%. The purpose of this paper is to illuminate the variability of the coverage probability of these nonparametric quantile estimators in a simulation study. Three further, possibly promising nonparametric quantile estimators from our former investigation [13] and the classical BA LoAs [3] (as a benchmark measure and for illustrative purposes) are added. In Section 2, all nonparametric quantile estimators and the simulation setup are described. In Section 3, the results are given and illustrated by boxplots. A discussion closes the paper.

2. Materials and Methods

The description of the six different methods of nonparametric quantile estimation considered here is kept brief. Further details can be found elsewhere [13].

2.1. Nonparametric Quantile Estimators

2.1.1. Sample Quantile Estimator

A random sample of n paired differences X 1 , X n is sorted in increasing order X ( 1 ) X ( 2 ) X ( n ) ; the symbols here denote the order statistics of the random sample. The SQ estimator is a weighted average of the two order statistics that are closest to including p percent of all the observations in the sample:
S Q = ( 1 α ) X ( r ) + α X ( r + 1 )
with α = p ( n + 1 ) r , r = p ( n + 1 ) , and x is the greatest integer that is less than or equal to x [10,12].

2.1.2. Harrell–Davis Estimator

The HD estimator, as well as those estimators that follow, employs linear combinations of all the available order statistics, weighting them according to their relative closeness to the target percentile. They can be given as L-statistics, which is, j = 1 n W j · X ( j ) , where W j and X ( j ) is the weight for the j-th order statistic and the j-th order statistic itself, respectively [16]. The Harrell–Davis estimator is given by:
H D = i = 1 n W i X ( i )
with weight function:
W i = I i / n p ( n + 1 ) , ( 1 p ) ( n + 1 ) I ( i 1 ) / n p ( n + 1 ) , ( 1 p ) ( n + 1 ) ,
where I i / n a , b is the incomplete beta function [9,14].

2.1.3. Bernstein Polynomial Estimator

The BP estimator employs the binomial probability of observing exactly i out of n events with an event probability of p, B ( i ; n , p ) , and is given by:
B P = i = 1 n B ( i 1 ; n 1 , p ) X ( i )
according to Cheng [17].

2.1.4. HD Estimator Using a Level Crossing Empirical Distribution

Huang [18] modified the Harrell–Davis estimator (2) by applying a weighted empirical distribution function instead of the empirical distribution with equal weights 1 / n . The Harrell–Davis estimator using a level crossing empirical distribution function can be written as:
H D l c = i = 1 n W i X ( i )
with weight function:
W i = I q i p ( n + 1 ) , ( 1 p ) ( n + 1 ) I q i 1 p ( n + 1 ) , ( 1 p ) ( n + 1 ) ,
the incomplete beta function I q i a , b , q i = j = 1 i w j , i = 1 , , n , and:
w j = 1 2 1 n 2 n ( n 1 ) if j = 1 , n 1 n ( n 1 ) if j = 2 , 3 , n 1 .

2.1.5. Sfakianakis–Verginis Estimator

Sfakianakis and Verginis [15] proposed a group of three estimators, from which we chose the first one due to the similarity of the results in our former investigation [13]. SV estimators are supposed to better estimate quantiles in the tails of a distribution when using small samples, and they employ also the binomial probabilities B ( i ; n , p ) as weights for the ordered statistics X ( i ) , i = 1 , , n :
S V = 2 B ( 0 ; n , p ) + B ( 1 ; n , p ) 2 X ( 1 ) + B ( 0 ; n , p ) 2 X ( 2 ) B ( 0 ; n , p ) 2 X ( 3 ) + i = 2 n 1 B ( i ; n , p ) + B ( i 1 ; n , p ) 2 X ( i ) B ( n ; n , p ) 2 X ( n 2 ) + B ( n ; n , p ) 2 X ( n 1 ) + 2 B ( n ; n , p ) + B ( n 1 ; n , p ) 2 X ( n ) .

2.1.6. Navruz–Özdemir Estimator

Recently, Navruz and Özdemir [19] introduced a new quantile estimator, which is also a linear function of the order statistics with weights of binomial probabilities:
N O = ( B ( 0 ; n , p ) 2 p + B ( 1 ; n , p ) p ) X ( 1 ) + B ( 0 ; n , p ) ( 2 3 p ) X ( 2 ) B ( 0 ; n , p ) ( 1 p ) X ( 3 ) + i = 1 n 2 ( B ( i ; n , p ) ( 1 p ) + B ( i + 1 ; n , p ) p ) X ( i + 1 ) B ( n ; n , p ) p X ( n 2 ) + B ( n ; n , p ) ( 3 p 1 ) X ( n 1 ) + ( B ( n 1 ; n , p ) ( 1 p ) + B ( n ; n , p ) ( 2 2 p ) ) X ( n ) .

2.2. Simulation Setup

We assumed six distributions (Figure 1), the choice and number of which were motivated by former simulation studies on nonparametric quantile estimation, i.e., 4–6 symmetric and asymmetric distributions [11,14,15,18,19]:
  • the standard normal distribution;
  • a lognormal distribution with meanlog = 1 and sdlog = 1;
  • a beta distribution with shape parameters α = 2 and β = 5 (non-centrality parameter λ = 0 );
  • a beta distribution with shape parameters α = 2 and β = 2 ( λ = 0 );
  • a chi-squared distribution with 4 degrees of freedom; and
  • an exponential distribution with rate parameter 1.
For sample sizes n = 50 ( 50 ) 200 , 250 ( 250 ) 1000 , each distribution, and quantile estimator, we simulated 5000 fictive trials and derived the LoAs. Their respective coverage probability c was derived by applying the cumulative distribution function F ( x ) to the LoAs, i.e.,
c = F ( u p p e r L o A ) F ( l o w e r L o A ) .
This resulted in the distributions of coverage probabilities, which we display with boxplots. We compared the nonparametric quantile estimators with respect to (a) the average (i.e., median and mean) coverage probability and the closeness to the nominal 95% level and (b) their variability in terms of the 5% quantile and the first quartile. The classical BA LoAs served as the benchmark indication of expectable variation under the standard normal distribution; their performance under the remaining distributions illustrated the inappropriateness under non-normal distributions. The Root Mean Squared Error (RMSE) for the estimation of the 2.5% and 97.5% quantiles was supplemented. Here, the true values for these quantiles were known, as were the assumed distributions. Note, though, that the investigation of the coverage probability enabled a simultaneous assessment of both LoAs, whereas the RMSE was related to one of the two LoAs at a time. For instance, underestimation of the lower LoA and underestimation of the upper LoA for one set of estimated LoAs can imply the very same coverage probability as an overestimated lower LoA and an overestimated upper LoA for another set of estimated LoAs. The R code is available as Supplementary Material Code S1 (R Version 4.0.2).

3. Results

3.1. BA LoA

For n = 50 , the median (mean) coverage probability was close to 0.95 for three distributions, but above the nominal level for the beta distributions (Distributions 3 and 4 in Figure 2; 0.957 (0.953) and 0.972 (0.967), respectively) and slightly below for the exponential distribution (Distribution 6 in Figure 2; 0.943 (0.940)). For Distribution 4, even the first quartile exceeded 0.95 (0.953), indicating overestimation of the BA LoA, which is visible by the box lying completely above 0.95 in Figure 2. The same applied in the case of Distribution 3 for sample sizes of n 200 and for Distribution 2 for sample sizes of n 500 (Figure 2 and Figure 3). In the case of Distribution 6, the median (mean) coverage probability fell short of the nominal level for all sample sizes, even for n = 1000 (0.948 (0.948)).

3.2. Nonparametric Quantile Estimators

3.2.1. SQ Estimator

The median (mean) coverage probability of the simple sample quantile estimator SQ was 0.958–0.959 (0.952–0.954) for n = 50 ; the first quartile was 0.938–0.939; and the third quartile was 0.972–0.975 (Figure 4). For larger sample sizes (Supplementary Figure S2), the coverage probability converged to the nominal level, always exceeding 0.95 on average (e.g., n = 150 : median 0.952–0.953 (mean 0.950–0.951), first quartile 0.939–0.94). The 5% quantile increased likewise from 0.896 ( n = 50 ) to 0.938 ( n = 1000 ); see Table 1. A distinguishing feature of SQ was the similarity of the boxplots across both distributions and sample sizes (Figure 4, Supplementary Figure S2).

3.2.2. HD Estimator

The median (mean) coverage probability of the HD estimator was 0.941–0.946 (0.936–0.941) for n = 50 ; the first quartile was 0.920–0.924; and the third quartile was 0.958–0.964 (Figure 5). For larger sample sizes (Supplementary Figure S3), the coverage probability converged, on average, to the nominal level (e.g., n = 150 : median 0.949–0.952 (mean 0.947–0.950), first quartile 0.938–0.94). The 5% quantile was slightly below that of SQ for n = 50 , 100 (Table 1), but identical for n 150 . The HD estimator performed very similar to the SQ estimator for n 150 (Figure 5, Supplementary Figure S3).

3.2.3. BP Estimator

The BP estimator fell short of the nominal coverage probability for n 250 (Supplementary Figures S4 and S5). For n = 250 , for instance, the median (mean) and first quartile coverage probability was 0.945–0.947 (0.944–0.946) and 0.936–0.938, respectively. The median (mean) coverage probability reached 0.949 (0.948–0.949) for n = 750 , 1000 .
Comparing the HD and BP estimator for n = 200 in terms of the RMSE, the RMSE was smaller for the HD than for the BP estimator for four distributions in the estimation of the 2.5% percentile (Table 2) and for one distribution in the estimation of the 97.5% percentile (Table 3). For three distributions, the RMSE was smaller for the BP than for the HD estimator in the estimation of the 97.5% percentile.
In general, all nonparametric LoAs performed considerably close to each other in the estimation of the 2.5% percentile for sample sizes of n 100 (Table 2). In the estimation of the 97.5% percentile, this was the case for n 200 (Table 3).

3.2.4. HD lc Estimator

The HD lc estimator performed quite similar to the BP estimator (Supplementary Figures S6 and S7). The median (mean) coverage probability reached, though, 0.949–0.95 (0.948–0.949) for n = 500 ; for n = 250 , the median (mean) and first quartile coverage probability was 0.947–0.948 (0.946–0.948) and 0.938–0.94, respectively.

3.2.5. SV Estimator

The median (mean) coverage probability of the SV estimator was 0.942–0.946 (0.937–0.941) for n = 50 ; the first quartile was 0.920–0.924; and the third quartile was 0.96–0.963 (Figure 6). For larger sample sizes (Supplementary Figure S8), the coverage probability converged, on average, to the nominal level (e.g., n = 150 : median 0.95–0.953 (mean 0.948–0.951), first quartile 0.938–0.942). The 5% quantile was slightly below that of SQ for n = 50 , 100 (Table 1), but identical for n 150 . The SV estimator performed very similar to the SQ estimator for n 150 and, in general, close to identical to the HD estimator with respect to the 5% quantile, the first quartile, and the median of the coverage probability for all sample sizes ( ± 0.001 ) .

3.2.6. NO Estimator

The NO estimator achieved a median (mean) and first quartile coverage probability of 0.95–0.951 (0.949–0.950) and 0.944–0.945 for n = 500 , but fell short of the nominal coverage probability of 0.95 for smaller sample sizes (Supplementary Figures S9 and S10). Only for sample sizes of n = 500 , 750 , 1000 , the NO estimator performed comparably to the SQ estimator.

4. Discussion

4.1. Key Finding

The SQ estimator, a simple sample quantile estimator based on two rank statistics, conservatively kept, on average, the nominal coverage level of 95% for n = 50 and converged rapidly towards it with increasing sample size. The variability of the SQ estimator, measured in terms of the 5% quantile and the first quartile of the coverage probability, was smallest amongst all considered nonparametric quantile estimators for n = 50 , 100 . For sample sizes of n 150 , both the HD and the SV estimator performed likewise well in terms of average performance and small variability. The remaining nonparametric estimators considered in the study did so for larger sample sizes only (BP: n 750 , HDlc: n 500 , and NO: n 500 ).

4.2. What Does This Add to What Is Known

Harrell and Davis [14] investigated the mean squared error of their estimator to measure its efficiency with sample sizes of a maximum n = 60 and did not recommend the HD estimator for small n and extreme p. Dielman, Lowry, and Pfaffenberger [12] compared several quantile estimators in terms of bias with sample sizes n = 10 , 15 , 25 , 30 and p as small as 0.02. They concluded that there was not one best estimator across scenarios, but the HD estimator performed well in a wide range of cases, except when p = 0.02 , 0.98 . They highlighted that the HD estimator makes use of many data points and, therefore, performed better in the estimation of middle quantiles. On the contrary, sample quantile estimators, like the SQ estimator, performed well at p = 0.02 , and the HD estimator performed better at larger sample sizes. Sfakianakis and Verginis [15] compared the SV estimator with the HD and SQ estimators in terms of bias and mean squared error for p = 0.01 , 0.05 ( 0.05 ) , 0.95 , 0.99 and n = 5–750. They concluded that the SV estimator performed better than the HD and SQ estimators in most of the examined cases did.
The nonparametric estimation of the LoAs involves two tail percentiles, p = 0.025 , 0.975 , which need to be considered simultaneously. Earlier, we indicated that the SQ estimator outperformed other quantile estimators for n = 50 in terms of the mean coverage probability for the next observation, but the HD estimator and estimators of the Sfakianakis–Verginis type performed likewise well for sample sizes exceeding 80 observations [13]. Here, we investigated both the average performance and variability of nonparametric quantile estimators for the LoA. The SQ estimator outperformed the HD and SV estimators for n = 50 and was slightly better for n = 100 , whereas the SQ, HD, and SV estimators performed identically well for n 150 . The similarity of the boxplots for the SQ estimator across both distributions and sample sizes was striking.

4.3. What Is the Implication, and What Should Change Now?

Whenever violated assumptions for the paired differences prohibit the immediate derivation of the classical BA LoA and neither an appropriate preparatory transformation nor a regression approach are practicable alternatives [3,8,20,21,22], the SQ estimator may serve as a basis for the derivation of nonparametric LoAs. For sample sizes n 150 , this holds also true for the HD and SV estimators, although all available differences contribute to the quantile estimation, and the target quantiles are extreme tail percentiles. The SQ estimator is not defined for n < 40 ; however, a reasonably sized method comparison study is likely to enroll at least 40 subjects anyway [23,24,25,26]. Finally, the 95% confidence intervals should supplement nonparametric LoAs to indicate the estimates’ uncertainty [27,28]. Bootstrapping techniques may serve this purpose for the SQ estimator.

Supplementary Materials

The following are available online at https://www.mdpi.com/1660-4601/17/22/8330/s1: Code S1: R source code for the simulation study, Figure S2: Boxplots of the coverage probability for the SQ estimator and n = 250 ( 250 ) 1000 , Figure S3: Boxplots of the coverage probability for the HD estimator and n = 250 ( 250 ) 1000 , Figure S4: Boxplots of the coverage probability for the BP estimator and n = 50 ( 50 ) 200 , Figure S5: Boxplots of the coverage probability for the BP estimator and n = 250 ( 250 ) 1000 , Figure S6: Boxplots of the coverage probability for the HD lc estimator and n = 50 ( 50 ) 200 , Figure S7: Boxplots of the coverage probability for the HD lc estimator and n = 250 ( 250 ) 1000 , Figure S8: Boxplots of the coverage probability for the SV estimator and n = 250 ( 250 ) 1000 , Figure S9: Boxplots of the coverage probability for the NO estimator and n = 50 ( 50 ) 200 , Figure S10: Boxplots of the coverage probability for the NO estimator and n = 250 ( 250 ) 1000 .

Funding

This research received no external funding.

Acknowledgments

The author would like to express his sincere gratitude to Claus Thorn Ekstrøm (University of Copenhagen) for an earlier discussion of the topic, the Editors of the Special Issue “Statistical and Epidemiological Methods in Public Health”, Sören Möller and Linda Juel Ahrenfeldt, for the kind invitation to contribute, and the three anonymous reviewers for their helpful comments on an earlier version of this manuscript.

Conflicts of Interest

The author declares no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:
BABland–Altman
BPBernstein Polynomial
HDHarrell–Davis
HD lcHarrell–Davis level crossing
LoAsLimits of Agreement
NONavruz–Özdemir
RMSERoot Mean Squared Error
SQSample Quantile
SVSfakianakis–Verginis

References

  1. Altman, D.G.; Bland, J.M. Measurement in medicine: The analysis of method comparison studies. Statistician 1983, 32, 307–317. [Google Scholar] [CrossRef]
  2. Bland, J.M.; Altman, D.G. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986, 1, 307–310. [Google Scholar] [CrossRef]
  3. Bland, J.M.; Altman, D.G. Measuring agreement in method comparison studies. Stat. Methods Med. Res. 1999, 8, 135–160. [Google Scholar] [CrossRef] [PubMed]
  4. Carkeet, A. Exact parametric confidence intervals for Bland–Altman limits of agreement. Optom. Vis. Sci. 2015, 92, e71–e80. [Google Scholar] [CrossRef] [PubMed]
  5. Carkeet, A.; Goh, Y.T. Confidence and coverage for Bland–Altman limits of agreement and their approximate confidence intervals. Stat. Methods Med. Res. 2018, 27, 1559–1574. [Google Scholar] [CrossRef] [PubMed]
  6. Abu-Arafeh, A.; Jordan, H.; Drummond, G. Reporting of method comparison studies: A review of advice, an assessment of current practice, and specific suggestions for future reports. Br. J. Anaesth. 2016, 117, 569–575. [Google Scholar] [CrossRef] [PubMed]
  7. Gerke, O. Reporting standards for a Bland–Altman agreement analysis: A review of methodological reviews. Diagnostics 2020, 10, 334. [Google Scholar] [CrossRef]
  8. Ludbrook, J. Confidence in Altman-Bland plots: A critical review of the method of differences. Clin. Exp. Pharmacol. Physiol. 2010, 37, 143–149. [Google Scholar] [CrossRef]
  9. Steinberg, S.M.; Davis, C.E. Comparison of nonparametric point estimators for interquantile differences in moderate sized samples. Commun. Stat. Theory Methods 1987, 16, 1607–1616. [Google Scholar] [CrossRef]
  10. Parrish, R.S. Comparision of quantile estimators in normal sampling. Biometrics 1990, 46, 247–257. [Google Scholar] [CrossRef]
  11. Sheather, S.J.; Marron, J.S. Kernel quantile estimators. J. Am. Stat. Assoc. 1990, 85, 410–416. [Google Scholar] [CrossRef]
  12. Dielman, T.; Lowry, C.; Pfaffenberger, R. A comparision of quantile estimators. Commun. Stat. Simul. Comput. 1994, 23, 355–371. [Google Scholar] [CrossRef]
  13. Frey, M.E.; Petersen, H.C.; Gerke, O. Nonparametric limits of agreement for small to moderate sample sizes: A simulation study. Stats 2020, 3, 343–355. [Google Scholar] [CrossRef]
  14. Harrell, F.E.; Davis, C.E. A new distribution-free quantile estimator. Biometrika 1982, 69, 635–640. [Google Scholar] [CrossRef]
  15. Sfakianakis, M.E.; Verginis, D.G. A new family of nonparametric quantile estimators. Commun. Stat. Simul. Comput. 2008, 37, 337–345. [Google Scholar] [CrossRef]
  16. Serfling, R.J. Approximation Theorems of Mathematical Statistics; John Wiley & Sons: Hoboken, NJ, USA, 1980. [Google Scholar]
  17. Cheng, C. The Bernstein polynomial estimator of a smooth quantile function. Stat. Probab. Lett. 1995, 24, 321–330. [Google Scholar] [CrossRef]
  18. Huang, M.L. On a distribution-free quantile estimator. Comput. Stat. Data Anal. 2001, 37, 477–486. [Google Scholar] [CrossRef]
  19. Navruz, G.; Özdemir, A.F. A new quantile estimator with weights based on a subsampling approach. Br. J. Math. Stat. Psychol. 2020, 73. [Google Scholar] [CrossRef]
  20. Dekkers, I.A.; de Heer, P.; Bizino, M.B.; de Vries, A.P.J.; Lamb, H.J. 1 H-MRS for the assessment of renal triglyceride content in humans at 3T: A primer and reproducibility study. J. Magn. Reson. Imaging 2018, 48, 507–513. [Google Scholar] [CrossRef]
  21. Sadler, W.A. Using the variance function to generalize Bland–Altman analysis. Ann. Clin. Biochem. 2019, 56, 198–203. [Google Scholar] [CrossRef]
  22. Simonsen, J.A.; Thilsing-Hansen, K.; Høilund-Carlsen, P.F.; Gerke, O.; Andersen, T.L. Glomerular filtration rate: Comparison of simultaneous plasma clearance of 99mTc-DTPA and 51Cr-EDTA revisited. Scand. J. Clin. Lab. Investig. 2020, 80, 408–411. [Google Scholar] [CrossRef] [PubMed]
  23. Lu, M.J.; Zhong, W.H.; Liu, Y.X.; Miao, H.Z.; Li, Y.C.; Ji, M.H. Sample size for assessing agreement between two methods of measurement by Bland–Altman method. Int. J. Biostat. 2016, 12, 20150039. [Google Scholar] [CrossRef] [PubMed]
  24. Jan, S.L.; Shieh, G. The Bland–Altman range of agreement: Exact interval procedure and sample size determination. Comput. Biol. Med. 2018, 100, 247–252. [Google Scholar] [CrossRef] [PubMed]
  25. Shieh, G. The appropriateness of Bland–Altman’s approximate confidence intervals for limits of agreement. BMC Med. Res. Methodol. 2018, 18, 45. [Google Scholar] [CrossRef] [PubMed]
  26. Shieh, G. Assessing agreement between two methods of quantitative measurements: Exact test procedure and sample size calculation. Stat. Biopharm. Res. 2020, 12, 352–359. [Google Scholar] [CrossRef]
  27. Drummond, G.B. Limits of Agreement with confidence intervals are necessary to assess comparability of measurement devices. Anesth. Analg. 2017, 125, 1075. [Google Scholar] [CrossRef]
  28. Drummond, G.B. Limits of agreement may have large confidence intervals. Br. J. Anaesth. 2016, 116, 430–431. [Google Scholar] [CrossRef]
Figure 1. Density functions of the assumed distributions.
Figure 1. Density functions of the assumed distributions.
Ijerph 17 08330 g001
Figure 2. Boxplots of the coverage probability for the BA LoA and n = 50 ( 50 ) 200 .
Figure 2. Boxplots of the coverage probability for the BA LoA and n = 50 ( 50 ) 200 .
Ijerph 17 08330 g002
Figure 3. Boxplots of the coverage probability for the BA LoA and n = 250 ( 250 ) 1000 .
Figure 3. Boxplots of the coverage probability for the BA LoA and n = 250 ( 250 ) 1000 .
Ijerph 17 08330 g003
Figure 4. Boxplots of the coverage probability for the SQ estimator and n = 50 ( 50 ) 200 .
Figure 4. Boxplots of the coverage probability for the SQ estimator and n = 50 ( 50 ) 200 .
Ijerph 17 08330 g004
Figure 5. Boxplots of the coverage probability for the HD estimator and n = 50 ( 50 ) 200 .
Figure 5. Boxplots of the coverage probability for the HD estimator and n = 50 ( 50 ) 200 .
Ijerph 17 08330 g005
Figure 6. Boxplots of the coverage probability for the SV estimator and n = 50 ( 50 ) 200 .
Figure 6. Boxplots of the coverage probability for the SV estimator and n = 50 ( 50 ) 200 .
Ijerph 17 08330 g006
Table 1. Five percent quantiles of the coverage probability by the estimator and sample size n. For BA LoA, the 5% quantiles are shown assuming a standard normal distribution (benchmark values); for the nonparametric estimators, the minimum of the 5% quantiles across all 6 distributions are shown.
Table 1. Five percent quantiles of the coverage probability by the estimator and sample size n. For BA LoA, the 5% quantiles are shown assuming a standard normal distribution (benchmark values); for the nonparametric estimators, the minimum of the 5% quantiles across all 6 distributions are shown.
Estimator501001502002505007501000
BA LoA0.8930.9140.9220.9270.9290.9360.9390.941
SQ0.8960.9130.9190.9220.9250.9330.9360.938
HD0.8800.9100.9190.9220.9260.9330.9360.938
BP0.8590.8970.9110.9170.9210.9310.9350.937
HDlc0.8710.9030.9140.9200.9230.9320.9360.938
SV0.8800.9110.9200.9230.9260.9330.9360.939
NO0.4910.7280.8580.9030.9200.9340.9360.938
Table 2. RMSE of the estimated 2.5% quantiles by sample size, estimator, and distribution.
Table 2. RMSE of the estimated 2.5% quantiles by sample size, estimator, and distribution.
nEstimatorNormal (0,1)Lognormal (0,1)Beta (2,5,0)Beta (2,2,0)Chi-Squared (4)Exponential (1)
50BA LoA0.202.340.0700.0451.970.94
SQ0.340.0400.0150.0310.170.015
HD0.280.0370.0140.0280.160.017
BP0.260.0430.0160.0320.180.023
HD lc0.260.0390.0140.0290.170.019
SV0.270.0380.0140.0290.160.017
NO0.280.0490.0180.0370.210.024
100BA LoA0.142.480.0700.0382.000.96
SQ0.220.0290.0110.0230.1240.012
HD0.200.0260.0100.0210.110.012
BP0.180.0280.0100.0220.120.014
HD lc0.190.0270.0100.0210.120.013
SV0.200.0260.0090.0200.110.012
NO0.190.0290.0110.0220.130.014
150BA LoA0.112.520.0700.0352.010.97
SQ0.180.0240.0090.0190.1040.010
HD0.160.0220.0080.0170.0930.009
BP0.150.0230.0090.0180.0980.011
HD lc0.160.0220.0080.0170.0950.010
SV0.160.0220.0080.0170.0920.009
NO0.150.0230.0090.0180.0990.011
200BA LoA0.0952.550.0700.0342.010.98
SQ0.160.0210.0080.0170.0920.009
HD0.140.0190.0070.0150.0820.008
BP0.140.0200.0070.0160.0850.009
HD lc0.140.0190.0070.0150.0830.008
SV0.140.0190.0070.0150.0810.008
NO0.140.0200.0080.0160.0850.009
250BA LoA0.0862.590.0710.0342.020.98
SQ0.140.0190.0070.0150.0800.008
HD0.130.0170.0060.0140.0730.007
BP0.120.0180.0070.0140.0740.008
HD lc0.120.0170.0070.0140.0730.008
SV0.130.0170.0060.0140.0720.007
NO0.120.0180.0070.0140.0750.008
500BA LoA0.0622.650.0700.0332.030.98
SQ0.0960.0130.0050.0110.0590.006
HD0.0910.0120.0050.0100.0550.005
BP0.0900.0120.0050.0100.0560.006
HD lc0.0900.0120.0050.0100.0560.006
SV0.0910.0120.0050.0100.0550.005
NO0.0900.0120.0050.0100.0560.006
750BA LoA0.0502.680.0710.0332.020.98
SQ0.0770.0110.0040.0090.0480.005
HD0.0730.0100.0040.0090.0450.004
BP0.0730.0100.0040.0090.0460.005
HD lc0.0730.0100.0040.0090.0460.004
SV0.0730.0100.0040.0090.0450.004
NO0.0730.0100.0040.0090.0460.005
1000BA LoA0.0432.680.0710.0332.020.99
SQ0.0680.0100.0040.0080.0420.004
HD0.0650.0090.0030.0070.0400.004
BP0.0640.0090.0040.0070.0400.004
HD lc0.0640.0090.0030.0070.0400.004
SV0.0650.0090.0030.0070.0400.004
NO0.0640.0090.0030.0070.0400.004
Table 3. RMSE of the estimated 97.5% quantiles by sample size, estimator, and distribution.
Table 3. RMSE of the estimated 97.5% quantiles by sample size, estimator, and distribution.
nEstimatorNormal (0,1)Lognormal (0,1)Beta (2,5,0)Beta (2,2,0)Chi-Squared (4)Exponential (1)
50BA LoA0.202.160.0520.0461.750.81
SQ0.343.400.0590.0322.060.87
HD0.282.600.0490.0291.620.68
BP0.262.010.0480.0331.400.59
HD lc0.262.300.0470.0301.490.63
SV0.282.540.0490.0291.610.68
NO0.732.810.230.334.051.38
100BA LoA0.141.830.0460.0391.630.76
SQ0.221.850.0390.0231.270.54
HD0.201.870.0340.0201.160.49
BP0.191.470.0330.0221.020.43
HD lc0.191.600.0330.0211.070.45
SV0.201.950.0340.0201.180.50
NO0.381.620.130.182.050.70
150BA LoA0.111.690.0440.0361.630.74
SQ0.181.370.0310.0190.990.43
HD0.161.360.0280.0170.920.40
BP0.161.170.0270.0180.840.36
HD lc0.161.230.0280.0170.870.38
SV0.161.450.0280.0170.940.41
NO0.181.240.0480.0740.990.39
200BA LoA0.0961.600.0430.0341.620.74
SQ0.161.180.0280.0170.870.37
HD0.141.140.0250.0150.790.34
BP0.141.020.0250.0160.750.32
HD lc0.141.070.0250.0160.770.33
SV0.141.190.0250.0150.800.34
NO0.141.180.0260.0260.790.34
250BA LoA0.0881.550.0420.0331.620.74
SQ0.141.020.0240.0150.760.33
HD0.130.990.0230.0140.710.30
BP0.130.910.0220.0140.680.29
HD lc0.130.940.0220.0140.690.30
SV0.131.020.0220.0140.710.31
NO0.131.100.0220.0140.740.32
500BA LoA0.0621.390.0430.0321.610.73
SQ0.0970.690.0180.0110.530.22
HD0.0910.670.0170.0100.500.21
BP0.0900.640.0160.0100.490.21
HD lc0.0900.650.0160.0100.490.21
SV0.0910.670.0170.0100.500.21
NO0.0940.720.0170.0100.530.22
750BA LoA0.0501.340.0430.0331.600.73
SQ0.0790.570.0140.0090.440.18
HD0.0750.550.0140.0080.420.17
BP0.0740.530.0140.0080.410.17
HD lc0.0750.540.0140.0080.410.17
SV0.0750.550.0140.0080.420.17
NO0.0760.570.0140.0080.430.18
1000BA LoA0.0431.310.0420.0321.600.73
SQ0.0680.480.0120.0080.370.16
HD0.0650.470.0120.0070.360.15
BP0.0640.460.0120.0070.350.15
HD lc0.0650.460.0120.0070.350.15
SV0.0650.470.0120.0070.360.15
NO0.0660.480.0120.0070.360.16
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Back to TopTop