Nonparametric Limits of Agreement for Small to Moderate Sample Sizes: A Simulation Study

Abstract

The assessment of agreement in method comparison and observer variability analysis of quantitative measurements is usually done by the Bland–Altman Limits of Agreement, where the paired differences are implicitly assumed to follow a normal distribution. Whenever this assumption does not hold, the 2.5% and 97.5% percentiles are obtained by quantile estimation. In the literature, empirical quantiles have been used for this purpose. In this simulation study, we applied both sample, subsampling, and kernel quantile estimators, as well as other methods for quantile estimation to sample sizes between 30 and 150 and different distributions of the paired differences. The performance of 15 estimators in generating prediction intervals was measured by their respective coverage probability for one newly generated observation. Our results indicated that sample quantile estimators based on one or two order statistics outperformed all of the other estimators and they can be used for deriving nonparametric Limits of Agreement. For sample sizes exceeding 80 observations, more advanced quantile estimators, such as the Harrell–Davis and estimators of Sfakianakis–Verginis type, which use all of the observed differences, performed likewise well, but may be considered intuitively more appealing than simple sample quantile estimators that are based on only two observations per quantile.

1. Introduction

The classical Bland–Altman Limits of Agreement (BA LoA) define a range within which approximately 95% of normally distributed differences between paired measurements are expected to lie [1,2,3]. In cases of non-normally distributed differences, the use of empirical quantiles has been proposed as a robust alternative [2,4,5]; however, extensive research endeavors in the past have suggested the application of nonparametric quantile estimation to the assessment of 2.5% and 97.5% percentiles as nonparametric LoA. We performed a simulation study on 15 nonparametric quantile estimators to derive nonparametric prediction intervals and assessed their performance by means of the coverage probability for one newly generated observation. The aim of this study was to suggest a nonparametric and robust alternative to the classical BA LoA when the normality assumption does not hold and/or the sample sizes are small to moderate. Our findings are illustrated by an application to data from a previously published clinical study on coronary artery calcification measured by the Agatston score [6].

2. Methods

Let the differences of paired observations be independent observations of a random variable X with a cumulative distribution function (CDF) $F:\mathbf{R}\to [\mathbf{0},\mathbf{1}]$. If F is continuous from the right, then the quantile function of X is continuous from the left: $Q\left(u\right):=inf\{x:F(x)\ge u\},u\in (0,1)$; hence, at least $100u$ percent of the values of X are below $Q\left(u\right)$ [7,8]. In the following, we seek to estimate the 2.5% and 97.5% percentiles of F, corresponding to $u=0.025,0.975$, by different methods of nonparametric quantile estimation.

Databases, such as JSTOR (Journal Storage), ScienceDirect, the online journal platform “Taylor & Francis Online”, and those maintained by PubMed/Medline in the NCBI (National Center for Biotechnology Information) were searched for quantile estimator, nonparametric quantile estimator, nonparametric kernel quantile estimator, subsampling quantile estimator, and new quantile estimator. Fifteen nonparametric quantile estimators were chosen, three of which are sample quantile estimators, four are subsampling quantile estimators, two are kernel quantile estimators, and six are other quantile estimators. Hence, we chose very different types of nonparametric quantile estimators, which use one, two, or all the observations in a sample.

2.1. Sample Quantile Estimators

The simplest way to estimate quantiles nonparametrically is by using sample quantile estimators. A random sample $X_1,\dots X_n$ of size n is sorted in increasing order $X_{\left(1\right)}\le X_{\left(2\right)}\le \dots \le X_{\left(n\right)}$; the symbols here denote the order statistics of the random sample. The CDF of F can then be estimated by the step function

$$\tilde{F}\left(x\right)=\frac{1}{n}\sum _{i=1}^n{I}_{\left[x_i,\infty \right)}\left(x\right),$$

(1)

where $I_A\left(x\right)$ is the indicator function that takes the value 1 if $x\in A$ and 0 if $x\notin A$; the sample quantile function is defined as $\tilde{Q}\left(u\right):=inf\{x:\tilde{F}\left(x\right)\ge u\},u\in (0,1)$ by Cheng [7]. The quantile estimator

$$\begin{array}{c}\hfill S{Q}_{p1}=\left\{\begin{array}{cc}{X}_{\left(np\right)}\hfill & \mathrm{if}\left[np\right]=np\hfill \\ X_{(\left[np\right]+1)}\hfill & \mathrm{if}\left[np\right]<np\hfill \end{array}\right.\end{array}$$

(2)

is based on a single order statistic, where $\left[x\right]$ is the greatest integer that is less than or equal to x [9,10]. $S{Q}_{p1}$ is the smallest observation for which at least p percent of the observed values in the sample are smaller than or equal to $S{Q}_{p1}$.

The second sample quantile estimator $S{Q}_{p2}$ is a weighted average of the two order statistics that are closest to including p percent of all the observations in the sample:

$$\begin{array}{c}\hfill S{Q}_{p2}=(1-\alpha )X_{\left(r\right)}+\alpha X_{(r+1)}\end{array}$$

(3)

with $\alpha =p(n+1)-r$ and $r=\left[p(n+1)\right]$ [9,11].

Finally, we considered a weighted average of $X_{\left[np+0.5\right]}$ and $X_{\left[np+0.5\right]+1}$:

$$\begin{array}{c}\hfill SQ{I}_p=(-np+0.5+i)X_{\left(i\right)}+(np+0.5-i)X_{(i+1)}\end{array}$$

(4)

with $i=[np+0.5]$ and $0.5\le np\le (n-0.5)$ [9,12].

2.2. Subsampling Quantile Estimators

The abovementioned sample quantile estimators are based on only one or two order statistics whereas subsampling, kernel, and other quantile estimators employ linear combinations of all the available order statistics, weighting them according to their relative closeness to the target percentile. Based on the sample quantile function $\tilde{Q}$, linear smooth nonparametric estimators of $Q\left(u\right)$ can be written as

$$Q\left(u\right)={\int }_0^1\tilde{Q}\left(t\right)d_{t}G(u;t),$$

(5)

where $G(u;·)$ is a CDF with support on the unit interval. Many distributions $G(u;·)$ have been proposed, the choice of which depends on the sample size and, typically, a smoothing parameter. Depending on the choice of $G(u;·)$, there are two major classes of quantile estimators according to Cheng [7]: subsampling quantile estimators and kernel quantile estimators. Both can be given as L-statistics, which is, ${\displaystyle \sum _{j=1}^n{W}_j·X_{\left(j\right)}}$, where $W_j$ and $X_{\left(j\right)}$ is the weight for the j-th order statistic and the j-th order statistic itself, respectively [13]. For subsampling quantile estimators, a discrete distribution is chosen for $G(u;·)$, interpreted as a resampling distribution from the set of the observed order statistics $X_{\left(j\right)},j=1,\dots ,n$ [7].

The Harrell-Davis estimator is given by

$$\begin{array}{c}\hfill H{D}_p=\sum _{i=1}^n{W}_i{X}_{\left(i\right)}\end{array}$$

(6)

with weight function

$$\begin{array}{cc}\hfill W_i& {\displaystyle =\frac{1}{\beta \left\{(n+1)p,(n+1)(1-p)\right\}}{\int }_{(i-1)/n}^{i/n}{y}^{(n+1)p-1}{(1-y)}^{(n+1)(1-p)-1}dy}\hfill \\ \hfill & =I_{i/n}\left\{p(n+1),(1-p)(n+1)\right\}-I_{(i-1)/n}\left\{p(n+1),(1-p)(n+1)\right\},\hfill \end{array}$$

where $I_{i/n}\left\{a,b\right\}$ is the incomplete beta function [10,14,15].

The quantile estimator of Kaigh and Lachenbruch

$$\begin{array}{c}\hfill K{L}_p=\left[\sum _{j=r}^{r+n-k}\left(\genfrac{}{}{0pt}{}{j-1}{r-1}\right)\left(\genfrac{}{}{0pt}{}{n-j}{k-r}\right)/\left(\genfrac{}{}{0pt}{}{n}{k}\right)\right]X_{\left(j\right)},\end{array}$$

(7)

where $r=\left[\right(k+1\left)p\right]$, is obtained by averaging a subsample quantile estimate over all $\left(\genfrac{}{}{0pt}{}{k}{n}\right)$ subsamples of size $k,1\le k\le n$, which are sampled without replacement. The subsample size k is an arbitrary smoothing (or reduction) parameter, and Kaigh and Lachenbruch proposed choosing k, so as to minimize the mean squared error (MSE), which is, $MSE=E{(K{L}_p-{\epsilon }_p)}^2$, where ${\epsilon }_p$ is the true value of the p-th quantile [7,16,17].

Kaigh and Cheng [18] proposed the quantile estimator

$$\begin{array}{c}\hfill K{C}_p=\sum _{j=1}^n\left[\left(\genfrac{}{}{0pt}{}{r+j-2}{r-1}\right)\left(\genfrac{}{}{0pt}{}{n-j+k-r}{k-r}\right)/\left(\genfrac{}{}{0pt}{}{n+k-1}{k}\right)\right]X_{\left(j\right)}\end{array}$$

(8)

with $r=⌈kp⌉$, where $⌈x⌉$ denotes the smallest integer greater than or equal to x [7]. The value of k is again determined by minimizing the MSE of the estimator.

Finally, the Bernstein polynomial quantile estimator is given by

$$\begin{array}{c}\hfill B{P}_p=\sum _{j=1}^n\left(\genfrac{}{}{0pt}{}{n-1}{j-1}\right)p^{j-1}{(1-p)}^{n-j}{X}_{\left(j\right)}\end{array}$$

(9)

according to Cheng [7,19].

2.3. Kernel Quantile Estimators

Like subsampling quantile estimators, kernel quantile estimators can be written in the form of Equation (5). Here, $G(u;.)$ is a location-scale family CDF with density function K, location parameter u, and scale parameter $h\left(n\right)$:

$$\begin{array}{c}\hfill G(u;t)=\frac{1}{h\left(n\right)}K\left(\frac{t-u}{h\left(n\right)}\right).\end{array}$$

Subsequently, Equation (5) becomes the kernel quantile estimator introduced by Parzen [8]:

$$\begin{array}{c}\hfill \widehat{Q}\left(u\right)={\int }_0^1\tilde{Q}\left(t\right)·\frac{1}{h\left(n\right)}·K\left(\frac{t-u}{h\left(n\right)}\right)dt\end{array}$$

with its L-statistic representation being

$$\begin{array}{c}\hfill \widehat{Q}\left(u\right)=\sum _{j=1}^n\left[{\int }_{(j-1)/n}^{j/n}\frac{1}{h\left(n\right)}·K\left(\frac{t-u}{h\left(n\right)}\right)dt\right]X_{\left(j\right)}.\end{array}$$

The density K, called the kernel, is symmetric around zero and it has a chosen bandwidth$h\left(n\right)$, which satisfies $h\left(n\right)\to \infty$ when $n\to \infty$ [7,20]. Yang [21] proposed a discretized version, which we used as the first of the two kernel quantile estimators in our study due to its closed form:

$$\begin{array}{c}\hfill K{Q}_{p1}=\frac{1}{n}\sum _{j=1}^n\frac{1}{h\left(n\right)}·K\left(\frac{(j/n)-p}{h\left(n\right)}\right)X_{\left(j\right)}.\end{array}$$

(10)

As $K\left(\frac{1/n-p}{h\left(n\right)}\right),\dots ,K\left(\frac{n/n-p}{h\left(n\right)}\right)$ do not generally provide a (discrete) probability distribution on $[0,1]$, monotonicity, translation and scale equivariance, and the symmetry relation do not hold. Translation and scale equivariance would imply that $K{Q}_{p1}$ applied to $(X_1+c,\dots ,X_n+c)$ is equal to c plus $K{Q}_{p1}$ applied to $(X_1,\dots ,X_n)$; the asymmetry means that $K{Q}_{p1}$ applied to $(-X_1,\dots ,-X_n)$ is not equal to $-K{Q}_{p1}$ applied to $(X_1,\dots ,X_n)$. The Nadaraya–Watson type estimator

$$\begin{array}{c}\hfill K{Q}_{p2}=\sum _{j=1}^n\frac{K\left(\frac{\left(\right(j-0.5)/n)-p}{h\left(n\right)}\right)}{{\sum }_{i=1}^{n}K\left(\frac{\left(\right(i-0.5)/n)-p}{h\left(n\right)}\right)}{X}_{\left(j\right)}\end{array}$$

(11)

Overcomes these drawbacks and was used as the second of the two kernel quantile estimators in this study [7,22,23]. Its name originates from the Nadaraya–Watson estimator in kernel regression [24,25].

In the following, we chose the standard Gaussian kernel for K and used the value of the bandwith $h\left(n\right)$ that minimized the MSE to find k for both $K{L}_p$ in Equation (7) and $K{C}_p$ in Equation (8).

2.4. Other Quantile Estimators

The kernel quantile estimators that are presented in the previous section are all based on the usual empirical distribution (1) with equal weights $1/n$ assigned to each observation. To improve the performance of quantile estimators, Huang and Brill [26] proposed using a weighted empirical distribution, for instance, the level crossing empirical distribution function

$$\begin{array}{c}\hfill F_{lc}\left(x\right)=\sum _{i=1}^n{w}_{i,n}{I}_{(-\infty ,x]}\left(X_{\left(i\right)}\right)\end{array}$$

(12)

with weight function

$$\begin{array}{c}\hfill w_{i,n}=\left\{\begin{array}{cc}\frac{1}{2}\left[1-\frac{n-2}{\sqrt{n(n-1)}}\right]\hfill & \mathrm{if}i=1,n\hfill \\ \frac{1}{\sqrt{n(n-1)}}\hfill & \mathrm{if}i=2,3...,n-1.\hfill \end{array}\right.\end{array}$$

(13)

One such kernel quantile estimator using level crossing empirical distributions is

$$\begin{array}{c}\hfill K{Q}_{plc}=\sum _{i=1}^n\left[n^{-1}\frac{1}{h\left(n\right)}K\left(\frac{{\displaystyle \sum _{j=1}^i{w}_{j,n}-p}}{h\left(n\right)}\right)\right]X_{\left(i\right)}.\end{array}$$

(14)

Huang [27] modified the Harrell–Davis estimator (6) by applying a weighted empirical distribution function instead of the empirical distribution with equal weights $1/n$. The Harrell–Davis estimator using a level crossing empirical distribution function can be written as

$$\begin{array}{cc}\hfill H{D}_{plc}& =\frac{1}{\beta \left(\right(n+1)p,(n+1\left)q\right)}{\int }_0^1{F}_{lc}^{-1}\left(y\right)y^{(n+1)p-1}{(1-y)}^{(n+1)q-1}dy\hfill \\ \hfill & =\sum _{i=1}^n\left[{\int }_{q_{i-1,n}}^{q_{i,n}}\frac{1}{\beta \left(\right(n+1)p,(n+1\left)q\right)}{y}^{(n+1)p-1}{(1-y)}^{(n+1)q-1}dy\right]X_{\left(i\right)},\hfill \end{array}$$

(15)

where $\beta (·,·)$ is the beta function, $q=1-p$, $F_{lc}(·)$ is given by (12), ${\displaystyle q_{i,n}=\sum _{j=1}^i{w}_{j,n}}$, $i=1,\dots ,n$, with $w_{j,n}$, as defined in (13), and $q_{0,n}\equiv 0$.

Sfakianakis and Verginis [28] proposed a group of estimators, motivated by the fact that nonparametric quantile estimation of extreme quantiles close to 0 and 1 requires large samples for sufficient accuracy. These three quantile estimators are supposed to better estimate quantiles in the tails of a distribution when using small samples and they employ the Binomial probability of observing exactly i out of n events with an event probability of p, $B(i;n,p)$:

$$\begin{array}{cc}\hfill S{V}_{p1}& =\frac{2B(0;n,p)+B(1;n,p)}{2}{X}_{\left(1\right)}+\frac{B(0;n,p)}{2}{X}_{\left(2\right)}-\frac{B(0;n,p)}{2}{X}_{\left(3\right)}\hfill \\ & +\sum _{i=2}^{n-1}\frac{B(i;n,p)+B(i-1;n,p)}{2}{X}_{\left(i\right)}\hfill \\ & -\frac{B(n;n,p)}{2}{X}_{(n-2)}+\frac{B(n;n,p)}{2}{X}_{(n-1)}+\frac{2B(n;n,p)+B(n-1;n,p)}{2}{X}_{\left(n\right)},\hfill \end{array}$$

(16)

$$S{V}_{p2}=\sum _{i=0}^{n-1}B(i;n,p)X_{(i+1)}+(2X_{\left(n\right)}-X_{(n-1)})B(n;n,p),$$

(17)

$$S{V}_{p3}=\sum _{i=1}^{n}B(i;n,p)X_{\left(i\right)}+(2X_{\left(1\right)}-X_{\left(2\right)})B(0;n,p).$$

(18)

Finally, Navruz and Özdemir [29] introduced a new quantile estimator, which is a weighted average of all order statistics:

$$\begin{array}{cc}\hfill N{O}_p& =(B(0;n,p)2p+B(1;n,p)p)X_{\left(1\right)}+B(0;n,p)(2-3p)X_{\left(2\right)}-B(0;n,p)(1-p)X_{\left(3\right)}\hfill \\ \hfill & +\sum _{i=1}^{n-2}(B(i;n,p)(1-p)+B(i+1;n,p)p)X_{(i+1)}-B(n;n,p)p{X}_{(n-2)}\hfill \\ \hfill & +B(n;n,p)(3p-1)X_{(n-1)}+(B(n-1;n,p)(1-p)+B(n;n,p)(2-2p))X_{\left(n\right)}.\hfill \end{array}$$

(19)

2.5. Simulation Setup

We contrasted nonparametric LoA as constructed with the 15 abovementioned quantile estimators by comparing their coverage probabilities for the next paired difference under the given distributional assumption. Here, we employed the standard normal distribution (ND), a standard normal distribution with 1%, 2%, and 5% outliers (ND 1%, ND 2%, and ND 5%, respectively), an exponential distribution (ED) with a rate of 1, and a lognormal distribution (LND) with meanlog = 0 and sdlog = 1. For normal distributions comprising outliers, simulated data were replaced with a probability of 1%, 2%, and 5% by data sampled from a normal distribution with a mean of 0 and a standard deviation of 3. To examine small to moderate sample sizes, the sample size was set to 30, 50, 80, 100, and 150. For each combination of distribution, sample size, and nonparametric quantile estimator, 20,000 simulated trials of size $(n+1)$ were generated with R (the code is available as Supplemental Material S1). Here, a seed was set in order to use the same simulated data for each combination of distribution and sample size across nonparametric quantile estimators. The first n observations in each simulated trial were used to derive nonparametric LoA, to which the last observation was compared. The coverage probability was then the proportion of cases out of the 20,000 trials where the nonparametric LoA included the last observation. All of the figures were generated with Stata/MP 16.1 (College Station, TX 77845, USA).

3. Results

For $n=30$ (

Table 1. Coverage probabilities for nonparametric Limits of Agreement ($n=30$). Neither $S{Q}_{p2}$ nor $K{L}_p$ are defined for $n<40$.

Estimator	ND	ND 1%	ND 2%	ND 5%	ED	LND
$S{Q}_{p1}$	0.937	0.938	0.937	0.937	0.934	0.937
$S{Q}_{p2}$	-	-	-	-	-	-
$SQ{I}_p$	0.926	0.927	0.927	0.928	0.921	0.926
$H{D}_p$	0.911	0.923	0.923	0.924	0.916	0.920
$K{L}_p$	-	-	-	-	-	-
$K{C}_p$	0.900	0.890	0.877	0.851	0.885	0.880
$B{P}_p$	0.905	0.908	0.908	0.909	0.897	0.904
$K{Q}_{p1}$	0.904	0.882	0.865	0.832	0.936	0.922
$K{Q}_{p2}$	0.912	0.893	0.880	0.857	0.916	0.906
$K{Q}_{plc}$	0.915	0.893	0.874	0.838	0.923	0.919
$H{D}_{plc}$	0.916	0.917	0.917	0.919	0.910	0.915
$S{V}_{p1}$	0.924	0.925	0.925	0.926	0.920	0.924
$S{V}_{p2}$	0.925	0.927	0.926	0.927	0.914	0.919
$S{V}_{p3}$	0.925	0.926	0.925	0.926	0.923	0.929
$N{O}_p$	0.813	0.814	0.817	0.821	0.808	0.834

), none of the estimators reached the nominal coverage probability of 0.95. The coverage probability of $S{Q}_{p1}$ was closest to 0.95, ranging from 0.934 to 0.938. Note that, for sample sizes of up to $n=40$ observations, the smallest and largest difference are used as nonparametric quantile estimates for the 2.5% and 97.5% percentiles, respectively. For $SQ{I}_p$, $H{D}_p$, $H{D}_{plc}$, $S{V}_{p1}$, $S{V}_{p2}$, and $S{V}_{p3}$, the coverage probabilities were at least 0.921, 0.911, 0.910, 0.920, 0.914, and 0.923, respectively. Neither $S{Q}_{p2}$ nor $K{L}_p$ are defined for $n<40$.

$S{Q}_{p2}$ was the only estimator with coverage probabilities oscillating closely around 0.95 for all the investigated sample sizes $n\ge 50$ (

Table 2. Coverage probabilities for nonparametric Limits of Agreement ($n=50$). Bold figures indicate coverage probabilities exceeding the nominal level of 0.95.

Estimator	ND	ND 1%	ND 2%	ND 5%	ED	LND
$S{Q}_{p1}$	0.919	0.918	0.920	0.919	0.919	0.919
$S{Q}_{p2}$	0.951	0.952	0.953	0.955	0.951	0.951
$SQ{I}_p$	0.934	0.935	0.937	0.938	0.931	0.934
$H{D}_p$	0.939	0.940	0.942	0.945	0.935	0.938
$K{L}_p$	0.938	0.932	0.930	0.917	0.937	0.929
$K{C}_p$	0.924	0.906	0.898	0.879	0.915	0.912
$B{P}_p$	0.922	0.925	0.927	0.931	0.919	0.922
$K{Q}_{p1}$	0.919	0.902	0.896	0.861	0.931	0.933
$K{Q}_{p2}$	0.924	0.909	0.900	0.881	0.924	0.916
$K{Q}_{plc}$	0.924	0.912	0.901	0.867	0.944	0.930
$H{D}_{plc}$	0.933	0.935	0.936	0.940	0.928	0.931
$S{V}_{p1}$	0.940	0.941	0.942	0.945	0.935	0.939
$S{V}_{p2}$	0.937	0.939	0.939	0.943	0.933	0.934
$S{V}_{p3}$	0.938	0.941	0.942	0.944	0.938	0.941
$N{O}_p$	0.839	0.843	0.845	0.855	0.845	0.870

,

Table 3. Coverage probabilities for nonparametric Limits of Agreement ($n=80$). Bold figures indicate coverage probabilities exceeding the nominal level of 0.95.

Estimator	ND	ND 1%	ND 2%	ND 5%	ED	LND
$S{Q}_{p1}$	0.939	0.939	0.940	0.938	0.934	0.939
$S{Q}_{p2}$	0.950	0.951	0.951	0.949	0.945	0.950
$SQ{I}_p$	0.941	0.941	0.942	0.941	0.935	0.941
$H{D}_p$	0.950	0.952	0.954	0.955	0.940	0.949
$K{L}_p$	0.943	0.939	0.935	0.933	0.938	0.939
$K{C}_p$	0.936	0.923	0.916	0.895	0.925	0.929
$B{P}_p$	0.937	0.939	0.941	0.942	0.929	0.937
$K{Q}_{p1}$	0.934	0.926	0.921	0.887	0.934	0.940
$K{Q}_{p2}$	0.936	0.925	0.917	0.897	0.930	0.931
$K{Q}_{plc}$	0.933	0.928	0.922	0.890	0.942	0.939
$H{D}_{plc}$	0.943	0.945	0.947	0.948	0.934	0.943
$S{V}_{p1}$	0.951	0.952	0.954	0.955	0.940	0.949
$S{V}_{p2}$	0.950	0.951	0.953	0.953	0.940	0.948
$S{V}_{p3}$	0.949	0.951	0.952	0.954	0.940	0.950
$N{O}_p$	0.888	0.893	0.896	0.901	0.891	0.910

,

Table 4. Coverage probabilities for nonparametric Limits of Agreement ($n=100$). Bold figures indicate coverage probabilities exceeding the nominal level of 0.95.

Estimator	ND	ND 1%	ND 2%	ND 5%	ED	LND
$S{Q}_{p1}$	0.941	0.941	0.941	0.942	0.941	0.941
$S{Q}_{p2}$	0.952	0.952	0.952	0.954	0.953	0.952
$SQ{I}_p$	0.941	0.941	0.941	0.942	0.941	0.941
$H{D}_p$	0.950	0.951	0.953	0.957	0.948	0.950
$K{L}_p$	0.947	0.941	0.939	0.938	0.946	0.941
$K{C}_p$	0.940	0.929	0.920	0.902	0.935	0.935
$B{P}_p$	0.940	0.942	0.944	0.948	0.937	0.940
$K{Q}_{p1}$	0.937	0.935	0.921	0.896	0.942	0.943
$K{Q}_{p2}$	0.939	0.930	0.920	0.903	0.940	0.935
$K{Q}_{plc}$	0.938	0.935	0.932	0.905	0.940	0.941
$H{D}_{plc}$	0.944	0.946	0.948	0.951	0.942	0.944
$S{V}_{p1}$	0.950	0.952	0.954	0.959	0.948	0.950
$S{V}_{p2}$	0.950	0.952	0.953	0.957	0.948	0.949
$S{V}_{p3}$	0.950	0.951	0.953	0.959	0.947	0.950
$N{O}_p$	0.911	0.914	0.917	0.924	0.913	0.924

and

Table 5. Coverage probabilities for nonparametric Limits of Agreement ($n=150$). Bold figures indicate coverage probabilities exceeding the nominal level of 0.95.

Estimator	ND	ND 1%	ND 2%	ND 5%	ED	LND
$S{Q}_{p1}$	0.945	0.945	0.944	0.946	0.946	0.945
$S{Q}_{p2}$	0.949	0.949	0.948	0.951	0.950	0.949
$SQ{I}_p$	0.942	0.942	0.941	0.943	0.944	0.942
$H{D}_p$	0.948	0.949	0.950	0.954	0.948	0.948
$K{L}_p$	0.945	0.941	0.939	0.936	0.947	0.945
$K{C}_p$	0.941	0.934	0.930	0.906	0.939	0.940
$B{P}_p$	0.942	0.944	0.943	0.947	0.941	0.942
$K{Q}_{p1}$	0.940	0.936	0.928	0.903	0.943	0.943
$K{Q}_{p2}$	0.940	0.932	0.923	0.908	0.941	0.937
$K{Q}_{plc}$	0.939	0.935	0.928	0.904	0.941	0.943
$H{D}_{plc}$	0.944	0.946	0.945	0.949	0.944	0.944
$S{V}_{p1}$	0.949	0.951	0.952	0.957	0.949	0.949
$S{V}_{p2}$	0.949	0.952	0.952	0.956	0.948	0.950
$S{V}_{p3}$	0.949	0.950	0.951	0.954	0.947	0.949
$N{O}_p$	0.934	0.937	0.937	0.943	0.936	0.940

); for $n=50$, $S{Q}_{p2}$ was the only one to do so.

For $n=80$ (Table 3), the coverage probabilities of $H{D}_p$, $S{V}_{p1}$, $S{V}_{p2}$, and $S{V}_{p3}$ fluctuated closely around the nominal level except for the simulations with an ED (0.94). For $n\ge 100$ (Table 4 and Table 5), these estimators performed close to the 0.95 nominal level for all of the investigated distributions.

The coverage probabilities of the recently proposed $N{O}_p$ estimator varied between 0.945 and 0.950 for $n=200$ and they were very close to 0.95 for $n=250$ (results not shown here).

4. Example

Diederichsen et al. [6] compared coronary artery calcification measurements using the Agatston score with the measurements using Framingham Heart Score in Danes of 50 and 60 years of age. Of 1825 randomly sampled citizens, 1257 consented to participation in the study, and 1156 of them were eligible. Agatston scores were independently reanalyzed for 129 randomly chosen study participants, and the agreement measures were the proportions of agreement and the kappa statistics for dichotomized calcification status (absence vs. presence) to assess intra- and inter-rater agreement. In the following, the intra-rater differences are used for exemplification purposes.

Approximately half of the 129 participants had an Agatston score of 0. The paired intra-rater differences ranged from −683 to 130, with a first, second, and third quartile being equal to 0; the 5th, 10th, 90th, and 95th percentiles were −23, −12, 1.1, and 5, respectively. The empirical distribution of the paired differences was, therefore, characterized by its denseness around 0 and a single, comparatively extreme outlier, clearly indicating the inappropriateness of the normality assumption in this setting (see also a histogram including an approximating normal distribution as Supplemental Material S2).

Using $S{Q}_{p2}$, $H{D}_p$, and $S{V}_{p1}$, the nonparametric, asymmetric, and robust LoA are −61.5, 12.8; −96.2, 26.7; and, −122.1, 30.6, respectively, whereas the symmetric BA LoA of −129.8, 116.1 are equidistant from the estimated mean difference of −6.9 (Figure 1). The upper LoA for $H{D}_p$ and $S{V}_{p1}$ are similar, but the respective lower LoA are differently affected by the single outlier (3942.5, −683). $S{Q}_{p2}$ appears to be most robust to few outliers due to its definition. The R source code for the derivation of these nonparametric LoA as well as the example data can be found as Supplemental Material S3 and S4, respectively.

The sensitivity of the classical BA LoA to outliers becomes crystal-clear when excluding the single outlier here. Subsequently, the symmetric BA LoA are −37.4 and 34.3, and the estimated mean difference reduces to −1.6 (results not shown here). In practice, outliers are, though, kept in the analysis dataset if there is no reasonable explanation for an exclusion. This underlines the importance of robust alternatives to the BA LoA.

5. Discussion

5.1. Statement of Principal Findings

The simple sample quantile estimators that are based on one and two order statistics performed closest to the nominal level in terms of the coverage probability for the next observation across six distributional scenarios for $n=30$ and $n=50,80,100,150$. The Harrell–Davis subsampling estimator and estimators of the Sfakianakis–Verginis type followed closely for sample sizes of at least $n=80$ and may be considered intuitively more appealing, as they use the entire sample, whereas more simple and outlier-robust sample quantile estimators are only based on a few observations from the sample.

5.2. Strengths and Limitations of The Study

The choice of distributions for the simulation study was motivated by our own experience with agreement assessments in clinical studies, especially roughly normal distributions with a few percent outliers. We investigated a wide range of quantile estimators, comprising sample, subsampling, and kernel quantile estimators as well as other methods for quantile estimation with sample sizes between 30 and 150. As a measure of performance, we considered the coverage probability of nonparametric LoA for the next observation, interpreting the nonparametric LoA as a prediction interval, as the lower and upper LoA need to be simultaneously assessed. Therefore, we did not pursue evaluations using, for instance, mean squared errors.

5.3. Strengths and Limitations in Relation to other Studies

A peculiarity of LoA is the sole focus on the 2.5th and 97.5th quantiles, two extreme quantiles. Dielman, Lowry and Pfaffenberger [9] investigated 0.02 and 0.98, but only for small samples ($n=10,15,25,30$). Others examined 0.05 and 0.95 [11,14,16,21,22,29], whereas Kaigh and Cheng [17,18] assessed 0.1 and 0.9. Only Huang and Brill [26,27] also targeted 0.025 and 0.975, but only for samples of maximum size $n=30$. Sfakianakis and Verginis [28] analyzed 0.01 and 0.99 as well as 0.05 and 0.95 in various sample sizes.

When compared to the usual number of 2000 iterations, the chosen number of 20,000 iteration runs translated for a given nonparametric estimator and sample size into a reduced range of the coverage probabilities across distributions by approximately 0.005 and is deemed appropriately accurate. However, the increased number of iterations did implicate considerably longer running times in creating the data for one Table (12 as opposed to 2 h). The abovementioned studies employed between 1000 and 10,000 iterations [16,22].

Harrell and Davis [14] did not recommend $H{D}_p$ for small n and extreme p, and Dielman, Lowry and Pfaffenberger [9] concluded that there was not one best estimator across scenarios, based on maximum sample sizes of 30 and 60; however, Dielman, Lowry and Pfaffenberger [9] suggested that $H{D}_p$ performs well in a wide range of cases, except when $p=0.02,0.98$. Our findings for $H{D}_p$ are in line with these former conclusions, but extend to larger sample sizes of $n=80,100,150$, in which $H{D}_p$ appears to be a preferable choice for estimating extreme quantiles.

5.4. Meaning of the Findings: Possible Mechanisms and Implications

Our findings suggest using $S{Q}_{p1}$ in small samples with approximately $n=30$ but $SQ{I}_p$, $S{V}_{p1}$, or $S{V}_{p3}$ may be preferential alternatives as $S{Q}_{p1}$ simply reduces to the smallest and largest observations as estimates for the 2.5% and 97.5% quantiles, respectively. The latter is, in turn, unfortunate in the case of outliers due to their unabated impact on the estimates. $S{Q}_{p2}$ performed closest to the nominal level for all samples with $n\ge 50$ and appeared to be less prone to the single outlier in our clinical example than $H{D}_p$ and $S{V}_{p1}$. However, the latter two estimators do involve all the observations. $S{Q}_{p2}$ can, therefore, be considered the first choice for samples of approximately $n=50$, but, for larger n, both $H{D}_p$ and Sfakianakis–Verginis type estimators are equally applicable and actually preferable if the researcher seeks to include the entire dataset in quantile estimation and not only pairs of order statistics.

The normality assumption of the paired differences may often be considered to be reasonable in the planning stage; however, alternative quantile estimators should be equally specified in the planning stage as empirical distributions may deviate notably from ideal assumptions. Moreover, our investigation suggests several beneficial nonparametric alternatives to BA LoA instead of the simple percentile estimators that currently seem to prevail.

5.5. Unanswered Questions and Future Research

In the case of normally distributed paired differences, Bland and Altman [1,2] have already proposed approximate confidence intervals for the BA LoA. Recently, Vock [30] emphasized that only a tolerance interval or the outer confidence limits for BA LoA can provide a range that will contain a specified percentage of future differences with a known certainty. Carkeet and Goh [31,32] proposed exact confidence intervals for BA LoA, while using two-sided tolerance factors for a normal distribution.

In the case of any given distribution for the paired differences, several approaches for the construction of nonparametric confidence intervals for quantiles have been proposed over half a century [33,34,35,36,37,38,39]. For both $H{D}_p$ and $K{L}_p$, confidence intervals for quantiles have been proposed [10,14,16]. In the context of nonparametric LoA, future research will naturally lie in the proposal and evaluation of confidence intervals for the 2.5% and 97.5% quantiles in small-to-moderate samples, especially with regard to $S{Q}_{p2}$, $SQ{I}_p$, $H{D}_p$, and Sfakianakis–Verginis type estimators.

Regression procedures for method comparison analysis have not been considered here [40,41,42,43]. Robust methods designed for data configurations with outliers, such as S- or MM-estimation, Least Trimmed Squares, or the Forward Search, are of interest in this context [44,45,46,47,48].