Correcting Diagnostic Test Sensitivity and Specificity for Patient Misclassifications Resulting from Use of an Imperfect Reference Standard

Abstract

Investigational diagnostic tests are validated by using a reference standard (RS). If the RS is imperfect (i.e., it has sensitivity [Se] and/or specificity [Sp] < 1), incorrect values for the investigational test’s Se and Sp may result because of patient misclassification by the RS. Formulas were derived to correct a test’s Se and Sp that were determined by using an imperfect RS. The following derived formulas correct for misclassification and give the true numbers of disease-positive [n_DP] and disease-negative patients [n_DN] from the apparent number of disease-positive and disease-negative patients (an_DP and an_DN), and the Se and Sp of the RS (Se_R, Sp_R): n_DP = (an_DP × Sp_R + an_DN × Sp_R − an_DN)/J_R; n_DN = (an_DP × Se_R + an_DN × Se_R − an_DP)/J_R, where J_R is Youden’s Index for the RS (J_R = Se_R + Sp_R − 1). The following derived formulas give the correct Se and Sp of an investigational test (Se_I and Sp_I): Se_I = (an_TPI × Sp_R − n_DP × Se_R × Sp_R + n_DP × J_R + n_DN × Sp_R² − n_DN × Sp_R − Sp_R × an_TNI + an_TNI)/(n_DP × J_R); Sp_I = (an_TPI − an_TPI × Se_R + n_DP × Se_R² −  n_DP × Se_R − Se_R × n_DN × Sp_R + n_DN × J_R + Se_R × an_TNI)/(n_DN × J_R), where an_TPI is the apparent number of true-positive test results, and an_TNI is the apparent number of true-negative test results. The derived formulas correct for patient misclassification by an imperfect RS and give the correct values of a diagnostic test’s Se and Sp.

1. Introduction

Sensitivity (Se) and specificity (Sp) are fundamental measures of the performance of a diagnostic test. They, respectively, provide the probability of a positive test result in patients with the index disease (the disease that the test is designed to detect or exclude) and a negative test result in patients without the disease [1]. When combined with pre-test probability (prevalence) of disease in a tested population, they can be used to determine the more clinically useful metrics positive predictive value (PPV) and negative predictive value (NPV) [2]. Alternatively, Se and Sp can be used to determine likelihood ratios, which can be used with the pre-test odds (rather than probability) of disease to estimate the post-test odds of disease [3].

The Se and Sp of an investigational test (IT) must therefore be accurately determined before the test can be useful in clinical practice. Typically, accuracy is determined through one or more clinical studies in which study patients undergo both the IT and another diagnostic test, which serves as a reference standard (RS). The RS is used to classify each patient as disease-positive (DP) or disease-negative (DN), and these classifications are used to classify the patient’s IT result as true positive (TP), true negative (TN), false positive (FP), or false negative (FN). The respective numbers of patient classifications (n_TP, n_TN, n_FP, and n_FN) are used to determine the sensitivity and specificity of the investigational test (Se_I and Sp_I, respectively; Formulas (1) and (2)).

Se_I = n_TP/n_DP

(1)

Sp_I = n_TN/n_DN

(2)

If the RS is perfect (Se_R = Sp_R = 1 [100%]), then it is a true standard of truth, and each patient is correctly classified as DP or DN. Se_I and Sp_I can then be determined straightforwardly and correctly. When a perfect RS is used to evaluate a patient, there are only two possible outcomes for the RS (either a true-positive RS result classifying the patient as DP or a true-negative RS result classifying the patient as DN); and when a perfect RS is used to classify positive and negative IT results, there can only be four possible IT classifications (TP, TN, FP, and FN).

However, if the RS is imperfect (i.e., Se_R and/or Sp_R < 1 [<100%]), then it can misclassify patients, resulting in incorrect numbers of patients with and without disease (n_DP, n_DN, n_TP, and n_TN), and thus incorrect values of Se_I and Sp_I. In this case, there can be four (rather than two) possible classifications of a patient by the imperfect RS: a true-positive result classifying the patient as DP, a true-negative result classifying the patient as DN, a false-positive result misclassifying the patient as DP, or a false-negative result misclassifying the patient as DN. When the imperfect RS is used to classify the positive and negative IT results, the IT can yield true or false results, which could agree or disagree with the RS, which in turn could be true or false. As a result, there could be eight categories: TP, TN, FP, and FN from comparison of the IT results to accurate RS results, plus apparent TP, TN, FP, and FN from comparison of the IT results to inaccurate RS results. Consequently, determination of the true Se and Sp of the IT when an imperfect RS is used is not as straightforward as when the RS is perfect.

An example of an imperfect reference standard that inspired this paper was the report by McKeith and coworkers of a clinical study that determined the Se and Sp of [¹²³I]ioflupane with single-photon emission computed tomography (SPECT) for assessing patients with suspected dementia with Lewy bodies (DLB) [4]. In that study, the International Consensus Criteria (ICC [5]) for diagnosing DLB were used as the RS. However, it was known from a validation study [6] that the Se and Sp of the ICC were less than perfect (0.83 and 0.95, respectively). Thus, the true Se and Sp of ioflupane for DLB were uncertain. For this reason, I sought ways to adjust the apparent values of Se and Sp of ioflupane imaging, as reported by McKeith et al. [5], to account for the known Se and Sp of the diagnostic criteria used as the reference standard.

A literature search yielded several relevant articles. Nihashi et al. [7] reported using a Bayesian latent class model for adjusting the Se and Sp of DaTscan for DLB for eight clinical studies (including follow-up data from the McKeith 2007 study [4], but not the original data), although neither the published article nor the Supplemental Materials provided sufficient details to allow one to reproduce their results.

Umemneku Chikere et al. [8] discussed three methods of correcting for the effects of an imperfect RS: Brenner [9], Gart and Buck [10], and Staquet et al. [11]. All of these authors took different approaches and reported different equations. They did not report enough detail to allow one to determine if their derivations were correct.

Trikalinos et al. mentioned the possibility of adjusting results that are based on an imperfect RS but did not report derivation of the formulas needed to do so [12].

Therefore, this work was initiated to derive formulas needed to correct for patient misclassifications by an imperfect RS and to determine the true values of a diagnostic test’s Se and Sp when they were determined by using an imperfect RS, in diagnostic terms readily understandable to clinicians, with full transparency of the derivations. Those results are reported here; application of the results to the McKeith ioflupane study [4] along with a review of relevant literature, will be reported separately.

2. Materials and Methods

Formulas were derived on the basis of an analysis of how a reference standard (RS) is used to classify patients as disease-positive or disease-negative and how misclassifications by an imperfect RS affect the apparent values of Se_I and Sp_I. Throughout, conditional independence of the RS and IT is assumed; i.e., the RS and IT misclassify patients independently. This assumption is reasonable if, for example, the RS and IT work by different mechanisms.

Two diagrams were created to depict patient misclassifications by the RS (Figure 1) and their effect on the apparent values of Se_I and Sp_I (Figure 2). In both figures, the prefix a was used to denote an apparent value. In Figure 2, to differentiate between the reference and investigational tests with respect to the numbers of true positives (n_TP), true negatives (n_TN), false positives (n_FP), false negatives (n_FN), Se, and Sp, these variables had the subscripts R (for reference test) or I (for investigational test) added.

Figure 1 shows how an imperfect RS results in patient misclassifications: multiplying Se_R and Sp_R by the true numbers of disease-positive (n_DP) and disease-negative (n_DN) patients results in the apparent number of disease-positive patients (an_DP) and the apparent number of disease-negative patients (an_DN). In Figure 1, n_TPR is the number of patients with true-positive RS results, n_FNR is the number of patients with false-negative RS results, n_TNR is the number of patients with true-negative RS results, n_FPR is the number of patients with false-positive RS results, Se_R is the sensitivity of the reference standard, Sp_R is the specificity of the reference standard, an_DP is the apparent number of disease-positive patients, and an_DN is the apparent number of disease-negative patients.

Figure 2 shows how the patient misclassifications result in incorrect values of the IT’s sensitivity (Se_I) and specificity (Sp_I): multiplying an_DP and an_DN by the true (but initially unknown) values of Se_I and Sp_I gives the apparent numbers of true-positive (an_TPI) and true-negative (an_TNI) IT results, which, when, respectively, divided by an_DP and an_DN (in accordance with Equations (1) and (2) above), give incorrect apparent values of the IT’s Se and Sp (aSe_I and aSp_I). In Figure 2,

• n_DP is the true number of disease-positive patients
• n_DN is the true number of disease-negative patients
• n_TPR is the number of patients with true-positive RS results
• n_FNR is the number of patients with false-negative RS results
• n_TNR is the number of patients with true-negative RS results
• n_FPR is the number of patients with false-positive RS results
• Se_R is the sensitivity of the RS
• Sp_R is the specificity of the RS
• an_DP is the apparent number of disease-positive patients according to the RS
• an_DN is the apparent number of disease-negative patients according to the RS
• Se_I is the sensitivity of the investigational test
• Sp_I is the specificity of the investigational test
• an_TPI1 is the apparent number of true-positive IT results based on n_TPR
• an_TPI2 is the apparent number of true-positive IT results based on n_FPR
• an_FNI1 is the apparent number of false-negative IT results based on n_TPR
• an_FNI2 is the apparent number of false-negative IT results based on n_FPR
• an_FPI1 is the apparent number of false-positive IT results based on n_FNR
• an_FPI2 is the apparent number of false-positive IT results based on n_TNR
• an_TNI1 is the apparent number of true-negative IT results based on n_FNR
• an_TNI2 is the apparent number of true-negative IT results based on n_TNR
• an_TPI is the apparent total number of true-positive IT results
• an_TNI is the apparent total number of true-negative IT results.

The two diagrams were analyzed to develop formulas that were then solved to give n_DP, n_DN, Se_I and Sp_I starting from the apparent results of a clinical study.

3. Results

3.1. Correction for Patient Misclassifications by an Imperfect RS: Calculation of True Numbers of Disease-Positive and Disease-Negative Patients

Figure 1 shows the relationship between n_DP, n_DN, Se_R, Sp_R, an_DP, and an_DN. From Figure 1, Formulas (3) and (4) can be deduced:

an_DP = n_DP × Se_R + n_DN × (1 − Sp_R)

(3)

an_DN = n_DP × (1 − Se_R) + n_DN × Sp_R

(4)

If an_DP, an_DN, Se_R, and Sp_R are all known, then Formulas (3) and (4) constitute a system of equations with two unknowns (n_DP and n_DN). This was solved by using an online system-of-equations calculator [13]. However, it was first necessary to substitute single-letter variables (e.g., x for n_DP, y for n_DN, and a, b, c, etc. for the other variables), because the online calculator interpreted two-letter variables as two variables rather than as a single variable. Solution of the system of equations leads to Formulas (5) and (6) (details shown in Supplementary Materials).

n_DP = (an_DP × Sp_R + Sp_R × an_DN − an_DN)/(Se_R + Sp_R − 1)

(5)

n_DN = (an_DP × Se_R − an_DP + Se_R × an_DN)/(Se_R + Sp_R − 1)

(6)

The denominators in Formulas (5) and (6) are equal to Youden’s J statistic [14] (Youden’s Index) for the RS (Equation (7)):

J_R = Se_R + Sp_R − 1

(7)

Therefore, Formulas (5) and (6) can be rewritten as Formulas (8) and (9):

n_DP = (an_DP × Sp_R + Sp_R × an_DN − an_DN)/J_R

(8)

n_DN = (an_DP × Se_R + Se_R × an_DN − an_DP)/J_R

(9)

Note that because the sum of n_DP and n_DN must equal N (the number of patients in the study; Equation (10)), one could also calculate just one value (either n_DP or n_DN), and subtract it from N to get the other value (Equation (11) or Equation (12)):

N = n_DP + n_DN

(10)

n_DP = N − n_DN

(11)

n_DN = N − n_DP

(12)

3.2. Calculation of Sensitivity and Specificity of the Investigational Test

Next, expressions for the true numbers of patients with true-positive and true-negative investigational test results (n_TPI and n_TNI, respectively) were derived. Figure 2 shows the possible outcomes of a clinical study of an IT using a RS. From Figure 2, it can be deduced that:

an_TPI = n_DP × Se_R × Se_I + n_DN × (1 − Sp_R) × (1 − Sp_I)

(13)

an_TNI = n_DP × (1 − Se_R) × (1 − Se_I) + n_DN × Sp_R × Sp_I

(14)

If n_DP, n_DN, Se_R, and Sp_R are all known, then Equations (13) and (14) constitute a system of equations with two unknowns (Se_I and Sp_I). Substituting single-letter variables and using the same online system-of-equations calculator referenced above leads to Equation (15) for the sensitivity of the investigational test (Se_I) and Equation (16) for the specificity of the investigational test (Sp_I).

Se_I = (an_TPI × Sp_R − n_DP × Se_R × Sp_R + n_DP × Se_R + n_DP × Sp_R − n_DP + n_DN × Sp_R² − n_DN × Sp_R − Sp_R × an_TNI + an_TNI)/(n_DP × (Se_R + Sp_R − 1))

(15)

Sp_I = (−an_TPI × Se_R  + an_TPI + n_DP × Se_R² − n_DP × Se_R − Se_R × n_DN × Sp_R + Se_R × n_DN + Se_R × an_TNI + n_DN × Sp_R − n_DN)/(n_DN × (Se_R + Sp_R − 1))

(16)

These can be simplified somewhat by substituting some of the terms with Youden’s Index. Thus, Equation (15) becomes Equation (17):

Se_I = (an_TPI × Sp_R − n_DP × Se_R × Sp_R + n_DP × J_R + n_DN × Sp_R² − n_DN × Sp_R − Sp_R × an_TNI + an_TNI)/(n_DP × J_R)

(17)

Similarly, substituting some terms with Youden’s Index converts Equation (16) into Equation (18):

Sp_I = (−an_TPI × Se_R + an_TPI + n_DP × Se_R² − n_DP × Se_R − Se_R × n_DN × Sp_R + n_DN × J_R + Se_R × an_TNI)/(n_DN × J_R)

(18)

As a quick check of the validity of the equations, if Se_R = Sp_R = 1, then Equations (17) and (18) should simplify to show that Se_I equals an_TPI/n_DP and that Sp_I equals an_TNI/n_DN—and they do (See Supplementary Materials).

3.3. Calculation of Apparent Sensitivity and Specificity of the Investigational Test

For the sake of completeness, Equations (19) and (20), which show the relationship between n_DP, n_DN, Se_R, Sp_R, Se_I, and Sp_I and the apparent sensitivity (aSe_I) and specificity (aSp_I) of the investigational test were deduced from Figure 2. Details are provided in the Supplementary Materials.

aSe_I = (n_DP × Se_R × Se_I + n_DN × (1 − Sp_R) × (1 − Sp_I))/(n_DP × Se_R + n_DN × (1 − Sp_R))

(19)

aSp_I = (n_DP × (1 − Se_R) × (1 − Se_I) + n_DN × Sp_R × Sp_I)/(n_DP × (1 − Se_R) + (n_DN × Sp_R))

(20)

3.4. Proportion-Based Equations

The above equations are in terms of patient counts. They can be converted to equations that are based on proportions by dividing by N, the total number of patients in the study. For example, starting with Equation (17) and dividing every term by N (equivalent to multiplying both the numerator and denominator by 1/N), one gets Equation (21):

Se_I = ((an_TPI × Sp_R/N) − (n_DP × Se_R × Sp_R/N) + (n_DP × J_R/N) + (n_DN × Sp_R²/N) − (n_DN × Sp_R/N) − (Sp_R × an_TNI/N) + (an_TNI/N))/(n_DP × J_R/N)

(21)

Since n_DP/N = Pr (the prevalence of the index disease), and since n_DN/N = 1 − Pr, Equation (21) can be rewritten as Equation (22):

Se_I = (pa_TPI × Sp_R − Pr × Se_R × Sp_R + Pr × J_R + (1 − Pr) × Sp_R² − (1 − Pr) × Sp_R − Sp_R × pa_TNI + pa_TNI)/(Pr × J_R)

(22)

In Equation (22), pa_TNI is the proportion of patients with an apparently true-negative investigational test result, Pr is the prevalence of the index disease, J_R is Youden’s Index for the RS, pa_TPI is the proportion of patients with an apparently true-positive investigational test result, Sp_R is the specificity of the RS, and Se_R is the sensitivity of the RS.

Similarly, dividing every term in Equation (18) by N gives Equation (23):

Sp_I = (pa_TPI − pa_TPI × Se_R + Pr × Se_R² − Pr × Se_R − Se_R × (1 − Pr) × Sp_R + (1 − Pr) × J_R + Se_R × pa_TNI)/((1 − Pr) × J_R)

(23)

In Equation (23), the variables have the same meaning as in Equation (22).

3.5. Example Calculation Using Test Data

As an example, suppose a clinical study of an investigational diagnostic test using an imperfect RS finds that 50 patients are apparently disease-positive by the RS (an_DP = 50), and 60 are apparently disease-negative (an_DN = 60). Suppose that Se_R = 0.90 and Sp_R = 0.85, and that the numbers of patients with apparently true IT classifications (an_TPI and an_TNI) are, respectively, 38 and 48. These data correspond to aSe_I = 0.76 and aSp_I = 0.80. Youden’s Index (J_R) for the reference standard = 0.75. Using Formula (8), n_DP = 45 after rounding. Since N = n_DP + n_DN, then n_DN = 110 − 45, or 65. The number of patients misclassified by the RS is the absolute value of the difference between an_DP and n_DP (or an_DN and n_DN):

Misclassifications = |an_DP − n_DP| = |an_DN − n_DN| = |50 − 45| = |60 − 65| = 5

Se_I and Sp_I are found by using Formulas (17) and (18): Se_I = 0.91 and Sp_I = 0.86. In this example, adjustment for the Se_R and Sp_R (0.90 and 0.85, respectively) of the imperfect RS led to an increase from aSe_I = 0.76 to Se_I = 0.91 and an increase from aSp_I = 0.80 to Sp_I = 0.86.

Suppose that in the above example Se_R equaled 0.75 and Sp_R equaled 0.90, and all of the other numbers were the same. Using Formula (8), n_DP = 60 after rounding. Since N = n_DP + n_DN, then n_DN = 110 − 60, or 50. The number of patients misclassified by the RS is then 10. Se_I and Sp_I are found by using Formulas (17) and (18): Se_I = 0.85 and Sp_I = 1.00. In this example, adjustment for the Se_R and Sp_R (0.75 and 0.90, respectively) of the imperfect RS led to an increase from aSe_I = 0.76 to Se_I = 0.85 and an increase from aSp_I = 0.80 to Sp_I = 1.00.

4. Discussion

In assessing an investigational new diagnostic test, it is not always feasible to use a perfect RS, and an imperfect RS (one with Se and/or Sp < 1) must sometimes be used, which can result in patient misclassifications and incorrect values of Se_I and Sp_I. Such situations raise questions about the accuracy of the investigational test’s Se and Sp determined using the imperfect RS. In this work, formulas for correctly calculating the investigational test’s true Se and true Sp from any reference standard were derived.

Three prior studies [9,10,11] reported derivations of formulas for correcting Se and Sp for misclassification by an imperfect RS. Their approaches differed from each other and from the approach taken here. In addition, they did not report enough detail to allow one to determine if their derivations were correct. Therefore, comparison of this work to theirs was difficult but was successful for the approaches by Gart and Buck [10] and Staquet et al. [11] (See Supplementary Materials for details).

Brenner [9] reported equations for aSe_I and aSp_I for a case–control study if Se_I, Sp_I, Se_R, Sp_R and the exposure Pr were all known, but did not report solving for Se_I and Sp_I, contrary to the paper by Umemneku Chikere et al. [8] who I believe may have misinterpreted the Brenner equations for aSe_I and aSp_I as being for Se_I and Sp_I.

Gart and Buck [10] discussed the use of screening and reference tests for estimating disease Pr in epidemiologic studies. They derived equations for what they termed co-positivity and co-negativity (which I determined to be equivalent to aSe_I and aSp_I), and solved these for Se_I and Sp_I if Pr, Se_R and Sp_R, aSe_I and aSp_I are known. I was able to show that my equations for Se_I and Sp_I (after transformation into proportion-based variables) were equivalent to theirs (see Supplementary Materials for details).

Staquet and colleagues [11] reported equations for calculating Se_I and Sp_I provided that Se_R, Sp_R, and an_TPI, an_TNI, an_FPI, and an_FNI are known. I was able to show that my equations for Se_I and Sp_I were equivalent to theirs (see Supplementary Materials for details).

Trikalinos et al. [12] did not report equations for Se_I or Sp_I, but I compared their equations for the cells of their 2 × 2 contingency table (corresponding to pa_TPI and pa_TNI), and they matched what I had derived (data not shown).

Strengths and Weaknesses of This Work

This work builds on that of Trikalinos et al. [12], Gart and Buck [10], Staquet et al. [11] and Brenner [9]. These authors discussed potential methods of handling diagnostic studies that use an imperfect RS but did not provide sufficient detail to allow easy replication of their methods. Although Trikalinos et al. [12] mentioned the possibility of adjusting results that are based on an imperfect RS, they did not report derivation of the formulas needed to do so, as I have. One advantage of my work is that I provide full details of the derivations (see Supplementary Materials) so that others may easily reproduce and confirm my work. In addition, I report formulas that can use either absolute patient counts or proportions (e.g., prevalence), in contrast to prior authors, who reported formulas for only one or the other approach.

5. Conclusions

Validation of a new diagnostic test by use of an imperfect RS (one with Se and/or Sp < 1) introduces patient misclassifications that result in deviation of the apparent sensitivity and specificity of the index test (aSe_I and aSp_I, respectively) from the true values. By analyzing the role of the reference standard in the determination of the sensitivity and specificity of an index test, it is possible to derive formulas that correct for patient misclassification by an imperfect RS, as well as for the subsequent error introduced into aSe_I and aSp_I. The analysis showed that the more imperfect the RS (i.e., the lower the Se and Sp of the RS), the greater the error introduced into aSe_I and aSp_I. Therefore, when an imperfect RS is used to validate a diagnostic test, it may be necessary to apply corrections to arrive at accurate values of Se and Sp for the test.

This work builds on that of prior authors who discussed potential methods of handling diagnostic studies that use an imperfect RS but did not provide sufficient detail to allow easy replication of their methods. In contrast, full details of the derivations are provided (in Supplementary Materials) to provide transparency, so that others may confirm and perhaps build upon this work. In addition, formulas based on both patient counts and patient proportions are provided, in contrast to prior authors, who provided either one or the other.

For this corrective method to be feasible, two conditions must be met. First, the assumption of conditional independence of the index test and the RS must be true; this assumption is reasonable if the two tests work by different mechanisms (e.g., if the index test relies on laboratory methods and the RS relies on autopsy). Second, one obviously needs to know the values of the Se and Sp of the RS, which may not always be the case. However, if they are known, then the derived formulas can help provide needed corrections to the apparent values of an index test’s Se and Sp.