A Software Tool for Calculating the Uncertainty of Diagnostic Accuracy Measures

Screening and diagnostic tests are applied for the classification of people into diseased and non-diseased populations. Although diagnostic accuracy measures are used to evaluate the correctness of classification in clinical research and practice, there has been limited research on their uncertainty. The objective for this work was to develop a tool for calculating the uncertainty of diagnostic accuracy measures, as diagnostic accuracy is fundamental to clinical decision-making. For this reason, the freely available interactive program Diagnostic Uncertainty has been developed in the Wolfram Language. The program provides six modules with nine submodules for calculating and plotting the standard combined, measurement and sampling uncertainty and the resultant confidence intervals of various diagnostic accuracy measures of screening or diagnostic tests, which measure a normally distributed measurand, applied at a single point in time to samples of non-diseased and diseased populations. This is done for differing sample sizes, mean and standard deviation of the measurand, diagnostic threshold and standard measurement uncertainty of the test. The application of the program is demonstrated with an illustrative example of glucose measurements in samples of diabetic and non-diabetic populations, that shows the calculation of the uncertainty of diagnostic accuracy measures. The presented interactive program is user-friendly and can be used as a flexible educational and research tool in medical decision-making, to calculate and explore the uncertainty of diagnostic accuracy measures.


Introduction
Diagnosis in medicine is the determination of the nature of a disease condition [1].The term diagnosis is derived from the Greek word "διάγνωσις" meaning "discernment".It is assumed that there is a dichotomy between the populations with and without a disease condition.Diagnostic tests or procedures are applied for the classification of people into the respective disjoint groups.The probability distributions of the measurand of a quantitative diagnostic test in each of the diseased and non-diseased populations are overlapping.The results of a test though can be dichotomized, by assigning a diagnostic threshold or cutoff point (Figure 1) [1].The possible test results are summarized in Table 1.It is assumed that there is a reference ("gold standard") diagnostic method correctly classifying a subject as diseased or non-diseased [2].The ratio of the diseased to the total population (diseased and non-diseased) at a single point in time is the prevalence rate (r) of the disease.
There is a persistent need of estimating the uncertainty of diagnostic accuracy measures, especially regarding screening and diagnostic tests of life-threatening diseases.The current pandemic of novel corona virus disease 2019 (COVID- 19) has exposed this unequivocally [3][4][5][6][7].There has been extensive research on either diagnostic accuracy or uncertainty, however, extremely limited research has been done on both subjects [8][9][10][11].
The program Diagnostic Uncertainty has been developed to explore the combined, measurement and sampling uncertainty of diagnostic accuracy measures as: 1.
Defining the permissible measurement uncertainty is critical to quality and risk management in laboratory medicine [13].

3.
Sampling uncertainty is decisive for clinical study design to evaluate a screening or diagnostic test [14].
Diagnostics 2021, 11, 406 2 of 40  Uncertainty is an expression of imperfect or deficient information.When quantifiable it can be represented with probability [16].The following components of the combined uncertainty of the diagnostic accuracy measures will be considered:

Measurement Uncertainty
As measurements are inherently variable, measurement uncertainty is defined as a "parameter, associated with the result of a measurement, that characterizes the dispersion of the values that could reasonably be attributed to the measurand" [17].Measurement uncertainty is replacing the total analytical error concept [18].

Sampling Uncertainty
Diagnostic accuracy measures are estimated by applying a screening or diagnostic test to samples of populations.Sampling heterogeneity contributes to the combined uncertainty of the diagnostic accuracy measures [19].Even when simple random sampling is applied, there is inherent sample heterogeneity [20].A sample of size n is considered as simple random sample, if all possible samples of the same size are equally probable [21].

Computational Methods
For the calculation of the uncertainty of the diagnostic accuracy measures of a screening or diagnostic test based on a measurand, it is assumed that: 1.
There is a reference ("gold standard") diagnostic method classifying correctly a subject as diseased or non-diseased [22].

2.
Either the values of the measurand or their transforms [23,24] are normally distributed in each of the diseased and non-diseased populations.

3.
Measurement uncertainty is normally distributed and homoscedastic in the diagnostic threshold's range.4.
The sampling is simple random.5.
If the measurement is above the threshold the patient is classified as test-positive, otherwise as test-negative.

Calculation of Diagnostic Accuracy Measures
The calculation of the diagnostic accuracy measures is based on their probability definitions (Table 2).The sensitivity and specificity can be defined in terms of the error function and the complementary error function (see Appendix B).The other diagnostic accuracy measures can be expressed in terms of sensitivity, specificity and prevalence rate and calculated as shown in Appendix B.

Calculation of Uncertainty of Diagnostic Accuracy Measures
The uncertainty of an input parameter or a diagnostic accuracy measure x can be expressed in the forms of standard and expanded uncertainty.The former, denoted as u(x) equals the standard deviation of x.The later, denoted as U(x), is defined as an interval around x including x with probability p [25].

Measurement Uncertainty
The standard measurement uncertainty u m of a measurand is estimated from a sample of n u measurements, as described in "Guide to the expression of uncertainty in measurement"(GUM) and "Expression of measurement uncertainty in laboratory medicine" [17].Bias may be considered as a component of the standard measurement uncertainty [26].

Sampling Uncertainty of Means and Standard Deviations
If m P and s P the mean and standard deviation of a measurand in a population sample of size n P , then the standard sampling standard uncertainties of m P and s P are:

Combined Uncertainty of Means and Standard Deviations
If u m the standard measurement uncertainty of a screening or diagnostic test measuring a measurand and m P and s P the mean and standard deviation of the measurand in a population sample of size n P , then the standard combined uncertainties of the mean m P and standard deviation s P are: Sampling Uncertainty of Prevalence Rate If n D and n D the respective numbers of non-diseased and diseased in a population sample, then the standard uncertainty of the prevalence rate r = n D n D +n D of the disease can be approximated as: according to the Agresti-Coull adjustment of the Waldo interval [27].

Combined Uncertainty of Diagnostic Accuracy Measures
The standard combined uncertainty u c (x) of each diagnostic accuracy measure x is calculated by applying the rules of uncertainty propagation from the input values to the calculated diagnostic accuracy measure (see Appendix B), according to GUM [28,29], with a first-order Taylor series approximation to uncertainty propagation [30].
When there are l components of uncertainty, with standard uncertainties u i (x) respec- tively, then:

Expanded Uncertainty of Diagnostic Accuracy Measures
The effective degrees of freedom v e f f of the standard combined uncertainty u c (x) are calculated using the Welch-Satterthwaite formula [31,32]: If v min the minimum of the respective degrees of freedom v 1 , v 2 , . . ., v l then: If F v (z) the cumulative distribution function of the Student's t-distribution with v degrees of freedom and u c (x) the standard combined uncertainty of a diagnostic accuracy measure x, its expanded combined uncertainty, at a confidence level p, is calculated as: The resultant confidence interval (CI) of x, at the same confidence level p, is:

The Program
To calculate the uncertainty of the diagnostic accuracy measures, the interactive program Diagnostic Uncertainty was developed in the Wolfram Language [33], using Wolfram Mathematica ® Ver.12.2, Wolfram Research, Inc., Champaign, IL, USA [34].The program was designed to provide six modules with nine submodules, for calculating and plotting the standard combined, measurement and sampling uncertainty and the resultant confidence intervals of various diagnostic accuracy measures of a screening or diagnostic test, applied at a single point in time in non-diseased and diseased population samples.The test measures a measurand in the population samples, for varying values of their sizes, mean and standard deviation and standard measurement uncertainty of the measurand.It is assumed that the measurands and measurement uncertainty are normally distributed and that measurement uncertainty is homoscedastic.
The program is freely available as a Wolfram Mathematica Notebook (.nb) (Supplementary File: Uncertainty.nb).It can be run on Wolfram Player ® or Wolfram Mathematica ® (see Appendix C).

Flowchart of the Program
The flowchart of the program is presented in Figure 2.

Interface of the Program
The modules and submodules of the program include panels with controls which allow the interactive manipulation of various parameters, as described in detail in Supplementary File: Diagnostic Uncertainty Interface.pdf.These are the following:

Plots vs. Diagnostic Threshold Module Diagnostic Accuracy Measures Standard Uncertainty Plots Submodule
The values of the standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test are plotted versus the diagnostic threshold of the test (Figure 3).

Diagnostic Accuracy Measures Relative Standard Uncertainty Plots Submodule
The values of the relative standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test are plotted versus the diagnostic threshold of the test (Figure 4).

Confidence Intervals of Diagnostic Accuracy Measures Plots Submodule
The values of the lower and upper bounds of the confidence intervals of diagnostic accuracy measure of a screening or diagnostic test, at a selected confidence level, are plotted versus the diagnostic threshold of the test (Figure 5).

Plots vs. Measurement Uncertainty Module Diagnostic Accuracy Measures Standard Uncertainty Plots Submodule
The values of the standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test are plotted versus the measurement uncertainty of the test (Figure 6).

Diagnostic Accuracy Measures Relative Standard Uncertainty Plots Submodule
The values of the relative standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test are plotted versus the measurement uncertainty of the test (Figure 7).The values of the standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test are plotted versus the diagnostic threshold of the test (Figure 3).  3.

Settings
Figure 3 3.The symbols are explained in Appendix A.

Diagnostic Accuracy Measures Relative Standard Uncertainties Plots Submodule
The values of the relative standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test are plotted versus the diagnostic threshold of the test (Figure 4).The values of the lower and upper bounds of the confidence intervals of diagnostic accuracy measure of a screening or diagnostic test, at a selected confidence level, are plotted versus the diagnostic threshold of the test (Figure 5).  3.

Plots vs. Measurement Uncertainty Module Diagnostic Accuracy Measures Standard Uncertainties Plots Submodule
The values of the standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test are plotted versus the measurement uncertainty of the test (Figure 6).  3.  3.

Diagnostic Accuracy Measures Relative Standard Uncertainties Plots Submodule
The values of the relative standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test are plotted versus the measurement uncertainty of the test (Figure 7).  3.

Confidence Intervals of Diagnostic Accuracy Measures Plots Submodule
The values of the lower and upper bounds of the confidence intervals of diagnostic accuracy measures of a screening or diagnostic test, at a selected confidence level, are plotted versus the measurement uncertainty of the test (Figure 8).The respective parameter settings are also shown in Table 3.

Plots vs. Population Sample Size Module Diagnostic Accuracy Measures Standard Uncertainties Plots Submodule
The values of the standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test are plotted versus the total population sample size (Figure 9).  3.

Plots vs. Population Sample Size Module Diagnostic Accuracy Measures Standard Uncertainty Plots Submodule
The values of the standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test are plotted versus the total population sample size (Figure 9).

Diagnostic Accuracy Measures Relative Standard Uncertainty Plots Submodule
The values of the relative standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test are plotted versus the total population sample size (Figure 10).3.

Diagnostic Accuracy Measures Relative Standard Uncertainties Plots Submodule
The values of the relative standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test are plotted versus the total population sample size (Figure 10).3.  3.

Confidence Intervals of Diagnostic Accuracy Measures Plots Submodule
The values of the lower and upper bounds of the confidence intervals of diagnostic accuracy measures of a screening or diagnostic test, at a selected confidence level, are plotted versus the total population sample size (Figure 11).3.

Confidence Intervals of Diagnostic Accuracy Measures Plots Submodule
The values of the lower and upper bounds of the confidence intervals of diagnostic accuracy measures of a screening or diagnostic test, at a selected confidence level, are plotted versus the total population sample size (Figure 11).3.

Diagnostic Accuracy Measures Standard Uncertainties Calculator Module
The values of the standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test, at a selected diagnostic threshold, are calculated and presented in a table (Figure 12).The respective parameter settings are also shown in Table 3.

Diagnostic Accuracy Measures Standard Uncertainty Calculator Module
The values of the standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test, at a selected diagnostic threshold, are calculated and presented in a table (Figure 12).

Diagnostic Accuracy Measures Relative Standard Uncertainty Calculator Module
The values of the relative standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test, at a selected diagnostic threshold, are calculated and presented in a table (Figure 13).

Diagnostic Accuracy Measures Relative Standard Uncertainties Calculator Module
The values of the relative standard combined, measurement and sampling uncertainties of diagnostic accuracy measures of a screening or diagnostic test, at a selected diagnostic threshold, are calculated and presented in a table (Figure 13).The respective parameter settings are also shown in Table 3.

Diagnostic Accuracy Measures Confidence Intervals Calculator Module
The point estimations and the lower and upper bounds of the confidence intervals of diagnostic accuracy measures of a screening or diagnostic, at a selected confidence level and diagnostic threshold, are calculated and presented in a table (Figure 14).

Diagnostic Accuracy Measures Confidence Intervals Calculator Module
The point estimations and the lower and upper bounds of the confidence intervals of diagnostic accuracy measures of a screening or diagnostic, at a selected confidence level and diagnostic threshold, are calculated and presented in a table (Figure 14).  3.

Illustrative Example
The program was applied to a bimodal distribution of log-transformed blood glucose measurements in samples of non-diabetic and diabetic populations.The data were derived from a national health survey conducted in Malaysia in 1996 [35].A glucose tolerance test (OGTT) was performed on 2667 Malay adults, aged 40-49 years.The respective sizes of the samples of the diseased and non-diseased populations were 179 and 2488.Glucose was measured with reflectance photometry, after the ingestion of 75 g glucose monohydrate.It was assumed that the measurement coefficient of variation and bias were equal to 4% and 2%, respectively.The log-transformed measurands of each population were normally distributed, as shown in Figure 1.The standardized log-transformed measurand means and standard deviations of the samples of the diseased and non-diseased populations, the standard measurement uncertainty and the diagnostic threshold were expressed in units equal to the standard deviation of the log-transformed measurand of the sample of the non-diseased population.The standardized log-transformed standard measurement uncertainty 0.046 of the test corresponds to coefficient of variation equal to 2%.The standardized log-transformed American Diabetes Association (ADA) diagnostic threshold for diabetes of the 2-h postprandial glucose during OGTT is equal to 2.26 [36].
The results of the illustrative example are presented: 1.
In the chart of Figure 22 3.
In the tables of Figures 12-14.3.

Illustrative Example
The program was applied to a bimodal distribution of log-transformed blood glucose measurements in samples of non-diabetic and diabetic populations.The data were derived from a national health survey conducted in Malaysia in 1996 [35].A glucose tolerance test (OGTT) was performed on 2667 Malay adults, aged 40-49 years.The respective sizes of the samples of the diseased and non-diseased populations were 179 and 2488.Glucose was measured with reflectance photometry, after the ingestion of 75 g glucose monohydrate.It was assumed that the measurement coefficient of variation and bias were equal to 4% and 2%, respectively.The log-transformed measurands of each population were normally distributed, as shown in Figure 1.The standardized log-transformed measurand means and standard deviations of the samples of the diseased and non-diseased populations, the standard measurement uncertainty and the diagnostic threshold were expressed in units equal to the standard deviation of the log-transformed measurand of the sample of the non-diseased population.The standardized log-transformed standard measurement uncertainty 0.046 of the test corresponds to coefficient of variation equal to 2%.The standardized log-transformed American Diabetes Association (ADA) diagnostic threshold for diabetes of the 2-h postprandial glucose during OGTT is equal to 2.26 [36].
The results of the illustrative example are presented: 1.

2.
In the chart of Figure 22 3.
In the tables of Figures 12-14.4.  In the illustrative example, combined uncertainty u c (x)has (see Figures 13 and 22): 1.
Greater effect (18% < (u c (x)/x) < 39%) on diagnostic odds ratio, on likelihood ratio for a positive or negative result and Euclidean distance, in accordance with previous findings [37,38].
In addition, measurement uncertainty is the main component of the combined uncertainty of specificity, overall diagnostic accuracy, positive predictive value, diagnostic odds ratio and likelihood ratio for a positive result.In the illustrative example, combined uncertainty has (see Figures 13 and 22): 1.
In addition, measurement uncertainty is the main component of the combined uncertainty of specificity, overall diagnostic accuracy, positive predictive value, diagnostic odds ratio and likelihood ratio for a positive result.4.

Discussion
The program Diagnostic Uncertainty explores the combined, measurement and sampling uncertainty of diagnostic accuracy measures of a screening or diagnostic test (Figures 3, 4 The complexity of the calculations of the confidence intervals of the diagnostic accuracy measures is considerable.In antithesis of the complexity of the calculations, the program simplifies its exploration with a user-friendly interface.Furthermore, it provides calculators for the calculation of the components of uncertainty of diagnostic accuracy measures and the resultant confidence intervals (Figures 12-14).
As demonstrated by the illustrative example described above, in this instance uncertainty has relatively little effect on specificity, overall diagnostic accuracy and negative predictive value.It affects more sensitivity, positive predictive value, Youden's index and concordance probability, while it has a considerable impact on diagnostic odds ratio, likelihood ratio for a positive or negative result and Euclidean distance (Figure 22).However, further research is needed to explore the uncertainty of diagnostic accuracy measures with different clinically-and laboratory-relevant parameter settings.
Limitations of this program, that could be improved by further research, are the following: (1) The assumptions used for the calculations: a.
The existence of a "gold standard" diagnostic method.If a "gold standard" does not exist, there are alternative approaches for the estimation of diagnostic accuracy measures [39].b.
The normality of either the measurements or their applicable transforms [23,24,40,41], however, this is usually valid.There is related literature on the distribution of measurements of diagnostic tests, in the context of reference intervals and diagnostic thresholds or clinical decision limits [42][43][44][45][46]. c.
The bimodality of the measurands, that is generally accepted, although unimodal distributions could be considered [47,48].d.
The simple random sampling.e.
The measurement uncertainty homoscedasticity in the diagnostic thresholds range.Nevertheless, if measurement uncertainty is heteroscedastic, thus skewing the measurements distribution, appropriate transformations may restore homoscedasticity [49].
If the above assumptions are not valid, there are other components of uncertainty which are not calculated by this program.
(2) The first order Taylor series approximations for the uncertainty propagation calculations [28,30].Higher order approximations may improve the accuracy.(3) The uncertainty of prevalence rate approximation by the Agresti-Coull adjusted Waldo interval [25], although there are more exact methods [50].
However, addressing these limitations, would increase exponentially the computational complexity.
The program presented in this work complements our previously published software [11], which explores the effects of measurement uncertainty on diagnostic accuracy measures applied to populations.This program calculates the standard and expanded combined, measurement and sampling uncertainty and the resultant confidence intervals of diagnostic accuracy measures of diagnostic tests, applied to samples of populations, providing 99 different types of plots and three different types of comprehensive tables (Figure 2), many of which are novel.To the best of our knowledge, no software, including all major general or medical statistical and uncertainty related software packages (Matlab ® , NCSS ® , R, SAS ® , SPSS ® , Stata ® , MedCalc ® , NIST Uncertainty Machine, UQLab, metRology), provides this range of plots and tables without advanced programming.

Conclusions
The presented program Diagnostic Uncertainty calculates the combined, measurement and sampling uncertainty of diagnostic accuracy measures and the resultant confidence intervals and can be used as a flexible, user-friendly, interactive educational or research tool in medical decision-making.
Informed Consent Statement: Not applicable.

Data Availability Statement:
The data presented in this study are available in [35].

Conflicts of Interest:
The authors declare no conflict of interest.

Appendix
where: where: where: where:

Figure 1 .
Figure 1.Probability density function plots.The probability density functions plots of a measurand in a non-diseased and diseased population.

Figure 2 .
Figure 2. Program flowchart.The flowchart of the program with the number of the input parameters and of the output types for each module or submodule.

Figure 2 .
Figure 2. Program flowchart.The flowchart of the program with the number of the input parameters and of the output types for each module or submodule.

Figure 3 .
Figure 3. Plots vs. diagnostic threshold module, DAM uncertainty plots submodule screenshot.Standard combined, measurement and sampling uncertainty of diagnostic odds ratio (u(DOR)) versus diagnostic threshold (d) curve plot, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Figure 3 .
Figure 3. Plots vs. diagnostic threshold module, DAM uncertainty plots submodule screenshot.Standard combined, measurement and sampling uncertainty of diagnostic odds ratio (u(DOR)) versus diagnostic threshold (d) curve plot, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Figure 4 .
Figure 4. Plots vs. diagnostic threshold module, DAM relative uncertainty plots submodule screenshot.Relative standard combined, measurement and sampling uncertainty of overall diagnostic accuracy (u(ODA)/ODA) versus diagnostic threshold (d) curve plot, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Figure 4 .
Figure 4. Plots vs. diagnostic threshold module, DAM relative uncertainty plots submodule screenshot.Relative standard combined, measurement and sampling uncertainty of overall diagnostic accuracy (u(ODA)/ODA) versus diagnostic threshold (d) curve plot, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Figure 5 .
Figure 5. Plots vs. diagnostic threshold module, DAM CI plots submodule screenshot.Confidence intervals of likelihood ratio for a negative test result (LR -) versus diagnostic threshold (d) curves plot, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Figure 5 .
Figure 5. Plots vs. diagnostic threshold module, DAM CI plots submodule screenshot.Confidence intervals of likelihood ratio for a negative test result (LR−) versus diagnostic threshold (d) curves plot, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Figure 6 .
Figure 6.Plots vs. measurement uncertainty module, DAM uncertainty plots submodule screenshot.Standard combined, measurement and sampling uncertainty of likelihood ratio for a negative test result (u(LR -)) versus standard measurement uncertainty (um) curve plot, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Figure 6 . 40 Figure 7 .
Figure 6.Plots vs. measurement uncertainty module, DAM uncertainty plots submodule screenshot.Standard combined, measurement and sampling uncertainty of likelihood ratio for a negative test result (u(LR−)) versus standard measurement uncertainty (u m ) curve plot, with the settings shown on the left.The respective parameter settings are also shown in Table 3. Diagnostics 2021, 11, 406 12 of 40

Figure 7 .
Figure 7. Plots vs. measurement uncertainty module, DAM relative uncertainty plots submodule screenshot.Relative standard combined, measurement and sampling uncertainty of likelihood ratio for a positive test result (u(LR+)/LR+) versus measurement uncertainty (u m ) curves plot, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Figure 8 .
Figure 8. Plots vs. measurement uncertainty module, DAM CI plots submodule screenshot.Confidence intervals of concordance probability (CZ) versus standard measurement uncertainty (um) curves plot, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Figure 8 .
Figure 8. Plots vs. measurement uncertainty module, DAM CI plots submodule screenshot.Confidence intervals of concordance probability (CZ) versus standard measurement uncertainty (u m ) curves plot, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Figure 9 .
Figure 9. Plots vs. total population sample size module, DAM uncertainty plots submodule screenshot.Standard combined, measurement and sampling uncertainty of diagnostic odds ratio (u(DOR)) versus total population sample size (n) curves plot, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Figure 9 .
Figure 9. Plots vs. population sample size module, DAM uncertainty plots submodule screenshot.Standard combined, measurement and sampling uncertainty of diagnostic odds ratio (u(DOR)) versus total population sample size (n) curves plot, with the settings shown on the left.The respective parameter settings are also shown in Table3.

40 Figure 10 .
Figure 10.Plots vs. population sample size module, DAM relative uncertainty plots submodule screenshot.Relative standard combined, measurement and sampling uncertainty of Youden's index (u(J)/J) versus total population sample size (n) curves plot, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Figure 10 .
Figure 10.Plots vs. population sample size module, DAM relative uncertainty plots submodule screenshot.Relative standard combined, measurement and sampling uncertainty of Youden's index (u(J)/J) versus total population sample size (n) curves plot, with the settings shown on the left.The respective parameter settings are also shown in Table3.

40 Figure 11 .
Figure 11.Plots vs. population sample size module, DAM CI plots submodule screenshot.Confidence intervals of likelihood ratio for a positive test result (LR+) versus total population sample size (n) curves plot, with the settings shown at the left.The respective parameter settings are also shown in Table3.

Figure 11 .
Figure 11.Plots vs. population sample size module, DAM CI plots submodule screenshot.Confidence intervals of likelihood ratio for a positive test result (LR+) versus total population sample size (n) curves plot, with the settings shown at the left.The respective parameter settings are also shown in Table3.

Figure 12 .
Figure 12.Calculators module, DAM uncertainty calculator submodule screenshot.Calculated standard combined, measurement and sampling uncertainty of diagnostic accuracy measures, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Figure 12 . 40 Figure 13 .
Figure 12.DAM uncertainty calculator module screenshot.Calculated standard combined, measurement and sampling uncertainties of diagnostic accuracy measures, with the settings shown on the left.The respective parameter settings are also shown in Table 3. Diagnostics 2021, 11, 406 18 of 40

Figure 13 .
Figure 13.DAM relative uncertainty calculator submodule screenshot.Calculated relative standard combined, measurement and sampling uncertainty of diagnostic accuracy measures, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Diagnostics 2021, 11 , 406 19 of 40 Figure 14 .
Figure 14.Calculators module, DAM CI calculator module screenshot.Calculated point estimations and confidence intervals of diagnostic accuracy measures, with the settings shown on the left.The respective parameter settings are also shown in Table3.
present the standard combined, measurement and sampling uncertainty and the resultant confidence intervals of sensitivity, specificity, positive and negative predictive value versus diagnostic threshold, measurement uncertainty and total population sample size.

Figure 14 .
Figure 14.DAM CI calculator module screenshot.Calculated point estimations and confidence intervals of diagnostic accuracy measures, with the settings shown on the left.The respective parameter settings are also shown in Table3.

Figure 15 .
Figure 15.Histogram of standard combined, measurement and sampling uncertainties of diagnostic accuracy measures, with the respective parameter settings in Table4.
present the standard combined, measurement and sampling uncertainty and the resultant confidence intervals of sensitivity, specificity, positive and negative predictive value versus diagnostic threshold, measurement uncertainty and total population sample size.The combined uncertainty and the resultant confidence intervals increase with measurement uncertainty (Figures 6-8, 18 and 19) and decrease with total population sample size (Figures 9-11, 20 and 21).

Figure 17 .
Figure 17.DAM confidence intervals versus diagnostic threshold plots.Plots of confidence intervals of (a) sensitivity (Se), (b) specificity (Sp), (c) positive predictive value (PPV) and (d) negative predictive value (NPV) versus diagnostic threshold (d) curves, with the respective parameter settings in Table4.

Figure 18 .
Figure 18.DAM confidence intervals versus diagnostic threshold plots.Plots of confidence intervals of (a) sensitivity (Se), (b) specificity (Sp), (c) positive predictive value (PPV) and (d) negative predictive value (NPV) versus diagnostic threshold (d) curves, with the respective parameter settings in Table4.

Figure 19 .
Figure 19.DAM confidence intervals versus measurement uncertainty plots.Plots of confidence intervals of (a) sensitivity (Se), (b) specificity (Sp), (c) positive predictive value (PPV) and (d) negative predictive value (NPV) versus standard measurement uncertainty (um) curves, with the respective parameter settings in Table4.

Figure 20 .
Figure 20.DAM confidence intervals versus measurement uncertainty plots.Plots of confidence intervals of (a) sensitivity (Se), (b) specificity (Sp), (c) positive predictive value (PPV) and (d) negative predictive value (NPV) versus standard measurement uncertainty (u m ) curves, with the respective parameter settings in Table4.

Figure 21 .
Figure 21.DAM confidence intervals versus population sample size plots.Plots of confidence intervals of (a) sensitivity (Se), (b) specificity (Sp), (c) positive predictive value (PPV) and (d) negative predictive value (NPV) versus total population sample size (n) curves, with the respective parameter settings in Table4.

Figure 22 .
Figure 22.DAM relative standard uncertainties of diagnostic accuracy measures.Histogram of standard combined, measurement and sampling uncertainties of diagnostic accuracy measures, with the respective parameter settings in Table 4.The combined uncertainty and the resultant confidence intervals increase with measurement uncertainty (Figures 6-8, 18 and 19) and decrease with total population sample size (Figures 9-11, 20 and 21).In the illustrative example, combined uncertainty has (see Figures13 and 22):

Figure 22 .
Figure 22.DAM confidence intervals versus population sample size plots.Plots of confidence intervals of (a) sensitivity (Se), (b) specificity (Sp), (c) positive predictive value (PPV) and (d) negative predictive value (NPV) versus total population sample size (n) curves, with the respective parameter settings in Table4.

Table 3 . The parameter settings of Figures 3-14. Settings Figure 3 Figure 4 Figure 5 Figures 6 and 7 Figure 8 Figures 9 and 10 Figure 11 Figures 12 and 13 Figure 14
The symbols are explained in Appendix A.

Table 4 .
The parameter settings of Figures 15-22.The parameter settings of Figures 3-14 are presented in Table 3 and of Figures 15-22 in Table 4. Figures 15-21 A ): Likelihood ratio for a positive test result nlr(. ..):Likelihood ratio for a negative test result dor(. ..):Diagnostic odds ratio of a test ed(. ..):Euclidean distance of a test D , s D