# Estimating Detection Limits in Chromatography from Calibration Data: Ordinary Least Squares Regression vs. Weighted Least Squares

## Abstract

**:**

_{res}), or the intercept standard deviation (s

_{b}

_{0}). In this study, multiple experimental calibrations are evaluated, applying both ordinary and weighted least squares. Moreover, the analyses of replicated blank matrices, spiked at 2–5 times the lowest calculated limit values with the two regression methods, are performed to obtain the standard deviation of the blank. The limits of detection obtained with ordinary least squares, weighted least squares, the signal-to-noise ratio, and replicate blank measurements are then compared. Ordinary least squares, which is the simplest and most commonly applied calibration regression methodology, always overestimate the values of the standard deviations at the lower levels of calibration ranges. As a result, the detection limits are up to one order of magnitude greater than those obtained with the other approaches studied, which all gave similar limits.

## 1. Introduction

_{LOD}, that can be detected with a reasonable certainty for a given analytical procedure [1,2]. Despite the simplicity of this definition, the LOD is a troublesome concept from a practical point of view. This is due to the different approaches that can be applied to calculate this parameter with “reasonable certainty” [2,3,4,5,6,7,8,9,10], which leads to differences that can reach several orders of magnitude depending on the approach used [11,12,13]. For this reason, validation guidelines leave the analyst free to choose, but suggest that the method used for determining the LOD should be documented and supported, and that an appropriate number or samples should be analyzed at the limit to validate the level [10].

_{bl}) and on two risk values: α (probability of false positives, type I error) and β (false negatives, type II error or power). This procedure requires the determination of two analytical parameters: the standard deviation of the blank, and the slope of a regression function (i.e., analytical sensitivity) [5].

_{bl}can only be estimated when blank measurement gives a signal, which is not the common situation in chromatographic methods. To solve this problem, the analysis of matrix blanks fortified at a level close to the LOD (usually <5 times the LOD) is accepted by many validation guidelines for the estimation of σ

_{bl}[4,5,16]. This procedure is labor intensive, as ≥20 blank measurements [7,12] are required to calculate a value of the standard deviation of the blank (s

_{bl}) that can be taken as a good estimator of σ

_{bl}. However, from a practical point of view a minimum of seven [6] or ten blank analyses [7,16] are typically used.

_{bl}, the Hubaux-Vos approach has been proposed when the instrument response is linearly related to the concentration [17]. This approach indicates that σ

_{bl}can be estimated from the linear calibration curve, either by the regression residual standard deviation (s

_{res}) or the standard deviation of the y-intercept (s

_{b}

_{0}). To fulfil the requirements of this approach, the error distribution of all standards used in the calibration must be constant (homoscedasticity). This procedure is also accepted by ISO [18].

_{bl}from the standard deviations obtained applying ordinary least squares to analytical calibrations—enables useful LODs to be obtained for routine work.

## 2. Experimental and Statistical Calculations

_{i}) as inverse of s

_{i}

^{2}was applied in the WLS regressions [14,23]. Other empirical w

_{i}such as 1/x

_{i}

^{2}and 1/y

_{i}

^{2}were also evaluated.

_{bl}), a minimum of seven replicates of a blank matrix were obtained for each method and spiked at a level between 2–5 times the smallest LOD value calculated through OLS and WLS regressions. The LODs from the standard deviation of the fortified blanks were also determined.

## 3. Statistical Considerations

_{LOD}) is given by the following equation:

_{bl}is the signal of the “true” blank mean (which is usually zero in chromatography in absence of bias in the procedure), z

_{1−α}and z

_{1−β}are the z-values of the one-sided standardized normal distribution at given significance levels α and β, σ

_{bl}is the standard deviation at the blank level when the component is not present in the sample, and σ

_{LOD}is the standard deviation at the LOD level.

_{bl}= 0, and assuming normal distribution for the blank and LOD signals and a constant dispersion between blank and LOD range (i.e., σ

_{bl}= σ

_{LOD}), Equation (1) can be rewritten as:

_{1−α}= z

_{1−β}= 1.645:

_{1}is the slope of the linear regression function.

_{bl}is unknown and has to be estimated from the standard deviation of a limited number of blank measurements (s

_{bl}). Therefore, the z

_{1−α}value should be replaced by the one-tailed Student’s t for ν degrees of freedom and α = 0.05 (t

_{(1−α,ν)}) [1,3,11]:

_{(1−α,ν)}) multiplying s

_{bl}should range from 3.89 (for 7 blank replicates) to 3.67 (n = 10).

_{(0.99,ν)}ranges from 3.14 (n = 7) and 2.82 (n = 10).

_{LOD}= 3 s

_{bl}that is usually applied in many studies. It should be taken into account that 3 s

_{bl}corresponds either to α = 0.00135 and β = 0.50 [24], which means that there is no control of false negative errors, or to α = 0.05 and β = 0.16 (84% power) [10], which may be considered as an acceptable β level, but is higher than the recommended β = 0.05 (95% power) by ISO and IUPAC. In general, it is accepted that y

_{LOD}≥ 3 s

_{bl}, but the use of y

_{LOD}= 2 s

_{bl}, as proposed by some studies, should not be applied for estimating LODs as this value corresponds to the critical level (L

_{c}) and would result in a concentration level where, assuming a normal distribution, there is only a 50% probability of the analyte being detected [3,11,12,13,25].

## 4. Results

_{1}) obtained were significant (p < 0.001 for the null hypothesis b

_{1}= 0) for both OLS and WLS regressions. The y-intercept values (b

_{0}) did not differ significantly from zero (p > 0.05 for the null hypothesis b

_{0}= 0) for OLS regression functions. However, ten calibrations (50%) did not yield intercepts equivalent to zero with WLS regression. It is important to point out that the b

_{0}value of the linear function must not be significantly different from the mean blank signal, which means that b

_{0}must not differ statistically from zero for chromatographic methods with no bias.

_{0}was found to differ from zero by:

_{b}

_{0}and s

_{res}) are usually accepted as estimates of σ

_{bl}when linear calibrations are applied for the determination of LODs [5,8,10]. In the case of WLS, the calculated s

_{res}is significantly rounded to near unity due to the inverse variance weighting scheme [26,27] (Table 1, Table 2, Table 3 and Table 4) and cannot be used directly as an estimate of σ

_{bl}, with s

_{b}

_{0}being the only estimator for this regression model. For OLS regressions with appropriate determination coefficients for quantitative purposes and ≥6 calibration standards, it is common to find that s

_{b}

_{0}< s

_{res}[5,8,28,29], which also happened with the calibrations analyzed in the present study. Therefore, s

_{b}

_{0}has been chosen to make the comparison between OLS and WLS regressions. In most calibrations (n = 16), LODs determined with OLS were significantly higher than those obtained with WLS (from 1.4–15 times higher, Table 5). In four GC-FID calibrations, LOD values calculated by OLS and WLS were equivalent.

_{bl}was used as a new estimate of σ

_{bl}. All LOD values obtained from s

_{bl}were equivalent to those obtained by WLS regression (Table 5).

## 5. Discussion

_{bl}from the linear calibration curve applying the Hubaux-Vos approach, the error distribution of all standards used in the calibration must be constant (homoscedasticity) [8,13,17]. However, despite the fact that many researchers often do not take it into account, heteroscedasticity is more frequent than might be expected in experimental sciences. Many analytical methods yield non-constant variances over the calibration range [8,26,27,30,31,32,33,34], as was the case with the calibrations evaluated in the present study (Figure 1). In these conditions, the absolute errors of the instrument tend to be proportional to the concentrations, and the relative standard deviation is the constant parameter across the curve instead of the standard deviation [33,35,36,37,38].

_{bl}without considering whether or not the calibration presents heteroscedasticity [28,29,40,41,42,43,44]. Unfortunately, OLS assumes constant variance over the whole calibration range and the standard deviations calculated by OLS can differ greatly from the true standard deviation, particularly at low concentration levels [35,36,37,45]. Moreover, as indicated by Meites et al. [46], in experimental calibrations where the independent variable can also be subject to random measurement errors, OLS always lead to biased estimates of the intercept. As can be seen in the results obtained in the present study, s

_{b}

_{0}values were always higher when OLS regression was applied, leading to an overestimation of LOD values (Table 5). This was corroborated by the fact that when fortified blanks were prepared at the LOD level estimated by OLS regression, the signals obtained gave S/N > 8 (as auto-integrated by the software of the instruments, Figure 2c); and fortified blanks prepared below this limit gave chromatograms with clearly identified peaks (S/N ≥ 3) (Figure 2b). Moreover, in many of the calibrations evaluated the value of the signal obtained for the first standard and the S/N confirmed that this standard gave a signal clearly above the LOD, but its concentration was below the LOD determined on applying OLS.

_{b}

_{0}values determined by WLS were significantly smaller (2–23 times, p < 0.05, Fisher F-test) than by OLS (Table 1, Table 2, Table 3 and Table 4), which agrees with the results obtained by other studies comparing OLS and WLS with experimental calibrations involving heteroscedastic data [8,26,27,47].

_{bl}approach were equivalent to the LODs determined by WLS (Table 5). In general, the limits determined with OLS were up to one order of magnitude higher than those obtained with WLS, S/N and fortified blank measurements.

_{i}= 1/s

_{i}

^{2}). Taking into account that standard deviation is usually proportional to the concentration [33,35,36,37,38], different experimental approaches have been proposed to avoid the requirement of replicate measurements at each calibration level [33,35,38,48]. Therefore, different empirical weighting factors, such as 1/x

_{i}

^{1/2}, 1/x

_{i}, 1/x

_{i}

^{2}, 1/y

_{i}

^{1/2}, 1/y

_{i}and 1/y

_{i}

^{2}, have been proposed, from which 1/x

_{i}

^{2}and 1/y

_{i}

^{2}seem to yield the best results. The WLS regressions were evaluated applying these two empirical weighting factors and the corresponding regression parameters, and their standard deviations were used to determine the LODs (Table 5). It was observed that, in general, there were no significant differences between the LODs determined by WLS independently of the weighting factor used. Only in two calibrations were the LODs determined applying both 1/x

_{i}

^{2}and 1/y

_{i}

^{2}weights significantly smaller than those obtained by 1/s

_{i}

^{2}or s

_{bl}.

## 6. Conclusions

_{b}

_{0}through OLS is not a good estimate of σ

_{bl}(s

_{b}

_{0}can be up to one order of magnitude higher than the real σ

_{bl}).

_{bl}.

_{i}

^{2}and 1/y

_{i}

^{2}. In the present study, it has been found that, in general, the use of these empirical weighting factors allows equivalent LODs to those determined by 1/s

_{i}

^{2}to be obtained.

## Supplementary Materials

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Currie, L.A. Nomenclature in evaluation of analytical methods including detection and quantification capabilities. Pure Appl. Chem.
**1995**, 67, 1699–1723. [Google Scholar] [CrossRef] - International Union of Pure and Applied Chemistry. IUPAC Compendium of Chemical Terminology—The Gold Book; IUPAC. Available online: https://goldbook.iupac.org (accessed on 8 March 2018).
- Heyden, Y.V.; Boqué, R. The limit of detection. LCGC Eur.
**2009**, 22, 82–85. [Google Scholar] - International Conference on Harmonisation (ICH). Q2(R1): Validation of Analytical Procedures—Text and Methodology. ICH Harmonised Tripartite Guideline. 1994/1996. Available online: http://www.ich.org/products/guidelines/quality/quality-single/article/validation-of-analytical-procedures-text-and-methodology.html (accessed on 31 July 2018).
- Shabir, G.A. Validation of high-performance liquid chromatography methods for pharmaceutical analysis: Understanding the differences and similarities between validation requirements of the US Food and Drug Administration, the US Pharmacopeia and the International Conference on Harmonization. J. Chromatogr. A
**2003**, 987, 57–66. [Google Scholar] [CrossRef] [PubMed] - US-EPA. 40 CFR Appendix B to Part 136—Definition and Procedure for the Determination of the Method Detection Limit; Revision 2, EPA 821-R-16-006; US Environmental Protection Agency: Washington, DC, USA, 2016. Available online: https://www.gpo.gov/fdsys/granule/CFR-2011-title40-vol23/CFR-2011-title40-vol23-part136-appB/content-detail.html (accessed on 31 July 2018).
- Wenzl, T.; Haedrich, J.; Schaechtele, A.; Robouch, P.; Stroka, J. Guidance Document on the Estimation of LOD and LOQ for Measurements in the Field of Contaminants in Feed and Food; EUR 28099; Publications Office of the European Union: Luxembourg, 2016; ISBN 978-92-79-61768-3. [Google Scholar]
- Vial, J.; Jardy, A. Experimental comparison of the different approaches to estimate LOD and LOQ of an HPLC method. Anal. Chem.
**1999**, 71, 2672–2677. [Google Scholar] [CrossRef] - Shrivastava, A.; Gupta, V.B. Methods for the determination of limit of detection and limit of quantification of the analytical methods. Chron. Young Sci.
**2011**, 2, 21–25. [Google Scholar] [CrossRef] - Araujo, P. Key aspects of analytical method validation and linearity evaluation. J. Chromatogr. B Anal. Technol. Biomed. Life Sci.
**2009**, 877, 2224–2234. [Google Scholar] [CrossRef] [PubMed] - Currie, L.A. Limits for qualitative detection and quantitative determination: Application to radiochemistry. Anal. Chem.
**1968**, 40, 586–593. [Google Scholar] [CrossRef] - Long, G.L.; Winefordner, J.D. Limit of detection: A closer look at the IUPAC definition. Anal. Chem.
**1983**, 55, 712A–724A. [Google Scholar] [CrossRef] - Bernal, E. Limit of detection and limit of quantification: Determination in gas chromatograpy. In Advances in Gas Chromatography; Guo, X., Ed.; IntechOpen: Rijeka, Croatia, 2014; pp. 57–81. ISBN 978-953-51-1227-3. Available online: https://www.intechopen.com/books/advances-in-gas-chromatography (accessed on 8 March 2018).
- Desimoni, E.; Brunetti, B. About estimating the limit of detection by the signal to noise approach. Pharm. Anal. Acta
**2015**, 6, 1000355. [Google Scholar] [CrossRef] - International Organization for Standardization (ISO). Capability of Detection, Part 1: Terms and Definitions; ISO 11843-1; ISO: Geneve, Switzerland, 1997. [Google Scholar]
- Magnusson, B.; Ornemark, U. (Eds.) EURACHEM Guide: The Fitness for Purpose of Analytical Methods—A Laboratory Guide to Method Validation and Related Topics, 2nd ed.; LCG: Teddington, UK, 2014; ISBN 978-91-87461-59-0. [Google Scholar]
- Hubaux, A.; Vos, G. Decision and detection limits for linear calibration curves. Anal. Chem.
**1970**, 42, 849–855. [Google Scholar] [CrossRef] - International Organization for Standardization (ISO). Capability of Detection. Part 2: Methodology in the Linear Calibration Case; ISO 11843-2; ISO: Geneve, Switzerland, 2000. [Google Scholar]
- Massart, D.L.; Vandeginste, B.G.M.; Buydens, L.M.C.; De Jong, S.; Lewi, P.H.; Smeyers-Verbeke, J. Handbook of Chemometrics and Qualimetrics: Part A; Elsevier: Amsterdam, The Netherlands, 1997; ISBN 0-444-89724-0. [Google Scholar]
- Mandel, J. The Statistical Analysis of Experimental Data; Dover Pub. Inc.: New York, NY, USA, 1964; ISBN 978-0486646664. [Google Scholar]
- Levene, H. Robust test for equality of variances. In Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling; Olkin, I., Ghurye, S.G., Hoedffding, W., Madow, W.G., Mann, H.B., Eds.; Stanford University Press: Palo Alto, CA, USA, 1960; pp. 278–292. ISBN 978-0804705967. [Google Scholar]
- Brown, M.B.; Forsythe, A.B. Robust test for the equality of variances. J. Am. Stat. Assoc.
**1974**, 69, 364–367. [Google Scholar] [CrossRef] - Jacquez, J.A.; Norusis, M. Sampling experiments on the estimation of parameters in heteroscedastic linear regression. Biometrics
**1973**, 29, 771–780. [Google Scholar] [CrossRef] - Miller, J.N.; Miller, J.C. Statistics and Chemometrics for Analytical Chemistry, 6th ed.; Prentice Hall: Harlow, UK, 2010; ISBN 978-0273730422. [Google Scholar]
- Voigtman, E. Limits of Detection in Chemical Analysis; Wiley: Hoboken, NJ, USA, 2017; ISBN 978-1119188971. [Google Scholar]
- Oppenheimer, L.; Capizzi, T.P.; Weppelman, R.M.; Mehta, H. Determining the lowest limit of reliable assay measurement. Anal. Chem.
**1983**, 55, 638–643. [Google Scholar] [CrossRef] - Zorn, M.E.; Gibbons, R.D.; Sonzogni, W.C. Weigthed least-squares approach to calculating limits of detection and quantification by modeling variability as a function of concentration. Anal. Chem.
**1997**, 69, 3069–3075. [Google Scholar] [CrossRef] [PubMed] - Sanagi, M.M.; Ling, S.L.; Nasir, Z.; Wan Ibrahim, W.A.; Naim, A.A. Comparison of signal-to-noise, blank determination, and linear regression methods for the estimation of detection and quantification limits of volatile organic compounds by gas chromatography. J. AOAC Int.
**2009**, 92, 1833–1838. [Google Scholar] [PubMed] - Ismail, R.; Lee, H.Y.; Mahyudin, N.A.; Bakar, F.A. Linearity study on detection and quantification limits for the determination of avermectins using linear regression. J. Food Drug Anal.
**2014**, 22, 407–412. [Google Scholar] [CrossRef] [PubMed] - Zorn, M.E.; Gibbons, R.D.; Sonzogni, W.C. Evaluation of approximate methods for calculating the limit of detection and quantification. Environ. Sci. Technol.
**1999**, 33, 2291–2295. [Google Scholar] [CrossRef] - Desimoni, E.; Brunetti, B. About estimating the limit of detection of heteroscedastic analytical systems. Anal. Chim. Acta
**2009**, 655, 30–37. [Google Scholar] [CrossRef] [PubMed] - Sanchez, J.M. Ordinary least squares with laboratory calibrations: A practical way to show students that this fitting model may easily yield biased results when used indiscriminately. World J. Anal. Chem.
**2017**, 5, 1–8. [Google Scholar] [CrossRef] - Raposo, F. Evaluation of analytical calibration based on least-square linear regression for instrumental techniques: A tutorial review. TRAC Trends Anal. Chem.
**2016**, 77, 167–185. [Google Scholar] [CrossRef] - Uhrovcik, J. Strategy for determination of LOD and LOQ values—Some basic aspects. Talanta
**2014**, 119, 178–180. [Google Scholar] [CrossRef] [PubMed] - Almeida, A.M.; Castel-Branco, M.M.; Falcao, A.C. Linear regression for calibration revisited: Weighting schemes for bioanalytical methods. J. Chromatogr. B Anal. Technol. Biomed. Life Sci.
**2002**, 774, 215–222. [Google Scholar] [CrossRef] - Tellinghuisen, J. Weighted least-squares in calibration: What difference does it make? Analyst
**2007**, 132, 536–543. [Google Scholar] [CrossRef] [PubMed] - Zeng, Q.C.; Zhang, E.; Dong, H.; Tellinghuisen, J. Weighted least squares in calibration: Estimating data variance functions in high-performance liquid chromatography. J. Chromatogr. A
**2008**, 1206, 147–152. [Google Scholar] [CrossRef] [PubMed] - Gu, H.; Liu, G.; Wang, J.; Aubry, A.F.; Arnold, M.E. Selecting the correct weighting factor for linear and quadratic calibration curves with least-square regression algorithm in bioanalytical LC-MS/MS assays and impacts of using incorrect weighting factors on curve stability, data quality, and assay performance. Anal. Chem.
**2014**, 86, 8959–8966. [Google Scholar] [CrossRef] [PubMed] - Mulholland, M.; Hibbert, D.B. Linearity and the limitations of least squares calibration. J. Chromatogr. A
**1997**, 762, 73–82. [Google Scholar] [CrossRef] - Vanatta, L.E.; Coleman, D.E. Calculation of detection limits for a single-laboratory ion-chromatographic method to determine parts-per-trillion ions in ultrapure water. J. Chromatogr. A
**1997**, 770, 105–114. [Google Scholar] [CrossRef] - Luo, W.; Li, H.; Zhang, Y.; Ang, C.Y.W. Determination of formaldehyde in blood plasma by high-performance liquid chromatography with fluorescence detection. J. Chromatogr. B Biomed. Sci. Appl.
**2001**, 753, 253–257. [Google Scholar] [CrossRef] - Ribani, M.; Collins, C.H.; Bottoli, C.B.G. Validation of chromatographic methods: Evaluation of detection and quantification limits in the determination of impurities in omeprazol. J. Chromatogr. A
**2007**, 1156, 201–205. [Google Scholar] [CrossRef] [PubMed] - Proestos, C.; Sereli, D.; Komaitis, M. Determination of phenolic compounds in aromatic plants by RP-HPLC and GC-MS. Food Chem.
**2006**, 95, 44–52. [Google Scholar] [CrossRef] - Muranaka, L.S.; Giorgiano, T.E.; Takita, M.A.; Forim, M.R.; Silva, L.F.C.; Coletta-Filho, H.D.; Machado, M.A.; de Souza, A.A. N-Acetylcysteine in agriculture, a novel use for an old molecule: Focus on controlling the plant-pathogen Xylella fastidiosa. PLoS ONE
**2013**, 8, e72937. [Google Scholar] [CrossRef] [PubMed] - Zabell, A.P.R.; Lytle, F.E.; Julina, R.K. A proposal to improve calibration and outlier detection in high-throughput mass spectrometry. Clin. Mass Spectrom.
**2016**, 2, 25–33. [Google Scholar] [CrossRef] - Meites, L.; Smit, H.C.; Kateman, G. The effects of errors in measuring the independent variable in least-squares regression analysis. Anal. Chim. Acta
**1984**, 164, 287–291. [Google Scholar] [CrossRef] - Gibbons, R.D.; Jarke, F.H.; Stoub, K.P. Detection limits for linear calibration curves with increasing variance and multiple future detection decisions. In Waste Testing and Quality Assurance: Third Volume; STP 1075; ASTM Publication Code Number 04-010750-56; Friedman, D., Ed.; ASTM: Philadelphia, PA, USA, 1991; pp. 377–390. ISBN 978-0-8031-1294-0. [Google Scholar]
- Kiser, M.M.; Dolan, J.W. Selecting the best curve fit. LCGC Eur.
**2004**, 17, 138–143. [Google Scholar]

**Figure 1.**Calibration curve and standardized residuals plot obtained for the method HPLC-UV#3 (determination of caffeine in coffee samples). Each calibration point is the mean of 8–10 replicate standards prepared and measured in different days. The error bars show the standard deviations obtained at each level. Solid line shows the linear trend obtained applying OLS (y = 121,810x + 224,242; R

^{2}= 0.99926), whereas the dashed line corresponds to the linear trend obtained by WLS (y = 124,904x + 1678; R

^{2}= 0.99896).

**Figure 2.**Chromatograms obtained to assess S/N for two of the methods evaluated in the present study (HPLC-UV #1, #2: determination of theobromine and caffeine in tea beverages): (

**a**) chromatogram for a matrix blank; (

**b**) the same blank spiked at 0.04 mg·L

^{−1}, a level close to the LOD value determined by WLS; (

**c**) chromatogram for the blank spiked at 0.30 mg·L

^{−1}, close to the LOD level calculated by OLS.

**Table 1.**Regression parameters obtained with different GC-FID methods. All calibrations were performed with a minimum of -six calibrations standards evenly distributed along the working range. The signal value for each calibration was determined as the mean value obtained for at least seven replicates prepared and measured in different days. s

_{res}= regression residual standard deviation; b

_{0}= y-intercept; s

_{b}

_{0}= y-intercept standard deviation; b

_{1}= slope of the calibration function; s

_{b}

_{1}= slope standard deviation; s

_{bl}= standard deviation of the blank.

Method | LOF | Levene | Model | s_{res} | b_{0} | b_{1} | b_{0} = 0 | b_{1} = 0 |
---|---|---|---|---|---|---|---|---|

(p-Value) | (p-Value) | (s_{b}_{0}) | (s_{b}_{1}) | (p-Value) | (p-Value) | |||

#1 | 0.419 | 0.001 | OLS | 0.0043 | 0.0029 | 6.73·10^{−4} | 0.981 | <0.001 |

(0.0029) | (6·10^{−6}) | |||||||

WLS | 0.4294 | 0.0053 | 6.64·10^{−4} | 0.004 | <0.001 | |||

(0.0009) | (6·10^{−6}) | |||||||

Blank | s_{bl}= | 0.0012 | ||||||

#2 | 0.952 | 0.008 | OLS | 0.0263 | 0.0023 | 1.73·10^{−3} | 0.901 | <0.001 |

(0.0177) | (4·10^{−5}) | |||||||

WLS | 1.5490 | 0.0147 | 1.67·10^{−3} | 0.079 | <0.001 | |||

(0.0063) | (3·10^{−5}) | |||||||

Blank | s_{bl}= | 0.0015 | ||||||

#3 | 0.107 | <0.001 | OLS | 0.01367 | 0.0096 | 1.69·10^{−3} | 0355 | <0.001 |

(0.0092) | (2·10^{−5}) | |||||||

WLS | 0.9568 | 0.0154 | 1.67·10^{−3} | 0.016 | <0.001 | |||

(0.0039) | (2·10^{−5}) | |||||||

Blank | s_{bl}= | 0.0045 | ||||||

#4 | 0.362 | <0.001 | OLS | 0.0143 | 0.0163 | 1.72·10^{−3} | 0.166 | <0.001 |

(0.0096) | (2·10^{−5}) | |||||||

WLS | 0.6781 | 0.0147 | 1.72·10^{−3} | 0.005 | <0.001 | |||

(0.0027) | (2·10^{−5}) | |||||||

Blank | s_{bl}= | 0.0043 | ||||||

#5 | 0.168 | <0.001 | OLS | 0.0088 | 0.0125 | 1.72·10^{−3} | 0.098 | <0.001 |

(0.0058) | (3·10^{−5}) | |||||||

WLS | 0.4307 | 0.0150 | 1.70·10^{−3} | 0.0004 | <0.001 | |||

(0.0013) | (2·10^{−5}) | |||||||

Blank | s_{bl}= | 0.0025 | ||||||

#6 | 0.745 | 0.002 | OLS | 5.78·10^{-3} | 8·10^{−3} | 3.94·10^{−3} | 0.158 | <0.001 |

(5·10^{−3}) | (4·10^{−5}) | |||||||

WLS | 0.3604 | 6·10^{−3} | 3.91·10^{−3} | 0.022 | <0.001 | |||

(2·10^{−3}) | (4·10^{−5}) | |||||||

Blank | s_{bl}= | 3·10^{−3} |

**Table 2.**Regression parameters obtained with GC-MS methods. Experimental conditions as indicated in Table 1.

Method | LOF | Levene | Model | s_{res} | b_{0} | b_{1} | b_{0} = 0 | b_{1} = 0 |
---|---|---|---|---|---|---|---|---|

(p-Value) | (p-Value) | (s_{b}_{0}) | (s_{b}_{1}) | (p-Value) | (p-Value) | |||

#1 | 0.473 | <0.001 | OLS | 233,473 | 249,259 | 860,379 | 0.152 | <0.001 |

(140,875) | (13,251) | |||||||

WLS | 0.6620 | 27,495 | 887,238 | 0.515 | <0.001 | |||

(38,502) | (24,696) | |||||||

Blank | s_{bl}= | 33,580 | ||||||

#2 | 0.372 | <0.001 | OLS | 93,959 | 41,563 | 149,078 | 0.505 | <0.001 |

(56,865) | (2679) | |||||||

WLS | 0.7872 | 53,521 | 148,848 | 0.030 | <0.001 | |||

(16,267) | (4955) | |||||||

Blank | s_{bl}= | 22,673 | ||||||

#3 | 0.188 | <0.001 | OLS | 72,065 | −50,424 | 179,903 | 0.312 | <0.001 |

(43,615) | (3131) | |||||||

WLS | 1.3649 | −25,615 | 174,245 | 0.133 | <0.001 | |||

(13,614) | (8048) | |||||||

Blank | s_{bl}= | 9628 |

**Table 3.**Regression parameters obtained with HPLC-UV methods. Experimental conditions as indicated in Table 1.

Method | LOF | Levene | Model | s_{res} | b_{0} | b_{1} | b_{0} = 0 | b_{1} = 0 |
---|---|---|---|---|---|---|---|---|

(p-Value) | (p-Value) | (s_{b}_{0}) | (s_{b}_{1}) | (p-Value) | (p-Value) | |||

#1 | 0.378 | <0.001 | OLS | 52,459 | 57,359 | 385,884 | 0.083 | <0.001 |

(27,656) | (1804) | |||||||

WLS | 2.2863 | −2136 | 393,588 | 0.302 | <0.001 | |||

(1894) | (8524) | |||||||

Blank | s_{bl}= | 840 | ||||||

#2 | 0.147 | <0.001 | OLS | 62,530 | 47,160 | 400,992 | 0.194 | <0.001 |

(32,266) | (3074) | |||||||

WLS | 0.8077 | 5079 | 407,693 | 0.012 | <0.001 | |||

(1431) | (4671) | |||||||

Blank | s_{bl}= | 2458 | ||||||

#3 | 0.621 | <0.001 | OLS | 68,178 | 24,242 | 121,810 | 0.653 | <0.001 |

(50,056) | (1653) | |||||||

WLS | 0.5428 | 1678 | 124,904 | 0.845 | <0.001 | |||

(8024) | (2013) | |||||||

Blank | s_{bl}= | 13,719 | ||||||

#4 | 0.537 | <0.001 | OLS | 77,517 | 113,661 | 129,902 | 0.117 | <0.001 |

(56,913) | (1879) | |||||||

WLS | 0.7255 | 13,710 | 134,518 | 0.574 | <0.001 | |||

(23,893) | (2889) | |||||||

Blank | s_{bl}= | 32,336 | ||||||

#5 | 0.761 | <0.001 | OLS | 46,038 | −19,495 | 11,310 | 0.615 | <0.001 |

(35,823) | (118) | |||||||

WLS | 0.2274 | −50,238 | 11,444 | 0.004 | <0.001 | |||

(8227) | (87) | |||||||

Blank | s_{bl}= | 7538 | ||||||

#6 | 0.741 | 0.004 | OLS | 48,296 | −101,407 | 10,799 | 0.054 | <0.001 |

(37,580) | (124) | |||||||

WLS | 0.2353 | −73,067 | 10,682 | 0.002 | <0.001 | |||

(9659) | (85) | |||||||

Blank | s_{bl}= | 11,925 | ||||||

#7 | 0.600 | 0.002 | OLS | 68,095 | −93,167 | 10,995 | 0.154 | <0.001 |

(52,987) | (175) | |||||||

WLS | 0.7003 | −43,109 | 10,628 | 0.195 | <0.001 | |||

(27,742) | (229) | |||||||

Blank | s_{bl}= | 31,162 | ||||||

#8 | 0.774 | 0.007 | OLS | 104,152 | −163,207 | 18,540 | 0.114 | <0.001 |

(81,043) | (267) | |||||||

WLS | 0.8237 | −65,272 | 17,849 | 0.234 | <0.001 | |||

(46,590) | (383) | |||||||

Blank | s_{bl}= | 44,535 |

**Table 4.**Regression parameters obtained with CZE-UV methods. Experimental conditions as indicated in Table 1.

Method | LOF | Levene | Model | s_{res} | b_{0} | b_{1} | b_{0} = 0 | b_{1} = 0 |
---|---|---|---|---|---|---|---|---|

(p-Value) | (p-Value) | (s_{b}_{0}) | (s_{b}_{1}) | (p-Value) | (p-Value) | |||

#1 | 0.868 | <0.001 | OLS | 205 | 105 | 692 | 0.338 | <0.001 |

(99) | (6) | |||||||

WLS | 0.4109 | 89 | 709 | 0.008 | <0.001 | |||

(21) | (10) | |||||||

Blank | s_{bl}= | 48 | ||||||

#2 | 0.486 | <0.001 | OLS | 321 | 3 | 1031 | 0.986 | <0.001 |

(155) | (9) | |||||||

WLS | 1.2439 | −140 | 1049 | 0.112 | <0.001 | |||

(72) | (24) | |||||||

Blank | s_{bl}= | 66 | ||||||

#3 | 0.534 | <0.001 | OLS | 135 | 140 | 509 | 0.084 | <0.001 |

(65) | (4) | |||||||

WLS | 1.0296 | 44 | 536 | 0.166 | <0.001 | |||

(27) | (8) | |||||||

Blank | s_{bl}= | 29 |

Method | Blank | OLS | WLS (s_{b}_{0}) | ||
---|---|---|---|---|---|

(s_{bl}) | (s_{b}_{0}) | w_{i} = 1/s_{i}^{2} | w_{i} = 1/x_{i}^{2} | w_{i} = 1/y_{i}^{2} | |

GC-FID #1 | 6 mg·L^{−1} | 14 mg·L^{−1} | 12 mg·L^{−1} | 10 mg·L^{−1} | 10 mg·L^{−1} |

GC-FID #2 | 9 mg·L^{−1} | 34 mg·L^{−1} | 13 mg·L^{−1} | 10 mg·L^{−1} | 11 mg·L^{−1} |

GC-FID #3 | 9 mg·L^{−1} | 18 mg·L^{−1} | 17 mg·L^{−1} | 10 mg·L^{−1} | 11 mg·L^{−1} |

GC-FID #4 | 8 mg·L^{−1} | 19 mg·L^{−1} | 14 mg·L^{−1} | 8 mg·L^{−1} | 11 mg·L^{−1} |

GC-FID #5 | 5 mg·L^{−1} | 11 mg·L^{−1} | 11 mg·L^{−1} | 10 mg·L^{−1} | 10 mg·L^{−1} |

GC-FID #6 | 3 mg·L^{−1} | 4 mg·L^{−1} | 3 mg·L^{−1} | 3 mg·L^{−1} | 3 mg·L^{−1} |

GC-MS #1 | 0.2 ppbv | 0.5 ppbv | 0.1 ppbv | 0.1 ppbv | 0.1 ppbv |

GC-MS #2 | 0.5 ppbv | 1.3 ppbv | 0.7 ppbv | 0.6 ppbv | 0.6 ppbv |

GC-MS #3 | 0.2 ppbv | 0.8 ppbv | 0.3 ppbv | 0.2 ppbv | 0.2 ppbv |

HPLC-UV #1 | 0.01 mg·L^{−1} | 0.24 mg·L^{−1} | 0.02 mg·L^{−1} | 0.02 mg·L^{−1} | 0.02 mg·L^{−1} |

HPLC-UV #2 | 0.02 mg·L^{−1} | 0.27 mg·L^{−1} | 0.02 mg·L^{−1} | 0.02 mg·L^{−1} | 0.02 mg·L^{−1} |

HPLC-UV #3 | 0.4 mg·L^{−1} | 1.4 mg·L^{−1} | 0.2 mg·L^{−1} | 0.1 mg·L^{−1} | 0.1 mg·L^{−1} |

HPLC-UV #4 | 0.8 mg·L^{−1} | 1.4 mg·L^{−1} | 0.6 mg·L^{−1} | 0.2 mg·L^{−1} | 0.2 mg·L^{−1} |

HPLC-UV #5 | 2 μM | 10 μM | 2 μM | 3 μM | 3 μM |

HPLC-UV #6 | 4 μM | 11 μM | 3 μM | 3 μM | 3 μM |

HPLC-UV #7 | 10 μM | 16 μM | 9 μM | 5 μM | 6 μM |

HPLC-UV #8 | 8 μM | 14 μM | 9 μM | 8 μM | 8 μM |

CZE-UV #1 | 0.2 mg·L^{−1} | 0.5 mg·L^{−1} | 0.2 mg·L^{−1} | 0.2 mg·L^{−1} | 0.2 mg·L^{−1} |

CZE-UV #2 | 0.2 mg·L^{−1} | 0.5 mg·L^{−1} | 0.2 mg·L^{−1} | 0.1 mg·L^{−1} | 0.1 mg·L^{−1} |

CZE-UV #3 | 0.2 mg·L^{−1} | 0.8 mg·L^{−1} | 0.2 mg·L^{−1} | 0.1 mg·L^{−1} | 0.1 mg·L^{−1} |

© 2018 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Sanchez, J.M.
Estimating Detection Limits in Chromatography from Calibration Data: Ordinary Least Squares Regression vs. Weighted Least Squares. *Separations* **2018**, *5*, 49.
https://doi.org/10.3390/separations5040049

**AMA Style**

Sanchez JM.
Estimating Detection Limits in Chromatography from Calibration Data: Ordinary Least Squares Regression vs. Weighted Least Squares. *Separations*. 2018; 5(4):49.
https://doi.org/10.3390/separations5040049

**Chicago/Turabian Style**

Sanchez, Juan M.
2018. "Estimating Detection Limits in Chromatography from Calibration Data: Ordinary Least Squares Regression vs. Weighted Least Squares" *Separations* 5, no. 4: 49.
https://doi.org/10.3390/separations5040049