Optimisation of Cotinine Extraction from Fingernails Using Response Surface Methodology for Fourier Transform Infrared Spectroscopy Analysis

Abstract

The increasing use of electronic cigarettes (e-cigarettes) highlights the need for accessible and reliable biomarkers to assess nicotine exposure. Fingernails represent a non-invasive and stable keratin matrix capable of capturing the long-term incorporation of xenobiotics such as cotinine, the primary metabolite of nicotine. This study aimed to optimise cotinine extraction from fingernails using Response Surface Methodology (RSM) with a central composite design prior to quantification by attenuated total reflectance Fourier transform infrared (ATR-FTIR) spectroscopy. Three extraction variables were evaluated: NaOH concentration, extraction temperature, and extraction time. Numerical optimisation identified the optimal conditions as 3.74 M NaOH, 50 °C, and 40 min, yielding a predicted recovery of 84.06% with a high desirability value of 0.973. The calibration curve demonstrated excellent linearity (R² = 0.9998), with a limit of detection of 14.5 µg kg⁻¹ and a limit of quantification of 43.8 µg kg⁻¹. The RSM model exhibited strong predictive performance, with an R² of 0.9990, an adjusted R² of 0.9982, and a predicted R² of 0.9958, supported by a non-significant lack of fit and robust residual diagnostics. Application of the optimised protocol to real fingernail samples successfully differentiated e-cigarette smokers from non-smokers based on characteristic cotinine-associated FTIR spectral features and quantitative measurements, demonstrating the practical utility of the proposed method. Overall, this study establishes a rapid, chromatography-free, and cost-effective analytical approach for monitoring long-term nicotine exposure using keratin-based matrices.

1. Introduction

The use of electronic cigarettes (e-cigarettes) has increased substantially in Malaysia, with national data reporting a rise in prevalence from 9.8% in 2017 to 14.9% in 2022, particularly among current smokers and individuals aged 18–24 years [1,2]. E-cigarettes are designed to deliver nicotine without the combustion products generated by conventional tobacco smoking [3]. Nevertheless, nicotine remains the principal addictive constituent in vaping liquids, and repeated exposure may lead to dependence [4]. Following inhalation, nicotine is rapidly absorbed and subsequently metabolised into its more stable primary metabolite, cotinine [5].

Nicotine and cotinine are most commonly quantified in conventional biological matrices such as urine, saliva, and blood owing to their relatively high analyte concentrations and established analytical protocols [6,7]. However, nicotine has a short biological half-life of approximately 6–8 h, while cotinine typically persists for only 16–18 h [8], thereby limiting their suitability for long-term exposure assessment. In addition, biological sampling is inherently invasive and may reduce participant compliance [9,10]. These limitations have driven increasing interest in keratin-based matrices, such as fingernails and hair, which offer several advantages, including non-invasive collection, resistance to degradation, minimal biohazard risk, and extended retrospective detection windows resulting from their slow growth rates [7,11,12].

Meanwhile, chromatographic and immunoassay techniques remain the most established approaches for the detection of nicotine and cotinine in keratinaceous matrices. However, these methods typically require extensive sample preparation, specialised instrumentation, and substantial solvent consumption [13]. Fourier transform infrared (FTIR) spectroscopy offers a practical alternative by enabling rapid and non-destructive analysis with minimal reagent use. When combined with appropriate spectral pre-processing and calibration strategies, FTIR spectroscopy is capable of delivering both qualitative and quantitative analytical information [14]. Although chromatographic techniques are widely recognised as the gold standard for cotinine detection in keratin matrices, FTIR spectroscopy provides a complementary strategy for first-line screening and pattern recognition based on spectral variability. Accordingly, the novelty of this study lies in the development of a practical and rapid analytical framework for cotinine detection among e-cigarette users, enabling efficient discrimination between distinct user groups.

By comparison, chromatographic and immunoassay techniques exhibit excellent specificity and sensitivity in drug detection, attributable to their instrumental design. These methods enable quantitative and semi-quantitative analyses, in which the differentiation between smoker and non-smoker groups is primarily dependent on the concentration of cotinine or nicotine present in keratinised samples [15,16,17]. In contrast, FTIR spectroscopic analysis is capable of providing quantitative measurements while revealing characteristic spectral differences that discriminate between smokers and non-smokers. Such spectral variations may serve as valuable supporting data for future classification and pattern-recognition studies.

Because cotinine is typically present at low concentrations within the rigid keratin matrix, optimisation of extraction conditions is essential to enhance analytical performance. In this study, extraction parameters were optimised using Response Surface Methodology (RSM) with a central composite design (CCD). The CCD approach enables simultaneous evaluation of main effects, interaction effects, and curvature among experimental variables, thereby providing a more comprehensive optimisation strategy than conventional one-factor-at-a-time methods [18,19]. The effects of extraction-solution concentration, temperature, and extraction time were systematically investigated to establish a robust and efficient protocol for cotinine extraction from fingernails.

2. Methods

2.1. Optimising Cotinine Extraction Using RSM

Fingernail samples were collected from verified non-smokers following the provision of written informed consent. All experimental procedures were approved by the Ethics Committee of Management and Science University (MSU) (Ethics Code: MSU: EA-L2-01-FHLS-2024-06-0002). A total of 1 g of fingernail material was decontaminated using dichloromethane (DCM) under sonication for 90 min and subsequently dried at 50 °C [20]. To simulate cotinine incorporation into the keratin matrix, the cleaned fingernail samples were immersed in 10 mL of a 25 µg mL⁻¹ cotinine standard solution (Merck, Darmstadt, Germany; 99.9% purity) for seven days at room temperature (25 °C). Following equilibration, the spiked samples were used for extraction optimisation.

Extraction conditions were optimised using Response Surface Methodology (RSM) implemented through a central composite design (CCD). Three experimental factors were investigated: sodium hydroxide (NaOH) concentration, extraction temperature, and extraction time. The factor ranges were selected to ensure that all experimental runs remained within chemically feasible limits. In the RSM model, each factor was coded from −1 to +1 across the selected experimental range, as presented in

Table 1. Five-level-three-factor of CCD condition variables.

Independent Variables	Coded Symbol	Coded Level
		−α	−1	0	+1	+α
Concentration of NaOH (M)	A	−0.704482	1	3.5	6	7.70448
Temperature (°C)	B	9.54622	30	60	90	110.454
Time (min)	C	9.54622	30	60	90	110.454

. Using a face-centred central composite design approach, an alpha (α) value of 1.68179 was applied, resulting in a total of 20 experimental runs comprising 14 non-centre-point runs and six centre-point replicates.

The response variable measured was the percentage recovery of cotinine following the extraction process.

Table 2. Combination of different parameters suggested by RSM.

Run	Factor 1: NaOH (M)	Factor 2: Time (min)	Factor 3: Temperature (°C)
1	1	30	30
2	3.5	60	60
3	3.5	60	60
4	6	30	90
5	7.70448	60	60
6	1	30	90
7	3.5	60	60
8	3.5	60	9.54622
9	3.5	60	60
10	3.5	9.54622	60
11	0.704482	60	60
12	6	90	30
13	1	90	90
14	6	90	90
15	3.5	110.454	60
16	3.5	60	60
17	3.5	60	60
18	6	30	30
19	1	90	30
20	3.5	60	110.454

summarises the 20 experimental conditions generated by the RSM design. For each run defined by the CCD matrix, 20 mg of spiked fingernail sample was digested in a NaOH solution at the specified concentration, temperature, and extraction time. Following digestion, 1 mL of dichloromethane (DCM) and 0.2 mL of methanol were added for liquid–liquid extraction, after which 0.5 mL of 25% potassium hydroxide was introduced to reduce water content. The resulting organic phase was subsequently subjected to FTIR analysis for the quantification of cotinine recovery.

Experimental design, model fitting, and optimisation analyses were performed using Design-Expert version 13 (Stat-Ease Inc., Minneapolis, MN, USA). Model evaluation comprised analysis of variance (ANOVA), regression modelling, residual diagnostics, three-dimensional response surface visualisation, and confirmation experiments to validate the optimum conditions.

2.2. Quantification of Cotinine Using FTIR

Spectral measurements were acquired using an FTIR spectrometer (IRAffinity-1, Shimadzu, Kyoto, Japan) equipped with an attenuated total reflectance (ATR) accessory. Spectra were recorded in absorbance mode using 20 co-added scans per sample, a spectral resolution of 8 cm⁻¹, and a wavenumber range of 600–4000 cm⁻¹.

A calibration series comprising concentrations of 50, 100, 300, 500, and 1000 µg kg⁻¹ was prepared from a 1000 µg kg⁻¹ cotinine stock solution. Calibration performance was evaluated using the coefficient of determination (R²), limit of detection (LOD), and limit of quantification (LOQ). The LOD and LOQ values were calculated in accordance with the International Council for Harmonisation (ICH) Q2(R2) guidelines [21] using the following equations:

$$\mathrm{LOD}=3.3 \times \left({\displaystyle \frac{\sigma }{S}}\right)$$

(1)

$$\mathrm{LOQ}=10\times \left({\displaystyle \frac{\sigma }{S}}\right)$$

(2)

where $\sigma$ represents the standard deviation of the response and $S$ denotes the slope of the calibration curve.

In addition, performance parameters were evaluated by assessing the linearity of the calibration curve across a defined concentration range through examination of the relationship between analyte concentration and instrumental response, from which the correlation coefficient was obtained. The LOD is defined as the lowest amount of analyte that can be detected under the specified measurement conditions, whereas the LOQ corresponds to the lowest amount of analyte that can be quantified with acceptable accuracy and precision. Both parameters were determined based on the standard deviation of the calibration response and the slope of the calibration curve.

Spectral pre-processing, including baseline correction, vector normalisation, and Savitzky–Golay smoothing (second-order polynomial, 11-point window), was performed using OriginPro 2025 (OriginLab Corporation, Northampton, MA, USA) to standardise the spectra. Peak identification was conducted on both standard and extracted sample spectra to confirm the presence of cotinine and to ensure the absence of overlapping spectral contributions from the nail matrix.

3. Results and Discussion

3.1. Calibration Curve of Cotinine Standard

The cotinine standard was analysed using ATR-FTIR spectroscopy to obtain reference spectra for subsequent identification and quantification in extracted fingernail samples. Figure 1 presents the FTIR spectrum of the pure cotinine standard. Several distinct spectral features, together with their corresponding functional group assignments, are summarised in

Table 3. Spectra region detected for cotinine in serum sample and their respective bond characteristics. Reference: Borden et al. [22].

Spectra Region (cm−1)	Bond Characteristics
950–1200	In plane C-H and N-H bending
1277	Vibrations of C-N stretching
1400–1500	Bending vibrations of the CH3 and CH2 groups of cotinine (scissoring)
1570–1600	Stretching vibration of the C=C ring or the benzene ring in the backbone of cotinine
1690	C=O stretching vibration

.

Among the detected bands, the peak at 1277 cm⁻¹ was selected as the analytical marker owing to its strong intensity, spectral stability, and minimal interference from the keratin matrix, making it suitable for cotinine identification. Cotinine standards were analysed to construct a six-point linear calibration curve, which exhibited excellent linearity with a coefficient of determination (R² = 0.9998), as shown in Figure 2. The calculated LOD was 14.5 µg kg⁻¹, while the limit of quantification LOQ was 43.8 µg kg⁻¹, demonstrating that ATR-FTIR spectroscopy is capable of detecting cotinine at low concentrations in keratin matrices.

3.2. RSM Experimental Design

A total of 20 extraction experiments were conducted using three experimental factors: NaOH concentration, extraction temperature, and extraction time. The experimental runs and their corresponding cotinine recovery values are presented in

Table 4. Cotinine recovery yield for each experimental run.

Run	Factor 1: NaOH (M)	Factor 2: Time (min)	Factor 3: Temperature (°C)	Response 1: Cotinine Recovery (%)
1	1	30	30	81
2	3.5	60	60	81
3	3.5	60	60	80
4	6	30	90	82
5	7.70448	60	60	71
6	1	30	90	75
7	3.5	60	60	80
8	3.5	60	9.54622	63
9	3.5	60	60	82
10	3.5	9.54622	60	80
11	0.704482	60	60	82
12	6	90	30	50
13	1	90	90	65
14	6	90	90	54
15	3.5	110.454	60	50
16	3.5	60	60	80
17	3.5	60	60	79
18	6	30	30	74
19	1	90	30	70
20	3.5	60	110.454	70

. These experiments were designed to evaluate the influence of each factor on extraction efficiency and to identify the optimal extraction conditions. The resulting data were analysed using a CCD within the RSM framework.

3.3. Output Interpretation

RSM combined with a CCD was employed to evaluate the interactions among the three experimental factors and their effects on cotinine recovery. Model performance was assessed using ANOVA, regression modelling, diagnostic plots, and three-dimensional response surface visualisations. A confirmation experiment was subsequently conducted to validate the optimal extraction conditions predicted by the model.

3.3.1. ANOVA

The ANOVA results for the RSM model are summarised in

Table 5. Output of ANOVA analysis for the model.

Source	Sum of Squares	df	Mean Square	F-Value	p-Value
Model	2450.86	9	272.32	1166.09	<0.0001	significant
A-NaOH	58.77	1	58.77	251.65	<0.0001
B-Time	1027.42	1	1027.42	4399.49	<0.0001
C-Temperature	48.64	1	48.64	208.27	<0.0001
AB	220.50	1	220.50	944.20	<0.0001
AC	32.00	1	32.00	137.03	<0.0001
BC	24.50	1	24.50	104.91	<0.0001
A2	70.85	1	70.85	303.38	<0.0001
B2	567.17	1	567.17	2428.66	<0.0001
C2	474.95	1	474.95	2033.75	<0.0001
Residual	2.34	10	0.2335
Lack of Fit	1.00	5	0.2004	0.7515	0.6192	not significant
Pure Error	1.33	5	0.2667
Cor Total	2453.20	19
Std. Dev.		0.4833		R2		0.9990
Mean		72.80		Adjusted R2		0.9982
C.V. %		0.6638		Predicted R2		0.9958
				Adeq Precision		104.2447

. The model exhibited a very high F-value of 1166.09 with a corresponding p < 0.0001, indicating that the fitted model is highly significant for predicting cotinine recovery from fingernail samples. The F-value reflects the ratio of variance explained by the model to the residual variance; therefore, a higher F-value denotes stronger explanatory power. The extremely low p-value confirms that the likelihood of obtaining such a strong model due to random noise is negligible.

All linear terms (A: NaOH concentration, B: extraction time, and C: extraction temperature), interaction terms (AB, AC, and BC), and quadratic terms (A², B², and C²) were statistically significant (p < 0.0001), demonstrating that both individual factors and their combined and non-linear effects exert a substantial influence on cotinine extraction efficiency. These findings support the suitability of a full quadratic model for describing the response surface behaviour. The lack-of-fit test was not significant (F = 0.75, p = 0.6192), indicating that the discrepancy between the model predictions and experimental observations is not statistically meaningful. A non-significant lack of fit is desirable, as it suggests that the model adequately represents the experimental domain [23]. Furthermore, the residual error was low (mean square = 0.2335), providing additional evidence of the model’s accuracy and reliability.

Overall, the ANOVA results confirm that the quadratic RSM model is statistically robust, demonstrates strong predictive capability, and is well suited for optimising cotinine extraction from fingernails.

Model predictability was assessed using several statistical indicators. The coefficient of determination (R² = 0.9990) indicates that the fitted model explains more than 99% of the variability in cotinine recovery, reflecting an exceptionally strong model fit. The adjusted R² (0.9982), which accounts for the number of model terms, is in close agreement with the predicted R² (0.9958), with a difference in less than 0.2. This close alignment demonstrates excellent internal consistency and robust predictive performance.

The model also exhibited a very low standard deviation (0.4833) and a coefficient of variation in only 0.6638%, indicating high precision and excellent repeatability of the experimental data. Furthermore, the adequate precision value of 104.24, which far exceeds the recommended threshold of 4, signifies an exceptionally high signal-to-noise ratio. This result confirms that the model has sufficient resolution to reliably navigate the design space. Collectively, the high R² values, strong agreement between the adjusted and predicted R² values, low variability, and excellent adequate precision demonstrate that the quadratic RSM model is statistically robust and highly reliable for predicting cotinine recovery [24,25].

3.3.2. Regression Equation

The model coefficients, expressed in terms of coded factors, are presented in

Table 6. Coefficient in terms of coded factor of the model.

Factor	Coefficient Estimate	df	Standard Error	95% CI Low	95% CI High	VIF
Intercept	82.68	1	0.1931	82.25	83.11
A-NaOH	−2.29	1	0.1446	−2.62	−1.97	1.08
B-Time	−8.67	1	0.1308	−8.96	−8.38	1.0000
C-Temperature	1.89	1	0.1308	1.60	2.18	1.0000
AB	−5.25	1	0.1709	−5.63	−4.87	1.0000
AC	2.00	1	0.1709	1.62	2.38	1.0000
BC	−1.75	1	0.1709	−2.13	−1.37	1.0000
A2	−2.73	1	0.1567	−3.08	−2.38	1.08
B2	−6.25	1	0.1267	−6.53	−5.96	1.01
C2	−5.72	1	0.1267	−6.00	−5.43	1.01

. These coefficients describe both the magnitude and direction of the relationship between each experimental factor and the response variable. Coded units were employed to standardise factor scales and to minimise multicollinearity among the variables. A positive coefficient indicates that an increase in the corresponding factor results in higher cotinine recovery, whereas a negative coefficient denotes an inverse relationship [26].

The precision of each model term was evaluated using the standard errors of the coefficient estimates. Smaller standard errors indicate more precise parameter estimation and greater confidence in the influence of each factor on the response variable. In this model, the standard errors were consistently low across all linear, interaction, and quadratic terms, ranging from 0.1267 to 0.1931, demonstrating excellent estimation precision.

Among the linear terms, extraction time (B) and extraction temperature (C) exhibited the lowest standard errors (0.1308), indicating that these coefficients were estimated with the highest precision. The interaction terms (AB, AC, and BC) and quadratic terms (A², B², and C²) likewise displayed comparably low standard errors, further confirming the stability of the model. In addition, the 95% confidence intervals for all coefficients were narrow, reinforcing the reliability of the parameter estimates.

Multicollinearity was assessed using the variance inflation factor (VIF). Multicollinearity is generally considered problematic when VIF values exceed 5–10 or when condition indices exceed 10–30 [27]. In the present model, all VIF values ranged from 1.00 to 1.08, which is well below the accepted thresholds. This indicates that the predictor variables are largely independent and that the regression coefficients were not inflated by inter-factor correlations. Collectively, the low standard errors, narrow confidence intervals, and minimal multicollinearity demonstrate that the regression coefficients were estimated with high precision, further supporting the robustness and reliability of the fitted RSM model.

The final regression equation, expressed in terms of actual (uncoded) factors, describes the combined influence of each experimental variable on cotinine recovery. In this model, A represents NaOH concentration, B denotes extraction temperature, and C corresponds to extraction time. The equation comprises linear (A, B, C), quadratic (A², B², C²), and interaction (AB, AC, BC) terms. Collectively, these terms enable the prediction of cotinine recovery across any specified combination of factor levels within the experimental domain. The final regression equation in actual factor units is presented below:

$$Cotinine Recovery = 82.68-2.29A-8.67B+1.89C-5.25AB+2.00AC-1.75BC-2.73A^2-6.25B^2-5.72C^2$$

(3)

Further interpretation of the regression Equation (3) indicates that the intercept term of 82.68 represents the predicted cotinine recovery when all variables are set at their respective centre levels. Cotinine recovery increases proportionally with increasing extraction time, decreasing NaOH concentration, and decreasing extraction temperature. Among the linear terms, extraction temperature exhibits the strongest influence on the response, as reflected by the magnitude of its coefficient.

The interaction term AB indicates an antagonistic effect between NaOH concentration and extraction temperature, whereas the positive AC interaction coefficient reveals a synergistic interaction between NaOH concentration and extraction time. Although the BC interaction term is negative, its magnitude is smaller, indicating a comparatively weaker antagonistic effect relative to AB.

All quadratic terms are negative, demonstrating pronounced curvature in the response surface and indicating the presence of a well-defined maximum within the experimental domain. Among these quadratic effects, extraction temperature and extraction time contribute more strongly to the curvature than NaOH concentration, suggesting that these variables play a dominant role in defining the optimal extraction conditions.

3.3.3. Residuals Plot

Residual plots were employed to assess the adequacy of the fitted model (Figure 3). A residual is defined as the difference between the observed and predicted response values, and its behaviour provides critical information regarding model validity [28]. The first diagnostic evaluated was the normal probability plot, which assesses whether the residuals follow a normal distribution. When this assumption is satisfied, the residuals are expected to align closely along a straight line with only minor and random deviations [29]. In contrast, pronounced curvature or an S-shaped pattern would indicate deviation from normality and may suggest the need for transformation of the response variable. As illustrated in Figure 3A, the residuals closely follow a straight line with minimal scatter, confirming that the normality assumption is met.

The second diagnostic examined was the plot of residuals versus predicted values, which evaluates the assumption of constant variance (homoscedasticity). Ideally, the residuals should be randomly distributed with no discernible pattern. A funnel-shaped or megaphone-shaped distribution would indicate non-constant variance, potentially necessitating corrective data transformation [30]. As shown in Figure 3B, the residuals are randomly scattered and confined within a consistent range across all predicted values, indicating stable variance throughout the dataset.

The residuals versus run order plot was also examined to identify any hidden or time-dependent factors that may have influenced the experimental outcomes. A random distribution of residuals around the zero line indicates that no external or time-related variables affected the response [31]. As shown in Figure 3C, the residuals are scattered without systematic trends, confirming that the experimental sequence did not introduce bias or instability.

Figure 3D presents the plot of predicted values against experimentally observed values. The close alignment of the data points along the reference line demonstrates strong agreement between predicted and measured responses, indicating that the model accurately captures the relationship between the extraction parameters and cotinine recovery.

Collectively, the normal probability plot, residuals versus predicted values plot, residuals versus run order plot, and predicted versus actual values plot confirm that the developed model is valid, well fitted, and statistically robust.

3.3.4. Three-Dimensional Graphs

Three-dimensional response surface plots and corresponding contour plots were employed to further interpret the behaviour of the fitted model. These graphical tools provide a visual representation of the response surface and offer deeper insight into the individual effects of the experimental factors, as well as their interactions, on cotinine recovery. In this study, the three investigated factors were NaOH concentration, extraction temperature, and extraction time. The shape and curvature of the response surfaces reflect the combined influence of paired factors on cotinine recovery, with steeper surfaces indicating stronger interaction effects and flatter surfaces suggesting weaker interactions. The contour plots provide additional clarity by delineating regions of higher and lower predicted recovery. Together, these visual representations facilitate the identification of favourable factor combinations that lead to enhanced extraction efficiency.

NaOH concentration with extraction time.

Figure 4 illustrates the three-dimensional response surface depicting the interaction between NaOH concentration (A) and extraction time (B) on cotinine recovery, with extraction temperature (C) held constant at 60 °C. The response surface exhibits a clear upward trend in cotinine recovery with increasing NaOH concentration and extraction time. The red-shaded region corresponds to the highest predicted recovery values, indicating favourable extraction performance, whereas the green and blue regions represent lower recovery yields. The pronounced curvature of the surface reflects a strong interaction between NaOH concentration and extraction time.

At lower NaOH concentrations (approximately 1–2 M), extending the extraction time results in only modest improvements in cotinine recovery. In contrast, as the NaOH concentration increases towards 5–6 M, the response surface rises sharply, particularly when the extraction time exceeds approximately 60 min. This behaviour suggests that efficient cotinine release from the fingernail keratin matrix requires both sufficient alkalinity and adequate digestion time to facilitate keratin breakdown.

The contour plot at the base of the surface further highlights this interaction, showing a progressive transition from low-recovery regions to high-recovery zones as both factors increase. Overall, the figure indicates that optimal cotinine extraction is achieved under conditions combining higher NaOH concentrations with longer extraction durations, consistent with the statistical significance of the linear and interaction terms identified in the RSM analysis.

NaOH concentration with extraction temperature.

Figure 5 illustrates the combined effects of NaOH concentration (A) and extraction temperature (C) on cotinine recovery, with extraction time (B) held constant at 60 min. The response surface shows a clear increase in cotinine recovery with increasing NaOH concentration, with the highest predicted values occurring at elevated NaOH levels and moderate temperatures. The warm-coloured region (orange to red) denotes the optimal extraction conditions, where cotinine recovery exceeds 80%.

The curvature of the surface reveals a pronounced interaction between NaOH concentration and extraction temperature. At lower NaOH concentrations (approximately 1–2 M), increasing the temperature results in only modest improvements in recovery. In contrast, at higher NaOH concentrations (5–6 M), temperature exerts a more substantial influence: moderate temperatures (approximately 50–70 °C) promote enhanced cotinine release, whereas excessively high temperatures (≥80–90 °C) lead to a decline in recovery. This behaviour suggests that very high temperatures, particularly when combined with strong alkalinity, may promote thermal degradation or adversely affect cotinine stability.

The contour plot at the base of the surface highlights a broad, curved region corresponding to the zone of maximum predicted recovery. As both NaOH concentration and extraction temperature approach their optimal mid-to-high values, the contour lines form a smooth, elevated plateau, corroborating the significant interaction and quadratic effects identified in the statistical analysis.

The figure demonstrates that optimal cotinine extraction is achieved under conditions of high NaOH concentration coupled with moderate extraction temperatures, consistent with the model’s prediction of a broad and stable optimum region within this factor space.

Extraction temperature with extraction duration.

Figure 6 depicts the interactive effects of extraction time (B) and extraction temperature (C) on cotinine recovery, with NaOH concentration maintained at 3.5 M. The response surface exhibits a pronounced curvature, indicating that cotinine recovery initially increases with both factors but begins to decline once the upper limits of extraction time or temperature are exceeded. The red-shaded region denotes the optimal zone of the surface, corresponding to cotinine recovery values exceeding 80%. The plot demonstrates that moderate extraction times (approximately 50–70 min) combined with mid-range temperatures (approximately 50–70 °C) yield the highest recovery.

At shorter extraction times, increasing temperature alone is insufficient to maximise cotinine release, as alkaline digestion requires adequate time to facilitate keratin breakdown. Conversely, extending the extraction time beyond approximately 70–80 min or increasing the temperature above approximately 75–80 °C results in a downward slope of the response surface, indicating reduced recovery. This decline may be attributed to thermal instability or partial degradation of cotinine under prolonged heating conditions.

The contour plot at the base of the figure reinforces these observations by revealing a well-defined elliptical region that represents the optimal combination of extraction time and temperature. This pattern confirms a significant interaction between the two factors, consistent with the RSM analysis in which both the linear and quadratic terms for extraction time and temperature were found to be statistically significant.

Overall, the figure highlights that optimal cotinine extraction is achieved under moderate extraction times and temperatures, whereas excessively long extraction durations or high temperatures lead to diminished recovery.

3.4. Optimisation and Validation

Numerical optimisation was conducted to identify the most favourable combination of extraction conditions for maximising cotinine recovery. In this procedure, the response variable, cotinine recovery expressed as a percentage, was set to be maximised, while NaOH concentration, extraction temperature, and extraction time were constrained within their experimentally investigated ranges. This approach was adopted to simplify the optimisation process while ensuring practical feasibility and high recovery efficiency. Based on these criteria, the optimisation algorithm generated 24 potential solutions, as summarised in

Table 7. Solutions table of different optimisation parameters provided by RSM.

Number	NaOH	Time	Temperature	Cotinine Recovery	Desirability
1	3.744	40.001	50.000	84.061	0.973	Selected
2	3.705	40.001	50.000	84.060	0.973
3	3.796	40.005	50.000	84.060	0.973
4	3.630	40.006	50.000	84.055	0.973
5	3.716	41.026	50.000	84.046	0.973
6	3.951	40.001	50.000	84.043	0.973
7	3.532	41.017	50.000	84.040	0.973
8	3.504	40.000	50.000	84.035	0.972
9	3.997	40.000	50.000	84.034	0.972
10	3.568	41.717	50.000	84.032	0.972
11	3.511	42.009	50.000	84.023	0.972
12	3.395	40.003	50.000	84.007	0.972
13	3.312	40.577	50.000	83.988	0.971
14	3.261	42.368	50.000	83.977	0.971
15	3.142	45.150	50.000	83.887	0.968
16	2.892	41.803	50.000	83.811	0.966
17	2.680	46.848	49.999	83.686	0.962
18	2.584	46.472	50.000	83.646	0.961
19	3.835	46.974	50.000	83.612	0.960
20	2.418	49.625	50.000	83.452	0.956
21	2.367	49.584	50.000	83.429	0.955
22	2.414	40.001	50.000	83.285	0.951
23	2.202	43.465	50.000	83.278	0.951
24	2.001	54.671	50.000	82.892	0.940

.

Figure 7 presents the desirability scores corresponding to the optimised extraction conditions. The desirability value obtained for the most favourable solution was close to 1.0, indicating that the selected parameter set effectively satisfied the optimisation criteria. Based on this analysis, the final optimised conditions for cotinine extraction were identified as a NaOH concentration of 3.74 M, an extraction temperature of 50.0 °C, and an extraction time of 40 min.

The predictive capability of the statistical model was evaluated through validation experiments. A confirmation test was conducted in triplicate using the optimised extraction conditions. The results of the confirmation analysis at the 95% confidence level are presented in

Table 8. Confirmation analysis of the optimised parameters.

Response	Predicted Mean	Predicted Median	Observed	Std Dev	SE Mean	95% PI Low	95% PI High
Cotinine Recovery	84.06	84.06	84.53	0.48	0.34	83.3	84.8

. The mean cotinine recovery obtained from the three replicates was 84.5%, which lies within the model-predicted range of 83.3% to 84.8%. This close agreement between the predicted and experimental values confirms the reliability of the optimised extraction conditions and demonstrates that the developed model is effective for predicting cotinine recovery under the selected experimental conditions.

3.5. Application of the Optimised Extraction Conditions to Real Nail Samples

To evaluate the practical utility of the optimised parameters, the extraction protocol was applied to real fingernail samples obtained from e-cigarette smokers and non-smokers. A total of 20 fingernail samples were collected, comprising 10 e-cigarette smokers and 10 non-smokers. The e-cigarette smokers included in this study had a minimum of six months of exclusive e-cigarette use and reported no consumption of conventional cigarettes. In addition to validating the optimised extraction parameters, this application served as a pilot study to generate preliminary data for future investigations, in which a larger sample size should be considered. The ATR-FTIR spectra obtained under the optimised conditions, as shown in Figure 8, revealed clear spectral differences between fingernail extracts from e-cigarette smokers and non-smokers, particularly within the diagnostic fingerprint region of 1700–900 cm⁻¹. In samples from e-cigarette smokers, cotinine-associated bands, notably at 1277 cm⁻¹ and within the 1200–950 cm⁻¹ region corresponding to C–N and C–C stretching vibrations, appeared sharper and more pronounced. These findings confirm the successful release and detection of cotinine from the keratin matrix, even under relatively mild alkaline extraction conditions.

Importantly, clear discrimination between e-cigarette smoker and non-smoker profiles was evident in fingernail samples based on spectral observations. Quantification of the peak at 1277 cm⁻¹ was performed to determine the absolute concentration of cotinine in the nail matrix, as presented in

Table 9. Concentration of cotinine in smokers and non-smokers by quantification of peak 1277 cm⁻¹. (n = 20), p < 0.05.

	Smokers	Non-Smokers
Concentration of cotinine (µg/kg)	1277.96	ND
Standard deviation	279.63	N/A

. The mean cotinine concentration in the smoker group was 1277.96 ± 279.63 µg kg⁻¹, whereas cotinine was not detected in samples from the non-smoker group. In addition,

Table 10. Precision (n = 3) and accuracy of the optimised extraction method.

	Precision	Accuracy
Concentration of cotinine (µg/kg)	1184.34 ± 16.73	47.75
RSD (%)	0.86	N/A
Recovery (%)	N/A	95.5

summarises the method validation performance of the optimised extraction protocol. All validation parameters were within acceptable limits, with a relative standard deviation of 0.86% and a recovery of 95.5%. Precision was assessed by repeating the extraction procedure three times using the same sample, expressed as the relative standard deviation and referred to as repeatability. Accuracy was evaluated by spiking a known amount of cotinine standard at 50 µg kg⁻¹ during extraction and subsequently determining the percentage recovery.

The combined evidence from qualitative spectral differentiation, enhanced cotinine-associated spectral features, and absolute quantification confirms the effectiveness of the optimised extraction conditions. These findings further demonstrate the practical applicability of the proposed method for long-term monitoring of nicotine consumption in clinical settings, enabling rapid detection of nicotine exposure and efficient discrimination in large-scale scenarios where conventional biological fluids may be unavailable. Moreover, the method shows potential for retrospective assessment of nicotine intake in occupational and sports-related contexts.

While the present study demonstrates the feasibility of the optimised extraction method with satisfactory performance outcomes, several aspects warrant further investigation to fully establish its robustness. Future work should incorporate chemometric approaches to enhance cotinine discrimination and quantification, particularly by capturing subtle spectral variations between groups, thereby improving model interpretability. In addition, systematic stability studies addressing analyte integrity under varying storage, processing, and environmental conditions are necessary. Expansion of the validation to larger and independent sample sets, as well as broadening the scope of analysis to include toenails, will be essential to confirm reproducibility and support wider implementation of the method.

Finally, although the optimisation of cotinine extraction from fingernails was based on immersion of cleaned fingernails in a cotinine solution to simulate drug incorporation into the keratin matrix, this approach represents a limitation of the present study, as it does not fully replicate the physiological incorporation process occurring in vivo.

3.6. Comparative Review of Cotinine Extraction Methods

Extensive research has reported the quantification of cotinine and nicotine in keratin-based matrices such as hair and nails, predominantly using chromatographic techniques. However, many of these studies place limited emphasis on the systematic optimisation of extraction conditions. Cotinine is a basic compound with a pKa value of approximately 4.8. Under alkaline conditions, it exists predominantly in its neutral form, which enhances its extractability during alkaline digestion and subsequent partitioning processes [32]. For this reason, numerous studies have employed sodium hydroxide solutions of varying concentrations, typically ranging from 0.1 M to 2.5 M. Reported extraction durations vary widely, from 30 min to 24 h, while extraction temperatures are commonly set at either room temperature or elevated values. Selected extraction strategies reported in the literature are summarised in Table 11.

A review of the published literature indicates that one of the most widely adopted extraction procedures involves alkaline digestion using 1 M NaOH at 50 °C overnight, which has been reported to yield acceptable analyte recovery [33,34,35,36,37,38,39]. These findings suggest that moderate heating combined with prolonged digestion facilitates the release of cotinine from the keratin matrix. In contrast, the present study demonstrates that comparable extraction performance can be achieved under substantially milder conditions, specifically using 3.74 M NaOH at 50 °C with an extraction time of 40 min, as identified through numerical optimisation.

This outcome indicates that extended heating or prolonged digestion is not necessarily required for efficient cotinine release when extraction parameters are systematically optimised using a statistical design approach. The effectiveness of the optimised mild conditions suggests that controlled alkalinity, even in the absence of high temperatures or lengthy extraction periods, is sufficient to disrupt the keratin structure and facilitate cotinine liberation. In addition to improving analytical efficiency, this approach reduces energy consumption and processing time while minimising the risk of thermal degradation or loss of analyte integrity, advantages that are not typically addressed in conventional extraction protocols.

Table 11. Summary of extraction parameters and alternative methods from various studies.

Author(s)	Extraction Parameters			Alternative	LOD; LOQ
	NaOH Concentration	Temperature (°C)	Duration
Kim et al. [39]	1 M	ambient	n/a	QuEchERS (dispersive SPE)	LLOQ: 10 pg/mg
Cashman and Nutt [15]	0.5 N	ambient	4 h	Extrelut-3 glass column (SPE)	n/a
Inukai et al. [40]	Using water	80	30 min	In tube-SPME	0.13 pg/mL; n/a
Tzatzarakis et al. [16]	1 M	60	90 min	-	0.015 ng/mg; 0.05 ng/mg
Lukrica et al. [41]	1 M	80	60 min	SPME	0.02 ng/mg; n/a
Yang et al. [42]	1 M	50	14 h	MISPE	0.2 ng/mL; 0.5 ng/mL

Table 11. Summary of extraction parameters and alternative methods from various studies.

Author(s)	Extraction Parameters			Alternative	LOD; LOQ
	NaOH Concentration	Temperature (°C)	Duration
Kim et al. [39]	1 M	ambient	n/a	QuEchERS (dispersive SPE)	LLOQ: 10 pg/mg
Cashman and Nutt [15]	0.5 N	ambient	4 h	Extrelut-3 glass column (SPE)	n/a
Inukai et al. [40]	Using water	80	30 min	In tube-SPME	0.13 pg/mL; n/a
Tzatzarakis et al. [16]	1 M	60	90 min	-	0.015 ng/mg; 0.05 ng/mg
Lukrica et al. [41]	1 M	80	60 min	SPME	0.02 ng/mg; n/a
Yang et al. [42]	1 M	50	14 h	MISPE	0.2 ng/mL; 0.5 ng/mL

Solid phase microextraction (SPME) has been applied particularly in cases where analyte concentrations are low [41]. This technique enables efficient preconcentration and sample clean-up under mildly alkaline conditions and allows shorter extraction times, with flexibility to operate at either ambient or elevated temperatures. Although SPME represents a promising alternative for cotinine extraction, it often requires specific combinations of sorbent materials and specialised fibres, which may increase operational costs and introduce method-specific variability [43]. Overall, the literature demonstrates a wide range of digestion and extraction conditions employed for nicotine and cotinine analysis in biological matrices. While this diversity reflects methodological flexibility, it also highlights the lack of standardised procedures across different analytical platforms.

When compared with limits of detection and quantification reported in previous studies, the current optimised method exhibits relatively higher values, primarily due to instrumental constraints. Nevertheless, the ability of the method to discriminate effectively between smoker and non-smoker groups is not compromised. Future research should therefore focus on direct comparisons of extraction techniques using the same analytical instrumentation in order to evaluate their relative efficiency and practical feasibility. Such studies may contribute towards the establishment of harmonised extraction protocols. In addition, investigations comparing data generated from different analytical platforms, including chromatographic techniques and FTIR spectroscopy, would be valuable for assessing agreement, accuracy, and reliability across methods.

4. Conclusions

The detection of nicotine and its primary metabolite, cotinine, in conventional biological fluids remains challenging due to their short detection windows and susceptibility to degradation, limiting their suitability for long-term exposure assessment. In this study, cotinine in nail matrices was successfully determined using ATR-FTIR spectroscopy following systematic optimisation of the extraction process through RSM and a CCD. Numerical optimisation identified 3.74 M NaOH, 50.0 °C, and an extraction time of 40 min as the most favourable conditions, achieving a predicted cotinine recovery of 84.06% with a high desirability value. The final RSM model demonstrated excellent predictive capability, supported by high R², adjusted R², and predicted R² values, together with a non-significant lack of fit, confirming that the model accurately describes the relationship between extraction parameters and cotinine recovery.

Application of the optimised method to real fingernail and toenail samples successfully differentiated e-cigarette smokers from non-smokers based on characteristic cotinine-associated FTIR bands within the fingerprint region. This finding demonstrates that the optimised protocol is effective not only under controlled experimental conditions but also in practical scenarios involving authentic biological specimens. The ability to analyse both fingernails and toenails further supports the suitability of keratin-based matrices as stable, non-invasive, and long-term biomarkers for nicotine exposure.

In summary, this study demonstrates that ATR-FTIR spectroscopy combined with an optimised mild alkaline extraction procedure provides a rapid, cost-effective, and chromatography-free approach for cotinine determination in nail samples. These findings establish a foundation for the expansion of nail-based nicotine biomonitoring and highlight the potential for future development of fully quantitative calibration models and standardised extraction protocols applicable to both clinical and forensic settings.