Classifying Dry Eye Disease Patients from Healthy Controls Using Machine Learning and Metabolomics Data

Abstract

Background: Dry eye disease is a common disorder of the ocular surface, leading patients to seek eye care. Clinical signs and symptoms are currently used to diagnose dry eye disease. Metabolomics, a method for analyzing biological systems, has been found helpful in identifying distinct metabolites in patients and in detecting metabolic profiles that may indicate dry eye disease at early stages. In this study, we explored the use of machine learning and metabolomics data to identify cataract patients who suffer from dry eye disease, a topic that, to our knowledge, has not been previously explored. As there is no one-size-fits-all machine learning model for metabolomics data, choosing the most suitable model can significantly affect the quality of predictions and subsequent metabolomics analyses. Methods: To address this challenge, we conducted a comparative analysis of eight machine learning models on two metabolomics data sets from cataract patients with and without dry eye disease. The models were evaluated and optimized using nested k-fold cross-validation. To assess the performance of these models, we selected a set of suitable evaluation metrics tailored to the data set’s challenges. Results: The logistic regression model overall performed the best, achieving the highest area under the curve score of $0.8378$, balanced accuracy of $0.735$, Matthew’s correlation coefficient of $0.5147$, an F1-score of $0.8513$, and a specificity of $0.5667$. Additionally, following the logistic regression, the XGBoost and Random Forest models also demonstrated good performance. Conclusions: The results show that the logistic regression model with L2 regularization can outperform more complex models on an imbalanced data set with a small sample size and a high number of features, while also avoiding overfitting and delivering consistent performance across cross-validation folds. Additionally, the results demonstrate that it is possible to identify dry eye in cataract patients from tear film metabolomics data using machine learning models.

1. Introduction

Dry Eye Disease (DED) is a multifaceted disorder characterised by a disruption in the composition, integrity, and stability of the tear film due to various internal and external factors. It is one of the most common reasons people seek eye care, with a severity spectrum ranging from minor, fleeting discomfort to severe, persistent pain and visual function impairment. This progression not only presents a substantial economic and healthcare challenge but also significantly impacts the quality of life of sufferers and the broader community. The incidence of DED notably increases following cataract surgery, highlighting the critical need for ophthalmologists to thoroughly evaluate for existing DED and to implement proactive and active treatment approaches. The presence of DED before surgery, can also complicate the precision of pre-surgical measurements, necessitate the reduction of intra-operative factors that could harm the ocular surface, and require the adoption of post-surgical care protocols to prevent the worsening of DED symptoms [1,2,3,4,5,6]. Clinical signs and symptoms are currently used to diagnose dry eye disease; however, the correlation between signs and symptoms is weak, leading to challenges in diagnosing and monitoring DED [7].

Previous research on diagnosing dry eye disease has utilized machine learning models applied to diverse data sets, including clinical data [8,9,10], imaging data [11,12,13], and patient-reported outcomes [14]. These studies have demonstrated the potential of machine learning to enhance diagnostic accuracy and identify relevant patterns associated with dry eye disease [14]. Different types of data represent distinct sources of information and can provide different insights into dry eye disease [15,16]. Omics data sets, such as genomics, proteomics, and metabolomics, can offer a deeper understanding of the disease by providing detailed molecular information related to genetic predisposition, underlying pathophysiology and current physiochemical status of the tear film as the end results of endogenous and exogenous processes. These molecular data provide a broader view of individuals, allowing for a variety of applications such as detection, personalized treatment, and disease monitoring with greater accuracy [1].

Advancements in omics technologies allow researchers to explore the genome, transcriptome, proteome, and more, providing in-depth insights into the molecular mechanisms underlying diseases. Despite their utility, single omics approaches are insufficient for comprehensively understanding the intricate interactions between genes, RNA, proteins, and environmental factors. Metabolomics distinguishes itself by revealing how metabolites, the dynamic output of gene, mRNA, and protein function, respond to various internal and external stimuli. This makes metabolomics a crucial instrument for unraveling complex biological processes and deepening our understanding of the origins, progression, and treatment outcomes of diseases [1].

In the context of DED, metabolomics holds the promise of identifying disease-specific metabolite profiles. These profiles can play an important role in enhancing the early diagnosis of DED, and in elucidating its etiology and pathology [1]. By identifying specific metabolic pathways and therapeutic targets, metabolomics can guide the choice of personalized treatment plans and improve the prediction of patient outcomes [1]. Furthermore, this approach can significantly enhance the monitoring of disease progression and the evaluation of treatment efficacy, ultimately leading to more effective management of DED [1,17]. By comparing metabolomic data from patients with that of a control group, researchers and clinicians can gain valuable insights, further enriching the biomedical and clinical understanding of DED.

In metabolomics research, the use of machine learning models is important for unraveling complex metabolic networks and identifying biomarkers. Each machine learning algorithm offers unique capabilities, necessitating careful selection to ensure the effectiveness of a study [18]. The choice of a machine learning algorithm significantly impacts the investigation’s success, as it must align with the specific characteristics of the data, including its dimensionality and the biological complexity [18]. Researchers dedicate efforts to evaluate the predictive performance of various machine learning algorithms, comparing them to traditional statistical methods to identify the most suitable approach for their specific needs [19].

However, selecting the optimal machine learning algorithm for metabolomics studies is challenging. Comparative studies often reveal that the effectiveness of a machine learning model can vary greatly depending on the research context and data attributes. This inconsistency underlines the fact that there is no one-size-fits-all machine learning algorithm for metabolomics [20,21,22,23,24]. Researchers are encouraged to develop a nuanced understanding of the advantages and operational intricacies of each machine learning model. By tailoring their choice of algorithm to the specific demands and nuances of their data, scientists can enhance the accuracy and interpretability of their findings [18]. Despite the lack of a universal guideline for the selection of machine learning algorithms, this iterative process of assessment and application is essential to advance metabolomics research and discover new biological insights.

This study focused on evaluating the efficacy of various machine learning models to classify cataract patients based on the presence or absence of DED, utilising metabolomics data collected from individuals awaiting cataract surgery. This approach marks an effort to explore the application of machine learning methodologies to this specific group of patients, to our knowledge, not previously undertaken.

Identifying effective machine learning models can help more accurately classify cataract patients with and without DED. This can also provide more reliable machine learning models for use by explainable artificial intelligence methods in identifying metabolomics signatures.

The following are our main contributions:

• Conducting a comparative study of machine learning techniques on challenging data sets characterized by imbalanced classes, low sample size, and high dimensionality.
• Classifying cataract patients with and without dry eye disease using metabolomics data from the participants’ tear films.
• Fine-tuning models using a nested k-fold cross-validation framework to identify the most effective models
• Employing early-stage integration strategy to combine metabolomics data from positive and negative modes to enhance model performance.
• Using a set of performance metrics suitable for the data set’s challenges to appropriately evaluate the developed models.

The rest of the paper is organized as follows: Section 2 details the data collection process, preprocessing steps, and the machine learning models used in this study, along with the evaluation metrics applied. Section 3 presents the performance of the machine learning models, including the impact of hyperparameter tuning, model consistency, and the interpretation of the most effective model. Section 4 presents the conclusions, followed by the limitations of the work and future directions in Section 3.5.

2. Materials and Methods

Our research methodology involved an initial preprocessing phase, in which the data sets were standardised and normalised. Subsequently, we employed the k-fold cross-validation technique to evaluate the performance of different machine learning models. Furthermore, for hyperparameter tuning, we ran an inner cross-validation with in the overall cross-validation. This research methodology, depicted in Figure 1, allowed us to assess both the immediate and optimised effects of machine learning applications on our data sets.

2.1. Data Description

The metabolomics data sets used in this study come from a clinical study of 222 patients scheduled for cataract surgery conducted from August 2020 to January 2022 at Ifocus Eye Clinic in Haugesund, Norway [25,26,27,28,29]. Due to a rigorous protocol, good mental health was required. Moreover, no systemic diseases that could affect the corneal surface were allowed. Patients were excluded if they had any manifestations of corneal disease, corneal scarring, or if they had previously undergone corneal refractive procedures [28]. Prior to cataract surgery, all participants were examined for dry eye disease using a set of clinical tests, with specific cut-off values detailed in previously published studies [25,28]. This selection process ensured that only eligible participants were included in the study, based on both their general health and the condition of their corneal surface.

The patients were sub-grouped into dry versus normal eyes based on clinical examination in a standardised manner. Of the 218 participants, tear samples harvested using Schirmer strips from 54 of the dry eye positive (DED+) group and 27 from the dry eye negative (DED-) group were selected randomly and subjected to complete global metabolomics analysis based on a well-validated global Liquid Chromatography–Mass Spectrometry (LC-MS) method at Oslo universitetssykehus [30]. The random selection of 81 patients from the larger pool of 218 ensured that the patients chosen for metabolomics analysis were not biased in any particular direction. Comprehensive metabolome coverage was obtained by analyzing the samples in both positive and negative ionization modes on the metabolomic instrument: Electrospray Ionization Positive (ESI+) and Electrospray Ionization Negative (ESI−) [30]. As most participants in the clinical study had dry eye disease, the number of samples with dry eye is greater than those without dry eye.

In the ESI+ mode, a total of 1922 metabolites were detected, of which 611 were given a proposed annotation by the Compound Discoverer software. The remainder are metabolite features without annotation. Similarly, the ESI− mode revealed 939 metabolites, of which 401 were annotated and the rest currently unidentified. The annotated metabolites include both metabolites identified at the highest level of confidence, and metabolites with lower levels of confidence of identification. Since identifying metabolites is a complex and resource-intensive task that requires careful examination and much work by biochemical experts, our aim is to first use machine learning to identify the most important metabolite features. Subsequently, this small group of key metabolite features will be analyzed in greater detail in our lab to try to accurately identify them at the highest level of confidence.

The metabolomics study followed the tenets of the Declaration of Helsinki and was approved by the Regional Committee for Medical and Health Research Ethics in Norway (Reference number 2020/140664). The data sets were anonymized prior to the machine learning experiments by removing all information that could potentially identify the participants. Ultimately, the data set consisted solely of the metabolomics data for the participants, ensuring that the anonymization process did not introduce any limitations to this study. Hence, the regional ethics committee was requested, and due to the true anonymity of the data sets, additional approval was not needed.

Previous research in the omics field has demonstrated that combining ESI+ and ESI− modes in mass spectrometry enables a more comprehensive analysis of samples, aimed at broadening the scope of analyte detection [31,32,33,34,35]. This dual-mode approach enhances the detection of diverse molecular components, increases the total number of assigned molecular features, and provides complementary data. These combined data sets result in a more complete chemical profile [36,37,38].

Merging the data sets from ESI+ and ESI− modes is complementary, as each mode ionizes and detects different types of molecules, although some metabolites can be detected in both ionization modes. The two modes together can access molecular features that are not detectable when using only one mode. This diversity in detected features enriches the data set, allowing machine learning models to learn from a broader and more varied set of data, which in turn enables the identification of more complex patterns and relationships. Consequently, this integration enhances the predictive capabilities of the machine learning models, although it increases the demand for memory and computational resources during model training. We employed an early integration approach to merge the data sets from both ESI+ and ESI− modes [39].

The metabolomics data sets from both ionization modes were merged, aiming to utilize all features of the patients for feeding the machine learning model. The integration of metabolomics data sets from both ionization modes was designed to augment the data set and enhance the predictive capability of machine learning models for dry eye disease in cataract patients. Information about the metabolomics data sets, categorized by each ionization mode is provided in

Table 1. Metabolomics datasets information.

	M1 ESI+	M1 ESI−
Samples	81	81
Metabolites	1922	939
DED−	27	27
DED+	54	54

Abbreviations: DED+ = Participants with dry eye disease, DED− = Participants without dry eye disease.

.

2.2. Data Preprocessing

Missing values were imputed using the default gap-filling algorithm in Compound Discoverer, which applies methods such as filling based on spectrum noise, simulated peaks, or trace area. As a result, no missing values remain in the data sets. Normalization is important in metabolomics data sets to mitigate the effects of variation arising from instrument drift [40,41]. The use of automatic normalization for each metabolite, based on an identical pooled quality control analyzed at regular intervals throughout the experiment, and logarithmic transformation helps mitigate this issue by stabilizing the variance across the data set. By taking the logarithm of the metabolite intensities, we reduce the influence of outliers and compress the dynamic range, making the data more homoscedastic. To address disparities in high and low-intensity features and to decrease the variability in data spread (heteroscedasticity), we standardized each metabolite’s logarithmic value. This standardization process consisted of subtracting the mean $\overline{x}$ of the logarithm of each metabolite value and dividing by its standard deviation (s) [42] as described in Equation (1). Next, quantile normalization was applied with the aim of further minimizing sample-to-sample variation, thus ensuring a more uniform data structure to subsequent analyses [43,44,45,46]. By applying quantile normalization, we adjust the distribution of metabolite intensities across all samples to a common reference distribution, thereby reducing sample-to-sample variability.

$${\widehat{x}}_{ij}=\left({\displaystyle \frac{{log}_2\left(x_{ij}\right)-\overline{x_i}}{s}}\right)$$

(1)

2.3. Machine Learning Models and Evaluation Metrics

For classifying cataract patients based on dry eye disease status, six supervised machine learning techniques such as XGBoost [47], Random Forest [48], Support Vector Machine [49], Multi Layer Perceptron [50], Logistic Regression [51], and K-Nearest Neighbor [52] were employed, and two dummy classifiers with different strategies were used as baselines for the performance benchmark. These particular supervised machine learning models were chosen because they represent a range of common methods in metabolomics research [18]. Furthermore, machine learning models, particularly tree-based models, are more appropriate for tabular data sets than deep learning models because they handle complex and non-linear relationships better by dividing the data into regions and fitting models within each region, effectively capturing irregularities [53]. Additionally, tree-based models require much less tuning and computational resources compared to deep learning models, which is particularly important in studies with limited data.

Dummy classifiers with a uniform strategy and a most frequent strategy are selected to predict each class with equal probability and to predict the most frequently observed class in the training data set, respectively, to use them as baselines for interpreting the machine learning model’s performance. The dummy classifiers do not use information from the features and are quite basic. Therefore, if a machine learning model performs better than the dummy classifiers, it indicates that the model has been able to learn from the features.

To accurately assess the effectiveness of the eight machine learning models deployed, it is important to employ suitable evaluation metrics. Consequently, in this study, a set of criteria was utilized for the evaluation, including the Area Under the Curve (AUC), Balanced Accuracy, Matthews Correlation Coefficient (MCC), Specificity, and the F1-Score. The selected metrics provide a clear view of the performance of the machine learning models when applied to metabolomics data sets, ensuring a reliable analysis of their predictive capabilities and overall accuracy. More details about the machine learning models and evaluation metrics can be found in Appendix A.

2.4. Code and Availability

The experiments for this study were conducted using the Python programming language within Google Colaboratory. This environment provided a Central Processing Unit (CPU) backend, equipped with 13 GB of RAM and 108 GB of storage space. The machine learning libraries utilized in the research included sklearn version $1.2.2$, numpy version $1.23.5$, seaborn version $0.13.1$, pandas version $1.5.3$, matplotlib version $3.7.1$, and scipy version $1.11.4$. The Jupyter notebooks employed in this research are accessible at the following location: https://github.com/sajadamouei/classification-metabolomics (accessed on 10 November 2024).

3. Results and Discussion

To explore the predictive capacity of machine learning models in imbalanced metabolomics binary classification data sets, we adopted an evaluation methodology. Data sets reflect a high-dimensional data structure common to metabolomics studies. Given the challenges posed by the data set’s imbalance and complexity, we implemented an approach to model development and validation to ensure the reliability of our findings.

Initially, we developed eight machine learning models on the entire data set. These models were selected based on their ability to handle challenges in the data set and their various learning mechanisms, providing a broad range of evaluations. The development and evaluation of these models were carried out using a stratified 10-fold cross-validation technique. We selected 10 for k in the k-fold cross-validation as it offers a good balance between bias and variance in model evaluation, considering the sample size [54]. Stratification in the 10-fold cross-validation involves dividing the data set into ten folds of equal size, ensuring that the distribution of labels is consistent across all folds, thereby maintaining the representativeness of the data set. In each iteration, nine folds were used to train the model, and the remaining fold served as a test set. This process was repeated ten times, each fold serving as a test set once, ensuring that every data point was used for both training and testing. The performance of each model was then evaluated based on the mean of the metrics obtained from the ten iterations, providing an aggregate measure of the model’s effectiveness across the entire data set.

In the field of machine learning, hyperparameters are variables that define the model’s structure or the characteristics of the learning algorithm, and are set before training begins. Unlike hyperparameters, other parameters like coefficients or weights change during training. The need and number of hyperparameters differ depending on the algorithm. To examine the impact of the model optimization, we enhanced our evaluation methodology by adopting a nested stratified cross-validation strategy for fine-tuning these hyperparameters.

Given the specific characteristics and challenges of our data sets, we opted for a limited search space tailored to each model to prevent overfitting, utilizing a grid search method for this purpose. In this approach, an additional stratified 5-fold cross-validation dedicated to hyperparameter tuning was conducted within each fold of the primary stratified 10-fold cross-validation. This nested setup enabled us to further partition the training folds from the outer loop into smaller sub-folds, helping to refine the model parameters and stabilize their performance across different folds. Here, four sub-folds were employed for training with different hyperparameter settings, while the fifth sub-fold acted as a validation set to evaluate these settings. The best-performing hyperparameter set on the validation sets from the five inner folds was then selected as the optimal configuration for each model. With these optimal hyperparameters, the model was trained anew on the complete training data set from the outer loop, prior to its assessment on the test fold.

Algorithm 1 illustrates the process of stratified 10-fold cross-validation, including the incorporation of nested stratified 5-fold cross-validation for hyperparameter tuning.

Algorithm 1: Stratified 10-Fold Cross-Validation

Input: Dataset D with N instances, each comprising a feature set and a class label. Output: Mean performance metrics of the models. Procedure: Stratify D by class labels to ensure class distribution is mirrored across folds. Divide D into 10 equal parts, $F_1,F_2,\dots ,F_{10}$, maintaining stratification. For each fold $F_i$: (a) Treat $F_i$ as the test set, and the remaining folds as the training set. (b) If tuning is needed, perform a stratified 5-fold CV on the training set to optimize hyperparameters. (c) Train the model on the training set, using optimized parameters if applicable. (d) Predict the true label for each data point in the test set, $F_i$ and store the predictions. Evalute the performance by comparing the the predictions from all the test sets $F_1,\dots ,F_{10}$ with true labels.

3.1. Results of Hyperparameter Tuning

To improve the performance of machine learning models, optimization and hyperparameter tuning for models such as Extreme Gradient Boosting (XGBoost), Random Forest (RF), Support Vector Machine (SVM), Multilayer Perceptron (MLP), Logistic ridge Regression (LR), and K-Nearest Neighbors (k-NN) were performed using the grid search in a nested stratified k-fold cross-validation approach. AUC metrics were used in the grid search to obtain the optimal hyperparameters for the machine learning models. In this study, the LR model was selected as the most suitable based on its performance across all evaluation metrics, which will be presented in detail in the next subsection. For the hyperparameters of the LR, the L2 penalty was chosen as the regularization method to help prevent overfitting. The ‘max_iter’ hyperparameter was set to 100 to define the maximum number of iterations allowed for the solvers to converge. Other hyperparameters selected for tuning included ‘C’, which controls the inverse of the regularization strength; ‘solver’, the algorithm used to optimize the model parameters; and ‘class_weight’, which addresses imbalances in the classes. These hyperparameters were tuned using the nested k-fold cross-validation approach and a grid search strategy. The search space and the optimal hyperparameters identified for the LR model are presented in

Table 2. Hyperparameter Tuning for Logistic Regression Model.

Hyperparameter	Search Area	Optimal Value
C	1.0, 0.1	1.0
Solver	lbfgs, newton-cg	lbfgs
Class Weight	None, balanced	None

.

3.2. Performance of Machine Learning Models

Table 3. Comparative Performance of Machine Learning Models on the ESI+ data set. The values presented are the averages obtained from the 10-fold cross-validation process.

Model	AUC	B. A.	MCC	Spec.	F1-Score
XGB	0.8167	0.6667	0.3345	0.5167	0.7937
RF	0.7631	0.7083	0.4776	0.4333	0.8669
SVM	0.7733	0.6517	0.3495	0.3833	0.8304
MLP	0.7494	0.6117	0.2579	0.55	0.6885
LR	0.8111	0.735	0.5147	0.5667	0.8513
K-NN	0.6292	0.6217	0.2832	0.3	0.8236
DCM	0.5	0.5	0	0	0.7987
DCU	0.5	0.4867	−0.0319	0.1	0.7522

Abbreviations: B. A. = Balanced Accuracy; Spec. = Specificity. Bold values indicate the best performance for the respective metric.

,

Table 4. Comparative Performance of Machine Learning Models on the ESI− data set. The values presented are the averages obtained from the 10-fold cross-validation process.

Model	AUC	B.A.	MCC	Spec.	F1-Score
XGB	0.7883	0.7117	0.4346	0.5167	0.8495
RF	0.5683	0.5283	0.0717	0.1333	0.7837
SVM	0.5372	0.4433	−0.1366	0	0.7367
MLP	0.7056	0.6333	0.2592	0.7167	0.637
LR	0.7311	0.615	0.2135	0.4167	0.7725
K-NN	0.4506	0.4533	−0.103	0.15	0.6859
DCM	0.5	0.5	0	0	0.7987
DCU	0.5	0.4867	−0.0319	0.1	0.7522

Abbreviations: B. A. = Balanced Accuracy; Spec. = Specificity. Bold values indicate the best performance for the respective metric.

and

Table 5. Comparative Performance of Machine Learning Models on the merged ESI+ and ESI− data set. The values presented are the averages obtained from the 10-fold cross-validation process.

Model	AUC	B. A.	MCC	Spec.	F1-Score
XGB	0.7861	0.72	0.4639	0.5667	0.8337
RF	0.7878	0.675	0.42	0.3667	0.8577
SVM	0.76	0.61	0.2583	0.3	0.8162
MLP	0.7811	0.6567	0.3419	0.45	0.8071
LR	0.8378	0.735	0.5147	0.5667	0.8513
K-NN	0.6614	0.6183	0.2756	0.3333	0.8083
DCM	0.5	0.5	0	0	0.7987
DCU	0.5	0.4867	−0.0319	0.1	0.7522

Abbreviations: B. A. = Balanced Accuracy; Spec. = Specificity. Bold values indicate the best performance for the respective metric.

summarize the efficacy of different machine learning models across three metabolomics data sets, using five evaluation metrics. The values presented in these tables are the averages obtained from the 10-fold cross-validation process. First, we examine performance obtained by machine learning models in three metabolomics data sets based on each metric, so that we can identify which data set is most suitable for machine learning models to detect dry eye in cataract patients.

The best AUC result achieved by machine learning models on the merged data set was $0.8378$, surpassing its effectiveness on the ESI− and ESI+ data sets by $0.0495$ and $0.0211$, respectively. The highest balanced accuracy recorded by machine learning models was $0.735$ in both the merged and ESI+ data sets, outperforming the ESI− data set by $0.0233$. Similarly, for the MCC metric, models performed better on the merged and ESI+ data sets, achieving a score of $0.5147$, which is an improvement of $0.0801$ over the ESI− data set. In terms of specificity, models scored highest on the ESI− data set at $0.7167$, exceeding scores in the merged and ESI+ data sets by $0.15$. For the F1-Score, the peak performance was observed in the ESI+ data set at $0.8669$, higher than in the merged and ESI− data sets by $0.0092$ and $0.0174$.

These results suggest that models generally performed better on the merged data set in terms of AUC, balanced accuracy, and MCC, though they fell slightly short in specificity and F1-Score. Therefore, these outcomes indicate that the merged data set can enhance model performance across some metrics, and achieve more balanced performance across all metrics, suggesting a beneficial effect of merging the ESI+ and ESI− data sets on model efficacy.

Following, we explore the effectiveness of various machine learning models, focusing on the merged data set, as detailed in Table 5, to find the most suitable models for identifying dry eye in cataract patients. The results in Table 5 reveal that, compared to baseline approaches that involve random guesses or selecting the most frequent class (represented by Dummy Classifiers), almost all the models demonstrated better performance across multiple evaluation metrics.

According to Table 5, the LR model has the highest AUC at $0.8378$. The AUC metric, commonly used in medical machine learning applications for disease prediction problems, shows that the LR model effectively differentiates between the two classes. Compared to other tested machine learning methods, this model achieves a higher AUC. After LR, the RF and XGBoost models recorded the next highest scores of $0.7878$ and $0.7861$, respectively. Due to the imbalance in the data set’s classes, other metrics besides the AUC were also considered to provide a clearer comparison of the models.

In terms of balanced accuracy, LR achieved the top score of $0.735$, outperforming other models in the merged data set. This metric highlights the model’s effectiveness on an imbalanced data set by valuing the performance across both minority and majority classes equally. XGBoost and RF followed with scores of $0.72$ and $0.675$, respectively.

For the MCC, LR again led with a score of $0.5147$, indicating its better performance in handling imbalanced data sets compared to other models. XGBoost and RF followed with scores of $0.4639$ and $0.42$, respectively.

For specificity, the LR and XGBoost achieved the highest score of $0.5667$, and MLP followed with a score of $0.45$. The negative samples are the minority classes in the data sets, making this metric important for demonstrating the models’ ability to correctly detect negative samples.

Regarding the F1-score, the RF model outperformed the others with a score of $0.8577$, closely followed by LR and XGBoost with scores of $0.8513$ and $0.8337$, respectively.

In summary, this comparative analysis indicates that the LR model achieved the highest performance in AUC, balanced accuracy, MCC, and specificity compared to other machine learning models in the merged data set. With a performance close to highest score in F1-score, it maintained balanced performance across all metrics compared to other machine learning models in this study on the merged metabolomics data set.

Previous research shows that the LR model was successful in biological and clinical contexts [55,56]. The results of this study demonstrated that the LR model outperforms other machine learning models and suggests that it is possible to achieve high performance even with a quite simple model. The more complex models, can quickly suffer from overfitting on the fairly small data set in this study. Additionally, LR is generally computationally less intensive and achieves convergence faster than more complex methods. This could be another reason for more stable model training and prediction, especially in this study’s data set, characterised by high-dimensional spaces and a limited number of samples. This is further discussed in Section 3.3.

The next best machine learning models following the LR model on the merged data set are XGBoost and RF, which are ensemble models. They achieved satisfactory performance on some evaluation metrics, but underperformed on others, and overall did not achieve consistent performance across all evaluation metrics.

Confusion matrix and ROC curve for the two best models, LR and XGBoost, are presented in Figure 2 to complement the reported metrics by providing detailed insights into the model’s classification results and their ability to distinguish between DED-positive and DED-negative patients.

In this study, the weakest results were observed with k-NN and SVM. Despite the theoretical benefits often associated with SVM, especially with an RBF kernel, in the field of bioinformatics and for high-dimensional data sets [18,57,58], the results presented in Table 5 demonstrate performance comparable to that of baseline dummy classifiers across several evaluation metrics on the merged data set. Both the k-NN and SVM models demonstrated suboptimal outcomes in this study. These results emphasize the need for a careful and nuanced approach when selecting models for metabolomics data sets.

Furthermore, integrating data sets from different ionization modes into a single data set has proven to improve model training by providing a wider range of features and patterns. This method enhances the data pool and helps models generalize better and increase prediction accuracy.

3.3. Model Performance Consistency

Figure 3 shows classification performance in terms of AUC along the y-axis and the standard deviation of the measured AUC from the different cross-validation folds along the x-axis. The standard deviation therefore gives an indication of the level of consistency in performance for different machine learning methods under repeated training of the algorithm. If a method has high classification performance over the folds, we also expect high classification constancy over the folds. Or said in another way, it is not possible to achieve high classification performance and at the same time have low consistency. Inspecting the three panels in Figure 3, we see that this is the overall trend, but there are for sure also differences in consistency for methods with about the same classification performance. Intuitively, we can expect that algorithms that are simple to train with a convex loss landscape, such as logistic ridge regression (based on a linear classifier), might document higher constancy compared to models that have a more complex loss landscape. We see that logistic ridge regression documents high consistency relative to classification performance for all the three data sets (Figure 3a–c).

Among the methods with a mean AUC over $0.75$ for the ESI+ data set (Figure 3a), we see that, in addition to logistic ridge regression, XGBoost also documents high constancy, while MLP, SVM, and RF document less consistency. Further, k-NN also documents poor constancy, which is as expected since the mean AUC is low.

Among the methods with a mean AUC over $0.7$ for the ESI− data set (panel Figure 3b), we see that only logistic ridge regression documents high consistency, while MLP and XGBoost document medium consistency. The other methods document poorer classification performance, and as expected, consistency is also low.

Among the methods with a mean AUC over $0.75$ for the merged data (Figure 3c), we see that in addition to logistic ridge regression, XGBoost also documents high consistency, while MLP, RF, and SVM document medium consistency.

To summarize, logistic ridge regression documents high constancy, XGBoost medium to high consistency in classification performance while the other methods document medium to low consistency.

3.4. Logistic Regression Model Interpretation

Table 6. Top 10 molecular features (metabolites) most strongly associated with dry eye disease, sorted by the size of the estimated logistic regression parameters.

Rank	M\Z	RT	Coefficient
1	460.26924	15.83	0.06868
2	227.08249	15.831	0.06539
3	149.09612	15.833	0.0641
4	236.0881	15.83	0.05915
5	499.16388	15.832	0.05755
6	245.09347	15.827	0.05497
7	137.04576	4.177	0.05428
8	350.98829	23.703	−0.0534
9	587.28264	15.829	0.0533
10	459.20277	15.866	0.05314

displays Mass-to-Charge Ratio (M/Z) and Retention Time (RT) values of features, highlighting the ten highest estimated regression parameters, which refer to the molecules most strongly associated with dry eye. Among the ten highest values, all molecules, except one, specifically features with ranks 1, 2, 3, 4, 5, 6, 7, 9, and 10 from Table 6, have positive estimates, indicating their association with an increased risk of dry eye. One molecule, with rank 8 from Table 6, has a negative estimate, suggesting its association with a reduced risk of dry eye. This was expected, as the data sets are imbalanced, with a majority being cataract patients with dry eye disease.

3.5. Limitations and Future Works

The machine learning models used in this study were evaluated on data sets obtained from a single clinic. While this limits the generalizability of the findings, it ensures that the data set analyzed is based on the biological characteristics of the samples rather than on variations in sampling techniques or sample handling procedures. However, we are not aware of any other data set with the same properties as the one used in this study (tear film metabolomics and dry eye disease). Additionally, the size of the metabolomics data could be increased by collecting more patient samples. This would enable the exploration of more complex models, such as deep learning and transfer learning, potentially improving the classification of cataract patients with and without dry eye disease.

Machine learning models trained on metabolomics data cannot currently be used directly in clinical medical systems due to the complexity of data collection and the high thresholds required for metrics such as specificity, precision, and sensitivity in clinical applications. However, it is important to note that our study was designed to identify the best-performing machine learning model and gain insights into metabolomic features associated with dry eye disease, rather than to produce a clinically deployable tool. These insights can then be further explored in detail through biological studies to identify biomarkers associated with dry eye disease. The identified biomarkers can in the future potentially be used in clinical settings in combination with other patient information, and together reach the required thresholds for a reliable medical decision system.

4. Conclusions

In this study, we focused on applying machine learning techniques to classify cataract patients, distinguishing between those with and without dry eye disease in a cohort of patients scheduled for cataract surgery. Dry eyes adversely affect patients’ quality of life and represent a risk factor for patients undergoing cataract surgery. Selecting the most suitable machine learning models for metabolomics data sets is crucial because no single model excels in all scenarios, and their accuracy and interpretability can influence subsequent steps in the application of these models in the metabolomics field. Therefore, in response to the challenge of selecting the optimal machine learning model for our specific imbalanced binary classification metabolomics data sets, we evaluated eight machine learning models to determine the most effective method for classifying cataract patients with and without dry eye disease.

To assess these models, we employed k-fold cross-validation across the entire data sets and used nested k-fold cross-validation for tuning hyperparameters. Additionally, multiple evaluation metrics were utilized to gain a clearer understanding of the models’ behavior and effectiveness. This is particularly important because the data sets in this study were imbalanced.

The results highlighted that the LR model outperformed other machine learning models on the merged metabolomics data set, achieving an AUC of $0.8378$, balanced accuracy of $0.735$, and MCC of 0$.5147$. It also achieved a specificity of $0.5667$ and an F1-score close to that of the best model, with values of $0.8513$, demonstrating overall balanced performance across all evaluation metrics. The LR model indicates consistent performance across 10-fold cross-validation, making it a reliable machine learning model in this study compared to other models in the metabolomics data set. In real-world clinical settings, the consistency of models is important, as it implies that the model’s performance is reliable across different patient populations and can generalize well to new, unseen samples. XGBoost and RF showed good performance on some evaluation metrics but could not maintain consistent performance across all metrics and 10-folds. SVM with RBF kernel and k-NN methods also showed poor performance in this study.

Furthermore, the results of this study demonstrate that the integration of data sets with positive and negative ionization modes enriches the models’ training process. This contributes to a more nuanced feature set that enhances generalization capabilities and prediction accuracy, positioning data set merging as a strategic approach to improve machine learning model efficacy for these specific metabolomics data sets.

The LR model outperformed other models in this study, and the results obtained were promising. We believe that the LR model could aid medical experts in detection of dry eye disease in cataract patients with metabolomics data sets.