A Novel Method for Medical Predictive Models in Small Data Using Out-of-Distribution Data and Transfer Learning

Abstract

Applying deep learning to medical research with limited data is challenging. This study focuses on addressing this difficulty through a case study, predicting acute respiratory failure (ARF) in patients with acute pesticide poisoning. Commonly, out-of-distribution (OOD) data are overlooked during model training in the medical field. Our approach integrates OOD data and transfer learning (TL) to enhance model performance with limited data. We fine-tuned a pre-trained multi-layer perceptron model using OOD data, outperforming baseline models. Shapley additive explanation (SHAP) values were employed for model interpretation, revealing the key factors associated with ARF. Our study is pioneering in applying OOD and TL techniques to electronic health records to achieve better model performance in scenarios with limited data. Our research highlights the potential benefits of using OOD data for initializing weights and demonstrates that TL can significantly improve model performance, even in medical data with limited samples. Our findings emphasize the significance of utilizing context-specific information in TL to achieve better results. Our work has practical implications for addressing challenges in rare diseases and other scenarios with limited data, thereby contributing to the development of machine-learning techniques within the medical field, especially regarding health inequities.

1. Introduction

Machine learning has emerged as a prominent field in the current medical research landscape. However, constructing effective machine-learning models, particularly with deep-learning (DL) techniques, often requires large amounts of data [1]. Acquiring the necessary volume of datasets can be time consuming and resource intensive, involving significant resource costs, including financial expenses. This challenge is especially pronounced in specialized medical fields, where data acquisition may be hindered by low-prevalence diseases, regional inequalities or standardization challenges [2]. When the ratio of training samples to the Vapnik–Chervonenkis (VC) dimensions of a learning machine is less than 20, it is considered a small sample size [3,4,5,6]. VC dimensions measure the capacity of a classifier, representing the cardinality of the greatest collection of points, which the procedure can break [4]. However, these theories do not apply to real-world scenarios with limited datasets for machine-learning models, as they primarily focus on generic machine learning with a high number of training samples [7]. Using limited datasets risks inadequate model training, potentially reducing the likelihood of achieving global minima [8]. Additionally, random weight initialization in machine-learning models may introduce further uncertainty, thereby highlighting the necessity for meticulous weight initialization in scenarios with limited data [9,10,11,12]. Moreover, small datasets pose various challenges, including overfitting problems, the significant impact of noise components, missing values, outliers and sharp fluctuations in variables within the dataset, resulting in low generalization ability [13].

Despite these challenges, small datasets possess intrinsic value, and ongoing research endeavors are addressing the mentioned issues [14,15]. Traditionally, augmentation methods, such as changing direction or adjusting angles, have been prevalent in the image domain [16]. Additionally, oversampling strategies are commonly employed in tabular data to increase the number of patient data, especially for minority groups [17]. Recent approaches utilizing generative adversarial networks or diffusion models for data synthesis aim to overcome these issues [18,19,20]. However, these strategies have yet to resolve trust issues related to generated data and require substantial resources. They are also predominantly limited to the field of images [21,22]. Other various algorithms, such as ensemble learning and input-doubling method [13,23,24,25,26], have also been actively researched. However, they mostly face similar challenges, and there is no clear consensus on technically feasible solutions, necessitating further research [13,27,28].

Furthermore, prior knowledge has improved predictive accuracy over random weight initialization in data-deficient contexts. As a result, the medical field has been actively exploring transfer learning (TL) to achieve this [29]. TL involves refining a pre-trained model on extensive datasets by adapting insights from one related task to another. TL can produce robust models capable of operating with sparse data, shortening training durations and improving model generalization [30]. However, its primary medical sector use is limited to imaging tasks, while electronic health record (EHR) data are often structured in tabular form, making it challenging to acquire compatible large-scale datasets for TL applications [29,30,31,32].

Meanwhile, the medical field frequently encounters diverse instances of out-of-distribution (OOD) data [33]. OOD data are generated from a distribution, which deviates from the one on which the model was initially trained [34]. In medical practice, OOD data are common in various scenarios, e.g., when data from different hospitals are used for external validation, when training and evaluation data are segregated, or when integrating data reflective of varying patient conditions or environments, even within the same medical condition [35]. OOD data often follow a distinct statistical distribution compared to in-distribution data and may exhibit contextual disparities [34]. The utilization of OOD data has enhanced machine models’ overall performance and robustness when employed in the right context [9,10,36].

This study presents an innovative method, which uses OOD data and TL to overcome data scarcity in machine-learning model development. We propose a machine-learning model for predicting acute respiratory failure (ARF) in patients with acute pesticide poisoning, incorporating a unique model development approach. Acute pesticide poisoning is a public health concern worldwide, and it is often accompanied by fatal outcomes [37]. ARF is a major cause of mortality in patients with acute pesticide poisoning, and it is known that the clinical course differs based on the category of pesticide, ingestion amount and underlying disease [38]. Since acute pesticide poisoning is rare, a lack of clinical experience can prevent predicting the prognosis of patients. Therefore, an ARF prediction model is essential for the timely treatment of patients with acute pesticide poisoning [39,40]. Acute pesticide poisoning is more prevalent in rural areas than in urban areas, leading to variations in data based on the location of medical institutions. Collecting data is exceptionally challenging due to the infrequency of cases, making our novel approach well suited for this problem [30].

The main contributions of this paper can be summarized as follows:

• Introducing a highly intuitive and simple idea and assessing the potential utility of OOD data in creating pre-trained models for TL.
• Experimentally validating the effectiveness of OOD and TL in small medical datasets while minimizing artificial data manipulations, such as data generation.
• Developing a predictive model for ARF in patients with acute pesticide poisoning using the proposed method, showcasing low bias and high performance.

The remainder of the paper is organized as follows. Section 2 encompasses the study population, labeling, feature selection, handling of outliers and missing values, and modeling. Section 3 presents the study participants’ characteristics, model performance and model interpretation. Section 4 engages in a thorough analysis of the outcomes, addressing limitations and future research. Lastly, Section 5 provides a summary of the study’s contributions and implications for the field. Additionally, in Appendix A, we present abbreviation descriptions (

Table A1. Abbreviation descriptions.

Category	Abbreviation	Full Form
Model	LGBM	Light Gradient-Boosting Machine
	LR	Logistic Regression
	MLP	Multi-Layer Perceptron
	RF	Random Forest
	XGB	Extreme Gradient Boosting
Metrics	AUROC	Area Under the Receiver Operating Characteristic
	AUPRC	Area Under the Precision–Recall Curve
	MCC	Matthews Correlation Coefficient
	SHAP	ShaHley Additive exPlanations
Features	BE	Base Excess
	DBP	Diastolic Blood Pressure
	GCS	Glasgow Coma Scale
	SBP	Systolic Blood Pressure
	Total CO2	Total Carbon Dioxide
ETC	ARF	Acute Respiratory Failure
	DL	Deep Learning
	EHR	Electronic Health Records
	IRB	Institutional Review Boards
	MICE	Multiple Imputation by Chained Equations
	MV	Mechanical Ventilation
	OOD	Out-of-Distribution
	TL	Transfer Learning

) along with tables and figures, which may aid in the understanding of the paper. In Appendix B, we offer additional experiments, which support and reinforce the experiments conducted in the manuscripts.

2. Materials and Methods

2.1. Study Population

The study was conducted on 129,953 patients aged 19 years and older admitted to the general ward at Korea University Anam Hospital between January 2015 and December 2021. To distinguish patients who experienced ARF from those who did not, we excluded 1508 patients who experienced ARF but had unclear onset times. These patients had not ingested pesticides and were considered as OOD data collected from different regions or hospitals.

A retrospective observational cohort study was conducted on 1081 patients with acute pesticide poisoning who were admitted to Soonchunhyang University Cheonan Hospital between January 2015 and December 2020. To ensure reliable results, exclusion criteria were established based on previous studies [37,41]. First, patients under the age of 19 were excluded, as were those who had been poisoned by paraquat-based pesticides, which are known to cause ARF within a short time. Considering the pattern of ARF occurrence and the study design, patients who were diagnosed with ARF within 1 h of admission or 72 h after admission were also excluded, as were patients with a “Do Not Resuscitate” status due to mechanical ventilator refusal (Figure 1). The final study cohort included 803 patients with acute pesticide poisoning.

The Institutional Review Boards (IRB) of Korea University Anam Hospital (IRB number: 2023AN0145) and Soonchunhyang University Cheonan Hospital (IRB number: 2020-02-016) reviewed and approved this study. The study was conducted following the principles outlined in the Helsinki Declaration.

2.2. Labeling

We considered the time of receiving mechanical ventilation as the onset of acute ARF. The study utilized data from two hospitals, each exhibiting distinct characteristics. As a result, specific research designs were implemented tailored to each dataset. Korea University Hospital data showed a notable scarcity of ARF cases, leading to a data imbalance issue.

The specific approach adopted to address this issue is illustrated in Figure 2. Patients who experienced ARF were labeled “1”, with a prediction time of 1–72 h before the onset of the condition. Patients who did not experience ARF were labeled “0”, with a prediction timeframe of 143–72 h before discharge to mitigate the uncertainty associated with the potential later onset after discharge. Data points outside the defined prediction timeframe were excluded, effectively rectifying the data imbalance issue.

Conversely, Soonchunhyang University Cheonan Hospital patients exhibited a different pattern, with ARF cases being more prevalent within 72 h of admission. This resulted in a less severe data imbalance. To maintain rigorous evaluation criteria and account for this pattern, a prediction timeframe of 1–72 h after admission was applied. Patients who experienced ARF were labeled “1”, while those who did not experience this condition were labeled “0”.

2.3. Feature Selection

The feature selection process was informed by prior studies [15,19]. We additionally consulted with experts to determine relevant features. We then excluded features, which were not commonly applicable to TL. For example, the Glasgow Coma Scale (GCS), considered a critical feature in past research, was excluded from our study due to its high rate of missing data in the Korea University Anam Hospital dataset. Considering the limited sample size and the need for rapid data assessment, we prioritized features, which are typically measured in the majority of patients within an hour, ensuring minimal missing data. As a result, we only selected variables missing from <5% of the Soonchunhyang University Cheonan Hospital data. The selected features included age, sex, systolic blood pressure (SBP), diastolic blood pressure (DBP), respiratory rate, body temperature, serum creatinine, hemoglobin, total carbon dioxide (Total CO₂), pH, pCO₂, pO₂, base excess (BE), lactate, category of pesticide and amount of ingestion. In this context, “sex” refers to the sex assigned at birth.

2.4. Handling of Outliers and Missing Values

To tackle potential outliers, values falling below the 2.5th percentile or exceeding the 97.5th percentile for each attribute were considered outliers and treated as missing values to eliminate their potential influence on the whole dataset. Subsequently, the multiple imputation by chained equations (MICE) algorithm was used to impute the missing data. MICE is widely used to generate imputations, which closely resemble true distributions when the rate of missing values is low [42]. Following this, robust scaling was applied. Notably, MICE and robust scaling computations were exclusively performed on the training data throughout all phases of the learning process.

Before performing the above pre-processing, the data obtained from Korea University Anam Hospital contained numerous missing values, necessitating additional pre-processing. First, we organized the features daily. Systolic blood pressure (SBP), diastolic blood pressure (DBP), respiratory rate and body temperature were arranged daily using the highest recorded values. The remaining attributes were assigned the last recorded values. Despite these efforts, any remaining missing data were then imputed by referencing the most recent observations. Furthermore, the pesticide category and ingestion amount were not available in the Korea University Anam Hospital dataset and were uniformly set to zero.

2.5. Modeling and Performance Evaluation

Predicting ARF in patients with acute pesticide poisoning is a challenging task due to the limited availability of such cases. This study used a large-scale OOD dataset of patients without acute pesticide poisoning to overcome this challenge to enhance ARF prediction. The study employs various machine-learning models, including logistic regression (LR), random forest (RF), extreme gradient boosting (XGB), light gradient-boosting machine (LGBM) and a multi-layer perceptron (MLP). The regression analysis, ensemble method and neural network models considered in our study are among the most commonly utilized models in the development of clinical prediction models [43,44]. Furthermore, since the data we are working with are not in the form of time series or images, we did not consider models such as recurrent neural networks or convolutional neural networks. Our novel approach to TL using OOD data is illustrated in Figure 3.

The first step of the approach is to develop an MLP model, which can predict ARF in patients without acute pesticide poisoning. This MLP model is a pre-trained model, serving as a foundation for fine-tuning using data from patients with acute pesticide poisoning. During the fine-tuning process, the number of trainable layers is adjusted, and each model variant is evaluated systematically. The initial MLP model consists of five dense layers, including the output layer. The number of trainable dense layers is varied systematically to create models ranging from TL1 to TL5. The specific model architecture can be examined in Figure A3.

To ensure the reliability of the models, cross-validation is carried out due to the limited number of patients in the cohort. The dataset is divided into five groups, with the ratio of patients with ARF to patients without ARF maintained in each group. Group 5 underwent early stopping in DL, while the remaining groups (Groups 1–4) were used for four-fold cross-validation, as shown in Figure A4.

We considered key performance metrics during model evaluation, such as the area under the receiver operating characteristic (AUROC) and the F1 score. For the final evaluation, a comprehensive range of performance metrics is considered for the best performing model, which includes accuracy, precision, recall, F1 score, negative predictive value (NPV), Matthews correlation coefficient (MCC), AUROC and the area under the precision–recall curve (AUPRC). In addition to these quantitative metrics, a visual assessment is conducted to comprehensively understand the model’s performance. This visual inspection involves the examination of confusion matrices, AUROC curves and AUPRC curves. It provides valuable insights into the model’s performance, behavior and strengths.

2.6. Statistical Analysis and Model Interpretation

The basic statistics of the datasets from both hospitals were thoroughly examined. A t-test was conducted at a significance level of 0.05 to determine potential between-hospital differences. Each hospital’s cases were designated as “1” (indicating ARF) and “0” (indicating non-ARF) and examined separately. Subsequent t-tests were performed at a significance level of 0.05 on data subsets to determine the significance of the observed differences.

To better understand how the model works and which features are most important, we identified the features, which received high evaluations from the model. We also used Shapley additive explanation (SHAP) values to confirm the clinical significance of the model’s learning process. These SHAP values help us understand how each feature contributes to the model’s predictions, making it easier to interpret model-guided decisions. This step is important in determining the practical relevance of the model’s findings in a clinical setting.

3. Results

3.1. Study Participants’ Characteristics

After addressing the outliers, we provide in

Table A2. Number and proportion of missing values by feature.

Feature	Korea University Anam Hospital (n = 12,059)		Soonchunhyang University Cheonan Hospital (n = 803)
	N	%	N	%
Age	0	0.00	0	0.00
Sex 1	0	0.00	0	0.00
Systolic BP 2	53,138	19.61	22	2.74
Diastolic BP	52,928	19.53	29	3.61
Respiratory	14,632	5.40	27	3.36
Heart rate	13,648	5.04	38	4.73
Serum Cr 3	14,047	5.18	28	3.49
Hemoglobin	13,172	4.86	41	5.11
Total CO2	11,695	4.32	65	8.09
Arterial pH	13,024	4.81	43	5.35
pCO2	13,307	4.91	39	4.86
pO2	13,531	4.99	43	5.35
BE 4	13,421	4.95	43	5.35
Lactate	10,295	3.80	50	6.23
GCS 5	251,420	92.78	24	2.99

¹ Sex, male; ² BP, blood pressure; ³ Cr, creatinine; ⁴ BE, base excess; ⁵ GCS; Glasgow Coma Scale.

the missing values for each feature and their corresponding proportions. Our study approach comprised selecting features with clinically significant relevance while ensuring that the proportion of missing values did not exceed 5%. Soonchunhyang University Cheonan Hospital initially selected features with less than 5% missing data. However, additional missing values were introduced after handling the outliers, causing some features to exceed the 5% threshold. Table A2 includes the GCS—which was otherwise excluded from feature selection—for comparative purposes.

Table 1. Mean and standard deviation according to the feature. “*” means statistically significant under a significance level of 0.05.

Feature	Korea University Anam Hospital (n = 12,059)		Soonchunhyang University Cheonan Hospital (n = 803)		p-Value
	Mean	SD	Mean	SD
Age, year	68.48	16.13	61.54	15.70	<0.0001 *
Sex (%) 1	6762	56.07	500	62.27	0.0007 *
Systolic BP, mmHg 2	123.38	16.26	133.90	23.86	<0.0001 *
Diastolic BP, mmHg	73.28	10.80	78.18	12.97	<0.0001 *
Respiratory rate, bpm	19.36	2.72	19.36	1.83	<0.0001 *
Heart rate, bpm	36.92	0.45	36.42	0.56	<0.0001 *
Serum Cr, mg/dL 3	1.12	0.83	0.86	0.27	<0.0001 *
Hemoglobin, g/dL	10.79	1.97	14.04	1.65	<0.0001 *
Total CO2, mmol/L	23.45	3.97	22.26	3.39	<0.0001 *
Arterial pH	7.43	0.04	7.38	0.06	<0.0001 *
pCO2, mmHg	32.07	5.80	37.11	5.72	<0.0001 *
pO2, mmHg	93.02	27.65	85.66	18.11	<0.0001 *
BE, mmol/L 4	−1.51	3.28	−2.33	4.08	<0.0001 *
Lactate, mmol/L	1.98	1.18	2.88	1.90	<0.0001 *

¹ Sex, male; ² BP, blood pressure; ³ Cr, creatinine; ⁴ BE, base excess.

includes each feature’s mean and standard deviation. A statistical analysis was conducted using p-values obtained from the t-tests to verify between-hospital differences. The t-test results indicated significant differences for all features, suggesting the two datasets were OOD. For the sex feature, the table presents the number and percentage of males. A chi-squared test confirmed the differences detailed in the table. This table offers a comprehensive overview of the statistical differences between the datasets, emphasizing the OOD relationship and the specific comparison for the sex feature.

Table 2. Differences between patients with ARF and patients without ARF at Korea University Anam Hospital. “*” means statistically significant under a significance level of 0.05.

Feature	Patients without ARF (n = 645)		Patients with ARF (n = 158)		p-Value
	Mean/N	SD/%	Mean/N	SD/%
Age, year	68.66	16.42	67.59	14.53	<0.0001 *
Sex (%) 1	54.05	54.61	1357	62.80	<0.0001 *
Systolic BP, mmHg 2	123.51	15.89	122.67	18.14	<0.0001 *
Diastolic BP, mmHg	73.48	10.61	72.21	11.69	<0.0001 *
Respiratory rate, bpm	19.15	2.40	20.51	3.86	<0.0001 *
Heart rate, bpm	36.92	0.44	36.94	0.53	<0.0001 *
Serum Cr, mg/dL 3	1.07	0.77	1.41	1.03	<0.0001 *
Hemoglobin, g/dL	10.87	1.95	10.37	2.03	<0.0001 *
Total CO2, mmol/L	23.58	3.90	22.75	4.26	<0.0001 *
Arterial pH	7.43	0.04	7.43	0.05	<0.0001 *
pCO2, mmHg	32.03	5.69	32.31	6.40	<0.0001 *
pO2, mmHg	92.92	27.09	93.54	30.41	<0.0001 *
BE, mmol/L 4	−1.46	3.20	−1.75	3.70	<0.0001 *
Lactate, mmol/L	1.92	1.12	2.26	1.42	<0.0001 *

¹ Sex, male; ² BP, blood pressure; ³ Cr, creatinine; ⁴ BE, base excess.

and

Table 3. Differences between patients with ARF and patients without ARF at Soonchunhyang University Cheonan Hospital. “*” means statistically significant under a significance level of 0.05.

Feature	Patients without ARF (n = 645)		Patients with ARF (n = 158)		p-Value
	Mean/N	SD/%	Mean/N	SD/%
Age, year	59.91	15.81	68.18	13.35	<0.0001 *
Sex (%) 1	408	63.26	92	58.23	0.2815
Systolic BP, mmHg 2	133.63	23.25	135.00	26.32	<0.5583
Diastolic BP, mmHg	78.55	12.67	76.63	14.06	<0.1257
Respiratory rate, bpm	19.36	1.75	19.34	2.19	0.9141
Heart rate, bpm	36.45	0.52	36.28	0.67	<0.0047 *
Serum Cr, mg/dL 3	0.83	0.27	0.97	0.27	<0.0001 *
Hemoglobin, g/dL	14.09	1.63	13.83	1.70	0.0988
Total CO2, mmol/L	22.69	3.19	20.51	3.61	<0.0001 *
Arterial pH	7.39	0.06	7.36	0.08	<0.0001 *
pCO2, mmHg	37.20	5.71	36.68	5.76	0.3319
pO2, mmHg	85.37	16.97	87.01	22.65	0.4274
BE, mmol/L 4	−1.85	3.88	−4.38	4.28	<0.0001 *
Lactate, mmol/L	2.84	1.82	3.03	2.24	0.3526

¹ Sex, male; ² BP, blood pressure; ³ Cr, creatinine; ⁴ BE, base excess.

present key statistics, such as means, standard deviations and statistical significance, highlighting the differences between patients with ARF and patients without ARF in both hospital datasets. For the Korea University Anam Hospital dataset, data pre-processing intentionally introduced distinctions between patients with ARF and patients without ARF, resulting in significant disparities in all features. However, in the Soonchunhyang University Cheonan Hospital dataset, some features were not significantly different between the patients with ARF and patients without ARF.

Table A3. Characteristics of pesticide exposure at Soonchunhyang University Cheonan Hospital.

Feature	N	%
Pesticide category
Not otherwise specified	227	28.27
Glyphosate	213	26.53
Glufosinate	186	23.16
Organophosphate	90	11.21
Pyrethroid	78	9.71
Carbamate	9	1.12
Amount of ingestion
≤50 cc	168	20.92
>50 cc, ≤100 cc	160	19.93
>100 cc, ≤200 cc	157	19.55
>200 cc, ≤300 cc	131	16.31
>300 cc	97	12.08
Unknown	90	11.21

provides an insightful overview of the distribution of pesticide-related features in the Soonchunhyang University Cheonan Hospital dataset, offering a better understanding of the features’ characteristics. These tables comprehensively illustrate the differences between patients with ARF and patients without ARF and provide insights into the distribution of pesticide-related features in the dataset.

3.2. Model Performance

Each model’s major performance metrics, including their AUROC and F1 scores, are included in

Table 4. Model performance. In transfer learning, the term “numbers” refers to the number of trainable dense layers.

Model	AUROC		F1
	Mean	95% CI	Mean	95% CI
LR	0.8665	0.8384–0.8946	0.4936	0.3565–0.6308
RF	0.8767	0.8189–0.9346	0.4173	0.2567–0.5780
XGB	0.8620	0.8238–0.9003	0.5179	0.4039–0.6319
LGBM	0.8627	0.8101–0.9153	0.5511	0.4420–0.6602
MLP	0.8361	0.7282–0.9439	0.4411	0.1924–0.6898
TL
5	0.9023	0.8760–0.9286	0.5539	0.4413–0.6665
4	0.8884	0.8513–0.9255	0.5672	0.4127–0.7216
3	0.8679	0.8344–0.9014	0.5435	0.4477–0.6392
2	0.8654	0.8107–0.9201	0.5513	0.5152–0.5873
1	0.8562	0.8008–0.9115	0.5228	0.4362–0.6094

. The models have generally high AUROC values, but the wide range of confidence intervals raises concerns about the reliability of their performance. This variability in performance can be attributed to differences between the training and test sets, especially in limited datasets. The RF model has the highest AUC among the traditional models, while the LR model has the narrowest confidence interval. The MLP model has the lowest mean and the widest confidence interval.

Notably, the TL model, which has the same structures as the MLP model, significantly outperforms the existing models. The TL approach remarkably narrows the confidence interval, substantially enhancing overall model performance. Figure 4 visually illustrates the comparative performance of each model.

Table A4. Model performance with Glasgow Coma Scale.

Model	AUROC		F1
	Mean	95% CI	Mean	95% CI
LR	0.9076	0.8872–0.9279	0.6800	0.5946–0.7655
RF	0.9115	0.8687–0.9544	0.5783	0.5106–0.6460
XGB	0.9039	0.8695–0.9383	0.6316	0.5459–0.7174
LGBM	0.9056	0.8966–0.9146	0.6339	0.5401–0.7277
MLP	0.8842	0.8460–0.9225	0.6321	0.4985–0.7658

illustrates the performance when GCS is included as a feature. Incorporating GCS as a feature significantly enhances the performance across all models. This confirms the crucial importance of GCS as a feature in patients with acute pesticide poisoning.

Table 5. Model performance for Group 4.

Model	Accuracy	Precision	Recall	F1	NPV	MCC	AUROC	AUPRC
MLP	0.83	0.83	0.16	0.26	0.83	0.31	0.77	0.57
TL5	0.87	0.87	0.41	0.55	0.87	0.54	0.91	0.79

provides detailed performance metrics for the MLP model, which has the same structure as the high-performing TL5 model and uses Group 4 evaluation data. These metrics offer a more detailed view of the model’s performance and effectiveness in the specific evaluation context. The table includes a comprehensive set of performance metrics, such as accuracy, precision, recall, F1 score, NPV, MCC, AUROC and AUPRC. Figure A1 visually compares the models’ performance, highlighting the confusion matrix, AUROC curve and AUPRC curves. Notably, there is a significant improvement in precision and recall for the TL5 model, highlighting substantial enhancements in its overall performance.

3.3. Model Interpretation

Figure A2 displays cases where there is a probability difference of 0.1 or more between the MLP and TL5 models. The red area represents patients with ARF, while the blue area represents patients without ARF. Therefore, if the model’s performance is high, the probability should be higher in the red area and lower in the blue area. The observed trend suggests that, except for some cases, the probability increases for patients with ARF and decreases for patients without ARF. Ultimately, the model predicts with greater confidence that patients who experience ARF are more likely to do so, and patients who do not experience ARF are more likely to remain free from it. This indicates an increased discriminative ability of the model.

Figure 5 displays the model’s SHAP values, highlighting the significant factors contributing to the development of ARF. The analysis provides the following insights:

• High Cr, low TCO₂ and low DBP significantly contributed to the development of ARF.
• Older age, low BE, high pCO₂ and high SBP may contribute to the development of ARF.
• Glufosinate and organophosphates were more likely to contribute to the development of ARF than other pesticides.
• Ingesting less than 100 cc carried a lower likelihood of developing ARF, while those who ingested 100–200 cc showed higher likelihood.

4. Discussion

Our study is a pioneering effort by the authors to apply OOD and TL techniques, commonly used in the image domain, to EHR, aiming to improve the performance of models with small sample sizes. Specifically, the authors developed a model for predicting ARF in patients with acute pesticide poisoning with minimized bias [37,41]. In cases of acute pesticide poisoning, MLP models face limitations due to insufficient data, resulting in low performance and wide confidence intervals [45]. In contrast, the newly proposed approach outperforms the MLP model and exhibits narrower confidence intervals. Additionally, the highest performance was achieved when the number of trainable layers was maximized. Maximizing the utilization of information tailored to the intended purpose is more advantageous than simply using information from a pre-trained model. The study also confirms the potential benefits of initializing weights using OOD data, particularly in cases of limited data, instead of commonly used initialization [9,10,11,12]. Moreover, transfer learning showed its ability to enhance performance, even when data are scarce, as illustrated in Figure A1.

Simply examining retrospective data does not allow for a discussion on the mechanisms of ARF between patients with acute pesticide poisoning and those without. However, leveraging OOD data, the model may have learned more generalized and rough patterns regarding the deteriorating respiratory condition of patients. The weights configured in this manner are expected to be effective in facilitating TL effectively. To assess the importance of features, SHAP values were employed. Most of the results were consistent with the trends of feature importance identified in previous research. By combining the importance of individual features as indicated by SHAP values with factors such as pesticide category and ingestion amounts, future research can contribute to a better understanding of the mechanisms underlying ARF resulting from acute pesticide poisoning.

However, this study has some limitations. First, it is retrospective and based on data from a single institution. Future studies should address these limitations and expand the scope of data collection. Second, further study is needed to examine the best ways to use OOD data, investigate various TL application methods and develop strategies for handling differing features in different application contexts. For instance, there is a need for discussion on how to address challenges when important features, such as GCS in this study, are mostly missing, making them difficult to leverage in pre-training. Despite its limitations, this study contributes valuable new methodologies for managing limited data in studying rare diseases and comparable conditions, highlighting the significant promise of machine-learning techniques in advancing medical research.

5. Conclusions

This study pioneers the application of OOD and TL techniques in the EHR, particularly in scenarios characterized by limited data. We conducted this research with a focus on predicting acute respiratory failure in patients with acute pesticide poisoning. Our proposed approach surpasses conventional predictive models by leveraging OOD data in conjunction with pre-trained models, highlighting the substantial benefits of OOD data for weight initialization in settings where data are scarce. The outcomes of our experimentation suggest that our method holds promise as a viable alternative for effectively training models with limited data. When an appropriate OOD dataset is adeptly utilized, it introduces a compelling methodology for addressing data limitations in rare diseases and analogous scenarios. Future research should be expanded beyond these preliminary findings to refine transfer learning applications and formulate strategies for handling diverse data attributes across various medical scenarios. A key emphasis should be placed on addressing the challenge of managing crucial yet dissimilar features in prediction. In conclusion, our study makes a significant contribution by presenting innovative methodologies to navigate challenges posed by limited data in the study of rare diseases and similar conditions. We will conduct additional research to overcome the limitations discussed in this paper.