Cervical Cancer Prediction Based on Imbalanced Data Using Machine Learning Algorithms with a Variety of Sampling Methods

Abstract

Cervical cancer affects a large portion of the female population, making the prediction of this disease using Machine Learning (ML) of utmost importance. ML algorithms can be integrated into complex, intelligent, agent-based systems that can offer decision support to resident medical doctors or even experienced medical doctors. For instance, an experienced medical doctor may diagnose a case but need expert support that related to another medical specialty. Data imbalance is frequent in healthcare data and has a negative influence on predictions made using ML algorithms. Cancer data, in general, and cervical cancer data, in particular, are frequently imbalanced. For this study, we chose a messy, real-life cervical cancer dataset available in the Kaggle repository that includes large amounts of missing and noisy values. To identify the best imbalanced technique for this medical dataset, the performances of eleven important resampling methods are compared, combined with the following state-of-the-art ML models that are frequently applied in predictive healtchare research: K-Nearest Neighbors (KNN) (with k values of 2 and 3), binary Logistic Regression (bLR), and Random Forest (RF). The studied resampling methods include seven undersampling methods and four oversampling methods. For this dataset, the imbalance ratio was 12.73, with a 95% confidence interval ranging from 9.23% to 16.22%. The obtained results show that resampling methods help improve the classification ability of prediction models applied to cervical cancer data. The applied oversampling techniques for handling imbalanced data generally outperformed the undersampling methods. The average balanced accuracy for oversampling was 77.44%, compared to 62.28% for undersampling. When detecting the minority class, oversampling achieved an average score of 60.80%, while undersampling scored 41.36%. The logistic regression classifier had the greatest impact on balanced techniques, while random forest achieved promising performance, even before applying balancing techniques. Initially, KNN2 outperformed KNN3 across all metrics, including balanced accuracy, for which KNN2 achieved 53.57%, compared to 52.71% for KNN3. However, after applying oversampling techniques, KNN3 significantly improved its balanced accuracy to 73.78%, while that of KNN2 increased to 63.89%. Additionally, KNN3 outperformed KNN2 in minority class performance, scoring 55.72% compared to KNN2’s 33.93%.

1. Introduction

Cervical cancer is the fourth most commonly diagnosed cancer in women around the world [1]. According to [2], the most important risk factors and potential predictors of cervical cancer are smoking, infection with sexually transmitted diseases (HIV, syphilis, etc.), and the use of hormonal contraceptives. These risk factors underline the importance of using demographic and medical information as indicators for cervical cancer risk prediction. Analyzing these factors can help us to understand how lifestyle choices and medical conditions correlate with the development of cervical cancer.

Prediction, particularly classification, and, in general, decision making based on imbalanced datasets can lead to diverse difficulties in data mining [3,4]. The studies reported in [5,6,7] presented recent state-of-the-art methods for healthcare data-balancing issues, with promising results.

In this work, the main focus is on balancing issues in cervical cancer prediction based on messy, imbalanced data. One of the aims of this work was to study and compare a broad range of frequently applied data-balancing methods, combined with ML classification methods, and to evaluate the performance of these methods before and after balancing on the cervical cancer dataset. In the selection of the ML classification method, several key factors were taken into consideration. K-nearest neighbors makes local decisions, not using global patterns [8], and computationally efficient logistic regression is frequently used in medical research [9], while random forest can handle non-linear relationships well with ensemble learning [10]. These algorithms are widely used in healthcare research when dealing with imbalanced data. However, if scientists do not treat this problem appropriately, could lead to wrong research design.

This study investigated the following eight undersampling methods: Condensed Nearest Neighbor (CNN), Tomek Links (TLs), Edited Nearest Neighbor (ENN), Repeated Edited Nearest Neighbors (RENN), All K-Nearest Neighbors (All-KNN), NearMiss (NM), Neighborhood Cleaning Rule (NCR), and Instance Hardness Threshold (IHT). The following four oversampling methods were also studied: the Synthetic Minority Oversampling Technique (SMOTE); the Adaptive Synthetic Sampling Approach for Imbalanced Learning (ADASYN); methods based on the decision boundary, such as Support Vector Machine (SVM) and Borderline; and ML algorithms KNN, LR, and RF. By comparing these methods, we provide knowledge to improve classification accuracy while offering both experimental results and statistical characterization of the data.

This research subject is also motivated by the fact that in many studies include experimental evaluations of algorithms without characterization of the data to which they are applied. Such characterizations could provide a clear indication for other researchers regarding the applicability of algorithms to their specific data. Specifically, if the data have the respective characteristics, performance evaluation results similar to those reported in the research can be expected.

This research is presented under six main sections. Section 2 presents a state-of-the-art bibliographic study on balancing techniques with state-of-the-art classification algorithms in healthcare prediction. Section 3 outlines the steps of data preprocessing and the proposed classification methodology. Based on this, Section 4 presents the results achieved when applying the eleven studied sampling methods with the three studied ML methods, and Section 5 highlights the results in comparison with similar works. Finally, Section 6 summarizes the main ideas and proposes future work in the medical field.

2. Bibliographic Study

This section analyzes current state-of-the-art ML models in healthcare prediction (including cancer prediction) that are frequently mentioned among the best. Sampling methods, including oversampling and undersampling, can be applied in combination with these methods to improve accuracy and model performance. The main difference between over- and undersampling is that oversampling increases the size of the minority class with synthetic samples, while undersampling reduces the size of the majority class, which can lead to data loss. While prediction includes the act of predicting both numerical and categorical values, the goal of classification is specifically to predict a category, such as determining if a patient has a certain type of cancer or not.

2.1. Classification Algorithms

ML models are widely used in health care to predict diseases and analyze health records with the purpose of improving medical diagnoses. In this work, the following ML methods are studied: KNN, LR, and RF. These ML methods are widely used in medicine for disease prediction, disease classification, and risk assessment [11].

2.1.1. K-Nearest Neighbors Algorithm

KNN is independent of data distribution [8], since it makes predictions based on the local neighbors (where k is an integer, such as 2, 3, 4… and, in the case of KNN2 and KNN3, 2 and 3 are considered neighbors) of the data points without making assumptions about the sample size and the characteristics of the data. Algorithm A1 presents the process of KNN, including parameter initialization, computation of the distance, and majority vote classification. KNN might have problems with distance calculation when working with imbalanced data because it could capture more observations from the majority class [12].

Wang and Han [13] proposed a novel ensemble algorithm for an imbalanced Parkinson’s disease dataset, where the KNN is used as the base classifier to calculate the ensemble weights, which overcomes the challenges posed by differing severity levels given an imbalanced distribution of the data [14]. Fuzzy KNN combined with a Bonferroni mean classifier makes multiple comparisons between the input data, resulting in easier evaluation for high-dimensional datasets (microarray and COVID-19), with minimal numbers of features [15]. In Fuzzy KNN, the membership degree is calculated with fuzzy membership functions. The Bonferroni mean classifier is an aggregation operator.

2.1.2. Logistic Regression Algorithm

LR is used to determine the outcomes of a particular data point. In binary classification, LR predicts two possible outcomes, while in multiclass classification, the model can predict multiple classes. Binary (binomial) LR (bLR) is a specific form of LR [16], where the dependent variable has only two categories. The applicability of LR depends on the size of the dataset, meaning that when small, imbalanced datasets are used, LR can have biases towards the majority class, leading to incorrect prediction performance for the minority class.

Algorithm A2 presents the main method of LR. The advantages of the simple bLR algorithm is that there only simple hyperparameter tuning is required, such as tuning of the learning rate and the number of epochs. This algorithm might struggle with highly imbalanced data because it results in high accuracy and low recall or precision for the minority class. More advanced LR models include regularizations with penalty terms in order to keep the model’s weights small to prevent overfitting, but this leads to a decrease in accuracy [17].

To address this issue, Firth’s penalized LR introduces a penalty term in the traditional LR model to create parameter estimates and standard errors, resulting in improved model fitting in small, health survey-related datasets [9]. Ref. [18] presented a bLR classifier with 70% Principal Component Analysis (PCA) (dimensionality reduction of the dataset with 70% proportion of variance explained), which achieved the highest precision and sensitivity values when compared with Elastic Net, SVM with a linear kernel, SVM with a Radial Basis Function kernel, RF, and XGBoost. The researchers concluded that bLR performs well with quantitative imaging (CT and MRI scans to measure and analyze quantitative data) features as predictors. Besides predicting binary outcomes, LR can also be incorporated with Least Absolute Shrinkage and Selection Operator (Lasso) regularization. Lasso LR adds a rule that restricts the model from assigning importance to less relevant features [19].

2.1.3. Random Forest Algorithm

RF is an ensemble learning method used in classification that creates multiple decision trees during the process of training. Algorithm A3 represents the creation of bootstrap (random replacement subsets) samples for each decision tree for the forest. While creating a decision tree for each bootstrap, the algorithm selects a random subset of features and votes on the best split while it grows to the maximal depth. This random selection of features works better with imbalanced data because it reduces the overfitting of the majority class [20].

Yang and Fridgeirsson [10] used Lasso LR, RF, and XGBoost classifiers with random under- and oversampling methods and varied the target imbalance ratio. The study used four large health databases—three U.S. claims databases and one German EHR database, all mapped to the OMOP Common Data Model to investigate outcomes in patients with treated depression, where the initial sample was 100,000. After applying 58 prediction tasks on electronic health data, the results showed that only the models for random oversampling with RF showed more variation in the Receiver Operating Characteristic—Area Under the Curve (ROC-AUC) difference (which compares the performance of two classification models by quantifying how much better a model is able to separate positive and negative classes compared to another model).

Other works collected data through surveys, as demonstrated by Xin and Rashid [21], whose main goal was to predict depression among women, who may experience anxiety due to their life roles and physiological differences in Malaysia’s environment whereby women may face issues with money, family, health, and work. In this research, the authors used the SMOTE oversampling method with RF and concluded that the imbalance ratio affected the sensitivity of the RF model, which predicted disease accurately. RF can be applied in identifying longitudinal predictors of health and was able to discriminate poor from good self-perceived health outcomes in a 30-year cohort study [22].

According to the bibliographic study, in the medical field, LR is mostly used in cases of smaller datasets, while KNN and ensemble RF algorithms are applied in complex datasets with multiple attributes. The difference between them lies in their methodological approaches. The simple form of bLR’s parameters can be tuned easily, which works well with a lower number of features, while KNN and RF can capture interactions among features when patient data in medical research convey different types of information.

2.2. Sampling Methods

2.2.1. Undersampling Methods

To handle class imbalance, the following nearest neighbor-based methods were considered appropriate and were chosen for this study: CNN, TL, ENN, RENN, All-KNN, NM, NCR, and the iterative IHT method. For an accurate analysis, undersampling reduces the number of abundant observations to create two equally sized classes [13].

The CNN undersampling technique reduces the size of the majority class by keeping the most informative instances to maintain the decision boundary. The work reported in [23] used a one-pass variation of CNN called Instance-Based Learning 2 (IB2) and proposed a novel variation algorithm based on it. The researchers tested it on nine datasets, including variables with different types of data, some of which were related to the medical field (such as yeast, congenital heart disease, and water quality), achieving decreased computational cost for the multilabel classification process.

TL identifies pairs of observations from different classes that are the nearest neighbors to each other, but when combined with the SMOTE method, it can balance the training dataset and eliminate components that are on the wrong side of the decision. On a lung cancer dataset, this hybrid TL approach with balanced RF resulted in the highest mean AUC compared to other hybrid methods, including the baseline and SMOTE-Edited Nearest Neighbors (SMOTE-ENN) [24].

All-KNN removes observations from the majority class that have at least one nearest neighbor in the minority class to increase the separability of the classes. In a case study of the Parkinson’s Disease Tremor Severity Classification (a clinical dataset collected from a wearable sensor in laboratory and home environments), All-KNN with Artificial NN based on multi-layer perceptron (ANN-MLP) metrics such as accuracy, precision, and sensitivity improved, but the Index of Balanced Accuracy (IBA) and geometric mean stayed low [25].

RENN removes misclassified observations from the majority class until a desired class balance is achieved. The study reported in [11] used four imbalanced health datasets concerning diabetes, anemia, lung cancer, and obesity, featuring high-dimensional data with non-linear relationships. The researchers underlined that RENN with LR is more effective in handling the lack of symmetry in dataset distribution.

ENN is a cleaning technique but eliminates misclassified observations in one step without any iterative approach. A possible application of the ENN undersampling technique is in Medicare fraud detection. In [26], ENN was combined with SMOTE methods on the Medicare Part B dataset. The results showed that this hybrid approach effectively balanced synthetic observations while eliminating noisy data, and Decision Tree (DT) outperformed the following ML classifiers: Extreme Gradient Boosting (XGBoost), Adaptive Boosting (Adaboost), Light Gradient Boosting Machine (LGBM), LR, and RF.

The NCR technique removes misclassified majority observations and improves decision boundaries. In [27], a novel clustering approach was presented that improved the quality of classification. The authors used the K-Means algorithms to cluster the data; then, clusters of only the majority class at a specified distance from the center were removed. For the Yeast5 protein–protein interaction network dataset, the NCR method provided the best results in terms of the Matthews Correlation Coefficient (MCC) and Cohen’s Kappa statistic (Kappa) metrics.

The NM technique selectively removes majority-class instances based on the distances to minority-class instances (NM-1 removes the closest element, and NM-2 removes the farthest element from the minority class). Combining NM with PCA, Tumuluru and Daniel [28] presented a novel approach to address the class imbalance problem in healthcare data. Using a real-world dataset, the proposed model outperformed baseline classifiers in metrics like precision, recall, F1 Score, and AUC, highlighting possible applications in disease diagnosis, patient risk stratification, and treatment prediction. The work reported in [29] studied the problem of establishing the optimal number of factors in an Exploratory Factor Analysis (EFA) in general and PCA in particular, highlighting the complexity of the issue.

The IHT technique removes instances from the majority class and focuses on instances that are easier to classify while removing harder-to-identify observations. The work of Lopo and Hartomo [6] presented an evaluation of sampling techniques in healthcare insurance-related fraud detection, where the IHT method (with 90% class distribution) outperformed the XGBoost, SMOTE, and random oversampling methods overall, highlighting the ability to generalize well to new and unseen data in the minority and majority classes.

2.2.2. Oversampling Methods

For datasets with less information in particular classes to be balanced, artificial samples must be generated to increase the number of rare instances [30]. To create an equal class distribution by resizing the classes in this study, the following types of oversampling methods were used: methods based on the minority class, namely SMOTE and ADASYN, and methods based on the decision boundary, namely SVM and Borderline.

The SMOTE technique generates new instances of the minority class, resulting in improved performance in minority-class prediction. There are many varieties of SMOTE preprocessing techniques, namely traditional SMOTE, Borderline-SMOTE1, Borderline-SMOTE2, SMOTE-NC, and SVM-SMOTE, which were used in a hospital mortality prediction study of 126 patients with traumatic injuries. The data were extracted from the patients’ medical records. The researchers focused on the trauma patients’ status (alive/dead) as an outcome and six risk factors (age, sex, type of trauma, location of injuries, Glasgow coma scale, and white blood cells). The results showed that among all SMOTE-based ML methods, RF and ANN with SMOTE and XGBoost with SMOTE-NC achieved the highest values for all evaluation metrics [30]. Sinha [31] proposed a novel Data Augmented SMOTE Multi-Class Classifier (DASMcC) to predict Cardiovascular Diseases (CVDs). To ensure that the classifier’s performance was not biased, SMOTE was combined with a 10-fold cross-validation technique, while XGBoost performed the best overall.

SVM is an ML algorithm for binary classification, according to which the input vector performs a non-linear mapping process to create linearly separable data in a higher-dimensional feature space. Removing correlated features and simplifying the model-training process with PCA whitening results in data with unit variance along each dimension in the pretraining process of an imbalanced dataset [13]. Bektas [32] presented a novel ensemble model based on SVM’s hyperplane to calculate the optional boundaries between the signed distances integrated with bLR, resulting in higher accuracy compared to SVM kernel selection using datasets related to the ionosphere, High Time-Resolution Universe Survey (HTRU2) pulsar candidate, diabetes, and liver disorders.

Borderline oversampling is an advanced variation of SMOTE that generates samples near the decision boundaries of classes. Jo and Kim [5] proposed a novel method called minority oversampling near the borderline with a generative adversarial network (OBGAN) that uses Borderline with a generative adversarial network to focus on avoiding the mode-collapse problem on small datasets. Among 21 relatively small imbalanced datasets, examples of majority/minority outcomes include the following: liver cancer (have/not), with 10 features; breast cancer (benign/malignant), with 9 features; Pima Indian diabetes (diabetes/not), with 10 features; blood transfusion (not/donate), with 4 features (from the UCI ML Repository, Kaggle, and DataHub); and 6 benchmark methods (SMOTE, Borderline-SMOTE, ADASYN, k-means SMOTE (kmSMOTE), conditional GAN, and Generative Adversarial Minority Oversampling (GAMO). The experiments show OBGAN is competitive with the SMOTE-based methods and achieves stable performance for multiclass problems with various majority–minority ratios. The ADASYN technique adapts the generation process based on the density of the distribution of minority-class instances. Ahmed [33] proposed a DAD-net system to detect Alzheimer’s disease from images for which ADASYN was able to generate new samples to balance the number of instances in every category.

This section underlines the fact that databases of common health problems like cancer, Parkinson’s disease, and depression are widely used in the medical field for predictive purposes. However, some datasets are affected by common data issues such as missing values, outliers, and imbalanced data. To address this challenge, it is necessary to apply data manipulation techniques such as undersampling and oversampling. Some of available and innovative approaches incorporate algorithm modifications that have achieved promising results in healthcare prediction. For instance, SMOTE-ENN can eliminate noisy data in medical fraud detection, NCR with clustering can eliminate class objects in protein–protein datasets, and SMOTE data augmentation performs well for multiclass classification of CVDs.

3. Materials and Methods

3.1. Conceptual Description

Figure 1 presents the research methodology and visually summarizes an ML workflow that includes data preprocessing, sampling, feature engineering, ML training, and model analysis. Our study outlines the performance of the KNN, LR, and RF methods combined with various undersampling and oversampling methods on a cervical cancer dataset with an Imbalance Ratio (IR) of 12.73 (the ratio of the number of instances in the majority class to the number of instances in the minority class). With only a 7% proportion with cancer, the 95% Confidence Interval (CI) for IR ranges between 9.23% and 16.22%. The dataset has over 2500 missing values.

3.2. Dataset Description

This study utilizes the cervical cancer risk factor dataset available on the UCI ML platform [34]. The data were collected from “Hospital Universitario de Caracas” in Caracas, Venezuela, comprising 858 records and 36 attributes (variables). The data reveal patients’ demographic information, habits, and medical history. The records contain the following four target variables: Hinselmann (screening test for abnormalities in the cervix) score, Schiller (identifying areas of concern with iodine test) score, and cytology (examination under a microscope for cell abnormality) and biopsy (for detection of the presence of cancer on removed tissue or cell samples) metrics. In this study, we selected biopsy as the only target variable. Because there were no data for cervical condylomatosis or AIDS, these two attributes were removed, leaving 30 variables as input.

3.3. Data Preprocessing

Clinical data are generally affected by factors such as missing values, inconsistent data, and exceptions, highlighting the need for preprocessing. Preprocessing is a very important step, as it includes both data preparation and data transformation. Tumuluru [28] recommended processing in different situation to change the distribution of the data and direct the algorithm.

Table 1. Summary statistics of the dataset attributes (integers).

Attribute	Min	Max	Mean	Std Dev	Median	95% CI Mean
Age	13	84	26.82	8.49	25	(26.25, 27.39)
Number of sexual partners	1	28	2.53	1.67	2	(2.41, 2.64)
First sexual intercourse	10	32	16.99	2.80	17	(16.81, 17.18)
Number of pregnancies	0	11	2.28	1.45	2	(2.18, 2.38)
Smokes (years)	0	37	1.22	4.09	0	(0.94, 1.50)
Smokes (packs/year)	0	37	0.45	2.23	0	(0.30, 0.60)
Hormonal contraceptives (years)	0	30	2.26	3.76	0.5	(1.99, 2.53)
IUD (years)	0	19	0.51	1.94	0	(0.37, 0.65)
STDs (number)	0	4	0.18	0.56	0	(0.14, 0.22)
STDs: number of diagnoses	0	3	0.09	0.30	0	(0.07, 0.11)

and

Table A2. Count of binary data for attributes.

Attribute	Negative	Positive	Missing	Attribute	Negative	Positive	Missing
Smokes	722	123	13	STDs: HIV	735	18	105
Hormonal contraceptives	269	481	108	STDs: Hepatitis B	752	1	105
IUD	658	83	117	STDs: HPV	751	2	105
STDs	674	79	105	Dx:Cancer	840	18	0
STDs: condylomatosis	709	44	105	Dx:CIN	849	9	0
STDs: cervical condylomatosis	753	0	105	Dx:HPV	840	18	0
STDs: vaginal condylomatosis	749	4	105	Dx	834	24	0
STDs: vulvo-perineal condylomatosis	710	43	105	Hinselmann	823	35	0
STDs: syphilis	735	18	105	Schiller	784	74	0
STDs: pelvic inflammatory disease	752	1	105	Citology	814	44	0
STDs: genital herpes	752	1	105	Biopsy	803	55	0
STDs: molluscum contagiosum	752	1	105	STDs: AIDS	753	0	105

present the descriptive statistics of the dataset attributes, in integer and Boolean form, respectively. To ensure that every attribute contributed equally to the model training of the selected algorithms, the values were normalized in the [0, 1] interval. The calculation of the mean is presented in Equation (A1).

Missing information in the dataset can affect the statistical analysis. In the case of the used dataset, several patients refused to provide certain information; therefore, there are many missing values, especially in the case of very intimate questions like those concerning the use of hormonal contraceptives and intrauterine devices, as well as Sexually Transmitted Diseases (STDs).

After analyzing the data, it was evident that several missing values, especially intimate information, belong to the same patients, resulting in the elimination of 105 records, leaving 753 records from the original dataset. The remaining missing values were replaced with the median of the integer data and the most frequently occurring value for the Boolean data.

Figure 2 shows a graphical representation of the frequency distributions of several important attributes that were found to be the best predictors in the dataset. The figure also presents the result of the Lilliefors [35,36] numerical statistical test and estimations of the distribution in the case of the most important variables. Asymmetry to the right is visible, with a sudden decrease in the number of pregnancies per patient, i.e., women with zero, one, or two births. Similarly, for the number of years in which the patient used hormonal contraceptives, there is also an asymmetry to the right, caused by the generally young age of the women in the study. On the other hand, although the patients are relatively young, their age distribution is near symmetrical. The predominance of young age in the dataset may be explained by efforts to a culture of health awareness, where younger women are encouraged to participate in medical examinations and early cancer detection programs.

The Lilliefors test shows, experimentally, that the data follow a non-normal distribution, and because of this, the median was preferred over the mean [15]. The median is less sensitive to extreme values (outliers) than the mean, but there were no outliers in our dataset. In the case of binary variables, this approach is not suitable. A visual validation of the non-normal distribution is presented in Figure 3 using a Q-Q plot [37].

Another important aspect that can be identified immediately after data preprocessing is the number of instances in each class. The data distribution comprised 92% healthy patients and 7% of patients with a positive biopsy.

After data preprocessing, it is the imbalance ratio (IR) was calculated to measure the imbalance degree of the dataset, as expressed by $IR={\displaystyle \frac{n_{\mathrm{negative}}}{n_{\mathrm{positive}}}}$, where $n_{\mathrm{negative}}$ is the number of negative samples and $n_{\mathrm{positive}}$ is the number of positive samples, yielding a value of 12.73.

After this, the variance of the imbalance ratio can be approximated using the formula $\mathrm{Var}\left(IR\right)=I{R}^2\times \left({\displaystyle \frac{1}{n_{\mathrm{positive}}}}+{\displaystyle \frac{1}{n_{\mathrm{negative}}}}\right)$.

The Standard Error (SE) of the IR is expressed by $SE\left(IR\right)=IR\times \sqrt{{\displaystyle \frac{1}{n_{\mathrm{positive}}}}+{\displaystyle \frac{1}{n_{\mathrm{negative}}}}}$.

The 95% confidence interval (CI) for the IR is calculated as $CI=IR\pm Z\times SE\left(IR\right)$, where Z is the Z score corresponding to the desired confidence level (for 95%, $Z=1.96$) and $SE\left(IR\right)$ is the SE of the IR.

Finally, the lower and upper bounds of the CI are expressed by $\mathrm{Lower}\mathrm{Bound}=IR-Z\times SE\left(IR\right)$ and $\mathrm{Upper}\mathrm{Bound}=IR+Z\times SE\left(IR\right)$, yielding values of 9.23 and 16.22, respectively.

3.4. Model Training and Evaluation

These predictive models were implemented in the Python programming language using ML libraries. The Pandas library (version 1.5.3) was used for data manipulation and analysis, and Numpy (version 1.23.5) was employed for numerical calculations. For data visualization, Matplotlib (version 3.7.1) and Seaborn (version 0.12.2) were utilized. Interactive visualizations were created with Plotly (version 5.15.0), utilizing the Plotly.express module and the Plotly.io interface.

After processing the dataset, it was used to train NNs with four ML models (KNN with k values of 2 (KNN2) and 3 (KNN3), LR, and RF) utilizing Scikit-learn and Keras. The typical values for k in KNN are 2 or 3, but there is no unique, well-established strategy for determining the best k value [38]. We conducted experiments with a higher number of integers for K, but only values of 2 and 3 achieved notable results.

For the undersampling methods, the most relevant parameters were the sampling_ strategy (how many instances to sample for all methods), random_state (seed for random number generation like in a CNN), n_neighbours (number of neighbors to consider), and max_iter (limits the number of iterations), threshold_cleaning (decides which instance to keep). Oversampling methods include important parameters such as k_neighbors (number of nearest neighbors to use for synthetic sample generation for SVM and Borderline), m_neighbours (defining the number of neighbors to consider for SVM), and n_jobs (number of processors to use for parallel processing like in ADASYN). The values of the algorithms were selected based on the size of the data and class imbalance. Using guidelines established in [18] and empirical testing also improved model performance.

To measure model performance, we used a common ML approach where the data were split into a 75% training set and a 25% testing set. When evaluating models for the diagnosis of contagious diseases, such as sexually transmitted infections, which are a primary risk factor for cervical cancer, the selected metrics must align with the goals of diagnosis. In this case, identifying all the infected individuals (even at the risk of some false positives) is more critical than missing infected persons. In order to measure this imbalanced data learning, the following five evaluation metrics were selected as performance evaluation criteria: Accuracy (%) (A4), Precision (%) (A5), Recall (%) (A6), F1 Score (%) (A7), Balanced Accuracy (%) (Ba Accuracy) (A8) ROC_AUC score (%) [39], Class distribution (Majority (MajClass) or Minority Class (MinClass)), and Number of instances [40].

The confusion matrix presented in

Table A1. Confusion matrix for binary classification.

Actual	Prediction
	Positive	Negative
Positive	TP	FN
Negative	FP	TN

represents the obtained classification results. Correctly classified instances are True Positive (TP) and True Negative (TN), and incorrectly classified instances are False Positive (FP) and False Negative (FN) [41].

The ROC curve and the corresponding AUC illustrate how well the model is capable of distinguishing between classes. The higher the AUC value, the more likely the model is to distinguish class 1 as 1 and class 0 as 0 (class 1 denotes cervical cancer; class 0 denotes cervical cancer not identified). To calculate it, a classifier and the probabilities obtained from predictions are needed. With multiple thresholds and based on each threshold, it can categorize the predicted probabilities as false or true and calculate the true-positive rate and false-positive rate.

An AUC closer to 1 indicates a better measure of separability, as well as which threshold is most suitable for a classifier, while the ROC curve is a probability curve. The X axis indicates the false-positive rate, and the Y axis represents the true-positive rate. Each point on the curve represents a probability threshold used to determine whether an example belongs to a certain category.

Metrics such as accuracy, F1 Score, and recall, both weighted and macro forms were also calculated to address the impact of imbalace. Macro metrics treat imbalanced classes as equals, highlighting the performance of the minority class. Weighted metrics represent the proportion of every class in the dataset, providing a measure of performance based on class frequency [42].

MacroPrecision (1), MacroRecall (2), and MacroF1score (3) represent the average precision, recall, and F1 Scores across all classes, respectively, as follows:

$$\mathrm{MacroPrecision}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\displaystyle \frac{T{P}_i}{T{P}_i+F{P}_i}}$$

(1)

$$\mathrm{MacroRecall}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\displaystyle \frac{T{P}_i}{T{P}_i+F{N}_i}}$$

(2)

$$\mathrm{MacroF}1\mathrm{Score}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\displaystyle \frac{2\times \frac{T{P}_i}{T{P}_i+F{P}_i}\times \frac{T{P}_i}{T{P}_i+F{N}_i}}{\frac{T{P}_i}{T{P}_i+F{P}_i}+\frac{T{P}_i}{T{P}_i+F{N}_i}}}$$

(3)

where N is the number of classes.

WeightedPrecision (4), MacroRecall (5), and WeightedF1score (6) represent the precision, recall and F1 Scores of each class weighted by the number of instances of that class, respectively, as follows:

$$\mathrm{WeightedPrecision}={\displaystyle \frac{1}{{\sum }_{i=1}^N\left|C_i\right|}}\sum _{i=1}^N{\displaystyle \frac{T{P}_i}{T{P}_i+F{P}_i}}\times \left|C_i\right|$$

(4)

$$\mathrm{WeightedRecall}={\displaystyle \frac{1}{{\sum }_{i=1}^N\left|C_i\right|}}\sum _{i=1}^N{\displaystyle \frac{T{P}_i}{T{P}_i+F{N}_i}}\times \left|C_i\right|$$

(5)

$$\mathrm{WeightedF}1\mathrm{Score}={\displaystyle \frac{1}{{\sum }_{i=1}^N\left|C_i\right|}}\sum _{i=1}^N{\displaystyle \frac{2\times \frac{T{P}_i}{T{P}_i+F{P}_i}\times \frac{T{P}_i}{T{P}_i+F{N}_i}}{\frac{T{P}_i}{T{P}_i+F{P}_i}+\frac{T{P}_i}{T{P}_i+F{N}_i}}}\times \left|C_i\right|$$

(6)

where $|C_i|$ is the number of instances in class i.

4. Results

The first experiment involved applying ML models using the original imbalanced dataset. For the other experiments, the balanced data were used with under- and oversampling techniques. The calculated weighted average (which gives more importance to values appearing more often in the data) used the numbers of data points in the majority and minority class as weights. On the other hand, the results of the macro average show the arithmetic mean of the metric values for each class, treating each class equally, regardless of the class size. Both averages are presented in

Table 2. Performance results of ML models (before applying sampling techniques).

Metric/Sampling	KNN2	KNN3	bLR	RF
Accuracy	93.12	91.53	92.59	95.76
Ba Accuracy	53.57	52.71	50.00	71.42
Precision (Mac;Wght)	96.54;93.59	58.98;87.93	46.29;85.73	97.81;95.95
Recall (Mac;Wght)	53.57;93.12	52.71;91.53	50.00;92.59	71.42;95.76
MajClass	100	98.28	100	100
MinClass	7.14	7.14	0	42.85
F1-Score (Mac;Wght)	54.87;90.26	53.33;89.30	48.07;89.03	78.88;94.96
ROC_AUC	0.61	0.63	0.71	0.99

,

Table 3. Performance results of KNN2 (after applying oversampling techniques).

Metric/Sampling	SMOTE	SVM SMOTE	Borderline SMOTE	ADASYN
Accuracy	88.35	91.01	90.47	87.83
Ba Accuracy	60.85	65.57	68.57	60.57
Precision (Mac;Wght)	59.60;89.09	66.67;90.70	66.43;91.08	58.85;87.83
Recall (Mac;Wght)	60.85;88.35	65.57;91.00	68.57;90.47	60.57;87.83
MajClass	93.14	95.43	94.29	92.57
MinClass	28.57	35.71	42.86	28.57
F1 Score (Mac;Wght)	60.17;88.71	66.09;90.85	67.41;90.76	59.58;88.36
ROC_AUC	0.72	0.74	0.74	0.72

Note: Bolded values indicate significant performance for a metric.

,

Table 4. Performance results of KNN2 (after applying undersampling techniques).

Metric/Sampling	CNN	TL	All-KNN	RENN	ENN	NCR	NM1	IHT
Accuracy	93.65	92.59	87.30	87.30	87.30	90.47	75.66	45.50
Ba Accuracy	60.42	53.28	50.42	50.42	50.42	55.42	77.00	64.00
Precision (Mac;Wght)	84.52;92.64	71.52;89.85	50.49;86.40	50.49;86.40	50.49;86.40	59.18;88.30	59.07;92.04	53.99;90.94
Recall (Mac;Wght)	60.42;93.65	53.28;92.59	50.42;87.30	50.42;87.30	50.42;87.30	55.42;90.47	77.00;75.66	64.00;45.50
MajClass	99.43	99.43	93.71	93.71	93.71	96.57	75.43	42.29
MinClass	21.43	7.14	7.14	7.14	7.14	14.29	78.57	85.71
F1 Score (Mac;Wght)	65.00;91.97	54.31;89.93	50.43;86.84	50.43;86.84	50.43;86.84	56.56;89.25	58.75;81.24	38.93;55.99
ROC_AUC	0.70	0.61	0.61	0.62	0.62	0.63	0.80	0.65

Note: Bolded values indicate significant performance for a metric.

,

Table 5. Performance Results of KNN3 (after applying undersampling techniques).

Metric/Sampling	CNN	TL	All-KNN	RENN	ENN	NCR	NM1	IHT
Accuracy	90.47	91.53	86.24	86.24	87.30	88.35	69.84	43.91
Ba Accuracy	68.57	52.71	49.85	49.85	50.42	54.28	77.14	63.14
Precision (Mac;Wght)	66.43;91.08	58.98;87.93	49.85;86.24	49.85;86.24	50.49;86.40	54.94;87.54	58.13;92.40	53.80;90.82
Recall (Mac;Wght)	68.57;90.47	52.71;91.53	49.85;86.24	49.85;86.24	50.42;87.30	54.28;88.35	77.14;69.84	63.14;43.91
MajClass	94.29	98.29	92.57	92.57	93.71	94.29	68.57	40.57
MinClass	42.86	7.14	7.14	7.14	7.14	14.29	85.71	85.71
F1 Score (Mac;Wght)	67.41;90.76	53.33;89.30	49.85;86.24	49.85;86.24	50.43;86.84	54.56;87.94	55.21;77.01	37.85;54.38
ROC_AUC	0.67	0.63	0.60	0.60	0.61	0.62	0.79	0.64

Note: Bolded values indicate significant performance for a metric.

,

Table 6. Performance results of bLR (after applying oversampling techniques).

Metric/Sampling	SMOTE	SVM SMOTE	Borderline SMOTE	ADASYN
Accuracy	86.77	88.35	86.77	85.18
Ba Accuracy	89.57	51.00	86.28	85.42
Precision (Mac;Wght)	67.72;94.66	50.89;86.50	66.49;93.92	65.12;93.70
Recall (Mac;Wght)	89.57;86.77	51.00;88.35	86.28;86.77	85.42;85.18
MajClass	86.86	94.29	86.86	85.14
MinClass	92.86	7.14	85.71	85.71
F1 Score (Mac;Wght)	72.34;89.66	50.74;87.15	70.69;89.19	68.78;88.05
ROC_AUC	0.92	0.87	0.89	0.91

,

Table 7. Performance results of RF (after applying oversampling techniques).

Metric/Sampling	SMOTE	SVM SMOTE	Borderline SMOTE	ADASYN
Accuracy	98.41	98.41	98.41	98.41
Ba Accuracy	92.57	92.57	92.57	98.41
Precision (Mac;Wght)	95.58;98.37	95.58;98.37	95.58;98.37	95.58;98.37
Recall (Mac;Wght)	92.57;98.41	92.57;98.41	92.57;98.41	92.57;98.41
MajClass	99.43	99.43	99.43	99.43
MinClass	85.71	85.71	85.71	85.71
F1 Score (Mac;Wght)	94.01;98.38	94.01;98.38	94.01;98.38	94.01;98.38
ROC_AUC	0.99	0.99	0.99	0.99

Note: Bolded values indicate significant performance for a metric.

and

Table 8. Performance results of RF (after applying undersampling techniques).

Metric/Sampling	CNN	TL	All-KNN	RENN	ENN	NCR	NM1	IHT
Accuracy	98.94	95.76	95.23	97.35	96.29	96.29	95.76	76.19
Ba Accuracy	96.14	71.42	67.85	82.14	75.00	75.00	97.71	83.85
Precision (Mac;Wght)	96.14;98.94	97.81;95.95	96.14;98.94	98.07;96.43	98.07;96.43	98.07;96.43	81.81;97.30	61.02;93.58
Recall (Mac;Wght)	96.14;98.94	71.42;95.76	67.85;95.23	82.14;97.35	75.00;96.29	75.00;96.29	97.71;96.43	83.85;76.19
MajClass	99.43	100	100	100	100	100	95.43	74.86
MinClass	92.86	42.86	64.29	64.29	50	50	100	92.86
F1 Score (Mac;Wght)	96.14;98.94	78.88;94.96	75.06;94.17	82.35;95.71	82.35;95.71	82.35;95.71	87.71;96.18	60.98;81.73
ROC_AUC	0.99	0.99	1.0	1.0	0.99	0.99	0.99	0.95

Note: Bolded values indicate significant performance for a metric.

.

Table 2 reports the metric values obtained for all algorithms with the data in their state after initial preprocessing. We are interested in the highest values (numerical values represent percentages (%)). The obtained results show that the best performance was achieved by RF. The other classifiers, in general, are limited to predicting the minority class. The performance results of the ML models were obtained before applying sampling techniques.

Table 3 and Table 4 show the results obtained by the KNN algorithm with two neighbors after the application of under- and oversampling techniques. The highest sampling values for KNN were achieved by the Borderline SMOTE technique, but all methods generated much higher values than the models applied to the original dataset. ADASYN showed the least effectiveness overall, particularly in terms of accuracy and ROC_AUC score. These insights also suggest that SVM SMOTE is an effective oversampling technique for enhancing the performance of the KNN2 model. Among the undersampling techniques, NM1 and IHT predicted 78.57% and 85.71%, respectively. NCR stands out with a statistically significant ROC_AUC score of 0.80, which is notably higher than that of the rest. When excluding NCR, the remaining values do not show notable differences. The rest of the metric values were not higher than the original values.

Table A3. Performance results of KNN3 (after oversampling techniques).

Metric/Sampling	SMOTE	SVM SMOTE	Borderline SMOTE	ADASYN
Accuracy	86.24	89.94	88.35	87.30
Ba Accuracy	72.85	74.85	74.00	73.42
Precision (Mac;Wght)	62.42;91.25	67.26;92.10	64.48;91.69	63.54;91.46
Recall (Mac;Wght)	72.85;86.24	74.85;89.94	74.00;88.35	73.42;87.30
MajClass	88.57	92.57	90.86	89.71
MinClass	57.14	57.14	57.14	51.47
F1 Score (Mac;Wght)	65.17;88.24	70.08;90.84	67.81;89.72	66.44;88.98
ROC_AUC	0.71	0.73	0.73	0.72

and Table 5 present the results achieved by KNN with three neighbors. For KNN3, every oversampling method worked well. In the case of undersampling, NM1 and IHT produced the best results, notably in the case of minority-class prediction. NM1 also excelled in class separation, with an ROC_AUC score of 0.79.

Table 6 and

Table A4. Performance Results of bLR (after undersampling techniques).

Metric/Sampling	CNN	TL	All-KNN	RENN	ENN	NCR	NM1	IHT
Accuracy	92.59	92.59	92.59	92.59	92.59	92.59	58.20	41.79
Ba Accuracy	50.00	50.00	50.00	50.00	50.00	50.00	57.71	58.71
Precision (Mac;Wght)	46.29;85.73	46.29;85.73	46.29;85.73	46.29;85.73	46.29;85.73	46.29;85.73	52.16;88.18	52.54;89.37
Recall (Mac;Wght)	50.00;92.59	50.00;92.59	50.00;92.59	50.00;92.59	50.00;92.59	50.00;92.59	57.71;58.20	58.71;41.79
MajClass	100	100	100	100	100	100	58.29	38.86
MinClass	0	0	0	0	0	0	57.14	78.57
F1 Score (Mac;Wght)	48.07;89.03	48.07;89.03	48.07;89.03	48.07;89.03	48.07;89.03	48.07;89.03	44.46;67.99	35.97;67.99
ROC_AUC	0.61	0.73	0.71	0.71	0.72	0.72	0.66	0.59

Note: Bolded values indicate significant performance for a metric.

present the results achieved by bLR. The good results for the AUC curve of all SMOTE methods must be noted. The undersampling techniques were not spectacular. CNN, TL, All-KNN, RENN, ENN, and NCR maintained high accuracy but at the cost of ignoring the minority class, resulting in poor balanced accuracy and macro metrics. NM1 and IHT handled the minority class better, but this came at the expense of overall accuracy.

Table 7 and Table 8 present the results obtained by RF. Although this algorithm proved effective and suitable for the initial data, after applying both types of sampling, it obtained further superior results. CNN and RENN offer a better balance, achieving high accuracy while maintaining reasonable performance for minority classes. It is also notable that the ADASYN and NM1 techniques achieved values of 85.71%, and 100%, respectively, in the detection of minority classes and ROC_AUC scores of more than 99% with all sampling methods.

A smaller area under the ROC curve is visible on the right side of Figure 4, and the left side shows the increase in the area under the ROC curve after applying the oversampling technique.

5. Discussions

5.1. Benefits of Prediction in Health Care

Using methods based on ML to solve diverse types of prediction problems can help health care to be more efficient and effective. In sophisticated data analytics, ML-based autonomous and intelligent systems are improved by intelligent communication technologies [43]. ML algorithms for prediction can help to develop preventive strategies in decision making, ensuring that the development of the algorithm is interpretable and transparent and takes into consideration ethical concerns [44].

Accuracy and real-time predictive analysis of disease prediction may lead to patients’ lives being saved, but incorrect predictions could endanger patients’ lives [45]. By integrating efficacious biomedical and health data, researchers can make accurate diagnoses based on ML methods to improve patient-centric treatments [46].

5.2. Advances in the Scientific Literature

Alsmariy [47] presented similar ML algorithms for cervical cancer prediction. The same dataset (used within our article) in this research was analyzed with PCA, balanced with SMOTE, and distributed with 10-fold cross-validation to prevent the overfitting problem. This approach raised the accuracy, sensitivity, and ROC_AUC score of the proposed model, a voting method that combines state-of-the-art classifiers: decision tree, logistic regression and random forest. Other cervical cancer work [1] used the dataset processed in this research, proposing a hybrid strategy based on the combination of an oversampling method (SMOTE) and two undersampling methods (CNN and ENN) to balance the dataset. Genetic algorithms were applied for feature selection, and RF classifier provided the highest performance in terms of G mean (geometric mean of the recall), sensitivity, and specificity.

In the case of a larger volume of data (more than 100,000 samples) or a large number of missing data (more than 10% of the data), diverse and sophisticated methods are available, including analyses to the replacement of missing values. For example, Entropy-Based (EB) algorithms, imputation of missing values, Independent Component Analysis (ICA), Linear Discriminant Analysis (LDA) [48], or the removal of outliers with covariance-based Mahalanobis distance, treating them as special values can be performed. A combination of these techniques is described in [49], the authors of which concluded that the ensemble model performed well under the proposed preprocessing and EB feature engineering method on a heart disease-related dataset including 14 features.

5.3. Results of This Research

We chose to use the traditional forms of KNN, LR, and RF because they are classic algorithms in ML. The bibliographic study showed that these algorithms have many improved versions, but they can also be complex or have potential disadvantages. Focusing on these algorithms, a clear comparison served as a basis for understanding their applicability.

The data analyzed in the experimental evaluation of this study were highly imbalanced, with 93% of the attributes belonging to the majority class and only 7% belonging to the minority class (positive biopsy cases). The CI with a 95% confidence level ranges between 9.23 and 16.22, while the the IB is 12.73. ML algorithms applied to the imbalanced data had difficulties detecting the minority class. One of the effects of imbalanced learning is that oversampling methods are more stable than other sampling methods (undersampling and hybrid sampling), while undersampling has is the least stable [24].

Evaluation of the imbalanced learning techniques applied to the dataset revealed that the use of oversampling techniques increased the performance of the models, proving to be inspired choices. KNN2 and KNN3 with undersampling initially predicted 7% of the minority class; after the application of sampling techniques, 86% of the majority class was predicted. LR improved from 0% to 93% with the SMOTE technique. Even for RF, which worked well even before applying the techniques, some metrics peaked. Decreasing the dimensionality of the dataset did not produce major changes, but some obtained metric values increased.

Based on the research that we performed, it can be concluded that the relationship between the minority and majority class is important in capturing each region in an imbalanced dataset while taking into consideration the sensitivity of ML in general and Neural Networks (NNs) in particular, which are more effective at learning the minority class with a focus near the borderline.

6. Conclusions

Based on the performed research, we conclude that datasets in a class with significantly fewer instances lead to overfit classifiers that favor the majority class. Based on this fact, the observations in the minority class are hardly recognizable or incorrectly identified. This issue is present in the medical field because, in many cases, the number of records corresponding to healthy patients (majority class) is much larger than the number of records for patients diagnosed with a specified disease (minority class). This leads to significant information related to minority classes corresponding to real-word medical problems being ignored, information that requires careful examination for prediction.

In our case, none of the studied oversampling methods (SMOTE, SVM SMOTE, Borderline SMOTE, or ADASYN), in combination with ML methods (KNN, LR, or RF) provided the best results in the case of any of the investigated evaluation metrics. KNN obtained higher metric values with the Borderline SMOTE method and RF than with the ADASYN method. It is also notable that in experiments, KNN3 performed better than KNN2. KNN3 achieved a balanced accuracy of 73.78%, while KNN2 scored 63.89%. KNN3 outperformed KNN2 in minority-class performance, scoring 55.72% compared to KNN2’s 33.93%. RF achieved good results with all oversampling methods, even before applying sampling techniques. This fact could lead to the use of compound classifiers in the future, although it is also important to pay attention to relevant attributes and eliminate irrelevant ones with the help of RF [22].

This paper presents a comprehensive research study of a large number of sampling methods used for data balancing with both over- and undersampling methods and discusses different modified versions. This study also serves as a practical guide for stakeholders because when using data with these characteristics, similar results can be expected when applying the investigated algorithms and balancing methods.

In future work, a set of hybrid classifiers (combining different classifiers and applying ensemble methods or feature-level hybridization with similar feature selection methods to PCA), in combination with under- and oversampling methods, could be be applied to the data used in this study.