Optimizing Cervical Cancer Diagnosis with Feature Selection and Deep Learning

Abstract

The main purpose of cervical cancer diagnosis is a correct and rapid detection of the disease and the determination of its histological type. This study investigates the effectiveness of combining handcrafted feature-based methods with convolutional neural networks for the determination of cancer histological type, emphasizing the role of feature selection in enhancing classification accuracy. Here, a data set of liquid-based cytology images was analyzed and a set of handcrafted morphological features was introduced. Furthermore, features were optimized through advanced selection techniques, including stepwise and significant feature selection, to reduce feature dimensionality while retaining critical diagnostic information. These reduced feature sets were evaluated using several classifiers including support vector machines and compared with CNN-based approach, highlighting differences in accuracy and precision. The results demonstrate that optimized feature sets, paired with SVM classifiers, achieve classification performance comparable to those of CNNs while significantly reducing computational complexity. This finding underscores the potential of feature reduction techniques in creating efficient diagnostic frameworks. The study concludes that while convolutional neural networks offer robust classification capabilities, optimized handcrafted features remain a viable and cost-effective alternative, particularly when the data count is limited. This work contributes to advancing automated diagnostic systems by balancing accuracy, efficiency, and interpretability.

1. Introduction

Looking at the latest statistics on cervical cancer, one can notice that the number of incidents of this disease decreases. However, it remains a significant challenge both in oncology and in social contexts. According to the World Health Organization (WHO), more than 570,000 new cases of cervical cancer are reported each year, and approximately 311,000 of these cases result in death [1]. In Poland, the five-year survival rate in years 2010–2015 was 48.3% and in Europe it was 62.1% [2]. For cancer, it is a well-known fact that a faster diagnosis will increase the chances of successful treatment for patients [3]. For cervical cancer, risk factors are well known and can be taken into account for a better diagnosis process. These factors can be divided into two categories [4]:

• Main factors that include age of the patient, chronic inflammation of the HPV virus, early sexual relationship experience, large number of sex partners, large number of child deliveries, long smoking habits, and low economic status.
• Possible factors that include long use of contraceptive pills, a diet rich in antioxidants, HIV infection, as well as frequent, untreated inflammations.

The main purpose of diagnosis is to quickly and accurately detect the disease and determine its histological type. The process starts with a patient interview and a physical examination. If a suspicious region is found, a cytological examination is performed.

Current diagnostic workflows often rely on cytological examinations, such as Pap smears or liquid-based cytology (LBC), to detect cervical abnormalities. However, these methods are often subjective and depend heavily on the expertise of pathomorphologists. Despite offering improved slide quality and reduced sample variability, LBC still suffers from interpretive limitations, leading to inconsistencies in diagnosis. Most existing methods try to solve the problem with deep learning models trained on large data sets or focus on slide-level classification, which limits their applicability in resource-constrained environments. To address these gaps, this study proposes a novel approach that integrates handcrafted feature-based methods, derived from clinically relevant diagnostic criteria. Unlike existing methods that focus on slide-level classification or rely on resource-intensive deep learning models, this framework works at the per-cell level, aligning closely with the diagnostic process used by pathologists. By focusing on per-cell classification, this study aims to advance automated diagnosis in resource-constrained environments. In addition, traditional diagnostic systems, such as the Bethesda system, often focus on broad classifications, such as normal versus abnormal, which may lack the specificity required for detailed treatment planning.

Recent developments in computer vision and machine learning, especially in the area of convolutional neural networks, have demonstrated high accuracies in medical image analysis and classification. However, most of these approaches focus on distinguishing between normal and abnormal samples, rarely providing the granular categorization necessary for effective treatment planning. These limitations make them less practical in resource-constrained environments or for granular classifications such as Normal Squamous Intraepithelial Lesion (NSIL), Low-Grade Squamous Intraepithelial Lesion (LSIL) and High-Grade Squamous Intraepithelial Lesion (HSIL).

To address these challenges, this study proposes a framework that combines handcrafted feature-based methods with deep learning techniques, such as convolutional neural networks, to classify LBC images into specific diagnostic categories: NSIL, LSIL or HSIL. According to the literature, hybrid approaches that integrate handcrafted features with deep learning have shown significant promise in the advancement of diagnostic frameworks. For example, Anurag et al. proposed a feature blending approach that combined handcrafted techniques with CNN features, resulting in improved classification accuracy when applied to histopathological images [5]. The results obtained by Daoud et al. also show that combined features can achieve high precision in breast ultrasound image classifications [6].

Unlike existing methods, the proposed approach reduces dimensionality through the incorporation of advanced feature selection methods, enabling high diagnostic accuracy while minimizing computational demands. This allows retaining critical diagnostic information, achieving a balance between computational efficiency, accuracy, and interpretability. Feature reduction is particularly important for LBC data, where extracted morphological features such as nucleus size and nucleus-to-cell ratio capture key diagnostic patterns without requiring extensive processing power. Furthermore, extraction of handcrafted features provides a practical solution for automated cytological diagnostics, offering scalability, cost effectiveness, and adaptability to a variety of clinical settings.

This work bridges the gap between traditional diagnostic methods and modern AI-driven solutions, offering a robust and scalable system tailored to the nuanced requirements of clinical practice. The details of the data set, feature extraction, and classification methods are described in the following section.

2. Materials and Methods

This section describes the proposed integration of machine learning for pattern recognition in the detection of cervical cancer using liquid-based cytology (LBC) slides. The study uses a combination of high-resolution LBC imaging and sophisticated algorithmic models to systematically identify and classify cellular patterns. The proposed scheme takes an image as input and assigns one of the squamous intraepithelial lesion grades as described in Section 2.1. The framework involves morphometric and convolutional neural network feature extraction, dimensionality reduction through feature selection, and classification. Each phase is described in detail in the following subsections, highlighting the methodologies used to achieve efficient and accurate classification. The hyperparameters for each classifier were selected using grid search optimization in the training data.

2.1. Diagnostic Process Overview

Cervical cancer diagnosis traditionally involves a multistage process that begins with a patient interview and physical examination. If abnormalities are suspected, cytological samples are collected from the cervix for further analysis. This subjective analysis often determines the patient’s subsequent treatment plan. Over time, efforts have been made to improve the traditional method in order to increase its objectivity. According to Kitchener et al., incorporating HPV testing into cervical screening protocols improved the sensitivity of detection of high-grade cervical intraepithelial neoplasia, indicating significant advances in the methods’ ability to identify serious diseases [7].

In this paper, we use images from liquid-based cytology (LBC), which is the latest advancement in cytological examination techniques. Liquid-based cytology is becoming more popular than traditional cytology as it reduces variability in sample preparation and improves the quality of diagnostic slides [7]. Unlike the conventional smear method, LBC involves suspending cells in a liquid medium before processing them on a glass slide, resulting in cleaner and more uniform preparations (see Figure 1).

In the next step, a slide is analyzed by a pathomorphologist. To standardize the interpretation and reporting of cervical cytological results, the Bethesda system is used. This system was developed in 1988 and was primarily used to report the results of cervical and vaginal cytology. Today, this system acts as a framework for consistent communication between healthcare professionals, helping to effectively manage and treat irregularities in cervical cytology.

George N. Papanicolaou played a crucial role in the development of cytological diagnostics, who, in 1941, described the usage of a vaginal smear in the diagnosis of uterine carcinoma [8] and in 1942 introduced a new staining procedure for these smears [9]. These advances have brought substantial improvements in cytological examinations, mainly by identifying atypical cells in smears. This technique, named after its creator, is now universally recognized as the Pap smear. Building on these basic techniques, the Bethesda system was established as a uniform method of reporting cervical and vaginal cytology findings. Having been revised on multiple occasions since its invention, it continues to facilitate efficient communication and handling of cervical cytological irregularities. In 2012, Solomon and Kurman described the Bethesda system (TBS) that details definitions, criteria, and terminological clarifications [10]. The TBS format is adaptable and designed to integrate both evolving diagnostic approaches for cervical cancer and advances in cervical pathology. This system employs a two-level reporting structure based on the condition of uterine cervix cells, which can be categorized into two distinct cases:

• Low-Grade Squamous Intraepithelial Lesion—also called LSIL. This grade describes a non-cancerous lesion in which cells of the uterine cervix are slightly abnormal (see Figure 2b).
• High-Grade Squamous Intraepithelial Lesion—also called HSIL. Describes these cases that can become cancerous. Here, uterine cervix cells are moderately or severely abnormal (see Figure 2c).

In addition, the Bethesda system contains information about the method that was used for the slide preparation. To distinguish between LSIL and HSIL cases, pathomorphologists evaluate morphological differences, including cell size, nucleus size, and nucleus-to-cytoplasm ratio as key diagnostic features [4]. These features are crucial for an accurate diagnosis and the appropriate management of cervical cytological abnormalities. Furthermore, the Bethesda system also provides detailed descriptions regarding the slide preparation techniques used, enhancing the reproducibility and reliability of cytological evaluations. This additional information helps ensure that the diagnostic process is consistent and standardized in different healthcare settings.

In this study, we take the comprehensive details captured in the Bethesda system to propose a robust machine learning framework. Using this well-documented and standardized information, we can suggest algorithms for accurate LBC image classification into the appropriate Bethesda system categories. This integration of machine learning with classical diagnostic criteria holds promise in enhancing diagnostic accuracy and efficiency in cytology.

2.2. Cervical Cancer Classification

As previously noted, advancements in diagnostic techniques are essential for improving patient outcomes. Recent studies have highlighted the potential for machine learning integration with traditional diagnostic techniques as a direction to enhance accuracy and efficiency of cervical cancer diagnosis. This section explores various cervical cancer classification approaches. Over the years, a variety of studies have explored the classification of cervical cancer using various techniques. In 2011, Rahmadwati et al. described a texture description based on a Gabor filter to analyze nuclei structure and determine the feature vector [11]. For cancer grade classification, the authors use a clustering algorithm applied to the feature vector and calculate a ration between normal and abnormal cells. Based on that ratio, they assign a grade based on the grading scheme used by pathomorphologists with an accuracy of 89%. Mariarputham and Stephen described a classification system for Pap smear evaluation based on the gray-level co-occurrence matrix (GLCM) and local binary patterns (LBP) [12]. The extracted features were then classified with support vector machines (SVM) and single-layer neural networks to reveal better performance of the SVM classifier. Amole and Osalusi [13] explored a comparable method to evaluate the performance of k-NN and SVM classifiers. Their framework, utilizing GLCM features, successfully classified Pap smear data with an accuracy of 90% for SVM and 88.3% for k-NN. In 2019, William et al. described a fuzzy clustering method for cancer classification that achieved an accuracy of over 96%. The most recent papers report on the combination of serum infrared spectroscopy with machine learning and the application of deep learning techniques [14,15]. Various other studies have also explored the use of deep learning in the classification of cervical cancer. Khoulqi and Nailae, as well as Hemalatha and Vetriselvi, achieved high accuracies with deep learning models [16,17]. Khoulqi and Nailae focused on the classification of MRI images and Hemalatha and Vetriselvi on cervical images. The approach of Subarna and Sukumar combines edge detection, wavelet transform, and CNN architecture for classification [18]. In 2023, Majeed et al. used transfer learning to improve classification performance.

The above discussion has highlighted the integration of machine learning with traditional diagnostic techniques, significantly enhancing the accuracy and efficiency of cervical cancer diagnosis. However, most studies have focused on the classical Pap smear method rather than liquid-based cytology (LBC). Considering the differences between these methods, it is essential to develop a new specialized framework for LBC to enhance both the accuracy and efficiency of cervical cancer detection [19,20]. Klug et al. described their comparative study showing the supremacy of the liquid-based cytology method over the conventional method [20]. The authors presented a comprehensive statistical study demonstrating the superiority of LBC in decision-making processes. Such studies provided new challenges, sparking a new research area. The recent literature review shows how these challenges are being addressed. In 2019, Sornapudi et al. provided a description of a deep learning approach that was able to extract features and classify the LBC slides with accuracy of 95% in the best case [21]. Hut et al. and Kanavati et al. both demonstrated the potential of machine learning in this context, achieving good classification performance [22,23]. Hut et al. used a VGG16 convolutional network to classify between low- and high-risk cases achieving an accuracy of 70.8%, while the accuracy of Kanavati et al. reached 90.7%. In 2023, Wong et al. described the application of ResNet architecture for the classification of cervical cancer [24]. On the other hand, Tan et al. evaluated pre-trained DenseNet-201 models for Pap smear image classification, underscoring the value of transfer learning in leveraging large-scale data sets for specific medical applications [25]. Further improvements showed very good classification results when combining 3D convolutional neural networks with Vision Transformer modules to achieve a remarkable 98.6% accuracy [26].

In the literature, we can also see a variety of studies exploring the application of feature-based approaches but focusing mainly on texture descriptors [27]. These studies have shown that LBP enhances the efficiency of deep learning architectures, especially in the context of content-based image retrieval (CBIR) systems [28]. The development of new feature extraction techniques, including the square symmetric LBP texture descriptor (SSLBP) and the merging two-class linear discriminant analysis (M2CLDA) approach, have improved the performance of image retrieval and classification systems [29]. These studies collectively emphasize the capabilities of LBP and additional feature extraction techniques to improve the performance of deep learning models.

From the above results, we can conclude that the introduction of liquid-based cytology considerably improved the cancer diagnosis process. This process extended features taken into account, and therefore, the morphometric analysis of the LBC features is important. To our knowledge, from the works described above only the attempts of Sornapudi et al. [21] and Zhang et al. [30] provide the calculation of cervical cancer features from Pap smears. The initial and most recent method employs convolutional neural networks to evaluate features. This study proposes an alternative approach that integrates handcrafted feature-based methods with feature selection techniques to address these limitations. By employing stepwise and significant feature selection, the dimensionality of feature vectors is reduced, retaining critical diagnostic information while enabling efficient and interpretable classification. The approach demonstrates competitive accuracy with reduced computational complexity, offering a cost-effective solution particularly suited for low-data environments. In Section 2.4, we propose a set of features resembling the traditional LBC features.

2.3. Image Segmentation

This paper investigates the morphometric characteristics of a cell and its nucleus. For this purpose, binary representations of both objects are required and, therefore, input color image segmentation is required. As segmentation of medical images is not easy, it has also become an increasingly active research field [31,32]. To reduce the noise in the images, median filtering was applied as a pre-processing step. This technique is well known to effectively remove noise while preserving important structural features such as the nucleus boundary. Subsequently, a K-means clustering algorithm was used to segment the images, isolating the nucleus and cell regions and minimizing the influence of irrelevant background data. These pre-processing steps mitigate noise and enhance the reliability of subsequent feature extraction.

K-means clustering is one of the simplest unsupervised learning algorithms to solve a clustering problem [33]. It has a single input parameter that represents a k input cluster. Therefore, prior to the beginning of the clustering process, knowledge of the number of clusters is required. The main idea behind this algorithm is to determine a k center of each cluster. This heuristic algorithm may not be able to converge to the global optimum, therefore the centroids should be placed in a cunning manner. When a centroid is assigned, the next step is to take each data point and associate it with the closest center. When this step is completed, the first step of the algorithm is completed, and the initial grouping is finished. The following procedure is to recalculate the new centroids defined as the centers of the groups initially calculated. This results in a new k centroids and the association procedure of all data must be repeated [33]. The consecutive steps of the generated loop change the location of the k centroids until convergence is achieved. The aim of the algorithm is to minimize the objective square error function (see Equation (1)).

$$J=\sum _{j=1}^k\sum _{i=1}^n\left|\right|x_i^j-c_j{\left|\right|}^2,$$

(1)

where $\left|\right|x_i^j-c_j{\left|\right|}^2$ is a chosen distance measure between a data point $x_i^j$ and the cluster center, $c_j$ is an indicator of the distance of the n data points from their respective cluster centers. Our research uses the RGB color distance between a pixel and a mean RGB color cluster as a measure of distance. The initial centroids are randomly selected, and the cluster with the highest mean RGB value is identified as the result of segmentation. In this study, the optimal number of clusters was empirically determined to be three. Selecting a larger k-value led to the loss of significant data, occasionally forming gaps within nuclei or uneven clusters. However, using two clusters introduced too much meaningless data, leading to unsatisfactory image segmentation. Given this context, opting for three clusters represents a balance between analyzing irrelevant data and ignoring valuable information.

For complete segmentation of a cell and its nucleus, a space partition was required to obtain separate binary images for both cell and nucleus (see Figure 3). These images were produced by thresholding the segmented image ($im{g}_{seg}$) as described in Equations (2) and (3). The thresholds were determined based on the values obtained from the k-means segmentation, which served as a representation of the identified regions. This representation allows for further determination of the morphological features of the cell and nucleus in Section 2.4 and will be treated as an entry point for the feature extraction subsystem.

$$cell(x,y)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}im{g}_{seg}(x,y)>t_c\hfill \\ 1\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$$

(2)

$$nucleus(x,y)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}im{g}_{seg}(x,y)>t_n\hfill \\ 1\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$$

(3)

where $cell(x,y)$ is a binary representation of a cell shape (see Figure 3b), $nucleus(x,y)$ is a binary representation of a nucleus (see Figure 3c), $t_c$ and $t_n$ are thresholds for the cell and the nucleus, respectively.

2.4. Feature Extraction

A feature vector serves as a parametric representation of objects within an image, making its development a crucial phase in every classification system. Thus, accurate feature selection can significantly enhance classification performance. This section introduces a collection of features that characterize cells on LBC slides and reflect their morphological aspects. For effective feature extraction, it is essential to identify the features that distinguish between uterine cervix cell conditions.

Table 1 describes these features, using a normal cell (NSIL) as the reference. Based on this description, we propose the following 27 morphological features:

• Nucleus size ($\mathit{N}$)—is defined as the sum of all nuclei pixels in the nucleus.
• Cell size ($\mathit{C}$)—is calculated as the total number of pixels of the cell in the segmented image.
• Nucleus/cell ratio ($\mathit{NCr}$)—this feature was calculated as a ratio of the nucleus size and the cell size, providing insights into nuclear enlargement, a hallmark of malignancy:

  $$NCr={\displaystyle \frac{N}{C}}.$$

  (4)
• Nucleus perimeter ($\mathit{Np}$)—we define $Np$ as the length of the nuclear boundary of a nucleus N that is approximated by a length of the polygonal approximation of the boundary [34].
• Cell perimeter ($\mathit{Cp}$)—similar to $Np$, cell diameter is defined as the length of the cell membrane of a cell C that is approximated by the polygonal approximation length of the membrane.
• Nucleus/cell perimeter ratio ($\mathit{NCp}$)—this feature was calculated as a ratio of nucleus diameter and cell diameter:

  $$NCpr={\displaystyle \frac{Nd}{Cd}}.$$

  (5)
• Min Axis ($\mathit{MinA}$)—$MinA$ is calculated as a shortest Euclidean distance between extreme points of the segmented region both for nucleus and a cell.
• Min Axis ratio ($\mathit{MinAr}$)—represents a ratio between Min Axis for nucleus and a cell
• Max Axis ($\mathit{MaxA}$)—$MaxA$ is calculated as a largest Euclidean distance between extreme points of the segmented region both for nucleus and a cell.
• Max Axis ratio ($\mathit{MaxAr}$)—determined as a ratio between $MaxA$ for the nucleus and a cell.
• Nucleus Aspect Ratio ($\mathit{Nar}$)—to compute the aspect ratio, the bounding box ($BB$) of a nucleus is obtained. Feature is calculated as a ratio between width ($B{B}_w$) and height ($B{B}_h$) of the bounding box.
• Cell Aspect Ratio ($\mathit{Car}$)—this feature is similar to $Nar$. Here, the bounding box is determined according to the cell image rather than the nucleus.
• Nucleus/cell Aspect Ratio ($\mathit{NCar}$)—here, we calculate the ratio of $Nar$ and $Car$.
• Extent ($\mathit{Ext}$)—extent is a feature that represents the amount of space that the nucleus or a cell occupies with respect to its bounding box. We calculate this feature according to Equation (6). It provides information about how compact or irregular a shape is.

  $$Ext={\displaystyle \frac{area}{B{B}_w\ast B{B}_h}},$$

  (6)

  where $area$ represents the size of a nucleus or a cell.
• Nucleus/cell extent ($\mathit{NCExt}$)—is a ratio of nucleus extent and cell extent.
• Solidity ($\mathit{Sol}$)—solidity measures the smoothness or irregularity of the nucleus boundary. It represents the number of pixels occupied by the object but with respect to its convex hull ($CH$). Solidity is defined by Equation (7).

  $$extent={\displaystyle \frac{area}{Area\left(CH\right)}},$$

  (7)

  where $area$ represents the size of a nucleus or a cell and $Area\left(CH\right)$ is an area of $CH$.
• Nucleus/cell solidity ($\mathit{NCs}$)—defined as a ratio between $NSol$ and $CSol$
• Equivalence diameter ($\mathit{Eq}$)—equivalence diameter is simply a diameter of a circle which area is the same as N or C.
• Equivalence diameter ratio ($\mathit{NCEq}$)—is a ratio between equivalence diameter of the nucleus and the equivalence diameter of a cell.
• Orientation ($\mathit{Or}$)—this feature is known as an axis of the least second moment and provides the information about the angle between the row axis when the coordinate system is placed and the centroid of the nucleus/cell. We calculate the orientation $Or$ according to Equation (8) [35].

  $$Or=tan\left(2{\theta }_i\right),$$

  (8)

  where the angle ${\theta }_i$ is measured counterclockwise from the x–axis.
• Orientation ratio ($\mathit{NCOr}$)—similar to the previous features, it is defined as a ratio between the nucleus and cell orientations.

All of the features mentioned above were calculated for each cell in a binary image ($\mathcal{I}$) obtained during segmentation. The constructed feature vector is then presented as an input of the classification system as described in Section 2.6.

Table 1. LSIL and HSIL in comparison to a healthy cell [4].

Feature	NSIL	LSIL	HSIL
Nucleus size	Very small	Smaller than $1/2$ of cell size	Larger than $1/2$ of cell size
Cell size	Large cell	Modest decrease	Significant decrease
Nucleus/cell ratio	1:10	Change in favor of nucleus	Significant change in favor of nucleus
Stainability	Normal	Hyper-stainability	Hyper-or-hypo-stainability
Brightness around nucleus	None	Can be visible	None

Table 1. LSIL and HSIL in comparison to a healthy cell [4].

Feature	NSIL	LSIL	HSIL
Nucleus size	Very small	Smaller than $1/2$ of cell size	Larger than $1/2$ of cell size
Cell size	Large cell	Modest decrease	Significant decrease
Nucleus/cell ratio	1:10	Change in favor of nucleus	Significant change in favor of nucleus
Stainability	Normal	Hyper-stainability	Hyper-or-hypo-stainability
Brightness around nucleus	None	Can be visible	None

2.5. Feature Selection and Ranking

Guyon and Elisseeff noted that using a large number of features is not always beneficial [36]. They suggested that when classification accuracy is not significantly compromised and the error rate remains relatively stable, it is preferable to use fewer features in the feature vector. In this study, we have embraced this approach to determine the optimal combination of the features proposed for the feature vector in the classification of cervical cancer. To achieve this, we have employed five feature selection methods, as described below. The statistical methods are robust to noise, as they evaluate feature distributions rather than raw data values, reducing the impact of outliers and irrelevant variations. Combined with pre-processing steps, such as median filtering, these feature selection techniques ensure the selection of highly informative and resilient features for classification.

• Kruskal–Wallis Test
  This is a non-parametric statistical test used to rank features based on their discriminatory power across diagnostic categories. It determines if the samples originate from the same distribution [37]. It is often regarded as the one-way ANOVA method on ranks or groups. The null hypothesis assumes that the cumulative distribution functions in the populations are equal, for $k>2$. For this test, we recorded a probability value (p-value) for the null hypothesis.
• Kormogolov–Smirnov Test
  This statistical metric represents characteristics through a distribution associated with a data sample [38,39]. This test determines whether the samples are drawn from the same distribution. If the samples come from two different distributions, then the KS test returns a value 1, otherwise it returns a value 0.
• Friedman Test
  A test proposed by Milton Friedman [40] that can be used for feature ranking. It assesses differences between at least three correlated classes. Here, we recorded a probability value (p-value) for this nonparametric Friedman test.
• Stepwise Feature Selection
  This is an iterative approach that adds or removes features to maximize classification performance. Given a set of extracted features, the stepwise feature selection method (SFS) chooses a subset of features that provide the best performance. The method performs a forward feature selection that in an iterative way adds the best features to the subset of selected features. In every iteration, the best prediction features are used and their significance is estimated with the F-test (see [41]). Here, we use the backward version of this algorithm (SBS), where the assumption is made that initially all features of the original feature set are used. At each iteration, the features with the worst prediction are eliminated.
• Significant Feature Selection
  This retains features with statistically significant differences between classes. Significant feature selection (SBS) uses statistical significance to reduce the full vector of features. This method estimates a $\mathit{p}-\mathrm{values}$ for each feature in the feature vector. This calculation is based on the two-sample Student’s t-test for the difference of the population means with unequal variances with the assumption that samples are normally distributed. Usually, we consider a feature being statistically significant if its $\mathit{p}-\mathrm{value}$ is smaller than 0.05 which is a typical significance value in biomedical sciences and elsewhere.

Since careful selection and ranking of features can be essential for optimizing the performance of the LBC cervical cancer classification system, we adopted the above feature selection and ranking procedures to construct several reduced feature vectors. By implementing the described feature selection methods, we aim to enhance the model’s efficiency, while maintaining its high accuracy of early cancer detection. Furthermore, the choice of feature selection methods was made to identify features with both high clinical relevance and strong discriminatory power. Features such as nucleus-to-cytoplasm ratio, nucleus size, and cell texture were selected because they align with established diagnostic criteria used by pathologists for cervical cancer diagnosis. These features were validated for their relevance and statistical significance using robust tests like Kruskal–Wallis and Kolmogorov–Smirnov. This approach ensures that the selected features not only contribute to classification accuracy but also enhance the interpretability and clinical applicability of the results.

2.6. Feature Classification

To analyze the selected features, we explore various classification algorithms and present their accuracy in distinguishing between TBS grades. We start with three popular and simple classifiers:

• Linear discriminant analysis—here, we define a function that is a linear combination of the observations x (see Equation (9)):

  $$g\left(x\right)=w^{t}x+w_0$$

  (9)

  where w is the weighting vector and $w_0$ is a bias [42]. If $g\left(x\right)=0$, then Equation (9) defines a decision surface and separates observations of one class from another class. For a two-category classifier, we classify the data into first class ($C_1$) if $\left(x\right)>0$ and second class ($C_2$) if $\left(x\right)<0$. For $\left(x\right)=0$, the observation is classified into either class.
  For a multi-class problem we need to define several linear discriminant functions as in Equation (9). The number of functions should be the same as the number of classes used for classification. In this case, a classification rule will be defined by Equation (10).

  $$g_i\left(x\right)=w_i^{t}x+w_{i0}fori=1,\dots nc$$

  (10)

  where $nc$ is the number of classes. Equation (10) defines the decision regions $R_i$ for $nc$ classes where $g_i\left(x\right)$ is the largest discriminant for $x-es$ in that region.
• K-Nearest Neighbors (KNN)—a popular and simple classifier that classifies a new data point based on the majority class among its k closest neighbors in the feature space [42]. Training of the KNN classifier consists of saving a feature vector values along with class label. The classification is based on the calculation of the distance metric, typically the Euclidean distance, to find the k closest points [43]. In this work, we also adopted the same metric, but additionally we have tested different numbers (between 1 and 10) of neighbors taken for label estimation. In the result section (see Section 4) we showed accuracy values for $k=1$ and $k=10$.
• Support Vector Machines (SVM)—used to separate two or more classes of observations by constructing a boundary between them using border points from each class called support vectors [44]. Here, we employ a kernel-based approach to segregate data, which ensures a robust generalization of the problem. In this way, the classification of an unknown point is classified by its position with respect to the boundary. Training of the SVM model involves an iterative minimization of an error function (Equation (11)).

  $${\displaystyle \frac{1}{2}}{w}^{T}w+C\sum _{i=1}^N{\epsilon }_i$$

  (11)

  with the following restrictions:

  $$y_i(w^T\varphi \left(x_i\right)+b)\ge 1-{\epsilon }_{i}and{\epsilon }_i\ge 0,i=1,\cdots ,N$$

  (12)

  where C and b are constants, w is the weight vector, ${\epsilon }_i$ is a bias value that deals with overlapping cases and $\varphi$ is a kernel function that transforms input data into the feature space. The constant C has a significant influence on the error rate and must be carefully estimated during the training process. In this study, we perform a grid search on values varied between ${10}^{-3}$ and ${10}^3$.
• Neural Networks (NN)—designed to simulate the behavior of the human brain to enable computers to learn from data. They consist of multiple layers of interconnected neurons, each connected through connections similar to synapses that carry weighted information. These weights are adjusted during training. The network prediction ability is then optimized based on input features. In this work, we adjust the internal parameters (weights and biases) based on the differences between the actual output and the desired output. This process, known as backpropagation, uses gradient descent to minimize network error across 200 iterations. For optimization, Adam’s optimizer was used and the rectified linear unit (ReLu) was used as an activation function in hidden layers. Here, the learning rate, number of hidden layers, and the number of neurons per layer were fine-tuned, with the learning rate ranging from 0.001 to 0.1 and the number of neurons per layer ranging from 10 to 100.

In addition to the above traditional classifiers, we explore how features extracted with Convolutional Neural Networks (CNN) perform on the same data set. We will exercise their ability to recognize spatial hierarchies and patterns in images. In our research, networks that have been shown to have high effectiveness in histological image recognition tasks were applied [45,46]. These are the VGG16 and VGG19 CNN models pre-trained on the ImageNet data set. VGG16 consists of 16 layers with weights, while VGG19 includes 19 weighted layers, both having a deep architecture of convolutional layers followed by fully connected layers. These are simple architectures that use only 3 × 3 convolutional layers stacked on top of each other in increasing depth. Due to their ability to resemble complex spatial hierarchies in images, they have the potential to handle the intricacies and variations specific to cervical cytology, providing robust classifications that support a timely and accurate medical diagnosis.

To maintain consistency and comparability, the classification of CNN features extracted was evaluated using the same classifiers as used for traditionally extracted features. This approach takes advantage of the powerful feature representation of convolutional neural networks while avoiding their computational burden during the classification phase. As shown in Section 4.2, this method achieved high classification performance, highlighting the robustness of the proposed framework.

2.7. Classification Metrics

This section will introduce the classification metrics used for LBC slide classification. To comprehensively evaluate the obtained results, it is essential to define metrics that can demonstrate the effectiveness of the proposed model. This approach allows for further comparison of the classification methods [47].

The fundamental method includes defining the confusion matrix, which records the classifiers’ output [48]. In this research, we employ a multi-class variant of the confusion matrix. This matrix includes three values on its diagonal that indicate the count of correct responses (“true”). The other six values represent misclassifications. Based on the recorded values, the values of “true positive” (TP), “false negative” (FN), and “false positive” (FP) can be calculated. These metrics reflect the classifier’s decision-making accuracy. The “false negative” metric is determined by summing all incorrectly classified examples of a class. Conversely, the “false positive” metric represents the total number of instances where other classes have been incorrectly identified as this particular class. For further evaluation of model accuracy, Precision and Recall measures are defined. They help to understand how well a model performs in terms of its predictive capabilities, particularly when dealing with data sets where the balance between classes is a critical factor.

Based on the confusion matrix values, we define the measures Precision and Recall for our system. Precision measures the accuracy of positive predictions made by the model. It is calculated as the ratio of true positive observations to the total predicted positives. Recall, on the other hand, measures system’s ability to make true positive classifications and is defined as a ratio of true positives and all actual positive values. These metrics are often used together to provide a complete picture of the performance of a model. In many practical applications, there is a trade-off between precision and recall, which can be balanced depending on the specific requirements. In the oncological classification, keeping both values as high as possible is very important. For the comprehensive assessment of the classifiers, Error and Accuracy were also calculated. Error is the proportion of false positives and false negatives to the total number of samples, while Accuracy is simply 100% − Error.

For a reliable assessment of model performance, a five-fold stratified cross-validation technique was applied. It involves dividing the data set into five equal parts, ensuring that each fold maintains the same proportion of classes as the original data set, which preserves the integrity of the distribution. During each of the five iterations of model assessment, four of the five folds are used for training, and the remaining fold is used to test the performance. By repeatedly training and testing the model across these folds, we gain insight into the stability of the model and its ability to generalize across different data samples.

3. Cervical Cancer Database

To evaluate a proposed framework, we assembled a database of cytological images of cervix uteri smears prepared using the liquid-based cytology (LBC) method according to the following procedure:

• Specimen Collection—the samples were collected with a brush and suspended in a preservation solution.
• Specimen processing—the solution was then processed through an automated separator to isolate cells from debris and placed on glass slides.
• Slide staining—slides were stained using the Papanicolaou technique, which employs specific staining agents to highlight cellular details for morphological analysis. In general, staining methods like hematoxylin and eosin or Papanicolaou staining enhance the visibility of cells and nucleus structures.
• Slide digitization—the stained slides were acquired from the Department of Patomorphology of the Pomeranian Medical University in Szczecin, Poland, and digitized in the Department of Pathology and Oncological Cytology at the Medical University of Wrocław, Poland, using an Olympus BX 51 microscope equipped with an Olympus DP72 camera attached to the microscope head. The images were captured using the Olympus CellD software at a resolution of 300 dpi, with dimensions of 2510 × 1540 pixels. The collected database contains 263 LBC cytology images, with 158 classified as Normal Squamous Intraepithelial Lesion, 70 as High Squamous Intraepithelial Lesion, and 35 as Low Squamous Intraepithelial Lesion. For cell shape analysis, we utilized ImageJ software to extract single cell images, resulting in 428 images: 233 NSIL, 110 HSIL and 85 LSIL, all annotated at the per-cell level. For data curation, the following inclusion and exclusion criteria were applied:

  –

  Inclusion criteria: Slides of adequate quality and clear diagnostic classification by expert pathologists

  –

  Exclusion criteria: Low slides quality, ambiguous diagnoses, or insufficient cellular material for accurate segmentation.

  The images were retrospectively collected at a single institution to ensure consistency in sample preparation and diagnostic annotations. Although this limits the diversity of the data set, it provides a high-quality foundation for per-cell classification, which is not available in most public data sets.
• Case validation—in the next step, an expert pathomorphologist examined the slides to categorize the samples into NSIL, LSIL, or HSIL based on the Bethesda system criteria. This step provided ground-truth labels for training and validating the classification algorithms.

The methods described in this section establish a framework for automated classification of cervical cancer cytology data. By integrating liquid-based cytology imaging, advanced feature extraction, dimensionality reduction techniques, and machine learning, this study aims to improve the accuracy and precision of cervical cancer diagnosis. The next section provides a description of the results obtained from the implementation of the proposed methodology. It includes an evaluation of the effectiveness of feature selection methods in reducing dimensionality, classification performance across different machine learning approaches, and comparisons between handcrafted feature-based methods and a convolutional neural network. Performance metrics such as accuracy, precision, recall, and computational efficiency are analyzed to validate the proposed approach.

4. Results

To check the behavior of the proposed features, the performance of different classifiers in different testing scenarios was checked. The scenarios were defined according to the feature selection methods described in Section 2.5. The features that were chosen in the scenarios are described in

Table 2. Reduction rates for feature selection methods.

Test	Chosen Features	Reduction Rate
Kruskal–Wallis	N, C, NCr, Np, Cp, NCp, NMinA, CMinA, NCMinAr, NMaxA, CMaxA, NCMaxr, NExt, NSol, CSol, NCSol, Neq, Ceq, NCeq	29.6%
Kormogolov–Smirnov	N, C, NCr, Np, Cp, NCp, NMinA, CMinA, NCMinAr, NMaxA, CMaxA, NCMaxr, NExt, CExt, NCExt, NSol, CSol, NCSol, Neq, Ceq, NCeq	22.2%
Friedman	N, C, NCr, Np, Cp, NCp, NMinA, CMinA, NCMinAr, NMaxA, CMaxA, NCMaxr, NExt, NCExt, NSol, CSol, NCSol, Neq, Ceq, NCeq	25.9%
Stepwise feature selection	N, C, NCMinAr, CMaxA, NExt, Neq, Ceq	74.1%
Significant feature selection	NCr, Cp, NCp, NMinA, CMinA, NCMinAr, NMaxA, CMaxA, NCMaxr, CAr, NCExt, NSol, NCSol, Neq, Ceq, NCeq	40.7%

. Additionally, for comparison purposes, classification for a full feature set was performed, as well as classification with features extracted with convolutional neural networks. These results are summarized in

Table 3. Classification results with respect to feature selection methods.

	CNN	Baseline	KW	KS	Fr	SFS	SBS
NN	92.9%	82.7%	87.7%	86.3%	88.1%	89.7%	88.1%
SVM	93.3%	90.1%	90.9%	91.8%	91.4%	90.1%	93.0%
KNN1	87.1 %	80.7%	88.9%	88.5%	88.9%	86.8%	85.6%
KNN10	84.4 %	84.0%	88.5%	87.7%	88.1%	86.8 %	86.8%
Disc. An.	90.9%	87.2%	86.4%	87.2%	87.7%	87.2%	87.7%

. In subsequent sections, we will present results of the systematic performance evaluation of the proposed scenarios across various metrics, including accuracy, precision, and recall. The results highlight the effectiveness of feature selection methods in reducing the feature vector.

4.1. Feature Vector Reduction Results

To improve the computational efficiency and performance of our classifiers, we implemented five techniques to decrease the dimensions of our feature vectors. For this purpose, we implemented a multifaceted approach, integrating statistical tests and algorithmic feature selection techniques to streamline our feature set. Specifically, the implementation of the Kruskal–Wallis test, Kolmogorov–Smirnov test, and Friedman test provided a robust statistical framework to identify features with significant differences. In addition, stepwise feature selection and significant feature selection methods were utilized to refine the feature vectors. The following description details the results of these reduction methods and highlights their impact on model performance.

At first, features reflecting the real morphometric features of the cells were calculated. These features resembled the characteristics evaluated by a specialist when making a diagnosis, which led to the calculation of 27 morphometric features. Upon examining the characteristics of these features, it becomes evident that the importance of extracted features varies when classifying cervical cancer. In Table 2, we summarize the features that are the most important according to the test used for size reduction. The purpose of our experiments was to check whether the reduced feature vector provides information for accurate cancer classification in the automated diagnosis task. To accurately assess the classification performance of the reduced feature vectors, a reduction rate metric has been introduced. This metric indicates the extent of the reduction by representing the percentage of features omitted from the entire feature vector. The reduction rate is visualized in Figure 4.

From the results presented in Table 2, it is evident that both stepwise feature selection and significant feature selection provide the most substantial reduction of the set, at 74.1% and 40.7%, respectively. A detailed examination of the classification results for the reduced feature set will be discussed in the next subsection.

4.2. Classification Results

With the feature vectors created according to the description mentioned above, we performed a test to check the ability of the handcrafted features to distinguish correctly between cervical cancer grades. The tests were performed on images from the database described in Section 3. At first, we checked the classification accuracy for the 27 features as a baseline. Then the reduced feature vector classification was performed. Furthermore, we compared these results with the classification results for features extracted with VGG16 convolutional neural networks. The obtained results are summarized in Table 3. Each classifier was presented with a full feature vector (baseline), five reduced feature vectors, and a CNN feature vector. Looking at the results, we can easily notice that the SVM classifier outperformed other classifiers for all feature vectors (see Figure 5). We cannot see this dependency for all remaining classifiers.

In

Table 4. Precision and recall rates with respect to feature selection method, including confidence intervals [in %].

	Precision (95% CI)					Recall (95% CI)
	NN	SVM	KNN1	KNN10	Disc An.	NN	SVM	KNN1	KNN10	Disc An.
Baseline	82.7 (81.5–84.0)	89.3 (88.0–91.5)	81.1 (80.0–82.5)	85.4 (84.0–86.7)	85.5 (84.1–86.9)	82.7 (81.5–83.9)	89.6 (88.5–91.4)	81.1 (80.0–82.3)	83.9 (83.0–85.0)	84.8 (83.2–85.9)
KW	87.6 (85.3–89.7)	90.9 (89.2–92.7)	87.5 (85.0–89.8)	88.5 (86.0–90.8)	86.6 (85.0–88.1)	87.7 (85.3–89.9)	90.9 (89.0–92.9)	87.2 (84.5–89.6)	88.5 (86.2–90.4)	86.4 (84.2–88.0)
KS	86.3 (83.7–88.5)	88.5 (86.4–90.3)	86.7 (84.2–88.9)	89.7 (87.5–91.6)	87.3 (85.0–89.3)	86.4 (84.0–88.6)	88.5 (86.5–90.6)	86.8 (84.3–88.8)	89.3 (87.2–91.0)	86.8 (84.3–88.7)
Fr	88.2 (85.5–90.4)	89.7 (87.6–91.5)	85.5 (82.5–88.1)	90.1 (87.7–92.0)	87.9 (85.2–90.1)	88.1 (85.3–90.3)	89.7 (87.4–91.8)	85.6 (82.8–87.9)	89.7 (87.2–91.7)	87.6 (84.8–90.0)
SFS	89.8 (87.3–92.1)	88.8 (86.2–91.0)	88.9 (86.3–91.1)	87.0 (84.4–89.2)	86.9 (84.3–89.0)	89.7 (87.0–92.0)	88.8 (86.0–91.2)	88.8 (86.0–91.2)	86.8 (84.0–89.3)	86.8 (84.0–89.0)
SBS	88.0 (85.2–90.4)	91.4 (89.2–93.2)	84.9 (82.0–87.6)	89.7 (87.4–91.9)	85.2 (82.5–87.7)	88.1 (85.2–90.6)	91.4 (89.1–93.4)	85.2 (82.3–87.8)	89.3 (87.0–91.6)	84.8 (82.1–87.2)

, the recorded precision and recall values for all classifier types are presented. From the table we can see that the highest precision was noted for Friedman, stepwise and significant feature selection methods. In case of recall (see Figure 6) the highest values were observed for the significant feature selection method reaching 91.40% when the SVM classifier was used. The same observation was made for precision, as can be seen in Figure 7. Another important discovery is that feature reduction not only highly increased the accuracy of the proposed framework but also enhanced the precision and recall of the system. Only in the case of Kormogolov–Smirnov test and stepwise feature selection method did we not observe such an increase.

Taking the above into account, we can see that SVM is the best classifier for which we have recorded the highest accuracy values in all cases. When we combine this classifier with the significant feature selection method (highest accuracy for SVM), we can see that the precision and recall values are also very high. What is also important to note is that this combination allows one to reach the accuracy similar to CNN feature vector and SVM classifier. Furthermore, an efficiency comparison between CNN-based feature extraction and conventional methods showed that the latter is more efficient. For a single cell image, the processing time for conventional methods was approximately 3.25 s, compared to 4.47 s for VGG16, making conventional methods 1.38 times faster. At the same time, memory usage was also slightly lower for conventional methods (0.0208 GB vs. 0.0246 GB). Considering this, we can propose a classification system based on SVM and a feature vector consisting of the features selected with the significant feature selection method. Taking that into account, it can be noted that the presented findings demonstrate the potential of combining handcrafted features with feature selection for accurate and efficient diagnosis.

5. Discussion and Conclusions

In this paper, we present a classification framework for liquid-based cytology (LBC) slides for automated cervical cancer diagnosis. The methodology described here uses the conventional classification approach in which morphological features are calculated. To our knowledge, this is the first automated classification approach for slides prepared using the LBC mmethod rather than the typical Pap smear method that uses hand-crafted features. Using feature reduction methods, the framework reduces computational demands without sacrificing diagnostic accuracy. This approach addresses critical challenges in resource-constrained regions, where access to high-end computational resources and large data sets for deep learning is limited.

Furthermore, this study shows the significant clinical implications of integrating machine learning with traditional cytological techniques in the diagnosis of cervical cancer. The application of machine learning, especially a Support Vector Machine (SVM) classifier, improves diagnostic accuracy and consistency. This approach reduces the risk of misdiagnosis and ensures standardized diagnostic processes by simplifying the examination of cytological slides. The results clearly demonstrate that the presented framework is capable of making accurate predictions. As observed, the SVM classifier achieved the best performance, with an error rate of only 7%, while also providing the highest precision rate. The recall values are at an acceptable level, allowing us to conclude that among the five tested classifiers, SVMs are the optimal choice.

Additionally, our findings highlight the computational efficiency of the proposed framework. A comparison between conventional methods and CNN-based feature extraction revealed that conventional methods are approximately 1.38 times faster for processing a single cell image and require slightly less memory. This efficiency makes conventional methods a better choice for resource-constrained clinical environments, further emphasizing the practicality of handcrafted features combined with SVM classifiers.

Moreover, our study shows that the stepwise and significant feature selection methods offer the best trade-off between feature vector size and classification accuracy. As mentioned in Section 2.5, feature selection can enhance the efficiency and interpretability of the diagnostic framework. Recent research has introduced sophisticated algorithms such as Opposition-based Harmony Search to select features, which improved performance on Pap smear image classification tasks [49]. Similarly, the integration of deep learning techniques, such as transfer learning with DenseNet and Vision Transformers, allowed the achievement of up to 98.6% classification accuracy [25,26]. Although these studies describe very good results, they also face challenges, including computational demands and reliance on large data sets. These constraints can limit their applicability in resource-constrained environments or for granular classifications like HSIL and LSIL. In contrast, our approach focuses on handcrafted morphological features combined with advanced feature reduction techniques, which significantly minimize computational requirements without sacrificing diagnostic accuracy. This makes our method particularly advantageous in clinical settings where resources and data are limited.

One limitation of the proposed methodology is the relatively small size of the data set (263 images), as it may affect the generalizability of the results. Furthermore, the data set was collected from a single institution, which may introduce biases due to the lack of diversity in patient demographics or preparation methods. However, the data set was specifically curated for per-cell classification, ensuring high-quality annotations aligned with diagnostic workflows. To address these limitations, future research should focus on validating the framework on a larger and more diverse data set. The expansion of the database with images from multiple institutions and the incorporation of diverse patient populations would improve the robustness and clinical relevance of the framework. Furthermore, integrating hybrid approaches that combine the interpretability of handcrafted features with the robustness of advanced deep learning models could further enhance performance and adaptability. Future work should explore the optimization of such hybrid frameworks for cervical cytology diagnostics.

Taking all of the above into consideration, we can deduce that the proposed novel LBC classification framework is suitable for automatic liquid-based cytology classification, and the described features are effective in characterizing the morphological properties of both the nucleus and the cell. Furthermore, we have shown that the features chosen by the stepwise and significant feature selection methods provided the highest quality of the features, reducing the feature vector by 74.1% and 40.7%, respectively.

In conclusion, the study presents a robust and clinically relevant framework for the automated classification of cervical cancer using LBC images. Significant improvements in diagnostic accuracy and reduced subjectivity underscore the potential of integrating machine learning with traditional cytological methods. The clinical impact of this research is significant, offering a pathway to more reliable and timely diagnoses, which is critical in the treatment and management of cervical cancer. Further research and adaptation of this framework can extend its benefits to other areas of medical diagnostics, demonstrating the versatility and value of this innovative approach.