Comparative Study of Cell Nuclei Segmentation Based on Computational and Handcrafted Features Using Machine Learning Algorithms

Background: Nuclei segmentation is the first stage of automated microscopic image analysis. The cell nucleus is a crucial aspect in segmenting to gain more insight into cell characteristics and functions that enable computer-aided pathology for early disease detection, such as prostate cancer, breast cancer, brain tumors, and other diagnoses. Nucleus segmentation remains a challenging task despite significant advancements in automated methods. Traditional techniques, such as Otsu thresholding and watershed approaches, are ineffective in challenging scenarios. However, deep learning-based methods exhibit remarkable results across various biological imaging modalities, including computational pathology. Methods: This work explores machine learning approaches for nuclei segmentation by evaluating the quality of nuclei image segmentation. We employed several methods, including K-means clustering, Random Forest (RF), Support Vector Machine (SVM) with handcrafted features, and Logistic Regression (LR) using features derived from Convolutional Neural Networks (CNNs). Handcrafted features extract attributes like the shape, texture, and intensity of nuclei and are meticulously developed based on specialized knowledge. Conversely, CNN-based features are automatically acquired representations that identify complex patterns in nuclei images. To assess how effectively these techniques segment cell nuclei, their performance is evaluated. Results: Experimental results show that Logistic Regression based on CNN-derived features outperforms the other techniques, achieving an accuracy of 96.90%, a Dice coefficient of 74.24, and a Jaccard coefficient of 55.61. In contrast, the Random Forest, Support Vector Machine, and K-means algorithms yielded lower segmentation performance metrics. Conclusions: The conclusions suggest that leveraging CNN-based features in conjunction with Logistic Regression significantly enhances the accuracy of cell nuclei segmentation in pathological images. This approach holds promise for refining computer-aided pathology workflows, potentially leading to more reliable and earlier disease diagnoses.