A Smartphone-Based Cell Segmentation to Support Nasal Cytology

Abstract

Rhinology studies the anatomy, physiology, and diseases affecting the nasal region—one of the most modern techniques to diagnose these diseases is nasal cytology, which involves microscopic analysis of the cells contained in the nasal mucosa. The standard clinical protocol regulates the compilation of the rhino-cytogram by observing, for each slide, at least 50 fields under an optical microscope to evaluate the cell population and search for cells important for diagnosis. The time and effort required for the specialist to analyze a slide are significant. In this paper, we present a smartphones-based system to support cell segmentation on images acquired directly from the microscope. Then, the specialist can analyze the cells and the other elements extracted directly or, alternatively, he can send them to Rhino-cyt, a server system recently presented in the literature, that also performs the automatic cell classification, giving back the final rhinocytogram. This way he significantly reduces the time for diagnosing. The system crops cells with sensitivity = 0.96, which is satisfactory because it shows that cells are not overlooked as false negatives are few, and therefore largely sufficient to support the specialist effectively. The use of traditional image processing techniques to preprocess the images also makes the process sustainable from the computational point of view for medium–low end architectures and is battery-efficient on a mobile phone.

1. Introduction

Thanks to the numerous studies in the field of computer vision applied to the medical and biomedical field, we now have many additional tools to support specialists in their tasks [1,2,3,4,5]. Modern technologies have improved the acquisition, transmission, and analysis of digital images. A growing benefit is also provided thanks to the spread of fast network connections for smartphones, allowing for the exchange of large amounts of clinical data also useful for remote diagnosis or follow-up [6,7,8].

Segmentation and contour extraction are important steps towards the analysis of digital images in the medical field, where such images are routinely used in a multitude of different applications [9]. Segmentation algorithms, based on structural analysis, continue to be used, often as an ensemble of segmentation techniques, especially in critical applications, such as lesion localization [10,11]. Other approaches, based on biased normalized cuts or light techniques, are also devised [12,13]. Many studies have also been conducted in the segmentation and classification of cells from digital images. Almost all studies are in the field of hematology. An interesting study into the classification of white blood cells (WBCs) is reported in [14]. In some studies, only segmentation aspects are discussed [15,16], while a neural network-based classifier of cytotypes in the hematological smear of a healthy subject was described in [17]: starting from digital scans of hematological preparations, it showed over 95% accuracy. Many other papers report interesting results about this last theme [18,19,20].

One of the fields that can benefit from the above technologies is nasal cytology, a branch of otolaryngology, which is gaining increasing importance in the diagnosis of nasal diseases due to the simplicity of the diagnostic examination and its effectiveness. In fact, the global spread of the nasal diseases is significant: allergic rhinitis is estimated to affect 35% of the world’s population and the World Health Organization considers it a growing epidemic form as, in a few years, 50% of children may be allergic. Rhinosinusitis affects 4% of the world’s population, and nasal polyposis 5%. Non-allergic vasomotor rhinitis affects 15% of people [21].

To the best of our knowledge, to date, there are no public or private laboratories that carry out the examination of the cell population of the nasal mucosa routinely, as instead it is done for hematological tests. This is for different reasons: firstly, because diagnostics based on nasal cytology have grown recently; secondly, because economic interest is still residual; finally, because the spectrum of diagnosable pathologies is not as extensive as in other fields of medicine. Typically, a rhinocytologist who wants to benefit from a cytological study must independently arrange a set of personal instruments, or, more frequently, carry out direct microscopic observation and manual cell counting using a special rhinocytogram.

Methods and techniques designed for hematology cannot be used directly for nasal cytology; for example, the WBC appear in almost all cases as isolated from each other, while nasal mucosa cells often appear amassed in the smear.

The first studies about the automatic extraction and classification of the cells of the nasal mucosa are reported in [22,23,24] where a diagnostic support system provides cell counting automatically—it uses segmentation algorithms to extract cells and a convolutional neural network to classify them. The sampling process and the diagnosis remain human activities, carried out by the specialist, but the whole time and effort are reduced considerably, letting the accuracy of the diagnoses remain unchanged or even be improved. To the best of our knowledge, there are no further contributions in the literature.

The further request of the stakeholders is to considerably reduce the cost of the analysis and of the instrumentation with the aim of increasing the capillarity of the analysis itself. Therefore, the challenge we have been given is to carry out the entire evaluation of the cell population on a mass device, such as a smartphone, fully automatically (as shown in Appendix A). Devices with limited resources will interact with the surrounding environment and users. Many of these devices will be based on machine learning models to decode meaning and the behavior behind sensors’ data, to implement accurate predictions and make decisions [25]. Several research papers have focused on the possibility of bringing artificial intelligence to devices with limited resources and there have been efforts in decreasing the model’s inference time on the device. Machine learning developers focus on designing models with a reduced number of parameters in the Deep Neural Network model, thus reducing memory and execution latency, while aiming to preserve accuracy, as far as possible. It is evident that, at the moment, there are several problems to overcome, first among which is the limitation of the computational capacity of mobile architectures [26,27,28,29,30,31,32,33,34].

In this paper a novel system based on a smartphone is presented to support rhinocytologists during cell observation. It carries out cell extraction from the digital image of the microscopic fields. Once this is done, the specialist can independently evaluate the segmented cells or send them to the Rhino-cyt platform [2], which will also perform the fully automatic classification, giving back the final rhinocytogram. This way he significantly reduces the time for diagnosis.

2. Rhino-Cytology

Nasal cytology is a very useful diagnostic method in rhino-allergology—it allows for the detection of cellular variations of an epithelium exposed to acute/chronic irritations or inflammations of different nature and makes it possible to diagnose some nasal pathologies [35,36]. The strengths of this methodology lie in the simplicity of the diagnostic examination; in fact, it is totally painless, safe, and fast, as it can be conducted in an outpatient clinic. Starting from the assumption that the nasal mucosa of the healthy individual consists normally of only four cytotypes, cytological diagnostics is based on a fundamental axiom that states that, if other cells, such as eosinophils, mastcells, bacteria, spores, and fungal hyphae, are present in the rhinocytogram, then the individual can be affected by a nasal pathology. A quantitative analysis of the pathological cells contained in the nasal mucosa and their state of rest or activation allows for the indication of a targeted therapy to the patient [37].

2.1. A. The Cytodiagnostic Technique

The diagnostic examination is accomplished through the following three main phases:

• Sampling: Consists of collecting a sample of nasal mucosa containing the superficial cells. It is carried out using a disposable plastic curette, called nasal scraping, or a simple nasal swab is preferred for smaller patients;
• Processing: The material collected is placed on a slide and dried in the open air. Then, the slide is stained using the May Grunwald–Giemsa method, which provides the cells with the classic purple staining and highlights all the cytotypes present in the nasal mucosa. Usually, the complete staining procedure takes about 20–30 s with rapid staining techniques;
• Microscopic observation: An optical microscope is used, mainly connected to a special camera to view the cells on a monitor. The diagnostic protocol involves viewing and analyzing 50 digital images for each slide, called fields, usually at 1000X magnification.

The cell count allows a diagnosis to be made simply by counting the cells present in the 50 analyzed fields. This process allows the specialists to draw up a diagnostic report.

2.2. B. Types of Cells Involved

Different types of cells are considered in the diagnosis of nasal diseases. Considering the diversity of the cells present in the nasal mucosa, it is, therefore, appropriate to draw up a classification of the different cytotypes present both in a healthy individual and in an individual with a pathology. The nasal mucosa cells belonging to a cytotype show some elements with high similarity; however, each cytotype appears quite different from all the others. These features allow their automatic classification [23]. A brief description of the appearance of each of the cells located in the nasal mucosa is reported below, and corresponding sample images are shown in Figure 1:

Ciliated: Among the most common cytotypes of the nasal mucosa are ciliated cells. They have a polygonal shape and a nucleus situated at various heights from the basement membrane. The apical region, the seat of the ciliary apparatus, is recognized as a well-represented body that includes a large part of the cytoplasm and the nucleus;

Muciparous (goblet cells): The muciparous cell is in the shape of a cup and is a unicellular gland. The nucleus is always situated in the basal position (the strengthener of the nuclear chromatin is typical) while the vacuoles, containing mucinous granules, are located above the nucleus, giving the mature cell its characteristic chalice shape;

Neutrophil: characterized by a polylobate nucleus, whose lobes are joined by very thin strands of nuclear material within the cytoplasm, which contains finely colored granules;

Eosinophil: usually has a bilobed nucleus and acidophilous granules that intensely stain with eosin (hence the name) as an orange-red color;

Mast cell: a granulocyte with an oval nucleus, covered in purple.

The nasal mucosa of a healthy individual normally contains ciliated, mucipara, striated, basal, and sporadic neutrophils cells. In the nasal epithelium, there can also be different types of inflammatory cells, where each of them can be a sign of a nasal pathology. They are known as immunophlogosis cells (eosinophils, mast cell, lymphocyte). Additionally, a significant presence of neutrophils is interesting—knowing the functions they perform helps motivate different therapeutic strategies [37]. Here, metaplastic cells have been merged into one class (epithelial) with ciliated cells because their nuclei are similar and this merging does not influence the diagnostic protocol.

3. Image Acquisition and Processing

Thanks to the large number of contexts in which digital image processing has been successfully in experimentation, its use has also increased in medicine that is becoming highly dependent on it and represents fundamental pillars of modern diagnosis [38,39,40,41,42].

The images of the smears used in this experimentation, supplied by the Policlinico di Bari, have been acquired with a Samsung Galaxy S6 Edge smartphone with a 16 Mpixel digital rear camera, with a photo resolution of 5312 × 2988 pixels and an aperture of F / 1.9. A specific smartphone adapter was also used, as shown in Figure 2. The system proposed here is based on image enhancement, segmentation, and morphological processing [43], which allows for the extraction of the cells present in the photo acquired by the smartphone camera and will be dealt with in this paper shortly.

3.1. Image Enhancement

There are several definitions of image enhancement in the literature but the one that best fits the context states that this process allows for the improvement of the quality and information contained in an original image before it is processed [44,45,46]. The result of this pre-process represents an improved image that highlights some features more relevant than others both for the visual and automated systems, which otherwise would not be visible in the original image; therefore, an image will be easier to interpret in certain contexts.

Image enhancement involves several aspects of an image: those that will be dealt with in this work concern brightness (or luminance), contrast (the difference between the pixel of higher and lower intensity), and saturation.

In Figure 3, the effects of image enhancement techniques on an image of nasal cells with low brightness and contrast are evident. The central image appears sharper and this brings many advantages, as the cells appear more visible and highlighted due to the increased contrast. The image on the right is too bright and needs the so-called gamma correction.

Gamma correction hides brightness defects in an image using a non-linear function based on the following transformation:

$$o={\left(\frac{I}{255}\right)}^Y\cdot 255$$

(1)

where the γ is called gamma and the I and O values indicate the input value of the pixel and the output value of the non-linear function, respectively. This correction is often used to manipulate contrast in medical images, especially to highlight specific characteristics in an image with low lighting and low contrast.

3.2. Image Segmentation

Image segmentation partitions a digital image into a finite number of different regions, where region means a set of interconnected pixels. A significant number of image segmentation techniques allow the partitioning of a digital image [12,47], some of which have been considered in this project.

Images from the whole smear were taken and analyzed, as explained above, in smaller regions, called fields. In terms of pixels, all fields have the same size. Many attempts were made to choose an optimal dimension of each digital image to speed up processing—ultimately, the fields were resized to 1024 × 768 pixels, which proved to be a fair compromise.

Cell extraction was essentially based on the chromatic characteristics of cells, especially nuclei. For example, neutrophils show a blue-violet core, eosinophils show pink granules, and lymphocytes show a very large nucleus of blue color. Mean Shift filtering makes an image with color gradients and fine-grain texture flattened. In order to set up the system to recognize images of slides prepared with different techniques in the future, experiments were conducted here using grayscale images for the segmentation phase based on the Otsu algorithm. Then, morphological operations and the watershed algorithm were applied, followed by labeling, marking the different “objects” with different shades of color to facilitate subsequent classification. The Canny algorithm was considered as an alternative in rare cases when watershed provides unsatisfying results (e.g., split cells). In these cases, giving the responsibility to the user to manually adjust thresholds, segmentation showed better results than watershed. Of course, this option is considered a marginal one.

3.2.1. Mean Shift

The unsupervised learning Mean Shift algorithm is based on clustering and is also applied to digital images [48,49,50,51]. This algorithm, transforms the digital image in Figure 4a, passing through surface construction, as shown in Figure 4b, and cluster detection, in a multidimensional space, as in Figure 4c, where the points represent all pixels assigned to a specific cluster.

3.2.2. Otsu Segmentation

The Otsu method is a global threshold algorithm. The result obtained represents a binary image. To ensure optimal separation between background pixels and object pixels, and thus effective segmentation, it is necessary to maximize the inter-class variance [52].

3.2.3. Watershed

Watershed performs a digital image partitioning in different regions, especially when there are image elements that are very close to each other or even connected. The resulting image shows higher pixel intensities of each object in the center areas.

3.2.4. Canny Edge Detector

The Canny algorithm finds and recognizes the contours of objects. It takes five steps during which the grayscale input image undergoes several intermediate transformations. The result obtained represents a binary digital image with only the contours highlighted by strongly marked pixels [53,54].

3.3. Morphological Image Processing

Morphological image processing alters the structure and geometric shape of an object and applies morphological operations to that portion of the image at each kernel position [55,56]. The morphological operation used in this work is dilation. It acts mainly near the contours of the cells by adding pixels and making it thicker. Expansion reduces and eliminates possible holes inside the cells, often due to binarization defects.

4. Methods

The software we have designed executes the image processing introduced above, allowing for the identification of cellular elements and their extraction from an RGB digital image, acquired with the smartphone camera. In particular, the following steps are applied.

4.1. Increase in Brightness and Contrast

A preliminary process of image enhancement improves the quality of the original image. In particular, both brightness and contrast are increased so as to reduce or eliminate the light color halos around the cells caused by the staining process of the cytodiagnostic examination, which would otherwise have compromised the detection process in the subsequent stages. The transformation applied to each RGB channel for each pixel (x,y) in the starting image is:

$$g\left(x,y\right)=\alpha \cdot f\left(x,y\right)+\beta$$

(2)

where α and β are the so-called gain and bias, respectively, which are parameters that regulate brightness and contrast, as shown in Figure 5. They were empirically determined: α = 1.5e, β = 6.

4.2. Gamma and Mean Shift Correction

Image brightness is “gamma-corrected”, further increasing image contrast by making the color shades of the nuclei more saturated; moreover, the Mean Shift algorithm is applied to make the coloring of the cell nuclei more homogeneous, as shown in Figure 6.

4.3. Otsu Binarization

After grayscale conversion, segmentation is made with automatic threshold to separate cells, as shown in Figure 7. To improve image quality and correct any defects due to Otsu’s binarization, such as holes in cell nuclei, the process benefits from the use of aperture in combination with dilation.

4.4. Identification of Markers, Watershed

After marker identification (Euclidean Distance Transform and local maxima detection), the Watershed algorithm is applied, as shown in Figure 8. To improve the performance of the Watershed algorithm, the bandwidth h was defined by studying the range of variation of the cell size by means of a micrometer, a high precision gauge with a typical sensitivity of a hundredth of a millimeter.

4.5. Cropping

The final step of the proposed system carries on ROI detection basing on their area, in order to reduce non-cell regions that can be improperly highlighted. In fact, only regions that have an area included in a specific range (a₁, a₂) are extracted; range values depend on the image resolution. As explained above, we resized the original images to 1024 × 768 pixels and then determined the range experimentally, setting a₁ = 80 and a₂ = 250. Examples of the cropped cells after applying this operation are given in Figure 9. Figure 10 shows images of the designed app, and in Figure 11 the software pipeline related to cell extraction is reported.

5. Experimental Results

The cell extractor here described has been tested on 75 digital images representing fields, first performing a standard cell observation and manual counting for each field, and then taking into consideration the cells detected through the system proposed in this paper.

In Figure 12 a qualitative example of the working system for one of the 75 images is reported. The result in terms of the detected cells is shown with a blue bounding box around the segmented objects. The performance of this system is reported in

Table 1. Cell detecting performance.

Confusion Matrix
	True Condition
Predicted	positive	negative
Positive	1224	166
Negative	52	113

—all cells and non-cells on the 75 slides were also manually labeled by domain experts, to obtain the ground truth.

With reference to Table 1, TP represents cells correctly extracted, FN lost cells, FP non-cells improperly extracted, and TN non-cells discarded.

Starting from these assumptions, the system performances are summarized here:

Accuracy	0.860
Sensitivity (Recall)	0.959
Specificity	0.405
Precision	0.881
F-score	0.918

The measure that must mainly be taken into consideration is certainly sensitivity, which quantifies the avoidance of false negatives. The value 0.96 is satisfactory because it shows that actual positives are not overlooked, as false negatives are few. The FN detected refer, for the vast majority, to heavily-massed cells that the same experts do not consider during the observation. In fact, the manual protocol defined by the experts is tolerant of the typical presence of clusters and specifies that at least 50 fields must be taken into account, increasing them during the observation if they find the excessive presence of clusters or almost empty fields. All of this takes significant effort. In reference to this, and also with the system we have designed, the specialist can increase the number of fields to be acquired and analyzed, proving to be flexible. Even the number of false positives does not worry us because the cells and the other “objects” extracted are classified manually or through the Rhino-cyt platform, which discards the FP with great accuracy.

A final remark should be given about the execution time. Time to process a set of 50 fields manually may exceed half an hour or more. It depends largely on the expertise of the specialist and on the specific field density and cell agglomeration [57].

Time to process a single field automatically may vary depending on how dense the field is. We observed an elapsed time of 4.2 s to process the field in Figure 4, 4.1 s to process the field in Figure 5, and 2.1 s for the field in Figure 9. We estimated the average processing time on 10 slides of differing densities, obtaining 2.9 ± 1.1 s. This result was obtained with a low-end/low-cost smartphone, Xiaomi Redmi Note 7, but of course, it largely depends on the device hardware. In this phase, we really focused on demonstrating the feasibility of the proposed approach in terms of segmentation effectiveness (i.e., the extraction of the cells and getting the approval of specialists about the efficacy and usefulness of this system). Then, it is worthwhile to invest in research and the development of technologies, such as those presented in this paper, while software efficiency can be pursued, but it might not be necessary, given that higher-end smartphones are increasingly more powerful and cheaper.

6. Conclusions

The advancements in the nasal cytology field and the evolution of smartphone technology have allowed for the realization of this project. The aim of designing a system that would support the specialist during the observation phase of the slides has been reached through the development of this system, able to acquire an image from the digital microscope and to extract the cellular elements. The main advantages of this application is that the cell counting activity is faster than the manual process, together with its ease of use and the possibility of sharing images obtained from the observed fields. In fact, the cell images extracted can be sent directly to a specific server, which automatically classifies and counts them, such as the Rhino-Cyt system [23]. A possible use of this system could also be in combination with a microscope, which allows for the automatic sliding of the slide. The specialist could manage the sliding and acquire the photo, as necessary. We are now setting ourselves two main goals. The first is to pursue effective full classification on the board and the second is to integrate other diagnostic tools, such as the one just published in the literature, which aims to diagnose dyskinesia of the hair cells of the nasal mucosa [58].