# The Active Segmentation Platform for Microscopic Image Classification and Segmentation

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

- histogram based
- edge-based
- morphological operations
- region-based, such as watersheds
- deformable model fitting

- (1)
- deterministic, in order to ensure reproducibility of the results;
- (2)
- providing an uncertainty measure of the segmentation outcome;
- (3)
- able to provide some insight into how the morphology is objectively described
- (4)
- able to learn new information whenever such is presented

- numerical feature engineering method (NFE);
- Artificial Neural Networks (ANNs);
- Transport based morphometry (TBM);

## 2. Theory

#### 2.1. Geometrical Image Features

#### 2.2. Differential Invariants

#### 2.3. Curvature Invariants

#### 2.4. Structure Tensor

#### 2.5. Scale Space Theory

#### 2.6. Image Moments

#### 2.7. Random Forest Classifier

#### 2.8. Support Vector Machines Classifier

## 3. User Interaction and Architecture

#### 3.1. Architecture

#### 3.2. Operation of the Platform

#### 3.3. Metadata

## 4. Materials and Methods

#### 4.1. Data Sets

#### 4.1.1. Synthetic Data Sets

#### 4.1.2. EM ISBI Challenge Data Set

#### 4.1.3. HeLa Data Set

#### 4.1.4. HEp-2 Data Set

#### 4.2. Image Segmentation Task

#### 4.3. Segmentation Metrics

#### 4.4. Image Classification Task

## 5. Results

#### 5.1. Image Segmentation

#### 5.1.1. Experiment 1: Batch Segmentation and Comparison

#### 5.1.2. Experiment 2: Three Class Segmentation

#### 5.2. Image Classification

#### 5.2.1. Case Study 1: Synthetic Data Set Classification

#### 5.2.2. Case Study 2: HeLa Cells Classification

#### 5.2.3. Case Study 3: HEp2 Cells Classification

#### 5.3. Influence of the Classification Method

## 6. Discussion

#### 6.1. Platform Comparison

#### 6.2. The Problem of Ground Truth

#### 6.3. The Problem of Feature Selection

#### 6.4. Image Classification Aspects

#### 6.5. Outlook

#### 6.6. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Appendix A. Weak Differentiation

## Appendix B. Orthogonal Laplacian Decomposition

## References

- Meijering, E. Cell Segmentation: 50 Years Down the Road [Life Sciences]. IEEE Signal Process. Mag.
**2012**, 29, 140–145. [Google Scholar] [CrossRef] - Vicar, T.; Balvan, J.; Jaros, J.; Jug, F.; Kolar, R.; Masarik, M.; Gumulec, J. Cell segmentation methods for label-free contrast microscopy: Review and comprehensive comparison. BMC Bioinform.
**2019**, 20, 360. [Google Scholar] [CrossRef] - Vu, Q.D.; Graham, S.; Kurc, T.; To, M.N.N.; Shaban, M.; Qaiser, T.; Koohbanani, N.A.; Khurram, S.A.; Kalpathy-Cramer, J.; Zhao, T.; et al. Methods for Segmentation and Classification of Digital Microscopy Tissue Images. Front. Bioeng. Biotechnol.
**2019**, 7, 53. [Google Scholar] [CrossRef] [PubMed] - Dumur, T.; Duncan, S.; Graumann, K.; Desset, S.; Randall, R.S.; Scheid, O.M.; Prodanov, D.; Tatout, C.; Baroux, C. Probing the 3D architecture of the plant nucleus with microscopy approaches: Challenges and solutions. Nucleus
**2019**, 10, 181–212. [Google Scholar] [CrossRef] [PubMed][Green Version] - Prodanov, D.; Verstreken, K. Molecular Imaging. In Chapter Automated Segmentation and Morphometry of Cell and Tissue Structures. Selected Algorithms in ImageJ; InTech: London, UK, 2012; pp. 183–208. [Google Scholar]
- Poulet, A.; Arganda-Carreras, I.; Legland, D.; Probst, A.V.; Andrey, P.; Tatout, C. NucleusJ: An ImageJ plugin for quantifying 3D images of interphase nuclei. Bioinformatics
**2014**, 31, 1144–1146. [Google Scholar] [CrossRef] [PubMed][Green Version] - Arganda-Carreras, I.; Kaynig, V.; Rueden, C.; Eliceiri, K.W.; Schindelin, J.; Cardona, A.; Seung, S.H. Trainable Weka Segmentation: A machine learning tool for microscopy pixel classification. Bioinformatics
**2017**, 33, 2424–2426. [Google Scholar] [CrossRef] [PubMed] - Berg, S.; Kutra, D.; Kroeger, T.; Straehle, C.N.; Kausler, B.X.; Haubold, C.; Schiegg, M.; Ales, J.; Beier, T.; Rudy, M.; et al. ilastik: Interactive machine learning for (bio)image analysis. Nat. Methods
**2019**, 16, 1226–1232. [Google Scholar] [CrossRef] [PubMed] - De Chaumont, F.; Dallongeville, S.; Chenouard, N.; Hervé, N.; Pop, S.; Provoost, T.; Meas-Yedid, V.; Pankajakshan, P.; Lecomte, T.; Montagner, Y.L.; et al. Icy: An open bioimage informatics platform for extended reproducible research. Nat. Methods
**2012**, 9, 690–696. [Google Scholar] [CrossRef] - Lamprecht, M.R.; Sabatini, D.M.; Carpenter, A.E. CellProfiler™: Free, versatile software for automated biological image analysis. BioTechniques
**2007**, 42, 71–75. [Google Scholar] [CrossRef][Green Version] - Bankhead, P.; Loughrey, M.B.; Fernández, J.A.; Dombrowski, Y.; McArt, D.G.; Dunne, P.D.; McQuaid, S.; Gray, R.T.; Murray, L.J.; Coleman, H.G.; et al. QuPath: Open source software for digital pathology image analysis. Sci. Rep.
**2017**, 7, 16878. [Google Scholar] [CrossRef][Green Version] - Haralick, R.M.; Shanmugam, K.; Dinstein, I. Textural Features for Image Classification. IEEE Trans. Syst. Man Cybern.
**1973**, SMC-3, 610–621. [Google Scholar] [CrossRef][Green Version] - Bianchetti, G.; Ciccarone, F.; Ciriolo, M.R.; Spirito, M.D.; Pani, G.; Maulucci, G. Label-free metabolic clustering through unsupervised pixel classification of multiparametric fluorescent images. Anal. Chim. Acta
**2021**, 1148, 238173. [Google Scholar] [CrossRef] [PubMed] - Shifat-E-Rabbi, M.; Yin, X.; Fitzgerald, C.; Rohde, G. Cell Image Classification: A Comparative Overview. Cytom. Part A
**2020**, 97, 347–362. [Google Scholar] [CrossRef] [PubMed][Green Version] - Rueden, C.T.; Schindelin, J.; Hiner, M.C.; DeZonia, B.E.; Walter, A.E.; Arena, E.T.; Eliceiri, K.W. ImageJ2: ImageJ for the next generation of scientific image data. BMC Bioinform.
**2017**, 18, 529. [Google Scholar] [CrossRef] [PubMed] - Abramoff, M.; Magalhaes, P.; Ram, S. Image Processing with ImageJ. Biophotonics Int.
**2004**, 11, 36–42. [Google Scholar] - Schneider, C.A.; Rasband, W.S.; Eliceiri, K.W. NIH Image to ImageJ: 25 years of image analysis. Nat. Methods
**2012**, 9, 671–675. [Google Scholar] [CrossRef] - Schindelin, J.; Arganda-Carreras, I.; Frise, E.; Kaynig, V.; Longair, M.; Pietzsch, T.; Preibisch, S.; Rueden, C.; Saalfeld, S.; Schmid, B.; et al. Fiji: An open-source platform for biological-image analysis. Nat. Methods
**2012**, 9, 676–682. [Google Scholar] [CrossRef][Green Version] - Florack, L.; Romeny, B.T.H.; Viergever, M.; Koenderink, J. The Gaussian scale-space paradigm and the multiscale local jet. Int. J. Comput. Vis.
**1996**, 18, 61–75. [Google Scholar] [CrossRef] - Larsen, A.B.L.; Darkner, S.; Dahl, A.L.; Pedersen, K.S. Jet-Based Local Image Descriptors. In Proceedings of the European Conference on Computer Vision—ECCV 2012, Florence, Italy, 7–13 October 2012; Springer: Berlin/Heidelberg, Germany, 2012; pp. 638–650. [Google Scholar] [CrossRef][Green Version]
- Prodanov, D.; Konopczynski, T.; Trojnar, M. Selected Applications of Scale Spaces in Microscopic Image Analysis. Cybern. Inf. Technol.
**2015**, 15, 5–12. [Google Scholar] [CrossRef][Green Version] - Brox, T.; Weickert, J.; Burgeth, B.; Mrazek, P. Nonlinear Structure Tensors; Technical Report; Universitat des Saarlandes: Saarbrücken, Germany, 2004. [Google Scholar]
- Iijima, T. Theory of pattern recognition. In Electronics and Communications in Japan; Wiley: Hoboken, NJ, USA, 1963; pp. 123–134. [Google Scholar]
- Marr, D.; Hildreth, E. Theory of edge detection. Proc. R. Soc. Lond. B
**1980**, 207, 187–217. [Google Scholar] [PubMed] - Witkin, A. Scale-space Fltering. In Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI 83), Karlsruhe, Germany, 8–12 August 1983; Volume 2, pp. 1019–1022. [Google Scholar]
- Koenderink, J.J. The structure of images. Biol. Cybern.
**1984**, 50, 363–370. [Google Scholar] [CrossRef] - Lindeberg, T. Scale-space. In Wiley Encyclopedia of Computer Science and Engineering; Wiley: Hoboken, NJ, USA, 2007; pp. 2495–2504. [Google Scholar]
- Alvarez, L.; Guichard, F.; Lions, P.L.; Morel, J.M. Axioms and fundamental equations of image processing. Arch. Ration. Mech. Anal.
**1993**, 123, 199–257. [Google Scholar] [CrossRef] - Weickert, J. Anisotropic Diffusion in Image Processing; ECMI Series; Teubner-Verlag: Stuttgart, Germany, 1998. [Google Scholar]
- Duits, R.; Felsberg, M.; Florack, L.; Platel, B. Alpha-scale spaces on a bounded domain. In Scale Space Methods in Computer Vision; Springer: Berlin/Heidelberg, Germany, 2003; pp. 494–510. [Google Scholar]
- Shakibaei, B.H.; Paramesran, R. Recursive formula to compute Zernike radial polynomials. Opt. Lett.
**2013**, 38, 2487. [Google Scholar] [CrossRef] [PubMed][Green Version] - Ho, T.K. Random Decision Forests. In Proceedings of the 3rd International Conference on Document Analysis and Recognition, Montreal, QC, Canada, 14–16 August 1995; pp. 278–282. [Google Scholar]
- Cortes, C.; Vapnik, V.N. Support vector networks. Mach. Learn.
**1995**, 20, 273–297. [Google Scholar] [CrossRef] - Platt, J.C. Sequential Minimal Optimization: A Fast Algorithm for Training Support Vector Machines. In Technical Report; Advances in Kernel Methods—Support Vector Learning; MIT Press: Cambridge, MA, USA, 1998. [Google Scholar]
- Fan, R.E.; Chang, K.W.; Hsieh, C.J.; Wang, X.R.; Lin, C.J. LIBLINEAR: A Library for Large Linear Classification. J. Mach. Learn. Res.
**2008**, 9, 1871–1874. [Google Scholar] - Chang, C.C.; Lin, C.J. LibSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol.
**2011**, 2, 1–27. [Google Scholar] [CrossRef] - Florack, L.M.J.; ter Haar Romeny, B.M.; Koenderink, J.J.; Viergever, M.A. Scale and the differential structure of images. Image Vis. Comput.
**1992**, 10, 376–388. [Google Scholar] [CrossRef] - Cardona, A.; Saalfeld, S.; Preibisch, S.; Schmid, B.; Cheng, A.; Pulokas, J.; Tomancak, P.; Hartenstein, V. An Integrated Micro- and Macroarchitectural Analysis of the Drosophila Brain by Computer-Assisted Serial Section Electron Microscopy. PLoS Biol.
**2010**, 8, e1000502. [Google Scholar] [CrossRef][Green Version] - Jain, V.; Bollmann, B.; Richardson, M.; Berger, D.R.; Helmstaedter, M.N.; Briggman, K.L.; Denk, W.; Bowden, J.B.; Mendenhall, J.M.; Abraham, W.C.; et al. Boundary Learning by Optimization with Topological Constraints. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 13–18 June 2010; IEEE: Piscataway, NJ, USA, 2010. [Google Scholar] [CrossRef]
- Murphy, R.; Boland, M.; Velliste, M. Towards a systematics for protein subcelluar location: Quantitative description of protein localization patterns and automated analysis of fluorescence microscope images. Proc. Int. Conf. Intell. Syst. Mol Biol.
**2000**, 8, 251–259. [Google Scholar] - Boland, M.V.; Murphy, R.F. A neural network classifier capable of recognizing the patterns of all major subcellular structures in fluorescence microscope images of HeLa cells. Bioinformatics
**2001**, 17, 1213–1223. [Google Scholar] [CrossRef][Green Version] - Hobson, P.; Lovell, B.C.; Percannella, G.; Vento, M.; Wiliem, A. Classifying Anti-nuclear Antibodies HEp-2 Images: A Benchmarking Platform. In Proceedings of the 2014 22nd International Conference on Pattern Recognition, Stockholm, Sweden, 24–28 August 2014; pp. 3233–3238. [Google Scholar] [CrossRef]
- Qi, X.; Zhao, G.; Chen, J.; Pietikäinen, M. Exploring illumination robust descriptors for human epithelial type 2 cell classification. Pattern Recognit.
**2016**, 60, 420–429. [Google Scholar] [CrossRef] - Hall, M.A. Correlation-Based Feature Selection for Discreate and Numeric Class Machine Learning; University of Waikato: Hamilton, New Zealand, 1999. [Google Scholar]
- Geurts, P.; Irrthum, A.; Wehenkel, L. Supervised learning with decision tree-based methods in computational and systems biology. Mol. BioSyst.
**2009**, 5, 1593. [Google Scholar] [CrossRef] - Arganda-Carreras, I.; Turaga, S.C.; Berger, D.R.; Cireşan, D.; Giusti, A.; Gambardella, L.M.; Schmidhuber, J.; Laptev, D.; Dwivedi, S.; Buhmann, J.M.; et al. Crowdsourcing the creation of image segmentation algorithms for connectomics. Front. Neuroanat.
**2015**, 9, 142. [Google Scholar] [CrossRef] [PubMed] - Stringer, C.; Wang, T.; Michaelos, M.; Pachitariu, M. Cellpose: A generalist algorithm for cellular segmentation. Nat. Methods
**2020**, 18, 100–106. [Google Scholar] [CrossRef] [PubMed] - Schmidt, U.; Weigert, M.; Broaddus, C.; Myers, G. Cell Detection with Star-Convex Polygons. In Medical Image Computing and Computer Assisted Intervention MICCAI 2018; Springer International Publishing: Cham, Switzerland, 2018; pp. 265–273. [Google Scholar] [CrossRef][Green Version]

**Figure 4.**Circles/Triangles classification. (

**A**) An image generated with variable size triangles and circles on black background; (

**B**) Individual objects are classified as triangles (green) and circles (blue) on noiseless background; (

**C**) Individual objects are classified as triangles (green) and circles (blue) on noisy background.

**Figure 5.**A representative image of each protein class and the average number of correctly classified instances in 10 cross validation (

**A**) filamentous actin labeled with rhodamine-phalloidin (Actin); (

**B**) DNA labeled with DAPI (DNA); (

**C**) endoplasmic reticulum protein (ER); (

**D**) transferrin receptor (endosome); (

**E**) Golgi protein giantin (Golgia); (

**F**) Golgi protein GPP130 (Golgpp); (

**G**) lysosomal protein LAMP2 (lysosomes); (

**H**) microtubules; (

**I**) mitochondrial protein (mitochondria); (

**J**) nucleolar protein nucleolin (nucleolus).

**Figure 6.**A representative image of each cell class and the average number of correctly classified instances in 10 cross validation. (

**A**) Centromere (CT); (

**B**) Golgi apparatus (GG); (

**C**) Homogeneous (HM); (

**D**) Nucleolar (NCL); (

**E**) Anc nuclear membrane (ANM); (

**F**) Speckled (SP).

**Figure 7.**Overview of the Image Classification Workflow. (

**A**) A sample image of filamentous actin labeled with rhodamine-phalloidin (Actin) from the HeLa data set; (

**B**) Representative images of ALoG, LoG and BoG at scale 4; (

**C**) Compound image transforms on original and scaled images, Feature vector is computed; (

**D**) CFS based feature selection and SVM’s are used for training the model; (

**E**) Several evaluation metrics are computed e.g., ROC curve, confusion matrix etc.

**Figure 8.**Characteristic features compared to the segmentation outcome. (

**A**) Initial expert annotation. (

**B**) Outcome of the segmentation; (

**C**) Gradient amplitude, $r=8$; (

**D**) Largest Hessian eigenvalue, $r=8$; (

**E**) Normal component ${\Delta}_{\perp}$ of the LoG, $r=8$; (

**F**) Mean curvature, $r=8$. The scale bar represents 0.5 $\mathsf{\mu}$m.

**Figure 9.**Characteristic features compared to the segmentation outcome. (

**A**) RF classifier. (

**B**) SMO classifier.

First Order Invariants | |
---|---|

Gradient amplitude | $A=\sqrt{{G}_{x}^{2}+{G}_{y}^{2}}$ |

Gradient orientation | $sin\varphi ={G}_{y}/\sqrt{{G}_{x}^{2}+{G}_{y}^{2}}$ |

$cos\varphi ={G}_{x}/\sqrt{{G}_{x}^{2}+{G}_{y}^{2}}$ | |

Second order invariants | |

Laplacian | ${\Delta}_{G}=\mathrm{Tr}\phantom{\rule{0.166667em}{0ex}}{\mathbb{H}}_{G}={G}_{xx}+{G}_{yy}$ |

Determinant of the Hessian | $\mathrm{det}\phantom{\rule{0.166667em}{0ex}}{\mathbb{H}}_{G}={G}_{xx}{G}_{yy}-{G}_{xy}^{2}$ |

Curvature Invariant | Formula |
---|---|

Line curvature | $\left(\right)open="("\; close=")">{G}_{xx}{G}_{y}^{2}-2{G}_{x}{G}_{y}{G}_{xy}+{G}_{x}^{2}{G}_{yy}$ |

Mean curvature | $1/2\left(\right)open="("\; close=")">\left(\right)open="("\; close=")">1+{G}_{x}^{2}{G}_{yy}-2{G}_{x}{G}_{y}{G}_{xy}+\left(\right)open="("\; close=")">1+{G}_{y}^{2}$ |

Gaussian curvature | $\left(\right)open="("\; close=")">{G}_{xx}{G}_{yy}-{G}_{xy}^{2}$ |

Filter | Functionality | Feature Order |
---|---|---|

Gauss2D | Gaussian smoothing, Equation (4) | 0 |

Gradient | Gradient amplitude and orientation | 1 |

Gaussian Structure | Structure tensor, Equation (2) | 1 |

LoG | Laplacian of Gaussian (LoG) | 2 |

ALoG | Anisotropic decomposition of LoG, Equations (A2) and (A3) | 2 |

Gradient amplitude and orientation | 1 | |

Hessian | Eigenvalues of the Hessian | 2 |

Determinant of the Hessian | 2 | |

Curvature 2D | Line curvature + Hessian determinant | 2 |

Curvature 3D | Mean + Gauss curvature of surfaces | 2 |

BoG | Bi-Laplacian of Gaussian, Equation (5) | 4 |

Gaussian Jet | Gaussian Jet of order n | n |

LoGN | n-th order PoL, Equation (5) | 2 n |

Regional Feature | Functionality |
---|---|

Legendre | Legendre moments |

Zernike | Zernike moments |

ImageJ | ImageJ statistics |

Haralick | Haralick features [12] |

Data Set | Filters | Min Scale | Max Scale |
---|---|---|---|

HeLa | LoG, ALoG, HoG, Structure | 2 | 8 |

HEp-2 | LoG, BoG, ALoG, HoG, Structure | 2 | 8 |

Regional Feature | Parameters |
---|---|

Legendre | order (0–6) |

Zernike | order (0–6) |

ImageJ | – |

Haralick | distance (1–3), directions (90${}^{\circ}$, 180${}^{\circ}$, 270${}^{\circ}$, 360${}^{\circ}$) |

Metric | Training Set | Test Set | ||
---|---|---|---|---|

AS/IJ | Ilastik | AS/IJ | Ilastik | |

${V}^{rand}$ | 1.0 | 1.0 | 0.87 | 0.91 |

${V}^{info}$ | 1.0 | 1.0 | 0.93 | 0.94 |

TP Rate | FP Rate | Precision | Recall | ${\mathit{F}}_{1}$ | ROC Area | PRC Area | Class |
---|---|---|---|---|---|---|---|

0.954 | 0.081 | 0.942 | 0.954 | 0.948 | 0.943 | 0.932 | cells |

0.722 | 0.020 | 0.808 | 0.722 | 0.763 | 0.935 | 0.667 | boundaries |

0.909 | 0.048 | 0.897 | 0.909 | 0.903 | 0.934 | 0.847 | nuclei |

**Table 9.**Comparison of mean average true positive rate (TP), false positive rate (FP), precision, recall, ${F}_{1}$ score, ROC area and PR area of different classification approaches in different data sets.

Data Set | TP | FP | Precision | Recall | ${\mathit{F}}_{1}$ | ROC Area | PRC Area |
---|---|---|---|---|---|---|---|

Regional Features | |||||||

Hela | 0.82 | 0.02 | 0.82 | 0.81 | 0.81 | 0.96 | 0.75 |

HEp-2 | 0.76 | 0.06 | 0.77 | 0.76 | 0.76 | 0.90 | 0.68 |

Regional features + Scale space pixel-features | |||||||

Hela | 0.92 | 0.01 | 0.92 | 0.92 | 0.92 | 0.98 | 0.88 |

HEp-2 | – | – | – | – | – | – | – |

Regional features + Scale space pixel-features+ discriminative features selection | |||||||

Hela | 0.93 | 0.01 | 0.93 | 0.93 | 0.93 | 0.99 | 0.90 |

HEp-2 | 0.88 | 0.03 | 0.88 | 0.88 | 0.88 | 0.95 | 0.83 |

Classifier | TP | FP | Precision | Recall | ${\mathit{F}}_{1}$ | ROC Area | PRC Area |
---|---|---|---|---|---|---|---|

SMO | 0.806 | 0.021 | 0.805 | 0.806 | 0.804 | 0.955 | 0.729 |

RF | 0.717 | 0.031 | 0.712 | 0.717 | 0.713 | 0.950 | 0.758 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Vohra, S.K.; Prodanov, D.
The Active Segmentation Platform for Microscopic Image Classification and Segmentation. *Brain Sci.* **2021**, *11*, 1645.
https://doi.org/10.3390/brainsci11121645

**AMA Style**

Vohra SK, Prodanov D.
The Active Segmentation Platform for Microscopic Image Classification and Segmentation. *Brain Sciences*. 2021; 11(12):1645.
https://doi.org/10.3390/brainsci11121645

**Chicago/Turabian Style**

Vohra, Sumit K., and Dimiter Prodanov.
2021. "The Active Segmentation Platform for Microscopic Image Classification and Segmentation" *Brain Sciences* 11, no. 12: 1645.
https://doi.org/10.3390/brainsci11121645