Spatiotemporal Identification of Cell Divisions Using Symmetry Properties in Time-Lapse Phase Contrast Microscopy

Abstract

A variety of biological and pharmaceutical studies, such as for anti-cancer drugs, require the quantification of cell responses over long periods of time. This is performed with time-lapse video microscopy that gives a long sequence of frames. For this purpose, phase contrast imaging is commonly used since it is minimally invasive. The cell responses of interest in this study are the mitotic cell divisions. Their manual measurements are tedious, subjective, and restrictive. This study introduces an automated method for these measurements. The method starts with preprocessing for restoration and reconstruction of the phase contrast time-lapse sequences. The data are first restored from intensity non-uniformities. Subsequently, the circular symmetry of the contour of the mitotic cells in phase contrast images is used by applying a Circle Hough Transform (CHT) to reconstruct the entire cells. The CHT is also enhanced with the ability to “vote” exclusively towards the center of curvature. The CHT image sequence is then registered for misplacements between successive frames. The sequence is subsequently processed to detect cell centroids in individual frames and use them as starting points to form spatiotemporal trajectories of cells along the positive as well as along the negative time directions, that is, anti-causally. The connectivities of different trajectories enhanced by the symmetry of the trajectories of the daughter cells provide as topological by-products the events of cell divisions together with the corresponding entries into mitoses as well as exits from cytokineses. The experiments use several experimental video sequences from three different cell lines with many cells undergoing mitoses and divisions. The quantitative validations of the results of the processing demonstrate the high performance and efficiency of the method.

1. Introduction

Many scientific biological applications as well as preclinical pharmaceutical studies for drug development including for cancer treatment require the investigation of cell responses to a variety of stimuli. Some of these responses are cell migration, cell proliferation, as well as cell differentiation. This study, in particular, is about the detection of the spatiotemporal events of mitoses and of the accompanying cell divisions and cytokineses. It requires the observation of these events in samples for long periods of time to be able to draw the necessary experimental conclusions. An effective way to achieve this is with phase contrast microscopy.

1.1. Background on Phase Contrast Imaging for Cell Division

Phase contrast microscopy is the most widely used modality for long term imaging of cellular processes such as cell migration and cell division. Phase Contrast (PC) is a transmitted light based technique in which phase shifts created by the cells are translated into amplitude changes. Unlike fluorescence based methodologies, in which the cell is illuminated with very high intensity light to excite the fluorophores, phase contrast relies on the use of low intensity illumination. Thus, the serious issues of phototoxicity and photodamage elicited under fluorescence excitation are eliminated making phase contrast a far less invasive and gentle method for long term cellular imaging. In addition, phase contrast does not require labeling of cells and thus any cell type can easily be imaged while fluorescence based methodologies require the use of cells which are constitutively expressing fluorescent proteins, an important limitation. The above coupled with the higher cost of fluorescence microscopy systems as well as the higher cost of alternative transmitted light modalities such as Differential Interference Contrast (DIC) have established phase contrast the method of choice for live cell imaging.

The division of a cell starts with the division of the nucleus, namely, mitosis. It completes with the cleavage (division) of the cytoplasm into two daughter cells, namely, with cytokinesis. A cell rounds up starting with mitosis. In phase contrast imaging this results in a halo effect of increased contrast at the cell boundary. The contrast of the halos at the boundaries of the daughter cells remains high until cytokinesis completes.

The acquired phase contrast data sets of cell divisions are large and their manual analysis is tedious, subjective, and restrictive. Thus, this study proposes an automated method for their analysis. The method requires the reconstruction and restoration of phase contrast images of cells and the processing of large data sets of time-lapse movies. The processing is complicated by several factors. The first factor are the imaging artifacts. Other complicating factors are the varying sizes of the cells in different phases of mitosis and cytokinesis as well as the high cell density and the varying shapes of the surrounding cells. A complicating factor is also the motions of the cells that even result in some cells entering or leaving the field or plane of view.

1.2. Literature Review

An acquired image sequence is first preprocessed to remove imaging artifacts. This involves noise removal and restoration for intensity non-uniformity [1]. It also involves preprocessing for cell reconstruction [2,3,4,5]. This is followed by the analysis of the data to detect mitoses and cell divisions. The cell division detection methods can be classified into tracking-free, tracking-based, hybrid methods, as well as appearance-based with machine learning and neural networks [6,7]. The tracking-free methods involve the segmentation and detection of mitotic cells or of cell division events in individual frames. The tracking-based ones may still involve the segmentation or detection of cells from different frames of a sequence independently, for example from the first frame. However, tracking-based methods also involve a subsequent tracking of the cells throughout a sequence by updating the cells positions and detecting cellular cycle events.

The detection and even segmentation of cells in phase contrast microscopy has been done using a variety of low level features. It has been performed with morphological operations either directly [8,9,10,11] or in combination with texture features [12]. Other features that have been used are the intensity histogram [13] and gradient features. More involved features have been the Histogram of Oriented Gradients (HOG), GIST [14,15], and the Scale-Invariant Feature Transform (SIFT) [16]. More global shape characterizations and segmentation methods have been used. These include watersheds [17], symmetry [18], template matching [19], and statistical decomposition [20,21]. The segmentation has been performed with explicit modeling of the halo effect, which surrounds cells imaged with phase contrast microscopy, with active contours [9,11]. In some cases segmentation is semi-automatic [9,22]. Many of these techniques are tracking-free, independent of spatiotemporal trajectories, and hence allow causal online tracking [13].

The results of cell segmentation have been used as candidates to initialize automatic tracking of the motion of the cells in an image sequence to detect mitoses. The detection has been performed in a variety of ways. An approach estimates the cell division rate globally and implicitly from an image sequence based on cell area coverage without the explicit detection of individual cells or mitotic events [23,24]. Another method uses mean shift directly on an image sequence for cell event detection or motion tracking performed in forward or in reverse time direction and hence motion tracking may not be causal [8]. Spatial segmentation has also been addressed with active contours propagated in time to follow motion with linear prediction based on Kalman filtering [13,24,25,26,27,28]. Another representation for cell division has been with linear programming [22] or linear programming combined with level sets [29]. Some tracking-based approaches detect the lineage of each cell [30].

An alternative representation of tracking-based methods has been with global data associations between frames in a sequence. These have been established based on features of the detected or segmented cellular regions. Some methods in this context perform cell detection with the circle Hough transform for isolated cells [31] or its extension, the local circle Hough transform, as medial axis transform, to deal with cell detection in dense samples [1]. The temporal associations have been followed to form spatiotemporal trajectory segments that may then be linked together [1,32,33]. Such methods result in implicit detections of mitoses as a topological by-product [1,33]. Some of these tracking-based methods segment and follow cells using level sets-based active contours methods; however, they are sensitive to dense samples and have only been able to detect isolated dividing cells [25,31].

The distinction between the spatial and the temporal dimension is diminished using global flow networks [30,34]. In an approach resembling stochastic signal processing, a graph has been used for the detection of the sequence of the different phases of mitosis and of cell division explicitly with Markov Models (MM) and Hidden Markov Models (HMM) [35,36]. More elaborate Conditional Random Fields (CRFs) classifiers have also been used for mitosis detection [37,38]. Further elaborations of these types of classifiers that have been used are the Event Detection Conditional Random Field (EDCRF) that is able to not only detect mitosis events, but also to detect the completion of mitosis and the birth of the daughter cells [39]. Another elaboration has been the Multi-Grained Random Fields (MGRF) [40]. In addition, Max-Margin Hidden Conditional Random Field (MM-HCRF) with Max-Margin Semi-Markov Model (MM-SMM) classifiers have been used [41]. These classifiers require extensive training. They also involve at least a time-delay for the processing of a sequence. Stochastic Markov models have also been used at nano scale [42].

The methods that have been described thus far are geometric, graphical, or statistical. Beyond these methods that attempt to model the form of the cells and quantify the associated remaining error for the cellular events [31], many other techniques develop classifiers directly from the data. Some are based on machine learning methods and others are based on neural networks. In the context of machine learning, mitoses have been detected with the AdaBoost ensemble classifier [28]. As an extension to graph-based methods, graph neural networks have also been used [43]. Convolutional neural networks in 2D (2D CNN) provide the detection of mitoses in individual frames; however, the method is likely to lead to either repeated or missed mitoses detections depending on the temporal sampling points [44]. Features from a 3D Convolutional Neural Network (3D CNN) have been combined with a Support Vector Machine (SVM) classifier from the context of machine learning to detect the presence of a single mitosis, however without the ability to temporally resolve it [45]. A neural architecture that combines Fully Convolutional Networks (FCN) and 3D CNNs, namely, the F3D-CNN has also been developed [46]. The first step, FCN, detects areas with candidate divisions. A sequence of these candidates is formed, since a cell division can span multiple frames, and is then provided to a 3D CNN for cell division detection.

To enhance the temporal resolution of the detection, a 2D-U-Net [47] has been used to extract candidates in space that are then stacked and filtered in time with a Bidirectional Long Short-Term Memory (BLSTM) network to reject false positives [48]. To temporally resolve a mitosis in a spatiotemporal sequence, convolutional neural networks have been combined with a Two-Stream Bidirectional Long Short-Term Memory (TS-BLSTM) network [49,50,51]. In that method, candidate regions are first extracted and then processed with the TS-BLSTM using appearance and motion information to jointly detect the presence of a mitosis as well as to temporally localize it. In a similar vein, to improve temporal mitosis resolution, patch sequence candidates are extracted that are then processed by a convolutional neural network in space and subsequently with an LSTM (CNN-LSTM) along time. The combined results are used to build a 3D probability heat map that is processed to locate mitotic events. Its training is end-to-end and enables the joint detection of mitotic event in space and time [52,53]. These methods based on neural networks can only detect a single mitosis in a candidate spatiotemporal patch, which is a limitation particularly for dense samples. To address this limitation a method based on 3D CNN, namely, V-Net [54], computes a mitosis likelihood throughout a spatiotemporal sequence even for dense samples [55,56]. In another work, combining processing in space and time, a deep reinforcement learning-based Progressive Sequence Saliency Discovery (PSSD) network is used to train a Markov Decision Process (MDP) to discover salient frames and temporally locate mitoses [57].

Machine learning-based methods and neural networks-based methods have several characteristics in common. They both require extensive training that is computationally intensive. Machine learning-based methods are sometimes preferred, because they are often more efficient computationally [28]. Techniques belonging to both of these classes have been shown to be effective for samples from only a single cell line. This has been demonstrated particularly from the benchmark that has been developed for validation that only consists of myoblast cells and involves extensive manual annotations [7]. The neural networks detections are also sensitive to dense samples and do not provide explicit spatiotemporal cell information for a complete mitotic event cycle that leads to a cell division and the associated cytokinesis. This limits the applicability and sufficiency of neural networks based detections for biological studies of this phenomenon.

1.3. Overview of the Proposed Method and Contributions

An early version of the work on this method has been extended, made more robust, and validated in this study [1]. The proposed method starts with extensive preprocessing for restoration and reconstruction. It involves the detection of candidate mitotic cells or daughter cells centroids in individual frames that are tracked for cell displacements between frames. This is followed by the detection of candidate cell divisions in individual frames that are tested to potentially establish a cell division event. The detection and tracking of mitosis and of the accompanying cell division is the eventual objective of the proposed method.

The acquired video sequences are first preprocessed to correct for intensity non-uniformity artifacts due to non-uniform illumination. Subsequently, the halo effect surrounding mitotic cells is detected with Hessian-based image filtering. The principal components of the pixelwise Hessians and their eccentricity are used to perform a circle Hough transform that takes advantage of the circular symmetry of the cells and has several novelties as well [58,59,60]. The main novelty is that it identifies the pixel-based direction towards the center of curvature that is used to exclusively “vote” for the interior circle center using the expected radius of the cells. This makes the transform robust to a dense cell clutter in a sample. The emphasis of the circle Hough transform, that is also a novelty of the method, is to be particularly sensitive to the detection of the daughter cells that remain circular and with a high contrast for a short period of time during cytokinesis. The circle Hough transform images are then corrected for possible spatial misregistrations due to shifts among consecutive frames of a sequence.

A direct processing of the circle Hough transform images provides several cell centroids. The registered centroids are the starting points for cells trajectories extraction. This is based on pixel associations between pixels locations of successive frames in a sequence. The pixel associations are established and the cell trajectories are subsequently tracked in both the positive as well as in the negative time direction. That is, the trajectory extraction is also anti-causal. Candidates for cell division events are identified as a topological by-product of joining an already extracted trajectory of a parent cell together with one of its daughter cell with the trajectory corresponding to the second daughter cell. The trajectory of the second daughter cell is detected later along the progression of the method. The segment of the new trajectory joining the pre-existing one is extracted anti-causally. The candidate cell division events are further tested to establish the division. The method takes advantage of the symmetry between the daughter cells in cytokinesis and even of the symmetry of the cell cycle. The implementation of the proposed method emphasizes efficiency. The validation used fourteen time-lapse phase contrast movie sequences from real biological experiments of HeLa, Kyoto, and Fibonectin cell lines undergoing mitoses over long periods of time. The outputs of the method for these sequences were validated quantitatively that demonstrated the high performance and robustness of the method.

2. Data and Methods

The phase contrast cellular image sequences are processed in a pipeline. The first step is the restoration of the intensity uniformity along time. This is followed by intensity uniformity restoration in space. The images are then transformed with an extension of the circle Hough transform. Both the original and the circle Hough transform image sequences are used in subsequent processing steps. These are the extraction of cell trajectories as well as the detection of cell divisions. A summary of the preprocessing steps is given in Figure 1 and a summary of the steps for cell division detection is given in Figure 7.

2.1. Data

Each image sequence corresponds to a different cell line. The sequences are from a total of three different cell lines, namely, of HeLa, Kyoto, and Fibronectin cells (the HeLa cell line was a kind gift from Dr. Katerina Strati (University of Cyprus), which acquired it from ATCC. The HeLa Kyoto cell line was kindly gifted to our lab by Dr. Niovi Santama (University of Cyprus). Regarding the Fibronectin null fibroblast cell line, it was a kind gift from Dr. Reinhard Fässler (Max Planck Institute of Biochemistry)). The cell divisions for the third type of sequences of Fibronectin cells in

Table 1. Description of the acquisitions of the time-lapse sequences.

Property\Seq.	Sequcences 1	Sequcences 2	Sequcences 3
Cell type	HeLa	Kyoto	Fibronectin null fibroblast
Inhibitor synchronized	No	No	RO3306 10 mM
Number of movies	5	4	5
Inter-frame time	3 min	3 min	5 min
Average number of frames	54	90	60
Average of total time	2 h 42 min	4 h 30 min	5 h
Magnification microscope	First three movies: Lens $20\times$ (200×) Last two movies: Lens $10\times$ (100×)	Lens $10\times$ (100×)	Lens $10\times$ (100×)
Image size (pixels)	$1300\times 1030$	$1300\times 1030$	$1300\times 1030$

were synchronized using the cell division inhibitor RO3306. The imaging was performed using a Carl Zeiss Axiovert 200M inverted microscope with a motorized stage, which was time sharing among several wells of different samples. Plan-Apochromatic lenses of magnification $10\times$ and $20\times$ were used, which gave total image magnification factors together with those of the ocular lens of $100\times$ and $200\times$, respectively. The microscope was used for positive phase contrast microscopy. It was also linked to a Carl Zeiss AxioCam HRc digital camera. The resolution of the CCD and hence the size of the images is $1300\times 1030 pixels$ or approximately $1.3 megapixels$. Images were collected at intervals of 3–5 min for several hours to provide a time-lapse movie sequence.

There are five sequences of the first cell line. They include two sequences that are from different positions (Fields of View (FOVs)) within the same sample that were imaged simultaneously during the experiment. Two more sequences were from different positions within another sample and were imaged simultaneously during the experiment. The last sequence was from a third sample. There are four sequences of the second cell line. They include two different positions within the same sample that were imaged simultaneously during the experiment. The other two sequences were acquired immediately afterwards from the same sample and they again correspond to different positions that were imaged simultaneously during the experiment. There are five sequences of the third cell line. All five of them show different positions within the same sample that were imaged simultaneously during the experiment.

The acquisition resulted in a total of $\tau$ time-lapse frames, $t=0,\dots \tau -1$. A phase-contrast image sequence is represented by $I_{PC}(\mathbf{x},t):(D,t)\to \Re$, where spatial coordinates are $\mathbf{x}\in D$ and D is the domain of the images. The description of the acquisitions is in Table 1. The processing method also provides an option to decrease the spatial resolution of the input data to improve the efficiency of subsequent analysis.

2.2. Restoration of Intensity Uniformity along Time

The illumination of the lamp in phase contrast microscopy may vary along time. The lamp may also flicker. As a result the brightness of the images not only varies smoothly along time, but also some of the images may be much darker or brighter than the immediately adjacent ones in time both before and after.

It is necessary to restore the uniformity of the image intensities along time. This is achieved by making the average intensity along time uniform. That is, the mean intensity per frame is preserved along time. However, the total average sequence intensity is also preserved. To this end, the average intensity of each frame as well as the average intensity of the entire image sequence are calculated. The intensities of each pixel in a frame are multiplied by a correction factor, which is the ratio of the average intensity of the entire image sequence over the average intensity of the corresponding image. That is,

$$\begin{array}{c}\hfill I_{PC}(\mathbf{x},t)\leftarrow I_{PC}(\mathbf{x},t)\frac{{〈I_{PC}({\mathbf{x}}^{\prime },t^{\prime })〉}_{{\mathbf{x}}^{\prime },t^{\prime }}}{{〈I_{PC}({\mathbf{x}}^{\prime },t)〉}_{{\mathbf{x}}^{\prime }}}\\ \hfill \mathrm{where}t=0,\dots ,\tau -1\end{array}$$

(1)

gives the sequence with time normalized intensity.

2.3. Restoration of Intensity Uniformity in Space

The intensity of the illumination is assumed to be and to remain uniform in space for all the frames of a sequence. However, in practice, the illumination in space across the sample is non-uniform. Hence, the acquired images suffer from spatial intensity non-uniformities. The non-uniformities are multiplicative and spatially smooth. Moreover, they cannot be computed directly. This is because they interfere with the inherent variation of the intensities primarily in the foreground of a sequence. In the analyzed sequences, the foreground is varying due to the dynamics and proliferation of the many cells present in a sample. In some frames, a spatial location may correspond to the background and in other frames the same spatial location may correspond to a cell.

The first objective is to remove the variability of the foreground to isolate a preliminary version of the background that is assumed constant along time. The background is considered equivalent to the intensity non-uniformity. The preliminary version of the background, $B\left(\mathbf{x}\right)$, is estimated by projecting it along time. To this end the method considers a circle around every spatial location $\mathbf{x}$ of radius approximately equal to that of a prespecified cell radius $\approx r_{cell}$. This circle is the base of a cylinder covering all time in spatiotemporal space. The median intensity of this spatiotemporal volume becomes the value of $B\left(\mathbf{x}\right)$.

The preliminary estimate of the background $B\left(\mathbf{x}\right)$ from the projection may still contain cells or their remnants from the foreground, particularly in regions occupied by stationary cells. Hence, $B\left(\mathbf{x}\right)$ is further processed to estimate a more accurate version of the background as its inverse that is the spatial intensity non-uniformity correction field. This is the objective of the remaining steps of the restoration performed with an adaptation of a Bayesian non-parametric method [61,62,63,64]. The histogram of the projection image $B\left(\mathbf{x}\right)$ is computed. The effect of the spatial intensity non-uniformity on the histogram is deconvolved to obtain a gain factor for each intensity in $B\left(\mathbf{x}\right)$. The gain factors are back-projected in space to the preliminary background image $B\left(\mathbf{x}\right)$ to compute a spatial gain field. The gain field is spatially filtered with a Gaussian filter to give an incremental smooth correction image. This then multiplies a cumulative correction field $V\left(\mathbf{x}\right)$, initialized to unity. The pixelwise multiplication of the correction field $V\left(\mathbf{x}\right)$ with each frame in the sequence gives the restored image sequence,

$$\begin{array}{c}\hfill I_{PC}(\mathbf{x},t)\leftarrow I_{PC}(\mathbf{x},t)V\left(\mathbf{x}\right)\\ \hfill \mathrm{where}t=0,\dots \tau -1.\end{array}$$

(2)

The cumulative restoration field $V\left(\mathbf{x}\right)$ is also scaled to ensure that it has a unit mean and that the intensity of the $95\%$ of the cumulative histogram of the spatiotemporal volume is preserved.

The spatial intensity restoration is iterative. The criterion to terminate the restoration uses the fact that the correction field has approximately unit mean and terminates the iterations when the standard deviation of the ratio between the correction fields $V\left(\mathbf{x}\right)$ of two consecutive iterations is very low, ≈$0.3\%$. The restoration is also terminated at a maximum allowed number of iterations. This correction is the third step in the block diagram of the preprocessing in Figure 1. An overview of the steps of the spatial intensity restoration method is in Figure 2. An example of a frame from the intensity restoration of a sequence is shown in Figure 3.

2.4. Extended Circle Hough Transform of a Phase Contrast Sequence

The shape of the cells during mitosis and cytokinesis becomes rounded with a well formed boundary. The imaging of these cells with phase contrast microscopy creates an approximately circularly symmetric halo effect of high contrast around them. The pixels along the contour of the halo effect are detected with pixelwise Hessian-based filtering. This is a convolution of the images with second order Gaussian derivative filters for every entry of the pixelwise Hessian matrices. The principal components of the Hessian matrix of a pixel along a cell boundary gives an eigenvector, $e_n$, normal to the boundary at that pixel with a corresponding eigenvalue of large magnitude, ${\lambda }_n$. The other eigenvector, $e_t$, is tangent to the boundary at that pixel with corresponding eigenvalue of a lower magnitude, ${\lambda }_t$. The largest absolute eigenvalue corresponds to the direction normal to the boundary and satisfies $|{\lambda }_n\left(\mathbf{x}\right)|>|{\lambda }_t\left(\mathbf{x}\right)|$.

In the phase contrast images in this study, the halo effect immediately surrounding the cells is bright as it is shown in the examples in Figure 5a,b. The bright cross-section of the contour along $e_n$ and considering the intensity as relief above the image plane is concave up and so it corresponds to an eigenvalue, ${\lambda }_n$, that is negative [65]. That is, along the cell contours the eigenvalue of largest magnitude is negative. Pixels where the eigenvalue of largest magnitude is positive are considered noisy and are pruned to zero. A summary of Hessian filtering over the entire sequence provides,

$$I_H(\mathbf{x},t)=\left\{\begin{array}{cc}-{\lambda }_n\left(\mathbf{x}\right)& \mathrm{if}{\lambda }_n\left(\mathbf{x}\right)<0,\mathrm{at}t\\ 0& \mathrm{otherwise}.\end{array}\right.$$

(3)

The entries of these images are used as modulating weights to extend the basic form of the circle Hough transform.

To further improve the efficiency of the circle Hough transform the set of pixels that participates in the transform is pruned even further. This is achieved using the statistics of the histogram of the sequence of the modulating weight images, $I_H(\mathbf{x},t)$, given in Equation (3). This histogram is unimodal with the mode corresponding to the dark background and the dark inner cytoplasm regions. The bright halo effects surrounding the mitotic cells correspond to the sidelobe of the higher intensity range. That sidelobe is detected as the intensity range above the intensity corresponding to the mode plus two times the full width half maximum of that mode. The pixels with intensities below that range are pruned and are not considered for the transform. This pruning only causes a minimal loss of accuracy to the transform and may even improve its accuracy by ignoring the contributions from noisy pixels.

The basic circle Hough transform [66] is extended to modulate the contribution of every pixel to the transform by a weight that is spatially localized with a two-dimensional Gaussian distribution, $G\left(\mathbf{x}\right)$. The covariance of the Gaussian is aligned with the two eigenvectors of the Hessian. The direction along $e_n$, across the contour for the centroid of the cell is assigned a standard deviation ${\sigma }_n$ equal to ${\sigma }_n=1.5r_{cell}pixels$. The direction along $e_t$, tangent to the boundary smooths the transform and is assigned a standard deviation ${\sigma }_t$ that is small and equal to a small fraction of the cell radius, but it still has a finite value, ${\sigma }_t=MAX(0.3r_{cell},3)pixels$.

The transform considers the Gaussians only locally around their mean points relative to the radius parameter of the cells, $r_{cell}$. Thus, it contributes to only a subset of the image domain. It truncates the Gaussian normal to the boundary $G_{e_n}$ to $1.75{\sigma }_n$ and the Gaussian tangent to the boundary $G_{e_t}$ to $1.75{\sigma }_t$. This not only expedites the transform, but also makes it more robust with respect to dense cells samples.

The proposed transform has two stages. The first stage for $I_{Hg}^c$ in Equation (5) transforms both along the positive as well as along the negative directions of the normal axes for the pixels assumed to be lying along cell contours. It is modulated with a Gaussian centered on the contour points at ${\mathbf{x}}_0$. The Hough transform of the image is:

$$\begin{array}{ccc}\hfill I_{Hg}^c(\mathbf{x},t)& =& \sum _{{\mathbf{x}}_0\in D}\sum _{(i,j)=(-r_{cell},-r_{cell}/4)}^{(r_{cell},r_{cell}/4)}{I}_H({\mathbf{x}}_0,t)\times \hfill \\ & & \times G_t({\mathbf{x}}_0+e_n\ast i|{\mathbf{x}}_0,{\sigma }_t)\times \hfill \\ \hfill & & \times G_n({\mathbf{x}}_0+e_t\ast j|{\mathbf{x}}_0,{\sigma }_n),\hfill \\ \hfill \mathrm{where}& & \mathbf{x}={\mathbf{x}}_0+e_n\ast i+e_t\ast j,\hfill \end{array}$$

(4)

$G_t$ is the Gaussian along $e_t$ and $G_n$ is the Gaussian along $e_n$. In this resulting transform space, close to the contours points of the cells there is a higher density value towards the interiors rather than towards the exteriors of the cells. Sampling this transform space normal and close to the contours points identifies the direction from the contour points corresponding to higher transform densities. In the transform, “voting” is along lines normal to the contours. Hence, the higher densities are towards the centers of curvature [67]. The centers of curvature for the circular cells are in the directions towards the interiors of the cells. That is, the first stage provides the directions from the contour points towards the centers of curvature that are the directions towards the interiors of the cells.

The second stage that gives the final extended transform, $I_{Hg}$, in Equation (6), is only towards the interiors of the cells. It involves exclusively the directions from the contour points towards the centers of curvature. Hence, it only uses the positive parts of the normal axes. The contributions along these semi-axes are also weighted with Gaussians centered at the estimated centers of the cells, which are at a positive distance $r_{cell}$ from the points of the contours. The extended circle Hough transform of an image is given by:

$$\begin{array}{ccc}\hfill I_{Hg}(\mathbf{x},t)& =& \sum _{{\mathbf{x}}_0\in D}\sum _{(i,j)=(0,0)}^{(r_{cell},r_{cell}/4)}{I}_H({\mathbf{x}}_0,t)\times \hfill \\ & & \times G_t({\mathbf{x}}_0+e_n\ast i|{\mathbf{x}}_0+e_n\ast r_{cell},{\sigma }_t)\times \hfill \\ \hfill & & \times G_n({\mathbf{x}}_0+e_t\ast j|{\mathbf{x}}_0,{\sigma }_n),\hfill \\ \hfill \mathrm{where}& & \mathbf{x}={\mathbf{x}}_0+e_n\ast i+e_t\ast j,\hfill \end{array}$$

(5)

$G_t$ is the Gaussian along $e_t$ and $G_n$ is the Gaussian along $e_n$. The weighting Gaussian, $G_n$, is shown schematically in the diagram in Figure 4.

The extended circle Hough transform is the fourth step in the block diagram of the preprocessing steps in Figure 1. Two example images that contain many rounded cells in mitosis and cytokinesis with a high contrast and the extended circle Hough transforms of these images are shown in Figure 5. It is recommended that the images in Figure 5 be viewed using the electronic/digital magnification functionality available. The extensions make the circle Hough transform images very intense and the cell centroids in these transform images to have a much higher contrast. After the cell division their contrast relaxes back to low levels.

2.5. Registration for Translation Transformations

The time-lapse microscopy movies analyzed are acquired over long periods of time. During that time, there can be a mechanical shift of the sample and the microscope can be interleaving among several samples. Both of these result in a drift in the relative location between the sample and the camera that may be accumulating over time. In turn, this causes cumulative spatial misplacements between the frames of a sequence that are mainly translations.

This spatial artifact is addressed by registering for global image translations between consecutive frames of a sequence. The registration is based on the extended circle Hough transform images that contain a reconstructed and more explicit information about the sample with the cells being convex as opposed to the phase contrast images where the cytoplasm has the same intensity as that of the background. The translation of the zeroth time point frame, $I_{Hg}(\mathbf{x},0)$, is the identity $f_0\equiv I\equiv (0,0)$. The registration computes the relative translation $f_{t-1\leftarrow t}$ between two consecutive frames. The reference frame is that of the previous time point, $I_{Hg}(·,t-1)$, and the moving frame is the next one, $I_{Hg}(·,t)$. The objective is to spatially normalize the moving frame to the reference one. The sequence of registrations starts from the first frame and proceeds to the last frame. This gives the total frame registrations $f_t\equiv (\Delta x_t,\Delta y_t)$, $t=1,\dots ,\tau -1$. Several possible in-plane translations are considered with maximum extend $2h$. The candidate translations are examined using a regular spatial subsampling factor $\delta h$ to improve computational efficiency. The optimal translation is selected.

The optimal translation is the one with minimal cost over the entire image. The basis of the cost of a candidate translation is the absolute value of the difference between a translated frame and the previous one that is the reference. This is the $L_1$-norm of the difference. The $L_1$-norm is further normalized with respect to the size of the common domain between the translated frame and the reference one. The trial translations $f_{*}$ are evaluated with exhaustive search to give the optimal one as:

$$\begin{array}{ccc}\hfill f_{t-1\leftarrow t}& =& argmi{n}_{f_{*}}\frac{\parallel I_{Hg}(f_{t-1}\mathbf{x},t-1)-I_{Hg}(f_{*}\mathbf{x},t){\parallel }_1}{\parallel f_{t-1}D\cap f_{*}{D\parallel }_1},\hfill \\ \hfill \mathrm{where}& & f_{*}\in ([-2h,2h],[-2h,2h]).\hfill \end{array}$$

(6)

The intersection of the domains of all the registered frames along time is the valid domain of the image sequence. That is, the valid domain is $D_f=D\cap \left(f_{1}D\right)\dots \cap \left(f_{\tau -1}D\right)$. The zeroth frame is not moved. The common domain of the sequence after registration with $D_f$ will be a subset of the valid domain of the zeroth image $I_0$, $D_f\subseteq D$.

The registration along a sequence starting from the first frame and finishing at the last one is repeated several times. This enables corrections for misregistrations that happen early in time to propagate to later time points. The repetitions of the registrations of the sequence follow a multiresolution approach. The search starts with a large displacement for the coarse resolution and completes with a small, local, displacement for the fine resolution. There are three levels of resolution and the extent of the maximum shift is $2h$ for all of them. The initial value is $h=8$pixels and the sampling is at $\delta h$. Both the extend of the maximum candidate shifts, 2h, and the number of sampling points for the displacement are refined to smaller values with the repetitions of the registration of the sequence. The decrease is linear and gradual to smaller values for finer resolutions with consecutive repetitions of the sequence registration. The sampling between candidate displacements is kept at $\delta h$ for all resolutions.

The iterations over the spatiotemporal volume, $t=1,\dots ,\tau -1$, continue for a certain resolution up to the iteration at which the percentage of non-zero to zero optimal translations between frames over the entire sequence is very low, which implies that it is close to the optimal translations for a sequence or up to a maximum number of iterations, $i_{max}=9$, for a specific resolution level. It then proceeds to the next finer resolution level until the finest resolution.

The transformations $f_{t-1\leftarrow t},t=1,\dots ,\tau -1$, computed with Equation (7), are applied to the corresponding frames of the phase contrast sequence $I_{PC}$, the Hessian filtered sequence $I_H$, and the circle Hough transform sequence $I_{Hg}$ for the purposes of subsequent processing steps. The phase contrast image sequence becomes $I_{PC}^f(\mathbf{x},t)=I_{PC}(f_t\mathbf{x},t)$, where $f_t\mathbf{x}=(x+\Delta x_t,y+\Delta y_t)$. The Hessian filtered image sequence becomes $I_H^f(\mathbf{x},t)=I_H(f_t\mathbf{x},t)$. Finally, the circle Hough transform sequence becomes $I_{Hg}^f(\mathbf{x},t)=I_{Hg}(f_t\mathbf{x},t)$. All three of them are defined in space only over the valid domain $D_f$. The registration is the last step of the preprocessing as it is shown in the block diagram in Figure 1. A schematic diagram of the sequence registration is shown in Figure 6.

2.6. Extraction of Cell Centroids

The intensity restored and circle Hough transform registered sequences, as shown in Figure 7, are used for the extraction of the cell trajectories and the identification of the cell divisions. The extended circle Hough transform images provide the conditions to detect the cell centroids. The first condition is that a cell centroid be a local maximum within a circle of radius $r_{cell}$ in a circle Hough transform image. This condition:

$${(\mathbf{x},t)}_{cent}=\{(\mathbf{x},t):I_{Hg}^f(\mathbf{x},t)>I_{Hg}^f(\mathbf{y},t)\forall \parallel \mathbf{y}-\mathbf{x}\parallel <r_{cell}\}$$

(7)

ensures a local maximum. The second condition uses the cumulative histogram of the 3D spatiotemporal volume of the framewise circle Hough transform sequence, ${\mathcal{H}}_{Hg}(·)$. It requires that a cell centroid be of value that corresponds to at least 98% of ${\mathcal{H}}_{Hg}(·)$. The condition

$${(\mathbf{x},t)}_{cent}=\{(\mathbf{x},t):{\mathcal{H}}_m\left(I_{Hg}^f(\mathbf{x},t)\right)>98\%\}$$

(8)

ensures that the candidate pixel has a value that is sufficiently high. Both of the two conditions above are necessary for a point to be classified as a cell centroid. These conditions in Equations (7) and (8) are drawn in Figure 8.

A spatiotemporal volume is formed $I_{Centroid}^f(\mathbf{x},t)$ with non-zero values only at detected cell centroid pixels with actual values equal to those of the circle Hough transform images at those pixels:

$$I_{Cent}^f(\mathbf{x},t)=\left\{\begin{array}{cc}{I}_{Hg}^f(\mathbf{x},t)& \mathrm{if}(\mathbf{x},t)={(\mathbf{x},t)}_{cent},\\ 0& otherwise.\end{array}\right.$$

(9)

This provides both the centroid pixels and their circle Hough transform values. The detection of cell centroids and their values is the second step of the processing pipeline in the block diagram shown in Figure 7.

2.7. Inter-Frame Associations between Pixels

The detected cell centroids are the starting points for the two trajectory segments. One proceeds in positive time direction and another in negative time direction. These establish the location of a cell centroid in previous and in subsequent time points, respectively. The tracking is performed not only along progressing time, but also anti-causally in the negative time direction. A cell division is detected as a join of two different trajectories at least one of which is almost always extracted in the negative time direction.

A cell motion trajectory is based on associations between pixels of successive frames. They are established by directing inter-frame pixel associations locally towards maxima of the circle Hough transform of the reference frame. Thus, the associations can be many to one and do form a function. Along the negative time direction the reference frame is the previous one and the association function is ${\mathcal{S}}_{-}$. In the positive time direction the reference frame is the next frame and the association function is ${\mathcal{S}}_{+}$.

An association of ${\mathcal{S}}_{-}$ from a pixel at time t, $(\mathbf{x},t)$, in reverse time direction to a pixel at time $t-1$, $t\to t-1$, is established with a local search in frame $t-1$. The search over candidate displacements $\Delta \mathbf{x}$ is within a window of radius $\parallel \Delta \mathbf{x}\parallel \le \frac{2}{3}{r}_{cell}$ in the frame of the previous time point $(\mathbf{x},t-1)$. The maximum with penalized displacement $|\Delta \mathbf{x}|$ establishes an association from pixel $(\mathbf{x},t)$ in the frame at time t to pixel $({\mathbf{x}}^{\prime },t-1)$. The utility function for the association to pixel $({\mathbf{x}}^{\prime },t-1)$ is:

$$u(\mathbf{x},\Delta \mathbf{x},t-1)=I_{Hg}^f(\mathbf{x}+\Delta \mathbf{x},t-1)\left(1-0.25\frac{\parallel \Delta \mathbf{x}\parallel }{r_{cell}}\right),$$

(10)

where $\parallel \Delta \mathbf{x}\parallel \le r_{cell}$. The association is to the pixel which maximizes the utility in Equation (10) at time $t-1$,

$$({\mathbf{x}}^{\prime },t-1)=argma{x}_{\parallel \Delta \mathbf{x}\parallel \le r_{cell}}u(\mathbf{x},\Delta \mathbf{x},t-1),$$

(11)

and location ${\mathbf{x}}^{\prime }$. This maximization establishes ${\mathbf{x}}^{\prime }$ to be the position of the association at the previous time point $t-1$. That is, the selection is the pixel of maximum circle Hough transform intensity with penalized displacement. This search, within $\parallel \Delta \mathbf{x}\parallel \le \frac{2}{3}{r}_{cell}$, is repeated three times. It creates a search path in $t-1$ that consists of a sequence of three local maxima and stops at the third maximum. The repetition contributes to representing not only the motion of a cell, but also possible remnants of misregistration between successive frames.

A reverse time association from $t\to t-1$ is an entry in the function ${\mathcal{S}}_{-}$. The associations are computed in a lazy fashion for computational efficiency. They are computed only if starting from a certain detected or tracked cell centroid it is necessary to compute an association to continue with the extraction of a cell trajectory.

The same process as in Equations (10) and (11) is used to compute causal associations ${\mathcal{S}}_{+}$ from a pixel at time t, $(\mathbf{x},t)$, in forward time direction to a pixel at time $t+1$, $t\to t+1$. Similarly to Equations (10) and (11), they are established by maximizing utility $u(\mathbf{x},\Delta \mathbf{x},t+1)$ to give $({\mathbf{x}}^{\prime },t+1)$ as the association at the next time point $t+1$. These inter-frame pixel association of ${\mathcal{S}}_{+}$ are also many to one and form a function. Thus, considering both functions ${\mathcal{S}}_{-}$ and ${\mathcal{S}}_{+}$ the pixel associations are many to many. They are mainly used to track cell centroids. The inter-frame pixel associations is the third step of the motion tracking part of the method summarized in Figure 7.

2.8. Extraction of Cell Trajectories and Detection of Cell Divisions

The temporal associations between pixels of adjacent time points are used to extract cell trajectories. The trajectories also establish cell centroids in frames where they might have been originally missed. Furthermore, joining two of these trajectories, extracted sequentially in the method, at a pixel in space and time gives a cell division candidate. That is, the joining of two trajectories identifies a candidate for cell division as a topological by-product. This happens when some of the extracted trajectories are of daughter cells during cytokineses. The cell division candidate, if verified, is actually immediately before the cytoplasm division to two daughter cells in cytokinesis and during the telophase of the mitotic cycle of the cell.

The extraction of the cell trajectories uses the inter-frame pixel associations from ${\mathcal{S}}_{-}$ and ${\mathcal{S}}_{+}$, the circle Hough transform images $I_{Hg}^f$, as well as the cell centroid images $I_{Cent}^f$. The starting point, ${(\mathbf{x},t)}_{start}$, for the extraction of a cell trajectory is the pixel of maximum intensity in the sequence of initially extracted centroids,

$${(\mathbf{x},t)}_{start}=ma{x}_{(\mathbf{x},t)}{I}_{Cent}^f(\mathbf{x},t).$$

(12)

This point, ${(\mathbf{x},t)}_{start}$, identified as starting point for a trajectory, has to also be at a distance of at least $3r_{cell}$ from all previously extracted trajectories.

The trajectory is extracted by progressing both reversely in time with ${\mathcal{S}}_{-}$ as well as forwards in time with ${\mathcal{S}}_{+}$. This continues as long as the intensity of the trajectory at the circle Hough transform image $I_{Hg}^f(\mathbf{x},\mathbf{t})$, from Equation (6) and registered according to Equation (7), is greater that 20% of the reference Hough intensity that corresponds to the 95% of the cumulative histogram of the spatiotemporal volume of the circle Hough transform, $0.2{\mathcal{H}}_{Hg}^{-1}\left(0.95\right)$.

In tracking reversely in time with ${\mathcal{S}}_{-}$, anti-causally, when the newly extracted trajectory approaches a previously extracted trajectory within a distance less than $1.5r_{cell}$, it undergoes a test to examine if this corresponds to a trajectory joining event due to reverse cell division. That is, whether the two trajectories in that time point are within a single mitotic cell at telophase immediately before a cell division, in which case they are merged. The tests involve the Hessian filtered image, $I_H^f$, as well as the circle Hough transform image, $I_{Hg}^f$, intensities of the point in the pre-existing trajectory and the point in the newly tracked one at the same time t. The two points are considered as poles interconnected by a line segment. The first condition involves the Hessian filtered values, $I_H^f$, and requires that any point along the line segment be less than that of the average of the two poles. The second condition involves the circle Hough transform values, $I_{Hg}^f$, and requires that any point along the line segment be greater than that of the average of the two poles.

The third and last condition is topological for the trajectories and involves the extend of the pre-existing trajectory. It requires that the pre-existing trajectory exists for at least one time point both before and one time point after the time point for which the two points are investigated for merging due to cell division. This condition ensures that the newly extracted trajectory segment is not a simple continuation of the pre-existing trajectory.

The cell centroids in $I_{Cent}^f$ from Equation (9) in the region around a newly extracted trajectory are invalidated within a distance of $3r_{cell}$ so that no other trajectory is initiated from that region. The centroid pixel of maximum intensity from the remaining valid spatiotemporal centroid volume $I_{Cent}^f$ is identified. If it is non-zero, this centroid becomes the starting point for the extraction of a new trajectory. The sequential extraction of trajectories continues until the entire of $I_{Cent}^f$ volume becomes zero.

This is the last step of the method, namely, the step of cell trajectory extraction and cell division identification and is summarized in Figure 7. Figure 9 shows an example schematic of trajectories both in forward and in reverse time directions as well as a detected cell division. A flow chart of the method for the sequential extraction of cell trajectories and cell division detections is shown in Figure 10.

3. Experiments

The method has been applied and validated with multiple image sequences of several cell lines that were acquired over many hours and hence have a large temporal size as described in Table 1. These sequences are from real biological experiments. The analysis of the data has multiple processing steps that normally have considerable computational requirements. To maintain a reasonable running time for the method, despite the extensive processing required, the implementation of the various steps of the method has been made efficient. The method is implemented in the C++ programming language. The spatial resolution of the imaging sequence used is $60\%$ over each spatial axis or $36\%$ over the whole spatiotemporal volume. The experiments were performed on a laptop with an Intel Core i7 processor of $2.60$ GHz and $16.0$ GB of RAM.

3.1. Validation Measures for Cell Division Detection

The objective of the method is to detect mitoses and accompanying cell divisions. These events are considered for the evaluation as long as they happen in the spatiotemporal interval imaged to obtain a sequence. In particular, a cell division is considered if the image sequence contains at least the late phases of mitosis, namely, anaphase and telophase, as well as the early part of cytokinesis including the cleavage of the cytoplasm into two daughter cells. The reference quantities considered for the performance evaluation of the cell division detection method for a sequence are the True Positive ($TP$), False Positive ($FP$), False Negative ($FN$), and True Negative ($TN$) detections. The number of $TN$ cells is the number of cells in the spatiotemporal sequence not undergoing mitosis.

The performance of the system is quantified in terms of the $Precision$ and $Recall$, the $Sensitivity$ and $Specificity$, as well as the $F1–score$. The recall is equivalent to the sensitivity. The $Precision$ is the fraction of the detected cell divisions that are actually cell divisions and is given by:

$$Precision=\frac{TP}{TP+FP}.$$

(13)

The $Recall$ is the fraction of the cell divisions that are actually detected out of all the cell divisions. This is equivalent to the $Sensitivity$ and are given by:

$$Recall\equiv Sensitivity=\frac{TP}{TP+FN}.$$

(14)

The $Specificity$ is the proportion of the cells not undergoing cell division that are not detected and is given by:

$$Specificity=\frac{TN}{TN+FP}.$$

(15)

The $F1–score$ is the harmonic mean of the $Precision$ and the $Recall$, that is, it is their average rate:

$$F1–score=\frac{2(Precision\times Recall)}{Precision+Recall}.$$

(16)

These quantities are computed for every sequence independently.

3.2. Experimental Performance of the Image Sequences

The data for the validation of the method were fourteen spatiotemporal sequences corresponding to three different cell lines. Each cell line belongs to a different group. The description of the first, second, and third groups are in the three columns of Table 1, respectively. The output of the method for a microscopy sequence is a new movie with background the original movie with preprocessing and foreground annotations for the automatically tracked cells that undergo cell division for the duration of the sequence. The annotation starts from mitosis during metaphase, anaphase, or telophase. The annotations continue for the tracking of the daughter cells during cytokinesis immediately following the division of the cytoplasm. A parent during mitosis and prior to cell division is annotated with a mark of a specific shape, color, and length. The same mark annotates the two resulting daughter cells. The actual point in space and time before the parent cell divides is indicated with a thick cross of the same color as that of the cell mark.

Some examples of representative frames from the output sequences from each of the first, second, and third groups of sequences are shown in Figure 11, Figure 12 and Figure 13. These frames show multiple cells in mitosis as well as in cytokinesis both before and after the division of the cytoplasm. It is recommended that the images in Figure 11, Figure 12 and Figure 13 be viewed in their original color and using the electronic/digital magnification functionality available. The sequence in Figure 11 is of the cell line of the first group of Table 1. It lasted for 3 h 15 min and gave 65 frames. The sequence in Figure 12 is of the cell line of the second group of Table 1. It lasted for 5 h and gave 100 frames. The sequence in Figure 13 is of the cell line of the third group of Table 1. It lasted for 5 h and gave 60 frames.

The results of the method for the fourteen sequences were evaluated by a scientist, MD, with specialty in radiology as well as a PhD in neuroscience and expertise in microscopy and biomedical quantitative image analysis. The evaluator annotated and counted the $TP$s, $FP$s, and $FN$s for every movie. These provided the $Precision$, and the $Recall$, which is equivalent to the $Sensitivity$, as well as the $F1–score$. The results of the evaluation of the method with these sequences are in

Table 2. Evaluation of the performance of the detections of cell divisions.

Seq.\Meas.	$\mathit{TP}$	$\mathit{FP}$	$\mathit{FN}$	$\mathit{Precision}$	$\mathit{Recall}$≡ ≡$\mathit{Sensitivity}$	$\mathit{F}1–\mathit{Score}$
Sequences 1
Sequence 1	8	0	2	1	0.8	0.89
Sequence 2	8	0	0	1	1	1
Sequence 3	9	0	2	1	0.82	0.9
Sequence 4	7	0	2	1	0.78	0.88
Sequence 5	18	0	2	1	0.9	0.95
Sequences 2
Sequence 1	7	0	0	1	1	1
Sequence 2	7	0	1	1	0.86	0.93
Sequence 3	15	0	0	1	1	1
Sequence 4	12	0	1	1	0.92	0.96
Sequences 3
Sequence 1	19	0	1	1	0.95	0.97
Sequence 2	23	0	1	1	0.96	0.98
Sequence 3	17	0	5	1	0.77	0.87
Sequence 4	11	0	3	1	0.79	0.88
Sequence 5	18	3	3	0.86	0.86	0.86

. The evaluations for the sequences described in the columns of Table 1 are given in the corresponding groups of rows of Table 2. The results show that the overall performance of the method is very good. This is shown particularly by the $Precision$ as well as the $F1–score$.

The cell divisions that happen before the beginning of the acquisition of the sequence are not considered even if apparent sibling cells are adjacent in the movies. Similarly, mitotic cells are ignored if the cleavage of their cytoplasms does not occur before the end of the sequence. Some $FN$ were identified in the outer field of the movies close to the borders of the images. The cells close to the boundaries of the images may be blurry due to artifacts such as spherical aberrations of the lenses present in the outer field as well as due to artifacts of the outer field specific to phase contrast imaging. The method may also not detect cell divisions that occur partly beyond the borders of the frames of the registered sequence, in which case the $FN$s are ignored. A limited number of $FP$s were identified from planes originating from below or above the one imaged due to cells moving in and out of the imaged plane.

In many sequences, particularly those with dense samples, the contrast of many of the cells not undergoing mitosis is low and the cells may even be unnoticeable. In these cases it is also difficult to discriminate between adjacent cells. Thus, the presence of $TN$ was only assessed qualitatively. The number of $TN$ cells was still found to be significantly higher than the number of $FP$ cells, that is, $TN>>FP$. Thus, the $Specificity$ was also very high.

The average time to process the sequences in the first group in Table 1 was ≈16 min, the average time to process the sequences of the second group in Table 1 was ≈26 min, and the average time to process the sequences of the third group in the same table was ≈17 min. This short running time for all the steps of preprocessing and processing of the whole method demonstrates its efficient implementation.

4. Discussion and Future Work

This study is in the context of interpreting dynamic cellular data from phase contrast microscopy to follow and quantify cell motility and proliferation. In particular, the objectives of the developed method have been to detect mitoses associated with cell divisions as spatiotemporal events from phase contrast cellular video data. The method takes advantage of the circular symmetry of mitotic cells, the symmetry of cell division and cytokinesis, as well as the symmetry of the entire cell cycle.

4.1. Summary

The method starts with a necessary preprocessing of the image sequences. Individual frames in a time sequence can be subject to different lighting due to flickering of the microscopy lamp. Thus, some frames can be brighter or darker with jumps in contrast. The method accounts for this with normalization of the mean intensities of the frames of the sequences along time. This is followed by restoration for the spatial distribution of the intensity non-uniformity of the microscope lamp that is assumed constant along time in a sequence. The intensity restored images are processed with a circle Hough transform enhanced with the ability to “vote” only towards the centers of curvature. That is, “vote” only around the cell centroid in the cell interior. This enhancement improves the robustness of the transform for a dense cell sample and the discriminability of the cell centroids.

A sufficient number of cell centroids of parents as well as necessarily of daughter cells are then identified and serve as starting points for the extraction of cell trajectories. Their extraction proceeds both towards the positive as well as towards the negative time directions. Therefore, the extraction of a cell trajectory is non-causal. The extraction of the trajectory of a daughter cell in negative time direction enables the identification of a candidate cell division as being a connection to an already extracted trajectory consisting of the motion of its sibling daughter cell together with that of their parent cell. That is, candidate cell divisions are detected as a by-product of topological connectivity of trajectories at least one of which is that of a daughter cell extracted in the negative time direction. Thus, the cell division detection is anti-causal. The candidate cell divisions are also verified by testing within the corresponding frames that they indeed belong to the same cell. In cancer cell samples, with abnormalities in the cell cycle, the cell divisions can lead to more than two daughter cells. Such abnormal cell divisions are also detected by the proposed method. This is made possible by the more than two to one inter-frame pixel associations that are possible since they preserve the function property of the associations. These associations are estimated and followed on a per need basis for trajectory extraction. The method can detect cell trajectories around the cell division time point as well as a limited cell ancestry from a grandparent.

A parameter of the method for a sequence is the expected radius of the cells, $r_{cell}$. The value of the radius is set by considering the approximate size of the cells in the sample as well as of the actual image magnifications both optical and digital. The prior knowledge of the value of $r_{cell}$ tunes the circle Hough transform to improve its discriminability for the daughter cells as well as its efficiency. The method has been found robust to the value of $r_{cell}$. The circle Hough transform is robust to cells which have a convex boundary as well. The value of $r_{cell}$ is also used to restrict the motion displacement estimates, since mitotic and dividing cells have limited mobility. The second parameter, set to $98\%$ in Equation (8) for the circle Hough transform image sequence, affects the cut-off value for consideration of the cell centroids. The implementation of all of the preprocessing and processing steps of the method make efficiency considerations. The sequences are spatially subsampled, the pixels used for the circle Hough transform are pruned, the optimization for the registration uses spatial subsampling, and the inter-pixel associations are computed in a lazy fashion. The experiments used fourteen sequences of three different cell lines. The sequences were processed with the developed cell division detection method. The outputs were validated quantitatively in terms of $Precision$, $Recall$ ($Sensitivity$), $F1–score$ and even in terms of $Specificity$. The validation demonstrated the high performance of the method.

4.2. Discussion

A cell that rounds up and increases its contrast may be mitotic, but may fail to undergo cytokinesis. However, not all cells that round up and increase their contrast are mitotic. Thus the method requires that a detected rounded cell also undergoes cytokinesis that leads to two detected daughter cells to classify the event as a mitotic cell division. This may miss some divisions, that is, it may create some $FN$. However, this is much less than the $FP$ that would result without the requirement for the presence of two daughter cells.

The method does not consider adjacent cells as being in cytokineses even if associated cell divisions appear to have happened immediately before the beginning of the sequence. It also does not consider mitotic cells even if they divide immediately after the end of the sequence. Cell divisions close to the image boundary may not be detected and lead to misses of the method for several reasons. Some cells, particularly close to the image boundaries, are blurry due to lens distortions and phase contrast imaging limitations. The boundary effects also limit Hessian filtering. Finally, due to compensation for misregistration shifts, the domain of some of the frames close to the image boundaries may not be part of the common domain of the sequence. In the latter two cases, the misses are ignored.

The cell division detection methods can be classified into tracking-based approaches, tracking-free approaches, and appearance-based approaches [6,31]. There exist trade-offs between these different types of approaches. Tracking-based methods can better represent the progression of mitoses and cell divisions over time. However, the performance of tracking-based methods can be lower for dense samples or in the presence of excessive cellular motion. They are also non-causal and involve delays in sequence processing. To address these issues, tracking-free approaches are also used. Tracking-free approaches suffer from limitations as well, specifically since mitoses and cell divisions are a temporal sequence of events that cannot be completely analyzed in each frame in time independently. On the other hand, appearance-based approaches with machine learning and neural networks, can detect cell divisions events, even spatiotemporally, robustly to their variations in appearance. However, appearance-based methods have several limitations. They require extensive training, they have not been shown able to generalize among cell lines to the authors knowledge, they are sensitive to dense samples, and do not characterize the complete mitoses and cytokineses events. In addition, in the last two types of approaches, namely, tracking-free and appearance-based, when treating spatiotemporal frame sequences as samples, the precise temporal resolutions for the cell divisions becomes weak.

The proposed method benefits from the spatial shapes and symmetries of the cells and the cell divisions rather than simply treating the sequences as signals or samples. It is also able to take advantage of the cell trajectories in space and time to robustly and uniquely detect cell divisions even in dense samples.

4.3. Future Work

The proposed method can be further improved and generalized in a variety of ways. The method was developed and validated with real experimental movies to demonstrate its applicability to the practice of a biological laboratory. A benchmark with movies of not only a single cell line [7,68], but of multiple cell lines, specifically acquired and annotated for method development can contribute to methodology improvements.

The imaging of the sequences analyzed was performed with positive phase contrast. This has been considered to interpret the Hessian-based filtering and to adjust the circle Hough transform as well as several other processing steps. In negative phase contrast, the appearance of the cells and of the halo effect around them is different. It requires a generalization in the interpretation of Hessian-based filtering in Equation (3) and of other subsequent processing steps. In low spatial resolution sequences, the thickness of the Halo effect at the boundary of the cells is larger and Hessian filtering must be adjusted accordingly.

The spatial intensity correction method computes the non-uniformity retrospectively from the entire sequence. To achieve real time intensity correction, the processing could be made causal by only considering preceding frames of the sequence. Further, the intensity uniformity restoration over time and over space could be performed jointly to better preserve contrast in the restored sequence.

The cell division detection with circle Hough transform based on shape symmetry can be made more specific by detecting circles of only a prespecified radius. That would avoid false positives from which conventional circle Hough transform in this context suffers due to “votes” into rings of smaller or larger radii as well as due to cells in interphase that may be elongated. A parent cell prior to its division has a larger radius compared to its daughter cells immediately following the division. Abnormal cell divisions that lead to three daughter cells have even smaller radii. Thus, it will be productive to extend the cell detection with multiresolution to consider prespecified cell radii for both before and after the cell division. Concerning the cell radii, another novelty in the extended circle Hough transform method would be to not only detect the direction towards the center of curvature, but also to detect the center of curvature itself. That would obviate the need to specify a cell radius parameter that is present explicitly or implicitly in most relevant methods in the literature. The cells in the sequences processed are convex. However, nerve cells can be concave and can have shapes that resemble some forms of starlike superquadrics [69]. The detection of such cells would require the development of a novel Hough transform methodology applicable to such superquadrics.

The circle Hough transform sequences are used to establish inter-frame pixel associations. For sequences with extensive cell motility, the associations can be established with the contribution of a stochastic motion tracking method as well [42]. The extraction of the trajectories of the daughter cells that drift apart can also use a stochastic motion tracking method [42]. The spatiotemporal angle between the trajectories of two daughter cells after a cell division is symmetric and is a “fingerprint” of a particular cell line. Abnormalities in the spatiotemporal division angle can also be a “fingerprint” of specific pathologies such as a wider neuronal precursor division angle for the Miller Diecker Lissencephaly syndrome that is causing retardation and epilepsy [70]. The spatiotemporal division angle can be considered as an additional condition in the methodology to determine whether to join two trajectories to form a cell division to improve the accuracy of the detection.

An abnormal cell division can lead to cell death, namely, apoptosis. The detection of such events would make the method more representative and further improve its performance. It would also be informative for high throughput screening applications to compute overall statistics for a movie, such as the number of cells dividing per minute, the duration of the cell cycle, the planar orientations of the mitotic axes, and the number of apoptoses. In a pharmaceutical context, the method can be used to quantify and study the response of treating cells with compounds whose objective is to affect the cell cycle by inhibiting mitotic progression or targeting abnormal cell divisions [42]. An example use would be for the development and screening of anti-cancer drugs.