Unsupervised Segmentation of Muscle Precursor Cell Images In Situ

Abstract

In vitro culture of muscle stem cells on a large scale could bring light to the treatment of muscle-related diseases. However, the current work related to muscle stem cell culture is still only performed in specialized biological laboratories that are very much limited by manual experience. There are still some difficulties to achieve an automated culture of complex morphological cells in terms of live cell observation and morphological analysis. In this paper, a set of bright-field cell in situ imaging devices is designed to perform non-contact and invasive imaging of muscle precursor cells in vitro, and a neural network structured lightweight unsupervised semantic segmentation algorithm is proposed for the acquired images to achieve online extraction of cell regions of interest without manual annotation and pre-training. The algorithm first uses a graph-based super-pixel segmentation to obtain a coarse segmentation, then aggregates the coarse segmentation results with the help of Laplace operators as a reference to a four-layer convolutional neural network (CNN). The CNN parameters learn to refine the boundaries of the cells which helps the final segmentation accuracy and mean intersection–merge ratio reach 88% and 77%, respectively.

1. Introduction

Muscle is an important motor and metabolic component in the human body, but there is no viable treatment for muscular degeneration caused by aging and disease, as well as muscle atrophy induced by long-term spaceflight in a weightless environment [1]. Alternatively, cell replacement therapy may be a breakthrough that injects healthy cells into the injury to replace dead cells to repair the injured tissue and restore the lost function [2,3]. The premise of the therapy is to obtain a large number of functional cells required for transplantation. The in vitro expansion technology of adult stem cells has been relatively mature in biological laboratories [4,5,6]. Cell culture depends greatly on manual experience, subject to subjective facts: The quality of cells cultured by different experimenters varies widely and the same experimenter’s judgement also varies with time. Manual control alone cannot achieve refined control in mass production. Therefore, the automatic culture of muscle stem cells in vitro by microscopy and computer technology must be considered for industrialization.

The most common method for gaining a suitable live cell image is fluorescence imaging which uses markers to label cells and capture the specific fluorescence for imaging. Guo et al. [7,8] have performed nuclear and cytoplasmic segmentation on the fluorescently stained images of smear muscle fibers by the threshold segmentation algorithm. However, fluorescence imaging also has conspicuous shortcomings: the dye is toxic and the fluorescence has a quenching time for that long-term observation cannot be guaranteed. Hence it is necessary to resort to a safe and long-lasting imaging method for long-term live cell monitoring.

It is critical to judge the cell culture state by computer measurement of cell morphological parameters like cell fusion length, and diameter after receiving the images. Thus many open-source cell image processing tools have been developed like CellProfiler [9], Image J [10], Cytokit [11] and most of them integrate cell segmentation functionality because the result of image segmentation directly affects the accuracy of measurement. The segmentation algorithms of these tools typically call packages from algorithm libraries such as OpenCV and SciPy. The current mainstream cell image segmentation algorithms can be divided into two major categories: one is based on heuristic image algorithms such as watershed, region-growing, level-set, etc. [12,13,14,15]. Since these algorithms perform well on regularly shaped circular and elliptical cells as well as nuclei segmentation. Users often need to continuously adjust parameters for specific images and the segmentation performance for irregular and overlapping cells may not always meet expectations. The outstanding semantic segmentation algorithms typically belong to another category: neural network-based image segmentation algorithms [16], many of which are primarily based on convolutional neural networks (CNN), such as U-NET [17], FCN [18], QCANet [19], etc. Because of CNN’s ability to automatically extract image features, these types of algorithms perform exceptionally well in cell segmentation. Nevertheless, high-performance segmentation algorithms rely on a large number of pixel-level labels that are quite time-consuming and labor-intensive. The training process of the model counts on GPU resources which is costly as well. Hence it is still worth trying to perform unsupervised segmentation of images in order to save the above resources to the greatest extent.

In this study, we address the problem of non-contact, non-invasive imaging of muscle stem cell in vitro culture and propose an unsupervised semantic segmentation algorithm to extract the portion of the muscle precursor cell (MPCs) images that need to be measured. We design a set of bright field imaging equipment for long-lasting cell monitoring in situ directly through a petri dish for that neither additional marking nor smearing is required. Furthermore, an unsupervised algorithm is applied to segment the unstained MPCs images online. The algorithm performs well in unstained cell images in which targets and backgrounds are highly similar due to the colorless transparency of both cells and culture medium. The superiorities of the algorithm are as follows:

A new idea using the super-pixel segmentation results aggregated by the Laplacian operator to guide the neural network learning is provided that effectively locates the edges of the strip cells.

We adopt the strategy of executing while training, and directly segmenting the target online on a shallow CNN network thus high-performance hardware resources are not required. The parameter quantity of the proposed network is only 22.8 K while the number of the most trendy U-net and FCN-GoogLeNet is 7.76 M and 6 M, respectively.

The algorithm is completely unsupervised and does not require manual annotation, which effectively saves expensive labor. The work of collecting and labeling thousands of samples is avoided here as well as the several days to weeks of pre-training that mainly depends on the scale of the image data.

2. Materials and Methods

2.1. Cell Culture and Image Acquisition

The muscle stem cells (MSCs) used for in vitro proliferation and culture are mainly muscle-derived cells like muscle satellite cells discovered by Mauro A [20]. After the muscle satellite cells are activated, their unipolar or bipolar poles will stretch to form spindle-shaped MPCs which later fuse to form strip myotube [21]. The cells used in the experiment are obtained from the muscle tissue of the limbs of mice. The MSCs were firstly cultured in the medium of F10 + 20% FBS + PS for 2–3 days and then were washed three times with PBS once the cells proliferated to a certain density on the third day. After that, they were transferred into a differentiation medium and cultured for another 4 days. The pictures used for segmentation are taken at the end of the fourth day in which both MPCs and myotubes are existing and the cell confluency is about 40%. The region of MPCs and myotubes are both of interest and in this paper, the myotube images are used to refer to the acquired images.

The device used for filming is a self-designed in situ microscopic imaging system for cells. As shown in Figure 1, the Petri dishes are simply placed on the culture rack made of a hollowed-out acrylic plate, and a white LED panel designed as the lighting source is suspended above the culture rack. A camera fixed on a three-axis motor is arranged under the culture rack by which the cells in the culture medium are directly filmed through the transparent culture dish and the three-axis motor drives the camera to find the exact focus and realize the field of view switching. The camera used here is a Basler-aca2500 equipped with a 10X optical lens as shown in

Table 1. Parameters of imaging system.

	Camera Type	Magnification	Resolution (Pixel)	Size (cm)	Number of Culture Dish	Pixel Size (µm)
Parameter	Basler-aca 2500	10×	2592 × 1944	W45 × L45 × H30	6	2.2 × 2.2

.

2.2. Image Segmentation

The proposed algorithm is a neural network learning method based on superpixel segmentation. As shown in Figure 2, the main structure of the algorithm can be divided into two key parts: one is the process of generating a reference image in an automatic way, and the other part is the learning network. At the outset, a graph-based segmentation is adopted to obtain appropriate superpixels. Then the Laplacian operator for image enhancement is employed in the original cell image to assist in merging superpixel regions to obtain a reference image for the CNN network. The original image is then put into the CNN and the cross entropy loss between the output and the reference image is calculated and back propagated for the network to study. The network’s final output is the last epoch’s training outcome at the same time and the ultimate segmentation of the cell image is the network’s output blended with the superpixels region.

3. Experiment and Results

3.1. Superpixel

Superpixels refer to irregular pixel blocks with certain visual significance composed of adjacent pixels with similar texture, color, brightness, and other characteristics [22]. The superpixel blocks generated in a proper way instead of a large amount of single pixels can effectively abstract the image information in line with human visual characteristics so that the volume of the image data will be remarkably compressed in the subsequent processing.

There are two main kinds of superpixel segmentation algorithms. One is based on graph theory in which images are regarded as a set of weighted undirected graphs. As well, segmentation is performed by evaluating the adjacent relationship of pixels, such as Felzenszwalb’s segment algorithm [23], Normolized Cut [24], and Superpixel lattice [25]. The other is based on gradient descent and adopts the basic idea of clustering. The clusters are iteratively updated by gradient descent on the basis of a coarse clustering, like Watershed [26], Quick-shift [27], and SLIC [28]. Different superpixel segmentation algorithms perform differently and the performance of the same algorithm also varies from natural images to medical images [29]. We compared several superpixel segmentation algorithms on muscle cell images. The results are shown in Figure 3, the Felzenszwalb’s graph-based algorithm gives the best cell edge and obtains the highest boundary recall.

3.2. Image Enhancement

The grayscale of the cell and background are too close to distinguish cells from the background simply through gray values due to the similar transparency of cells and the culture medium. The size 6 of 10 × 10 pieces of the cell and the background from one grayscale image are cropped independently and the histogram is plotted as shown in Figure 4. Although both of them are Gaussian distribution, there are some deviations in means and variances which shows that the distribution of values inside the cells is relatively dispersed with richer details while the background part has relatively concentrated values and lacks rich texture. According to this difference between cells and background, the image enhancement operation in pre-processing is chosen rationally.

As a second-order differential operator, the Laplacian operator can emphasize regions of the picture with rapidly changing density, which is often used for boundary detection. There are more drastic changes inside the cells than the background, which is possible since the fact that cells are three-dimensional and do not have a uniform liquid surface as the culture medium. As shown in Figure 5, it is clear that the cellular region has more “boundaries” relative to the background region after filtering by Laplace operator.

In the actual processing, the original map is firstly Gaussian filtered to reduce noise, followed by the Laplace operator. After taking the absolute value of the Laplace image, the median filter and binarization are executed so that the cell region and the background region can be initially distinguished. The results of each step are shown in Figure 6 for the Gaussian blur, Laplace processing, median filtering, and the final binarized image.

3.3. Image Merging

Each superpixel is a collection of neighboring pixels that are related to each other, and the enhanced image based on the structure of the cellular optical features is non-contiguous. By traversing the pixel value of the corresponding enhanced binarized image in each superpixel and assigning the pixel value with the largest number to the superpixel as a category helps to merge the superpixel image to obtain a new binarized image as the reference image in the following network, the image fusion equations are as follows:

$$n=\sum I\_binary \left(d_i\right)$$

(1)

$$I\_label\left(d_i\right)=\left\{\begin{array}{c}1, if n\ge N \left(d_i\right)/2\\ 0, else \end{array}\right.$$

(2)

where

$d_{i }$ is the pixel range of the i-th superpixel block,

I_binary is the enhanced binarized image,

N is the number of pixels per superpixel block,

I_label is the new binarized image for reference.

The initial superpixel segmentation image and the obtained fused reference image is shown in Figure 7a,c, respectively.

3.4. Convolutional Neural Network

The fused reference results are applied to a CNN to further enhance the segmentation effect. The network adopts an end-to-end network structure as shown in Figure 8, with a total of 4-layer network structures, and the acquired 3-channel RGB cell images are put into the neural network, successively going through four layers.

Different from the general CNN model which first trains the model with a large amount of labeled data and then infers the new input image. The network parameters trained here by a single image online are used as inference at the same time and there is no necessary connection between different images. Each image needs to be trained online to generate its own unique network parameters and the set of network parameters is only suitable for the current image.

Figure 9 shows the comparison between the reference image and the learning result after training 20 epochs. Although the learning result of CNN is obtained according to the reference label, the adjusted result is significantly better than the reference label. The red boxed areas in Figure 9 shows that the reference image incorrectly segments the background area as the foreground area of the cells while the CNN segmentation results can distinguish the foreground cells from the background more finely.

This is because the designed network is mainly composed of convolution with small kernels (3 × 3 and 1 × 1). Convolution is a local operation that can more finely perceive the difference between pixels when extracting features. Coupled with the process that the network training and inference are performed with the current image, the network can fit the mapping relationship between the current image and the reference features better. The trained network will not be totally over-fitted due to the network’s generalization capabilities, and even if more epochs are trained, the loss will only converge to a certain number rather than plummeting to zero. The finer parts of the original image can be correctly distinguished rather than giving unimproved segmentation results that are exactly the same as the reference features.

3.5. Results and Analysis

In this part, online semantic segmentation is performed on the images acquired by the cellular in situ imaging device. The original image size of 2592 × 1944 is acquired and cropped to 1944 × 1944, which is then resized to 448 × 448 with the purpose of improving computing efficiency. Since the cells are in the process of differentiation and fusion, there are relatively large gaps in cell size and density in different culture petri, myotube images with different fusion orientations and densities are selected for presentation. The segmentation results are shown in Figure 10. Meanwhile, the intersection ratio and segmentation accuracy for each step are calculated to prove that the algorithm process is practical and effective in

Table 2. Performance of every single step of the proposed algorithm for different myotube images.

	No. 1			No. 2			No. 3			No. 4
	Enhancement	Reference	CNN	Enhancement	Reference	CNN	Enhancement	Reference	CNN	Enhancement	Reference	CNN
PA	0.80	0.86	0.89	0.81	0.87	0.90	0.77	0.84	0.88	0.80	0.85	0.88
mIOU	0.65	0.73	0.77	0.68	0.78	0.81	0.63	0.72	0.78	0.66	0.74	0.77

. The performances on myotube images of an algorithm which is also using a single image doing unsupervised semantic segmentation are compared with the proposed method in

Table 3. Performance of Asako Kanezak’s and the proposed algorithm.

	PA	mIOU	Param
Asako Kanezak’s	0.62	0.35	103.6 k
The proposed	0.89	0.78	22.8 k

.

Pixel accuracy (PA) and mean intersection-merge ratio (mIOU) are used to evaluate the performance of segmentation. Pixel accuracy indicates the proportion of correctly segmented pixels to the total pixels and the mean intersection-merge ratio is calculated as the average of the ratio of the intersection and merge of the two sets of true and predicted values for each category:

$$pixel accuracy=\frac{{{\displaystyle \sum }}_{i=0}^k{p}_{ii}}{{{\displaystyle \sum }}_{i=0}^k{{\displaystyle \sum }}_{j=0}^k{p}_{ij}}$$

$$mIOU=\frac{1}{k+1}{{\displaystyle \sum }}_{i=0}^k\frac{p_{ii}}{{{\displaystyle \sum }}_{j=0}^k{p}_{ji}-p_{ii}}$$

where

$p_{ij}$ is the number of pixels whose true value is i and is predicted to be j,

k + 1 is the total number of categories (including empty categories),

$p_{ii}$ is the number of pixels with the same predicted and true values,

$p_{ji}$ and $p_{ij}$ denote false positives and false negatives, respectively.

The number of parameters (Param) refers to the total amount of weight parameters of all layers with parameters in the model, which is used to describe the space complexity of the model. While counting the parameters’ number, non-linearly varying layers such as Relu are not included since they do not have parameters to learn.

The segmentation algorithm proposed by Asako Kanezak [30], which is also an unsupervised semantic segmentation algorithm with a single image as input and the network structure is lightweight as well, has achieved okay results on natural images. However, as shown in Figure 11 and Table 3, Asako Kanezak’s algorithm is not significantly more dominant in the segmentation of myotube images. The proposed algorithm with image preprocessing and the appropriate superpixel segmentation algorithm shows more significant performance after improving the network training inference mechanism.

4. Discussion

The non-destructive observation and morphological analysis of cells are the key technologies for the automated culture of muscle cells in vitro. In order to achieve long-term non-destructive monitoring of the cell culture process, a set of cell in situ observation tools that perform non-staining and non-smear imaging of muscle cells in culture is developed. Accurate segmentation of such unstained cell images is crucial for future assessments of morphological features. While automatic segmentation is greatly hampered by the irregular shape of muscle precursor cells and the resemblance of unstained cells to the backdrop, the proposed method utilizes Laplacian to increase the contrast between cells and background and make use of the coarse segmentation output of superpixels to produce images that resemble pseudo-labels. The algorithm’s prediction component adopts the strategy of executing while training. In order to guarantee the unsupervised segmentation impact of this lightweight network, tiny convolution kernels are utilized in the network to improve its segmentation effect on the current input image.

The proposed algorithm has relatively low hardware platform requirements and can be more readily implemented into different embedded platforms in the future precisely because of the network’s lightness, which also makes it possible to be applied on integrated platform gadgets like automated cell culture systems and high-content screening. The strategy of executing while training causes it to take a few seconds to determine the outcome of each input, which restricts its applicability in videos of rapidly moving cells. Another flaw is that the algorithm only does semantic segmentation at the moment, is unable to differentiate the contour lines of the connected sections of the cells and only able to distinguish between the cells and the background. In the future, we will concentrate on the optimization of the algorithm’s running speed and make an attempt at segmenting instances.

5. Conclusions

In this paper, we proposed a novel unsupervised segmentation algorithm based on a shallow CNN to perform the segmentation of mixed images of myocytes and myotubes in culture medium backgrounds. The reference image without manual annotation is generated with the aid of an imaging feature caused by the difference in cellular thickness and homogeneous culture medium while in-situ imaging. The final segmentation accuracy and the mIOU are about 88% and 77%, respectively. Meanwhile, the network does not need to be trained in advance so that the online output of the segmented results can be obtained directly from the input of a single image, Expensive computational resources are saved since the algorithm model is quite lightweight.