A Method for Identification of Multisynaptic Boutons in Electron Microscopy Image Stack of Mouse Cortex

Abstract

Recent electron microscopy (EM) imaging techniques make the automatic acquisition of a large number of serial sections from brain samples possible. On the other hand, it has been proven that the multisynaptic bouton (MSB), a structure that consists of one presynaptic bouton and multiple postsynaptic spines, is closely related to sensory deprivation, brain trauma, and learning. Nevertheless, it is still a challenging task to analyze this essential structure from EM images due to factors such as imaging artifacts and the presence of complicated subcellular structures. In this paper, we present an effective way to identify the MSBs on EM images. Using normalized images as training data, two convolutional neural networks (CNNs) are trained to obtain the segmentation of synapses and the probability map of the neuronal membrane, respectively. Then, a series of follow-up operations are employed to obtain rectified segmentation of synapses and segmentation of neurons. By incorporating this information, the MSBs can be reasonably identified. The dataset in this study is an image stack of mouse cortex that contains 178 serial images with a size of 6004 pixels × 5174 pixels and a voxel resolution of 2 nm × 2 nm × 50 nm. The precision and recall on MSB detection are 68.57% and 94.12%, respectively. Experimental results demonstrate that our method is conducive to biologists’ research on MSBs’ properties.

1. Introduction

Electron microscopy (EM) connectomics is an ambitious research direction aimed at studying comprehensive brain connectivity maps using high-throughput, nanoscale microscopes [1]. The development of EM technologies has greatly promoted the progress of brain science and connectomics. Although EM provides sufficient resolution to reveal the invaluable information about structures such as neurons, mitochondria, and synapses [2], higher resolution results in a multiplication of the data volume. As is known to all, it is time-consuming and difficult to annotate large volumes of data manually. Therefore, there is an urgent need to develop automated algorithms to process the structures in EM images.

Much effort has been devoted to developing automated algorithms for analyzing the EM data. One of the main application scenarios is neuron segmentation. Recently, Januszewski et al. [3] used a method called flood-filling to segment and trace neurons in a dataset obtained by serial block-face scanning electron microscopy (SBF-SEM) from a male zebra finch brain. It is a recurrent neural network (RNN)-based method unifying the two steps of finding the boundary between synapses with edge detectors or machine learning classifiers and combining image pixels that are not separated by boundaries using algorithms such as watershed or image segmentation. Due to the important role of mitochondria in cell function, researchers have attempted to quantify important mitochondrial properties in recent years. Vitaladevuni et al. [4] designed a boosting-based classifier for texture features to detect and segment mitochondria in EM images. Lucchi et al. [5] considered not only texture features, but also features on the shape of mitochondria. Using the graph cut model, they performed high-precision stereo segmentation of mitochondria on superpixels. Synapses also play an important role in the nervous system, which allows neurons to transmit an electrical or chemical signal to other neurons. Staffler et al. [6] reported an automated detection method for synapses from conventionally en-bloc stained 3D electron microscopy image stacks, SynEM. It is based on a segmentation of the image data and focuses on classifying borders between neuronal processes as synaptic or non-synaptic. Xiao et al. [7] proposed a deep learning-based method for synapse 3D reconstruction, where the Dijkstra algorithm and GrabCut algorithm were used to segment the synaptic cleft.

In addition to the several structures just mentioned, multisynaptic boutons (MSBs) are also worthy of study. It has been proven that MSBs are closely related to sensory deprivation, brain trauma, and learning [8,9,10]. MSBs are boutons that make synaptic contacts with multiple postsynaptic structures. They were first seen in cat brain exactly 50 years ago [11] and have been observed in mice, rats, rabbits, cats, and monkeys [8,12,13,14,15,16]. Previous work showed that the formation of MSB can be induced by visual sensory deprivation, an enriched environment, brain lesion, motor skill learning, and auditory associative learning [12,13,14]. Dendritic spines are tiny protrusions from neuronal dendrites and form the postsynaptic component of the synapse in the brain [17,18,19,20]. Increasing the area or number of dendritic spines can result in more efficacious synaptic transmission and thus enhance the strength of neuronal connections [12,21,22]. Both dendritic spines and shafts can form MSBs [17,23]. Long-term potentiation (LTP), a major form of synapse plasticity, can result in increased spine number and elevated proportion of MSBs in rat hippocampus [12], consistent with the idea that MSB may represent a strengthened form of synaptic connection. Therefore, analyzing the structure and connectivity of MSBs is critical in understanding sensory experience- and learning-associated synaptic plasticity.

As mentioned above, MSBs are presynaptic boutons that are in contact with multiple postsynaptic structures. Unlike the single-synapse boutons, the special structure of MSBs determines that we have to identify them step by step. Inspired by the previous work focusing on bio-electron image processing, we propose an efficient way to identify the MSBs from serial EM images. We first use a CNN-based (convolutional neural network) algorithm to detect and segment the synapses. An effective algorithm for filtering the pseudo-synapses and the missed synapses is then used to optimize the results on synapses. Meanwhile, we design a residual network to predict the neuronal membrane and further obtain the segmentation of the neurons with an improved watershed-based algorithm. Based on the information of synapses and neurons obtained in the above steps, it is reasonable to identify the MSBs on serial EM images. Our method manages to automate MSBs’ detection and recognition, which will provide a powerful tool for neuroscience research on synaptic plasticity associated with learning and memory.

2. Materials

The biological specimen in this paper is mouse cortex (provided by the Institute of Neuroscience, Chinese Academy of Sciences). Automated tape-collecting ultra-microtomy scanning electron microscopy (ATUM-SEM) was used to obtain the image stack of the mouse cortex specimen of a volume of 12 µm × 10.35 µm × 8.9 µm (performed at the Institute of Automation, Chinese Academy of Sciences). The image stack consisted of 178 serial images with a size of 6004 pixels × 5174 pixels and a voxel resolution of 2 nm × 2 nm × 50 nm. Figure 1 presents the images and ground truth on the adjacent sections. The ground truth was manually labeled by three well-trained graduate students with cross-validation. A total of 1230 synapses were annotated for the whole image stack, and the neuron membrane of the former 5 images was annotated. The software FIJI was used for annotation, and the interface is shown in Figure 2. The database and the manually-labeled ground truth are available on the website (http://95.163.198.142/MiRA/synapse_deng/).

3. Methods

The main steps of MSBs detection are shown in Figure 3. Firstly, histogram equalization was performed on the image stack. Based on the equalized images, we located and segmented the synapses with the mask R-CNN. The contextual information was also applied to removing the false synapses and to make up the missed synapses. Meanwhile, we used a deep network to obtain the probability map of the neuronal membrane, from which we obtained the neuron segmentation with an improved marker-controlled watershed segmentation algorithm. Finally, the MSBs could be located by using the information of synapses and neurons.

3.1. Image Preprocessing

In order to reduce the effects of illumination and other factors on detecting and segmenting synapses and neurons, we transformed the intensity of the raw image to make the histogram equalized [24,25]. For each raw image $I_{raw}$, we first counted the gray-value histogram of the image.

$$h(r_k)=n_k,$$

(1)

where $r_k$ is the $k^{\mathrm{th}}$ gray scale ($k=0,1,\cdots ,255$) and $n_k$ is the number of pixels with a gray value of $r_k$. Then, we used the following transformation,

$$P(r_k)=n_k/n,$$

(2)

where $n={\displaystyle \sum _{k=1}^L}{n}_k$.

From Figure 4, we can notice that the contrast between foreground (synapses, the membrane of neurons) and background was more significant on the processed image.

3.2. Recognition of Synapse

In this section, we show the procedures for synapse detection and segmentation. Inspired by Hong et al. [26], we first adopted the MaskR-CNN (region-CNN) [27] to detect and segment synapses on the serial EM image stack. Then, an algorithm for filtering the pseudo-synapses and locating the missed synapses was used to rectify the preliminary results of synapses.

3.2.1. Detection and Segmentation with Mask R-CNN

The main idea of Mask R-CNN is to extend the original FasterR-CNN and add a branch to use the existing detection to predict the target in parallel. The architecture of the proposed network is illustrated in Figure 5. The first module is a fully-convolutional network (FCN) for extracting features over the input images. It outputs a feature map for each input image, where the foreground is more prominent. The next module, region proposal network (RPN), then generates region proposals over each feature map. It is a small network that slides over the feature maps. In the field of view of the sliding window, different possible regions (anchors) of different sizes are generated. All anchors are classified into foreground and background, where partial anchors belonging to the foreground are used to optimize regression parameters to correct the bounding boxes. The rectified foreground anchors that have great possibilities of containing targets are called “proposals”. Through an RoIAlign layer, the proposals on the corresponding feature maps are mapped into a fixed size. The fixed-size RoIs are imported in a network that consists of three branches, the classification branch, regression branch, and mask branch. The classification branch gives the probability that the object belongs to each class, while the regression branch fixes the position of the bounding box of the proposal once again. The mask branch is a small FCN applied to each RoI, predicting a segmentation mask.

3.2.2. Rectifying Detection Results of Synapses on the Serial EM Image Stack

Confirming the connection of objects on serial images in 3D is helpful in rectifying the detection results [28]. We present an algorithm for confirming the connection of synapses in 3D. We denote by $P_{i,j}$ the $j^{\mathrm{th}}$ synapse on the $i^{\mathrm{th}}$ section, in which the index j is randomly given. The algorithm includes the following steps:

• For the synapse on the first section $P_{1,j}$, we assign it a 3D serial number $N_{1,j}^{3d}=j$.
• For the synapse on the $i^{\mathrm{th}}$ section $P_{i,j},(i>2)$, we calculate the Euclidean distance between $P_{i,j}$ and ${\{P_{i^{\prime }}\}}_{i^{\prime }=i-3}^{i-1}$. Denote by ${\mathcal{D}}_{j,j^{\prime }}^{i,i^{\prime }}$ the distance between $P_{i,j}$ and $P_{i^{\prime },j^{\prime }}$:

  $${\mathcal{D}}_{j,j^{\prime }}^{i,i^{\prime }}={\parallel c_{i,j}-c_{i^{\prime },j^{\prime }}\parallel }_2,$$

  (3)

  where $c_{i,j}$ is the centroid of the bounding box of $P_{i,j}$.
• Find the closest synapse to $P_{i,j}$, and denote it by $P_{i^{″},j^{″}}$.

  $$\underset{i^{″},j^{″}}{\mathrm{argmin}}\{{\mathcal{D}}_{j,1}^{i,i-1},{\mathcal{D}}_{j,2}^{i,i-1},\cdots ,{\mathcal{D}}_{j,1}^{i,i-3},{\mathcal{D}}_{j,2}^{i,i-3},\cdots \}.$$

  (4)
• Verify if $P_{i,j}$ is the closest synapse to $P_{i^{″},j^{″}}$ on the $i^{\mathrm{th}}$ section. Find the closest synapse to $P_{i^{″},j^{″}}$ on the $i^{\mathrm{th}}$ section, and denote it by $P_{i,j^{‴}}$.

  $$\underset{i,j^{‴}}{\mathrm{argmin}}\{{\mathcal{D}}_{j^{″},1}^{i^{″},i},{\mathcal{D}}_{j^{″},2}^{i^{″},i},\cdots \}.$$

  (5)
  If $j^{‴}=j$ and the distance between $P_{i,j}$ and $P_{i^{″},j^{″}}$ is smaller than a given threshold $\theta$ (according to the thickness of sections of 50 nm, i.e., 25 pixels in the x-y direction, we set $\theta =100$),

  $${\mathcal{D}}_{j,j^{″}}^{i,i^{″}}\le \theta ,$$

  (6)

  we consider that $P_{i,j}$ and $P_{i^{″},j^{″}}$ are the same synapse appearing on different sections. Then, we assign the 3D serial number of $P_{i,j}$ to $P_{i^{″},j^{″}}$. If $j^{‴}\ne j$ or Equation (6) is not satisfied, we consider that $P_{i,j}$ and $P_{i^{″},j^{″}}$ are not the same synapse in the 3D perspective. We assign a new 3D serial number to $P_{i^{″},j^{″}}$.

With the algorithm for assigning the 3D serial number to synapses on the serial image stack, we give each synapse on the serial image stack a 3D serial number. The same synapses appearing on different sections have the same 3D serial number. Based on this, it is easy to count the number of synapses in the 3D perspective. A synapse is a spatial structure with a size of about 200 nm [7]. Therefore, one synapse should appear on at least 3 adjacent sections under the condition that the thickness of the section is 50 nm. We deleted the synapses that appeared on less than 3 layers, which were considered as pseudo-synapses. Meanwhile, in order to avoid the inaccuracy of synapse statistics caused by synapses that were missed, as shown in Figure 6, where the yellow rectangle indicates missed, while the red rectangles indicate correct detection, we also needed to locate the synapses that were missed. For the synapses with the same 3D serial number, if the layer numbers of the sections they are in was incoherent, it was likely that there were missed synapses on the the section with a discontinuous serial number.

3.3. Segmentation of Neuron

In this section, we introduce the procedures to obtain the segmentation of neurons. Firstly, a well-designed deep neural network is trained to output the probability map of the neuronal membrane. Then, a series of morphological operations are performed on the probability map to obtain neuron segmentation.

3.3.1. Probability Map of the Neuronal Membrane

For the recognition of the neuronal membrane, an efficient contextual residual network [29] was used in this study. It was first used on the public dataset ISBI (International Symposium on Biomedical Imaging) 2012, and the experimental results showed its effectiveness in the recognition of the neuronal membrane on EM images. The schematic diagram showing the training/testing process of the network that outputs the probability map of neuronal membrane is shown in Figure 7. The network consisted of two main parts: a ResNet38-like module and an expansive module. As for the ResNet38-like module, the first improvement is that a maximum pooling layer was added after the first convolutional layer, which greatly reduced the number of parameters. Following the max-pooling layer were 8 residual units, which can be divided into 6 blocks. Block 1 and Block 2 was composed of two residual units, respectively, each of which consisted of two convolutional layers with a kernel size of $3\times 3$. By setting the stride of kernels on the first convolutional layer of Block 1 and Block 2 to 2, they downsampled the image to 1/4 of the original size. For Block 3 and Block 4, each consisted of two convolutional layers. Correspondingly, Block 5 and Block 6 were composed of three residual units, respectively. In order to cope with the problem that the distribution of the training set was inconsistent with the distribution of the prediction set, that is the internal covariate shift (ICS) phenomenon, we used mini-batch normalization before the exponential linear units (ELUs) so that the result (each dimension of the output) had a mean of 0 and a variance of 1, which helped to increase the convergence speed and improve the prediction results. For the feature maps outputted by the max pooling layer, Block 1 and Block 6 were upsampled by fractional strided convolution with channels 64, kernel size $2N\times 2N$, and stride N ($N=2,4$, and 8 for upsampling layers, respectively). The global information and local cues from different scales were then incorporated by summation-based skip connections. Unlike concatenation-based skip connections, summation-based skip connection provides a more thorough integration of multi-scale context cues and overcomes the vanishing gradient problem more efficiently. Finally, two convolutional layers and dropout $(p=0.5)$ were used to refine the pixel-wise prediction and improve generalization ability.

3.3.2. Neuron Segmentation with the Marker-Controlled Watershed Algorithm

This section shows how to use a modified watershed-based segmentation method to separate neurons in a 2D EM image, and the workflow is shown in Figure 8. Based on the probability map of the neuronal membrane outputted from the network described in the above subsection, we first obtained the foreground marker. Then, we used watershed segmentation to separate the neurons under the guidance of the rectified foreground marker.

The watershed transform finds “catchment basins” and “watershed ridge lines” in an image by taking it as a surface where light pixels are high and dark pixels are low [30]. It is a natural way to segment neurons with the watershed-based algorithm under the hypothesis that the neurons and membranes are “catchment basins” and “watershed ridge lines”, respectively. Segmentation using the watershed transform works better if we can identify, or “mark”, foreground objects and background locations. Therefore, to make the segmentation result more accurate, we extracted a part of each neuron as the foreground mark. Here, we do not give the background mark. For each probability map of the neuronal membrane I (as shown in Figure 8A) whose pixel values in I varied from 0–255, we used the modified marker-controlled watershed segmentation following these basic procedures.

Firstly, we computed its complement $I^{\prime }$ (as shown in Figure 8B):

$$I^{\prime }=255-I.$$

(7)

Next, we found the markers of the foreground objects. These markers should be connected speckle pixels inside each of the neurons. We used opening-by-reconstruction and closing-by-reconstruction to make the internal part of each neuron “flatter” and denoted by $I_{obrcbr}^{\prime }$ the clean image. The opening-by-reconstruction is an erosion followed by a morphological reconstruction, while the closing-by-reconstruction is a dilation followed by a morphological reconstruction. Reconstruction-based opening and closing are more effective than standard opening and closing at removing small blemishes without affecting the overall shapes of the objects. The dark spots and stem marks on the probability map of the neuronal membrane can be removed efficiently by performing these two operations. The detailed process is shown in Equation (8):

$$\begin{array}{c}{I}_e^{\prime }=I^{\prime }\ominus se,\hfill \\ I_{obr}^{\prime }=I_e^{\prime }\odot I^{\prime },\hfill \\ I_{obrd}^{\prime }=I_{obr}^{\prime }\oplus se,\hfill \\ I_{obrcbr}^{\prime }=255-(255-I_{obrd}^{\prime })\odot (255-I_{obr}^{\prime }),\hfill \end{array}$$

(8)

where $\ominus ,\oplus ,\odot$ are the erosion, dilation, and reconstruction operations, respectively. $se$ is a disk-shaped structuring element with a radius of 10 pixels. The regional maxima of $I_{obrcbr}^{\prime }$ are foreground markers. We denoted by $I_{fgm1}^{\prime }$ (as shown in Figure 8C) the binary image of the foreground markers. To prevent the foreground markers in some objects from reaching the edge of the object, we shrunk them by a closing followed by an erosion:

$$\begin{array}{c}{I}_{fgm1}^{\prime }=regionalMaxima(I_{obrcbr}^{\prime }),\hfill \\ I_{fgm2}^{\prime }=I_{fgm1}^{\prime }•s{e}_2,\hfill \\ I_{fgm3}^{\prime }=I_{fgm2}^{\prime }\ominus s{e}_2,\hfill \end{array}$$

(9)

where • is the close operation and $s{e}_2$ is a disk-shaped structuring element with a radius of 5 pixels. In addition, some connected components that have fewer than a certain number of pixels should also be removed because this algorithm tends to leave some stray isolated pixels. We denoted by $I_{fgm}^{\prime }$ (as shown in Figure 8D) the binary image of the rectified foreground marker.

Then, as described in Equation (10), we took the probability map I as the gradient magnitude image $I_{grad}$ and modified it using morphological reconstruction, so only the regional minima where $I_{fgm}^{\prime }$ is nonzero were preserved.

$$\begin{array}{c}{I}_{grad}=I,\hfill \\ I_{grad2}=I_{grad}\odot I_{fgm}^{\prime }.\hfill \end{array}$$

(10)

Lastly, we can obtain the segmentation of I by performing the watershed transform on the modified gradient magnitude image $I_{grad2}$:

$$I^{label}=\mathrm{watershed}(I_{grad2}).$$

(11)

As can be seen from Figure 8E, the width of the boundary indicated by this red line was only one pixel. This was caused by the fact that we only gave the foreground mark without giving the background mark. This helps to integrate the synaptic segmentation information to locate multiple sites.

3.4. MSB Identification

As shown in Figure 9, the MSBs can be positioned logically by incorporating the segmentations of neurons and synapses. For each synapse, we can know the labels of pre- and post-synaptic neurons by projecting the bounding box of the synapse onto the labeled segmentation of neurons. According to the morphological characteristics of MSB, it can be concluded that the neuron shared by several synaptic clefts was an MSB. Firstly, we located each synapse on the labeled segmentation of neurons. The two neurons with maximum frequency labels were considered as pre- and post-synaptic neurons corresponding to the synapse. Naturally, a neuron corresponding to multiple synapses, that is a neuron whose label frequency was more than 2 times, was considered to be an MSB.

4. Results

To demonstrate the effectiveness of the proposed algorithm, we counted the number of detected MSBs from our dataset and compared the results with the manually-labeled ground truth. The measurements we selected to evaluate the results of synapse detection were Precision and Recall.. Precision, also referred to as the positive predictive value, is the fraction of relevant instances among the retrieved instances, while $Recall$ is the fraction of relevant instances that have been retrieved over the total amount of relevant instances:

$$\mathit{Precision}=\frac{\mathrm{True}\mathrm{Positives}}{\mathrm{True}\mathrm{Positives}+\mathrm{False}\mathrm{Positives}},$$

(12)

$$\mathit{Recall}=\frac{\mathrm{True}\mathrm{Positives}}{\mathrm{True}\mathrm{Positives}+\mathrm{False}\mathrm{Negatives}}.$$

(13)

This study adopted the Jaccard similarity [31] as the evaluation measurement for segmentation, which is defined as the size of the intersection divided by the union of the two segmentation results in the field of image processing:

$$J(S_i,S_j)=\frac{|S_i\cap S_j|}{|S_i\cup S_j|},$$

(14)

where $S_i$ and $S_j$ represent the ground truth and segmentation, respectively. Pixel-error and Rand-error [32] were used for evaluating the segmentation results of neurons. Pixel-error is the ratio of the number of misclassified pixels to the total number of pixels. Rand-error is the difference between one and the $Rand$-$index$. The $Rand$-$index$ is a similarity evaluation method for two clusters that can be used to measure segmentation due to the fact that segmentation can be regarded as a cluster of pixels.

$$\mathit{Pixel-error}=\frac{\mathrm{Number}\mathrm{of}\mathrm{False}\mathrm{Positive}\mathrm{Pixels}+\mathrm{Number}\mathrm{of}\mathrm{False}\mathrm{Negative}\mathrm{Pixels}}{\mathrm{Number}\mathrm{of}\mathrm{Pixels}},$$

(15)

$$\mathit{Rand-error}=1-\mathit{Rand-index}=1-\frac{N_1+N_2}{C_n^2},$$

(16)

where n is the number of pixels, $N_1$ is the number of pairs of pixels that are in the same connected object in the ground truth and in the same connected object in the segmentation results, and $N_2$ is the number of pairs of pixels that are in the different connected objects in the ground truth and in the different connected objects in the segmentation results

Due to the limitation of computer memory, we split each original image of size 6004 pixels × 5174 pixels into nine sub-images with a size of 2048 × 2048 in the experiment for training the Mask R-CNN. The overlapping pixels between adjacent sub-images in the x and y directions were 485 and 78, respectively.

For the 1602 sub-images, 630 sub-images were for training, while 270 sub-images were for verification, and the other 702 sub-images were for testing. The neuronal membrane of the former five images was manually annotated. The images with a size of 6004 pixels × 5174 pixels were split into sub-images with a size of 1024 pixels × 1024 pixels, each of which was further split into four images with a size of 512 pixels × 512 pixels. In total, 720 sub-images with a size of 512 pixels × 512 pixels were obtained, 450 sub-images for training, 150 sub-images for validation, and 120 sub-images for testing. To improve the robustness of the networks, we enlarged the training dataset by means of data augmentation, including rotation and flipping. In addition, reliability issues arise when the distribution of input training data differs from the distribution of the evaluation model [33]. Therefore, we randomly added Gaussian noise to the raw image (mean = 0, variance = 0.1) to make the model more robust during the training phase. The training and testing tasks were conducted on a server equipped with an Intel i7 CPU with 512 GB of main memory and a Tesla K40 GPU.

To illustrate the effectiveness of Mask R-CNN in the detection of synapses, we show the detection results of synapses using AdaBoost. From Figure 10, we can notice that the precision and recall were about 10.00% and 95.15%, respectively, under the default threshold (the default value of the minimum number of adjacent rectangles in which the target was detected was three). Besides, we show the comparison of the detection results on synapses in Figure 10 to illustrate the effectiveness of our algorithm for assigning a 3D serial number to synapses on the serial image stack and the algorithm for deleting pseudo-synapses and locating missed synapses. In total, we detected 13,036 synapses on the serial image stack. We found that there were 2027 synapses that only appeared on one section, which were considered as pseudo-synapses. With the algorithm for deleting pseudo-synapses and locating missed synapses, we removed these pseudo-synapses and located 445 missed synapses. As shown in Figure 10, the recall rate was increased from 97.15%–97.64%, and the accuracy rate was increased from 54.30%–62.12%.

For the comparison of synapse segmentation results, the morphology-based method and a variational model-based method [34,35,36] were selected. As shown in Figure 11, the mask R-CNN was superior to the other two methods in maintaining the basic shape and edge smoothness and was basically consistent with the manually-annotated ground truth. The Jaccard similarity of the segmentation results of synapses is shown in

Table 1. Comparison of synapse segmentation.

	Morphology Method	Variational Model Method [34,35,36]	Mask R-CNN
J(Synapses, Ground truth)	19.21%	18.49%	65.55%

. For the comparison of the segmentation results of neurons, the neuronal membrane prediction results of ResNet50 [37] are shown in Figure 12. Obviously, our approach was superior to ResNet50 in maintaining the neuronal membrane. The Pixel-error and Rand-error [32] of the segmentation results of neurons are shown in

Table 2. Comparison of neuron segmentation.

	Our Method	ResNet50 [37]
Pixel-error	5.61%	7.81%
Rand-error [32]	12.74%	27.34%

. In this way, the accuracy of the segmentation was guaranteed when the neurons were segmented based on the probability map of the neuronal membrane.

We list the number of misdetected MSBs and missed MSBs in 10 images in

Table 3. The numerical analysis of the identified results of MSBs in 10 images.

Image	Manual	Our Method
		Total	False Positive	False Negative
Layer 1	8	10	2	0
Layer 21	7	10	3	0
Layer 41	6	7	1	0
Layer 61	3	5	2	0
Layer 81	3	4	2	1
Layer 101	4	7	3	0
Layer 121	5	6	2	1
Layer 141	4	7	4	1
Layer 161	8	11	3	0
Layer 178	3	3	0	0
Average	5.1	7	2.2	0.3

, where the first image is from the training set and the others images are from the test set. Figure 13 shows the identification results of MSBs on the 121^th image. There were six MSBs identified, in which one MSB was missed (Figure 13A) and two MSBs were misidentified (Figure 13C). Figure 13B shows one of the four correctly-identified MSBs. The identified results on the whole image stack are available on the website mentioned above. According to our statistics, each section of a size of 12 µm × 11.3 µm from mouse cortex had about 5.1 MSBs. The precision and recall of our method on MSBs’ identification were 68.57% and 94.12%, respectively.

In Figure 14, we show three types of MSBs: Type A, one bouton to two synapses; Type B, one bouton to three synapses; and Type C, one bouton to four synapses. From left to right are raw images, bouton segmentations, synapse segmentations and the superimposed images of synapse segmentations and the manually-labeled ground truth of synapses. In the superimposed images, the yellow area is the overlapping area of segmentations and the manually-labeled ground truth, containing correctly-predicted pixels. The red area and green area consist of misclassified pixels and missing pixels, respectively. To evaluate the segmentation performance, we computed the Jaccard similarity between the synapse segmentations and the manually-labeled ground truth of synapses, as shown in

Table 4. Statistics of MSBs in Figure 14.

	J(Synapses, Ground truth)	Area(Neuron) (µm2)	Area(Synapse) (µm2)	Area(Neuron)/Area(Synapses)
A	62.54%	0.2050	0.0266	12.98%
B	57.48%	0.3634	0.0334	9.20%
C	48.40%	0.6497	0.0048	7.40%

.

Table 5. Statistics of different types of MSBs.

	Type A	Type B	Type C	Total
Numbers	946	50	7	1003
Ratio	94.31%	4.99%	0.7%	100%

shows the statistics of different types of MSBs: approximately 94.31% axon-two postsynaptic sites (dendrite spines or shafts), 4.99% axon-three postsynaptic sites (dendrite spines or shafts), and 0.70% axon-four postsynaptic sites (dendrite spines or shafts). The morphological changes in MSBs can induce pattern and functional strength alterations in neural connections, such as synchronous activity, learning, and memory [12,38]. Apart from the percentage ratio of different MSBs’ profiles, the area of the synapse, especially the presynaptic bouton and PSD (postsynaptic density) along the portion of the plasma membrane, are a characteristic feature of the synapses. The area of the presynaptic axon boutons and PSD are also shown in Table 4. The average area of the presynaptic axon boutons and PSD was 0.5731 µm² and 0.0121 µm², respectively. A previous study indicated that structural alterations of the presynaptic boutons and the PSD led to sustained changes in neurotransmitter release, which played an important role in synapse-specific forms of plasticity, as well as neuropathy [39]. Deep learning method application in MSBs’ detection and recognition will be a useful tool in neuroscience study. $\mu$ µ

5. Discussion

As shown in Figure 15, three types of MSBs were identified in mouse cortex: (1) axon-multiple dendritic shaft synapse, a single axon presynaptic terminal forming synapses with two or more postsynaptic dendritic shafts; (2) axon-multiple dendritic spine synapse, a single axon presynaptic bouton forming synaptic contact with two or more postsynaptic dendritic spines; (3) axon-dendritic shafts-dendritic spine, including a single axon terminal site forming synapses with one or more dendritic shafts and one or more dendritic spines. Kim et al. suggested that different types of MSBs have disparate effects on cerebellar synaptic transmission [40]. Therefore, it makes sense to classify the MSBs in more detail. For example, in many cases, it is necessary to analyze different groups of biological tissue data grouped by age or drug variables to further explore the role of different factors. Our method can be used to analyze the differences in the number or distribution of MSBs in different groups. Especially when the amount of data is large, the advantages of this automatic method are more obvious. Except for ATUM-SEM, current mainstream EM technologies include serial block-face scanning electron microscopy (SBF-SEM), serial section transmission electron microscopy (ssTEM), and focused ion beam scanning electron microscopy (FIB-SEM). Although each of these methods has its pros and cons, they all provide sufficient resolution to reveal invaluable information about structures such as neurons, mitochondria, and synapses. Therefore, our approach is also suitable for EM images of these types and data from other biological tissues.

Though the recognition accuracy of our method was acceptable under the premise of a high recall rate, which is of great help to biologists’ research on MSBs, the proposed method will also fail in some cases. As shown in Figure 16, the MSBs were incorrectly identified in the same area of three adjacent images. Due to the misdetection of synapses on more than two serial images, our algorithm for deleting pseudo-synapses and locating missed synapses failed to remove pseudo-synapses and even mistakenly added synapses on the image “Section 122”. This was caused by the fact that the edges of soma were too similar in morphology to the synapses, which made them difficult to distinguish. In order to solve this problem, we needed to improve the robustness of the algorithm in synaptic detection. Similarly, over-segmentation or under-segmentation of neurons can also lead to missed or misdetected MSBs.

As mentioned above, our method relied on synaptic detection and neuron segmentation, so improving algorithm performance was one of our directions for improvement. Meanwhile, two networks that handled synapses and membrane structures separately were trained in this study. From the perspective of complexity, if the two networks can be combined into one, that is designing a multi-class segmentation network, the efficiency of the recognition algorithm would be greatly improved. The difficulty is that the synaptic cleft is located on the membrane of the neuron and is very prone to confusion.

6. Conclusions

In this paper, we presented an approach for identifying MSBs on a serial EM image stack of mouse cortex. According to the morphological characteristics of MSB, synapses and neurons were segmented in parallel on each image. For the detection and segmentation of synapses, Mask R-CNN was used followed by an algorithm for filtering the pseudo-synapses and making up the missed synapses. For neuron segmentation, an effective residual network was first used for predicting the neuronal membrane. Then, a morphology-based method was adopted to obtain segmentations of neurons from the probability map of the neuronal membrane outputted from the previous step. The MSBs, finally, were identified by incorporating the segmentations of synapses and neurons. The experimental results showed that the accuracy of recognition was acceptable under the premise of a high recall rate, which is of great help to biologists’ research on MSBs. The application and directions for improvement of the proposed method were also discussed.