Super-Resolution Image Reconstruction Based on Single-Molecule Localization Algorithm

Abstract

Fluorescence imaging is an important and efficient tool in cell biology and biomedical research. In order to observe the dynamics of biological macromolecules such as DNA, RNA and proteins in live cells, it is extremely necessary to surpass the Abbe diffraction limit in microscopic imaging. Single-molecule localization microscopy (SMLM) is a sort of super-resolution imaging technique that can obtain a large number of images of sparse fluorescent molecules by the use of photoswitchable fluorescent probes and single-molecule localization technology. The center positions of fluorescent molecules in the images are precisely located, and then the entire sample pattern is reconstructed with super resolution. In this paper, we present a single-molecule localization algorithm (SMLA) that is based on blind deconvolution and centroid localization (BDCL) method. Single-molecule localization and image reconstruction of 15,000/9990 frames of original images of tubulins are accomplished. In addition, this fluorophore localization algorithm is used to localize high particle-density images. The results show that our BDCL-SMLA method is a reasonable attempt and useful method for SMLM imaging when the imaging system is unknown.

1. Introduction

Optical microscopy is widely used for imaging in life science with the advantages of non-invasion, high penetrability and non-destruction to biosamples. However, due to the Abbe diffraction limit [1], the spatial resolution of traditional optical microscope can only reach wavelength scale, usually 200 nm~500 nm, which enables the study of dynamic events and the fine structural details of cellular architecture, but limits its applications in studying structure, function and interaction of biological macromolecules such as DNA, RNA and proteins at nano level.

In the past decades, several imaging strategies to break the diffraction limit, including stimulated emission depletion microscopy (STED) [2], single-molecule localization microscopy (SMLM) [3,4] and super-resolution structured illumination microscopy (SIM) [5], have been developed with the progress of novel fluorescent probes and imaging theories. Their inventors, E. Betzig, S.W. Hell and W.E. Moerner were awarded the 2014 Nobel Prize “for the development of super-resolved fluorescence microscopy”. (Fluorescence) Photoactivated localization microscopy ((f)PALM) [3,6] and (direct) stochastic optical reconstruction microscopy ((d)STORM) [4,7] are two kinds of single-molecule localization microscopies, which employ photoswitchable fluorescent probes, localize single, blinking molecules in an extended image sequence and then produce a precise image. The spatial resolution of SMLM depends mainly on the number of detected photons from each molecule and can achieve approximately 20 nm or better [8,9,10,11].

It is concluded that obtaining a final super-resolution reconstructed image requires the following two steps: (1) Record the fluorescence signal of fluorophores with a detector; (2) Obtain the super-resolution reconstructed image by data analysis technique. As the key technology in SMLM, the data analysis technique, namely the single-molecule localization algorithm (SMLA), is extremely meaningful to obtain super-resolution images. Many algorithms have been currently developed [12,13], including nonlinear least squares (NLLS) [9,14], maximum likelihood estimate (MLE) [8,14], centroid method [13], Gauss fitting [13], gradient fitting [15] and fluoroBancroft (FB) method [16], etc. Among these algorithms, MLE, Gauss fitting and gradient fitting are considered more accurate, while centroid and fluoroBancroft methods are much faster in the localization of molecules [12,15]. Until now, scientists are still making unremitting efforts to improve the performance of localization algorithms in super-resolution imaging technology [17,18,19,20].

In order to localize a fluorophore precisely, it would be best to have a direct knowledge of the experimentally determined parameters such as point spread function (PSF) and background noise of the dataset to be processed. However, in many cases those factors are unknown to us, which brings many difficulties in reconstructing the image with super resolution. In this paper, we present a blind deconvolution and centroid localization-based single-molecule localization algorithm (BDCL-SMLA) that combines maximum likelihood estimation method and centroid method to resolve the problem mentioned above. Moreover, the BDCL-SMLA method can achieve the compatibly of both high accuracy and fast calculating speed. In order to prove the validity of the proposed method, single-molecule localization and image reconstruction of 15,000/9990 frames of original datasets of tubulins are accomplished. In addition, this algorithm is used to localize high particle-density images. The reconstructed images show the effectiveness and practicability of the BDCL-SMLA method.

2. Materials and Methods

In this section, the reference datasets we used are introduced and the BDCL-SMLA method is illustrated in detail.

2.1. Image Formation and Noise Models

The reference datasets that we used in this paper come from Suliana Manley’s laboratory: http://bigwww.epfl.ch/smlm/datasets/index.html (10 March 2021). The three experimental datasets are as follows:

(1)

Tublins, long sequence, 15,000 frames of 64 × 64 pixels, pixel size 100 nm;

(2)

Tubulin AF647, a fixed cell, stained with mouse anti-alpha-tubulin primary antibody and Alexa647 secondary antibody, 9990 frames of 128 × 128 pixels, pixel size 100 nm;

(3)

Tublins, high density, 500 frames of 64 × 64 pixels, pixel size 100 nm.

To estimate a fluorophore’s position, fitting models are assumed typically as this form [10]:

$$I(x,y)=I_{0}h(x-x_0,y-y_0)+b.$$

(1)

where h is PSF of the imaging system that describes the shape of the blur formed when a point source is imaged and is proportional to the average number of photons at a given position relative to the source; I₀ is the peak intensity and is proportional to the photon emission rate and the single-frame acquisition time; (x₀, y₀) are the fluorophore’s coordinates; b is the average background per pixel in the experiment. In most fitting algorithms, all four (or more) parameters (x₀, y₀, I₀, b, etc.) are estimated.

High-accuracy calculation of PSF requires accounting for numerous factors, particularly the collection angle (numerical aperture) of the lens, interfaces between the sample and lens (such as coverslips and immersion oil) and the dipole moment of the light source. Noise is inevitable in the image acquisition due to the effects of temperature, out-of-focus fluorophores or scattered light, shot noise of the photons in the spot and read-out noise of a detector in the process of converting an optical signal to an electrical signal. All of these factors may degrade the image quality.

2.2. Blind Deconvolution and Image Restoration

A degraded image is the deterioration in image quality during the process of image formation, propagation and preservation. The degradation of the experimental images we used in this paper may be caused by the aberration of the optical imaging system, the relative motion of the imaging system and the fluorophores, the overlap of the fluorophore diffuse spots, the distortion of the display device and a large amount of noise generated in the image acquisition.

It is subjective that people handle the image for different enhancement purposes, such as filtering, smoothing and sharpening, etc., to apply to a particular application. In contrast, image restoration is to rebuild or restore the original image based on the corresponding degradation model or knowledge, which is relatively objective. If the degradation process is known, the image can be restored by performing its inverse operation; if there is no known degradation information, such as the experimental data in this paper, it is necessary to establish the model of the degradation process (fuzzy and noise) to restore the original image and weaken its impact on fluorophore localization to the maximum extent.

Generally, the degradation process of the image is modeled as a degenerate function and an additive noise term. According to the linear system theory, the image degradation/restoration model can be expressed as:

$$g(x,y)=h(x,y)*f(x,y)+n(x,y).$$

(2)

where $f(x,y)$ is the input image, $h(x,y)$ is the degenerate function, $n(x,y)$ is the additive noise, and $g(x,y)$ is the generated degraded image.

Equation (2) is equivalent to the following frequency domain expression:

$$G(u,v)=H(u,v)F(u,v)+N(u,v).$$

(3)

Then the original image can be expressed as:

$$F(u,v)=\frac{G(u,v)}{H(u,v)}-\frac{N(u,v)}{H(u,v)}.$$

(4)

In other words, from the known degraded image and degradation transfer function we can get the frequency domain expression of the original image. Thus, by taking an inverse Fourier transform, also known as deconvolution technique, the restored image can be obtained. However, this method is only applicable for those problems with degenerate function but not considering additive noise.

The datasets we used in this paper are multi-frame degraded images; moreover, the PSF of the datasets is unknown. In order to restore the original images accurately, blind deconvolution algorithm [21,22] is chosen.

The blind deconvolution method is based on the maximum likelihood estimation, and it is an optimization strategy for estimating the parameters that are interfered by random noise. In blind deconvolution, the optimization is realized by using the specified constraints and assuming that the convergence is achieved by iterative procedure. Then, the resulting maximum likelihood functions, $f(x,y)$ and $h(x,y)$ are employed as the restored image and PSF.

In this way, the algorithm steps are summarized as follows: (1) Read the image matrix. The original grayscale images, which are Tiff format and uint8 type, are processed as a batch to Gif format and double type for convenience of use and high accuracy. (2) Calculate the weight of each pixel of the images above. (3) Estimate the point spread function. An all 1 matrix corresponding to the original image size is generated as an initial PSF and the weight value of each pixel are calculated. (4) Perform blind deconvolution iteration. The function in MATLAB, $[J,PSF]=deconvblind(I,InitPSF,NumIt,Damper,Weight)$, is used for blind deconvolution, and a new PSF is developed. Furthermore, the number of iterations in blind deconvolution being either too small or too large would affect the restoration result; therefore, the number should be tried according to the image processing effects.

Blind deconvolution is the first step of single-molecule localization, and it is used to suppress the noise and restore the image. Since the actual PSF is unknown, and there are a large number of frames in the original sequence, it is difficult to restore the image precisely; that is, each fluorescent molecule occupies only one pixel. Therefore, the question of how to locate the fluorescent molecule is the focus of the paper.

2.3. Centroid Localization and Image Reconstruction

The centroid method is one of the popular localization algorithms in fluorescence single-molecule localization, which locates the position of the fluorophore based on the circular symmetry of the fluorophore imaging model and the weighted average technique, or estimates the distance an object has moved by comparing the center of mass or centroid of two successive images of a particle. As the most commonly employed sub-pixel algorithm, the centroid method mainly locates the center symmetric object such as the circle and rectangle, but it can also be applied to the centroid localization of the irregular geometrically connected region efficiently and accurately. Equation (5) gives the centroid calculation for x and y axes [13].

$$C_x=\frac{{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{j=1}^n(x_i{I}_{i,j})}}}{{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{j=1}^n(I_{i,j})}}},\begin{array}{cc}& \end{array}{C}_y=\frac{{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{j=1}^n(y_i{I}_{i,j})}}}{{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{j=1}^n(I_{i,j})}}}$$

(5)

where $I_{i,j}$ is an image matrix of intensities, $n$ is the size of the extracted image area, $x_i$, $y_i$ denote the coordinates of the pixel $(i,j)$ in the $x$, $y$ axial directions, respectively. The obtained (C_x, C_y) is the centroid coordinate of the fluorophore along the $x$, $y$ axes.

Precise positioning algorithm and image reconstruction steps are as follows:

(1)

Denoise and restore images with blind deconvolution method;

(2)

Set a proper threshold for the grayscale binarization transformation to exclude background image and localize the position of the light spots (fluorescent molecules);

(3)

Perform removal operation to reduce the effects on accuracy due to the overlap of the fluorophores or the large exposure area, which is particularly necessary for densely labeled region with many fluorophores in close proximity [10];

(4)

Employ centroid localization algorithm as shown in Equation (5) to locate the single molecule precisely;

(5)

Output the positions of the fluorophores in a single frame image;

(6)

Reconstruct the positions of all the fluorophores in image sequence and obtain super-resolution image.

Figure 1 shows the flow chart of single-molecule localization and image reconstruction. The practical application of the SMLA is implemented in MATLAB platform.

3. Results and Discussion

In order to prove the validity of the proposed method, the BDCL-SMLA and image reconstruction method are applied to the three reference datasets downloaded from Suliana Manley’s laboratory. The computer operating system we used is Windows 7 with the processor model of Intel(R) Core(TM) I3-6100 CPU @3.70 GHz. The programming language is MATLAB R2020a 9.8.0.

Take the first dataset of long sequence Tubulins for an example. Figure 2a shows the 200th, 400th and 600th original frames coming from the 15,000 frames. Figure 2b–e are the frames processed sequentially by blind deconvolution, grayscale binarization transformation, removal operation and centroid localization, respectively, corresponding to the algorithm flow chart illustrated in Figure 1. From the comparative results of (a) and (b), it is clear that blind deconvolution can effectively suppress the noise and restore the image. Figure 2f shows the frames simply resulted from centroid localization. There are some differences between (e) and (f), which may affect the reconstructed image.

Then, using the BDCL-SMLA method, we obtain the reconstructed image of long sequence Tubulins of 15,000 frames with image size of 6.4 μm × 6.4 μm as shown in Figure 3b, comparing to the original image in Figure 3a. The inset pictures display the lateral profiles along the lines at same position. The reconstruction process needs 505 s. Moreover, we obtain the images reconstructed from the original 15,000 frames at intervals of 5, 10, 20 and 30 frames, as shown in Figure 3c–f, which take 115 s, 65 s, 37 s and 25 s, respectively. With the decrease in the frames, the reconstruction speed improves, while low image contrast and artificial disappearances of some structures are observed. From our calculation, 750 frames (at interval of 20 frames) or more are necessary for reconstruction to ensure the image quality.

Figure 4a shows the original image of Tubulin AF647 of 9990 frames with image size of 12.8 μm × 12.8 μm. Figure 4b is the corresponding reconstructed image with distinctive structure using BDCL-SMLA method, and 457 s are taken to obtain it. The inset pictures display the lateral profiles along the lines at same position. Figure 4c is the reconstructed image only by centroid localization with running time of 90 s. Although the centroid localization algorithm is very fast, there is more noise on the image compared to (b). Figure 4d is the reconstructed image obtained through the blind deconvolution and centroid localization but skipping removal operation, with running time of 500 s. Comparing (b) and (d), it can be concluded that removal operation can shorten the running time and reduce the noise, and it should be an important step in the BDCL-SMLA method.

In addition, the fluorophore localization method is used to localize high particle-density fluorophores. The original image and reconstructed image of high-density Tubulins of 500 frames are shown in Figure 5a and b, respectively, with an image size of 6.4 μm × 6.4 μm. The inset pictures display the lateral profiles along the lines at same position. It can be seen that the structure that is indistinguishable in the original data is resolved clearly after reconstruction. However, the disappearances of some structures can be observed in (b), mainly because of the small number of frames and the selection of elliptic eccentricity in the connected region in the algorithm. Figure 5c shows the reconstructed image obtained only through the centroid localization. It is obvious that the reconstructed image with BDCL-SMLA method is superior to that with only centroid localization in the aspect of image contrast and structure details.

4. Conclusions

In summary, super-resolution localization microscopy methods have increasingly wide applications in cell biology. As the two key innovations for these methods, switchable fluorophores and powerful localization algorithms have attracted more and more attention from scientists. In this paper, we have developed a single-molecule localization algorithm based on the blind deconvolution and centroid localization (BDCL-SMLA) method without much knowledge of experimentally determined parameters such as PSF and background noise. This BDCL-SMLA method takes account into both speed and precision at the same time in super-resolution image reconstruction by combining the maximum likelihood estimation method and the centroid method. The BDCL-SMLA method is used to reconstruct the images of several datasets and good results are obtained, which is proved to be a reasonable attempt for the SMLM imaging when the imaging system is unknown. However, the method still has some shortcomings, such as reconstruction speed with a large number of frames and artificial disappearances of some structures with a small number of frames, which will be considered and improved in the next step.

With the availability of new probes and improvement of detector technology, greater precision in the localization of point sources becomes more feasible. On the other side, further advances in localization algorithms and the development of deep learning technology will likely further promote the development of SMLM and expand its application scopes.