Study on Aberration Correction of Adaptive Optics Based on Convolutional Neural Network

Abstract

The existence of aberrations has always been an important limiting factor in the imaging field. Especially in optical microscopy imaging, the accumulated aberration of the optical system and the biological samples distorts the wavefront on the focal plane, thereby reducing the imaging resolution. Here, we propose an adaptive optical aberration correction method based on convolutional neural network. By establishing the relationship between the Zernike polynomial and the distorted wavefront, with the help of the fast calculation advantage of an artificial intelligence neural network, the distorted wavefront information can be output in a short time for the reconstruction of the wavefront to achieve the purpose of improving imaging resolution. Experimental results show that this method can effectively compensate the aberrations introduced by the system, agarose and HeLa cells. After correcting, the point spread function restored the doughnut-shape, and the resolution of the HeLa cell image increased about 20%.

1. Introduction

Adaptive optics (AO) was first proposed in astronomy to solve the problem of wavefront distortion caused by atmospheric turbulence in the observation of stars [1], using the wavefront corrector to compensate for the detected distortion wavefront in real time. Similarly, in optical microscopy imaging systems, signal light accumulates aberrations after passing through optical systems and biological samples, resulting in a decrease in the image resolution and signal-to-noise ratio. Therefore, adaptive optics has also been introduced into microscopy imaging technology [2] to address optical aberrations in the system and samples. According to the presence or absence of wavefront detectors, aberration correction methods are divided into two categories: aberration correction with wavefront detection and aberration correction without wavefront detection. In aberration correction with wavefront detection systems, fluorescent beads or fluorophore signals are generally used as guide stars to measure distorted wavefronts with wavefront detectors and then perform aberration compensation. In 2010, Azucena et al. used fluorescent microspheres seeded in the embryo as a guide star and corrected the aberrations measured by the Shack–Hartmann wavefront sensor [3,4]. Reinig et al. used a wavefront detector and deformable mirror to correct wavefront aberrations in mouse brains and improved the quality and signal-to-noise ratio of images [5]. Wang et al. recovered diffraction-limited imaging over large volumes (>240 mm per side) in the zebrafish brain by a descanned, laser-induced guide star and direct wavefront sensing [6]. In aberration correction without a wavefront detector system, wavefront measurement and aberration compensation are performed through iterative calculation to optimize a certain index of the image without a wavefront detector. The coherent optical adaptive technique [7], genetic algorithm [8] and hill-climbing algorithm [9] are all aberration corrected technologies without wavefront detection that correct aberrations by maximizing the image brightness and improve the signal-to-noise ratio and resolution of the images to a certain extent. Using other parameters as evaluation indicators, which is also a branch of aberration correction without wavefront detection, Débarre et al. increased the contrast and quality of images of a mouse intestine and embryo, and the process induced little photobleaching [10]. Ji et al. successfully restored the diffraction-limited resolution at a depth of 450 μm in the mouse cortex by dividing the pupil into multiple regions, calculating the wavefront slope in each region and reconstructing the wavefront [11].

Although AO with a wavefront detector has the advantages of fast speed and high accuracy, it increases the complexity of the optics system and the cost, and the method based on fluorescence guide stars involves complicated biological operations and damages the sample. AO without a wavefront detector is simple and low in cost, but its correction time is longer due to multiple iterative calculations and is prone to photobleaching, which restricts imaging in fluorescent samples to a certain extent. However, machine learning algorithms have solved these problems. As early as 1997, Montera et al. used a neural network to predict the measurement slope of wavefront sensors and compared them with the results of classic statistical-based methods to prove the advantages of neural networks [12]. Rivenson et al. used a single image taken with a standard optical microscope without the need for extra hardware or user-specified postprocessing, taking on average ~0.69 s to output an improved image by deep learning [13]. Turpin et al. used a neural network to establish the functional relationship between transmitted and reflected speckle patterns to correct the wavefront [14]. Khonina S.N. et al. studied a new type of wavefront aberration sensor, and at the same time used a neural network to recognition the type of wavefront aberrations [15,16]. Similarly, based on Zernike polynomial reconstruction, deep learning has improved the resolution and intensity of images in aberration correction of two-dimensional scattering samples [17] and three-dimensional beads [18]. Although the above methods have advantages in speed and accuracy, the methods [19,20,21,22] based on convolutional neural networks have also encountered difficulties in practicality. Some methods [19,20] used simulated synthetic data to train the network and directly output the aberration functions for aberration compensation while ignoring the collection of ground truth data from the microscope. Others [21,22] focused on the analysis of the data and the accuracy of the results and lacked the application of biological samples in microscopy imaging. Therefore, the practical applicability of these methods in microscopy experiments is still doubtful.

In this paper, we collect data from experiments for convolutional neural network training and compensate for the aberrations in microscopy. It is worth noting that in the shape-based evaluation strategy, the doughnut-shaped beam is not only more effective than complex patterns but also more sensitive to aberrations than simple Gaussian beams, so it is used to carry aberration information. Furthermore, in stimulated emission depletion (STED) super-resolution microscopy, the doughnut-shaped depletion beam is used to superimpose on the excitation beam to achieve super-resolution imaging, and the quality of the doughnut-shaped beam has a direct impact on the resolution of STED [23,24,25,26]. Therefore, correcting the aberrations in the donut-shaped beam can help improve the resolution of STED super-resolution imaging. Our self-created convolutional neural network can automatically extract the characteristic information of the input distorted spot image and establish a connection with the aberration fitted by the Zernike polynomial. After training, the Zernike polynomial coefficients can be output in a very short time for the reconstruction of the wavefront phase to compensate for aberrations. Based on this method, we corrected the system aberration, agarose aberration and HeLa cell aberration to verify the correction effect and practicability of this method in the experiments.

2. Methods

2.1. Experimental Setup

Figure 1 shows the experimental setup for adaptive optics aberration compensation based on a convolutional neural network, in which a mode-locked Ti–sapphire laser (Chameleon Ultra II, Coherent, Santa Clara, CA, USA) with a wavelength of 780 nm was used as the light source. A half-wave plate (Thorlabs, Newton, NJ, USA) is used to control the polarization direction of the laser and to adjust the power of light before entering the fiber. The single mode polarization maintaining fiber (DH-FP780-FC-1) is used to eliminate the influence of stray light and shape the beam spot. Then, the light is expanded by a pair of lenses to fit the spatial light modulator (SLM, PLUTO-NIR-011, HOLOEYE Photonics AG, Berlin, Germany), which has two functions: one is used to generate doughnut-shape beam, and the other is to load the compensation aberration calculated by the “neural network” algorithm to realize the aberration correction in the system. The collimated beam is modulated by the SLM with a spiral phase, which converts the Gaussian beam to a doughnut-shaped beam. After a reflector mirror, a quarter wave plate (Thorlabs, Newton, NJ, USA) converted the doughnut-shaped beam to right-hand circular polarization for better light field distribution. A pair of galvanometer scanning mirrors (6210H, Cambridge Technology Inc. Cambridge, MA, USA) was controlled by ScanImage (HHMI/Janelia Farm, Ashburn, VA, USA) and used to scan the beam across the sample, and an oil-immersion objective (HCX PL APO, 100×/1.40–0.70 OIL, Leica, Wetzlar, Germany) was used to focus the laser beams on the sample and collect the fluorescence signal. The fluorescence signal was separated from the excitation light path by a dichroic mirror (DM, transmission: 720–1200 nm, reflection: 350–720 nm) and then focused into a photomultiplier tube (PMT, H7422–40, Hamamatsu Photonics, Japan) by a lens (L5: 125 mm). A filter between the PMT and the lens is used to remove the residual excitation light and stray light. A charge-coupled device (CCD) is used to collect data for constructing images. The photograph of the experimental setup please see the Figure S1.

Aberration, that is, wavefront phase distortion, can be expressed as a set of phase matrices, which are generally decomposed into a set of linear combinations of orthogonal polynomials. Each term in the polynomials is called a wavefront mode in wavefront analysis. The Zernike polynomials are the most widely used aberration description basis functions in the field of adaptive optics. Zernike polynomials are composed of an infinite number of polynomials, which are continuous and orthogonal within the unit circle. In the aberration simulation experiment, it can be expressed by the terms of the Zernike polynomial or any combination of any terms. The Zernike polynomial polar coordinate form is often used to fit wavefront aberrations:

$$\mathsf{\phi }\left(\mathsf{\rho },\mathsf{\theta }\right)={\displaystyle \sum }_{i=1}^n{a}_i{Z}_i\left(\rho ,\theta \right)$$

(1)

Here, $\mathsf{\phi }\left(\mathsf{\rho },\mathsf{\theta }\right)$ is the phase distribution on the pupil plane, n is the highest order of the Zernike polynomial (up to 15 terms in our experiment), $Z_i$ and $a_i$ represent the i-th polynomial and its coefficients, respectively, $\rho$ is the polar radius, and $\theta$ is the polar angle. The first three modes of aberrations are often ignored. Thus, the first of the Zernike polynomials is the piston phase, which represents the overall displacement of the wavefront and does not affect the imaging quality; the second and third represent the tip and tilt of the wavefront in the x and y directions, respectively, and these can be estimated rapidly using centroiding algorithms or other registration [27].

2.2. Convolutional Neural Network Structure

The structure of the convolutional neural network (CNN) is shown in Figure 2. The CNN starts with 3 parts, and each part contains two convolutional layers and a max pooling layer. The convolutional filters have 32, 64 and 128 kernels for each part, and the size of all convolution kernels is 3 × 3, which is used to extract image features. Each of the convolutional layers links an activation function named the rectified linear unit (ReLU), which causes the sparsity of the network and reduces the possibility of overfitting. The essence of pooling is sampling, which can be regarded as a special convolution process, and max pooling with a size of 2 × 2 can remove the maximum value in pooling size, which can effectively reduce the image feature parameters and perform dimensionality reduction. The dropout operation randomly discards some neurons with a probability of 0.3 to avoid overfitting. The last two fully connected layers can integrate the extracted features, run nonlinear transformations and output the predicted wavefront distortion Zernike coefficients for reconstruction.

We randomly generated 12,000 phase patterns with aberrations fitted by the Zernike polynomials and loaded them on the SLM and used the CCD camera to collect the images and establish the data set. The convolutional neural network was implemented in Keras with the TensorFlow backend and trained on a standard computer equipped with a Nvidia GeForce GTX 1070 GPU. After the training is completed, it takes approximately 33 ms to input a distortion PSF and then output the Zernike coefficients to reconstruct the phase patterns. If a better configuration is used, it takes less time. Furthermore, it should be noted that it is not necessary to retrain unless fundamental parameters of the experimental setup are changed, such as the numerical aperture or a major realignment. The images used for machine learning are all raw images without preprocessing. To improve the display effect of the images, the greyscale of all the images in the paper have been adjusted in the same way by Imaged J.

3. Results and Discussion

Due to the aberrations of the optical system and the sample, the doughnut-shaped beam cannot be focused on in the ideal state. First, we used aberration correction of adaptive optics based on a convolutional neural network to correct the aberrations introduced by the system. To verify the correction effect, we made gold nanoparticles as a sample. The steps were as follows: drop an appropriate amount of gold nanoparticle solution (200 nm in diameter, Fisher Scientific, Vantaa, Finland) to a coverslip, and encapsulate the coverslip on a microscope slide with mounting medium (97% 2,2′-thiodiethanol, TDE, Fisher Scientific, Vantaa, Finland). We compared the backscattering intensity PSFs of the gold nanoparticles before and after the aberration correction. It can be clearly found that the symmetry of the doughnut-shaped beam intensity distribution was improved after the aberration correction (Figure 3).

Then, we used random phase patterns with aberrations to examine the effectiveness of the convolutional neural network. Figure 4a,b shows the comparison before and after correction. Even if random and complicated aberrations have been added, our method still demonstrates its effectiveness. Moreover, both the image symmetry and strength in the corrected PSF had a significant improvement compared to those in the uncorrected PSF, which can be observed from the normalized intensity profiles along the lines in Figure 4c,d. The initial and predicted wavefront Zernike coefficients of the first 15 orders (ignoring the first to third) are shown in Figure 4e. We find that the predicted result is extremely close to the actual value, which further shows that our algorithm can effectively perform aberration correction.

Furthermore, we prepared an agarose sample for the simulation of biological tissues. The sample was prepared according to published protocols [8]. We imaged the 3D PSFs of 200 nm gold nanoparticles before and after the correction. As shown in Figure 5, the PSF improved well and were restored to a doughnut-shape after the correction. The intensity distribution symmetry of the doughnut-shaped beam was improved in the y–z plane after performing the correction.

To further demonstrate the capability of machine learning in correcting aberrations in biological samples, we used a method to observe the structures of microtubules in HeLa cells labeled with Alexa Fluor 488 (Abcam, ab150077). The scattering signal of the gold nanoparticles in HeLa cells was used as a guide for the convolutional neural network. Figure 6a,b shows images of tubulin structures in the HeLa cells labeled with Alexa Fluor 488. It can be clearly found that the resolution of the image has been restored after the aberration correction. By fitting the intensity profiles along the white arrows in the magnified images with a Gaussian function, two full-widths at half-maximum (FWHMs) can be obtained, 534 nm in the uncorrected image and 397 nm in the corrected image, and the resolution increased about 20%. Similarly, the intensity of the image is also improved after aberration correction, as shown in Figure 6c,d. It should be noted that due to the difference in the internal structure of biological cells, the refractive index distribution is uneven, and the aberrations in different regions are also different. To further improve the calibration effect, the sample can be calibrated by region. The more regions the sample is divided into, the better the calibration effect is. However, because the aberration of our sample itself is limited, only one calibration point can meet the demand.

4. Conclusions

Here, a distorted wavefront compensation approach based on a sensorless wavefront and a convolutional neural network is proposed. Because the doughnut-shaped beam carries aberration information more easily than other beams, the convolutional neural network can extract aberration information carried by the distorted doughnut-shaped beam automatically and establish a connection with the aberration fitted by the Zernike polynomial. The well-trained convolutional neural network can calculate the aberration information in a very short time and output the Zernike polynomial for wavefront reconstruction. We have corrected the aberrations induced by the system, agarose and HeLa cells through experiments. The results show that our method can effectively restore the shape of the doughnut shape and restore the original resolution of the image. However, there are still some questions, such as when facing the situation of a low signal-to-noise ratio, the convolutional neural network that relies on the shape strategy may appear to have reduced accuracy or failure when extracting aberration information. Using other image processing methods to increase the signal-to-noise ratio may improve the correction effect of our method. For biological imaging, the aberrations of different regions are different due to the difference in refractive index, and the method of using multiple regions and different guide stars to compensate aberrations and merge images can achieve a certain improvement in resolution and signal-to-noise ratio. Therefore, our method may have an important application prospect in STED super-resolution imaging.