A CNN-GS Hybrid Algorithm for Generating Pump Light Fields in Atomic Magnetometers

Abstract

Atomic magnetometers (AMs), recognized for their ultra-high magnetic sensitivity, demand highly uniform pump light fields to maximize measurement accuracy. In this paper, a phase modulation-based method using convolutional neural networks (CNN) and the Gerchberg–Saxton (GS) algorithm is proposed to generate the pumping light field, and the model was trained using a supervised learning approach with a custom dataset. The specific training settings are as follows: the backpropagation algorithm was adopted as the training algorithm, and the Adam optimization method was used for network training, with a learning rate of 0.001 and a total of 100 training epochs, utilizing a liquid crystal spatial light modulator (LCSLM) to regulate the light field phase distribution dynamically. By transforming Gaussian beams into flat-top beams, the method significantly enhances polarization uniformity within vapor cells, leading to improved magnetometric sensitivity. The proposed hybrid algorithm reduces the mean square error from 35% to 19% and peak non-uniformity from 21% to 7.6%. A reflective LCSLM-based optical setup is implemented to produce circular and square flat-top beams with a measured non-uniformity of 5.1%, resulting in an enhancement of magnetic sensitivity from 14.04 fT/Hz^1/2 to 7.80 fT/Hz^1/2.

1. Introduction

Quantum precision measurement technology can be widely applied to research such as frontier physics exploration [1], magnetic anomaly detection [2], space magnetic field detection [3], biological magnetic field detection [4], and resource exploration [5]. With the development of quantum technology and laser technology, the spin-exchange relaxation-free (SERF) atomic magnetometer (AM) has become a research hotspot in the field of ultra-sensitive extremely weak magnetic field detection [6,7,8,9], which will enable magnetic field measurements at the $aT$ (10⁻¹⁸ T) magnitude level in the future [10]. In particular, AM possesses the advantages of easy miniaturization and wearability and has no need for liquid nitrogen cooling, thus holding significant research and application value in human magnetic field measurements such as magnetoencephalography (MEG) and magnetocardiography (MCG) [11,12,13,14,15].

The SERF AM is a high-precision measurement sensor that detects weak magnetic fields by measuring the spin effects of alkali metal atoms in a SERF state [16,17,18]. It polarizes alkali metal atoms through optical pumping and measures changes in laser polarization or absorption characteristics caused by Larmor precession induced by external magnetic fields to determine the magnetic field strength [19,20]. Therefore, the quality of the laser beam directly impacts atomic polarization and dynamic response detection. The Gaussian laser spot output by the laser enters the sensor’s vapor cell to polarize alkali metal atoms. However, Gaussian beams have higher intensity in the center than at the periphery. Due to the properties of Gaussian beams and beam propagation changes in the magnetic field measurement optical system, the pump light intensity distribution within the vapor cell becomes non-uniform. This non-uniformity in the pump laser spot directly causes uneven electron polarizability [21]. Therefore, beam shaping of the pump laser’s Gaussian beam is necessary to improve beam quality and prevent a decrease in AM sensitivity caused by lateral polarizability gradients generated by the pump laser.

Beam shaping technology can modulate laser beams with Gaussian intensity distribution into desired shapes and intensity distributions, enabling pump lasers to be adjusted according to the shape of the atomic vapor cell and the required polarization gradient. Traditional beam shaping technologies are mainly realized based on the principles of physical optics or geometric optics, such as the aspheric lens method [22] and birefringent lens group method [23]. These technologies are often accompanied by limitations such as complex devices, low energy conversion efficiency, and insufficient shaping accuracy. In recent years, the micro-lens array method [24], diffractive optical element method [25], and liquid crystal spatial light modulator (LC-SLM) shaping method [26] based on diffraction principles have developed rapidly. After analyzing the advantages and disadvantages of various beam shaping methods, this paper adopts the LC-SLM shaping method. By calculating the phase distribution of the beam superimposed on the liquid crystal screen, beams with different parameters are shaped into flat-top beams with uniform light intensity distribution. At present, the methods for calculating phase distribution include the Gerchberg–Saxton (GS) algorithm [27], simulated annealing method [28], genetic algorithm [29], and GS improved algorithms such as the GSW optimization operator and pseudo-random GS improved algorithm. The GS algorithm features an intuitive principle and extremely fast calculation speed, but it tends to fall into local optimal solutions [30]. Both the simulated annealing method and genetic algorithm require enormous computational power [31] and extremely long calculation times. In the interdisciplinary research field of optics and computer vision, deep learning models have been applied for some time [32] and numerous achievements and published articles regarding the applications of deep learning models in optics or atomic magnetometers (AMs) have provided strong theoretical support for the research presented in this paper [33,34,35].

In this study, an improved method combining GS and neural networks is proposed for phase modulation calculation to generate uniform beams. The beam characteristics of Gaussian beams and flat-top beams are analyzed. A convolutional neural network (CNN) is used to predict the initial phase of the beam [36] by supervised learning, which is then refined by the GS algorithm. The improved algorithm is divided into four stages: data preparation, model training, model prediction, and algorithm refinement, to obtain the beam phase distribution shaped by the improved GS algorithm. Simulation experiments are conducted on the GS algorithm and the improved algorithm combining GS and neural networks, and their mean square errors and top non-uniformities are compared. Finally, a beam shaping and SERF AM optical system is built. Based on the improved algorithm, the beam shaping of the pump light source for the atomic magnetometer is realized, improving the sensitivity of magnetic field measurement.

2. Flat-Top Beam Generation System

The default output of a laser typically follows a Gaussian intensity profile, exhibiting strong central intensity and weak peripheral intensity, and the modeling of beam profiles forms the basis of this effort.

The Gaussian intensity distribution is defined by the following:

$$I(x,y)=I_{0}exp\left[-{\displaystyle \frac{2\left(x^2+y^2\right)}{{\omega }_0^2}}\right]$$

(1)

where, $I_0$ is the maximum light intensity at the center of the beam, x and y are the spatial coordinates on the beam cross-section, and ${\omega }_0$ is the beam waist radius, defined as the radial distance at which the intensity drops to $1/e^2$ of central value, approximately 13.5%.

A flat-top beam is characterized by uniform intensity over a specified central region, with sharp intensity decay at the edges. In this study, the circular flat-top beam is modeled using the Super-Gaussian profile, defined as follows:

$$I(x,y)=I_{0}exp\left[-2{\left({\displaystyle \frac{x^2+y^2}{w^2}}\right)}^N\right]$$

(2)

where $I_0$ is the peak intensity, $\omega$ is the beam waist radius (where the intensity drops to $I_0/e^2$), and N is the Super-Gaussian order. When $N=1$, the model simplifies to a standard Gaussian. As N increases, the beam top becomes flatter and the edge transitions steeper. This model effectively represents a continuum from Gaussian to ideal flat-top distributions while maintaining mathematical simplicity, facilitating the implementation of phase retrieval algorithms.

Similarly, the square flat-top beam is described by the Super-Gaussian model:

$$I(x,y)=I_{0}exp\left[-2\left({\left({\displaystyle \frac{x}{a}}\right)}^{2N}+{\left({\displaystyle \frac{y}{b}}\right)}^{2M}\right)\right]$$

(3)

where a and b are the characteristic half-widths in the x and y directions, respectively, and N, M are the Super-Gaussian orders along each axis. When $a=b$ and $N=M$, the model represents a standard square beam; when $N=M=1$, it reduces to a standard Gaussian.

Figure 1 illustrates the three-dimensional intensity distributions of two typical flat-top beams: (a) circular flat-top beam; (b) square flat-top beam. As observed from the figure, the beams maintain a uniform intensity in the central region and exhibit a rapid decline to zero at the edges. The super-Gaussian order for both beams is set to 10.

To systematically evaluate the beam shaping performance, the mean square error e and the peak-to-valley uniformity $\sigma$ are adopted as evaluation metrics. The specific formulas are as follows:

$$e={\displaystyle \frac{{\sum }_{(x,y)}{\left|I(x,y)-I^{\prime }(x,y)\right|}^2}{{\sum }_{(x,y)}{\left|I^{\prime }(x,y)\right|}^2}}$$

(4)

$$\sigma =\sqrt{{\displaystyle \frac{{\sum }_{(x,y)\in w}{\left[\frac{I(x,y)-r}{r}\right]}^2}{n-1}}}$$

(5)

where $I(x,y)$ denotes the experimentally obtained intensity distribution, $I^{\prime }(x,y)$ represents the desired target distribution, $\omega$ indicates the region of interest on the observation screen, n is the number of sampling points within this region, and r is the average intensity in the region. The mean square error e reflects the similarity between the experimental and target distributions; a smaller value indicates better agreement and thus a more accurate beam shaping result. The peak-to-valley uniformity $\sigma$ measures the uniformity of the experimental intensity distribution; a lower value signifies a higher degree of uniformity across the beam profile.

Given the reflective nature of the spatial light modulator (SLM) employed in this experiment, the fundamental optical path is designed as illustrated in Figure 2. The laser beam is first emitted from the laser diode, passes through an isolator, and is coupled into an optical fiber. After exiting the fiber, the beam undergoes expansion to achieve the desired spot size, followed by polarization via a polarizer to produce a horizontally linearly polarized beam. This beam then enters the SLM, where its phase is modulated. The reflected beam is focused by a convex lens and observed using a beam quality analyzer.

2.1. Phase Algorithms and Simulation

2.1.1. Gerchberg–Saxton (GS) Algorithm

The GS algorithm, proposed by Gerchberg and Saxton in 1971, remains a foundational approach in holographic computation and phase retrieval. It is based on iterative Fourier transforms and is known for its simplicity and ease of implementation. Figure 3 illustrates the algorithm’s workflow.

Let the incident beam have amplitude $f_0(x_1,y_1)$ and initial phase ${\varphi }_0(x_1,y_1)$. After k iterative Fourier transforms, the output amplitude distribution is $g_o(x_k,y_k)$, and the corresponding phase is ${\varphi }_k(x_k,y_k)$. The initial optical field is E, and $E_k$ is expressed as the output optical field distribution. Substituting the initial phase into E yields the following:

$$E=f_0\left(x_1,y_1\right)exp\left[i{\varphi }_0\left(x_1,y_1\right)\right]$$

(6)

A Fourier transform yields the following:

$$E_1=\left|F\left(x_2,y_2\right)\right|exp\left[i\varphi \left(x_2,{\mathrm{y}}_2\right)\right]$$

(7)

Replacing the amplitude with the desired distribution:

$$E_1^{\prime }=\left|g_0\left(x_2,y_2\right)\right|exp\left[i\varphi \left(x_2,y_2\right)\right]$$

(8)

Performing another Fourier transform yields the following:

$$E_1=\left|f\left(x_1,y_1\right)\right|exp\left[i\varphi \left(x_2,y_2\right)\right]$$

(9)

The updated field is obtained by replacing the amplitude with the original $f(x_1,y_1)$ with the original $f_0(x_1,y_1)$, maintaining the phase:

$$E_1=\left|f_0\left(x_1,y_1\right)\right|exp\left[i\varphi \left(x_2,y_2\right)\right]$$

(10)

This iteration continues until the convergence condition (e.g., output phase ${\varphi }_k(x_k,y_k)$) is met.

In simulation, the following parameters are used: Gaussian and flat-top beam waist radius of 2 mm, resolution of 512 × 512 sampling points, SLM pixel size of 9.2 µm, focal length of Fourier lens of 100 mm, Super-Gaussian order of 10, and 50 iterations.

The simulation results are presented in Figure 4. The mean square error is 35.27%, and the top non-uniformity is 20.82%. The output beam demonstrates significant noise and suboptimal flatness.

The phase distribution obtained by the GS algorithm is converted into a grayscale image as shown in Figure 5a, indicating significant disorder in the phase distribution. The grayscale values of adjacent pixels exhibit a jump-like distribution, making it difficult to reproduce such results in experiments. Figure 5b presents the simulation result of the beam shaping experiment using this phase distribution, which also shows severe noise and an unsatisfactory flat-top effect.

The GS algorithm is highly sensitive to the initial values of iterations during experiments, prone to falling into local optimal solutions, and stopping iterations, so the experimental effects are generally unsatisfactory. However, due to its simple principle, it provides an idea for people to study iterative algorithms, thus holding very important significance in iterative algorithms.

2.1.2. Improved Algorithm Based on GS and CNN

In recent years, with the advancement of neural network-related technologies, an increasing number of fields have begun to adopt this machine learning theory to explain complex experimental phenomena. Owing to its superior mapping and simulation capabilities, significant achievements have been made in various domains. For instance, in the research field of computer-generated holography (CGH), Hossein proposed an innovative non-iterative algorithm based on an unsupervised learning convolutional neural network called DeepCGH, aiming to enhance the quality of holograms through computation [37]. P. Tsang proposed a deep learning-based holographic vision system (HVS) that identifies holograms through hologram classification and uses a handwritten digit dataset for performance evaluation [38]. Goi presented a deep learning-based method for generating binary holograms, which generates binary holograms in a non-iterative manner and compares the neural network-generated binary holograms with those from existing methods, showing significant improvements in both speed and quality [39]. In 2010, researchers proposed a deep learning-based computer-generated holography (CGH) technique that can quickly generate high-quality holograms, outperforming baseline methods that require more computational resources and time [40]. S.J. Lee et al. proposed a deep learning-based digital holographic microscopy (DHM), where the network is trained on thousands of blurred microparticle holograms, uses a convolutional neural network (CNN) to estimate the depth of microparticles, and employs Segnet and Hough transform to detect microparticles in the plane. After training, this method far surpasses iterative methods in tracking the 3D motion of microparticles [41].

CNN (Convolutional Neural Network) possesses unique working principles and characteristics. Its local perception property enables neurons in the network to perceive only local regions of the input data, which is consistent with the local correlation characteristics of data such as images [42]. For instance, when processing light intensity images, adjacent pixels usually exhibit strong correlations, and local perception can effectively extract these local features [43]. Another core feature of CNN is the weight-sharing mechanism, which is manifested in the convolutional layer as follows: the weight parameters of the same convolution kernel remain consistent across the entire input data. This design significantly reduces the number of network parameters, effectively lowering the model complexity and the risk of overfitting [44]. The standard structure of CNN consists of convolutional layers, pooling layers, and fully connected layers. The convolutional layer performs feature extraction on input data through filters; the pooling layer reduces the dimensionality of feature maps while retaining key feature information (e.g., max pooling achieves feature compression by selecting the maximum value in local regions); the fully connected layer ultimately completes feature integration and classification decisions [45]. This hierarchical structure enables CNN to automatically learn feature representations with translation invariance [42]. CNN has demonstrated tremendous advantages in the field of image processing, being capable of automatically learning complex feature patterns in images. For example, in image classification tasks, it can accurately identify images of different categories, and it also exhibits excellent performance in tasks such as image denoising and super-resolution reconstruction [46,47].

Since the GS algorithm is a local optimization method, it is significantly influenced by the initial phase, making it prone to convergence at local optima. A well-designed initial solution can substantially enhance the performance of the GS algorithm. Therefore, the core idea of this study is to employ a CNN model to predict the initial phase distribution, followed by refinement using the GS algorithm.

The improved algorithm comprises four stages: data preparation, model training, phase prediction, and GS-based refinement, as illustrated in Figure 6 and described in detail as follows.

• Data preparation

A high-quality dataset is essential for accurate neural network modeling. The following strategy is proposed:

Based on the equation $E_2\left(x_2,y_2\right)=F\left[{\mathrm{E}}_1\left(x_1,y_1\right)\times exp\left(i{\varphi }_1\left(x_1,y_1\right)\right)\right]$, there $E_m$$(m=1,2)$ stands for light field distribution. Given $E_1$, once a phase distribution ${\mathsf{\Phi }}_1$ is obtained, the corresponding intensity distribution $E_2$ can be computed. Thus, a set of random phase patterns ${\mathsf{\Phi }}_1$ is generated, and their corresponding intensity distributions $E_2(x_2,y_2)$ are obtained via simulation. The intensity $E_2$ serves as the input, and the phase ${\mathsf{\Phi }}_1$ as the ground-truth label output for training purposes to obtain the simulation of ${\widehat{F}}^{-1}$ by a neural network. A total of 500 data samples are synthesized and normalized to the grayscale range of $[0, 1]$.

• Model Training

The CNN is implemented using the PyTorch1.10.2 framework. The network takes intensity distributions $E_2$ as input and padding was used to preserve the spatial dimensions, then learns to reconstruct the corresponding initial phase distributions ${\mathsf{\Phi }}_1$. The Adam optimizer is employed, with a learning rate of 0.001 and 200 training epochs. The model minimizes the loss function, such as the Mean Squared Error (MSE), through backpropagation. During training, the validation set is periodically evaluated to monitor performance metrics such as phase correlation coefficients. Early stopping or regularization techniques may be applied if overfitting is detected.

• Phase Prediction

Loading the Trained Model:After completing model training and confirming its satisfactory performance on the validation and test sets, load the trained model weights.

Inputting the Intensity Image to be Processed: Input the intensity images required for beam shaping in practical applications into the loaded model. These images must undergo the same pre-processing steps as the training data.

Outputting the Predicted Initial Phase: The model performs forward propagation calculations to output the corresponding initial phase distribution, which will serve as the initial input for the improved GS algorithm.

• GS-Based Refinement

The predicted phase distribution $\phi (X,Y)$ is used to construct the initial complex amplitude for the GS algorithm. The algorithm then iteratively optimizes the phase to match the target intensity distribution. Convergence is defined by a predefined criterion—e.g., the error between the calculated and target intensity patterns remains below a threshold for consecutive iterations. Upon convergence, the resulting phase is considered optimal and can be used to drive spatial light modulators for beam shaping purposes.

Figure 7 presents the simulation results of the improved GS algorithm assisted by the CNN model. Sub-figure (a) shows the 3D intensity profile of the generated flat-top beam, while (b) illustrates the cross-sectional intensity distribution. The Mean Squared Error (MSE) is 18.95%, and the peak non-uniformity is 7.56%. Compared to the GS algorithm, the improved method significantly enhances beam shaping quality.

Figure 8 further evaluates the phase and beam quality. Sub-figure (a) shows the optimized phase hologram obtained by the improved method, characterized by smooth gray level transitions and reduced abrupt phase jumps. Sub-figure (b) displays the resulting two-dimensional flat-top beam intensity distribution, exhibiting superior uniformity compared to those produced by the GS.

Table 1. Comparison of Simulation Results among GS and CNN-GS Hybrid Algorithms.

Algorithm	Mean Squared Error (%)	Peak Non-Uniformity (%)
GS Algorithm	35.27	20.82
Proposed CNN-GS Hybrid	18.95	7.56

summarizes the comparative results among the algorithms above. The proposed method achieves the lowest MSE and peak non-uniformity, demonstrating its superior performance in beam shaping tasks.

3. Training and Experiments

3.1. Dataset Preparation

Due to the absence of publicly available datasets suitable for training deep neural networks in computer-generated holography, a custom dataset was generated.

According to the equation $E_2(x_2,y_2)=F[E_1(x_1,y_1)\times exp\left(i{\mathsf{\Phi }}_1(x_1,y_1)\right)]$, a known input amplitude $E_1$ and a ${\mathsf{\Phi }}_1$ can generate a corresponding $E_2$, so a series of phase distributions ${\mathsf{\Phi }}_1$ can be randomly generated, the output field intensity $E_2(x_2,y_2)$ can be computed and saved then. This enables the creation of input-output pairs: $E_2$ as input, ${\mathsf{\Phi }}_1$ as output, to train a supervised learning model to obtain the simulation of ${\widehat{\mathrm{F}}}^{-1}$.

In this simulation, 500 sets of 64 × 64 data were generated and normalized to grayscale values in the range $[0, 1]$. Data were saved in $.npy$ format for training and also exported as $.png$ images for visualization. Each sample was labeled by index; let $sample\_num\_phase$ and $sample\_num\_amplitude$ denote the phase distribution and its corresponding amplitude distribution, respectively. The phase and amplitude are numbered to establish their correspondence.

3.2. Network Architecture and Model Training

The model adopts an encoder–decoder structure similar to U-Net [48] that shows in Figure 9. In this paper, the model we used has a structure as follows: the encoder consists of two $3\times 3$ convolutional layers that increase the channel dimension, followed by a max pooling layer to reduce spatial resolution. The decoder includes a transposed convolution layer to restore resolution and a final convolution layer to produce a 64 × 64 phase map. A ReLU activation function follows each layer to maintain nonlinearity for the advantage of enabling efficient training of deep networks through simple piecewise nonlinearity, while avoiding the vanishing gradient and computational redundancy issues of sigmoid. In contrast, the strong and smooth nonlinearity of the sigmoid has only limited applications in shallow networks or specific output layers, and it has been replaced by ReLU and its variants (such as Leaky ReLU and Swish) in hidden layers. After the ReLU activation function, a Dropout layer with a dropout rate of 0.5 and L2 regularization was added.

The training procedure includes the following steps:

• Set parameters: path for datasets saving, input image size 64 × 64, batch₋size 8, learning rate 1 × 10⁻³, training epochs 200, and loss function as RMSE.
• Initialize model weights.
• For each epoch, input training data to perform forward propagation, compute output, and calculate loss against the ground-truth phase. To enhance the credibility of the trained model, in the absence of a separate validation set, we employed 5-fold cross-validation on the 500 generated samples during training. Specifically, the dataset was randomly partitioned into 5 subsets of equal size (100 samples each). In each fold, 4 subsets (400 samples) served as the training data, and the remaining 1 subset (100 samples) was used to monitor performance metrics such as phase correlation coefficients for hyperparameter tuning. This process was repeated 5 times, with each subset acting as the validation data exactly once. The average performance across all folds was taken to guide hyperparameter selection, ensuring that the model optimization was based solely on the training data distribution. Meanwhile, a separate test set, consisting of 100 additional samples generated using the same method but not involved in any training or cross-validation steps, was reserved to provide an unbiased final evaluation of the model’s generalization ability.
• Perform backpropagation to compute gradients.
• Update weights using the Adam optimizer in view of its advantage lies in its balance between convergence speed, stability, and parameter adaptability, with low dependence on hyperparameters, enabling it to quickly achieve favorable results in most deep learning tasks. Although in certain scenarios (such as image classification requiring extreme generalization performance), Stochastic Gradient Descent (SGD) combined with fine hyperparameter tuning may perform slightly better, Adam has become the default choice in both industry and academia due to its comprehensive performance.
• Repeat until convergence or max epochs reached.
• Save model weights upon training completion.

Detailed parameter settings in

Table 2. Model training main parameter table.

Parameters	Setting
convolutional Layer	3 × 3
max pooling layer	2 × 2
training epoch	200
batch−size	8
learning rate	0.001
optimization algorithm	Adam
loss function	MSE
activation function	ReLU
regularization	Dropout (0.5) + L2

.

Figure 10 shows the loss function curves of the training set and validation set of the proposed algorithm after 200 training epochs. The curve trends indicate the following: In the early training stage (0–25 epochs), the loss values of both sets decrease rapidly, demonstrating effective adjustment of network parameters. In the middle training stage (25–170 epochs), the loss values show a fluctuating slow decline, indicating that the network is undergoing fine–tuning. In the final training stage (170–200 epochs), the loss values tend to be stable and finally converge to around 0.01, proving that the network has reached a good convergence state. The blue curve represents the validation set, and the red curve represents the training set, enabling a direct comparison of their loss change trends during the training process. The shared y-axis enhances the clarity of comparing the magnitude and rate of loss reduction between the two curves, providing a more intuitive understanding of the model’s generalization ability during training.

Model inference steps are as follows: (1) Load trained model and weights. (2) Resize the target amplitude image to match the model input. (3) Feed into the network to obtain the predicted initial phase. (4) Input the phase into the GS algorithm for refinement. (5) Save the final phase hologram and load it onto the SLM for physical beam shaping.

3.3. Optical Setup

To reduce optical noise introduced by the LCSLM, a $4f$ filtering system is implemented. As shown in Figure 11, a pair of identical focal length lenses is placed after the LCSLM with a separation of $2f$. A spatial filter (aperture) at the Fourier plane blocks the zero-order diffraction component. An imaging lens is placed at one focal length after the $4f$ system.

The effectiveness of the $4f$ system is summarized in

Table 3. Comparison Before and After $4f$ System.

	Before	After
Circular Flat-top beam
Square Flat-top beam

. Without filtering, shaped beams contain high noise and large non-uniformity. Introducing the $4f$ system improves beam uniformity significantly.

3.4. Experimental Procedure

The experimental procedure for phase-modulated beam shaping is described as follows:

First, the optical setup is constructed. According to the optical path layout shown in Figure 11, all components are properly positioned, and the laser beam height and collimation are finely adjusted. The laser source emits a beam at a wavelength of 795 nm, with an initial Gaussian intensity profile. A beam expansion system composed of two convex lenses with focal lengths of 25 mm and 75 mm is used to enlarge the beam spot by a factor of three. The beam then passes through a half-wave plate and a polarizing beam splitter (PBS) to adjust the polarization state to horizontal linear polarization. After incidence on the liquid crystal spatial light modulator (SLM), the corresponding phase map is loaded onto the SLM. The reflected beam is then passed through a $4f$ filtering system and a Fourier lens, with a beam quality analyzer positioned at the back focal plane of the Fourier lens to observe the beam shaping results.

Second, the phase map calculation is conducted. A critical aspect of this experiment is the accurate computation of the phase distribution to be loaded onto the SLM. The required experimental parameters are input into the algorithm described in Section 2.1, for example: laser wavelength: 795 nm, flat-top beam waist radius: 80 mm, holographic phase map size: $[1000\times 1000]$, pixel size: 9.2 µm, Fourier lens focal length:100 mm. The algorithm is executed, and the resulting phase map $[-\pi , \pi ]$ is saved as a grayscale image $[0, 255]$ representing the target phase distribution over the designated region.

Finally, the grayscale phase map is uploaded to the SLM. Under the influence of the constructed optical system, the laser beam acquires the fixed phase modulation imparted by the SLM. After propagation through the Fourier lens, the intensity distribution of the beam is transformed into a flat-top profile with uniform intensity within a defined area. The resulting beam shape can be directly observed using the beam quality analyzer.

4. Results and Discussion

The phase hologram calculated by the proposed algorithm was uploaded to the liquid crystal spatial light modulator (LCSLM). By spatially separating the zero-order diffraction spot from the modulated beam, a circular flat-top beam was successfully generated, as shown in Figure 12.

To quantitatively evaluate beam shaping performance, the experimental data were imported into MATLAB R2024b for analysis. The cross-sectional intensity distribution along the x-axis of the circular flat-top beam is shown in Figure 13.

From the experimental data, the flat-top region is observed to lie within the interval $[360, 610]$. By extracting this segment and applying Equation (5), the peak non-uniformity was calculated to be 5.07%, consistent with theoretical expectations.

This study introduces an innovative enhancement to the classical Gerchberg–Saxton (GS) phase retrieval algorithm by incorporating an optimized computational model to obtain the phase solution. Numerical simulations showed a reduction in the peak non-uniformity of the output beam to 7.56%. A complete optical experimental system was then established based on the simulation framework, producing a circular flat-top beam with a measured peak non-uniformity of 5.07%. The experimental results exhibit strong agreement with simulation predictions. An error source analysis of the system indicates that discrepancies mainly arise from the following three aspects: $\left(1\right)$ Deviation between ideal and actual input beam profiles: The numerical model assumes an ideal Gaussian beam, whereas the experimentally generated beam exhibits non-ideal features. This asymmetry in input conditions significantly limits the achievable uniformity of the output beam. $\left(2\right)$ Phase quantization error of the LCSLM: The 8−bit (256−level) discrete grayscale modulation introduces quantization errors in phase representation. Additionally, the approximate linear relationship between grayscale and phase is only maintained within a limited range, resulting in phase reconstruction inaccuracies during phase mask generation. $\left(3\right)$ Geometric misalignments in optical alignment: (a) The laser beam center and the geometric center of the LCSLM’s modulation area are difficult to align precisely. (b) Diffraction effects from the edges of the aperture in the spatial filtering stage impose additional wavefront modulation on the output beam.

5. Conclusions

In conclusion, this study experimentally verifies the effectiveness of the proposed beam shaping method in enhancing the sensitivity of SERF atomic magnetometers with the Rb (rubidium) cell, and the experimental schematic and partial physical setup of a SERF atomic magnetometer are shown in Figure 14. The research demonstrates that modifying the Gaussian laser pump via a CNN-GS hybrid algorithm leads to significant performance improvements in ultra-sensitive magnetic field measurement devices based on atomic spin effects. The comparison results of sensitivity values between the method in this paper and traditional methods are shown in Figure 15. Specifically, when using the default Gaussian laser as the pump without phase modulation, the atomic magnetometer exhibited a sensitivity of 14.04 fT/Hz^1/2. Upon introducing the CNN-GS hybrid algorithm we proposed, the sensitivity was improved to 7.80 fT/Hz^1/2, representing a 44% enhancement. This result is in good agreement with theoretical predictions. Furthermore, the algorithm proposed not only boosts sensitivity but also improves laser quality, mitigating potential damage to optical components and devices. These findings highlight the method’s dual utility in optimizing measurement precision and extending device longevity.