An Improved Dual-Sorting NSGA-II Method for Optimal Radiation Shielding Design

Abstract

The Non-dominated Sorting Genetic Algorithm II (NSGA-II) is a widely used approach for solving multi-objective radiation shielding optimization problems, but it struggles to distinguish equally ranked solutions near the Pareto front, leading to reduced selection pressure and slower convergence. This study proposes an improved dual-sorting NSGA-II method that incorporates a novel dominance determination strategy and an O-based sorting mechanism to overcome these limitations. By leveraging the concave characteristics of Pareto-optimal solutions, the dual-sorting approach improves solution ranking accuracy and maintains population diversity. A case study on a multilayer shielding design demonstrates that the proposed method converges faster than the classical NSGA-II and achieves over 40% higher optimization efficiency in later evolutionary stages. These findings highlight the practical significance of the improved algorithm and its potential to accelerate radiation shielding optimization for nuclear facilities, offering a promising tool for efficient shielding design under multi-objective constraints.

1. Introduction

Radiation shielding design is a fundamental aspect of nuclear safety engineering, playing a critical role in minimizing radiation exposure while ensuring the structural and economic feasibility of shielding systems. The design of radiation shielding inherently involves multiple competing objectives, such as radiation attenuation, material weight minimization, mechanical stability, and cost-effectiveness. Given the complexity of these interrelated factors, traditional trial-and-error approaches based on empirical knowledge are often inadequate for achieving optimal shielding configurations. Consequently, the development of computational optimization techniques has become an essential research focus in radiation protection and shielding design [1,2,3,4,5,6].

Multi-objective optimization techniques, particularly evolutionary algorithms, have been widely employed to address the challenges associated with radiation shielding design [7,8,9,10]. Among them, non-dominated sorting-based genetic algorithms, such as multi-objective genetic algorithms (MOGA) [11] and the Non-dominated Sorting Genetic Algorithm II (NSGA-II) [12], have demonstrated remarkable effectiveness in generating diverse trade-off solutions within a single optimization run, without requiring predefined preference settings. Unlike conventional genetic algorithms that rely on multiple iterative computations under varying weight conditions, MOGA and NSGA-II efficiently construct a set of Pareto-optimal solutions, making them particularly well suited for complex multi-objective optimization problems. Compared to MOGA, the NSGA-II algorithm offers superior convergence speed in the later stages of evolution and demonstrates enhanced capability in identifying the extrema of objective functions [13,14], thereby making it especially suitable for radiation shielding design optimization [15,16,17,18].

However, despite its advantages, NSGA-II has inherent limitations that hinder its efficiency in radiation shielding optimization. Monte Carlo methods require extensive computational time for radiation shielding calculations, particularly when dealing with complex coupled models. At the same time, the NSGA-II algorithm encounters convergence difficulties when handling more than three optimization objectives [16,17,18]. These two factors limit the performance and practical applicability of NSGA-II in radiation shielding optimization, making it critically important to improve the algorithm’s intrinsic search efficiency. One of the primary shortcomings of NSGA-II is its inability to effectively distinguish among solutions with the same non-dominated rank in terms of their relative proximity to the Pareto front. As the optimization progresses, this limitation results in reduced selection pressure, slower convergence rates, and difficulties in achieving high-quality solutions within a reasonable computational time frame [19,20]. Addressing these shortcomings is essential to enhance the algorithm’s performance and practical applicability in radiation shielding design.

To overcome these challenges, this study proposes an enhanced dual-sorting NSGA-II method, incorporating a novel dominance determination strategy and an O-based sorting mechanism. By leveraging the concave characteristics of Pareto-optimal shielding solutions, the proposed approach refines the ranking process and maintains evolutionary pressure toward optimality. This dual-sorting mechanism not only improves convergence speed but also enhances solution diversity, ultimately leading to a more efficient and robust optimization process.

This paper is structured as follows: Section 2 presents the methodological framework of the proposed dual-sorting NSGA-II, detailing the integration of the O-based sorting mechanism with conventional NSGA-II ranking. Section 3 provides a case study on a multilayer shielding optimization problem, comparing the performance of the proposed method with the classical NSGA-II in terms of convergence rate and optimization efficiency. Section 4 discusses the key findings and their implications for radiation shielding applications, followed by conclusions and potential future research directions.

2. Improved NSGA-II Methodology with Dual-Sorting Approach

2.1. Workflow of the Dual-Sorting NSGA-II Method

In the classical NSGA-II method, the shield solutions in the population are sorted and ranked based on the dominance to find the Pareto front. However, it does not effectively differentiate solutions with the same rank in terms of their distance from the Pareto front. To address this limitation, we introduce a dual-sorting approach, as illustrated in Figure 1:

• The initial parent population P_t+1 is generated using traditional R-based sorting;
• A modified parent population P’_t+1 is then created using a new sorting method based on the fitness value O, which accounts for the concave characteristics of the Pareto front;
• This population undergoes crossover and mutation to produce the offspring population Q_t+1;
• The combined parent population P_t+1 and offspring population Q_t+1 form the next-generation population C_t+1.

For a detailed explanation of the rank number O and its associated crowding distance $I_o$, refer to Section 2.2.

2.2. O-Based Sorting Method

In combinatorial optimization, optimal radiation shield solutions often exhibit concave characteristics with respect to their objective values. Specifically, for a Pareto-optimal solution curve $y=F\left(x\right)$ defined by two objectives, x and y, any straight line $y=f\left(x\right)$ connecting two points ($x_1, y_1)$ and $\left(x_2, y_2\right)$ on the curve must satisfy Equation (1).

$$F\left(x\right)<f\left(x\right), x\in \left[x_1,x_2\right]$$

(1)

Building on the concave characteristics, a novel dominance determination method is introduced, considering a shielding solution $S_k:\left(x_k, y_k\right)$ and two arbitrary solutions, $S_{k-i}:\left(x_{k-i}, y_{k-i}\right)$ and $S_{k+j}:\left(x_{k+j}, y_{k+j}\right)$, located on either side of $S_k$. If the straight line $y=f_1\left(x\right)$ connecting $S_{k-i}$ and $S_{k+j}$ satisfies the condition $y_k>f_1\left(x_k\right)$, then $S_k$ does not lie on the Pareto-optimal solution curve and is dominated by $S_{k-i}$ and $S_{k+j}$. Conversely, if this condition is not met, $S_k$ is considered a non-dominated solution. Based on this dominance determination method, the fitness value O is introduced to determine the hierarchical ranking of the shielding solution within the population. The detailed computational procedure can be found in Reference [13].

To further enhance the O-value sorting algorithm, this study developed a novel crowding distance calculation method, as illustrated in Figure 2.

For shield solutions $S_{k,O=1}$ with O = 1, the crowding distance $I_o\left(S_{k,O=1}\right)$ is computed using the relative distance between two adjacent solutions, which is the same as the classical approach, as provided in Equation (2):

$$I_o\left(S_{k,O=1}\right)=dis\left(S_{k-1,O=1},S_{k+1,O=1}\right)=\left|{\displaystyle \frac{x_{k+1}-x_{k-1}}{x_n-x_{min}}}\left|+\right|{\displaystyle \frac{y_{k+1}-y_{k-1}}{y_1-y_{min}}}\right|$$

(2)

where $S_{k-1,O=1}$ and $S_{k+1,O=1}$ are the solutions adjacent to $S_{k,O=1}$ on the left and right, respectively, among all solutions with O = 1. The solutions ($x_{min}, y_1)$ and ($x_n, y_{min})$ are the two endpoints of the Pareto curve.

For shielding solutions $S_{l,O>1}:\left(x_l, y_l\right)$ with O > 1, the crowding distance $I_o\left(S_{l,O>1}\right)$ is instead determined by the relative distance to the optimal solution curve, as described in Equation (3):

$$I_o\left(S_{l,O>1}\right)=\left(1-{\displaystyle \frac{y_l}{f_k\left(x_l\right)}}\right)·{\displaystyle \frac{1}{dis\left(S_{k,O=1}, S_{k+1,O=1}\right)}}$$

(3)

where $S_{k,O=1}$ and $S_{k+1,O=1}$ are neighboring shield solutions with objective values $x_k<x_l$ and $x_{k+1}>x_l,$ respectively. The function $y=f_k\left(x\right)$ represents the straight line passing through solutions $S_{k,O=1}$ and $S_{k+1,O=1}$. The crowding distance $I_O\left(S_{l,O>1}\right)$ not only indicates the relative proximity of the shield solution $S_{l,O>1}$ to the optimal solution curve but also signifies its potential contribution to refining the optimal solution upon evolution and inclusion in the optimal solution set.

In the newly proposed non-dominated sorting algorithm based on fitness O, all solutions are initially sorted in ascending order according to their O-values. Subsequently, solutions with the same O-value are further ranked in descending order based on their crowding distance $I_o$. This dual-sorting strategy effectively distinguishes and prioritizes solutions with O = 1—which are closest to the Pareto front (termed optimal solutions)—from solutions with R = 1 and O > 1 (termed suboptimal solutions), ensuring that optimal solutions are placed at the forefront of the population C_t. Additionally, the remaining solutions are ranked according to their relative proximity to the optimal solution curve, further refining the selection process.

2.3. Implementation of the Genetic Algorithm

The genetic algorithm was implemented entirely in MATLAB R2021b and coupled with the Monte Carlo-based particle transport code JMCT [21] to evaluate the radiation dose objective functions for various shielding configurations. MATLAB was used to perform key genetic algorithm operations, including encoding, decoding, sorting, selection, and mutation, as illustrated in Figure 1. During the optimization process, the decoded design parameters were automatically converted into GDML-format geometry models, and simulation input files were generated based on a predefined structural hierarchy. JMCT was then used to calculate the radiation dose objective values for the specific shielding geometry. To enhance simulation efficiency, a global variance reduction technique based on weight windows was applied at each generation and integrated into the transport simulations. This strategy enabled optimized allocation of computational resources and accelerated convergence.

This study employs independent random numbers for coding. To evaluate the robustness of the dual-sorting NSGA-II method, two cases with different mutation strategies were considered:

• Case 1: Each gene segment within a shield solution is evaluated independently to determine whether it should undergo mutation, with a mutation probability of 10%;
• Case 2: In each shield solution, only one gene segment is randomly selected for mutation.

The selection of the next generation of N parent individuals follows a tournament selection algorithm, where the top N/4 shield solutions from the current population C_t are directly retained as part of the parent sets P_t+1 and P’_t+1. The remaining 3N/4 shield solutions are determined through five rounds of tournament selection.

3. Results and Discussion

3.1. Simplified Multilayer Flat Plate Shielding Problem

To assess the optimization performance of the enhanced NSGA-II method, this study considers a simplified multilayer flat plate shielding problem, as illustrated in Figure 3. In this scenario, a multilayer flat plate serves as a shielding barrier for a neutron point source, which follows a Watt spectrum with parameters a = 0.965 and b = 2.29. The shielding plate has a total thickness of 30 cm and fixed dimensions of 1 m × 1 m. The neutron source and the point detector are positioned on opposite sides of the plate, each located 15 cm from the plate surface.

The design optimization aims to minimize two objective functions:

• Total effective dose at the detector position;
• Average density of the multilayer shielding plate, which serves as an indicator of material utilization efficiency.

The neutron and γ-photon energy spectra are recorded for the detector and are converted into effective doses using dose conversion factors from ICRP-74. The Pareto-optimal solutions for this shielding configuration are obtained by adjusting the thickness distribution and selecting appropriate materials for each layer of the multilayer plate. The available shielding materials, selected based on their neutron and photon attenuation properties, are listed in

Table 1. Selectable materials for shielding optimization.

Material	Lithium	Polyethylene	Boron	Aluminum	Iron	Lead
Density g/cm3	0.535	0.95	2.34	2.70	7.86	11.34

.

3.2. Comparison of Convergence: Classical NSGA-II vs. Dual-Sorting NSGA-II

In this section, the shielding problem described in Section 3.1 is optimized using both the classical NSGA-II and the dual-sorting NSGA-II methods. The mutation strategy described in Case 1 was applied. In the multilayer flat plate shielding problem, each generation comprised 20 solutions, with a total of 50 evolutionary generations. A 6-bit random number was used for thickness encoding, with the thickness of each layer determined by Equation (4):

$$h_i=H{\displaystyle \frac{r_i}{{\sum }_{m=1}^6{r}_m}}$$

(4)

where $h_i$ represents the thickness of the ith layer, H denotes the total thickness of the six-layer shielding, and $r_i$ corresponds to the thickness encoding of the ith layer.

To reduce the influence of stochastic effects arising from the genetic algorithm’s selection and mutation mechanisms, five independent optimization runs were performed for both the classical NSGA-II and the dual-sorting NSGA-II methods, ensuring that each run commenced from an identical initial population. The results indicate that each run successfully converged and produced the optimal solution set (i.e., the Pareto front). Moreover, similar evolutionary processes and optimization outcomes were observed across all five runs. The distribution of shield solutions throughout a representative optimization process is depicted in Figure 4.

As shown in the figure, by the seventh generation, the population distributions of the two optimization methods exhibit a notable divergence. The classical NSGA-II method produces shield solutions that are densely concentrated in the central region, remaining relatively distant from the Pareto front. In contrast, the dual-sorting NSGA-II method, influenced by crowding distance settings, shifts the solution set toward lower dose levels, facilitating a more rapid convergence. By generation 10, the solution distribution of the classical NSGA-II method begins to move toward the Pareto front, with some solutions approaching it. However, the dual-sorting NSGA-II method achieves a closer alignment with the Pareto front, forming a more complete and continuous solution set across the density range of 0.8 g/cm³ to 3.6 g/cm³. By generation 20, most solutions from both methods converge toward the Pareto front. However, solutions obtained via the classical NSGA-II method still exhibit a noticeable gap from the optimal curve, whereas those from the dual-sorting NSGA-II method are predominantly concentrated along the Pareto front, covering a broader solution space as the evolution progresses. By generation 50, the shield solutions obtained by both methods are largely positioned along the optimal solution curves, making it increasingly difficult to discern significant differences in distribution between the two approaches based solely on solution positioning.

To further compare the solution capabilities of the classical NSGA-II and the proposed dual-sorting NSGA-II methods on the Pareto front, we examined the shielding configuration with the lowest dose value identified in each of five independent optimization runs and computed the average performance. For the classical NSGA-II method, the average density of the lowest-dose solutions was 1.82 g/cm³, with an average total dose of 1.18 × 10⁻¹⁰ μSv/n. A representative shielding design consisted of a composite structure: 174.1 mm polyethylene + 35.5 mm iron + 88.8 mm polyethylene + 1.6 mm lead. This configuration follows well-known shielding design principles, where iron is embedded between polyethylene layers to enhance neutron moderation, and lead is positioned at the rear to absorb secondary gamma photons.

In comparison, the dual-sorting NSGA-II method produced shielding solutions with an average density of 2.17 g/cm³ and an average dose of 1.14 × 10⁻¹⁰ μSv/n. A typical configuration featured a similar composite structure: 184.7 mm polyethylene + 40.0 mm iron + 66.7 mm polyethylene + 8.6 mm lead. These results confirm that both NSGA-II variants are effective in identifying optimal shielding configurations, while the dual-sorting method exhibits improved convergence accuracy and greater consistency in locating near-optimal low-dose solutions.

The number of solutions with O = 1 (i.e., optimal solutions) in population C_t, as well as the number of solutions with R = 1 and O > 1 (i.e., suboptimal solutions), serve as key metrics for assessing the effectiveness of an optimization method in approximating the Pareto front. Figure 5 presents the evolutionary progression of optimal and suboptimal solutions within population C_t over successive generations. The results demonstrate that the dual-sorting NSGA-II method consistently outperforms the classical NSGA-II method in maintaining a higher count of both optimal and suboptimal solutions beyond generation 15. By the final generation, the Pareto front obtained via the dual-sorting NSGA-II method consists of 29 optimal solutions and 74 suboptimal solutions, whereas the classical NSGA-II method yields only 22.6 optimal solutions and 49 suboptimal solutions. This outcome underscores the superior performance of the dual-sorting NSGA-II approach, which more effectively approximates the Pareto front and facilitates a more refined optimal solution set.

For t > 16, both optimization algorithms exhibit a predominance of suboptimal solutions over optimal solutions, necessitating an increased selection of optimal solutions for the next-generation parent population P_t+1 to sustain evolutionary pressure. In the classical NSGA-II method, the proportion of suboptimal solutions selected for P_t+1 gradually increases, as depicted in Figure 6. The inclusion of optimal solutions in P_t+1 begins to decline from t = 14, and by the later evolutionary stages, optimal and suboptimal solutions are selected in equal proportions, each accounting for 50% of P_t+1.

As shown in Figure 5 and Figure 6, although both the absolute number and relative proportion of suboptimal solutions in the Pareto front are higher for the dual-sorting NSGA-II method compared to the classical NSGA-II method, the O-value-based sorting strategy effectively filters out suboptimal solutions, leading to a greater concentration of optimal solutions in P_t+1. By t = 50, optimal solutions constitute over 80% of P_t+1 in the dual-sorting NSGA-II method, demonstrating its ability to maintain selection pressure on high-quality solutions. Additionally, in this method, solutions beyond the optimal and suboptimal sets are more likely to be selected for P_t+1, thereby improving population diversity and enhancing the overall robustness of the optimization process by preventing premature convergence and maintaining a more comprehensive exploration of the solution space.

3.3. Optimization Efficiency Analysis

3.3.1. Population Superiority Metrics

To quantitatively evaluate the optimization efficiency across different genetic algorithm configurations, this study introduces a normalized population superiority metric $V\left(C_t\right)$. For optimization problems with two objective functions, x and y, the normalized population superiority parameter $V\left(C_t\right)$ is defined by Equation (5):

$$V\left(C_t\right)={{\displaystyle \int }}_{x_{t,min}}^{x_{max}}{\displaystyle \frac{F\left(x\right)}{G\left(x\right)·\left(x_{max}-x_{min}\right)}}dx$$

(5)

where $y=F\left(x\right)$ represents the Pareto curve, which is constructed by selecting the optimal solutions from all simulation results and performing a nonlinear fit from ($x_{min}$,$y_{max}$) to ($x_{max}$,$y_{min}$). Conversely, $y=G\left(x\right)$ represents the step curve, comprised of optimal and suboptimal solutions in population C_t with the minimum objective value $x_{t,min}$. A schematic diagram of the population superiority parameter is shown in Figure 7.

As illustrated in Figure 7, the region where $f\left(x\right)>G\left(x\right)$ represents all shield solutions dominated by the population $C_t$. The relative distance between these dominated solutions and the Pareto-optimal solutions is quantified by ${\displaystyle \frac{F\left(x\right)}{G\left(x\right)}}$. As the optimal and suboptimal solutions within population $C_t$ converge toward the Pareto-optimal set, whether through tighter clustering or expansion across a broader region, the population superiority metric $V\left(C_t\right)$ increases accordingly. This metric simultaneously characterizes the proximity of shield solutions within a given population to the optimal solution and the completeness of the optimal solution set, thereby providing a comprehensive assessment of population superiority.

3.3.2. Comparison of Optimization Efficiency Across Different Methods

Based on the population superiority metric defined in Section 3.3.1, the overall evolutionary speed of the populations obtained using the classical NSGA-II and the dual-sorting NSGA-II methods, as described in Section 3.2, was analyzed. To reduce variability, the population superiority values were averaged over five independent optimization runs. Additionally, to verify the scalability of the dual-sorting NSGA-II method, the population evolution speed was further investigated under an alternative genetic encoding scheme (Case 2) with a different optimization efficiency. The results are presented in Figure 8.

Figure 8a illustrates the evolution of the population superiority metric $V\left(C_t\right)$ over generations t for both the classical NSGA-II method and the dual-sorting NSGA-II method. Additionally, Figure 8b provides a detailed comparison of the optimization efficiency between the two approaches. As shown in Figure 8, following an initial phase characterized by high randomness, the dual-sorting NSGA-II method surpasses the classical NSGA-II method in population superiority by generation 10 in both Case 1 and Case 2. By generation 15, a significant divergence in superiority becomes evident between the two approaches.

In Case 1, aside from minor fluctuations between generations 25 and 30, the dual-sorting method consistently exhibits over a 20% improvement in optimization efficiency compared to the classical NSGA-II method. Notably, in the final stages of evolution, when $V\left(C_t\right)$ > 0.8 (i.e., t > 28), the efficiency advantage of the dual-sorting algorithm becomes increasingly pronounced, reaching an enhancement of over 40% at $V\left(C_t\right)=0$.91.

A similar trend is observed in Case 2. Once $V\left(C_t\right)$ > 0.8 (i.e., t > 20), the dual-sorting method’s efficiency improvement exceeds 30%. Moreover, compared to Case 1, the mutation strategy in Case 2 results in higher optimization efficiency beyond generation 10.

These findings highlight the effectiveness of the O-based sorting mechanism and the new crowding distance metric in reducing interference from suboptimal solutions and maintaining evolutionary pressure on the Pareto-optimal front, particularly when the optimal solution set is more comprehensive.

4. Conclusions

The NSGA-II-based radiation shielding optimization method effectively constructs the Pareto front; however, it experiences a decline in evolutionary pressure beyond the early stages of evolution, primarily due to difficulties in selecting optimal solutions. To overcome this limitation, this study introduces a novel fitness value and its corresponding crowding distance, leveraging the concave function properties of the Pareto front in radiation shielding optimization. Accordingly, a dual-sorting NSGA-II method is proposed, integrating both O-based and R-based sorting mechanisms to refine Pareto front identification. This approach retains the R-based sorting method’s ability to preserve a diverse solution set while significantly enhancing efficiency in locating optimal solutions.

To assess the effectiveness of the dual-sorting NSGA-II method, this study applies it to a simplified multilayer flat plate shielding optimization problem and compares its performance against the classical NSGA-II method. Repeated computational experiments demonstrate that the dual-sorting NSGA-II method markedly improves the selection probability of optimal solutions. As a result, it achieves superior optimization efficiency, notably in the middle and late evolutionary stages, with performance improvements exceeding 40% in certain phases. This enhanced efficiency underscores the practical advantages of the dual-sorting NSGA-II method in radiation protection design, enabling a more effective determination of shielding configurations while reducing computational time and resource consumption.