Multi-Criteria Selection of Network Security Configuration Using NSGA-II

Abstract

The problem of multi-criteria selection of network security configurations (NSC) under resource constraints and the necessity to comply with information security (IS) policies is addressed in this study. A formal mathematical model of the problem has been developed, encompassing the definition of a set of possible security mechanism configurations, the formalization of objective functions reflecting security levels, throughput, and deployment costs, and the introduction of constraints on feasible solutions. The NSGA-II (Non-dominated Sorting Genetic Algorithm II) optimization algorithm is employed to generate a set of Pareto-optimal solutions, ensuring uniform coverage of compromise configurations. A software package implemented in Python 3 incorporates modules for population generation, fitness evaluation, selection, crossover, mutation operators, and result visualization. Computational experiments (CE) were conducted to validate the effectiveness of the proposed approach. The evolution dynamics of the Pareto hypervolume were analyzed, the uniformity of solution distribution in the objective space was studied, and the impact of algorithm parameters on convergence to the optimal solution was examined. The results demonstrate that the proposed methodology enables the formation of NSC sets that achieve a balanced trade-off between security, throughput, and IS system deployment costs.

1. Introduction

The selection of network security tools (NSTs) required for implementing information security policies presents a complex multi-criteria problem due to the vast number of possible configurations, a wide range of threats, and limited available resources. Information security policies should not be considered as a single static document but rather as a set of interrelated rules and requirements that define acceptable system behavior in various operational scenarios. In modern information and telecommunication networks, these policies vary depending on multiple factors, including the type of processed data, industry standards, legal regulations, and specific business processes of companies or organizations. For example, corporate networks may implement policies aimed at protecting employee personal data, preventing confidential information leaks, segmenting networks to limit the impact of potential incidents, and regulating remote access and authentication mechanisms.

The diversity of information security policies necessitates a comprehensive approach to selecting NSTs, as each tool has unique functional characteristics, potential limitations, and an impact on key system performance indicators, including security, performance, and deployment costs. The problem is further complicated by the fact that existing NST selection methods do not account for all possible combinations of security mechanisms and are often focused on a limited number of security solution classes. Specifically, some studies [1,2,3,4] propose approaches that group security tools based on generalized functional attributes; however, such classifications do not always capture differences between components that perform similar but not identical tasks. For instance, firewalls and packet filters may be classified under the same category, despite their differences in handling connection state analysis.

Optimization approaches presented in various studies [1,2,5,6] primarily focus on minimizing risks associated with the incomplete implementation of information security policies. However, these methods often overlook deployment costs, computational resource constraints, and performance requirements, reducing their applicability in real corporate network environments. Additionally, some works only address the problem of finding “nearly equivalent” solutions, yet they either lack formal criteria for equivalence or rely on heuristic definitions, complicating the objective classification of security tools.

In this study, NSTs are considered as a set of functional modules capable of transforming incoming network traffic according to predefined security policies. Each module, in turn, can be represented as an abstract computational automaton that generates not only output traffic but also auxiliary information, including event logs and policy violation alerts. The collection of these modules can be divided into equivalence classes, where identical data transformations occur under the same input conditions. This allows the NST selection problem to be formulated as a multi-criteria optimization task aimed at identifying a security tool configuration that best aligns with the given policies while maintaining a balance between security, performance, and deployment costs.

In the present study, the selection of NSC is considered in the context of generally accepted information security policy frameworks. In particular, the international standards ISO/IEC 27001 [7] and ISO/IEC 27002 [8] define the requirements for establishing an information security management system and formalize a set of controls aimed at asset protection, access management, network security, and incident monitoring. Similarly, the NIST Cybersecurity Framework and the NIST Risk Management Framework (RMF) provide a process-oriented risk management model that includes threat identification, selection of protective measures, their implementation, and subsequent effectiveness evaluation. In [2,4,6], these paradigms were used by the authors as a basis for formalizing requirements for security mechanisms and their configurations in industrial and corporate networks. Within the scope of this work, information security policy requirements are interpreted as formal constraints and objective functions of the optimization problem. This enables us to link regulatory requirements (compliance, certification, minimum levels of protection and throughput) with the procedure for automated selection of NSC.

This research focuses on developing a solution method for this problem using the NSGA-II (Non-dominated Sorting Genetic Algorithm II). This algorithm offers several advantages over traditional multi-criteria optimization methods, as it produces not a single optimal solution but a set of Pareto-optimal configurations that provide different trade-offs between key performance indicators. Unlike deterministic methods, NSGA-II can account for complex nonlinear dependencies between system parameters, making it applicable in multidimensional solution spaces. This study proposes a mathematical model for the NST selection problem, develops an algorithm for solving it using NSGA-II, and analyzes the obtained results in terms of their theoretical consistency and practical applicability in corporate information systems.

The structure of the article is organized as follows. Section 2 presents a review of the relevant literature and identifies the limitations of existing methods and models. Section 3 outlines the aims and objectives of the study. Section 4 presents the proposed mathematical model, where protection mechanisms are modeled as functional automata, along with the corresponding optimization algorithm. Section 5 presents the computational experiments and discusses the interpretation of the obtained Pareto-optimal solutions. The conclusions summarize the main results of the study.

2. Literature Review

The issue of multi-criteria optimization for elements of complex computer systems and networks, including information security (IS) challenges, has remained an active research area for decades. This field employs a wide range of scientific methods and approaches, ranging from classical multi-criteria decision-making methods to modern computational models.

In [1], the authors propose a method for prioritizing information security measures for cloud computing and wireless sensor networks using fuzzy AHP (Analytic Hierarchy Process), which accounts for uncertainties and complexities in decision-making. In [2], a review of multi-criteria decision-making algorithms for network selection in heterogeneous wireless networks is conducted, focusing on methods that improve service quality and adaptive connectivity. The authors of [3] provide a systematic literature review on the application of multi-criteria decision-making methods for information security risk assessment, highlighting key trends and gaps in existing research. In [4], the author explores cybersecurity challenges in smart grids and suggests a hierarchical multi-criteria analysis approach, including artificial intelligence techniques, to optimize cybersecurity decisions.

The study in [5] describes a methodology for multi-criteria decision-making in cybersecurity, based on evaluating system capabilities to enhance security strategies. In [6], the authors analyze the correctness of security-related decision-making using multi-criteria evaluation, emphasizing the importance of accuracy and reliability in security assessments. The work in [9] presents an improved TOPSIS method for evaluating research productivity in cloud security using multi-criteria assessment. The authors of [10] develop efficient numerical methods for multi-objective cybersecurity planning, demonstrating their application in security modeling and optimization.

In [11], a scalable min-max multi-criteria optimization model is proposed for cybersecurity, leveraging probabilistic attack graphs to consider complex threat scenarios and minimize risks. The study in [12] introduces a hybrid multi-criteria system for assessing the resilience and security of web applications based on fuzzy logic, enabling more precise security analysis while considering sustainability factors. The authors of [13] apply genetic algorithm-based multi-criteria optimization methods to design an optimal cybersecurity structure, enhancing the adaptability and efficiency of protective measures. The work in [14] proposes a hybrid MCDM model combining DANP and PROMETHEE II methods for cybersecurity assessment in Industry 4.0 environments, allowing for the consideration of complex dependencies and priorities in digital manufacturing.

Recent research has increasingly focused on evolutionary modeling techniques, such as genetic algorithms and their modifications, which demonstrate high efficiency in solving complex multi-criteria problems by finding balanced solutions amidst conflicting objectives. In [15], a genetic algorithm is used to select an optimal set of security tools for an educational information system, ensuring a balance between efficiency and resource constraints. The authors of [16] investigate the use of genetic algorithms for feature ranking in multi-criteria optimization, emphasizing improvements in selection accuracy. The comprehensive review in [17] explores the NSGA-II algorithm for multi-criteria optimization, highlighting its key achievements and application areas.

In [18], the authors propose a security-oriented task scheduling model for computational grid systems using NSGA-II, accounting for both computational and security criteria. The study in [19] introduces a multi-criteria framework for securing blockchain systems in fog-cloud networks for vehicle-to-infrastructure applications, focusing on both security and efficiency. The work in [20] presents an intelligent FoGMatch approach for multi-criteria planning in IoT-fog environments using game theory, optimizing resource allocation and improving network computing efficiency.

Several studies approach the optimization of NSC solely based on minimizing the risk of policy violations [1,2,4,7], without considering factors such as deployment costs and operational expenses.

Summarizing the analysis of prior research, we conclude that the NSGA-II (Non-dominated Sorting Genetic Algorithm II) method is well-suited for solving the problem of multi-criteria selection of NSTs in accordance with established information security policies. This algorithm is particularly effective for tasks involving multiple conflicting objectives, as it enables the generation of a Pareto-optimal solution set. The application of NSGA-II ensures an effective distribution of solutions along the Pareto front, which is essential for a relevant evaluation and selection of optimal security system configurations, considering various aspects such as security level, performance, and cost.

3. Research Objectives

This study aims to develop and formalize a multi-criteria model for selecting network security tools (NSTs) that ensures compliance with predefined information security policies while considering key parameters such as security level, throughput, and implementation cost.

To achieve the specified goal, the following tasks must be addressed:

To develop a theoretical automata-based approach for modeling network security tools by representing them as computational automata with internal states;

To adapt the NSGA-II genetic metaheuristic to solve a specific combinatorial optimization problem for network defense configuration;

To validate the proposed model via computational experiments and perform Pareto-front analysis to identify latent relationships between deployment cost and system resilience, providing decision-makers with a well-justified basis for selecting network security configurations.

4. Proposed Model and Method

4.1. Multi-Criteria Optimization Model

We formalize the model for solving the multi-criteria optimization problem of selecting an ISS (Information Security System).

We define the following notation:

$G=(H,L)$—a weighted directed graph modeling the network, where $H$ is the set of hosts and $L$ is the set of communication channels between them.

$F=\left\{F_1,F_2,\dots ,F_m\right\}$—A set of possible ISS (Information Security Systems), each possessing functional characteristics such as the type of protection mechanism, packet filtering level, encryption algorithms, and authentication control.

$P=\left\{P_1,P_2,\dots ,P_m\right\}$—A set of information security (IS) policies. IS policies are defined as a set of rules, each representing a set of requirements for processing, transmitting, and storing information. Each rule $\left(P_i\right)$ may include restrictions on the use of specific ISS classes, permissible trust levels for network nodes, and requirements for event logging.

$C\left(F_i\right)$—the cost of ISS $F_i$. The function $C\left(F_i\right)$, defined based on the model presented in [21], includes expenses for licensing, technical maintenance, computational resources, and traffic consumption.

$R\left(F_i\right)$—the security rating of ISS $F_i$, determined based on vulnerability mitigation criteria; for example, according to [22,23]. It is calculated based on the elimination of vulnerabilities and the ability to counter known types of attacks. This assessment can be expressed as a function depending on the number of neutralized threats, the update frequency of signatures, and the effectiveness of access control.

$T\left(F_i\right)$—the throughput of ISS $F_i$, determined based on the packet processing algorithms used and the hardware characteristics of the device. For example, one of the key challenges of modern ISS, including Next-Generation Firewalls (NGFW), is limited throughput. Therefore, this parameter serves as one of the critically important criteria to consider when selecting and deploying an ISS. Throughput requirements may vary for different network segments depending on their functional load and the volume of transmitted traffic. In the decision-making process for ISS selection, factors such as solution rating, throughput, economic feasibility, and regulatory requirements, including the availability of relevant compliance certificates, are considered.

$S\left(F_i\right)$—a binary indicator of the presence of a compliance certification. Let $S\left(F_i\right)\in \left\{0,1\right\}$ be the indicator of compliance certification with regulatory requirements, where $S\left(F_i\right)=1$ if the information security system has been certified according to specific standards (e.g., FIPS, ISO/IEC 27001), and MMM if such certification is absent.

As mentioned above, the network model can be represented as a weighted directed graph $G=(H,L)$ corresponding, for example, to Figure 1. Here, $L=\left\{\left(h_i,{\mu }_{ij},h_j\right)\right\}$ is the set of weighted arcs that model communication channels between hosts. The weights of the arcs ${\mu }_{ij}\in$ℝ are determined by the throughput of the corresponding communication channel (see Figure 1). A host refers to a device that has one or more dedicated IP addresses. Within this model, there can be no more than two directed arcs between each pair of hosts (one in each direction): $\left(h_i,{\mu }_{ij},h_j\right)$ and $\left(h_j,{\mu }_{ji},h_i\right)$. In this case, loops in the graph $G=(H,L)$ are excluded. Each vertex of the graph may correspond to one or more IP addresses. In general, ISS can be deployed in the network in various ways. For each class of these security solutions, specific requirements can be defined for their placement and relative positioning concerning other protection mechanisms. According to [13], there are no strict restrictions on the placement of firewalls (FW) within the network infrastructure. They can be installed at the boundaries of network segments, in front of specific groups of hosts (e.g., organizational units critical to information security), or in front of individual servers [24,25].

The network graph $G=(H,L)$ is employed to specify the domain of feasible solutions. The graph specifies the topological constraints of the system, ensuring that each protection mechanism is placed only on nodes or edges consistent with its functional purpose, for example, a firewall located at the boundary of network segments. Thus, the graph acts as a filtering mechanism that eliminates technically impossible device deployment configurations before the network security optimization process.

Within the scope of this work, a graph-based network model is used to conceptually justify the applicability of security mechanisms in a networked environment. The optimization problem is focused on selecting the composition of the security configuration rather than their specific placement. A detailed analysis of network topology and the optimization of device placement are considered as a separate problem and may be the subject of future research.

This approach establishes a multi-layered defense (defense-in-depth). The optimal placement of ISS within an organization’s infrastructure is determined by the location of vulnerable systems and potential attack sources. To ensure the proper deployment of ISS within the network, the following principles can be formulated:

ISS should be positioned along the network traffic path from a potential attacker to a vulnerable system. If the network architecture allows multiple possible placement points, the specific location can be selected from a set of feasible options.

If the information security (IS) policy allows ISS placement at multiple feasible points along the route between the attacking node and the protected system, the selection of a specific location may be based on additional criteria. The consideration of such criteria falls beyond the scope of this study. Henceforth, it is assumed that ISS is deployed in any of the permissible network segments.

If ISS can be deployed at multiple points within the network, a multi-layered defense strategy can be implemented. In this case, security measures can be installed on any subset of the permissible network segments, enhancing the overall level of protection by duplicating security functions. However, as the number of deployed nodes increases, the total costs of implementation and operation also rise. Therefore, the total number of deployed ISS should be limited by economic feasibility, which is determined by the acceptable level of risk that their deployment aims to mitigate.

The problem of selecting the optimal set of ISS in the context of multi-layered defense falls beyond the scope of this study and requires separate consideration.

Within the proposed model, the network is modeled as a graph $G=(H,L)$ to capture node connectivity and to assess the admissibility of protection-mechanism placement along traffic paths. While topology optimization is not considered in this work, the graph model provides the foundation for constraint validation. Accordingly, a protection mechanism is included in the configuration only when it resides in a permitted segment of the graph. Each security tool $F_i$ is formally modeled as an abstract automaton. The automaton processes and transforms the incoming packet stream. For the purposes of optimization, the automaton’s behavior is abstracted into a set of static attributes—such as security rating, processing delay (or throughput), and cost—defined as functionals of its internal state. Modeling security tools, including firewalls, intrusion detection systems (IDS), and proxies, as finite automata instead of static graph nodes is essential for accurately capturing their state-dependent behavior in response to network traffic. Each security mechanism $F_i$ is modeled as a tuple $(S,\Sigma ,\delta ,s_0,F)$, where the transition function depends not only on packet header attributes but also on the current load state of the device. Although the present optimization procedure relies on averaged parameters, the automata-theoretic model lays the groundwork for future extensions enabling more detailed and state-aware analysis. This framework enables adaptive network reconfiguration in response to evolving attack vectors, as transitions of the automaton from the “Normal Operation” state to the “Under Attack” state lead to dynamic adjustments of the objective function weight coefficients. Moreover, representing security mechanisms as abstract automata $F_i$ allows the analysis to extend beyond the limitations of static network security assessment. Each automaton is defined not only by its functional properties but also by its corresponding state graph. Within the state graph, transitions are governed by the type of incoming traffic as well as the current load conditions of the system. This enables the consideration of nonlinear packet delay effects. Specifically, when in the “High Load” state, the automaton adjusts its temporal characteristics. This has a direct impact on the throughput objective function ${\phi }_2$.

It should be noted that the security objective function ${\phi }_1$ is evaluated as an aggregated rating. An additive rating model is used as a reasonable approximation for comparing configurations, recognizing that actual security is inherently probabilistic. Accordingly, the security objective function is defined as follows:

$${\phi }_1\left(F^{\ast }\right)={\displaystyle \sum _{F_i\in F^{\ast }}R\left(F_i\right)}\to \max .$$

(1)

Maximization of Throughput.

When protection mechanisms are deployed sequentially along a network path, the overall throughput of the configuration is governed by the minimum throughput of the deployed mechanisms rather than their sum. Accordingly, the throughput of each protection chain is determined as follows:

$${\phi }_2\left(F^{\ast }\right)=\underset{F_i\in F^{\ast }}{\min }\left(T\left(T_i\right)\right)\to \max$$

(2)

Maximization of the Number of Certified ISS.

$${\varphi }_3\left(F^{\ast }\right)={\displaystyle \sum _{F_i\in F^{\ast }}S\left(F_i\right)}\to \max .$$

(3)

Minimization of Cost.

$${\varphi }_4\left(F^{\ast }\right)={\displaystyle \sum _{F_i\in F^{\ast }}C\left(F_i\right)}\to \min .$$

(4)

Considering the following constraints:

Minimum throughput for each communication channel.

$$T\left(F^{\ast }\right)\ge T_{\min }.$$

(5)

Budget constraint.

$$C\left(F^{\ast }\right)\le C_{\max }.$$

(6)

Minimum number of certified ISS.

$${\displaystyle \sum _{F_i\in F^{\ast }}S\left(F_i\right)}\ge S_{\min }.$$

(7)

The complexity of solving the multi-criteria selection problem for ISS is determined by several factors, including both computational and theoretical-methodological aspects. This problem belongs to the class of combinatorial multi-criteria optimization problems, where multiple conflicting criteria must be simultaneously considered when selecting the optimal security system configuration.

Firstly, the complexity is driven by the dimensionality of the solution space. Let there be a set of possible ISS configurations $F=\left\{F_1,F_2,\dots ,F_m\right\}$, where each configuration $F^{\ast }\subseteq F$ represents a permissible combination of security mechanisms. Then, the total number of possible solutions grows exponentially as the number of available security measures increases, which can be expressed as follows: $\left|\chi \right|=2^m,$ where $\left|\chi \right|$ is the number of possible ISS combinations. For large values of $m$, an exhaustive search of all solutions becomes computationally infeasible.

Secondly, the multi-criteria nature of the problem necessitates finding a set of Pareto-optimal solutions. This means that instead of a single optimal solution, the goal is to identify a set of solutions where no criterion can be improved without worsening at least one other criterion. Let a vector of objective functions be given as $F\left(x\right)=\left(F_1\left(x\right),F_2\left(x\right),\dots ,F_k\left(x\right)\right), x\in \chi ,$ where each function $F_i\left(x\right)$ represents the alignment of an ISS configuration with a specific criterion (e.g., security level, throughput, deployment cost). Then, it is necessary to determine the subset of Pareto-optimal solutions ${\chi }^{\ast }\subseteq \chi$, such that $\forall x^{\prime }\in {\chi }^{\ast }, \cancel{\exists }\in \chi , x\succ x^{\prime },$ where $x\succ x^{\prime }$ indicates that $x$ dominates $x^{\prime }$ across all criteria. The search for the Pareto-optimal set requires evaluating all possible solutions and performing dominance comparisons, which significantly increases computational complexity.

The third complicating factor is the presence of constraints (5)–(7), which impose restrictions on the set of feasible solutions. While these constraints reduce the solution space, they also make the problem even more complex, as the optimization process must simultaneously account for both the objective functions (1)–(4) and the constraints (5)–(7).

Furthermore, the complexity of solving the problem increases due to the need to ensure uniform coverage of the Pareto set. Traditional multi-criteria optimization methods may lead to a concentration of solutions in narrow regions of the Pareto front, reducing the variability of available trade-off solutions. Effective distribution of points along the Pareto set requires the use of mechanisms such as crowding distance, further increasing computational complexity.

Additionally, complexity is driven by the dynamic nature of information security (IS) policies. Security policies can change over time, periodically requiring the re-optimization of previously found solutions. Unlike static multi-criteria optimization problems, in this case, it is necessary to account for the adaptability of solutions to evolving system requirements.

Thus, the main sources of complexity in solving the multi-criteria selection problem for ISS are: the exponential growth of the solution space dimension; the multi-criteria nature of the problem and the need to form a set of Pareto-optimal solutions; the presence of strict constraints on the selection of ISS configurations; the challenge of ensuring uniform coverage of the Pareto set; and the dynamic changes in information security policies.

Considering the above, we assume that evolutionary algorithms, such as NSGA-II, would be a suitable approach for solving the given problem. NSGA-II demonstrates high computational efficiency in handling multi-criteria optimization problems with large solution spaces and complex constraints.

4.2. Applied Solution Method

The NSGA-II algorithm was selected due to its well-established efficiency in solving multi-objective optimization problems characterized by a small number of conflicting objective functions, typically between two and four. Unlike NSGA-III, which is intended for many-objective optimization problems, NSGA-II ensures a more uniform spread of solutions along the Pareto front in the given search domain.

The numerical implementation methodology of the proposed model consists of the following stages:

1. Encoding and population initialization. Each protection system configuration is encoded as a chromosome, represented by a vector of discrete variables, in which each gene denotes the index of the selected network security tool (NST) for a particular network node. An initial population of size $N$ was generated using a stochastic procedure to provide wide coverage of the search space.

2. Fitness evaluation. The objective function values for each individual in the population are computed in accordance with Equations (1)–(7). At this stage, a hard constraint handling mechanism is incorporated into the model. Specifically, solutions that violate information security policy requirements or budget limitations are assigned penalty coefficients or eliminated from the subsequent reproductive cycle.

3. Application of genetic operators to generate offspring solutions. Binary tournament selection was employed to generate the new population, promoting the dominance of the most fit individuals. Crossover and point mutation operators were applied to the selected parent pairs to generate offspring solutions. These operators were specifically adapted to address the combinatorial characteristics of the network defense configuration selection problem.

4. Non-dominated sorting of the population. All obtained solutions were categorized into hierarchical Pareto fronts based on dominance relationships. This makes it possible, during computational experiments, to identify a subset of configurations that cannot be improved in one objective without compromising another.

5. Calculation of the crowding distance for each individual. A density-based operator was applied to preserve genetic diversity and avoid premature convergence to local optima. This guarantees a well-distributed Pareto front, offering decision-makers a broad spectrum of optimal trade-off solutions.

6. Hypervolume indicator analysis. The hypervolume metric was employed as both a termination criterion and a performance evaluation measure for the algorithm. Stabilization of the hypervolume indicator throughout the evolutionary process confirms the convergence of the algorithm and the optimality of the resulting solution set.

The algorithm produces an approximation of the Pareto front, consisting of a set of optimal network protection configurations. This allows the transition from a subjective selection of security mechanisms to a formalized decision-making framework grounded in the analysis of quantitative trade-offs among security, cost, and network performance.

4.3. Algorithmic Description of the Proposed Optimization Method

Configurations are encoded as a binary vector $X$ of length $N$ (where $N=9$ denotes the number of available protection mechanisms in the database). The condition $x_i=1$ denotes the inclusion of the $i$-th protection mechanism in the configuration. The constraint-handling procedure is defined as follows. The standard NSGA-II algorithm is extended with a constrained dominance mechanism for handling constraints. Any solution that violates the budget constraint ($C_{max}$) or the minimum throughput requirement ($T_{min}$) is treated as worse than any feasible solution, irrespective of its objective function values. Equivalently, under this rule, any solution violating at least one constraint (e.g., $Cost>Budget$) is deemed inferior to all feasible solutions. This approach enables the NSGA-II algorithm to effectively discard infeasible configurations without the need for complex penalty functions.

• Input:
• Population size (N)
• Maximum generations (G)
• Crossover probability (Pc)
• Mutation probability (Pm)
• Output:
• Pareto-optimal set of NSC
• 1: Initialize population P(0) with N random NSC
• 2: Evaluate objective functions for each individual in P(0)
• 3: Perform non-dominated sorting to determine Pareto ranks
• 4: Compute crowding distance for diversity preservation
• 5: Set generation counter g ← 0
• 6: while g < G do
• 7: Generate offspring population Q(g) using selection, crossover (Pc), and mutation (Pm)
• 8: Evaluate objective functions for each individual in Q(g)
• 9: Merge P(g) and Q(g) into R(g) ← P(g)∪ Q(g)
• 10: Perform non-dominated sorting on R(g)
• 11: Select the best N individuals to form the next generation P(g+1) using Pareto dominance and crowding distance
• 12: g ← g + 1
• 13: end while
• 14: Return the final non-dominated set as the Pareto-optimal configurations of NSTs

The proposed pseudocode represents the implementation of NSGA-II for solving the problem of selecting the optimal configuration of an information security system (ISS) while considering information security policies. The input parameters of the algorithm include the population size $\left(N\right),$ the maximum number of generations $\left(G\right),$ the crossover probability $\left(P_c\right)$ and the mutation probability $\left(P_m\right)$, which regulate the evolutionary process. At the first stage, the population is initialized with random configurations of the information security system (ISS), each of which represents a potential solution to the problem. Next, the objective function values are evaluated, taking into account network security metrics, performance, and compliance with security policies. After evaluating the solutions, non-dominated sorting is performed to determine the Pareto fronts in the population. To maintain solution diversity, the “crowding distance” metric is computed, preventing the algorithm from prematurely converging to a local optimum.

In the main part of the algorithm, the population is evolved through selection, crossover, and mutation mechanisms. The new individuals form the offspring population $\left(Q(g)\right),$ which is combined with the current population, $\left(P(g)\right),$ for the subsequent execution of non-dominated sorting. The final population update is performed by selecting the top $N$ individuals based on Pareto dominance and crowding distance. The algorithm terminates upon reaching the maximum number of generations $\left(G\right).$ As a result, a set of Pareto-optimal configurations of the information security system (ISS) is formed, representing optimal trade-offs between security, throughput, and compliance with information security policies. The proposed methodology ensures the formation of balanced solutions for selecting network security measures, while the use of NSGA-II enables high efficiency and alignment of the optimization process with the real-world information security requirements of companies.

An adaptive mutation mechanism is employed in this study, in which the mutation probability varies depending on the generation number as follows:

$$p_m\left(g\right)=p_{\max }-{\displaystyle \frac{g}{G}}\cdot \left(p_{\max }-p_{\min }\right),$$

(8)

where $g$ denotes the index of the current generation and $G$ represents the maximum number of generations. As a result, intensive exploration is achieved in the early stages, while solution stabilization is ensured in the later stages of the evolutionary process.

4.4. Limitations

The developed approach to solving the considered problem, despite its effectiveness, has a number of limitations that need to be taken into account for practical application. Firstly, if the formalized criteria do not fully reflect the actual network security requirements, the solutions found may be theoretically optimal but practically ineffective [26]. Secondly, the NSGA-II algorithm is heuristic and does not guarantee finding the global optimum. While it effectively constructs a set of Pareto-optimal solutions, its performance depends on algorithm parameters such as population size, mutation probability, and the number of generations. Incorrect selection of these parameters may result in incomplete coverage of the possible solution set or excessively long algorithm execution time.

Thirdly, the computational complexity increases with the growing dimensionality of the problem. Consequently, for large corporate networks containing dozens of nodes and hundreds of possible security system configurations, computational complexity may become a critical factor limiting the applicability of the method without additional optimizations.

Additionally, the proposed approach does not take into account dynamic changes in the network, such as topology modifications, the emergence of new cybersecurity threats, or security policy updates. In real-world conditions, the security system configuration may require periodic adaptation, necessitating either a rerun of the algorithm or the development of an additional mechanism for dynamic solution adjustment.

It should also be noted that a constrained dominance mechanism was employed for constraint handling. Solutions that violate hard constraints (budget, minimum throughput, certification requirements) are considered dominated with respect to feasible solutions, regardless of the values of the objective functions. Therefore, during the evolutionary process, priority was always given to feasible configurations, which prevented economically or technically unacceptable information security solutions from appearing in the final Pareto set.

5. Computational Experiments and Obtained Results

For the correct execution of computational experiments (CE), input parameters defining the model configuration (1)–(7) were determined.

Table 1. Input parameters of the computational experiment.

Parameter	Notation	Value	Description
Population Size	$N$	100	The number of individuals in each population throughout all iterations of the evolutionary process.
Number of Generations	$G$	50	The number of algorithm iterations after which the set of Pareto-optimal solutions is formed.
Crossover Probability	$P_c$	0.7	The proportion of individuals undergoing recombination in each generation to maintain genetic diversity.
Mutation Probability	$P_m$	0.4	The initial mutation probability, which adapts dynamically during the evolutionary process.
Chromosome Length	$n$	9	The number of binary features defining the security system configuration in a single individual.
Minimum Quality Threshold	$Q_{\min }$	3	The minimum sum of active components in the solution that ensures an acceptable level of protection.
Security Assessment Variability	${\delta }_R$	[0.9, 1.1]	The range of the random variation coefficient in the security rating.
Throughput Assessment Variability	${\delta }_T$	[1.2, 1.8]	The range of variation for the random fluctuation coefficient of throughput capacity.
Certification Level Variability	${\delta }_S$	[0.8, 1.2]	The range of random variation in the security system certification.

presents the main parameters and their values used in the simulation process.

Table 1 includes parameters that define the numerical characteristics of the input data for the multi-objective optimization algorithm. The population size $N$ determined the number of individuals in each generation, while the number of generations $G$ specified the maximum number of iterations during which the NSGA-II algorithm performed selection, mutation, and crossover. The crossover and mutation probabilities $P_c$ and $P_m$ controlled the processes of recombination and variation of individuals. In this study, an adaptive mutation mechanism was used, where $P_m$ varied depending on the current generation, ensuring intense mutation at the early stages and its reduction at the later stages of evolution. The chromosome length $\left(n\right)$ determined the number of features in each individual, corresponding to the number of possible security system components included in the solution configuration to ensure the information security policy. The minimum quality threshold $Q_{\min }$ was introduced as a constraint on the number of active security mechanisms in each solution.

The variability range of random fluctuation coefficients (${\delta }_R$,${\delta }_T$,${\delta }_S$) defines the possible deviation range when calculating the values of the objective functions (1)–(4) (security, throughput, and certification). These parameters are used to model the uncertainty and stochastic nature of the evaluations of solution characteristics. The use of the aforementioned parameters allowed for the reproduction of a realistic optimization process for the configuration of NIS and provided an objective assessment of the effectiveness of the proposed method.

To conduct the computational experiments correctly, it is necessary to define the initial data representing the characteristics of possible NIS configurations.

Table 2. Initial data for multi-criteria selection of NIS.

№	Security Tool (Example)	Type	$\mathbf{Security} \mathbf{Rating}, \mathit{R}\left({\mathit{F}}_{\mathit{i}}\right)$	$\mathbf{Throughput} \mathit{T}\left({\mathit{F}}_{\mathit{i}}\right)$, Gbit/s	$\mathbf{Cost} \mathbf{Thousand} \mathbf{USD}, \mathit{C}\left({\mathit{F}}_{\mathit{i}}\right)$	$\mathbf{Certification}, \mathit{S}\left({\mathit{F}}_{\mathit{i}}\right)$
1	Firewall Cisco ASA	Firewall (FW)	8.5	10	150	1
2	Palo Alto PA-3220	FW	9.0	12	180	1
3	Fortinet FortiGate 100F	FW	8.2	9	140	1
4	Snort IDS/IPS	Intrusion Detection System (IDS)	7.5	5	80	0
	Suricata IDS/IPS	IDS	7.8	6	90	0
6	OpenVPN	Virtual Private Network (VPN)	6.5	8	50	1
7	WireGuard	VPN	7.0	10	60	0
8	Squid Proxy	Proxy server	6.0	7	40	1
9	Zscaler Cloud Proxy	Proxy server	8.0	15	200	1

presents examples of possible security tools with their corresponding parameters, which are used in the modeling.

Each protection mechanism $F_i$ in the proposed model is represented as an abstract computational automaton. The automaton thus maps the incoming network packet stream from its initial state to a secured state. The parameters reported in Table 2—including the security rating and processing delay—serve as integral performance characteristics of the considered automata. Consequently, network security configuration selection can be formulated as the problem of determining an optimal sequence of automata that minimizes total delay (i.e., maximizes throughput) under a specified trust level.

It is worth noting that the numerical values of the security-mechanism parameters employed in the experiments and presented in Table 2 are illustrative in nature. They were chosen to illustrate the applicability of the proposed model. For practical deployment in a real-world infrastructure, the illustrative parameters should be substituted with real performance measurements and expert-based risk evaluations corresponding to the organization’s network topology.

A notable limitation of this study is the use of a synthetic dataset to represent the parameters of the security mechanisms listed in Table 2. The numerical values employed in this study are intended exclusively for validating the convergence behavior of the NSGA-II algorithm and demonstrating the functionality of the proposed mathematical model. In practical deployment scenarios, the performance of the model is dependent on the accuracy and reliability of the input data. Thus, for real-world deployment, the parameter vectors $R(F_i)$ should be derived from empirical throughput measurements in the target network topology and from relevant vulnerability assessment metrics, including CVSS scores and penetration testing results obtained during security audits. Nevertheless, the stochastic nature of the employed data does not compromise the validity of the optimization mechanism, which consistently demonstrated the ability to identify non-dominated solutions independently of the magnitude of the input parameters.

The analysis of the presented visualizations (see Figure 2, Figure 3, Figure 4 and Figure 5) confirms the correctness of the NSGA-II algorithm’s operation. This is evidenced by the formation of a set of Pareto-optimal solutions, the uniform distribution of points along the front, the steady increase in hypervolume, and the preservation of population divergence throughout the evolutionary process. These results validate the effectiveness of the proposed method in solving the multi-criteria selection problem for security system configuration.

6. Discussion

Figure 2 illustrates the set of Pareto-optimal solutions obtained during the execution of the NSGA-II algorithm for the multi-objective security system configuration selection problem. The solutions on the left side of the graph in Figure 2 achieve the lowest cost while offering only a baseline level of network security, adequate for non-critical network segments. The solutions on the right side of Figure 2 illustrate the increasing cost associated with each incremental improvement in the security rating. This allows a security administrator (CISO) to justify the selection of a configuration, such as trading a 10% reduction in network throughput for a 30% improvement in security, while staying within the security budget. Accordingly, for the decision-maker, the derived Pareto front provides a basis for substantiating investments in network security. For instance, an analysis of the curve’s “knee” can reveal points where a modest budget increase results in the greatest gain in security. The boundary points of the Pareto front delineate the applicability limits of the system, namely the highest attainable level of security given the existing infrastructure and the minimally acceptable configuration necessary to ensure network operability.

From a practical standpoint, the analysis of the resulting Pareto front allows decision-makers to distinguish three clusters of configurations that are critical for cybersecurity strategy. The main cluster represents the knee point of the Pareto curve, where further marginal gains in security level result in disproportionately high increases in cost. Configurations positioned prior to this point represent the most economically efficient solutions for small- and medium-sized enterprises operating under budget constraints. The second important region represents the high compliance zone, where solutions meet stringent regulatory requirements, including mandatory certifications, despite potential reductions in network throughput. Finally, the extreme solutions correspond to configurations that maximize throughput, which are critical for DMZ segments and media server environments. In these scenarios, latency is unacceptable. The proposed model converts an abstract optimization problem into a practical decision-support tool for cybersecurity budgeting. Consequently, the CISO can provide a quantitative justification to the board of directors for increasing the cybersecurity budget, for instance by 20%, showing that this shift enables movement from a basic protection level to resilience against advanced persistent threats (APTs). In this process, the CISO bases their justification on the quantitative metrics provided by the model.

Figure 3 depicts the dynamics of the hypervolume change in the Pareto set across generations. The hypervolume, a quality metric for the obtained solutions, represents the volume of the space dominated by the Pareto-optimal configurations relative to a reference point. The increase in hypervolume with the number of generations, as shown in Figure 2, reflects a gradual improvement in solution quality and an expansion of the Pareto set. In the initial iterations, the hypervolume grows rapidly due to the active evolutionary phase of the population. However, as the algorithm progresses, the rate of growth slows, indicating convergence toward an optimal solution set. Furthermore, in practical use, the resulting Pareto front serves as a decision-support map for the CISO. For instance, the Pareto front in Figure 3 demonstrates that a budget increase of just 15% allows the integration of certified intrusion detection system (IDS) modules. This enables compliance with ISO 27001 standards without causing substantial reductions in network throughput. Thus, the modeling results justify cybersecurity investment decisions on the basis of mathematically derived trade-offs rather than subjective intuition.

Figure 4 illustrates the evolution of the average values of the objective functions (derived from model (1)–(7)) across generations. The graph tracks the average security rating, throughput, and certification level for all individuals in the population as a function of the iteration number. The observed trends confirm the proper functioning of the NSGA-II algorithm, as the average values of the criteria improve consistently throughout the optimization process. In the early stages, significant changes occur due to the high variability of solutions. As the process advances, the values stabilize, signaling convergence to a stationary distribution in the solution space.

Figure 5 presents the evolution of the solution set’s divergence during the algorithm’s execution. The divergence metric, calculated as the average Euclidean distance between individuals in the population within the objective function space (1)–(3), provides insight into the preservation of population diversity. A gradual decrease in divergence in later iterations indicates the formation of a stable Pareto front. This gradual reduction confirms the effectiveness of the mechanism employed to maintain solution diversity, which is critical for the success of evolutionary algorithms.

In summary, the visualizations presented in Figure 2, Figure 3, Figure 4 and Figure 5 demonstrate the correct operation of the NSGA-II algorithm. The results confirm the algorithm’s ability to generate a set of Pareto-optimal solutions, ensure uniform coverage of the Pareto front, improve solution quality as measured by hypervolume, and maintain population diversity throughout the optimization process. These findings validate the feasibility of the proposed approach for solving the multi-criteria selection problem for security system configurations.

For a more comprehensive evaluation of the algorithm’s characteristics and its applicability in diverse scenarios, future research will focus on analyzing the influence of evolutionary parameters on the structure of the solution set during the selection of network information security configurations. Additionally, the proposed approach will be compared with alternative multi-criteria optimization methods to further assess its effectiveness.

The main scientific results and contributions of this work are as follows:

A formalization of network security mechanisms is proposed in the form of abstract computational automata that transform network traffic and generate auxiliary security events, which enables the unification of heterogeneous security mechanisms within a unified modeling framework.

A multi-objective mathematical model for selecting NSC is developed, in which information security policy requirements are represented as constraints and optimization criteria.

A method for encoding the security system configuration as a binary decision vector is proposed, which ensures the scalability of the model when the set of available security mechanisms is expanded.

The NSGA-II algorithm is adapted to solve the configuration selection problem with hard constraints, including budgetary and operational requirements.

A computational verification of the model is conducted, including an analysis of hypervolume variation, convergence, and solution diversity, thereby confirming the applicability of the proposed method for decision support in information security problems.

7. Conclusions

A model for selecting a network security configuration has been developed, in which protection mechanisms are represented as graph nodes with attributes derived from their functions as computational automata. This approach enables the representation of non-additive system properties, including network throughput.

A modified NSGA-II algorithm with binary encoding and a constrained dominance mechanism has been implemented, which makes it possible to exclude the generation of economically or technically infeasible protection-mechanism configurations.

The effectiveness of the proposed method has been experimentally validated, demonstrating that the algorithm successfully identifies a set of Pareto-optimal solutions and provides the decision-maker with flexibility in balancing cost, performance, and network security. The analysis showed that adaptive mutation contributes to improved coverage of the solution space when using a small population size.