Entropy-Based Optimization in Chaotic Image Encryption Algorithms with Implementation of Artificial Intelligence ^†

Abstract

This paper addresses the challenge of determining optimal parameters in chaotic systems used for image encryption algorithms based on chaos theory. A baseline algorithm employing a third-order Lorenz chaotic system is examined, incorporating core procedures such as permutation (shuffling) and diffusion. Graphical results are presented to illustrate the variation of image entropy in relation to changes in system parameters. The analysis reveals a distinct region in the parameter space where entropy reaches its highest values. Based on these observations, an optimality criterion is formulated, defining an objective function that captures the entropy’s sensitivity to two key system parameters, including the bifurcation parameter. A complex objective function is derived, and the optimization problem is solved using a modified version of the Price algorithm enhanced with artificial intelligence techniques. The proposed modification demonstrates superior performance in locating the global extremum of the objective function, resulting in enhanced security of the encrypted image. Numerical and graphical results for various images are provided, along with a comparative analysis between the standard and the modified Price method.

1. Introduction

In an era of constant digital information sharing and with the growing volume of images generated daily, the need for efficient image encoding methods is more relevant than ever. These methods aim to ensure data privacy, provide resilience against various types of attacks, and, at the same time, be easily applicable and highly performant. Image encryption is a field within secure data transmission that finds broad and diverse applications in areas such as: Medicine—where the protection of sensitive information (medical images such as X-rays, MRIs, CT scans) [1,2,3], or remote diagnostics [4,5] is required; Military and Intelligence—encryption of tactical object images [6] and communication between military systems [7,8]; Art, Culture, and Non-Fungible Tokens (NFTs)—digital ownership and authentication [9,10], steganography [11,12]; Mobile and Web Applications—attachments in messages [13,14], cloud-stored images [15,16,17], security in video calls [18,19,20]; Finance and Legal Services—encryption of documents containing embedded images [21], protection of biometric images used in verification processes [22], industrial cameras and sensors [23]; and others.

The quality of encryption is evaluated using various metrics and indicators, such as information entropy, pixel correlation, the Number of Pixel Change Rate (NPCR), and the Unified Average Changing Intensity (UACI), among others [24].

Entropy, in particular, plays a key role as an indicator of predictability and the distribution of pixel values in the encrypted image, as it measures the degree of uncertainty or randomness in a given set of information [25]. High values of information entropy suggest stronger protection and a lower probability of successful attack through analysis.

The task of finding an optimal solution, from the perspective of modern scientific and technological advancements, is present in every problem related to the design, operation, or management of systems and objects. Optimization is a purposeful activity aimed at achieving the best possible result under specific conditions and in a given context. The increasing complexity of modern applications and the demand for high efficiency necessitate the optimization of cryptographic algorithms. The goal of optimization is to improve security, execution speed, energy efficiency, and adaptability to various types of data and devices.

In parallel, artificial intelligence (AI)—particularly techniques such as genetic algorithms [26,27], neural networks [28], machine learning [29], and large language models like GPT [30]—is actively entering the field of optimization. GPT models, in particular, could be employed to generate optimal parameters, to automatically design or fine-tune cryptographic schemes, as well as to analyze security by predicting vulnerabilities or the behavior of potential attackers.

The integration of chaotic systems into the image encryption process is a modern and effective strategy that relies on the unpredictability, sensitivity to initial conditions, and pseudo-randomness inherent in chaos. The literature presents a variety of approaches for incorporating chaos into the encryption process, which can generally be categorized as follows: the use of one-dimensional or multi-dimensional chaotic maps [31,32], chaotic masking [33], and algorithms based on chaotic synchronization [34,35].

This paper proposes the optimization of the image encryption process using a chaos-based algorithm. The optimization task is formulated as the search for a global extremum of the entropy of the encrypted image, represented as an objective function dependent on the parameters of the chaotic system. A modified Price algorithm is proposed, incorporating artificial intelligence to enhance computational efficiency.

2. Materials and Methods

2.1. Lorenz Chaotic System

The Lorenz system is considered the most well-known and one of the earliest mathematically formulated chaotic systems. It is a simplified model of the equations describing thermal convection, proposed by Edward Lorenz in 1963 [36].

The mathematical model of the Lorenz system is:

$$\begin{array}{l}{\dot{x}}_1=-\sigma x_1+\sigma x_2\\ {\dot{x}}_2=-x_1{x}_3+r{x}_1-x_2,\\ {\dot{x}}_3=x_1{x}_2-b{x}_3\end{array}$$

(1)

where $x_1$ represents the fluid velocity, $x_2$ and $x_3$ are the vertical and horizontal temperature differences, respectively, $\sigma$ is the Prandtl number, $b$—is a spatial constant, and $r$ is the Rayleigh number, which is proportional to the applied heat and can be considered the system input [36].

Lorenz discovered that, for a specific combination of system parameters $\sigma =10, b=8/3$ and $r>25$, the system exhibits a chaotic attractor when observed in state space, meaning that chaotic oscillations arise within the system. The characteristic chaotic attractor, given the initial conditions $x_0={\left[1 1 0\right]}^T$ is illustrated in Figure 1.

The parameter r acts as a bifurcation parameter for the Lorenz system, as increasing its value leads to qualitative changes in the system’s behavior.

The goal of bifurcation theory is to determine the existence and stability of different “branches” of solutions, such as fixed points and periodic orbits.

As the bifurcation parameter changes around the equilibrium point, different equilibrium states continuously emerge from each other. These are graphically represented for each of the system’s variables in Figure 2.

The main characteristic of chaotic dynamics is its exceptional sensitivity to initial conditions. When considering two trajectories $x_1\left(t\right)$ and $x_{1*}\left(t\right)$, starting from nearby initial conditions with a deviation $\mathsf{\Delta }$, an exponential increase in the divergence between the trajectories can be observed, such that $\mathsf{\Delta }=x_1\left(t\right)-x_{1*}\left(t\right).$

Figure 3 presents the trajectories of the same time characteristic, with different initial conditions—$x_0={\left[1 1 0\right]}^T$ (in blue) and $x_0={\left[1 1 0.05\right]}^T$ (in red).

Based on the definitions provided so far, it can be concluded that there are certain limitations regarding the chaotic nature of a system. In particular, the determination of appropriate values for both the initial conditions and the bifurcation parameter is crucial. Such tasks, aimed at finding an optimal set of parameters, are the subject of active research in the field of optimization.

2.2. Chaos-Based Encryption Algorithms

Due to their exceptional unpredictability, chaotic systems find wide application in image encryption algorithms, providing a high level of security. Because of their high sensitivity to initial conditions and parameters, where even small deviations lead to drastically different dynamics, chaos-based encryption schemes are also resilient to various cryptographic attacks.

A basic version of an encryption algorithm using a chaotic signal is considered. In nearly all image encryption algorithms based on chaotic systems, the signal generated by the system is used at least in two steps and several levels. Figure 4 shows the block diagram of a basic chaos-based image encryption algorithm.

Initially, the image to be encrypted is introduced and preprocessed. In the preprocessing step, the image is converted to grayscale (if the algorithm will work with grayscale images only) or is divided into three separate images, one for each color channel. Subsequently, the images are transformed from matrix form into one-dimensional arrays and combined, resulting in a numerical sequence of size N × M × 3.

After preprocessing, the steps of shuffling and diffusion are performed. The algorithms vary in terms of which of these actions is executed first. The shuffling process primarily involves working with the indices of the one-dimensional array obtained from the signal generated by the chaotic system. The dimensionality is the same as that of the array derived from the image. Subsequently, this array is sorted or another algorithm is applied to confuse the values of the elements. Using the indices, the pixel values from the image are rearranged, thus creating a new image with shuffled pixels.

For the diffusion process, a so-called chaotic key is generated. This key can be shared, or if not, chaotic synchronization is necessary. The method of obtaining the chaotic key can vary widely, utilizing a range of mathematical principles. Primarily, the pseudo-random nature of the chosen chaotic system is relied upon, combined with functional normalization of the values. The resulting key is then combined with the image through a bitwise XOR operation.

The combination of shuffling and diffusion can be applied multiple times and at several levels, both at the pixel and bit levels. After completing these primary processes, the image is reconstructed at the pixel level by forming three images for the three color channels, which are then combined.

The degree of security in this type of algorithm largely depends on the random nature of the chaotic signal generated, which is dependent on the dynamics of the chaotic system. The dynamics of the chaotic system are sensitive to changes in the parameters (especially the bifurcation parameters) and initial conditions, as mentioned in the previous section. However, the question remains unclear as to whether and which specific dynamics of the system yield greater “randomness” and, consequently, better protection, and whether this holds for the entire range of parameter variations. The answer to these questions constitutes a solution to an optimization problem, the solution to which will be presented in the next section.

3. Entropy-Based Optimization Through the Application of the Price Method and Artificial Intelligence

This section presents the problem related to finding the optimal parameters of the chaotic system, at which the highest entropy of the encrypted image is achieved. An optimization problem is formulated, which is solved through a modification of the Price method.

3.1. Entropy as a Criterion for Optimality

In solving optimization problems, one of the key steps is the determination of the objective function (criterion for optimality) and its analysis. The objective function provides a quantitative measure to evaluate the state of the object under consideration and allows for a comparative analysis between different states.

When discussing image encoding, one of the main metrics that inevitably appears in the statistical analysis is entropy. It is precisely entropy that will be used to form the objective function, through which the optimal value will be sought, expressed as a maximum.

Entropy is a physical quantity that represents a measure of randomness in the image [35]. It can be calculated through:

$$H\left(x\right)=-\sum _{i=0}^{255}p\left(x_i\right){\mathit{log}}_{2}p\left(x_i\right).$$

(2)

where $p\left(x_i\right)$ is the probability with which the i-th grayscale value $x_i$ appears in the image $x$. For an encrypted image, when each grayscale value $x_i$ appears with equal probability, i.e., ${\displaystyle \frac{1}{256}}$, the information entropy reaches its maximum value of 8. Therefore, the ideal approach for image encryption should aim for the information entropy to be close to 8.

For the current algorithm, the goal is not to achieve the best possible entropy value, but to find the optimal one. The chaotic system has three parameters, and entropy is calculated for three potential objective functions, where only two of the parameters are changed, and the third remains constant (the nominal value is selected).

Figure 5 presents three-dimensional graphs of the entropy of a test image when applying the considered encryption algorithm. The entropy is calculated for an image with dimensions 50 × 50 with a step size of $∆x=0.01$. The parameter changes are within the range in which the chaotic system can be solved, and are adjusted according to acceptable computation time, specifically $r=0÷40, \sigma =0÷20 \mathrm{a}\mathrm{n}\mathrm{d} b=0÷20$.

The obtained graphs show that the considered objective function is a complex function with multiple extrema and shapes. Such types of complex functions are difficult to differentiate and identify analytically. To find the extremum, an optimization problem is solved with an unknown objective function, as the computation time for even smaller ∆x is immensely large, and the form of the entropy function cannot be obtained in a reasonable time given the current computational capabilities.

As the working objective function, the one shown in Figure 5a with changes in parameters r and σ is chosen. With this configuration, a plateau is obtained again with multiple extrema, but overall, the entropy has the highest values, and thus the resulting level of security. The region with the highest entropy values is defined by parameters with $r>20$ and $\sigma >5$.

The choice of optimization method mainly depends on the nature of the objective function, the number of controlling parameters, and the type of constraints. The graphs presented in Figure 5 provide sufficient information to select an appropriate algorithm for nonlinear optimization.

3.2. Modified Price’s Method for Finding the Global Extremum

Finding the global maximum, in itself, means that the objective function has multiple extrema, which is a common phenomenon for complex high-order objective functions. To correctly solve the problem of finding the global extremum, all extrema must be identified, and the global one selected. However, in real problems, the number of local extrema can be enormous, making such an approach unrealistic. Therefore, it is necessary to seek a solution where not all extrema of the entire function are examined.

For tasks involving the search for a global extremum of this type of complex objective function, one of the best groups of methods are heuristic optimization methods. Among them, the Price method [37] is considered a light and efficient approach.

Price’s method utilizes elements of cluster analysis. Initially, a set of M evenly distributed points, also called agents, is generated within the feasible space. Each agent computes the value of the objective function. Groups of 3 agents are randomly selected. The arithmetic mean of the objective function values from these agents is calculated, and a new agent is created. If the value provided by the new agent is better than the worst in the current population, the new agent replaces the one with the worst result; otherwise, the new agent is rejected. The algorithm is organized so that the group of agents converges around the current extremum as iterations continue. As the worst result threshold increases, the agents concentrate around the global extremum area. The number of agents in the method is not clearly defined, but for 2 to 3 control parameters, M is chosen to be between 10 and 100.

Figure 6 shows the initial distribution of 100 agents across the objective function.

With the chosen initial agents M = 100 and 1000 iterations, the obtained maximum entropy is 7.9466 for $\sigma =12.72$ and $r=29.43$.

The Price method has several drawbacks, such as the fact that as the number of agents increases, the probability of finding the extremum grows, but the calculations also become too numerous. Additionally, from Figure 6, it is evident that more than 50% of the initial agents are distributed in a zone where even if they reach an extremum, it will be below the selected level.

For this reason, the use of artificial intelligence is proposed to determine the initial coordinates of the agents. The technology used is the GPT model by OpenAI. The implementation is set as a separate subprogram, where the input parameters are the image to be encrypted and an API key for access, and the output is the corresponding parameters of the system.

Working with GPT is semantic, with prompts being sent in a structured and organized JSON format. The parameter values are requested by providing information about the properties of the working image and the chosen chaotic system.

The prompt used to generate the parameters of the chaotic system has the following form in MATLAB (R2023a):

prompt = sprintf([‘Generate initial parameters (sigma, beta, rho)’, ‘for the Lorenz chaotic system based on the following image features:\n’, ’Entropy: %.4f\nMean intensity: %.4f\nContrast: %.4f\n\n’, ’Return only three floating point numbers separated by spaces.’], entropy_value, mean_intensity, contrast).

The output formatting is required with an additional prompt, such as:

‘Return only three floating point numbers (sigma, beta, rho) separated by spaces. Do not include units or explanation.’].

The parameters for sending the request are with the selected model ‘gpt-4’, with a maximum token count of no more than 50 and temperature = 1.9.

Figure 7 shows the distribution of agents when using AI.

By using the GPT model’s knowledge of the chaotic system and image properties, the coordinates of the points are primarily located in the region where the global extremum is most likely to be found. It can be said that the generated agents are “smart” and are concentrated around the extremum, allowing the algorithm to converge to it much faster, without unnecessary wandering, which increases productivity. M = 20 agents were used, again with 1000 iterations, and an extremum was found with a value of 7.9498 for $\sigma =10.97$ and $r=32.62$.

Despite the drastically smaller number of agents, the modification yields better results in solving the optimization problem.

4. Results and Analysis

The results presented in this section pertain to the performance of the encoding algorithm, the investigation of the objective function in relation to image size, and the identification of optimal parameter values for the chaotic system aimed at maximizing the entropy of the encoded image, achieved through a modified Price algorithm.

4.1. Results of the Encoding Algorithm

Similarly to other encryption algorithms, it is necessary to present results for the encoded images using standard security analysis metrics, namely entropy, correlation, and histograms. These results are provided both numerically and graphically.

Figure 8 presents graphical results for encrypting test image Lena with dimensions 256 × 256: (a)—original image; (b)—encrypted image; (c)—decrypted image; (d)—histogram of the original image; (e)—histogram of the encrypted image; (f)—correlation of the original image; (g)—correlation of the encrypted image.

Table 1. Numerical results.

	Entropy	Correlation
Input image	7.3283	0.9864
Encrypted image	7.9973	−0.0037

presents numerical results regarding the information entropy and the correlation of the original and encrypted image.

4.2. Results from the Analysis of the Objective Function with Respect to Image Size

In a MATLAB environment, the entropy of the encoded image is computed while varying the parameters of the chaotic system. This analysis is conducted for all three parameter combinations. The test image used for encoding is the standard “Lena” image with dimensions 50 × 50 (Figure 9 and Figure 10), 256 × 256 (Figure 11), and 343 × 343 (Figure 12), respectively.

The resolution step used in computing the objective function derived from entropy is denoted by Δ. For the 50 × 50 image, values of Δ = 1 and Δ = 0.1 (Figure 10) are applied, whereas for the larger images only Δ = 1 is used. This limitation arises from the significantly increased computational time and inefficiency when using finer steps for high-resolution images, given the hardware available at the time of the study. Figure 9, Figure 10, Figure 11 and Figure 12 illustrate three-dimensional plots of the objective function for images of different sizes.

4.3. Results from the Application of the Price Algorithm with AI for Determining the Optimal Parameters of the Chaotic System

The optimization of the proposed image encryption algorithm is based on the objective function presented in the previous section, which reflects the image entropy in relation to variations in the chaotic system parameters σ and r, while parameter b remains at its nominal value. The presented results aim to identify the parameter values that yield the highest entropy, without claiming that the algorithm provides an exceptionally high level of security. This type of optimization, implemented in this manner, can be applied to any image encryption algorithm based on chaotic systems.

Table 2. Additional numerical results.

	Entropy of the Original Image	With Nominal Parameters of the Chaotic System	With Standard Price Optimization Method	With AI-Enhanced Price Optimization Method
	7.3283	7.9975	7.9978 $\sigma$ = 11.5682 r = 28.8351	7.9979 $\sigma$ = 12.5817 r = 31.1784
	7.6968	7.9971	7.9978 $\sigma$ = 11.3439 r = 29.1481	7.9981 $\sigma$ = 11.8787 r = 29.1722
	7.7211	7.9976	7.9978 $\sigma$ = 10.0268 r = 28.8655	7.9979 $\sigma$ = 11.4699 r = 31.1835
	7.0839	7.9972	7.9976 $\sigma$ = 10.6537 r = 28.4883	7.9979 $\sigma$ = 11.9907 r = 28.7076
	7.4868	7.9961	7.9972 $\sigma$ = 11.8724 r = 30.9686	7.9974 $\sigma =$ 12.0712 r = 31.0978

presents numerical results for the entropy of the encoded image using nominal parameter values of the chaotic system, as well as values obtained using the standard Price algorithm and its AI-enhanced modification.

The test images were selected from pre-selected databases of test images and have a resolution of 256 × 256 pixels. Both color and grayscale images were chosen in order to ensure the comprehensiveness of the study.

4.4. Analysis

The presented results demonstrate that algorithms based on chaotic systems provide highly effective encryption performance, which can be further improved through appropriate selection of system parameters. The examined baseline version of such an algorithm confirms this, as the achieved results for the test image Lena include an entropy value of approximately 7.997, a correlation coefficient of −0.0037, and a normally distributed histogram. It is important to note that these numerical results were obtained using an algorithm in which the generation of the encryption key does not involve additional mathematical transformations, and both the permutation and diffusion processes are applied only once.

The graphical results related to the objective function clearly indicate that the entropy of the encoded image is not uniform across the entire range of parameter variations. This behavior is observed for both small and large images, with the effect being more pronounced in smaller images. Due to the extremely high computational time required to generate the objective function, results with a calculation step of 0.1 are presented only for the 50 × 50 image. Computing the objective function with such a fine step for images sized 256 × 256 or larger would take several hours with the currently available hardware. This makes the application of a fixed-step scanning method practically infeasible. For this reason, the proposed heuristic approach is employed.

The final set of results demonstrates that the encryption process can indeed be improved by formulating and solving an optimization problem. The maximization of entropy was carried out using both the standard Price algorithm and an AI-enhanced modification of the Price algorithm. The modified method achieved a better extremum value than the standard one, with the difference ranging from approximately 0.001 to 0.0002—a significant improvement in the context of image encryption. The superior results are primarily attributed to the assumption that the AI has prior knowledge of the chaotic system and its bifurcation parameter. As a result, the initial coordinates of the agents are better selected. The identified extrema are located within a so-called plateau of the objective function and can be considered global, although the Price algorithm does not guarantee the discovery of the absolute global optimum.

5. Conclusions

The present study addresses the optimization of image encryption processes using algorithms based on chaotic systems. A basic version of such an algorithm is examined, including its key components and operational procedures. A detailed analysis of the Lorenz chaotic system is presented, highlighting the role of parameter variation in generating different dynamic behaviors and associated temporal characteristics. Based on entropy as an optimality criterion, an objective function is defined. The resulting objective function is highly multi-extremal, likely exhibiting discontinuities and transcendental characteristics. To handle such complex functions, a heuristic approach based on the Price algorithm is proposed, further enhanced with artificial intelligence for the selection of initial agent coordinates. This modification enables faster convergence to an extremum of the objective function, while reducing the number of required initial agents. The graphical and numerical results presented in this work demonstrate the advantages of the proposed modification and confirm the potential for improving the security level of the encryption process through optimal parameter selection of the chaotic system.