Three-Dimensional and Multiple Image Encryption Algorithm Using a Fractional-Order Chaotic System

Abstract

The rapid development of communication in the last decade has heightened the necessity to create a secure platform for transferring data, including images, more than in previous years. One of the methods of secure image transmission is the encryption method. In this work, an encryption algorithm for multiple images is introduced. In the first step of the proposed algorithm, a key generation algorithm based on a chaotic system and wavelet transform is introduced, and in the next step, the encryption algorithm is developed by introducing rearrange and shift functions based on a chaotic system. One of the most important tools used in the proposed algorithm is the hybrid chaotic system, which is obtained by fractional derivatives and the Cat map. Different types of tests used to study the behavior of this system demonstrate the efficiency of the proposed hybrid system. In the last step of the proposed method, various statistical and security tests, including histogram analysis, correlation coefficient analysis, data loss and noise attack simulations, have been performed on the proposed algorithm. The results show that the proposed algorithm performs well in secure transmission.

1. Introduction

Images play a crucial role in maintaining security, especially considering that images can contain personal information. In previous decades, the tools for taking pictures were limited to analog cameras, and keeping private images was a relatively easy task, but in recent years, with the advancement of digital imaging technology, it is rare to find a person without a mobile phone capable of taking images. Analog images are stored on photographic film, and copying requires special equipment and can be hidden in a physical environment such as a safe. But digital images have a different nature and are stored on digital devices or can be copied by pressing a key. Digital images are also widely transmitted over the Internet. Due to the differences in their structures, these types of images present different security challenges. In this paper, we focus on the security of digital images. In some cases, it is necessary to create a secure environment for a sequence of images. For example, a medical image is taken from various parts of the body and creates a sequence of images or a three-dimensional image represents different layers of the image. An algorithm that encrypts these images collectively is more efficient in terms of both security and speed compared to an algorithm that encrypts each image individually. Various articles in the last few years tried to develop the multiple image encryption algorithm using means of various tools. In [1], blurring pixels with SHA-512 are employed to produce a random key, and then by using a new one-dimensional chaotic systems, the encryption algorithm is introduced. In [2], domino encryption strategy is used in the encryption algorithm. Radon transform and ghost imaging are introduced in [3] for the proposed algorithm. Also, quaternion discrete fractional Tchebyshev moment transform with cross-coupling operation is proposed for the encryption algorithm in [4]. However, since security can be enhanced with advancements in science, researchers continually develop different algorithms. In recent years, neural networks have been of interest to researchers in the field of security, and this tool has been used to create a secure image environment. For example, in [5], the fuzzy method and neural network have been introduced to create a secure environment for transmitting medical images. Another crucial tool employed in encryption algorithms is the utilization of chaotic systems. The primary reason for this lies in the ability of these systems to generate a sequence of numbers with a random structure, which can be reconstructed given the initial information. Although chaotic systems are a useful tool in image encryption, classical maps such as the logistic and sine maps have limitations, including a restricted chaotic range and non-uniform distribution. To address these issues, researchers have conducted various studies using different methods and tools. In [6], the chaotic system based on a fractional seed chaos generator and fractal theory is introduced; a 2D cascaded logistic map is studied in [7], a new n-dimensional conservative chaos based on a generalized Hamiltonian system and a conservative hyperchaotic system is studied in [8,9,10]. In the present work, the chaos system is improved based on fractional calculations and a combination with the Cat map, and then an encryption algorithm based on this chaos system is presented. The general structure of this paper is as follows: In Section 2, a 4D chaotic system is introduced using the Atangana–Baleanu fractional derivative [11]. Subsequently, this system is combined with the Cat map, and a new 2D chaotic system is introduced. The purpose of this process is to create complexity in the generation of random numbers in order to reduce the possibility of a reproduction of numbers by attackers to a minimum. In Section 3, the encryption algorithm based on the proposed chaps system for the multiple images is presented. This algorithm includes two parts: key generation and encryption. The key generation algorithm utilizes wavelet transform to reduce calculation time. Also, the algorithm is designed in such a way that some parts of this algorithm can be done in parallel to increase the speed. Since a 3D image can be converted into a multiple images by separating its layers, the proposed algorithm can be easily applied to this type of images. Following the presentation of the proposed algorithm, in Section 4, the chaotic behavior of the proposed systems is investigated by various types of tests such as statistical tests. Additionally, the security of the encryption algorithm is evaluated against different types of attacks using a range of tests.

2. Fractional-Based Hybrid Chaotic System

The concept of expanding based on Atangana–Baleanu fractional derivatives for a hybrid system was developed in [12]. In the remainder of this paper, following the presentation of preliminary definitions, this concept is generalized to introduce a chaotic system for a four-dimensional system. Subsequently, a hybrid system is created by utilizing this system and the Cat map, with the aim of generating sequences with a chaotic structure. Readers can find more information about the definitions mentioned in the next section in references related to fractional calculus, such as [13,14].

2.1. 4D Fractional Chaotic System

Definition 1. 

The left Atangana–Baleanu fractional derivative for $f\left(x\right)\in H^1(a,b)$, where $b>a$ of order $\alpha \in (0,1)$ by Caputo view, is defined as

$$\begin{array}{c}\hfill {}_a{}^{ABC}D_t^{\alpha }f\left(t\right)={\displaystyle \frac{\beta \left(\alpha \right)}{1-\alpha }}{\int }_a^t{\displaystyle \frac{df\left(s\right)}{ds}}{\epsilon }_{\alpha ,1}\left(-{\displaystyle \frac{\alpha }{1-\alpha }}{(t-s)}^{\alpha }\right)ds,t\in (a,b),\end{array}$$

(1)

where ε denotes Mittag–Leffler function that defined by using the Euler’s gamma function $(\Gamma )$ as

$$\begin{array}{c}\hfill {\epsilon }_{a,b}\left(x\right):=\sum _{i=0}^{\infty }{\displaystyle \frac{x^i}{\Gamma (ai+b)}},x\in \mathbb{C}.\end{array}$$

Also, in this definition, $\beta \left(\alpha \right)$ is consider as a normalization function and obtained by

$$\begin{array}{c}\hfill \beta \left(\alpha \right)=1-\alpha +{\displaystyle \frac{\alpha }{\Gamma \left(\alpha \right)}},\beta \left(0\right)=\beta \left(1\right)=1.\end{array}$$

Considering the information stated in Definition 1, the left Atangana–Baleanu fractional derivative in Riemann–Liouville sense can be stated similarly as below

$$\begin{array}{c}\hfill {}_a{}^{ABR}D_t^{\alpha }f\left(t\right)={\displaystyle \frac{\beta \left(\alpha \right)}{1-\alpha }}{\displaystyle \frac{d}{ds}}{\int }_a^{t}f\left(s\right){\epsilon }_{\alpha ,1}\left(-{\displaystyle \frac{\alpha }{1-\alpha }}{(t-s)}^{\alpha }\right)ds,t\in (a,b).\end{array}$$

(2)

Definition 2. 

By the conditions stated in Definition 1, the left Atangana–Baleanu fractional integral is defined as

$$\begin{array}{c}\hfill {}_a{}^{AB}I_t^{\alpha }f\left(t\right)={\displaystyle \frac{1-\alpha }{\beta \left(\alpha \right)}}f\left(t\right)+{\displaystyle \frac{\alpha }{\beta \left(\alpha \right)}}{}_a{}^{RL}I_t^{\alpha }f\left(t\right),t\in (a,b),\end{array}$$

(3)

where ${}_a{}^{RL}I_t^{\alpha }f\left(t\right)$ is used for the Riemann–Liouville fractional integral and defined as

$$\begin{array}{c}\hfill {}_a{}^{RL}I_t^{\alpha }f\left(t\right)={\displaystyle \frac{1}{\Gamma \left(\alpha \right)}}{\int }_a^{t}f\left(s\right){(t-s)}^{\alpha -1}ds.\end{array}$$

By using the above two definitions, the relation between Atangana–Baleanu fractional derivatives (1) and (2) and integral (3) can be expressed as [15]

$$\begin{array}{cc}\hfill & {}_a{}^{ABR}D_t^{\alpha }\left({}_a{}^{AB}I_t^{\alpha }f\left(t\right)\right)={}_a{}^{AB}I_t^{\alpha }\left({}_a{}^{ABR}D_t^{\alpha }f\left(t\right)\right)=f\left(t\right),\hfill \\ \hfill & {}_a{}^{AB}I_t^{\alpha }\left({}_a{}^{ABC}D_t^{\alpha }f\left(t\right)\right)=f\left(t\right)-f\left(a\right).\hfill \end{array}$$

(4)

Now, we continue to introduce a chaotic system based on the discussed topics. Consider the following system of Atangana–Baleanu fractional differential equations with the initial conditions $x\left(0\right)=x_0$, $y\left(0\right)=y_0$, $z\left(0\right)=z_0$ and $k\left(0\right)=k_0$.

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{}_a{}^{ABR}D_t^{\alpha }x\left(t\right)={\Phi }_1\left(x\left(t\right),y\left(t\right),z\left(t\right),k\left(t\right)\right),\hfill \\ {}_a{}^{ABR}D_t^{\alpha }y\left(t\right)={\Phi }_2\left(x\left(t\right),y\left(t\right),z\left(t\right),k\left(t\right)\right),\hfill \\ {}_a{}^{ABR}D_t^{\alpha }z\left(t\right)={\Phi }_3\left(x\left(t\right),y\left(t\right),z\left(t\right),k\left(t\right)\right),\hfill \\ {}_a{}^{ABR}D_t^{\alpha }k\left(t\right)={\Phi }_4\left(x\left(t\right),y\left(t\right),z\left(t\right),k\left(t\right)\right).\hfill \end{array}\right.\end{array}$$

(5)

For the simplicity, system (5) is written as ${}_a{}^{ARB}D_t^{\alpha }X\left(t\right)={\rm Y}\left(X\left(t\right)\right)$, where ${\rm Y}:=\left({\Phi }_1,{\Phi }_2,{\Phi }_3,{\Phi }_4\right)$ and $X\left(t\right):=\left(x\left(t\right),y\left(t\right),z\left(t\right),k\left(t\right)\right)$. In this system, the functions ${\Phi }_i(i=1,\dots ,4)$ are selected as the same maps that are introduced in [16] as follows

$$\begin{array}{c}\hfill {\Phi }_1^r(x,y,z,k):=\left\{\begin{array}{c}{\alpha }_1^x{f}_1^x\circ F_1^x(r,x)+g_1^x(r,x,y,z,k)\hfill \\ +h_1^x\left({\displaystyle \frac{({\beta }_1^x-r)z}{2}}\right)mod1,whenk<0.5,\hfill \\ \\ {\alpha }_2^x{f}_2^x\circ F_2^x(r,x)+g_2^x(r,x,y,z,k)\hfill \\ +h_2^x\left({\displaystyle \frac{({\beta }_2^x-r)(1-z)}{2}}\right)mod1,whenk\ge 0.5,\hfill \end{array}\right.\end{array}$$

(6)

$$\begin{array}{c}\hfill {\Phi }_2^r(x,y,z,k):=\left\{\begin{array}{c}{\alpha }_1^y{f}_1^y\circ F_1^y(r,y)+g_1^y(r,x,y,z,k)\hfill \\ +h_1^y\left({\displaystyle \frac{({\beta }_1^y-r){\zeta }_1}{2}}\right)mod1,when{\zeta }_1<0.5,\hfill \\ \\ {\alpha }_2^y{f}_2^y\circ F_2^y(r,y)+g_2^y(r,x,y,z,k)\hfill \\ +h_2^y\left({\displaystyle \frac{({\beta }_2^y-r)(1-{\zeta }_1)}{2}}\right)mod1,when{\zeta }_1\ge 0.5,\hfill \end{array}\right.\end{array}$$

(7)

$$\begin{array}{c}\hfill {\Phi }_3^r(x,y,z,k):=\left\{\begin{array}{c}{\alpha }_1^z{f}_1^z\circ F_1^z(r,z)+g_1^z(r,x,y,z,k)\hfill \\ +h_1^z\left({\displaystyle \frac{({\beta }_1^z-r){\zeta }_2}{2}}\right)mod1,when{\zeta }_2<0.5,\hfill \\ \\ {\alpha }_2^z{f}_2^z\circ F_2^z(r,z)+g_2^z(r,x,y,z,k)\hfill \\ +h_2^z\left({\displaystyle \frac{({\beta }_2^z-r)(1-{\zeta }_2)}{2}}\right)mod1,when{\zeta }_2\ge 0.5,\hfill \end{array}\right.\end{array}$$

(8)

$$\begin{array}{c}\hfill {\Phi }_4^r(x,y,z,k):=\left\{\begin{array}{c}{\alpha }_1^k{f}_1^k\circ F_1^k(r,k)+g_1^k(r,x,y,z,k)\hfill \\ +h_1^k\left({\displaystyle \frac{({\beta }_1^k-r){\zeta }_3}{2}}\right)mod1,when{\zeta }_3<0.5,\hfill \\ \\ {\alpha }_2^k{f}_2^k\circ F_2^k(r,k)+g_2^k(r,x,y,z,k)\hfill \\ +h_2^k\left({\displaystyle \frac{({\beta }_2^k-r)(1-{\zeta }_3)}{2}}\right)mod1,when{\zeta }_3\ge 0.5,\hfill \end{array}\right.\end{array}$$

(9)

where parameter $r\in (0,4]$, ${\zeta }_i(i=1,2,3)$ is chosen as $x,y,z$ and $F_{\varsigma }^{\tau }$ for $\tau =x,y,z,k,$$\varsigma =1,2$ are selected as Sin or Logistic maps. ${\alpha }_{\varsigma }^{\tau },{\beta }_{\varsigma }^{\tau }$ for $\tau =x,y,z,k,\varsigma =1,2$ are constant numbers in $\mathbb{R}$, and $g_{\varsigma }^{\tau },h_{\varsigma }^{\tau }$ are considered as sufficiently smooth functions. By taking $[0,+\infty )={\cup }_{i=0}^{+\infty }{I}_i$, where $I_i:=[ih,(i+1)h)$, $h>0$, and using the method described in [17], the discrete solution of this system can be found as follows. In the first step, let ${}_a{}^{ABR}D_t^{\alpha }X\left(t\right)={\rm Y}\left(X\left(h\lfloor {\displaystyle \frac{t}{h}}\rfloor \right)\right)$, then for $t\in I_n$, we obtain

$$\begin{array}{c}\hfill {}_a{}^{ABR}D_t^{\alpha }X\left(t\right)={\rm Y}\left(X_n\left(nh\right)\right).\end{array}$$

(10)

Assuming that $t=(n+1)h$ and using (4) and (10), the final solution is found as follows:

$$\begin{array}{c}\hfill X_{n+1}=\left({\displaystyle \frac{1-\alpha }{\beta \left(\alpha \right)}}+{\displaystyle \frac{h^{\alpha }}{\beta \left(\alpha \right)\Gamma \left(\alpha \right)}}\right){\rm Y}\left(X_n\right),\end{array}$$

The behavior of this chaotic system is evaluated in Section 4.2. In the following, the output of this system for input values $\alpha$ and r is shown by ${\Psi }_r^{\alpha }$.

2.2. Hybrid Chaotic System

In the following, a hybrid chaotic system is introduced using the fractional chaotic system from the previous subsection and the Cat map. The most important reason for producing this hybrid system is to create more complexity on the sequence produced in the previous section. This method increases the complexity of the proposed image encryption algorithm. In the proposed system, the Cat map is used. This map has been studied in various papers, and various methods have been provided to calculate this map. For example, see [18,19,20]. The multidimensional generalization Cat map can be written as

$$\begin{array}{c}\hfill X_{n+1}=C^{nD}{X}_{n}mod1,\end{array}$$

(11)

where in (11)

$$C^{nD}=\left[\begin{array}{cc}I& P_a\\ P_b& I+P_b{P}_a\end{array}\right].$$

In this matrix, $P_a\in {\mathbb{R}}^{m\times (n-m)},P_b\in {\mathbb{R}}^{(n-m)\times m}$ and $I\in {\mathbb{R}}^{m\times m}$ are the identity matrices. By simple calculations, it can be seen that $det\left(C^{nD}\right)=1$. In the proposed hybrid chaos, $C^{2D}$ is used as follows.

$$C^{2D}=\left[\begin{array}{cc}1& 1\\ 1& 2\end{array}\right].$$

The sequence production steps are presented in the following steps.

• Step 1. Consider $X_0,r,n,h,C^{2D},i=0$ and $\alpha$ as input values.
• Step 2. Define $X_i^j:=X_i,j=1$ and run the following algorithm.

1: repeat 2:     $Y_j:=C^{2D}{X}_i^j$; 3:     $Z_j:={\Psi }_r^{\frac{\alpha }{5}}\left({\Psi }_r^{\alpha }\left([X_i^j,Y_j]\right)\right)$; 4:     $X_i^j\left(1\right)=mod\left(Z_j\left(1\right)+Z_j\left(2\right),1\right)$; 5:     $X_i^j\left(2\right)=mod\left(Z_j\left(3\right)+Z_j\left(4\right),1\right)$; 6:     $j=j+1$; 7: until  $j\le 2$

• Step 3. Consider $i=i+1,X_i=X_i^2$.
• Step 4. If $i=n$, stop, orelse go to Step 2.

The effect of this algorithm on the fractional-based chaotic system and the analysis of the behavior of the hybrid system are evaluated in the next subsection. Additionally, a sequence of the proposed hybrid system with n elements is shown by

$$[X_0\cdots X_n]={\Omega }_{h,n}^{\alpha ,r}\left(X_0\right).$$

The structure of the proposed hybrid system is shown in Figure 1.

Figure 1. Flowchart of the proposed hybrid chaotic system.

3. 3D and Multiple Image Encryption Algorithm

To create a secure environment for image transfer, we need two components: one involving the key, and the other involving the transformation of the original data into unrecognizable data. In the discussion of data security, a secure path exists for the transmission of cryptographic keys. These keys are transferred between the sender and receiver of the message through this path. However, it is important to note that this path has a limited transfer volume and, as a result, cannot be utilized for transferring original data. In the following, first, an algorithm for generating the key space is introduced. The generated key space should not have a large volume to ensure easy transferability. Additionally, it should also be sensitive to small changes in the input image.

3.1. The Key of Encryption Algorithm

The keys utilized for encryption algorithms are categorized as symmetric and asymmetric. In the proposed algorithm, the symmetric key is used for encryption and decryption. The key used can be specified as a number of our own choice, and this fixed number can be used for encryption. However, one of the most secure methods is to choose the keys in a way that depend on the input data. This allows the key to be changed with small variations in the input data. Due to the fact that a large number of images are used in the proposed algorithm, the time taken to obtain the key should not be too long. Also, the keys should be produced as quickly as possible and should be sensitive to small changes. The flowchart of the key generation algorithm is illustrated in Figure 2. In the key space generation algorithm, some parts are designed in such a way that these parts can be calculated in parallel. If there is a parallel calculation system, the calculation time can be reduced. These specific parts are highlighted in the flowchart. The key generation algorithm is written in the following steps. Consider a sequence of images as ${\left\{I_i\right\}}_{i=1}^k$, where $I_i\in {\mathbb{R}}^{n_i\times m_i}$, and the input values $\alpha$, r, and h for the chaotic system.

Figure 2. Flowchart of the proposed key generation algorithm.

• Step 1. In this step, in order to reduce the volume of calculations, we reduce the size of the image used in the calculations by wavelet transformation. For $i=1,\dots ,k$, derive $I_i^{ll}\in {\mathbb{R}}^{n_i^{\prime }\times m_i^{\prime }}$ as the low–low pass filter of the Daubechies wavelet transform at level 5 (db5) for $I_i$.
• Step 2. Obtain the sequence ${\left\{({\nu }_i^x,{\nu }_i^y)\right\}}_{i=1}^k$ by using the following formulae.

  $$\begin{array}{cc}\hfill & {\nu }_i^x:={\displaystyle \frac{4}{256n_i^{\prime }{m}_i^{\prime }}}\sum _{j=1}^{\lfloor \frac{n_i^{\prime }}{2}\rfloor }\sum _{l=1}^{\lfloor \frac{m_i^{\prime }}{2}\rfloor }{I}_i^{ll}(j,l),\hfill \\ \hfill & {\nu }_i^y:={\displaystyle \frac{4}{256n_i^{\prime }{m}_i^{\prime }}}\sum _{\begin{array}{c}j=\lfloor \frac{n_i^{\prime }}{2}\rfloor \\ +1\end{array}}^{n_i^{\prime }}\sum _{\begin{array}{c}l=\lfloor \frac{m_i^{\prime }}{2}\rfloor \\ +1\end{array}}^{m_i^{\prime }}{I}_i^{ll}(j,l).\hfill \end{array}$$

Changes in the pixels of the input image cause the values of these parameters to vary, which affects other parts of the proposed algorithm and leads to changes in the final encrypted image.

• Step 3. Define ${\omega }^x:={\sum }_{i=1}^k{\nu }_i^x$ and ${\omega }^y:={\sum }_{i=1}^k{\nu }_i^y$.
• Step 4. Calculate the $c^a$ and $c^b\in {\mathbb{R}}^{1\times 7}$ using the following algorithm.

1: $c^a=0,c^b=0$; 2: for  $j=1:k$  do 3:     $a_{j,1}^x:={\nu }_j^x$, $a_{j,1}^y:={\nu }_j^y$; 4:     if $j=1$ then 5:         $b_{j,1}^x:={\omega }^x$, $b_{j,1}^y:={\omega }^y$; 6:     else 7:         $b_{j,1}^x:=a_{j-1,7}^x$, $b_{j,1}^y:=a_{j-1,7}^y$; 8:     end if 9:       $[a_{j,1:7}^x;b_{j,1:7}^x]:={\Omega }_{h,7}^{\alpha ,r}(a_{j,1}^x,b_{j,1}^x)$; 10:     $[a_{j,1:7}^y;b_{j,1:7}^y]:={\Omega }_{h,7}^{\alpha ,r}(a_{j,1}^y,b_{j,1}^y)$; 11:     $c^a={\displaystyle \frac{1}{3}}\left(c^a+a_{j,1:7}^x+b_{j,1:7}^y\right)$; 12:     $c^b={\displaystyle \frac{1}{3}}\left(c^b+a_{j,1:7}^y+b_{j,1:7}^x\right)$; 13: end for

Step 5. 

By using the bitxor operator (⊕), obtain $\{x_1,x_2,y_1,y_2\}$ as

$$\begin{array}{cc}\hfill & [x_1,x_2]:={\displaystyle \frac{1}{2\times {10}^{15}}}\left[\underset{i=2}{\overset{4}{⨁}}\lfloor c_i^a\times {10}^{15}\rfloor ,\underset{i=5}{\overset{7}{⨁}}\lfloor c_i^a\times {10}^{15}\rfloor \right],\hfill \\ \hfill & [y_1,y_2]:={\displaystyle \frac{1}{2\times {10}^{15}}}\left[\underset{i=2}{\overset{4}{⨁}}\lfloor c_i^b\times {10}^{15}\rfloor ,\underset{i=5}{\overset{7}{⨁}}\lfloor c_i^b\times {10}^{15}\rfloor \right].\hfill \end{array}$$

After performing the above steps, the final key space for the encryption algorithm is considered as follows

$$\begin{array}{c}\hfill \mathrm{Key}=\{\alpha ,r,h,x_1,x_2,y_1,y_2\}.\end{array}$$

(12)

3.2. Proposed Shift and Rearrange

Rearranging pixels of images at the appropriate time is a critical step in image encryption algorithms. In this subsection, we introduce two straightforward and efficient methods, one based on shifting image pixels and the other on swapping image pixels. For the first method, we begin by creating a matrix using a permutation that contains exactly one non-zero element (equal to one) in each row and column (Algorithm 1). Multiplying this matrix with any other matrix leads to shifting the elements of that matrix. In  Algorithm 2, the pixels of each layer in the multilayer matrix undergo shifts using distinct permutations. It is noteworthy that in this context, the layer corresponding to each pixel remains unchanged. To illustrate the operation of this function, refer to Figure 3 and Figure 4b.

Algorithm 1 PermutationMatrix

Input: A permutation $P\in {\mathbb{N}}^{1\times n}$ Output: A permutation matrix $M\in {\{0,1\}}^{n\times n}$ 1: $M=zerose\left(n\right)$ 2: for  $i=1,\dots ,n$  do 3:     $M(i,P(i\left)\right)=1$ 4: end for 5: return the matrix M

Figure 3. Illustration of Algorithm 2 execution.

Figure 4. Execution of Shift and Rearrange on an image: (a) 3D plain image, (b) execution of Shift on the plain image, (c) execution of Rearrange3D on the plain image, (d) execution of Shift and Rearrange, consecutively.

Algorithm 2 Shift

Input: Matrices $A_1,\dots ,A_l\in {\mathbb{R}}^{n\times n}$ and permutations $P_1,\dots ,P_l\in {\mathbb{N}}^{1\times n}$ Output: Matrices $B_1,\dots B_l\in {\mathbb{R}}^{n\times n}$ 1: for  $i=1,\dots ,l$  do 2:     $M_l\leftarrow PermutationMatrix\left(P_l\right)$ 3:     $B_l=M_l{A}_l{M}_l$ 4: end for 5: return Matrices $B_1,\dots B_l$

In the second method, pixel locations are altered based on permutations. As the first step, Algorithm 3 is defined. Then compared to the first method, this approach induces more disturbance in the neighborhood of pixels (Algorithm 4). To achieve a more favorable outcome, Algorithm 5 is introduced by incorporating the application of Algorithm 3 to combine the layers of a multilayer matrix. This involves not only changing the rows and columns within each layer but also modifying the arrangement of the layers themselves. To illustrate the operation of this function, refer to Figure 4c and Figure 5. Figure 4d shows the optimal performance of Shift and Rearrange3D when applied consecutively. Note that the second step in Figure 5 is based on the following permutation matrices.

$$P_1=\left[\begin{array}{ccc}8& 11& 1\\ 6& 9& 2\\ 7& 10& 12\\ 5& 3& 4\end{array}\right],P_2=\left[\begin{array}{ccc}8& 9& 2\\ 6& 10& 11\\ 5& 1& 3\\ 12& 7& 4\end{array}\right].$$

Algorithm 3 RowCol

Input: Natural umbers $n,m$ Output: Natural numbers $i,j\in [1,n]$ 1: $q=floor(m/n)$ 2: $r=mod(m,n)$ 3: if  $r=0$  then 4:     $(i,j)=(q,n)$ 5: else 6:     $(i,j)=(q+1,r)$ 7: end if 8: return Natural numbers $i,j$

Figure 5. Illustration of Function Rearrange3D execution.

Algorithm 4 Rearrange

Input: A matrix $A\in {\mathbb{R}}^{n\times m}$ and a permutation $P\in {\mathbb{N}}^{n\times m}$ Output: A matrix $A_P\in {\mathbb{R}}^{n\times m}$ 1: for  $i=1,\dots ,n$  do 2:     for $j=1,\dots ,m$ do 3:         $(v,w)\leftarrow RowCol\left(m,P(i,j)\right)$ 4:         $A_p(i,j)=A(v,w)$ 5:     end for 6: end for 7: return Matrix $A_P$

Algorithm 5 Rearrange3D

Input: A multilayer matrix $A\in {\mathbb{R}}^{n\times m\times l}$ and a permutation $P\in {\mathbb{N}}^{n\times l\times m}$ Output: A matrix $A_P\in {\mathbb{R}}^{n\times m\times l}$ 1: for  $k=1,\dots ,m$  do 2:     $B(:,:,k)\leftarrow Rerrange\left(A(:,k,:),P(:,k,:)\right)$ 3: end for 4: for  $k=1,\dots ,l$  do 5:     $A_p(:,:,k)=B(:,k,:)$ 6: end for 7: return Matrix $A_P$

It is worth noting that both defined operators are invertible, and their inverses are denoted by adding “Inv” to the beginning of their titles.

3.3. Proposed Encryption Algorithm

In this subsection, an encryption algorithm for multiple images is presented using the concepts introduced in the previous sections. Before presenting the encryption algorithm, the following notation is introduced. Let

$$\begin{array}{c}\hfill \left[\begin{array}{c}\begin{array}{ccc}{x}_1& \cdots & x_n\end{array}\\ \begin{array}{ccc}{y}_1& \cdots & y_n\end{array}\end{array}\right]={\Omega }_{h,n}^{\alpha ,r}\left([x_0;y_0]\right),\end{array}$$

then ${\overline{\Omega }}_{h,n}^{\alpha ,r}\left([x_0;y_0]\right)$ is defined as

$${\overline{\Omega }}_{h,n}^{\alpha ,r}\left([x_0;y_0]\right):=[x_1\dots x_n{y}_1\dots y_n].$$

Now, the encryption algorithm is being introduced. Consider ${\left\{I_i\right\}}_{i=1}^k$ as a sequence of n-by-m input images. In the initial phase of the algorithm, these images enter the key generation algorithm, and the key space is considered as (12). Subsequent to this step, the remaining steps of the algorithm are delineated as follows:

• Step 1. By using the input keys and hybrid chaotic system, obtain the vector $v^1\in {\mathbb{R}}^{1\times 2\vartheta }$ with $\vartheta :=\lfloor {\displaystyle \frac{nm}{4}}\rfloor$ as follows:

  $$v^1={\overline{\Omega }}_{h,\vartheta }^{\alpha ,r}\left([x_1;y_1]\right),$$

After performing these calculations, reshape the vector $v^1$ into an n-by-m matrix, denoted as $V^1$.

• Step 2. Obtain the sequence ${\left\{I_i^1\right\}}_{i=1}^k$ as

  $$I_i^1=I_i\oplus V_i^c,$$

  where

  $$\begin{array}{c}\hfill V_i^c:=\left\{\begin{array}{cc}{V}^1\hfill & i=1,\hfill \\ \mathrm{circshift}\left(V^1,\left[v_i^1{v}_{\vartheta +i}^1\right]\right)\hfill & i=2,\dots ,k.\hfill \end{array}\right.\end{array}$$

In the above formulas, “circshift” represents circular shifts.

• Step 3.
• (a) By using input keys $x_2$ and $y_2$ obtain $v^2\in {\mathbb{R}}^{1\times 2\varrho }$ with $\varrho =\lfloor {\displaystyle \frac{kn}{2}}\rfloor$ as

  $$v^2={\overline{\Omega }}_{h,\varrho }^{\alpha ,r}\left([x_2;y_2]\right).$$
• (b) Calculate $P^1\in {\mathbb{N}}^{k\times n}$ as follows:

  $$\begin{array}{c}\hfill P^1(i,:):=\left\{\begin{array}{cc}{\mathrm{Index}}_{\mathrm{sort}}\left(v^2\left(1,({\displaystyle \frac{i}{2}}-1)n+1:{\displaystyle \frac{i}{2}}n\right)\right),\hfill & \mathrm{when}i\mathrm{is}\mathrm{even},\hfill \\ {\mathrm{Index}}_{\mathrm{sort}}\left(v^2\left(1,({\displaystyle \frac{i+1}{2}}-1)n+1:{\displaystyle \frac{i+1}{2}}n\right)\right),\hfill & \mathrm{when}i\mathrm{is}\mathrm{odd}.\hfill \end{array}\right.\end{array}$$

In the above formula, ${\mathrm{Index}}_{\mathrm{sort}}$ represents a permutation of indices in $\mathbb{N}$, where the elements in this vector indicate the positions of the elements after sorting them from the smallest to the largest values.

• (c) Consider ${\left\{P^1(i,:)\right\}}_{i=1}^k$ and ${\left\{I_i^1\right\}}_{i=1}^k$ as input values for shift Algorithm 2 and obtain ${\left\{I_i^2\right\}}_{i=1}^k$ as output of this function.

Step 4. 

• (a) Obtain $v^3\in {\mathbb{R}}^{1\times 2\varrho }$ as

  $$v^3={\overline{\Omega }}_{h,\varrho }^{\alpha ,r}\left([x_1;y_2]\right).$$
• (b) Define $P^2\in {\mathbb{N}}^{1\times kn}$ as

  $$P^2={\mathrm{Index}}_{\mathrm{sort}}\left(v^3\right).$$

  and reshape the $P^2$ into an n-by-k matrix, denoted as $V^2$.
• (c) Define $M^v\in {\mathbb{N}}^{m\times n\times k}$ as

  $$\begin{array}{c}\hfill M^v(i,:,:):=\left\{\begin{array}{cc}{V}^2,\hfill & i=1,\hfill \\ \mathrm{circshift}\left(M^v(i-1,:,:),\left[\lfloor v_i^3\times {10}^2\rfloor \lfloor v_{i+1}^3\times {10}^2\rfloor \right]\right),\hfill & i=2,\dots ,m.\hfill \end{array}\right.\end{array}$$
• (d) Define a multilayer matrix $A(:,:,i):=I_i^2,i=1,\dots ,k$, and then consider $M^v$ and A as input values for Rearrange3D Algorithm 5 and obtain final encrypted images ${\left\{I_i^e\right\}}_{i=1}^k$.

A flowchart illustrating the proposed encryption algorithm is presented in Figure 6. The decryption algorithm is precisely the inverse of the encryption algorithm; therefore, the details of the decryption algorithm are not studied. When considering color images, each color space is selected as a separate input image for the algorithm. If the input image is three-dimensional, it is initially divided into two-dimensional images corresponding to the slices of the image. Each of these images is then treated as an input for the algorithm. This process is implemented using the Image3DSlices command in Wolfram Mathematica software 9.0.1.0

Figure 6. Flowchart of the proposed encryption algorithm.

4. Simulation Results and Security Analysis

In this section, the security of the algorithm presented in the previous section is investigated by means of different types of security tests.

4.1. Simulation Infrastructure

As mentioned in the previous sections, the algorithm is designed to operate on multiple images. The number of images it can process depends on the available computational resources. On systems with higher processing power and more memory, the proposed algorithm is capable of handling a larger number of images simultaneously. To generate numerical results, a Windows-10-based system equipped with an Intel(R) Core(TM) i3-5005U CPU running at 2.00 GHz and 4.00 GB RAM is employed. The simulations are conducted in the MATLAB R2014b and Wolfram Mathematica 9 programming environments. Additionally, the images utilized in this section were sourced from USC-SIPI (available at http://sipi.usc.edu/database/(accessed on 18 April 2025)), and the information-hiding homepage (available at https://www.petitcolas.net/steganography/index.html (accessed on 18 April 2025)), and  internal 3D images were sourced from Wolfram Mathematica software.

4.2. Analysis of the Proposed Systems

There are various tests to investigate the behavior of a chaotic system. In this paper, the Lyapunov exponent, bifurcation diagram, histogram analysis and sensitivity to the initial point are used to investigate the behavior of the chaotic system. The proposed 4D hybrid chaotic system with the following input values is used in the relevant calculations in this paper.

• $\{{\alpha }_1^x,{\alpha }_2^x,{\alpha }_1^y,{\alpha }_2^y,{\alpha }_1^z,{\alpha }_2^z,{\alpha }_1^k,{\alpha }_2^k\}=\{30,20,20,20,10,20,10,20\}$.
• $\{{\beta }_1^x,{\beta }_2^x,{\beta }_1^y,{\beta }_2^y,{\beta }_1^z,{\beta }_2^z,{\beta }_1^k,{\beta }_2^k\}=\{6,9,4,20,4,$$7,4,7\}$.
• $\{{\zeta }_1,{\zeta }_2,{\zeta }_3\}=\{x,y,z\}$.
• $f_1^x\left(p\right)=cosh\left(p\right)$, $f_2^x\left(p\right)=f_1^y\left(p\right)=f_1^z\left(p\right)=f_2^k\left(p\right)=p$, $f_2^y\left(p\right)=cos\left(p\right),f_2^z\left(p\right)=f_1^k\left(p\right)=sin\left(p\right)$.
• $g_1^x=2sin\left(rx\right)+2cosh(20rz+k)$, $g_2^x=-3sinh\left(\pi rx\right)+cos\left(xyz\right)$, $g_1^y=2cos(rx+y)-exp(y+z^2)+k$, $g_2^y=z^2+sin\left(k\right)+2sinh\left(\pi ry\right)$, $g_1^z=2exp(rz+y+z+k)$, $g_2^z=2ry+exp\left(2rxyk\right)$, $g_1^k=2exp(rz+y+z)+cos\left(rk\right)$, $g_2^k=sin(2rx+y)$.
• $h_1^x\left(p\right)=-2p,h_2^x\left(p\right)=h_2^y\left(p\right)=4p,h_1^y\left(p\right)=h_1^z\left(p\right)=h_1^k\left(p\right)=2p$, $h_2^z\left(p\right)=cosh\left(4p\right)$, $h_2^k\left(p\right)=exp\left(4p\right)$.

The Lyapunov exponent for a chaotic system as F is obtained by

$$\lambda =\underset{n\to \infty }{lim}{\displaystyle \frac{1}{n}}\sum _{i=0}^{n-1}ln\left|F^{\prime }\left(x_i\right)\right|.$$

For this value, three possible states occur: greater than zero, equal to zero or less than zero. By utilizing each of these states, the chaotic behavior of the system can be analyzed. If the Lyapunov exponent is greater than zero, then the system is chaotic. If the Lyapunov exponent is less than zero, then the orbit is periodic. When this value is equal to zero, a bifurcation occurs [21]. The graph of this value for two systems is given in Figure 7. According to the results in Figure 7a,b, it can be seen for $\alpha =0.7$ that some of the Lyapunov exponents of the 4D fractional chaotic system are negative. After using the proposed hybrid chaotic system on the 4D fractional chaotic system with $\alpha =0.7$ in Figure 7b, it can be seen that all the Lyapunov exponent values are positive. Therefore, the hybrid chaotic system improves the chaotic region of the 4D fractional chaotic system. Also, according to the results, it can be seen that the value of Lyapunov exponent for hybrid system is greater than this value for the 4D fractional chaotic system.

Figure 7. Lyapunov exponent results with $\alpha =0.7$ for (a) 4D fractional chaotic system and (b) hybrid chaotic system.

Another test that is used to study the behavior of the proposed systems is to draw the Bifurcation diagram. To draw this diagram, the first step is to divide the domain of r into several points. Then, for each of these numbers, the sequence produced by the chaotic system after a great repetition is plotted along the vertical line. In the event of convergence, the output sequence will approach a fixed point for specific points. Conversely, in the case of divergence, the output sequence will spread out. Bifurcation diagrams for the hybrid chaotic system with $\alpha =0.3$ are shown in Figure 8. The results of this diagram show the divergence of the sequences produced by the chaotic system. Another diagram used to investigate the chaotic behavior of a system is the cobweb plot. The output results of this plot for the proposed system are given in Figure 9. This figure shows that the outputs of the hybrid chaotic system do not approach a fixed point and represent chaotic orbits for the proposed systems.

Figure 8. Bifurcation diagram with $\alpha =0.3$ for (a,b) hybrid chaotic system.

Figure 9. Cobweb plots with $\alpha =0.3$, $r=0.2$ (red) and $r=0.8$ (blue) for (a,b) hybrid chaotic system.

A paramount feature that a chaotic system should have is the sensitivity to the initial point. To study this feature, the outputs of the proposed systems with $\alpha =0.3$ and $\alpha =0.7$, $r=0.7$ and different values of the initial point with a small difference, i.e., $X_0^1=(0.1,0.4)$ and $X_0^2=(0.1,0.4+{10}^{-15})$ are drawn in Figure 10. According to the results, it can be seen that although the change in the initial point is small, the outputs of the system are different. Consequently, minor changes in the input would result in modifications to the production sequence.

Figure 10. Two outputs of the proposed hybrid chaotic system with (a) $\alpha =0.3,r=0.7$, (b) $\alpha =0.7,$ $r=0.7$, and $X_0^1=(0.1,0.4)$ (red, green), $X_0^2=(0.1,0.4+{10}^{-15})$ (blue, black).

The nonuniform distribution can be considered as one of the reasons for the weakness of the encryption algorithm in dealing with statistical attacks. Since the distribution of the chaotic system affects the distribution of the encrypted image, the uniform distribution of the chaotic system used in encryption can be effective in enhancing the security of the encryption algorithm. The histogram of hybrid chaotic system is shown in Figure 11. By using this figure, we can see that the histogram plot of output values is flat.

Figure 11. Histogram of hybrid chaotic system with $\alpha =0.7$ and $r=0.5$: (a) $x_n$, (b) $y_n$.

The last test that is studied to check the behavior of the proposed systems is the $0-1$ test. The reader can find more information about this test in reference [22,23]. In this test for the time series as $T_n$, the following systems is obtained

$$\begin{array}{c}\hfill p_{n+1}=p_n+T_{n+1}cos\left(cn\right),\\ \hfill q_{n+1}=q_n+T_{n+1}sin\left(cn\right),\end{array}$$

where $c\in (0,2\pi )$ and considered as fixed number. Then, the following growth rate is defined

$$\begin{array}{c}\hfill K=\underset{n\to \infty }{lim}{\displaystyle \frac{log{M}_n}{logn}},\end{array}$$

where $M_n={lim}_{N\to \infty }{\sum }_{j=1}^N{(p_{j+n}-p_j)}^2+{(q_{j+n}-q_j)}^2$. Expected ideal values of K for a chaotic system are one and zero for a non-chaotic system. The plot of c versus K is drawing in Figure 12. The results of this output show that the calculated values of K are close to ideal value one. Also, in this test, the plot of p versus q is drawn in Figure 13. According to these results, irregularity and diffusion can be observed in the output sequences.

Figure 12. Plot of c versus K with $\alpha =0.3$ and $r=0.75$ for (a) 4D fractional chaotic system and (b) hybrid chaotic system.

Figure 13. Plot of p versus q with $\alpha =0.3$ and $r=0.75$ for (a–d) 4D fractional chaotic system and (e,f) hybrid chaotic system.

4.3. Key Space Analysis

In this subsection, the analysis of the key generation algorithm is discussed. The key space consists of two components: a set of fixed keys and a set of keys dependent on input images. This dual nature of key selection arises from practical limitations in securely transferring keys. To overcome this challenge, the sender can initiate the process by transmitting a subset of fixed keys. Subsequently, the variable keys are sent in correspondence with a sequence of different images. As an initial study, the sensitivity of the variable key space to minor changes in the input data is studied. For this purpose, the fixed key space equal to $\{\alpha ,r,h\}=\{0.2,1.1,0.001\}$ is considered. Also, four sequences of images are selected as

• ${\mathrm{I}}^1=\{{\mathrm{I}}_1^1=$ Aerial 5.1.10(256 × 256), ${\mathrm{I}}_2^1=$ Airplane 5.1.11(256 × 256), ${\mathrm{I}}_3^1=$ Chemical plant 5.1.14(256 × 256), ${\mathrm{I}}_4^1=$ Clock 5.1.12(256 × 256)}.
• ${\mathrm{I}}^2=\{{\mathrm{I}}_1^2=$ Female 4.1.02(256 × 256 × 3), ${\mathrm{I}}_2^2=$ Female 4.1.03(256 × 256 × 3), ${\mathrm{I}}_3^2=$ Jelly beans 4.1.08(256 × 256 × 3), ${\mathrm{I}}_4^2=$ House 4.1.05(256 × 256 × 3)}.
• ${\mathrm{I}}^3=\{{\mathrm{I}}_1^3=$ Airplane 5.1.11(256 × 256), ${\mathrm{I}}_2^3=$ Clock 5.1.12(256 × 256), ${\mathrm{I}}_3^3=$ Jelly Jellybeans 4.1.08(256 × 256 × 3), ${\mathrm{I}}_4^3=$ House 4.1.05(256 × 256 × 3)}.
• ${\mathrm{I}}^4=\{{\mathrm{I}}_1^4=$ Watch(512 × 512), ${\mathrm{I}}_2^4=$ Lena(512 × 512), ${\mathrm{I}}_3^4=$ Peppers(512 × 512), ${\mathrm{I}}_4^4=$ Fishing Boat(512 × 512)}.

Also, the Engine image is chosen as a 3D image. The applied changes are as follows:

Case i. 

Without changing the input data.

Case ii. 

The following changes have been applied to images ${\mathrm{I}}^1,{\mathrm{I}}^2,{\mathrm{I}}^3$ and ${\mathrm{I}}^4$: ${\mathrm{I}}_1^1(20,20)={\mathrm{I}}_1^1(20,20)+1,{\mathrm{I}}_1^2(20,20,1)={\mathrm{I}}_1^2(20,20,1)+1,{\mathrm{I}}_1^3(20,20)={\mathrm{I}}_1^3(20,20)+1$ and ${\mathrm{I}}_1^4(20,20)={\mathrm{I}}_1^4(20,20)+1$.

Case iii. 

$\alpha$ has been changed to $0.2+{10}^{-15}$.

Case iv. 

The value of r has been updated to $1.1+{10}^{-15}$.

Case v. 

The new value assigned to h is $0.001+{10}^{-15}$.

The results related to the input changes of the key generation algorithm mentioned above are given in Table 1. Based on this table, it is evident that the key generation algorithm is sensitive to very small changes, and any change in the input of the algorithm creates a new key space.

Table 1. The output of the key generation algorithm for different inputs.

Images	Key	Case i	Case ii	Case iii	Case iv	Case v
${\mathrm{I}}^1$	$x_1$	0.3699	0.3607	0.0053	0.2009	0.1936
	$y_1$	0.3302	0.3437	0.2468	0.2260	0.3965
	$x_2$	0.3310	0.1241	0.5127	0.3521	0.2145
	$y_2$	0.3041	0.3462	0.4062	0.1765	0.1814
${\mathrm{I}}^2$	$x_1$	0.4680	0.1866	0.1718	0.1439	0.1860
	$y_1$	0.3314	0.3966	0.1107	0.3376	0.3431
	$x_2$	0.3057	0.4664	0.2168	0.1822	0.3739
	$y_2$	0.2933	0.1508	0.4643	0.1746	0.3616
${\mathrm{I}}^3$	$x_1$	0.2307	0.1978	0.4691	0.4069	0.2901
	$y_1$	0.1547	0.1771	0.1116	0.3248	0.3889
	$x_2$	0.4242	0.2972	0.3734	0.3846	0.2588
	$y_2$	0.4377	0.5214	0.2964	0.3770	0.3176
${\mathrm{I}}^4$	$x_1$	0.2633	0.2028	0.2784	0.2385	0.3297
	$y_1$	0.0873	0.3831	0.3515	0.1748	0.0777
	$x_2$	0.1848	0.3284	0.5258	0.2342	0.2752
	$y_2$	0.2583	0.3127	0.4522	0.2607	0.2202
Engine	$x_1$	0.5381	0.3147	0.3814	0.2705	0.0153
	$y_1$	0.2818	0.5031	0.3812	0.2064	0.1799
	$x_2$	0.2737	0.1674	0.2143	0.0793	0.2559
	$y_2$	0.1766	0.4007	0.0623	0.0133	0.5334

The next aspect to analyze is the size of the key space. The size of the keys used should not be too small to be easily detected by a key recovery attack, such as the brute force attack, nor should it be too large that it cannot be transferred in a secure environment. In relation to the subjects discussed in source [24], it is noted that the algorithm can withstand the brute force attack if the key size is a minimum of $2^{100}$. Assuming that the calculation accuracy in the program execution environment has an accuracy of ${10}^{-15}$, the size of the key space will be equal to ${10}^{105}\approx 2^{348}$. Therefore, the size of the key space is large enough to resist this attack. In Table 2, the size of the key space of the proposed algorithm is compared with the size of the key space of other papers. It should be noted that this table is given only for comparison, and the magnitude of the key size does not necessarily indicate the superiority of the algorithm. As mentioned earlier, a key value greater than $2^{100}$ enhances the algorithm’s resistance against attacks. However, if the secure transmission path is constrained, the size of the key space may become problematic.

Table 2. Key space size for different algorithms.

Algorithm	Proposed	Ref. [25]	Ref. [26]	Ref. [27]	Ref. [28]
Key size	${10}^{105}\approx 2^{348}$	$1.936\times {10}^{59}$	${10}^{60}$	$4.34\times {10}^{96}$	${10}^{75}$
Algorithm	Ref. [29]	Ref. [30]	Ref. [31]	Ref. [32]	Ref. [33]
Key size	${10}^{56}$	$5.841\times {10}^{135}$	$1.2446\times 2^{327}$	$1.157\times {10}^{203}$	${10}^{56}$

As the last study in this subsection, the effect of a small change in the key space in the encryption and decryption algorithms is investigated. For this purpose, the sequence of images ${\mathrm{I}}^1$ and the following keys are considered.

$$\begin{array}{cc}\hfill & K_1:=\{\alpha ,r,h,x_1,x_2,y_1,y_2\}=\{0.2,1.1,0.001,0.3699,0.3310,0.3302,0.3041\},\hfill \\ \hfill & K_2:=\{\alpha ,r,h,x_1,x_2,y_1,y_2\}=\{0.2,1.1,0.001,0.3699+{10}^{-15},0.3310,0.3302,0.3041\}.\hfill \end{array}$$

It can be seen that these two key spaces exhibit very little difference. In the first case, the sequence of images ${\mathrm{I}}^1$ is encrypted by the key $K_1$, and then the sequence of decryption images is checked by the key spaces $K_1$ and $K_2$. The results of this process are illustrated in Figure 14. It can be seen that the original images cannot be recovered using a key space with only minor differences from the original key. The sequence of images encrypted by these two keys creates different encrypted images even though they are slightly different. In order to show the difference between these two sequences, pixel-by-pixel subtraction and its histogram are shown in Figure 15, and these results show their difference. Consequently, without comprehensive knowledge of the key space, image recovery is not feasible.

Figure 14. (a–d) Decrypted images by $K_1$ and (e–h) decrypted images by $K_2$.

Figure 15. (a–f) Encryption images by $K_1$, (e–h) encryption images by $K_2$, (i–l) pixel-by-pixel subtraction of the encryption images and (m–p) histogram of the pixel-by-pixel subtraction of the encryption images.

4.4. Analysis of Program Execution Time

In addition to ensuring secure data transmission, an encryption algorithm must also be efficient in terms of its execution time to be suitable for simulation and widespread use. To evaluate the execution time of the proposed algorithm, the execution times for some of the image sequences are listed in Table 3. According to the reported times, it can be concluded that the calculation time for the proposed algorithm is efficient.

Table 3. Program execution time (in seconds).

Images	Key Generation	Encryption	Decryption
${\mathrm{I}}^1$	0.2084	3.7334	3.8196
${\mathrm{I}}^2$	0.2677	6.6504	6.4169
${\mathrm{I}}^3$	0.2648	5.1215	5.0604
${\mathrm{I}}^4$	0.3203	14.1428	14.1293

4.5. Histogram and Correlation Coefficients Analysis

In unencrypted images, in most cases, the numerical values of the pixels are not uniformly distributed, and the number of the values of pixels varies according to the type of image. These differences contribute to the image’s meaningfulness. For example, in an image taken from the night sky, black color will be more prevalent than other colors. Additionally, in a meaningful, unencrypted image, the relationship between adjacent pixels is often significant. The relationship between pixels is analyzed using the statistical concept of correlation coefficients. Therefore, if an image is not well encrypted, the attacker may continue to decrypt it based on the numerical values of some pixels or the relationship between pixels. In practice, some attacks are designed to exploit this weak point, such as statistical attacks. In this subsection, the histogram and correlation coefficient are studied. The histogram plots for all images in sequences ${\mathrm{I}}^2$ and the sequence of encrypted images of these sequences are drawn in Figure 16. The results of this figure show that the encrypted image, unlike the original image, has a uniform distribution. In the calculations, the Pearson correlation coefficient is used to measure correlation. The value of this test for an encrypted image should be close to zero. A value near zero indicates that the pixels are uncorrelated, which is desirable for encryption. Conversely, a value close to one suggests a strong correlation between pixels, implying insufficient encryption To check the correlation coefficients, the sequence of images ${\mathrm{I}}^4$ is selected. Also, to avoid the need for calculating the correlation coefficients for each image, the images are placed side by side, and this value is computed. In Figure 17, the correlation distributions for different directions are illustrated. Additionally, the correlation coefficients are obtained and compared with other algorithms in Table 4. For another example, the sequence of images ${\mathrm{I}}^5=${Barbara$(512\times 512)$, Airplane$(512\times 512)$, Fishing boat$(512\times 512)$, Watch$(512\times 512)$} is considered, and then, using the fixed key $\{\alpha =0.2,r=1.1,h=0.001\}$, the sequence of encrypted images is obtained. In Table 5, after obtaining the sequence of encrypted images, the correlation coefficients for each image are calculated and compared by their average value. From these results, it can be observed that the values are close to zero, indicating no discernible relationship between the pixel intensities of adjacent pixels in various directions.

Figure 16. Histogram plot of the sequence of the images: (a) ${\mathrm{I}}^2$ and (b) encrypted ${\mathrm{I}}^2$.

Figure 17. (a) Input images, (b) encrypted images and correlation of neighborhood pixels at different directions (c–e) before encryption and (f–h) after encryption.

Table 4. Comparison of correlation coefficients for ${\mathrm{I}}^4$.

	Input images	Proposed	Ref. [34]
Horizontal	$0.9684$	$0.0001$	$-0.0003$
Vertical	$0.9783$	$0.0010$	$0.0011$
Diagonal	$0.9558$	$0.0003$	$0.0013$
	Ref. [35]	Ref. [36]	Ref. [37]
Horizontal	$-0.0036$	$0.0538$	$0.0034$
Vertical	$0.0016$	$0.0389$	$0.0015$
Diagonal	$0.0058$	$0.0307$	$0.0008$

Table 5. Comparison of correlation coefficients for ${\mathrm{I}}^5$.

	Input images	Proposed	Ref. [38]
Horizontal	$0.9272$	$-0.0001$	$0.0058$
Vertical	$0.9599$	$0.0014$	$0.0048$
Diagonal	$0.9070$	$0.0001$	$-0.0006$
	Ref. [39]	Ref. [40]	Ref. [41]
Horizontal	$0.0012$	$0.0256$	$-0.0010$
Vertical	$0.0018$	$0.0116$	$-0.0009$
Diagonal	$0.0014$	$0.0180$	$0.0006$

4.6. NPCR, UACI and Information Entropy Analysis

NPCR (Number of Pixel Change Rate), UACI (Unified Average Changing Intensity) and information entropy are commonly used metrics for performance evaluation and analysis of image encryption algorithms. The tests NPCR and UACI, as their names suggest, calculate the amount of pixel changes. Since a suitable encryption algorithm should be able to detect most of the changes for small changes in the input images, these tests are used to calculate the sensitivity of the changes in the algorithm. For this purpose, the following formulas are used. Suppose the image is encrypted by the algorithm for the first time and denoted as $d_1$, then a small change is made to the image, which is subsequently encrypted by the algorithm and denoted as $d_2$. NPCR and UACI are defined as

$$\begin{array}{ccc}\hfill & \mathrm{NPCR}={\displaystyle \frac{{\sum }_{i,j}{\delta }_{i,j}}{n\times m}}\times \%100,\hfill & \hfill \mathrm{UACI}={\displaystyle \frac{{\sum }_{i,j}|d_1-d_2|}{255\times n\times m}}\times \%100,\end{array}$$

where $n,m$ are the size of input image, and ${\delta }_{i,j}$ is defined as

$$\begin{array}{c}\hfill {\delta }_{i,j}:=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{d}_1(i,j)\ne d_2(i,j),\hfill \\ 0,\hfill & \mathrm{if}{d}_1(i,j)=d_2(i,j).\hfill \end{array}\right.\end{array}$$

The values $100\%$ and $33.\overline{33}$ are considered as ideal values for NPCR and UACI, respectively. Considering that in practice it is not possible for an algorithm to achieve the ideal value, a critical range for these values has been introduced in [42]. This range is checked in Table 6 for various sequences of images. Also, the results of the proposed algorithm are compared with those of another algorithm in Table 7. As the last result for this test, the average value of NPCR and UACI values for the sequence images ${\mathrm{I}}^5$ have been compared with other encryption algorithms in Table 8. By the results of these tests and comparing it with the ideal values, it can be seen that the reported values are close to the ideal values.

Table 6. Numerical results of UACI and NPCR for various sequences of images.

		UACI Critical Values [42]				NPCR Critical Values [42]
		${\mathbf{u}}_{\mathbf{0.05}}^{\ast -}$  = 33.2824	${\mathbf{u}}_{\mathbf{0.01}}^{\ast -}$  = 33.2255	${\mathbf{u}}_{\mathbf{0.001}}^{\ast -}$  = 33.1594		${\mathbf{N}}_{\mathbf{0.05}}^{\ast }=$	${\mathbf{N}}_{\mathbf{0.01}}^{\ast }=$	${\mathbf{N}}_{\mathbf{0.001}}^{\ast +}=$
Image	UACI	${\mathbf{u}}_{\mathbf{0.05}}^{\ast +}$  = 33.6447	${\mathbf{u}}_{\mathbf{0.01}}^{\ast +}$  = 33.7016	${\mathbf{u}}_{\mathbf{0.001}}^{\ast +}$  = 33.7677	NPCR	99.5693	99.5527	99.5341
${\mathrm{I}}^1$	33.3277	Pass	Pass	Pass	99.6063	Pass	Pass	Pass
${\mathrm{I}}^2$	33.4920	Pass	Pass	Pass	99.6190	Pass	Pass	Pass
${\mathrm{I}}^3$	33.4685	Pass	Pass	Pass	99.6002	Pass	Pass	Pass
Engine	33.4090	Pass	Pass	Pass	99.6107	Pass	Pass	Pass

Table 7. Comparison of NPCR, UACI and information entropy values.

	Proposed	Ref. [34]	Ref. [37]	Ref. [43]
NPCR	$99.6041$	$99.6060$	$99.1841$	$99.6534$
UACI	$33.4587$	$33.5126$	$33.5284$	$33.6772$
Information entropy	$7.9998$	$7.9998$	$7.9996$	$7.9994$

Table 8. Comparison of NPCR and UACI values for ${\mathrm{I}}^5$.

	Proposed	Ref. [38]	Ref. [39]	Ref. [40]
NPCR	$99.6162$	$99.6075$	$99.6100$	$99.6125$
UACI	$33.4485$	$33.4100$	$33.5450$	$33.4150$

A measure of the randomness or unpredictability of pixel values is studied by the information entropy. If l and p represent the gray level and the probability, and $I_i$ are the possible pixel values, this value is calculated by the formula:

$$H=-\sum _{i=0}^{l-1}p\left(I_i\right)lo{g}_2\left(p\left(I_i\right)\right).$$

The ideal value of information entropy is 8. The numerical results for this test are presented in Table 7 and Table 9. Also, the numerical results for the average value of information entropy of the encrypted sequence images have been compared with the results of other papers in Table 10 for ${\mathrm{I}}^5$. These results show that the value obtained from the proposed method is close to the ideal value.

Table 9. Numerical results of information entropy.

	${\mathbf{I}}^1$	${\mathbf{I}}^2$	${\mathbf{I}}^3$	Engine
Information entropy	$7.9992$	$7.9997$	$7.9996$	$7.9959$

Table 10. Comparison of information entropy for ${\mathrm{I}}^5$.

	Input Images	Proposed	Ref. [38]
Information entropy	$7.1252$	$7.9992$	$7.9983$
	Ref. [39]	Ref. [40]	Ref. [41]
Information entropy	$7.9974$	$7.9420$	$7.9984$

4.7. Data Loss and Noise Attacks

Because the route of encrypted data transmission lacks high security, if an attacker cannot decode the encrypted images, they may attempt to disrupt the receiver from obtaining information by introducing noise or destroying parts of the image. A suitable encryption algorithm should be able to prevent the loss of all the sent information after encryption, despite the loss of part of the information. The results of this test are shown in Figure 18, Figure 19 and Figure 20. According to the results, it can be seen that although part of the data is lost after decoding the images, the reconstructed images remain recognizable and are not entirely lost.

Figure 18. (a–h) Sequence of encrypted images destroyed by data loss attack and (i–l) decrypted images after data loss attack.

Figure 19. (a–h) Sequence of encrypted images destroyed by salt and pepper noise attack with noise density = 0.2 and (i–l) decrypted images after noise attack.

Figure 20. (a) Original 3D image of engine, (b) encrypted image, (c) destroyed image by data loss attack and (d) decrypted images after data loss attack.

5. Conclusions

In this work, we address a critical issue in image transmission: ensuring image transmission security. In the first part, one of the most important tools used in the proposed algorithm, the chaotic system, has been introduced. For this purpose, first the 4D chaotic system is calculated by the Atangana–Baleanu fractional derivative, and then a hybrid chaotic system is introduced by combining this system with the Cat map. The chaotic behavior of these systems has been studied by various types of tests, and the results demonstrate the appropriate behavior of the proposed systems. In the second part, the encryption algorithm is designed for multiple images that can be used for 3D images. In the simulation section, the security of the algorithm is studied through different types of statistics tests, the results of which indicate the efficiency of the proposed algorithm.