A Jigsaw Puzzle SolverBased Attack on Image Encryption Using Vision Transformer for PrivacyPreserving DNNs
Abstract
:1. Introduction
2. Preparation
2.1. PrivacyPreserving DNNs
2.2. Image Encryption for PrivacyPreserving DNNs
 Divide the RGB color image I into blocks of $M\times M$ pixels.
 Separate each pixel into upper and lower 4 bit pixel values to form sixchannel blocks.
 Reverse the intensities of the pixel values in each block randomly by a secret key.
 Shuffle the pixel values in each block randomly by a secret key.
 Combine six channels in each block into three channel to generate an encrypted image.
 Divide RGB color image I into blocks of $M\times M$ pixels.
 Permute randomly the divided blocks by using a secret key.
 The same procedure of LE is applied to the permuted blocks to generate an encrypted image.
 Divide RGB color image I into $X\times Y$ pixels.
 Apply negative–positive transformations to each pixel of the three color channels randomly by using a secret key. In this scheme, the secret key is independently used for all color components. In this step, a pixel q is transformed to ${q}^{\prime}$ by$${q}^{\prime}=\left\{\begin{array}{cc}q\hfill & \left(r\right(i)=0)\hfill \\ q\oplus ({2}^{8}1)\hfill & \left(r\right(i)=1)\hfill \end{array},\right.$$
 Shuffle three color components of each pixel by using a secret key.
 Combine $X\times Y$ pixels to generate an encrypted image.
 Divide RGB color image I into blocks of $M\times M$ pixels.
 Permute randomly the divided blocks using a secret key.
 Rotate and invert randomly each block by using a secret key.
 Apply negative–positive transformations to each block by using a secret key according to Equation (1).
 Shuffle three color components of each block by using a secret key.
 Integrate the encrypted blocks to generate an encrypted image.
 Divide an RGB color image I into blocks $B=\{{B}_{1},\dots ,{B}_{i},\dots ,{B}_{n}\}$, $i\in \{1,\dots ,n\}$ with $M\times M$ pixels, where n is the number of divided blocks calculated by$$n=\lfloor \frac{X}{M}\rfloor \times \lfloor \frac{Y}{M}\rfloor .$$
 Permute the divided blocks by using a secret key ${K}_{VTE1}$, where ${K}_{VTE1}$ is commonly used for all color components. Accordingly, the scrambled blocks ${B}^{\prime}=\{{B}_{1}^{\prime},\dots ,{B}_{i}^{\prime},\dots ,{B}_{n}^{\prime}\}$ are generated.
 Divide each scrambled block ${B}_{i}^{\prime}$ into four nonoverlapping square subblocks ${S}_{ij},j\in \{UL,UR,LL,LR\}$ with $\frac{M}{2}\times \frac{M}{2}$ pixels, where ${S}_{iUL}$ is defined as the upper left position of the ith blocks, ${S}_{iUR}$ as the upper right, ${S}_{iLL}$ as the lower left, and ${S}_{iLR}$ as the lower right. Thereby, scrambled blocks divided into subblocks $S=\{{S}_{1j},\dots ,{S}_{ij},\dots ,{S}_{nj}\}$ are generated. The number of subblocks m is described as$$m=4n.$$
 Shuffle the pixel position within a subblock by using a secret key ${K}_{VTE2}$ to generate pixel shuffled subblocks ${S}^{\prime}=\{{S}_{1j}^{\prime},\dots ,{S}_{ij}^{\prime},\dots ,{S}_{nj}^{\prime}\}$, where ${K}_{VTE2}$ is commonly used for all subblocks and color components. As a result, each scrambled block is divided into four encrypted subblocks, denoted by ${S}_{i}^{\prime}=\{{S}_{iUL}^{\prime},{S}_{iUR}^{\prime},{S}_{iLL}^{\prime},{S}_{iLR}^{\prime}\}$.
 Merge all blocks to generate an encrypted image.
3. Proposed Attacks
3.1. Threat Models
Algorithm 1 FRattack [14] 

3.2. Jigsaw Puzzle SolverBased Attack
4. Experiments
4.1. Experimental Conditions
4.2. Experimental Results
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
 Yann, L.; Yoshua, B.; Geoffrey, H. Deep Learning. Nature 2015, 521, 436–444. [Google Scholar]
 Huang, Q.X.; Yap, W.L.; Chiu, M.Y.; Sun, H.M. PrivacyPreserving Deep Learning With Learnable Image Encryption on Medical Images. IEEE Access 2022, 10, 66345–66355. [Google Scholar] [CrossRef]
 Zeng, W.; Lei, S. Efficient frequency domain selective scrambling of digital video. IEEE Trans. Multimed. 2003, 5, 118–129. [Google Scholar] [CrossRef]
 Ito, I.; Kiya, H. A new class of image registration for guaranteeing secure data management. In Proceedings of the IEEE International Conference on Image Processing, San Diego, CA, USA, 12–15 October 2008; pp. 269–272. [Google Scholar]
 Kiya, H.; Ito, I. Image matching between scrambled images for secure data management. In Proceedings of the 16th European Signal Processing Conference, Lausanne, Switzerland, 25–29 August 2008; pp. 1–5. [Google Scholar]
 Ito, I.; Kiya, H. Onetime Key Based Phase Scrambling for Phaseonly Correlation Between Visually Protected Images. EURASIP J. Inf. Secur. 2009, 2009, 841045. [Google Scholar] [CrossRef]
 Tang, Z.; Zhang, X.; Lan, W. Efficient image encryption with block shuffling and chaotic map. Multimed. Tools Appl. 2015, 74, 5429–5448. [Google Scholar] [CrossRef]
 Sirichotedumrong, W.; Chuman, T.; Imaizumi, S.; Kiya, H. GrayscaleBased Block Scrambling Image Encryption for Social Networking Services. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), San Diego, CA, USA, 23–27 July 2018; pp. 1–6. [Google Scholar]
 Kiya, H.; Nagamori, T.; Imaizumi, S.; Shiota, S. PrivacyPreserving Semantic Segmentation Using Vision Transformer. J. Imaging 2022, 8, 233. [Google Scholar] [CrossRef] [PubMed]
 Kiya, H.; Iijima, R.; AprilPyone, M.; Kinoshita, Y. Image and Model Transformation with Secret Key for Vision Transformer. IEICE Trans. Inf. Syst. 2023, 106, 2–11. [Google Scholar] [CrossRef]
 Tanaka, M. Learnable Image Encryption. In Proceedings of the IEEE International Conference on Consumer ElectronicsTaiwan (ICCETW), Taipei, Taiwan, 6–8 July 2018; pp. 1–2. [Google Scholar]
 Sirichotedumrong, W.; Kinoshita, Y.; Kiya, H. PixelBased Image Encryption Without Key Management for PrivacyPreserving Deep Neural Networks. IEEE Access 2019, 7, 177844–177855. [Google Scholar] [CrossRef]
 Madono, K.; Tanaka, M.; Onishi, M.; Ogawa, T. Blockwise Scrambled Image Recognition Using Adaptation Network. In Proceedings of the Workshop on AAAI Conference Artificial Intellignece, New York, NY, USA, 7–12 February 2020. [Google Scholar]
 Chang, A.H.; Case, B.M. Attacks on Image Encryption Schemes for PrivacyPreserving Deep Neural Networks. arXiv 2020, arXiv:2004.13263. [Google Scholar] [CrossRef]
 Ito, H.; Kinoshita, Y.; Aprilpyone, M.; Kiya, H. Image to Perturbation: An Image Transformation Network for Generating Visually Protected Images for PrivacyPreserving Deep Neural Networks. IEEE Access 2021, 9, 64629–64638. [Google Scholar] [CrossRef]
 Xu, J.; Ai, B.; Chen, W.; Yang, A.; Sun, P. Image Encryption Methods in Deep Joint Source Channel Coding: A Review and Performance Evaluation. In Proceedings of the IEEE International Conference on Computer and Communications (ICCC), Xiamen, China, 28–30 July 2021; pp. 240–244. [Google Scholar]
 Kiya, H.; AprilPyone, M.; Kinoshita, Y.; Imaizumi, S.; Shiota, S. An Overview of Compressible and Learnable Image Transformation with Secret Key and Its Applications. APSIPA Trans. Signal Inf. Process. 2022, 11, e11. [Google Scholar] [CrossRef]
 Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An Image is Worth 16 × 16 Words: Transformers for Image Recognition at Scale. arXiv 2020, arXiv:2010.11929. [Google Scholar]
 AprilPyone, M.; Kiya, H. Blockwise Image Transformation with Secret Key for Adversarially Robust Defense. IEEE Trans. Inf. Forensics Secur. 2021, 16, 2709–2723. [Google Scholar] [CrossRef]
 Qi, Z.; AprilPyone, M.; Kiya, H. PrivacyPreserving Image Classification Using ConvMixer with Adaptive Permutation Matrix. In Proceedings of the 30th European Signal Processing Conference(EUSIPCO), Belgrade, Serbia, 29 August–2 September 2022; pp. 543–547. [Google Scholar]
 Sholomon, D.; David, O.E.; Netanyahu, N.S. An automatic solver for very large jigsaw puzzles using genetic algorithms. Genet. Program. Evolvable Mach. 2016, 17, 291–313. [Google Scholar] [CrossRef]
 Son, K.; Hays, J.; Cooper, D.B. Solving Square Jigsaw Puzzle by Hierarchical Loop Constraints. IEEE Trans. Pattern Anal. Mach. Intell. 2019, 41, 2222–2235. [Google Scholar] [CrossRef] [PubMed]
 Chuman, T.; Kurihara, K.; Kiya, H. Security Evaluation for Block Scramblingbased ETC Systems against Extended Jigsaw Puzzle Solver Attacks. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), Hong Kong, China, 10–14 July 2017; pp. 229–234. [Google Scholar]
 Chuman, T.; Kiya, H. A jigsaw puzzle solverbased attack on blockwise image encryption for privacypreserving DNNs. In Proceedings of the International Workshop on Advanced Imaging Technology (IWAIT), Hong Kong, China, 4–6 January 2023; pp. 335–340. [Google Scholar]
 Simonyan, K.; Zisserman, A. Very Deep Convolutional Networks for LargeScale Image Recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
 He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. arXiv 2015, arXiv:1512.03385. [Google Scholar]
 AprilPyone, M.; Kiya, H. PrivacyPreserving Image Classification Using an Isotropic Network. IEEE Trans. Multimed. 2022, 29, 23–33. [Google Scholar] [CrossRef]
 Chuman, T.; Sirichotedumrong, W.; Kiya, H. Encryptionthencompression systems using grayscalebased image encryption for jpeg images. IEEE Trans. Inf. Forensics Secur. 2019, 14, 1515–1525. [Google Scholar] [CrossRef]
 Chuman, T.; Kurihara, K.; Kiya, H. On the Security of Block ScramblingBased EtC Systems against Extended Jigsaw Puzzle Solver Attacks. IEICE Trans. Inf. Syst. 2018, 101, 37–44. [Google Scholar] [CrossRef]
 Sirichotedumrong, W.; Kiya, H. Visual Security Evaluation of Learnable Image Encryption Methods against Ciphertextonly Attacks. In Proceedings of the 2020 AsiaPacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Auckland, New Zealand, 7–10 December 2020; pp. 1304–1309. [Google Scholar]
 Gallagher, A. Jigsaw Puzzles with Pieces of Unknown Orientation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA, 16–21 June 2012; pp. 382–389. [Google Scholar]
Plain  LE [11]  PE [12]  ELE [13]  EtC [28]  VTE [20]  

Block scrambling  √  √  √  
Key type  Same  Different  Different  Same  Same  
Key space  $({M}^{2}\xb76)!\phantom{\rule{0.166667em}{0ex}}\xb7{2}^{{M}^{2}\xb76}$  ${2}^{3\xb7X\xb7Y}\xb7{6}^{X\xb7Y}$  ${(({M}^{2}\xb76)!\phantom{\rule{0.166667em}{0ex}}\xb7{2}^{{M}^{2}\xb76})}^{n}$$\xb7n!\phantom{\rule{0.166667em}{0ex}}$  ${8}^{n}\xb7{2}^{n}\xb7{6}^{n}\xb7n!\phantom{\rule{0.166667em}{0ex}}$  $n!\phantom{\rule{0.166667em}{0ex}}\xb7{\left(\frac{M}{2}\right)}^{2}!$  
Example  
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. 
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chuman, T.; Kiya, H. A Jigsaw Puzzle SolverBased Attack on Image Encryption Using Vision Transformer for PrivacyPreserving DNNs. Information 2023, 14, 311. https://doi.org/10.3390/info14060311
Chuman T, Kiya H. A Jigsaw Puzzle SolverBased Attack on Image Encryption Using Vision Transformer for PrivacyPreserving DNNs. Information. 2023; 14(6):311. https://doi.org/10.3390/info14060311
Chicago/Turabian StyleChuman, Tatsuya, and Hitoshi Kiya. 2023. "A Jigsaw Puzzle SolverBased Attack on Image Encryption Using Vision Transformer for PrivacyPreserving DNNs" Information 14, no. 6: 311. https://doi.org/10.3390/info14060311