# Privacy-Preserving Deep Neural Network Methods: Computational and Perceptual Methods—An Overview

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Computational Methods

- Setup: generates the public key $mpk$ and the master secret key $msk$.
- KeyDrive: outputs the function secret key $s{k}_{f}$ for the function f using $msk$.
- Encrypt: encrypts the sample message x using $mpk$.
- Decrypt: computes $f(x)$, using $mpk$, $s{k}_{f}$, and the ciphertext of x generated in Encrypt step.

#### 2.1. CryptoNets

#### 2.2. CryptoDL

#### 2.3. CryptoNN

- The authority: generates the secret key $msk$, the public key $mpk$, and the function secret key $s{k}_{f}$ (KeyDrive step).
- The client: preprocesses and encrypts the data—input (x) and labels (y)—using $mpk$ and sends them to the server (Encrypt step). The labels must first be coded using the one-hot method, and then mapped to a random vector number r whose components are ${r}_{i}$.
- The server: trains and tests the NN model using the data received from the client(s). Having the data and the first hidden layer in the feed-forward process, and the labels and the output layer in the back-propagation process, the server gets from the authority the $s{k}_{f}$ corresponding to the specific function and then decrypts the result of the function (Decrypt step). The server can continue the feed-forward and back-propagation processes normally. The output of the network is ${p}_{i}$, which is the probability that the data x belongs to the class i.

#### 2.4. Comparison

## 3. Perceptual Methods

#### 3.1. Tanaka’s Scheme

#### 3.2. Pixel-Based Image Encryption

- Same encryption key: all training and test images are encrypted using the same encryption key K.
- Different encryption keys: different keys are independently assigned to training and test images.

#### 3.3. GAN-Based Image Transformation Scheme

#### 3.4. Model-Based Image Transformation Scheme

#### 3.5. Comparison

## 4. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Hussain, H. A Review of Artificial Intelligence Techniques in Image Steganography Domain. J. Eng. Sci. Technol.
**2017**, 12, 1835–1845. [Google Scholar] - Liu, J.; Ke, Y.; Zhang, Z.; Lei, Y.; Li, J.; Zhang, M.; Yang, X. Recent Advances of Image Steganography with Generative Adversarial Networks. IEEE Access
**2020**, 8, 60575–60597. [Google Scholar] [CrossRef] - Fu, Z.; Wang, F.; Cheng, X. The secure steganography for hiding images via GAN. EURASIP J. Image Video Process.
**2020**, 2020. [Google Scholar] [CrossRef] - So, J. Deep Learning-Based Cryptanalysis of Lightweight Block Ciphers. Secur. Commun. Netw.
**2020**, 2020, 1–11. [Google Scholar] [CrossRef] - Xiao, Y.; Hao, Q.; Yao, D.D. Neural Cryptanalysis: Metrics, Methodology, and Applications in CPS Ciphers. In Proceedings of the 2019 IEEE Conference on Dependable and Secure Computing (DSC), Hangzhou, China, 18–20 November 2019; pp. 1–8. [Google Scholar] [CrossRef][Green Version]
- Volna, E.; Kotyrba, M.; Kocian, V.; Janosek, M. Cryptography Based on Neural Network. ECMS
**2012**, 386–391. [Google Scholar] [CrossRef][Green Version] - Acar, A.; Aksu, H.; Uluagac, A.S.; Conti, M. A Survey on Homomorphic Encryption Schemes: Theory and Implementation. arXiv
**2017**, arXiv:1704.03578. [Google Scholar] [CrossRef] - Boneh, D.; Sahai, A.; Waters, B. Functional Encryption: Definitions and Challenges. In Theory of Cryptography; Ishai, Y., Ed.; Springer: Berlin/Heidelberg, Germany, 2011; pp. 253–273. [Google Scholar]
- Tanuwidjaja, H.C.; Choi, R.; Baek, S.; Kim, K. Privacy-Preserving Deep Learning on Machine Learning as a Service—A Comprehensive Survey. IEEE Access
**2020**, 8, 167425–167447. [Google Scholar] [CrossRef] - Bost, R.; Popa, R.; Tu, S.; Goldwasser, S. Machine Learning Classification over Encrypted Data. NDSS
**2015**, 4325. [Google Scholar] [CrossRef][Green Version] - Dowlin, N.; Gilad-Bachrach, R.; Laine, K.; Lauter, K.; Naehrig, M.; Wernsing, J. CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy. In Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA, 19–24 June 2016; pp. 201–210. [Google Scholar]
- Hesamifard, E.; Takabi, H.; Ghasemi, M. CryptoDL: Deep Neural Networks over Encrypted Data. arXiv
**2017**, arXiv:1711.05189. [Google Scholar] - Xu, R.; Joshi, J.B.D.; Li, C. CryptoNN: Training Neural Networks over Encrypted Data. arXiv
**2019**, arXiv:1904.07303. [Google Scholar] - Maekawa, T.; Kawamura, A.; Kinoshita, Y.; Kiya, H. Privacy-Preserving SVM Computing in the Encrypted Domain. In Proceedings of the 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Honolulu, HI, USA, 12–15 November 2018; pp. 897–902. [Google Scholar]
- Kawamura, A.; Kinoshita, Y.; Nakachi, T.; Shiota, S.; Kiya, H. A Privacy-Preserving Machine Learning Scheme Using EtC Images. IEICE Trans. Fundam. Electron. Commun. Comput. Sci.
**2020**, E103.A, 1571–1578. [Google Scholar] [CrossRef] - Tanaka, M. Learnable Image Encryption. In Proceedings of the 2018 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW), Taichung, Taiwan, 19–21 May 2018; pp. 1–2. [Google Scholar] [CrossRef][Green Version]
- Sirichotedumrong, W.; Kiya, H. Visual Security Evaluation of Learnable Image Encryption Methods against Ciphertext-only Attacks. In Proceedings of the 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Auckland, New Zealand, 7–10 December 2020; pp. 1304–1309. [Google Scholar]
- AprilPyone, M.; Sirichotedumrong, W.; Kiya, H. Adversarial Test on Learnable Image Encryption. In Proceedings of the 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE), Osaka, Japan, 15–18 October 2019; pp. 667–669. [Google Scholar] [CrossRef][Green Version]
- Sirichotedumrong, W.; Maekawa, T.; Kinoshita, Y.; Kiya, H. Privacy-Preserving Deep Neural Networks with Pixel-Based Image Encryption Considering Data Augmentation in the Encrypted Domain. In Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 22–25 September 2019; pp. 674–678. [Google Scholar] [CrossRef][Green Version]
- Sirichotedumrong, W.; Kinoshita, Y.; Kiya, H. Pixel-Based Image Encryption Without Key Management for Privacy-Preserving Deep Neural Networks. IEEE Access
**2019**, 7, 177844–177855. [Google Scholar] [CrossRef] - Sirichotedumrong, W.; Kiya, H. A GAN-Based Image Transformation Scheme for Privacy-Preserving Deep Neural Networks. In Proceedings of the 2020 28th European Signal Processing Conference (EUSIPCO), Amsterdam, The Netherlands, 18–21 January 2021; pp. 745–749. [Google Scholar] [CrossRef]
- Ito, H.; Kinoshita, Y.; Kiya, H. A Framework for Transformation Network Training in Coordination with Semi-trusted Cloud Provider for Privacy-Preserving Deep Neural Networks. In Proceedings of the 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Auckland, New Zealand, 7–10 December 2020; pp. 1420–1424. [Google Scholar]
- Ito, H.; Kinoshita, Y.; Kiya, H. Image Transformation Network for Privacy-Preserving Deep Neural Networks and Its Security Evaluation. arXiv
**2020**, arXiv:2008.03143. [Google Scholar] - Madono, K.; Tanaka, M.; Onishi, M.; Ogawa, T. Block-wise Scrambled Image Recognition Using Adaptation Network. arXiv
**2020**, arXiv:2001.07761. [Google Scholar] - Sirichotedumrong, W.; Kinoshita, Y.; Kiya, H. On the Security of Pixel-Based Image Encryption for Privacy-Preserving Deep Neural Networks. In Proceedings of the 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE), Osaka, Japan, 15–18 October 2019; pp. 121–124. [Google Scholar] [CrossRef][Green Version]

**Figure 1.**CryptoNN framework, reproduced from [13].

**Figure 2.**Tanaka’s block-based encryption scheme. © 2019 IEEE. Reprinted, with permission, from [18].

**Figure 3.**Pixel-based image encryption. © 2019 IEEE. Reprinted, with permission, from [19].

**Figure 4.**(

**a**) Cycle-GAN architecture. (

**b**) Training process of ${h}_{p}(.)$ in the TN-GAN-based method. © 2021 IEEE. Reprinted, with permission, from [21].

**Figure 5.**Training process of the transformation network in the TN-model-based method. © 2020 IEEE. Reprinted, with permission, from [23].

**Table 1.**Details and properties of the computational methods through the availability of the model training (T) and testing (t) using encrypted data, the dataset used, the depth of the NN (number of convolutional layers in the network), the accuracy of the classification, the accuracy of the original model(without modification and using simple images), the training time, and the number of predictions per hour that can be processed (# p/h).

T | t | Dataset | NN Depth | Accuracy | Original Accuracy | Training Time | # p/h | |
---|---|---|---|---|---|---|---|---|

CryptoNets [11] | ∘ | ⋆ | MNIST | 2 | 99% | - | - | 58,982 |

CryptoDL [12] | ∘ | ⋆ | MNIST | 5 | 99.52% | 99.56% | - | 163,840 |

CIFAR-10 | 8 | 91.5% | 94.5% | - | 2524 | |||

CryptoNN [13] | ⋆ | ⋆ | MNIST | 3 | 95.49% | 95.48% | 75 h | - |

**Table 2.**Details and properties of the perceptual methods through the availability of the model training (T) and testing (t) using encrypted data, the dataset used, the classifier network used to classify the encrypted data, the accuracy of the classification, the accuracy of the original model (using simple images), and the robustness against various COA.

T | t | Dataset | Classifier Network | Accuracy | Original Accuracy | Attacks | ||||
---|---|---|---|---|---|---|---|---|---|---|

FR-Attack | ITN-Attack | GAN-Attack | ||||||||

Tanaka’s Scheme [16] | ⋆ | ⋆ | CIFAR-10 | Pyramidal Residual | 86.3% | 88.4% | × | × | × | |

CIFAR-100 | Network | 56.8% | 59.1% | |||||||

Pixel-based | Same key [19] | ⋆ | ⋆ | CIFAR-10 | ResNet-18 | 91.76% | 95.53% | × | × | × |

Different key [20] | 91.39% | × | ✓ | × | ||||||

TN-GAN [21] | ⋆ | ⋆ | CIFAR-10 | ResNet-18 | 90.73% | 95.65% | ✓ | × | ✓ | |

CIFAR-100 | 67.36% | 77.24% | ||||||||

TN-model [22,23] | ∘ | ⋆ | CIFAR-10 | ResNet-20 | 91.72% | 91.23% | ✓ | ✓ | ✓ | |

CIFAR-100 | 70.78% | 67.9% |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

El Saj, R.; Sedgh Gooya, E.; Alfalou, A.; Khalil, M.
Privacy-Preserving Deep Neural Network Methods: Computational and Perceptual Methods—An Overview. *Electronics* **2021**, *10*, 1367.
https://doi.org/10.3390/electronics10111367

**AMA Style**

El Saj R, Sedgh Gooya E, Alfalou A, Khalil M.
Privacy-Preserving Deep Neural Network Methods: Computational and Perceptual Methods—An Overview. *Electronics*. 2021; 10(11):1367.
https://doi.org/10.3390/electronics10111367

**Chicago/Turabian Style**

El Saj, Raghida, Ehsan Sedgh Gooya, Ayman Alfalou, and Mohamad Khalil.
2021. "Privacy-Preserving Deep Neural Network Methods: Computational and Perceptual Methods—An Overview" *Electronics* 10, no. 11: 1367.
https://doi.org/10.3390/electronics10111367