# Fully Homomorphically Encrypted Deep Learning as a Service

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

#### 1.1. Motivation

- To test the feasibility of using FHE on a new application in agri-food, specifically dairy milk data for milk yield forecasting.
- To evaluate the performance of using encrypted deep learning as a service towards solving this agri-food milk yield forecasting problem/application.
- To show how sequence models can be built in an FHE compatible manner which is a void in the current encrypted deep learning field.

#### 1.2. Commutative Rings Formalisation

- $\mathbb{Z}$; integers, e.g., $(-1,0,1,2,\dots )$ Formally: An integer is any number that has no fractional part (not a decimal).
- $\mathbb{Q}$; rational numbers, e.g., $(5,1.75,0.001,-0.1,\dots )=(\frac{5}{1},\frac{7}{4},\frac{1}{1000},\frac{-1}{10},\dots )$ Formally; a rational number is a number that can be in the fractional form $\frac{a}{b}$ where a and b are integers and b is non-zero.
- $\mathbb{R}$; real numbers, e.g., $(0,-1.5,3/7,0.32,\pi )$ Formally; a real number is any non-imaginary, non-infinite number.
- $\mathbb{C}$; complex numbers, e.g., $(1+i,32+-2.2i,5,-6i)$ Formally: A number which is a combination of real and imaginary numbers, where either part can be zero.

- $\mathbb{I}$; imaginary numbers, e.g., where: $i=\sqrt{-1},(i,-i,39.8i,\dots )$ Formally: Imaginary numbers are any numbers which are multiplied by the imaginary unit i.

- addition axioms;
- given: $(x,y,z\in R)$, then:

- multiplication axioms;
- given: $(x,y,z\in R)$, then:

- multiplicative additive axioms;
- given: $(x,y,z\in R)$, then:

## 2. Materials and Methods

- Using FHE towards creating encrypted sequence models, which has only been peripherally explored at this point.

#### 2.1. Data Pipeline

#### 2.1.1. Data Wrangling

#### 2.1.2. Client/Data Source

#### 2.1.3. Server/Data Processor

#### 2.2. Interface

#### 2.3. Fully Homomorphic Encryption Library

#### 2.4. Fully Homomorphic Encryption in Deep Learning

#### 2.4.1. Forward Pass

- $\sigma $ = sigmoid
- x = some input vector x

- $\sigma $ = sigmoid/sigmoid approximation
- x = some input vector x
- e = eulers number

#### 2.4.2. Backward Pass

#### 2.4.3. Weight Update

## 3. Results and Discussion

## 4. Future Work

**Neural network components**: We are working on improving some of the approximations and components presented here, taking into account recent advancements made on sigmoid approximation and ReLU, as proposed by Ali [12]. There have also been some very recent techniques proposed, which are relevant to our work, such as Lee et al. batch normalisation and kernel implementation [5]. Nevertheless, the purpose of this paper was to consider FHE in conjunction with deep learning and show at least that it can be applied and use in practical client-server settings.

**FHE**: In this paper, we treat FHE and leveled-FHE (LFHE) as if they are the same, however, FHE includes the use of a bootstrapping function, which is an operation which Microsoft-SEAL does not yet support, however, this is a road-mapped feature which means in due course this initially LFHE implementation can be reused for FHE as it becomes available.

**Plaintext Backpropagration**: Readers will note that we have in several places mentioned that backpropagation is calculated in plaintext. This is a prevalent limitation in all FHE deep learning implementations that is often overlooked/attention is not drawn to. This is primarily due to both the aforementioned lack of bootstrapping in many implementations which makes such long computations untenable (due to cyphertext size and computation time), the lack of compatibility in loss functions (which we and the broader community are working towards improving), and finally the inoperability of encrypted weights. Here, by inoperability we mean the following:

- Cyphertext + Cyphertext operations take an order of magnitude longer to compute; if we maintained encryption throughout the backward pass and kept the data absolutely secret then we would also have to pay this computational cost.
- Cyphertext + Cyphertext operations can only be computed on identically parameterised and borne of the same secret key, which means we would require all EDLaaS data owners to encrypt their data using the same key, which would mean the point of encryption would be lost since the secret key would then be effectively openly accessible.
- We could decrypt the finalised weights to overcome these previous limitations but then we could still derive at least a generalised representation of the source data in which case there is not much to be gained by computing backpropagation while encrypted.
- There is nothing stopping us from only using the forward pass for data owners while having pre-trained or transferred models to various scenarios of which they can choose. This would lose some accuracy but at this current time, it seems like the optimal scenario if data privacy/sensitivity is of key importance but computation is still required.

## 5. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Gilad-Bachrach, R.; Dowlin, N.; Laine, K.; Lauter, K.; Naehrig, M.; Wernsing, J. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In Proceedings of the International Conference on Machine Learning, New York, NY, USA, 19–24 June 2016; pp. 201–210. [Google Scholar]
- Gentry, C. Fully homomorphic encryption using ideal lattices. In Proceedings of the Forty-First Annual ACM Symposium on Theory of Computing, Bethesda, MD, USA, 31 May–2 June 2009; pp. 169–178. [Google Scholar]
- Marcano, N.J.H.; Moller, M.; Hansen, S.; Jacobsen, R.H. On fully homomorphic encryption for privacy-preserving deep learning. In Proceedings of the 2019 IEEE Globecom Workshops (GC Wkshps), Waikoloa, HI, USA, 9–13 December 2019; pp. 1–6. [Google Scholar]
- Meftah, S.; Tan, B.H.M.; Mun, C.F.; Aung, K.M.M.; Veeravalli, B.; Chandrasekhar, V. DOReN: Towards Efficient Deep Convolutional Neural Networks with Fully Homomorphic Encryption. IEEE Trans. Inf. Forensics Secur.
**2021**, 16, 3740–3752. [Google Scholar] [CrossRef] - Lee, J.W.; Kang, H.; Lee, Y.; Choi, W.; Eom, J.; Deryabin, M.; Lee, E.; Lee, J.; Yoo, D.; Kim, Y.S.; et al. Privacy-Preserving Machine Learning with Fully Homomorphic Encryption for Deep Neural Network. arXiv
**2021**, arXiv:2106.07229. [Google Scholar] - Juvekar, C.; Vaikuntanathan, V.; Chandrakasan, A. GAZELLE: A low latency framework for secure neural network inference. In Proceedings of the 27th USENIX Security Symposium (USENIX Security 18), Baltimore, MD, USA, 15–17 August 2018; pp. 1651–1669. [Google Scholar]
- Krizhevsky, A.; Hinton, G. Learning Multiple Layers of Features from Tiny Images. 2009. Available online: https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf (accessed on 28 August 2021).
- Cheon, J.H.; Kim, A.; Kim, M.; Song, Y. Homomorphic encryption for arithmetic of approximate numbers. In International Conference on the Theory and Application of Cryptology and Information Security; Springer: Hong Kong, China, 2017; pp. 409–437. [Google Scholar]
- Cheon, J.H.; Han, K.; Kim, A.; Kim, M.; Song, Y. Bootstrapping for approximate homomorphic encryption. In Annual International Conference on the Theory and Applications of Cryptographic Techniques; Springer: Tel Aviv, Israel, 2018; pp. 360–384. [Google Scholar]
- Kingma, D.P.; Ba, J. Adam: A Method for Stochastic Optimization. arXiv
**2017**, arXiv:1412.6980. [Google Scholar] - LeCun, Y.; Cortes, C. MNIST Handwritten Digit Database. 2010. Available online: https://deepai.org/dataset/mnist (accessed on 28 August 2021).
- Ali, R.E.; So, J.; Avestimehr, A.S. On polynomial approximations for privacy-preserving and verifiable relu networks. arXiv
**2020**, arXiv:2011.05530. [Google Scholar] - Noble, A. Protecting Privacy in Practice; The Royal Society: London, UK, 2019. [Google Scholar]
- Alhnaity, B.; Pearson, S.; Leontidis, G.; Kollias, S. Using deep learning to predict plant growth and yield in greenhouse environments. Acta Hortic.
**2020**, 425–432. [Google Scholar] [CrossRef] - Alhnaity, B.; Kollias, S.; Leontidis, G.; Jiang, S.; Schamp, B.; Pearson, S. An autoencoder wavelet based deep neural network with attention mechanism for multi-step prediction of plant growth. Inf. Sci.
**2021**, 560, 35–50. [Google Scholar] [CrossRef] - Durrant, A.; Markovic, M.; Matthews, D.; May, D.; Enright, J.; Leontidis, G. The Role of Cross-Silo Federated Learning in Facilitating Data Sharing in the Agri-Food Sector. arXiv
**2021**, arXiv:2104.07468. [Google Scholar] - Hossain, M.S.; Al-Hammadi, M.; Muhammad, G. Automatic fruit classification using deep learning for industrial applications. IEEE Trans. Ind. Inform.
**2018**, 15, 1027–1034. [Google Scholar] [CrossRef] - Cheng, X.; Zhang, Y.; Chen, Y.; Wu, Y.; Yue, Y. Pest identification via deep residual learning in complex background. Comput. Electron. Agric.
**2017**, 141, 351–356. [Google Scholar] [CrossRef] - Pearson, S.; May, D.; Leontidis, G.; Swainson, M.; Brewer, S.; Bidaut, L.; Frey, J.G.; Parr, G.; Maull, R.; Zisman, A. Are Distributed Ledger Technologies the panacea for food traceability? Glob. Food Secur.
**2019**, 20, 145–149. [Google Scholar] [CrossRef] - Durrant, A.; Markovic, M.; Matthews, D.; May, D.; Leontidis, G.; Enright, J. How might technology rise to the challenge of data sharing in agri-food? Glob. Food Secur.
**2021**, 28, 100493. [Google Scholar] [CrossRef] - Ershov, M. Survey of Algebra. 2015. Available online: http://people.virginia.edu/~mve2x/3354_Spring2015/ (accessed on 10 November 2019).
- Onoufriou, G. Python Fully Homomorphically Encrypted Microsoft Seal Abstraction Libary, ReSeal Repository. 2020. Available online: https://github.com/DreamingRaven/python-reseal (accessed on 25 November 2020).
- Chen, H.; Gilad-Bachrach, R.; Han, K.; Huang, Z.; Jalali, A.; Laine, K.; Lauter, K. Logistic Regression over Encrypted Data from Fully Homomorphic Encryption. Available online: https://eprint.iacr.org/2018/462 (accessed on 28 August 2021).

**Figure 1.**The pipeline demonstrates the key stages of our project, from the client and raw data (

**upper left**) to the data processing and analytics (

**lower right**).

**Figure 2.**Serialised representation of encrypted data using CKKS scheme, and including all private, relin, and public keys, where objects here are byte arrays.

**Figure 3.**FHE dashboard, allowing simple upload, data view (of metadata since data is encrypted), and processing of data.

**Figure 4.**FHE compatible neural network graph implemented by Python-ReSeal [22], visualised using PyVis, deployed towards predicting time series milk yield data via 1D Convolutional Neural Network (CNN)/biased cross-correlation (CC) with activation. Further in this diagram, blue represents input nodes, yellow represents CC/CNN nodes/components, pink represents the dense layer to condense the feature vector from the CNN layer, green is all glue operations such as enqueue and dequeue to merge and split inputs along varying edges respectively, orange is predictions, and red is loss functions. Purple is a special/unique set of operations related to the encryption itself such as decryption before moving on to the final circuit.

**Figure 5.**Graphical comparison of the sigmoid (purple) and sigmoid approximation (green) functions, showing their similarity between the range of −5 and 5.

**Table 1.**Tabular-summarised neural network architecture, outlining the neural network components and how they are constructed using various constituent nodes, along with the parameters these nodes received such as filter shapes to randomly initialise the weights.

Neural Network Component | Constituent Nodes | Parameters |
---|---|---|

Inputs | x, y | |

One Dimensional Convolutional Neural Network (1D CNN) | 1D-CC, CC-dequeue, CC-enqueue, CC-sop- *, CNN-acti | filter shape: (5, 1, 7), bias: 0 |

Fully Connected Artificial Neural Network (ANN/Dense) | Dense, Dense-acti | weight shape: (16), bias: 0 |

Mean Squared Error (MSE) Loss | MSE | |

Adaptive moment (Adam) Optimiser | Not a Node * | alpha: 0.001, beta_1: 0.9, beta_2: 0.999 |

Outputs | y_hat |

**Table 2.**Time performance for different operations, both locally and remotely. It should be noted that N/A (not applicable) is used to identify operations which are not implemented atomically as remote API operations as they would not make sense since they are too atomic to warrant the transmission overhead alone. These operations of course still exist on the server side but embedded into much more complex operations such as part of inference in our neural network depicted in Figure 4.

Operation | Locally (Seconds 3 s.f) | Remotely (Seconds 3 s.f) |
---|---|---|

Encryption | 0.0136 | 0.454 |

Decryption | 0.0330 | 1.14 |

Inference | 0.966 | 3.13 |

Cyphertext + Cyphertext | 0.287 | N/A |

Cyphertext + Plaintext | 0.0480 | N/A |

Cyphertext*Cyphertext | 0.277 | N/A |

Cyphertext*Plaintext | 0.0500 | N/A |

**Table 3.**Output of loss functions for both validation and testing sets, using Mean Squared Error (MSE), and Mean Absolute Error (MAE) when using our neural network from Figure 4.

Data Set | MSE Cyphertext (4 s.f.) | MSE Plaintext (4 s.f.) | MAE Cyphertext (4 s.f.) | MAE Plaintext (4 s.f.) |
---|---|---|---|---|

validation | 0.02226 | 0.02225 | 0.1240 | 0.1240 |

testing | 0.02233 | 0.02233 | 0.1241 | 0.1241 |

**Table 4.**Space taken of different length vectors, unencrypted as NumPy arrays and encrypted as ReSeal vectors, including private keys and all meta-data required for operation.

Length | Polynomial Modulus Degree | Numpy Plaintext Size (bytes) | Encrypted Vector Size (bytes) |
---|---|---|---|

4096 | 8192 | 32,880 | 4,800,310 |

8192 | 16,384 | 65,648 | 9,600,592 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Onoufriou, G.; Mayfield, P.; Leontidis, G.
Fully Homomorphically Encrypted Deep Learning as a Service. *Mach. Learn. Knowl. Extr.* **2021**, *3*, 819-834.
https://doi.org/10.3390/make3040041

**AMA Style**

Onoufriou G, Mayfield P, Leontidis G.
Fully Homomorphically Encrypted Deep Learning as a Service. *Machine Learning and Knowledge Extraction*. 2021; 3(4):819-834.
https://doi.org/10.3390/make3040041

**Chicago/Turabian Style**

Onoufriou, George, Paul Mayfield, and Georgios Leontidis.
2021. "Fully Homomorphically Encrypted Deep Learning as a Service" *Machine Learning and Knowledge Extraction* 3, no. 4: 819-834.
https://doi.org/10.3390/make3040041