A Reinforcement Learning-Based Optimization Strategy for Noise Budget Management in Homomorphically Encrypted Deep Network Inference

Abstract

Homomorphic encryption provides a powerful cryptographic solution for privacy-preserving deep neural network inference, enabling computation on encrypted data. However, the practical application of homomorphic encryption is fundamentally constrained by the noise budget, a core component of homomorphic encryption schemes. The substantial multiplicative depth of modern deep neural networks rapidly consumes this budget, necessitating frequent, computationally expensive bootstrapping operations to refresh the noise. This bootstrapping process has emerged as the primary performance bottleneck. Current noise management strategies are predominantly static, triggering bootstrapping at pre-defined, fixed intervals. This approach is sub-optimal for deep, complex architectures, leading to excessive computational overhead and potential accuracy degradation due to cumulative precision loss. To address this challenge, we propose a Deep Network-aware Adaptive Noise-budget Management mechanism, a novel mechanism that formulates noise budget allocation as a sequential decision problem optimized via reinforcement learning. The core of the proposed mechanism comprises two components. First, we construct a layer-aware noise consumption prediction model to accurately estimate the heterogeneous computational costs and noise accumulation across different network layers. Second, we design a Deep Q-Network-driven optimization algorithm. This Deep Q-Network agent is trained to derive a globally optimal policy, dynamically determining the optimal timing and network location for executing bootstrapping operations, based on the real-time output of the noise predictor and the current network state. This approach shifts from a static, pre-defined strategy to an adaptive, globally optimized one. Experimental validation on several typical deep neural network architectures demonstrates that the proposed mechanism significantly outperforms state-of-the-art fixed strategies, markedly reducing redundant bootstrapping overhead while maintaining model performance.

1. Introduction

In the data-driven era, Deep Neural Networks (DNNs) have become a core technology for unlocking the value of massive data. However, a fundamental contradiction exists between this highly centralized data model and the growing societal and regulatory demands for privacy protection [1]. This conflict has catalyzed the rapid development of the privacy-preserving machine learning field [2]. Among the various technical approaches, Federated Learning (FL) and Multi-Party Computation (MPC) primarily address the issues of keeping data local or collaborative training, but they still have limitations in non-interactive inference scenarios between clients and a server [3,4]; Differential Privacy (DP), in contrast, focuses on protecting the statistical information of the model’s output rather than the input data itself [5]. In this context, Homomorphic Encryption (HE) offers a unique cryptographic solution. HE permits arbitrary computations on ciphertexts without decryption, fundamentally achieving data’s “usability without visibility” [6]. Although HE is theoretically promising, a significant gap exists in its practical application to modern deep networks [7]. The computation in HE schemes is based on a core constraint known as the noise budget [8]. Each multiplication operation consumes this budget, and the inherent high multiplicative depth of deep networks can quickly exhaust the initial budget. To prevent decryption failure, a periodic bootstrapping operation must be performed to “refresh” the budget [9]. However, bootstrapping itself is an extremely computationally expensive operation, with overhead far exceeding standard ciphertext computations [10]. Consequently, the bootstrapping operation has become the key bottleneck restricting HE inference performance.

Currently, mainstream HE schemes predominantly adopt static noise management strategies [11,12]. These strategies trigger bootstrapping by setting uniform noise thresholds at pre-defined, regular layer intervals. While the performance impact of this strategy is acceptable for shallow networks, its inherent limitations become prominent when applied to modern DNNs with substantial depth, leading to significant performance issues, especially when handling multiple inference tasks [13]. Furthermore, excessive bootstrapping introduces and accumulates minor computational precision losses; during the iterations of multiple training cycles, this cumulative effect can be sufficient to damage the model’s gradients and weights, which in turn affects inference accuracy [14]. Therefore, the core problem of this paper is to construct a globally optimal dynamic bootstrapping strategy that simultaneously optimizes the computational efficiency and model performance of deep networks while strictly adhering to the HE security thresholds.

To address the above problems, this paper proposes a Deep Network-aware Adaptive Noise-budget Management mechanism (DN-ANM). First, to tackle the lack of awareness in static strategies, we construct a layer-aware noise consumption prediction model capable of accurately estimating the heterogeneous computational overhead of different layers. Second, to address the lack of global optimality in traditional decision-making, we design an optimization algorithm driven by reinforcement learning. We utilize a Deep Q-Network (DQN) [15] to train an intelligent agent that can achieve a globally optimal policy allocation for the noise budget based on the output of the noise predictor and the current network state. To facilitate understanding, the key mathematical notations and symbols used throughout this paper are summarized in

Table 1. Summary of Key Notations.

Symbol	Description
$\mathcal{N}$	Deep Neural Network model
L	Total number of network layers
q	Ciphertext modulus
$\Delta$	Scaling factor in CKKS scheme
$B_{init}$	Initial noise budget
$B_{max}$	Maximum noise threshold for decryption correctness
${\left|\right|e\left|\right|}_{\infty }$	Infinity norm of the noise error
$\mathcal{S}$	State space in the Markov Decision Process
$\mathcal{A}$	Action space $\{a_0,a_1,a_2\}$
$s_t$	State vector at step t
$r_t$	Reward received at step t
${\pi }^{*}$	Optimal bootstrapping policy
$Q(s,a)$	Action-value function

. The main contributions of this paper are summarized as follows:

(1)

To address the problem that existing HE schemes use fixed noise budget allocation strategies, which cannot adapt to the heterogeneous computational characteristics between deep network layers, we propose a Deep Network-aware Adaptive Noise-budget Management mechanism. By constructing a layer-aware noise consumption prediction model, this mechanism achieves dynamic and precise allocation of the noise budget.

(2)

To solve the lack of global optimality and adaptability in traditional bootstrapping decision methods, we design a reinforcement learning-driven optimization algorithm for bootstrapping decisions. It uses a Deep Q-Network to learn the optimal timing for bootstrapping, minimizing the bootstrapping overhead while ensuring computational security.

(3)

Experimental validation on several typical deep networks demonstrates the effectiveness and scalability of the proposed mechanism. The results show that the mechanism can optimize the management strategy online according to the computational characteristics of network. While guaranteeing security requirements, it significantly reduces the redundant overhead of bootstrapping operations compared to fixed strategies.

This paper is organized as follows: Section 2 introduces the related work. Section 3 details the DN-ANM model architecture. Section 4 presents the reinforcement learning-based adaptive bootstrapping decision-making. Section 5 provides the experimental setup and results analysis. Section 6 concludes the paper.

2. Related Work

2.1. Homomorphic Encryption in Privacy-Preserving Machine Learning

HE enables computation on ciphertexts without prior decryption. This cryptographic primitive ensures that operations performed in the ciphertext space map directly to operations in the plaintext space. Formally, for an encryption scheme $\mathcal{E}$ and a binary operation ⊙, the scheme satisfies the homomorphic property if $Dec(\mathcal{E}\left(m_1\right)\odot \mathcal{E}\left(m_2\right))=m_1\odot m_2$ for plaintexts $m_1,m_2$ [6]. HE classifications depend on the supported operations and computation depth. Partially Homomorphic Encryption (PHE) supports a single operation type with unbounded depth. Somewhat Homomorphic Encryption (SHE) supports both addition and multiplication but limits the circuit depth due to noise accumulation. Fully Homomorphic Encryption (FHE) supports arbitrary computations on ciphertexts. FHE achieves this by introducing a bootstrapping mechanism to reduce noise, theoretically allowing infinite evaluation depth. Leveled HE, a subset of FHE, supports arbitrary depth provided the parameters are selected based on the circuit complexity.

A generic HE scheme comprises four algorithms: $(Gen,Enc,Dec,Eval)$. The key generation algorithm $(pk,sk)\leftarrow Gen\left(1^{\lambda }\right)$ outputs a public and private key pair based on the security parameter $\lambda$. The encryption algorithm yields a ciphertext $c\leftarrow Enc(pk,m)$, mapping the plaintext $m\in \mathcal{M}$ to the ciphertext space. The decryption algorithm recovers the message $m\leftarrow Dec(sk,c)$. The evaluation algorithm $c_{res}\leftarrow Eval(pk,f,c_1,...,c_k)$ computes the function f over the input ciphertexts. Modern schemes, including BFV [16] and CKKS [17], rely on the Ring Learning with Errors (RLWE) assumption [17] and operate as noise-based cryptosystems. The LWE problem can be formalized as: given samples $(a_i,b_i=a_i·s+e_{i}modq)$ over the polynomial ring $R_q={\mathbb{Z}}_q\left[X\right]/(X^N+1)$, where $a_i\in R_q$ is uniformly random, $s\in R_q$ is the secret, and $e_i$ is sampled from an error distribution $\chi$, the objective is to distinguish these samples from uniformly random pairs [18]. A freshly generated ciphertext $c\leftarrow Enc(\mathsf{pk},m)$ contains an initial, low-level noise $e_{\mathrm{init}}$, bounded by $|e_{\mathrm{init}}|<B_{init}$. Every homomorphic operation performed on the ciphertext causes this noise to grow [17]. Specifically, for homomorphic addition, the noise satisfies $|e_1+e_2| \le |e_1|+|e_2|$; for homomorphic multiplication, the noise growth is more significant, satisfying $|e_{\mathrm{mult}}| \lesssim |e_1|·|e_2| + |e_1| · |m_2| + |e_2| · |m_1|$. The correctness of the $\mathsf{Dec}$ algorithm is only guaranteed when the noise e is below a specific modulus or maximum bound $B_{\max }$. This characteristic poses a core challenge for DNN inference. A DNN is essentially an arithmetic circuit with a high multiplicative depth $\mathcal{L}$ [19]. To map DNNs onto HE, two paths have evolved in the academic community. One path is to use the BFV/BGV schemes, which quantize network weights and activations into integers and perform exact computations over the integer ring ${\mathbb{Z}}_t$. In the BFV scheme, the plaintext space is ${\mathbb{Z}}_t$, and the ciphertext space is $R_q^2$. The decryption process can be represented as $m=\lfloor {\displaystyle \frac{t}{q}}·{\left[\langle c,\mathsf{sk}\rangle \right]}_q\rceil modt$. The other path uses the CKKS scheme, which natively supports approximate arithmetic on real numbers $\mathbb{R}$. CKKS introduces a scaling factor $\Delta$ to control precision. The encoding process is $m\to \lfloor \Delta ·m\rceil$, and after decryption, rescaling is required: $m\approx {\displaystyle \frac{1}{\Delta }}·\mathsf{Dec}(\mathsf{sk},c)$. By adjusting the scaling factor $\Delta$ and the modulus q, CKKS allows for a flexible trade-off between precision and noise budget. This approximate nature enables CKKS to directly handle the floating-point weights and activations in neural networks without complex quantization procedures. Furthermore, while the integer quantization of BFV faces challenges with cumulative quantization errors when processing deep networks [20,21], the native support of CKKS for floating-point operations makes it more easily scalable to large-scale deep networks. Therefore, this paper selects the CKKS noise model as the foundation to construct a noise consumption predictor usable by the reinforcement learning agent, thereby achieving adaptive noise budget management for deep networks.

Regardless of the HE scheme chosen, non-polynomial activation functions (e.g., ReLU, Sigmoid, GeLU) pose a significant challenge. As HE natively supports only addition and multiplication, executing these functions requires relying on the costly bootstrapping $\mathsf{Boot}$ operation. This operation can “refresh” the ciphertext and reduce its noise: $c_{\mathrm{refresh}}\leftarrow \mathsf{Boot}(\mathsf{pk},c)$, such that $\mathsf{Dec}(\mathsf{sk},c_{\mathrm{refresh}})=\mathsf{Dec}(\mathsf{sk},c)$ but the noise is reset to its initial level. Alternatively, low-degree polynomials, such as $p\left(x\right)=x^2$ or $p\left(x\right)={\sum }_{i=0}^d{a}_i{x}^i$, are used as approximations [22].

2.2. Noise Budget Management in HE Inference

In HE schemes, the noise growth e is always constrained by an upper bound $B_{\max }$ to guarantee decryption correctness [23], requiring $|e_i| <B_{\max }$ for all $i\in [1,n]$, where n is the total number of computation steps. The performance of HE inference is essentially a constrained optimization problem: minimizing the computational overhead ${min}_{\theta }\mathcal{T}\left(\theta \right)$ subject to the noise constraint $e_{\mathrm{final}}<B_{\max }$, where $\theta$ denotes the HE parameter configuration and $\mathcal{T}$ is the total time overhead. Early work in HE-based machine learning, exemplified by CryptoNets [24], introduced the leveled homomorphic encryption paradigm. This strategy analyzes the total multiplicative depth ${\mathcal{L}}_{\mathrm{total}}={\sum }_{l=1}^L{d}_l$ of the entire DNN circuit C, where $d_l$ is the multiplicative depth of the l-th layer. It then selects a set of sufficiently large initial HE parameters $(\lambda ,q,N)$, such that the initial noise budget $B_{init}$ can accommodate the noise accumulation of the entire circuit, i.e., satisfying $B_{init}·{\prod }_{i=1}^{{\mathcal{L}}_{\mathrm{total}}}{\alpha }_i<B_{\max }$, where ${\alpha }_i$ is the noise growth factor of the i-th multiplication. This approach completely avoids the costly bootstrapping operation, exhibiting a time complexity of $O\left(T_{\mathrm{infer}}\right)$ without incurring the additional $O(k·T_{\mathrm{boot}})$ overhead. The trade-off, however, is a high-order dependency of the HE parameters on the multiplicative depth [25,26]. Specifically, the modulus q must satisfy $logq=O({\mathcal{L}}_{\mathrm{total}}·log{B}_{\mathrm{mult}})$, causing the ciphertext size $\left|c\right|=O(N·logq)$ and computational complexity $O(NlogN·logq)$ to grow exponentially. Consequently, this paradigm is only applicable to shallow networks such as LeNet-5, where ${\mathcal{L}}_{\mathrm{total}}\le 5$.

To overcome the depth limitation, Fan et al. [16] shifted the focus to automated optimization through static graph optimization and compile-time decisions. HE compilers, such as EVA [27], perform parameter pre-planning by conducting static analysis on the DNN’s dataflow graph $G=(V,E)$ at compile-time. The compiler constructs a noise propagation function $\varphi :V\to {\mathbb{R}}^{+}$, where $\varphi \left(v_i\right)$ denotes the estimated upper bound of the noise at node $v_i$. For each computational node $v\in V$, the compiler statically allocates a noise budget $b_v$ such that ${\sum }_{v\in V}{b}_v\le B_{\mathrm{total}}$, and inserts bootstrapping operations at predetermined locations $\{v_{j_1},v_{j_2},\dots ,v_{j_k}\}$. While these methods enhance the efficiency of HE applications, they remain fundamentally static. Consider three different layer types: the noise consumption of a $3\times 3$ convolutional layer is $\Delta e_{\mathrm{conv}3\mathrm{x}3}\approx 9·\left|w\right|·e_{\mathrm{prev}}$, that of a $1\times 1$ convolutional layer is $\Delta e_{\mathrm{conv}1\mathrm{x}1}\approx \left|w\right|·e_{\mathrm{prev}}$, and that of a non-linear activation layer using a degree-d polynomial approximation is $\Delta e_{\mathrm{act}}\approx O\left(e_{\mathrm{prev}}^d\right)$. These three operations consume the noise budget in a starkly different manner, satisfying $\Delta e_{\mathrm{conv}1\mathrm{x}1}\ll \Delta e_{\mathrm{conv}3\mathrm{x}3}\ll \Delta e_{\mathrm{act}}$. However, the static analyzer assigns them the same conservative budget $b_v=max\{\Delta e_{\mathrm{conv}1\mathrm{x}1},\Delta e_{\mathrm{conv}3\mathrm{x}3},\Delta e_{\mathrm{act}}\}$, resulting in suboptimal resource allocation efficiency $\eta ={\displaystyle \frac{B_{\mathrm{used}}}{B_{\mathrm{allocated}}}}<1$. As a complement to static compilers, some works employ runtime heuristic rules. This includes the fixed-interval strategy ${\mathcal{S}}_{\mathrm{interval}}:\mathrm{if}(imodk=0)\to \mathsf{Boot}$, which performs bootstrapping every k layers. Although this strategy possesses basic runtime awareness, enabling decisions based on the current noise state $e_t$, it disregards the noise consumption characteristics of the future computational path, $\mathbb{E}[\Delta e_{t+1:T}]$ [28]. Consequently, it triggers redundant bootstrapping operations at unnecessary locations, incurring additional time overhead.

3. Noise Model for HE Deep Networks

3.1. Problem Formulation Definition

Definition 1.

Algebraic Structure of CKKS. The CKKS homomorphic encryption scheme is defined over the quotient polynomial ring $R_q={\mathbb{Z}}_q\left[X\right]/(X^N+1)$, where $N=2^m$ is the ring dimension and $X^N+1={\Phi }_{2N}\left(X\right)$ is the $2N$-th cyclotomic polynomial, which is irreducible over the rational field $\mathbb{Q}$. The ciphertext modulus q is typically chosen as a product of distinct primes, $q={\prod }_{i=1}^k{q}_i$, where each prime $q_i$ satisfies $q_i\equiv 1(mod2N)$. This condition ensures the existence of a primitive $2N$-th root of unity in ${\mathbb{Z}}_{q_i}$, enabling an efficient implementation of negacyclic polynomial multiplication via the Number Theoretic Transform (NTT). The size and number of primes composing q are selected according to the required multiplicative depth, precision, and noise growth constraints of the homomorphic computation.

Although the polynomial $X^N+1$ is irreducible over $\mathbb{Q}$ and plays a central role in the security analysis based on the RLWE assumption, irreducibility modulo q is not required for homomorphic computation. In practice, due to the choice $q_i\equiv 1(mod2N)$, the polynomial $X^N+1$ splits into linear factors over each ${\mathbb{Z}}_{q_i}$. As a consequence, the ring $R_q$ admits a Chinese Remainder Theorem (CRT) decomposition $R_q\cong {\prod }_{i=1}^k\left({\mathbb{Z}}_{q_i}\left[X\right]/(X^N+1)\right)\cong {\prod }_{i=1}^k{\mathbb{Z}}_{q_i}^N$, which enables a Single Instruction Multiple Data (SIMD) representation and efficient parallel arithmetic in the CRT domain. Arithmetic operations in $R_q$ are defined as follows:

• Addition: Polynomial addition modulo q, performed coefficient-wise (or component-wise in the CRT representation),

  $$\begin{array}{c}\hfill (a+b)\left(X\right)=\sum _{j=0}^{N-1}(a_j+b_{j}modq)X^j.\end{array}$$

  (1)
• Multiplication: Negacyclic convolution modulo $X^N+1$ and q, which is efficiently implemented using the NTT,

  $$\begin{array}{c}\hfill (a·b)\left(X\right)=a\left(X\right)b\left(X\right)mod(X^N+1)modq.\end{array}$$

  (2)

The cryptographic primitives are defined over this structure as follows:

• $\mathsf{KeyGen}\left(1^{\lambda }\right)\to (\mathsf{pk},\mathsf{sk},\mathsf{evk})$: Generates the public key, secret key, and evaluation key.
• $\mathsf{Enc}(\mathsf{pk},m;\Delta )\to ct$: Encrypts a plaintext vector $m\in {\mathbb{C}}^{N/2}$ into a ciphertext $ct=(c_0,c_1)\in R_q^2$, where $\Delta$ is the scaling factor.
• $\mathsf{Dec}(\mathsf{sk},ct)\to m^{\prime }$: Decrypts the ciphertext to obtain an approximate plaintext $m^{\prime }\approx m$.

Definition 2.

Computational graph of deep neural networks. We formally represent a Deep Neural Network (DNN), denoted as $\mathcal{N}$, as a weighted Directed Acyclic Graph (DAG), $\mathcal{N}=(V,E,\Theta )$. In this graph, the vertex set $V=\{v_0,v_1,\dots ,v_L\}$ represents the nodes of the network, where $v_0$ is the input node, $v_L$ is the output node, and the intermediate vertices $v_i$ ($i\in [1,L-1]$) correspond to the i-th computational layer. The set of directed edges $E\subseteq V\times V$ defines the data dependencies between layers, where $(v_i,v_j)\in E$ indicates that data flows from layer i to layer j. $\Theta =\{{\theta }_1,{\theta }_2,\dots ,{\theta }_L\}$ is the set of parameters associated with each computational layer vertex. Each ${\theta }_i=(f_i,W_i,{\mathcal{M}}_i)$ comprises three components: the layer’s computational function $f_i:{\mathcal{X}}_i\to {\mathcal{X}}_{i+1}$; the layer’s weight parameters $W_i\in {\mathbb{R}}^{d_{i+1}\times d_i}$; and the meta-information ${\mathcal{M}}_i$ describing the layer’s topological structure, such as kernel size $k_i$, stride $s_i$, padding $p_i$, and the number of input/output channels $C_i^{\mathrm{in}},C_i^{\mathrm{out}}$, among others. Consequently, for any given input $x\in {\mathcal{X}}_0$, the network’s forward propagation process can be compactly expressed as the composite function $\mathcal{N}\left(x\right)=f_L\circ f_{L-1}\circ \dots \circ f_1\left(x\right)$ [29].

Definition 3.

Homomorphic network reasoning. Given a network $\mathcal{N}=(V,E,\Theta )$ and an input $x\in {\mathcal{X}}_0$, homomorphic network inference is defined as a state transition process ${\left\{(c{t}_i,e_i)\right\}}_{i=0}^L$ [24]. In the initialization phase, $c{t}_0=\mathsf{Enc}(\mathsf{pk},x;{\Delta }_0)$, with an initial noise $\parallel e_0{\parallel }_{\infty }=B_{init}$. For each layer $i\in [1,L]$, the homomorphic computation $c{t}_i=\mathcal{H}(f_i,c{t}_{i-1},W_i)$ is executed, where $\mathcal{H}$ denotes the homomorphic computation operator that maps the plaintext function $f_i$ to a sequence of ciphertext-domain operations. The output noise at i is determined by the noise growth function ${\Phi }_i$:

$$\begin{array}{c}\hfill \parallel e_i{\parallel }_{\infty }={\Phi }_i(\parallel e_{i-1}{\parallel }_{\infty },{\theta }_i)=\parallel e_{i-1}{\parallel }_{\infty }+\Delta {\mathcal{E}}_i(\parallel e_{i-1}{\parallel }_{\infty },W_i,{\mathcal{M}}_i)\end{array}$$

(3)

In this formulation, ${\Phi }_i$ represents the noise growth function that maps the input error state and layer configuration to the output error bound. The variable $\left|\right|e_{i-1}{\left|\right|}_{\infty }$ denotes the infinity norm of the noise inherited from preceding computations, serving as the baseline for the current step. The parameter set ${\theta }_i$ encapsulates the structural configuration of the layer, including the weight tensor $W_i$ and topological metadata ${\mathcal{M}}_i$. The term $\Delta {\mathcal{E}}_i$ defines the noise increment function, which quantifies the specific additive error generated by the arithmetic operations within layer i as a function of the input noise magnitude and the layer’s multiplicative complexity.

Problem 1.

The Adaptive Noise Budget Management Problem. Given an initial noise budget $B_{init}$, a maximum noise threshold $B_{\max }$, and the L computational layers of a network $\mathcal{N}$, the objective is to design an adaptive policy $\pi :\mathcal{S}\to \mathcal{A}$ with a ternary action space $\mathcal{A}=\{a_0,a_1,a_2\}$. This policy, for each state $s_i=(i,\parallel e_i{\parallel }_{\infty },{\left\{{\theta }_j\right\}}_{j>i})$ during inference, decides whether and when to execute the bootstrapping operation $\mathsf{Boot}$. Formally, we define a decision sequence $a=(a_1,a_2,\dots ,a_L)$, where $a_i=\pi \left(s_i\right)\in \{a_0,a_1,a_2\}$ ($a_i\in \{a_1,a_2\}$ indicates that bootstrapping is scheduled around layer i). The optimization goal is ${min}_{\pi }K\left(\pi \right)={\sum }_{i=1}^L\mathbb{I}[a_i\in \{a_1,a_2\}]$. Subject to the security constraint $\forall i\in [1,L],\parallel e_i{\parallel }_{\infty }<B_{\max }$. The noise evolution follows:

$$\begin{array}{c}\hfill \parallel e_i{\parallel }_{\infty }=\left\{\begin{array}{cc}{B}_{init}+\sum _{k=j+1}^i\Delta {\mathcal{E}}_k\hfill & \mathrm{if}{a}_i\in \{a_1,a_2\},j=max\{k<i:a_k\in \{a_1,a_2\}\}\hfill \\ \parallel e_{i-1}{\parallel }_{\infty }+\Delta {\mathcal{E}}_i\hfill & \mathrm{if}{a}_i=a_0\hfill \end{array}\right.\end{array}$$

(4)

The goal of this problem, given only the current noise state $\parallel e_i{\parallel }_{\infty }$, is to accurately predict the future noise trajectory $\{\parallel e_j{{\parallel }_{\infty }\}}_{j>i}$ to make optimal decisions, thereby avoiding both premature and delayed bootstrapping.

3.2. Deep Network-Aware Adaptive Noise-Budget Management (DN-ANM)

To address Problem 1, we propose the DN-ANM model. As shown in Figure 1, this model is composed of three core components, forming a complete closed loop for noise perception, prediction, and monitoring.

Network Structure Analyzer: The Network Structure Analyzer is defined as a mapping $\mathcal{A}:\mathcal{N}\to \mathcal{G}$, which transforms the input DNN model $\mathcal{N}$ into a structured graph representation $\mathcal{G}=(V,E,\Theta ,\Psi )$, where $\Psi =\{{\psi }_1,\dots ,{\psi }_L\}$ is the set of layer-wise feature vectors. For each layer i, the feature vector ${\psi }_i\in {\mathbb{R}}^d$ encodes the network’s structural features, including:

• Topological features: layer depth $d_i$, the set of predecessor nodes $\mathrm{pred}\left(v_i\right)=\{v_j:(v_j,v_i)\in E\}$, and the set of successor nodes $\mathrm{succ}\left(v_i\right)$.
• Operational features: the one-hot encoding of the operation type
  ${\tau }_i\in \{\mathrm{Conv},\mathrm{FC},\mathrm{Poly},\mathrm{Pool},\dots \}$.
• Parameter features: weight dimension $dim\left(W_i\right)$, weight norm $\parallel W_i{\parallel }_F$, and multiplicative complexity ${\mathcal{C}}_i^{\mathrm{mult}}$.
• Dataflow features: input/output tensor shapes $\mathrm{shape}\left(x_i^{\mathrm{in}}\right)$ and $\mathrm{shape}\left(x_i^{\mathrm{out}}\right)$.

This component has a time complexity of $O\left(\right|V|+|E\left|\right)$ and only needs to be executed once before the inference begins.

Noise Consumption Prediction: This component constructs a forward noise propagation model ${\mathcal{P}}_{\mathrm{pred}}:{\mathbb{R}}^{+}\times [1,L]\times [1,L]\to {\mathbb{R}}^{+}$. Given the current layer index i, the current noise level $\parallel e_i{\parallel }_{\infty }$, and a target layer index j ($j>i$), it predicts the cumulative noise consumption:

$$\begin{array}{c}\hfill {\mathcal{P}}_{\mathrm{pred}}(\parallel e_i{\parallel }_{\infty },i,j)=\sum _{k=i+1}^j\Delta {\mathcal{E}}_k^{\mathrm{pred}}(\parallel e_{k-1}{\parallel }_{\infty }^{\mathrm{pred}},{\theta }_k)\end{array}$$

(5)

The sequence of predicted noise states $\left\{\right||e_k{\left|\right|}_{\infty }^{pred}{\}}_{k=i}^j$ is computed recursively. The initialization is set by the current observed noise level, such that $\left|\right|e_i{\left|\right|}_{\infty }^{pred}=\left|\right|e_i{\left|\right|}_{\infty }$. For all subsequent layers $k>i$, the noise state evolves according to:

$$\begin{array}{c}\hfill \parallel e_k{\parallel }_{\infty }^{\mathrm{pred}}=\parallel e_{k-1}{\parallel }_{\infty }^{\mathrm{pred}}+\Delta {\mathcal{E}}_k^{\mathrm{pred}}(\parallel e_{k-1}{\parallel }_{\infty }^{\mathrm{pred}},{\theta }_k)\end{array}$$

(6)

The key to the predictor lies in accurately modeling the noise increment function $\Delta {\mathcal{E}}_i^{\mathrm{pred}}$ for each layer, the construction of which will be detailed in Section 3. Theoretically, if $\Delta {\mathcal{E}}_i^{\mathrm{pred}}=\Delta {\mathcal{E}}_i$, the prediction error would be zero.

Runtime Noise Monitor: This component tracks the current ciphertext noise level in real-time, as HE schemes typically provide functions to estimate the current noise level. The monitor integrates this real-time noise value, the current network layer index, and the future noise consumption predictions (provided by the prediction model) into a complete state vector. This vector is then supplied to the reinforcement learning decision agent, as detailed in Section 4, to support optimal decision-making. The component maintains a dynamic state space $\mathcal{S}=\mathcal{I}\times \mathcal{N}\times \mathcal{F}$, where $\mathcal{I}=[1,L]$ is the layer index space; $\mathcal{N}=[B_{init},B_{\max })$ is the noise level space; and $\mathcal{F}={\mathbb{R}}^{L-i}$ is the future noise consumption prediction space. At the i-th inference step, the monitor constructs the state vector $s_i=(i,\parallel e_i{\parallel }_{\infty }^{\mathrm{cur}},f_i)$. Here, $\parallel e_i{\parallel }_{\infty }^{\mathrm{cur}}$ represents the actual noise level of the current ciphertext, obtained via the CKKS scheme’s built-in function $\mathsf{GetNoise}\left(c{t}_i\right)$. The vector $f_i=({\mathcal{P}}_{\mathrm{pred}}(\parallel e_i{\parallel }_{\infty }^{\mathrm{cur}},i,i+1),{\mathcal{P}}_{\mathrm{pred}}(\parallel e_i{\parallel }_{\infty }^{\mathrm{cur}},i,i+2),\dots ,{\mathcal{P}}_{\mathrm{pred}}(\parallel e_i{\parallel }_{\infty }^{\mathrm{cur}},i,L\left)\right)$ is the prediction vector for the noise consumption of all subsequent layers. This state vector provides a complete decision-making context, serving as the informational foundation for the subsequent reinforcement learning agent.

3.3. Layer-Wise Sensing Noise Prediction Model Flowchart

The core of noise consumption prediction lies in the construction of a precise layer-wise noise increment function, $\Delta {\mathcal{E}}_i$. We adopt a bottom-up modeling strategy: commencing with a theoretical analysis of homomorphic primitives, progressively extending to composite layer-level operations, and culminating in an end-to-end noise propagation model.

3.3.1. Noise Modeling of Basic Homomorphic Operations

Consider the basic operations of the CKKS scheme. Let two ciphertexts, $c{t}_1$ and $c{t}_2$, have noises $e_1$ and $e_2$, plaintexts $m_1,m_2\in {\mathbb{C}}^{N/2}$, and modulus q. Homomorphic Addition: $c{t}_{add}=\mathrm{Add}(c{t}_1,c{t}_2)$. Its output noise satisfies $\parallel e_{add}{\parallel }_{\infty } \le \parallel e_1{\parallel }_{\infty } + {\parallel e_2\parallel }_{\infty }$. Homomorphic Multiplication: $c{t}_{mult}=\mathrm{Mult}(c{t}_1,c{t}_2)$ followed by relinearization $\mathsf{Relin}$. Its output noise satisfies:

$$\begin{array}{c}\hfill \parallel e_{mult}{\parallel }_{\infty }\lesssim \Delta ·(\parallel m_1{\parallel }_{\infty }· \parallel e_2{\parallel }_{\infty }+\parallel m_2{\parallel }_{\infty }· \parallel e_1{\parallel }_{\infty })+\parallel e_1{\parallel }_{\infty }· {\parallel e_2\parallel }_{\infty }+e_{relin}\end{array}$$

(7)

Here, $e_{relin}=O(N·{\sigma }^2)$ is the additional noise introduced by relinearization, where $\sigma$ is the standard deviation of the key distribution. Since $\parallel e_1{\parallel }_{\infty },{\parallel e_2\parallel }_{\infty }\ll \Delta$, the cross-term ${\Delta }^{-1}{e}_1{e}_2$ can be ignored. We thus define the dominant function for multiplicative noise growth as $h_{mult}{(\parallel e\parallel }_{\infty }{,\parallel m\parallel }_{\infty }{)=\Delta ·\parallel m\parallel }_{\infty }·{\parallel e\parallel }_{\infty }+e_{relin}$.

3.3.2. Noise Modeling of Neural Network Layer-Wise Operations

Based on the analysis of fundamental homomorphic operations, we differentiate between theoretical worst-case bounds and calibrated predictive models to establish precise noise increments for DNN layers.

Table 2. Summary of Calibrated Noise Increment Models by Layer Type.

Layer Type	Key Parameters	Noise Increment Model ($\mathit{\Delta }\mathcal{E}$)
Convolutional	$N_{mult},w_{max},{\alpha }_i,{\beta }_i$	${\alpha }_i{N}_{mult}(\Delta w_{max}+1)\left|\right|e_{in}\left|\right|+{\beta }_i{N}_{mult}$
Square Activation	$m_{max}$	$2\Delta m_{max}\left|\right|e_{in}\left|\right|+{\Delta }^{-1}\left|\right|e_{in}{\left|\right|}^2+e_{relin}$
Poly Activation	$m_{max},d$	${(2\Delta m_{max})}^{\lceil {log}_{2}d\rceil }\left|\right|e_{in}\left|\right|+O(d·e_{relin})$

provides a summary of these layer-wise noise models.

(1) Convolutional Layer Noise Increment: Consider a convolutional layer $l_i$ parameterized by kernel size $k\times k$, input channels $C_{in}$, output channels $C_{out}$, and a weight tensor $W_i$ with maximum magnitude $w_{max}=max\left|W_i\right|$. The computational cost is driven by the number of ciphertext-plaintext multiplications per activation, defined as $N_{mult}^{\left(i\right)}=k^2·C_{in}$. We first derive the theoretical upper bound for a single output activation, $\left|\right|e_{single}{\left|\right|}_{\infty }$, based on the worst-case interaction between weight scaling and input noise:

$$\begin{array}{c}\hfill \left|\right|e_{single}{\left|\right|}_{\infty }\le \Delta w_{max}{N}_{mult}^{\left(i\right)}\left|\right|e_{i-1}{\left|\right|}_{\infty }+N_{mult}^{\left(i\right)}{e}_{relin}+N_{mult}^{\left(i\right)}\left|\right|e_{i-1}{\left|\right|}_{\infty }\end{array}$$

(8)

Since this theoretical bound is often loose for efficient budgeting, we formulate a calibrated noise increment model. This model introduces learnable coefficients $\alpha i,{\beta }_i$ to map the theoretical components to the actual noise behavior:

$$\begin{array}{c}\hfill \Delta {\mathcal{E}}_i^{conv}\left(\right||e_{i-1}{\left|\right|}_{\infty },W_i)={\alpha }_i{N}_{mult}^{\left(i\right)}(\Delta w_{max}+1)\left|\right|e_{i-1}{\left|\right|}_{\infty }+{\beta }_i{N}_{mult}^{\left(i\right)}\end{array}$$

(9)

where ${\alpha }_i$ and ${\beta }_i$ are layer-specific calibration coefficients used to minimize the prediction error.

(2) Polynomial Activation Noise Increment: For a square activation $f\left(x\right)=x^2$, involving ciphertext-ciphertext self-multiplication, the noise increment is modeled as:

$$\begin{array}{c}\hfill \Delta {\mathcal{E}}_i^{square}\left(\right||e_{i-1}{\left|\right|}_{\infty })=2\Delta m_{max}\left|\right|e_{i-1}{\left|\right|}_{\infty }+{\Delta }^{-1}\left|\right|e_{i-1}{\left|\right|}_{\infty }^2+e_{relin}\end{array}$$

(10)

where $m_{max}$ denotes the maximum plaintext norm. For a general polynomial activation of degree d, the noise growth scales exponentially with the multiplicative depth $\lceil {log}_{2}d\rceil$:

$$\begin{array}{c}\hfill \Delta {\mathcal{E}}_i^{poly}\left(\right||e_{i-1}{\left|\right|}_{\infty },d)\approx {(2\Delta m_{max})}^{\lceil {log}_{2}d\rceil }\left|\right|e_{i-1}{\left|\right|}_{\infty }+O(d·e_{relin})\end{array}$$

(11)

3.4. Cross-Layer Noise Propagation and Cumulative Prediction

By integrating the noise models for each layer, we construct a complete inter-layer noise propagation equation. Let the network comprise L layers, with each layer’s noise increment function denoted as $\Delta {\mathcal{E}}_i$ (where the corresponding model is selected based on the layer type). Starting from the initial noise budget $\left|\right|e_0{\left|\right|}_{\infty }=B_{init}$, we derive the cumulative noise at layer i by iteratively expanding the recursive relationship established in Equation (6). Since each step adds a strictly positive noise increment $\Delta {\mathcal{E}}_k$, the total noise is the sum of the initial state and the increments of all preceding layers $k=1$ to i. This telescoping expansion yields the closed-form expression:

$$\begin{array}{c}\hfill \parallel e_i{\parallel }_{\infty }^{\mathrm{pred}}=\parallel e_0{\parallel }_{\infty }+\sum _{k=1}^i\Delta {\mathcal{E}}_k^{\mathrm{pred}}(\parallel e_{k-1}{\parallel }_{\infty }^{\mathrm{pred}},{\theta }_k)\end{array}$$

(12)

To evaluate the prediction accuracy, we define the layer-wise prediction error ${\xi }_i=|\parallel e_i{\parallel }_{\infty }^{\mathrm{pred}}-\parallel e_i{\parallel }_{\infty }^{\mathrm{actual}}|$ and the layer-wise relative error ${\eta }_i={\xi }_i/{\parallel e_i\parallel }_{\infty }^{\mathrm{actual}}$.

Theorem 1.

(Noise Prediction Error Bound): Suppose the noise increment prediction for each layer satisfies $|\Delta {\mathcal{E}}_i^{\mathrm{pred}}-\Delta {\mathcal{E}}_i|\le ϵ$, and the noise growth follows a linear pattern (i.e., $\Delta {\mathcal{E}}_i\propto {\parallel e_{i-1}\parallel }_{\infty }$, with a growth factor of ${\gamma }_i$). Then, the absolute prediction error at the i-th layer satisfies: ${\xi }_i\le ϵ·{\displaystyle \frac{{(1+{\gamma }_{max})}^i-1}{{\gamma }_{max}}}$ where ${\gamma }_{max}={max}_{k\in [1,i]}{\gamma }_k$.

Proof of Theorem 1. 

By mathematical induction. Base Case ($i=1$): ${\xi }_1=|\Delta {\mathcal{E}}_1^{\mathrm{pred}}-\Delta {\mathcal{E}}_1|\le ϵ$. Inductive Step: Assume the hypothesis holds for $i-1$, i.e., ${\xi }_{i-1}\le ϵ·{\displaystyle \frac{{(1+{\gamma }_{max})}^{i-1}-1}{{\gamma }_{max}}}$. Then: ${\xi }_i=|\parallel e_{i-1}{\parallel }_{\infty }^{\mathrm{pred}}+\Delta {\mathcal{E}}_i^{\mathrm{pred}}-\parallel e_{i-1}{\parallel }_{\infty }^{\mathrm{actual}}-\Delta {\mathcal{E}}_i|$ $\le {\xi }_{i-1}+|\Delta {\mathcal{E}}_i^{\mathrm{pred}}(\parallel e_{i-1}{\parallel }_{\infty }^{\mathrm{pred}})-\Delta {\mathcal{E}}_i(\parallel e_{i-1}{\parallel }_{\infty }^{\mathrm{actual}}\left)\right|$ $\le {\xi }_{i-1}+{\gamma }_i{\xi }_{i-1}+ϵ\le (1+{\gamma }_{max}){\xi }_{i-1}+ϵ$ Substituting the inductive hypothesis and expanding yields the result. Theorem 1 indicates that under the assumption of a noise growth factor ${\gamma }_{max}<1$, the growth of the prediction error is bounded. By minimizing $ϵ$ using the calibration coefficients ${\alpha }_i,{\beta }_i$, it can be ensured that the cumulative prediction error for the entire network remains within an acceptable range.    □

4. Reinforcement Learning-Based Adaptive Bootstrapping Decision

This section formalizes the adaptive bootstrapping problem as a Markov Decision Process (MDP) and proposes a multi-objective driven decision algorithm. This algorithm incorporates the noise prediction model from the preceding section as part of the environment. It aims to learn an optimal policy function ${\pi }^{*}:\mathcal{S}\to \mathcal{A}$, seeking to achieve a Pareto optimum [30] between computational security and the overhead of homomorphic operations.

4.1. MDP Problem Formulation

Definition 4

(Adaptive Bootstrapping MDP). We formalize the adaptive noise budget management problem defined in Problem 1 as a finite-horizon Markov Decision Process (MDP) $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma ,H)$. Here, $\mathcal{S}$ is the state space, representing all possible system configurations during inference; $\mathcal{A}=\{a_0,a_1,a_2\}$ is the action space, corresponding to different bootstrapping decisions; $\mathcal{P}:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to [0,1]$ is the state transition probability function, defined as $\mathcal{P}\left(s^{\prime }\right|s,a)=Pr[s_{t+1}=s^{\prime }|s_t=s,a_t=a]$; $\mathcal{R}:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ is the reward function, quantifying the immediate utility of each state-action pair; $\gamma \in [0,1]$ is the discount factor, balancing immediate rewards and long-term gains; and $H=L$ is the decision horizon, corresponding to the total number of network layers. The objective of this MDP is to find an optimal policy ${\pi }^{*}:\mathcal{S}\to \Delta \left(\mathcal{A}\right)$ that maximizes the expected cumulative reward: ${\pi }^{*}=arg{max}_{\pi }{\mathbb{E}}_{\tau \sim \pi }\left[{\sum }_{t=0}^{H-1}{\gamma }^t\mathcal{R}(s_t,a_t)\right]$ where $\tau =(s_0,a_0,s_1,a_1,\dots ,s_{H-1},a_{H-1},s_H)$ is a complete decision trajectory, and $\Delta \left(\mathcal{A}\right)$ denotes a probability distribution over the action space [10,31].

4.1.1. Multi-Dimensional State Space

The design of the state space is determinant of the quality of policy learning. Our designed state vector, $s_i\in \mathcal{S}\subseteq {\mathbb{R}}^{d_s}$, integrates multi-scale information provided by the DN-ANM model from Section 3. It is structurally defined as $s_i={[s_i^{\mathrm{instant}},s_i^{\mathrm{foresight}},s_i^{\mathrm{structural}}]}^{\top }$ This vector is composed of three functional modules, which capture the instantaneous state, foresight prediction, and structural features, respectively.

Instantaneous State Features $s_i^{\mathrm{instant}}\in {\mathbb{R}}^4$ encode the basic information of the current inference step: $s_i^{\mathrm{instant}}={[r_{\mathrm{budget}},r_{\mathrm{progress}},r_{\mathrm{distance}},r_{\mathrm{util}}]}^{\top }$ where each component is defined as: Noise Budget Ratio: $r_{\mathrm{budget}}={\displaystyle \frac{\parallel e_i{\parallel }_{\infty }^{\mathrm{cur}}}{B_{\max }}}\in [0,1)$, characterizing the proportion of the current noise relative to the security threshold. As this ratio approaches 1, the system nears the failure boundary. Inference Progress Ratio: $r_{\mathrm{progress}}={\displaystyle \frac{i}{L}}\in [0,1]$, quantifying the completeness of the inference process. Combined with the noise budget ratio, this feature helps the agent determine if the current rate of noise consumption is reasonable. Bootstrapping Distance: $r_{\mathrm{distance}}={\displaystyle \frac{d_i}{L}}$, where $d_i=i-max\{j<i:a_j\in \{a_1,a_2\}\}$ is the number of layers elapsed since the last bootstrapping. This feature provides an explicit signal regarding the duration of noise accumulation. Budget Utilization $r_{\mathrm{util}}={\displaystyle \frac{\parallel e_i{\parallel }_{\infty }^{\mathrm{cur}}-B_{init}}{B_{\max }-B_{init}}}\in [0,1]$, measuring the proportion of the effective noise budget used since the last bootstrapping, incentivizing the agent to fully utilize the budget provided by each bootstrapping operation.

Foresight Prediction Features $s_i^{\mathrm{foresight}}\in {\mathbb{R}}^3$ directly leverage the future information provided by the noise predictor ${\mathcal{P}}_{\mathrm{pred}}$ from Section 3 $s_i^{\mathrm{foresight}}={[r_{\mathrm{next}},r_{\mathrm{remain}},r_{\mathrm{horizon}}]}^{\top }$ Its components are defined as: Immediate Consumption Ratio: $r_{\mathrm{next}}={\displaystyle \frac{{\mathcal{P}}_{\mathrm{pred}}(\parallel e_i{\parallel }_{\infty }^{\mathrm{cur}},i,i+1)}{B_{\max }-{\parallel e_i\parallel }_{\infty }^{\mathrm{cur}}}}$, representing the proportion of the expected noise consumption of the next layer relative to the current remaining budget, providing a short-term risk assessment. Remaining Consumption Ratio: $r_{\mathrm{remain}}={\displaystyle \frac{{\mathcal{P}}_{\mathrm{pred}}(\parallel e_i{\parallel }_{\infty }^{\mathrm{cur}},i,L)}{B_{\max }-{\parallel e_i\parallel }_{\infty }^{\mathrm{cur}}}}$, the ratio of the total expected noise consumption from the current layer to the end of the network against the remaining budget, offering a long-term risk assessment. If $r_{\mathrm{remain}}>1$, it indicates that the inference will fail without bootstrapping. Foresight Horizon: $r_{\mathrm{horizon}}=min\left\{k:{\mathcal{P}}_{\mathrm{pred}}(\parallel e_i{\parallel }_{\infty }^{\mathrm{cur}},i,i+k)\ge B_{\max }-{\parallel e_i\parallel }_{\infty }^{\mathrm{cur}}\right\}$, representing the maximum number of layers that can be safely traversed with the current budget, providing the agent with a clear decision-making time window.

Structural Features $s_i^{\mathrm{structural}}\in {\mathbb{R}}^{\left|\mathcal{T}\right|+1}$ encode network topology information, where $\mathcal{T}$ is the set of all layer types $s_i^{\mathrm{structural}}={[r_{\mathrm{depth}},e_{\mathrm{layer}}]}^{\top }$ where Normalized Multiplicative Depth $r_{\mathrm{depth}}={\displaystyle \frac{{\sum }_{k=i+1}^L\mathbb{I}[{\tau }_k\in \{\mathrm{Conv},\mathrm{FC},\mathrm{Poly}\}]}{{\mathcal{L}}_{\mathrm{total}}}}$, characterizing the density of high-noise operations in the subsequent computations. Layer Type Encoding $e_{\mathrm{layer}}\in {\{0,1\}}^{\left|\mathcal{T}\right|}$ is the one-hot encoding of the next layer’s operation type, enabling the agent to distinguish the noise consumption patterns of different operations. Here, $\mathbb{I}[·]$ denotes the indicator function; ${\tau }_k$ denotes the type of layer k; and $\Theta$ denotes the parameter space of layers in the network. For typical linear layers (convolution and fully-connected), we denote the parameter at layer $i+1$ by $W_{i+1}\in \Theta$.

Theorem 2.

(Markov Property of the State Space). The defined state space $\mathcal{S}$ satisfies the Markov property. That is, given the current state $s_i$, the distribution of the future state $s_{i+1}$ is conditionally independent of the historical states $\{s_0,\dots ,s_{i-1}\}$: $Pr\left[s_{i+1}\right|s_0,\dots ,s_i,a_i]=Pr\left[s_{i+1}\right|s_i,a_i]$.

Proof of Theorem 2. 

The state $s_i$ completely encodes all sufficient statistics required for decision-making, as it contains the current noise level $\parallel e_i{\parallel }_{\infty }^{\mathrm{cur}}$, the current layer index i, and all future layer noise predictions computed via ${\mathcal{P}}_{\mathrm{pred}}$. The recursive nature of the noise predictor ${\mathcal{P}}_{\mathrm{pred}}$ (as defined in Section 3) ensures that future noise evolution depends only on the current noise and the network structure, and this network structure information is already encoded by $s_i^{\mathrm{structural}}$. Therefore, the influence of historical decisions on the future is fully captured by the current noise level, thereby satisfying the Markov property.    □

4.1.2. Extended Action Space

To empower the agent with fine-grained control capabilities, we design a ternary action space $\mathcal{A}=\{a_0,a_1,a_2\}$, which ameliorates the limitations of binary decisions.

• $a_0$ (Continue): No bootstrapping; compute the next layer directly.
• $a_1$ (Immediate Bootstrapping): Bootstrap first to reset noise and distance, then compute the next layer.
• $a_2$ (Pre-scheduled Bootstrapping): Compute the next layer, then bootstrap immediately to prepare subsequent layers.

• Formal transition compositions for these actions are provided Equation (21).

4.1.3. Multi-Objective Reward Function

The reward function $\mathcal{R}(s_i,a_i)$ serves as the sole signal to guide the agent’s learning. To balance the two primary objectives of efficiency and security, we design a decomposed multi-objective function, formally defined as: $\mathcal{R}(s_i,a_i)=w_{\mathrm{eff}}·{\mathcal{R}}_{\mathrm{eff}}(s_i,a_i)+w_{\sec }·{\mathcal{R}}_{\sec }\left(s_i\right)+w_{\mathrm{comp}}·{\mathcal{R}}_{\mathrm{comp}}\left(s_i\right)-P_{\mathrm{penalty}}(s_i,a_i)$ where $w_{\mathrm{eff}},w_{\sec },w_{\mathrm{comp}}>0$ are adjustable weight coefficients satisfying a normalization constraint, and $w_{\mathrm{eff}}+w_{\sec }+w_{\mathrm{comp}}=1$.

Efficiency Reward ($R_{\mathrm{eff}}$): This component incentivizes the agent to minimize the number of bootstrappings and to fully utilize the budget provided by each operation. It is expressed as: ${\mathcal{R}}_{\mathrm{eff}}(s_i,a_i)=r_{\mathrm{util}}\left(s_i\right)-\alpha ·\mathbb{I}[a_i\in \{a_1,a_2\}]-\beta ·\mathbb{I}[a_i=a_1\wedge r_{\mathrm{next}}\left(s_i\right)<{\tau }_{\mathrm{next}}]$ This function incorporates three mechanisms. The first term rewards the full utilization of the budget via the budget utilization $r_{\mathrm{util}}$. The second term imposes a fixed cost $\alpha$ on any bootstrapping action ($a_1$ or $a_2$), reflecting the computational cost of the $\mathsf{Boot}$ operation. The third term introduces an additional penalty $\beta$ for unnecessary immediate bootstrapping (i.e., selecting $a_1$ when the immediate next layer’s noise consumption $r_{\mathrm{next}}$ is below a threshold ${\tau }_{\mathrm{next}}$).

Security Reward ($R_{\sec }$): This component guides the agent to maintain a safe noise level through a soft constraint, preventing it from approaching the failure boundary:

$$\begin{array}{c}\hfill {\mathcal{R}}_{\sec }\left(s_i\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}{r}_{\mathrm{budget}}\left(s_i\right)<{\theta }_{\mathrm{safe}}\hfill \\ exp\left(-\kappa ·{\displaystyle \frac{r_{\mathrm{budget}}\left(s_i\right)-{\theta }_{\mathrm{safe}}}{1-{\theta }_{\mathrm{safe}}}}\right)\hfill & \mathrm{if}{\theta }_{\mathrm{safe}}\le r_{\mathrm{budget}}\left(s_i\right)<{\theta }_{\mathrm{danger}}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(13)

Here, ${\theta }_{\mathrm{safe}}$ is the safe threshold, ${\theta }_{\mathrm{danger}}$ is the danger threshold, and $\kappa >0$ controls the decay rate. This function creates three reward zones: a “safe zone” providing full reward, a “transition zone” where the reward decays exponentially, and a “danger zone” (beyond ${\theta }_{\mathrm{danger}}$) yielding zero reward. This structure guides the agent to maintain a safety margin. As shown in Equation (13), the security reward is defined piecewise over the safe, transition, and danger regions to enforce conservative behavior near the failure boundary.

Completion Reward ($R_{\mathrm{comp}}$): This reward is granted only upon the successful completion of the entire inference (i.e., at $i=L$ without any layer failing). Its magnitude is correlated with resource utilization efficiency:

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{comp}}\left(s_L\right)=\left\{\begin{array}{cc}{R}_{\mathrm{base}}·\left(1+{\displaystyle \frac{K_{\mathrm{baseline}}-K_{\mathrm{actual}}}{K_{\mathrm{baseline}}}}\right)\hfill & \mathrm{if}\mathrm{success}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(14)

where $R_{\mathrm{base}}$ is the base completion reward, $K_{\mathrm{baseline}}$ is the average number of bootstrappings for a baseline policy (e.g., a fixed-threshold strategy), and $K_{\mathrm{actual}}$ is the actual number of bootstrappings for the current trajectory. This design incentivizes the agent to find strategies that are more economical (i.e., use fewer bootstrappings) than the baseline. Equation (14) formalizes this completion reward, scaling the base reward by the efficiency gain over a baseline policy.

Penalty Term ($P_{\mathrm{penalty}}$): This term imposes a strong penalty on actions that lead to computation failure, thereby rapidly eliminating hazardous policies:

$$\begin{array}{c}\hfill P_{\mathrm{penalty}}(s_i,a_i)=\left\{\begin{array}{cc}{P}_{\mathrm{fail}}\hfill & \mathrm{if}\parallel e_{i+1}{\parallel }_{\infty }\ge B_{\max }\hfill \\ P_{\mathrm{risk}}·{(r_{\mathrm{budget}}\left(s_{i+1}\right)-{\theta }_{\mathrm{danger}})}_{+}\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(15)

where $P_{\mathrm{fail}}$ is a large failure penalty, far exceeding any positive reward, and ${\left(x\right)}_{+}=max(0,x)$. $P_{\mathrm{risk}}$ applies a progressive penalty for any action that causes the subsequent state $s_{i+1}$ to enter the danger zone ($r_{\mathrm{budget}}>{\theta }_{\mathrm{danger}}$). As specified in Equation (15), the penalty term sharply discourages trajectories that violate safety constraints or approach hazardous regions.

4.2. Policy Optimization Based on Deep Q-Learning

We employ the Deep Q-Network (DQN) approach to solve the defined MDP problem. This method approximates the optimal action-value function $Q^{*}(s,a)$ using a deep neural network $Q_{\theta }:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$. The optimal action-value function is defined as:

$$\begin{array}{c}\hfill Q^{*}(s,a)=\mathbb{E}\left[\sum _{t=0}^{H-1}{\gamma }^t\mathcal{R}(s_t,a_t)\mid s_0=s,a_0=a,\pi ={\pi }^{*}\right]\end{array}$$

(16)

This function represents the expected cumulative return obtained by taking action a in state s and subsequently following the optimal policy ${\pi }^{*}$ until the end of the task horizon. Equation (16) formalizes this optimal return under the policy ${\pi }^{*}$.

4.2.1. Value Network Architecture

We construct a fully connected neural network $Q_{\theta }:{\mathbb{R}}^{d_s}\to {\mathbb{R}}^{\left|\mathcal{A}\right|}$, with its architecture defined as:

$$\begin{array}{c}\hfill Q_{\theta }\left(s\right)=W^{\left(3\right)}\sigma (W^{\left(2\right)}\sigma (W^{\left(1\right)}s+b^{\left(1\right)})+b^{\left(2\right)})+b^{\left(3\right)}\end{array}$$

(17)

where $\theta ={\{W^{\left(l\right)},b^{\left(l\right)}\}}_{l=1}^3$ are the learnable parameters, and $\sigma (·)$ is the ReLU activation function, $\sigma \left(x\right)=max(0,x)$. The network contains two hidden layers, with dimensions $h_1=128$ and $h_2=64$, respectively. The output layer has a dimension of $\left|\mathcal{A}\right|=3$. This architectural design accounts for the small input dimension $d_s$; the decreasing dimensions of the hidden layers form an information bottleneck, which promotes feature abstraction. See Equation (17) for the explicit forward mapping of the value network.

4.2.2. Temporal Difference Learning and Bellman Optimality

The core of Q-learning utilizes the Bellman optimality equation to iteratively update value estimates. For a deterministic MDP, the Bellman equation is:

$$\begin{array}{c}\hfill Q^{*}(s,a)=\mathcal{R}(s,a)+\gamma \underset{a^{\prime }}{max}{Q}^{*}(s^{\prime },a^{\prime })\end{array}$$

(18)

where $s^{\prime }=\mathcal{T}(s,a)$ is the next state following the deterministic transition. For an observed transition $(s,a,r,s^{\prime })$, the Temporal Difference (TD) target is defined as: $y=r+\gamma {max}_{a^{\prime }\in \mathcal{A}}{Q}_{\widehat{\theta }}(s^{\prime },a^{\prime })$. Here, $Q_{\widehat{\theta }}$ denotes the target network, whose parameters $\widehat{\theta }$ are periodically copied from the online network $\theta$ (i.e., $\widehat{\theta }\leftarrow \theta$ every C steps). This mechanism addresses the “moving target problem” in bootstrapped updates and significantly improves training stability. The network parameters $\theta$ are updated by minimizing the expected Huber loss on the TD error: $\mathcal{L}\left(\theta \right)={\mathbb{E}}_{(s,a,r,s^{\prime })\sim \mathcal{D}}\left[{\ell }_{\mathrm{Huber}}\left(\delta \right)\right]$, where $\delta =y-Q_{\theta }(s,a)$ is the TD error, and $\mathcal{D}$ is the experience replay buffer. The Huber loss is defined as

$${\ell }_{\mathrm{Huber}}\left(\delta \right)=\left\{\begin{array}{cc}{\displaystyle \frac{1}{2}}{\delta }^2,\hfill & \mathrm{if}\left|\delta \right|\le c\hfill \\ c(|\delta |-{\displaystyle \frac{1}{2}}c),\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(19)

with smoothing threshold $c>0$, which improves robustness to outliers compared to a pure squared loss.

4.2.3. Experience Replay and Exploration Strategy

The training objective of DQN is to minimize a TD loss function. To enhance training stability, we utilize experience replay and an exploration strategy.

Experience Replay: A circular buffer $\mathcal{D}={\left\{(s_j,a_j,r_j,s_j^{\prime },d_j)\right\}}_{j=1}^N$ of fixed capacity is maintained, where $N={10}^5$ is the buffer size and $d_j\in \{0,1\}$ is a termination flag. At each training step, the agent stores the new transition $(s_t,a_t,r_t,s_{t+1},d_t)$ into $\mathcal{D}$. When the buffer is full, the oldest sample is discarded using a First-In, First-Out (FIFO) policy. During training, a mini-batch $\mathcal{B}\subset \mathcal{D}$ of size $\left|\mathcal{B}\right|=B=32$ is sampled uniformly at random from $\mathcal{D}$.

$ϵ$-Greedy Exploration Strategy: To balance exploration and exploitation, an adaptive $ϵ$-greedy strategy is adopted. At step t, the policy selects an action according to:

$$\begin{array}{c}\hfill a_t=\left\{\begin{array}{cc}\mathrm{random}\left(\mathcal{A}\right)\hfill & \mathrm{probability}{ϵ}_t\hfill \\ arg\underset{a\in \mathcal{A}}{max}{Q}_{\theta }(s_t,a)\hfill & \mathrm{probability}1-{ϵ}_t\hfill \end{array}\right.\end{array}$$

(20)

where $\mathrm{random}\left(\mathcal{A}\right)$ denotes the operation of sampling an action from the discrete action space $\mathcal{A}$ according to a uniform distribution $\mathcal{U}\left(\mathcal{A}\right)$, such that each action is chosen with probability $1/\left|\mathcal{A}\right|$. The exploration rate ${ϵ}_t$ decays exponentially over time. The exploration rate decays exponentially ${ϵ}_t=max\left({ϵ}_{\mathrm{end}},{ϵ}_{\mathrm{start}}·{\lambda }^{t/T_{\mathrm{decay}}}\right)$ The action selection rule is summarized in Equation (20).

4.2.4. Integration with the DN-ANM Model

The reinforcement learning agent is integrated into the DN-ANM model to form a complete perception-prediction-decision-execution loop. This enables the system to achieve intelligent noise budget management during complex homomorphic inference tasks.

(1) Construction Mechanism of the Environment State Space: The first step in reinforcement learning is to define the state space $\mathcal{S}$, which determines the information the agent possesses. First, the current noise level $\parallel e_i{\parallel }_{\infty }^{\mathrm{cur}}$ within the instantaneous state features $s_i^{\mathrm{instant}}$ is provided by the runtime noise monitor $\mathcal{M}$. In a simulated environment, $\mathcal{M}$ maintains a virtual noise state variable, which is updated after each action execution according to the DN-ANM’s noise increment function. In a real-world environment, $\mathcal{M}$ instead calls the CKKS scheme’s built-in function $\mathsf{GetNoise}\left(c{t}_i\right)$ to retrieve the actual noise level of the ciphertext. Based on this current noise level, $\mathcal{M}$ further computes derived features, such as the noise budget ratio $r_{\mathrm{budget}}={\parallel e_i\parallel }_{\infty }^{\mathrm{cur}}/B_{\max }$ and the budget utilization $r_{\mathrm{util}}$. These normalized features map the raw noise values to the $[0,1]$ interval, rendering states comparable across different HE parameter configurations and enhancing the policy’s generalization capability. The inference progress ratio $r_{\mathrm{progress}}=i/L$ and the bootstrapping distance $r_{\mathrm{distance}}$ then provide temporal context, helping the agent to understand the current stage of the inference process.

(2) State Transition Function: The DN-ANM provides a deterministic noise prediction model; consequently, the state transition is effectively deterministic and can be represented as a function $\mathcal{T}:\mathcal{S}\times \mathcal{A}\to \mathcal{S}$. The implementation of this transition function must handle the three distinct action types, as each action induces a different state evolution pattern. To clearly express this process, we define two fundamental transition operators: the computation transition operator ${\mathcal{T}}_{\mathrm{compute}}$ and the bootstrapping transition operator ${\mathcal{T}}_{\mathrm{boot}}$.

The computation transition operator ${\mathcal{T}}_{\mathrm{compute}}:\mathcal{S}\times \Theta \to \mathcal{S}$ simulates the effect of executing one HE layer on the state. This operation updates the noise level according to the DN-ANM’s noise increment model: $\parallel e_{i+1}{\parallel }_{\infty }\leftarrow \parallel e_i{\parallel }_{\infty }+\Delta {\mathcal{E}}_{i+1}^{\mathrm{pred}}(\parallel e_i{\parallel }_{\infty },W_{i+1})$ where the specific form of $\Delta {\mathcal{E}}_{i+1}^{\mathrm{pred}}$ is selected from the noise models in Section 3 based on the type of layer $i+1$. For instance, if layer $i+1$ is a $3\times 3$ convolutional layer, then $\Delta {\mathcal{E}}_{i+1}^{\mathrm{conv}}(\parallel e_i{\parallel }_{\infty },W_{i+1})={\alpha }_{i+1}·N_{\mathrm{mult}}^{(i+1)}·(\Delta ·w_{max}+1)·{\parallel e_i\parallel }_{\infty }+{\beta }_{i+1}·N_{\mathrm{mult}}^{(i+1)}$, where $N_{\mathrm{mult}}^{(i+1)}=9·C_{\mathrm{in}}$ is the number of multiplications, and $w_{max}$ is the maximum absolute value of the weights.

This state transition necessitates a re-computation of the foresight features based on the updated noise level $\left|\right|e_{i+1}{\left|\right|}_{\infty }$ and the advanced layer index $i+1$. We formally define the remaining consumption ratio ($r_{remain}$) and the foresight horizon ($r_{horizon}$) according to the following update rules:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill r_{\mathrm{next}}& \leftarrow {\displaystyle \frac{{\mathcal{P}}_{\mathrm{pred}}(\parallel e_{i+1}{\parallel }_{\infty },i+1,i+2)}{B_{\max }-{\parallel e_{i+1}\parallel }_{\infty }}}\hfill \\ \hfill r_{\mathrm{remain}}& \leftarrow {\displaystyle \frac{{\mathcal{P}}_{\mathrm{pred}}(\parallel e_{i+1}{\parallel }_{\infty },i+1,L)}{B_{\max }-{\parallel e_{i+1}\parallel }_{\infty }}}\hfill \\ \hfill r_{\mathrm{horizon}}& \leftarrow min\left\{k:{\mathcal{P}}_{\mathrm{pred}}(\parallel e_{i+1}{\parallel }_{\infty },i+1,i+1+k)\ge B_{\max }-{\parallel e_{i+1}\parallel }_{\infty }\right\}\hfill \end{array}\end{array}$$

(21)

The metric $r_{remain}$ functions as a global risk indicator. Unlike the immediate consumption ratio $r_{next}$, which restricts its assessment to the safety of the single upcoming layer, $r_{remain}$ evaluates the long-term viability of the inference task. It compares the cumulative predicted noise consumption aggregated from layer $i+1$ to the final layer L against the currently available noise budget. A value of $r_{remain}>1$ serves as a deterministic signal that the current budget is insufficient to reach the final layer without at least one future bootstrapping operation. Complementing this, $r_{horizon}$ quantifies the temporal safety window. By solving for the maximum integer k such that the cumulative noise remains within bounds, it provides the agent with an explicit countdown of safe computation steps available before a mandatory bootstrapping event is triggered. These foresight features collectively enable the agent to balance immediate safety with long-term planning.

The bootstrapping transition operator ${\mathcal{T}}_{\mathrm{boot}}:\mathcal{S}\to \mathcal{S}$ simulates the effect of executing the $\mathsf{Boot}$ operation on the state. The primary effect of bootstrapping is to reset the noise level to its initial value: $\parallel e_i{\parallel }_{\infty }\leftarrow B_{init}$ This reset operation has a cascading effect on multiple state features. First, the budget utilization $r_{\mathrm{util}}$ is reset to zero, as no budget has been consumed since the bootstrapping. The bootstrapping distance $r_{\mathrm{distance}}$ is also reset to zero, marking the start of a new noise accumulation cycle. Furthermore, due to the abrupt change in the noise level, all foresight features must be recomputed based on this new noise starting point. This implies that even at the same layer position, the state before and after bootstrapping will have entirely different foresight feature values, representing a complete reset of the noise budget.

Using these two base operators, the complete state transition function can be defined through composition. For the “Continue Computation” action $a_0$, the transition directly applies the computation operator: $\mathcal{T}(s_i,a_0)={\mathcal{T}}_{\mathrm{compute}}(s_i,W_{i+1})$ For the “Immediate Bootstrapping” action $a_1$, the transition first applies the bootstrapping operator, then the computation operator: $\mathcal{T}(s_i,a_1)={\mathcal{T}}_{\mathrm{compute}}({\mathcal{T}}_{\mathrm{boot}}\left(s_i\right),W_{i+1})$ This signifies that the noise is reset first, and the next layer’s computation is then executed with a clean budget. For the “Pre-scheduled Bootstrapping” action $a_2$, the transition first applies the computation operator, then the bootstrapping operator: $\mathcal{T}(s_i,a_2)={\mathcal{T}}_{\mathrm{boot}}\left({\mathcal{T}}_{\mathrm{compute}}(s_i,W_{i+1})\right)$ This represents executing the next layer’s computation using the current budget, and then immediately resetting the noise in preparation for subsequent layers. This strategy of deferred bootstrapping can be superior to immediate bootstrapping under certain conditions, as it avoids budget waste.

4.3. Algorithm Implementation

We integrate the aforementioned components into a complete training pipeline, as shown in Algorithm 1 In each training episode, the agent interacts with the simulated homomorphic inference environment according to the current policy. The data generated during this interaction is stored in the experience replay buffer, and the Q-network is updated by minimizing the TD loss. In a practical homomorphic inference task, the algorithmic procedure is shown in Algorithm 2.

Algorithm 1 DQN Training
Require: Network $\mathcal{N}$, Noise model ${\mathcal{P}}_{\mathrm{pred}}$, Episodes $E_{max}$
Ensure: Trained Q-network parameters ${\theta }^{*}$
1: Initialize Q-networks $\theta$, $\widehat{\theta }\leftarrow \theta$, replay buffer D, $\mathrm{step\_count}\leftarrow 0$
2: $ϵ\leftarrow {ϵ}_{\mathrm{start}}$
3: for  $\mathrm{episode}=1\mathrm{to}{E}_{max}$  do
4:        s ← InitialState$(G,B_{init})$; $K_{\mathrm{episode}}\leftarrow 0$; $\mathrm{done}\leftarrow \mathbf{false}$
5:        while not done do
6:              if Random < $ϵ$ then
7:                    a ← UniformSample$\left(\mathcal{A}\right)$
8:              else
9:                    $a\leftarrow arg{max}_{a^{\prime }\in \mathcal{A}}{Q}_{\theta }(s,a^{\prime })$
10:              end if
11:              $s^{\prime }\leftarrow \mathcal{T}(s,a)$; $r\leftarrow \mathcal{R}(s,a)$	▹ $\mathcal{T}$ is the DN-ANM simulation
12:              if $a\in \{a_1,a_2\}$; then $K_{\mathrm{episode}}\leftarrow K_{\mathrm{episode}}+1$
13:              end if
14:	▹ Check environment termination conditions
15:              if $s^{\prime }.\mathrm{layer\_index}=L$ then
16:                    $\mathrm{done}\leftarrow \mathbf{true}$; $r\leftarrow r+R_{\mathrm{comp}}\left(K_{\mathrm{episode}}\right)$
17:              else if $s^{\prime }.\mathrm{noise\_budget}\ge B_{max}$ then
18:                    $\mathrm{done}\leftarrow \mathbf{true}$; $r\leftarrow r-P_{\mathrm{fail}}$
19:              end if
20:              D. Store$\mathrm{\left(}(s,a,r,s^{\prime },\mathrm{done})\right)$
21:              $s\leftarrow s^{\prime }$; $\mathrm{step\_count}\leftarrow \mathrm{step\_count}+1$
22:              if $\left|D\right|\ge B$ then
23:                    UpdateNetworks$(D,\theta ,\widehat{\theta },\mathrm{step\_count})$	▹ Target-network updates with Huber loss
24:              end if
25:        end while
26:        $ϵ\leftarrow max({ϵ}_{\mathrm{end}},ϵ·{\lambda }^{(1/T_{\mathrm{decay}})})$	▹ Decay $ϵ$
27: end for
28: return  ${\theta }^{*}$

Algorithm 2 Online Adaptive Bootstrapping Decision-Making
Require: Input ciphertext $c{t}_0$, Network $\mathcal{N}$, Trained Q-network $Q_{{\theta }^{*}}$, DN-ANM model $(\mathcal{A},\mathcal{P},\mathcal{M})$
Ensure: Inference result $c{t}_L$, Total bootstrapping count K
1: G ← AnalyzeNetwork$\left(\mathcal{N}\right)$	▹ Offline pre-processing: Analyze network
2: n0 ← GetNoise$\left(c{t}_0\right)$	▹ Get initial noise
3: s ←InitializeState $(G,n_0)$	▹ Initialize state
4: $i\leftarrow 0,K\leftarrow 0$
5: while  $i<L$  do
6:       $a\leftarrow arg{max}_{a^{\prime }\in \mathcal{A}}{Q}_{{\theta }^{*}}(s,a^{\prime })$	▹ Greedy action selection
7:       if $a=a_1$ then	▹ Immediate bootstrap
8:              cti ← Boot$\left(c{t}_i\right)$
9:              $K\leftarrow K+1$
10:       end if
11:       $c{t}_{i+1}\leftarrow f_{i+1}(c{t}_i,W_{i+1})$	▹ Execute homomorphic computation
12:       $i\leftarrow i+1$
13:       if $a=a_2$ then	▹ Pre-scheduled bootstrap
14:              cti ← Boot$\left(c{t}_i\right)$	▹ Bootstrap the result $c{t}_i$ (now $c{t}_{i+1}$ logical)
15:              $K\leftarrow K+1$
16:       end if
17:       ni ← GetNoise$\left(c{t}_i\right)$	▹ Update noise information
18:       s ← $\mathcal{M}.$UpdateState$(n_i,i,G,\mathcal{P})$	▹ Update state
19: end while
20: return  $c{t}_L,K$

By organically integrating the noise perception capabilities from Section 3 with the adaptive decision-making capabilities from this section, we have constructed an end-to-end intelligent noise management system. This system provides a theoretically sound solution for the efficient homomorphic inference of deep neural networks.

Hyperparameter Configuration and Sensitivity Analysis

To facilitate reproducibility and provide a complete specification of the learning environment, we summarize the key hyperparameters used for training the DN-ANM agent in

Table 3. DN-ANM Hyperparameter Configuration (Empirically Optimized).

Category	Parameter	Value
Training Settings	Episodes ($E_{max}$)	2000
	Replay Buffer Size (N)	${10}^5$
	Batch Size (B)	32
	Learning Rate ($\eta$)	$1\times {10}^{-4}$
	Discount Factor ($\gamma$)	0.99
Reward Weights	Efficiency ($w_{eff}$)	0.4
	Security ($w_{sec}$)	0.4
	Completion ($w_{comp}$)	0.2
Thresholds	Safe Threshold (${\theta }_{safe}$)	0.80
	Danger Threshold (${\theta }_{danger}$)	0.95
	Lookahead Threshold (${\tau }_{next}$)	0.15
Penalties	Bootstrapping Cost ($\alpha$)	0.1
	Premature Boot. ($\beta$)	0.2

. As these parameters are not intrinsic to the theoretical model, their specific values were determined through an empirical grid search to optimize the stability of the Deep Q-Network. The discount factor is set to $\gamma =0.99$ to ensure the agent prioritizes long-term safety over immediate computational gains, which is critical given the cumulative nature of noise growth. The reward weights were balanced ($w_{eff}=w_{sec}=0.4$) to prevent policy collapse; we observed that unbalanced weights led to either reckless behavior or overly conservative strategies. The safety thresholds were calibrated based on the error margin of noise predictor, ${\theta }_{danger}$ was set to 0.95 rather than 1.0 to create a 5% buffer zone, accommodating potential worst-case deviations in the layer-wise noise prediction $\Delta {\mathcal{E}}^{pred}$.

Sensitivity Analysis: The performance of agent is primarily sensitive to the balance between $w_{eff}$ and $w_{sec}$. We observed that setting $w_{eff}\gg w_{sec}$ encourages the agent to delay bootstrapping until the noise budget is critically low, increasing the risk of decryption failure near the danger threshold (${\theta }_{danger}$). Conversely, dominant security weights ($w_{sec}>0.6$) lead to conservative policies with redundant bootstrapping operations, approximating static fixed-interval strategies. The safety threshold ${\theta }_{safe}$ determines the “alert zone”; lowering ${\theta }_{safe}$ below 0.7 triggers earlier interventions but reduces budget utilization efficiency ($r_{util}$). The chosen configuration ($w_{eff}=w_{sec}=0.4$) enables the agent to maximize budget usage while maintaining a 5% safety margin (${\theta }_{danger}=0.95$) against calculation errors.

5. Experimental Analysis

All experiments in this study were executed on a PC equipped with an Intel Ultra 7 258V 8-core processor and 32GB of RAM. The operating system was Windows 11, and the experimental environment was based on Python 3.12.3.

5.1. Network Architectures and Evaluation Metrics

To comprehensively evaluate the performance and generalization capability of the proposed DN-ANM mechanism, we employed two representative Convolutional Neural Network (CNN) architectures: LeNet-5 and ResNet-20. These models differ significantly in depth and computational patterns, enabling a robust evaluation of efficiency across varying complexities. We uniformly adopted $f\left(x\right)=x^2$ as the activation function to maintain compatibility with HE constraints.

To ensure the experimental results are transferable to practical privacy-preserving deployments, we configured the CKKS scheme to target a standard security level of $\lambda \ge 128$-bit. We selected a polynomial ring dimension of $N=16,384$, according to the HE security standard, supports a maximum modulus capacity sufficient for the depths of the tested networks while maintaining resistance against known lattice attacks (e.g., primal and dual hybrid attacks). Specifically, we utilized a coefficient modulus chain $Q=\{q_0,\dots ,q_L\}$ with bit-sizes $[60,30,\dots ,30,60]$. The initial and special moduli are set to 60 bits to preserve precision during key switching, while intermediate moduli are set to 30 bits to match the scaling factor $\Delta =2^{30}$. This configuration ($logQ\approx 210$ bits) strictly adheres to the security threshold for $N=16,384$, balancing computational efficiency with 128-bit security guarantees.

We compare the proposed DN-ANM with a representative baseline method, PyCrCNN [11], which exemplifies a framework adopting a static noise management strategy. It adheres to a fixed bootstrapping policy, triggering bootstrapping at predefined layer intervals, and lacks the capability for dynamic awareness of the network’s real-time state. Consequently, our evaluation primarily focuses on the number of bootstrapping operations, model accuracy, and parameter robustness.

5.2. Experimental Results and Analysis

5.2.1. Noise Prediction Effectiveness

We evaluate the accuracy of the DN-ANM noise prediction under the CKKS scheme. The encryption parameters are set as: polynomial dimension $n=16,384$, scaling factor $\Delta =2^{30}$, and coefficient modulus chain $q{i}_{sizes}=[60,30,30,30,60]$. The test network includes two convolutional layers (with 8 $3\times 3$ filters and 16 $5\times 5$ filters, respectively) and two fully-connected layers (with 64 and 10 output units). We compare DN-ANM against two methods: Worst-Case Static Analysis and Linear Extrapolation (LE). DN-ANM, the predictor for deep network-aware noise management, combines theoretical structures with empirical parameters to perform calibrated modeling of the noise increments for network layers such as convolution, fully-connected, and square (activation). Its objective is to estimate the noise consumption of each layer and maintain trend alignment. Worst-Case Static Analysis is a static estimation guided by worst-case upper bounds, which typically overestimates the noise and serves as a conservative baseline. Linear Extrapolation uses the average increment of the first few layers to perform linear extrapolation; it can track partial trends but is prone to deviation when encountering non-linear network layers and parameter changes.

The actual measured noise(MN) of each layer is obtained by calculating the difference between the decrypted value and the corresponding plaintext reference value, and is compared against the layer-wise predicted values from each method. As can be seen in

Table 4. Comparison of the performance of layer noise measurement and prediction methods.

Type	Params	MN $\left({10}^{-7}\right)$	DN-ANM $\left({10}^{-7}\right)$	Worst-Case	LE $\left({10}^{-7}\right)$
Conv2d	IC = 1, OC = 8	3.7559	7.0809	$3.53\times {10}^9$	9.1703
Conv2d	IC = 8, OC = 16	4.2449	7.6795	$8.21\times {10}^{10}$	1.1921
Square	—	8.1627	7.6811	$8.21\times {10}^{10}$	2.1458
Linear	In = 64, Out = 64	3.7899	7.8763	$1.06\times {10}^{11}$	3.1472
Linear	In = 64, Out = 10	3.5029	8.0765	$1.29\times {10}^{11}$	4.6159

, the proposed DN-ANM outperforms the baseline methods in both noise estimation accuracy and trend alignment. Its predictions adapt well to changes in convolutional kernel size, channel count, and the input dimensions of fully-connected layers, while maintaining profile stability after non-linear activations.

5.2.2. Bootstrapping Efficiency Analysis

Bootstrapping operations constitute the primary computational bottleneck in HE inference; minimizing their frequency is critical for performance. Figure 2a contrasts the bootstrapping overhead of DN-ANM against the static PyCrCNN baseline across varying network scales. While PyCrCNN exhibits a rigid, linear increase in overhead (fixed at 17 and 22 operations for ResNet-20 Narrow and Deep, respectively), DN-ANM dynamically optimizes the schedule, reducing overhead by approximately 47% to 65%. Figure 2b details the epoch-wise fluctuations during RL agent training. Unlike the flatline static schedule, DN-ANM actively explores the state space, converging to a minimal-cost strategy (approx. 6 operations for ResNet-20 Narrow) by the 7th epoch, validating the capability of RL agent to identify globally optimal execution paths.

5.2.3. Model Training Accuracy

Excessive bootstrapping not only incurs latency but also accumulates numerical precision loss, effectively adding noise to the gradient updates. Figure 3 illustrates the test accuracy for ResNet-20 over 80 training epochs. DN-ANM demonstrates superior convergence characteristics, reaching 80% accuracy by epoch 25—ten epochs earlier than the baseline. Furthermore, the model stabilizes at a higher peak accuracy (95–96% vs. 91–92%). This 4–5% performance gap is directly attributable to the reduced frequency of bootstrapping operations, which preserves the fidelity of the ciphertext data and mitigates the cumulative error inherent in the refresh procedure.

5.2.4. Parameter Robustness

To validate that the effectiveness of the DN-ANM mechanism is not confined to a specific parameter configuration; we further evaluate its performance robustness across different combinations of critical HE hyperparameters. The initial noise budget and the security threshold are two core variables influencing HE performance. Figure 4 presents the optimal average reduction in bootstrapping operations achieved by DN-ANM compared to PyCrCNN, under various initial budget settings. Here, optimal refers to the optimal security threshold configuration adaptively identified by the DN-ANM’s agent for a given budget. The experimental data reveals that DN-ANM demonstrates strong robustness across a broad parameter space. Whether in budget-constrained ($Budget=30$) or relatively ample-budget ($Budget=60$) scenarios, DN-ANM consistently delivers significant performance gains, achieving an average reduction of 14 to 17 bootstrapping operations. Particularly noteworthy is that DN-ANM’s adaptability is manifested in its selection of the optimal threshold for different budgets. For instance, the optimal threshold is 5 when $Budget=40$, whereas it is 7 when $Budget=50$. This demonstrates that the DN-ANM’s agent is capable of perceiving changes in HE parameters and dynamically adjusting its decision policy to match the current constraints, thereby finding a near-optimal execution path for any given parameter configuration.

5.2.5. Ablation Study

Table 5. Performance Comparison of Different Configurations (Ablation Study).

Configuration	Avg Reenc.	Violations	Safe Rate(%)	Adjusted Reenc. *
Full DN-ANM	18.67	0	76.2	19.08
No Foresight	18.97	0.16	52.4	19.75
Binary Action	5.85	0.24	38.1	7.06
No Completion Reward	18.40	0	76.2	18.82

* The number of re-encryption operations after introducing penalties for violations.

presents the ablation study results across 21 experimental configurations, evaluated on the ResNet-20 wide architecture under varying HE constraints (noise budgets $B\in \{40,50,60\}$ and thresholds ${\theta }_{safe}\in \{5,6,8\}$). The key finding is that the ternary action space contributes primarily to security rather than efficiency. While the binary action variant achieves fewer raw re-encryptions (5.85 vs 18.67), it does so at the cost of a 38.1% reduction in safe completion rate (76.2% → 38.1%). Unlike raw re-encryption counts, the Violations column quantifies how often the ciphertext noise budget falls below the security threshold. Such events render the inference invalid, making configurations with non-zero violations impractical for production HE deployments. The foresight features improve safe completion rate by 23.8% by anticipating future noise consumption and triggering preemptive re-encryptions. The completion reward mechanism maintains stable RL training convergence with minimal impact on final performance.

6. Conclusions

To address the limitation of existing HE schemes in Deep Neural Network (DNN) inference, which employ fixed noise strategies and thus fail to adapt to the heterogeneous computational characteristics across network layers, this paper proposes the DN-ANM. The core of this mechanism involves constructing a layer-aware noise consumption prediction model and designing an optimization algorithm for bootstrapping decisions driven by reinforcement learning. This approach enables DN-ANM to dynamically and precisely allocate the noise budget, utilizing a Deep Q-Network (DQN) to learn the globally optimal timing for bootstrapping, thereby minimizing overhead. We conducted exhaustive experimental validation on two typical deep networks (LeNet-5 and ResNet-20). The experimental results demonstrate that the proposed DN-ANM can dynamically optimize its management strategy based on the network’s computational characteristics. Compared to the fixed strategy of PyCrCNN, its adaptive nature yields significant efficiency improvements. Furthermore, by minimizing the bootstrapping operations that induce precision loss, DN-ANM preserves the fidelity of the ciphertext data to the greatest extent, thereby accelerating the convergence of the learned bootstrapping policy and improving the stability of encrypted inference accuracy. In conclusion, DN-ANM addresses the limitations of static strategies through awareness and adaptive decision-making, offering a novel approach to balancing efficiency and precision in complex privacy-preserving computations. Future work will focus on deploying this mechanism from the simulated environment into a practical HE cryptography library to validate its real-world application performance. Further work will explore the use of Graph Neural Networks (GNNs) to characterize the computational graphs of arbitrary DNNs, enabling the Reinforcement Learning agent to generalize to more complex architectures such as Transformers.