Layerwise Conditioned Backpropagation: A Curvature-Aware Reparameterization of the Backward Pass with Convergence Guarantees

Abstract

Backpropagation is less a single algorithm than a pipeline of choices: how the error signal is propagated, how the weight gradient is assembled, and how the update is applied. This paper revisits three consecutive steps and proposes small, mathematically transparent modifications that improve gradient scaling and conditioning without changing the represented function class. The resulting method, Conditioned Backpropagation(CBP), combines (i) a layerwise gradient-norm equalization that counters the geometric depth dependence of the backpropagated error; (ii) an activation-centering reparameterization that removes the dominant rank-one mean term from the per-layer curvature; and (iii) a damped diagonal preconditioner that is positive-definite by construction. The composite operator is a bounded positive-definite preconditioner, so the method inherits standard nonconvex, Polyak–Łojasiewicz, and stochastic convergence guarantees at the per-step cost of ordinary backpropagation. No prior method composes these three repairs into one operator with a joint boundedness and positive-definiteness guarantee. Two further results, both new, concern equalization. On a block-structured strongly convex model, and for the curvature-equalizing target that the implemented gradient-energy equalizer approximates up to a quantified heterogeneity factor, equalization makes the convergence rate depth-uniform; the bounded-clip version that is actually run stays depth-uniform up to a clip-determined depth and retains a constant-factor improvement beyond it. Controlled experiments, run over ten or more seeds with paired significance tests, confirm the mechanisms: Equalization compresses an order-of-magnitude per-layer gradient disparity, centering cuts the top curvature eigenvalue about threefold and yields the lowest training loss, and the configurations combining centering with the damped preconditioner, including the full method, converge fastest. The effects persist on MNIST and on CIFAR-10 with a small residual convolutional network, at a measured per-iteration overhead below about twice that of Adam. Generalization is comparable across methods, and no end-to-end depth-scaling advantage is claimed, keeping the contribution focused on optimization geometry.

1. Introduction

The backpropagation algorithm is so familiar that it is easy to forget how many distinct decisions it bundles together. At its core, it is a single application of the chain rule, but in practice, every implementation must also decide how the error signal is scaled as it traverses the layers, how that signal is combined with stored activations to form each weight gradient, and how the resulting gradient is turned into a parameter update. These are separate steps, and each of them shapes the geometry of the optimization problem that the learner actually faces. A large fraction of the engineering folklore of deep learning, including initialization rules, normalization layers, adaptive optimizers, gradient clipping, and learning-rate warmup, can be read as attempts to repair one or another of these steps so that the gradient signal is well scaled, stable, and informative [1,2,3].

Rather than proposing yet another monolithic optimizer, this paper isolates three consecutive steps of the backward pass and asks, for each, what minimal mathematical change improves the conditioning of the gradient while leaving the function computed by the network unchanged. The three steps and their corresponding modifications are as follows.

M1.

Error propagation. The backpropagated error has a second moment that grows or decays geometrically with depth; this is the classical vanishing/exploding-gradient phenomenon. Each layer’s gradient block is rescaled by a positive scalar estimated online so that all blocks carry a common second moment, a step referred to as layerwise gradient-norm equalization.

M2.

Gradient assembly. The weight gradient is an outer product of the error signal with the incoming activation. When the activation has a large mean (as rectified units do), this outer product inherits a dominant rank-one term that inflates the local curvature. Centering the incoming activation, an exact reparameterization, removes that rank-one term.

M3.

Update. A diagonal preconditioner built from a running estimate of the squared gradient is applied, regularized by a damping term that guarantees positive definiteness and bounded eigenvalues.

Individually, each ingredient is closely related to ideas that already exist in the literature: signal-propagation analyses of initialization [4,5], mean-removal and normalization schemes [1,6], and adaptive or curvature-aware optimizers [7,8]. The contribution here is to present them as three commuting operations on the backward pass, to give a unified and self-contained analysis showing that the combined operator is a bounded positive-definite preconditioner, and therefore a legitimate descent operator to which standard convergence guarantees apply, and to verify, in controlled experiments, the specific mechanisms the analysis predicts. The general convergence theorems themselves are not new: they are the textbook guarantees for preconditioned (stochastic) gradient descent with a bounded positive-definite preconditioner. The novelty lies in constructing such a preconditioner cheaply and provably from three interpretable steps and, for the equalization step, specifically, in a depth-uniform linear rate (Theorem 4) that the generic bounded-preconditioner results do not capture, since it depends on the block-diagonal structure M1 imposes, together with an analysis of the clipped equalizer that is actually run (Theorem 5), which turns the clip from an unanalyzed implementation detail into a sharp statement of the depth at which depth-uniformity is retained.

Stated explicitly, the novelty of this work is threefold. First, although each ingredient descends from a known family (depth-aware rescaling, mean removal, and diagonal adaptive scaling), no prior work composes the three as commuting operations on the backward pass and proves that the composite is a single bounded positive-definite preconditioner. Existing approaches modify the architecture (normalization), the objective (sharpness-aware training), or the update rule in isolation (adaptive optimizers), and not one supplies a joint guarantee for the assembled pipeline. Second, two of the theoretical results are new in themselves rather than inherited: the depth-uniform linear rate of Theorem 4, which converts a depth-dependent condition number into a depth-independent one on block-structured objectives, and the direct analysis of the clipped equalizer (Theorem 5), which to the best of the author’s knowledge is the first to treat the clipping that is actually run as an object of analysis and to derive from it a sharp crossover depth $L^{★}$. Third, the experimental design is mechanism-level rather than leaderboard-level: Every theoretical claim is paired with a falsifiable measurement (the per-layer gradient disparity, the alignment of the activation mean with the top curvature eigenvector, and depth sweeps on the analyzed model problem), a methodological stance that large-scale benchmarking studies have argued is precisely what optimizer research lacks [3,9].

The specific contributions are as follows. First, Conditioned Backpropagation (CBP) is formalized as the composition of three modifications to consecutive backpropagation steps (Section 4), and it is shown to be implementable with $O(P+\mathcal{A})$ extra work per iteration, where P is the number of parameters and $\mathcal{A}$ the number of activations; for dense layers this is the same order as a single backward pass, although for layer types in which the activation count exceeds the parameter count, the activation term dominates. Second, the analysis (Section 5) establishes that (a) the composite update is a descent direction because the induced preconditioner is positive definite (Proposition 1); (b) equalization makes the expected per-layer gradient second moment independent of depth (Proposition 2); (c) centering reduces the conditioning of the per-layer curvature factor in the aligned-mean regime characteristic of rectified activations (Proposition 3); and (d) the resulting bounded preconditioned scheme inherits the standard $O(1/T)$ nonconvex rate, linear convergence under a local Polyak–Łojasiewicz condition, and the $O(1/\sqrt{T})$ stochastic rate (Theorems 1–3); and (e) going beyond these inherited guarantees, on a block-structured strongly convex objective and for the curvature-equalizing target, the equalization step converts the depth-dependent global condition number into a depth-uniform one, yielding a linear rate whose constant does not grow with depth (Theorem 4); the gradient-energy equalizer that is actually run matches that target only up to a heterogeneity factor $\omega$, which the analysis quantifies and the experiments bound empirically, and the rate comes with an explicit exponential-in-depth separation from single-step-size descent on a model problem (Proposition 5); and (f) the bounded-clip equalizer that is actually implemented is analyzed directly: It stays depth-uniform up to a sharp clip-determined depth $L^{★}=\log R/\log \overline{\kappa }$ and retains a constant factor-R improvement of the condition number beyond it, where R is the clip range (Theorem 5, Corollary 1), a prediction that the measured disparity profile confirms. Third, each mechanism is corroborated with reproducible experiments run over ten or more seeds, with confidence intervals and paired significance tests (Section 6): a depth-24 network in which M1 compresses an order-of-magnitude per-layer gradient disparity toward parity; a measurement confirming that the mean is almost collinear with the top eigenvector ($\cos \approx 0.99$) and that centering cuts the top curvature eigenvalue by a median factor of about three; a head-to-head comparison with a common-base factorial ablation, indicating that centering is the modification most responsible for the lowest training loss, that the damped preconditioner supplies the speed, and that the configurations combining the two converge fastest; a three-part depth study that confirms the depth-uniform rate of Theorem 4 exactly on the block-structured model problem (single-step-size descent grows geometrically with depth, while the equalized scheme stays constant), confirms in networks that the per-layer disparity driving the rate grows geometrically with depth, and adds a training-level study that delineates where the gradient-scale mechanism governs end-to-end training and identifies the complementary forward-signal component whose treatment is charted as future work; and a real-data evaluation on MNIST and CIFAR-10, the latter with a small residual convolutional network, in which the optimization effects persist and the measured wall-clock overhead stays modest. The scope of the evidence is stated as plainly as the findings: Generalization is comparable across methods, and the practical clipped form of M1 removes a bounded, clip-determined factor of that disparity at extreme depth, with the remaining mapped out as future work.

It is worth being explicit about what is not claimed. CBP is not a new state-of-the-art optimizer, and on shallow, easy problems, it is essentially tied with well-tuned baselines. No generalization improvement is claimed, and while the depth-uniform rate is confirmed on the model problem it analyzes, no end-to-end depth-scaling advantage in generalization is claimed for deep networks; the path toward one is charted in Section 8.2. The point of the paper is mathematical: to show that three inexpensive, individually justifiable edits to the backward pass compose into a single object with clean guarantees, and that the place they demonstrably help is optimization geometry, where centering drives the lowest training loss and the damped preconditioner the fastest convergence in the ablations reported below.

The remainder of the paper is organized as follows. Section 2 fixes notation and recalls the backpropagation pipeline, identifying the two pathologies (depth-dependent scale and poor curvature conditioning) that motivate the method. Section 3 situates the three modifications within the existing literature on optimizers, curvature-aware methods, normalization, and loss landscapes. Section 4 defines the three modifications precisely and assembles them into a single algorithm, illustrated by the schematic in Figure 1 and the pseudocode in Algorithm 1.

Section 5 is the analytical core: It shows that the induced preconditioner is bounded and positive definite, analyzes the two structural modifications in turn, records the convergence guarantees (including the depth-uniform rate for the equalization step and the rate of the clipped equalizer that is actually run, both of which go beyond the inherited bounds), and bounds the computational overhead. Section 6 reports a sequence of controlled experiments, each probing one part of the analysis: the two mechanism studies, the head-to-head comparison with its common-base factorial ablation and hyperparameter sensitivity analysis, a three-part depth study covering the model problem, the geometric premise in networks, and the predicted clip-determined crossover together with a training-level study of its end-to-end scope, and a real-data evaluation on MNIST and CIFAR-10 with measured wall-clock overhead. Section 7 discusses scope, the relationship to normalization and adaptive methods, the price of the boundedness guarantee (where the clipped-equalizer rate is interpreted), and limitations, and Section 8 concludes with a summary and directions for future work. A reader interested primarily in the method may proceed from Section 4 to the experiments; a reader interested in the guarantees will find the self-contained proofs in Section 5, with extended derivations in Appendix A, Appendix B and Appendix C.

2. Background and Notation

This section fixes the notation used throughout the paper and recalls the mechanics of the backward pass, so that the three modifications of Section 4 can be stated as precise operations on well-defined quantities. Section 2.1 lays out the forward and backward equations of a feedforward network and names the three steps modified later. The two pathologies the method is designed to address are then explained in terms of those equations (Section 2.2): the depth-dependent scale of the backpropagated error and the poor conditioning of the per-layer curvature. Throughout, vectors are columns, $∥·∥$ is the Euclidean norm, and ${∥·∥}_F$ is the Frobenius norm.

2.1. Feedforward Networks and Backpropagation

Consider a feedforward network with L layers. Writing $a^{\left(0\right)}=x\in {\mathbb{R}}^{n_0}$ for the input, the network computes, for $l=1,\dots ,L$,

$$z^{\left(l\right)}=W^{\left(l\right)}{a}^{(l-1)}+b^{\left(l\right)},a^{\left(l\right)}=\varphi \left(z^{\left(l\right)}\right),$$

(1)

where $W^{\left(l\right)}\in {\mathbb{R}}^{n_l\times n_{l-1}}$, $b^{\left(l\right)}\in {\mathbb{R}}^{n_l}$, and $\varphi$ is an elementwise activation (with $\varphi$ the identity at the output layer). Let $\mathcal{L}$ denote a per-example loss and define the error signal ${\delta }^{\left(l\right)}:=\partial \mathcal{L}/\partial z^{\left(l\right)}\in {\mathbb{R}}^{n_l}$. The backward pass computes

$$\begin{array}{cc}\hfill {\delta }^{\left(L\right)}& ={\nabla }_{z^{\left(L\right)}}\mathcal{L},\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill {\delta }^{\left(l\right)}& ={\left(W^{(l+1)}\right)}^{\top }{\delta }^{(l+1)}\odot {\varphi }^{\prime }\left(z^{\left(l\right)}\right),l=L-1,\dots ,1,\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial \mathcal{L}}{\partial W^{\left(l\right)}}}& ={\delta }^{\left(l\right)}{\left(a^{(l-1)}\right)}^{\top },{\displaystyle \frac{\partial \mathcal{L}}{\partial b^{\left(l\right)}}}={\delta }^{\left(l\right)},\hfill \end{array}$$

(4)

where ⊙ denotes the elementwise (Hadamard) product. Equation (3) is the error propagation step, and (4) is the gradient assembly step. For a softmax output with cross-entropy loss, the output error simplifies to the convenient form ${\delta }^{\left(L\right)}=p-y$, where p is the predicted class distribution and y the one-hot label.

All trainable parameters are collected into a single vector $\theta \in {\mathbb{R}}^P$, and $f\left(\theta \right)$ denotes the empirical risk. The symbol $g^{\left(l\right)}:=\partial \mathcal{L}/\partial W^{\left(l\right)}$ denotes a per-layer gradient block and $g=\nabla f\left(\theta \right)$ the full gradient. The number of activations (summed hidden widths) is denoted $\mathcal{A}={\sum }_l{n}_l$. Table 1 summarizes the notation used throughout.

Table 1. Summary of notation.

Symbol	Meaning
L, $n_l$	number of layers; width of layer l
$a^{\left(l\right)},z^{\left(l\right)}$	activation and preactivation at layer l
${\delta }^{\left(l\right)}$	backpropagated error $\partial \mathcal{L}/\partial z^{\left(l\right)}$
$g^{\left(l\right)},g$	per-layer gradient block; full gradient
$P,\mathcal{A}$	number of parameters; number of activations ${\sum }_l{n}_l$
$s_l,S$	M1 per-layer scale; block-diagonal scaling matrix
${\mu }^{\left(l\right)}$	EMA mean activation (M2)
$P_t,\zeta$	M3 diagonal preconditioner; damping
$M_t=P_t{S}_t$	induced preconditioner
${\mu }_M,L_M$	lower/upper eigenvalue bounds of $M_t$
$\mathrm{\Sigma },\kappa \left(·\right)$	activation covariance; condition number
${\mu }_l,{\ell }_l$	per-block strong-convexity and smoothness constants
$\overline{m},\overline{L}$	equalized constants ${\min }_l{s}_l{\mu }_l$, ${\max }_l{s}_l{\ell }_l$
${\kappa }_{\mathrm{blk}}$	worst per-block condition number ${\max }_l{\ell }_l/{\mu }_l$
${\kappa }_{★}$	curvature spread ${\max }_l{\ell }_l/{\min }_l{\ell }_l$ (grows with depth)
$R,L^{★}$	clip range $s_{\max }/s_{\min }$; crossover depth $\log R/\log \overline{\kappa }$
$\overline{\kappa },\omega$	per-layer growth ratio; energy-vs-curvature factor

2.2. Why the Three Steps Matter

Two recurring pathologies motivate the modifications introduced below, and both can be read directly off the equations of the backward pass. The first concerns the scale of the gradient signal as it crosses many layers; the second concerns the conditioning of the local geometry that each layer’s update must navigate. Each is described in turn, because each is tied to a specific step of the pipeline and therefore admits a targeted repair.

Depth-dependent scale. The recursion (3) multiplies the error by a weight matrix and an activation-derivative mask at every layer. Under the usual signal-propagation model, the second moment of ${\delta }^{\left(l\right)}$ therefore evolves multiplicatively with depth, so that early layers can receive gradients that are many orders of magnitude smaller (or larger) than late layers [4,5,10]. A single global learning rate then under-trains some layers and destabilizes others.

Poor curvature conditioning. Even when scales are controlled, the local geometry seen by each layer can be badly conditioned. The Gauss–Newton/Fisher curvature block for $W^{\left(l\right)}$ factorizes (in the Kronecker-factored approximation) into an output factor and an input factor $A_{l-1}=\mathbb{E}\left[a^{(l-1)}{\left(a^{(l-1)}\right)}^{\top }\right]$ [8]. When the activation has a large mean, as it does for rectified units, which are nonnegative and so have a mean far from zero, this input factor is dominated by a rank-one mean term, and the condition number of the optimization problem suffers [1,11]. Recent work has tied such conditioning to training instability at the edge of stability and to the gap between adaptive and non-adaptive methods on modern architectures [2,12,13].

Conditioned Backpropagation attacks these two pathologies at the three steps where they originate. Each modification is local to one step, requires no architectural change, and is cheap enough to run at every iteration, so the repairs can be applied together without changing the asymptotic cost of training. The next section places the modifications in the context of the methods they build on, and Section 4 then states them precisely.

3. Related Work

Each of the three modifications has antecedents in a distinct strand of the literature, and the contribution of this paper lies as much in combining and analyzing them jointly as in any single piece. Four threads are relevant: rules for turning gradients into updates, curvature-aware optimization, normalization and initialization, and the study of optimization landscapes. The relationship of each thread to a specific modification is made explicit where it arises.

A long line of work modifies how gradients become updates. Adaptive moment methods descending from Adam [14], including the decoupled weight-decay variant AdamW [15], and their refinements remain the default for many architectures [16,17,18], and recent symbolic-search and schedule-free methods continue to push the design space [19,20,21], while sharpness-aware objectives reshape what is minimized to favor flat minima [22]. Large-scale benchmarks caution that, once tuned, many optimizers perform comparably, which raises the bar for claims of universal superiority and motivates the present emphasis on mechanism rather than leaderboard position [3]. A complementary thread explains why adaptive methods help: through curvature heterogeneity across parameter blocks [13,23], through sign-descent-like behavior [24], and through heavy-tailed class imbalance in language data [25]. The diagonal preconditioner M3 is an Adam-style second-moment scaler; what differs here is that it is treated as one of three composable steps and analyzed as a damped, bounded positive-definite operator, with the standard caveat that the empirical Fisher diagonal it uses is only a coarse curvature surrogate [24].

Curvature is used explicitly by second-order and natural-gradient methods [7,8]. Full or Kronecker-factored curvature is expensive; M2 and M3 together can be seen as a cheap diagonal-plus-centering surrogate that captures two specific, analyzable effects of curvature (the rank-one mean term and per-coordinate scale) without forming any matrix.

Normalization layers and careful initialization reshape the forward pass so that the backward pass behaves well [1,4,6,26]. Mean-removal in particular is a known ingredient of batch and layer normalization; the present treatment isolates it as an exact reparameterization (M2) and supplies a direct conditioning argument, though it captures only the mean-removal component of normalization, not variance scaling or a learned affine map. Signal-propagation analyses of depth and of transformers [5,10,27,28] motivate the equalization step M1, which is an adaptive, data-driven relative of depth-aware residual scaling and of layer-wise adaptive learning-rate schemes; M1 is not claimed to be wholly novel, but rather to be a correctness-preserving, analyzable instance of that family. The closest members of that family are the layerwise trust-ratio methods LARS [29] and LAMB [30], which scale each layer’s update by the ratio of the parameter norm to the update norm and were designed to stabilize large-batch training. M1 differs in target and in guarantee; it equalizes the estimated gradient second moment across layers rather than tying the step to the parameter norm, and its clipped form is part of a composite operator with a joint boundedness and positive-definiteness proof, together with the depth-uniform rate of Section 5.5 that trust-ratio methods do not provide. Both methods are included as baselines in the revised experiments of Section 6.3. Width-aware parameterizations that make hyperparameters transferable across scale [31] pursue the related goal of taming scale through a fixed reparameterization rather than online estimation.

Closest in spirit to M1 is the literature on gradient scaling and stability: Gradient clipping accelerates training under relaxed smoothness [32], and adaptive gradient clipping enables normalizer-free training [6]. M1 differs in intent, in that rather than bounding the global gradient norm it equalizes norms across layers, addressing depth-dependent disparity rather than outliers in time.

Finally, the convergence analysis draws on the structure of modern optimization landscapes. Over-parameterized networks frequently satisfy a Polyak–Łojasiewicz condition on neighborhoods of the interpolation manifold [33], which is the structure the local linear-rate result exploits. Conditioning and the loss-Hessian spectrum also govern training instability [2,12,34], the scientific-computing setting of physics-informed networks [35,36,37], and the practicalities of training very deep transformers [26,38]. Against this backdrop, CBP is a lightweight way to improve conditioning without architectural change.

Table 2 condenses this discussion and makes the position of the present work explicit. Each neighboring family modifies a single locus of the training pipeline and supplies, at best, a guarantee scoped to that locus. The combination offered here, namely three commuting and correctness-preserving operations on the backward pass, a joint boundedness and positive-definiteness guarantee for their composition, a depth-uniform rate for the equalization step, and a sharp analysis of its clipped implementation, is not available in any prior method of which the author is aware. This combination, rather than any individual ingredient, is the claim of novelty.

Table 2. Positioning of Conditioned Backpropagation (CBP) relative to the closest method families. Each row names what the family modifies, the guarantee it typically provides, and what CBP adds beyond it.

Family	What It Modifies	Typical Guarantee	What CBP Adds
Adaptive optimizers [16,17,18]	Update rule, in isolation	Convergence for the specific rule	The diagonal scaler becomes one of three composable steps with a joint positive-definiteness bound
Second-order and natural gradient [7,8]	Curvature matrix in the update	Convergence at high per-step cost	A matrix-free centering-plus-diagonal surrogate with an exact rank-one removal argument
Normalization and initialization [1,4,6]	Architecture and forward pass	Largely empirical	An exact reparameterization that leaves the function class unchanged, with a direct conditioning theorem
Residual scaling and signal propagation [5,10]	Initialization-time scale	Stability at initialization	Online, data-driven equalization with a depth-uniform rate and an analysis of the clipped form that is actually run
Layerwise trust ratios [29,30]	Per-layer step size via the parameter-to-update norm ratio	Large-batch stability, largely empirical	Equalization of gradient energy rather than parameter norm, with a depth-uniform rate and an analysis of the clipped form
Gradient clipping [6,32]	Global gradient norm over time	Acceleration under relaxed smoothness	Equalization of norms across layers, addressing depth-dependent disparity rather than temporal outliers

4. The Conditioned Backpropagation Framework

This section defines the three modifications precisely and assembles them into a single algorithm. Throughout, EMA denotes an exponential moving average with decay $\rho \in (0,1)$.

4.1. M1: Layerwise Gradient-Norm Equalization

Let $q_l$ be an EMA estimate of the expected squared Frobenius norm of the l-th gradient block, $q_l\approx \mathbb{E}{∥g^{\left(l\right)}∥}_F^2$, updated by

$$q_l\leftarrow \rho q_l+(1-\rho ){∥g^{\left(l\right)}∥}_F^2.$$

(5)

Fix a common target scale $c={\left(\frac{1}{L}{\sum }_j{q}_j\right)}^{1/2}$, and define the per-layer scale

$$s_l=clip\left({\displaystyle \frac{c}{\sqrt{q_l}}},s_{\min },s_{\max }\right),0<s_{\min }\le s_{\max }<\infty .$$

(6)

The equalized gradient block is ${\tilde{g}}^{\left(l\right)}=s_l{g}^{\left(l\right)}$. As an operation on the full gradient vector, M1 is multiplication by the block-diagonal matrix

$$S=blockdiag\left(s_1{I}_{d_1},\dots ,s_L{I}_{d_L}\right)\succ 0,$$

(7)

where $d_l$ is the number of parameters in block l. The clipping in (6) keeps S uniformly bounded and bounded away from zero. The scales applied at iteration t are computed from the estimates $q_l$ stored at the end of iteration $t-1$, with the convention $s_l=1$ at $t=1$; the estimates are refreshed with the current block norms only after the scales have been formed. This ordering, made explicit in Algorithm 1, keeps the equalization matrix a function of past information only, which Section 5.4 uses.

4.2. M2: Activation-Centering Reparameterization

Maintain an EMA of the mean activation entering layer l, ${\mu }^{(l-1)}\approx \mathbb{E}\left[a^{(l-1)}\right]$, and replace the forward map (1) with the centered map

$$z^{\left(l\right)}=W^{\left(l\right)}\left(a^{(l-1)}-{\mu }^{(l-1)}\right)+{\tilde{b}}^{\left(l\right)}.$$

(8)

As ${\tilde{b}}^{\left(l\right)}=b^{\left(l\right)}+W^{\left(l\right)}{\mu }^{(l-1)}$ can represent any bias, (8) parameterizes the same family of functions as (1); M2 is therefore an exact reparameterization, not an approximation. Backpropagation through (8) produces the exact gradient of the reparameterized loss,

$${\displaystyle \frac{\partial \mathcal{L}}{\partial W^{\left(l\right)}}}={\delta }^{\left(l\right)}{\left(a^{(l-1)}-{\mu }^{(l-1)}\right)}^{\top }.$$

(9)

The only change relative to (4) is that the stored activation is centered before the outer product.

The centering statistics are not static: ${\mu }^{(l-1)}$ is re-estimated as training proceeds. Due to the fact that the represented function depends on the pair $({\mu }^{(l-1)},{\tilde{b}}^{\left(l\right)})$, updating the mean alone would change the function even at fixed weights. Whenever the running mean moves from ${\mu }_t^{(l-1)}$ to ${\mu }_{t+1}^{(l-1)}$, the effective bias is therefore transported along with it,

$${\tilde{b}}_{t+1}^{\left(l\right)}={\tilde{b}}_t^{\left(l\right)}+W^{\left(l\right)}\left({\mu }_{t+1}^{(l-1)}-{\mu }_t^{(l-1)}\right),$$

(10)

which leaves $W^{\left(l\right)}(a-{\mu }_{t+1}^{(l-1)})+{\tilde{b}}_{t+1}^{\left(l\right)}=W^{\left(l\right)}(a-{\mu }_t^{(l-1)})+{\tilde{b}}_t^{\left(l\right)}$ identically in a. With (10) in place, a mean update is a pure change of coordinates that fixes the represented function, so M2 remains an exact reparameterization along the whole trajectory and not only at a frozen $\mu$; the convergence analysis of Section 5.4 relies on this invariance. The correction costs one matrix–vector product per layer per iteration, which is within the $O(P+\mathcal{A})$ budget of Proposition 6.

4.3. M3: Damped Diagonal Curvature Preconditioning

Maintain an EMA second-moment vector $v_t$ of the (equalized, centered) gradient, $v_t\leftarrow {\beta }_2{v}_{t-1}+(1-{\beta }_2){\tilde{g}}_t^2$ (elementwise), and form the diagonal preconditioner from the statistics available at the start of the iteration,

$$P_t=diag\left({\displaystyle \frac{1}{\sqrt{{\widehat{v}}_{t-1}}+\zeta }}\right),{\widehat{v}}_{t-1}={\displaystyle \frac{v_{t-1}}{1-{\beta }_2^{t-1}}},P_1=I,$$

(11)

with a positive damping scalar $\zeta$; the symbol is reserved for the damping throughout, so it cannot be confused with the backpropagated error signal ${\delta }^{\left(l\right)}$. The empirical-Fisher diagonal equals the per-coordinate mean-square gradient, so (11) is a damped diagonal Gauss–Newton/Fisher surrogate [7,8]. The damping makes $P_t$ positive definite and bounded for every t (Lemma 1 below).

4.4. The Composite Update

Let $m_t$ be the (bias-corrected) momentum of the equalized, centered gradient. The CBP update is

$${\theta }_{t+1}={\theta }_t-\eta P_t{\widehat{m}}_t,{\widehat{m}}_t={\displaystyle \frac{m_t}{1-{\beta }_1^t}},m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)S_t{g}_t^{\mathrm{c}},$$

(12)

where $g_t^{\mathrm{c}}$ is the centered gradient (9) and $S_t$ the equalization matrix (7). Ignoring momentum (i.e., ${\beta }_1=0$) the update direction is $-M_t{g}_t^{\mathrm{c}}$ with the induced preconditioner

$$M_t=P_t{S}_t,$$

(13)

a product of a positive diagonal and a positive block-scalar matrix, hence itself a positive diagonal matrix. Two points of precision are in order. The damped preconditioner $P_t$ is formed from the second moment of past equalized gradients $\tilde{g}=S{g}^{\mathrm{c}}$, so the composite acts on $g_t^{\mathrm{c}}$ as the ordered product $P_t{S}_t$ rather than as two independent factors; what the descent and convergence analysis requires is only that this product be a bounded positive-definite diagonal operator, which holds for any order. Centering, by contrast, is not a multiplicative factor at all: it changes the gradient $g_t^{\mathrm{c}}$ itself by changing the parameterization, and its role is analyzed separately in Section 5.3. Figure 1 shows where each modification enters the standard pipeline; Algorithm 1 gives pseudocode.

Figure 1. One iteration of Conditioned Backpropagation, read clockwise around the square: The standard pipeline is grey, and the three modifications are red. Along the top (forward pass), M2 centers the incoming activation ($z=W(a-\mu )+\tilde{b}$) before it is stored for the outer product. Down the right side (backward pass), the error signal ${\delta }^{\left(L\right)}$ produced at the loss is propagated back and combined with the stored activations to assemble the per-layer gradient blocks $g^{\left(l\right)}$. Along the bottom and up the left side (update construction), M1 rescales each block ($s_l=\mathrm{clip}(c/\sqrt{q_l})$), M3 applies a damped diagonal preconditioner ($P=\mathrm{diag}(1/(\sqrt{\widehat{v}}+\zeta ))$), and the bounded positive-definite operator $M_t=P_t{S}_t$ is formed before the parameters are updated. Edge labels show the tensor passed between steps; the dashed arrow on the left returns the updated parameters ${\theta }_{t+1}$ to the next iteration. Solid red boxes are transformations applied to the signal within the step; dashed grey chips are the exponential moving-average state variables $(\mu ,q_l,v)$, which are read when the step is formed and refreshed only after the parameter update (dashed arrows), together with the bias transport $\tilde{b}\leftarrow \tilde{b}+W\mathrm{\Delta }\mu$ of (10).

Two orderings in Algorithm 1 are deliberate. First, the scales $s_l$ and the preconditioner $P_t$ are formed from the statistics stored at the end of iteration $t-1$ and only then are those statistics refreshed with the current gradient, so the operator applied at step t is a deterministic function of the past. This one-step lag is what makes $M_t$ measurable with respect to the history and independent of the current minibatch noise, which is exactly the condition the stochastic analysis of Theorem 3 uses; without the lag, the preconditioner would be correlated with the noise it preconditions. Second, the bias transport of (10) follows every mean update, so the represented function is unchanged by the re-estimation of the centering statistics. In practice, the lag is invisible; it delays each statistic by a single minibatch under decay rates $\rho , {\beta }_2$ close to one, and the experiments in Section 6, all run with the lagged form, show no measurable difference from the eager form on any study.

Algorithm 1 Conditioned Backpropagation (one iteration)
Require: parameters ${\theta }_t$, minibatch $(X,Y)$, step $\eta$, decays ${\beta }_1,{\beta }_2,\rho$, damping $\zeta$, clip $[s_{\min },s_{\max }]$
1: Forward pass with inputs centered by the current (past) means ${\mu }^{(l-1)}$ (8)
2: Backward pass (3); assemble centered blocks $g^{\left(l\right)}={\delta }^{\left(l\right)}{(a^{(l-1)}-{\mu }^{(l-1)})}^{\top }$	▹ M2
3: $s_l\leftarrow clip\left(c/\sqrt{q_l},s_{\min },s_{\max }\right)$ with $c={\left(\frac{1}{L}{\sum }_j{q}_j\right)}^{1/2}$, using the stored $q_j$ (information through step
$t-1$; at $t=1$ set $s_l=1$)	▹ M1
4: $q_l\leftarrow \rho q_l+(1-\rho ){∥g^{\left(l\right)}∥}_F^2$ for all l	▹ energy estimate for the next step
5: ${\tilde{g}}^{\left(l\right)}\leftarrow s_l{g}^{\left(l\right)}$ for all l; stack into ${\tilde{g}}_t$
6: $m_t\leftarrow {\beta }_1{m}_{t-1}+(1-{\beta }_1){\tilde{g}}_t$; ${\widehat{m}}_t\leftarrow m_t/(1-{\beta }_1^t)$
7: ${\widehat{v}}_{t-1}\leftarrow v_{t-1}/(1-{\beta }_2^{t-1})$, using the stored $v_{t-1}$ (at $t=1$ use $P_1=I$)
8: ${\theta }_{t+1}\leftarrow {\theta }_t-\eta {\widehat{m}}_t/(\sqrt{{\widehat{v}}_{t-1}}+\zeta )$	▹ M3 + update
9: $v_t\leftarrow {\beta }_2{v}_{t-1}+(1-{\beta }_2){\tilde{g}}_t^2$	▹ second moment for the next step
10: ${\mu }^{\left(l\right)}\leftarrow \rho {\mu }^{\left(l\right)}+(1-\rho )\overline{a^{\left(l\right)}}$ for all l	▹ update means for the next step
11: ${\tilde{b}}^{(l+1)}\leftarrow {\tilde{b}}^{(l+1)}+W^{(l+1)}\left({\mu }_{\mathrm{new}}^{\left(l\right)}-{\mu }_{\mathrm{old}}^{\left(l\right)}\right)$ for all l	▹ transport (10)

5. Theoretical Analysis

This section establishes the claims advertised in the introduction. Section 5.1 first shows that the composite operator is a bounded positive-definite preconditioner; Section 5.2 and Section 5.3 then analyze the two structural modifications M1 and M2; and Section 5.4 establishes convergence, with Section 5.6 bounding the computational overhead.

5.1. The Induced Preconditioner Is Bounded and Positive Definite

Lemma 1 

(Boundedness). Suppose the (equalized, centered) gradients are bounded, ${∥{\tilde{g}}_t∥}_{\infty }\le G$ for all t. Then, the diagonal preconditioner (11) satisfies

$${\displaystyle \frac{1}{G+\zeta }}I⪯P_t⪯{\displaystyle \frac{1}{\zeta }}I,$$

(14)

and the induced preconditioner $M_t=P_t{S}_t$ in (13) satisfies ${\mu }_{M}I⪯M_t⪯L_{M}I$ with ${\mu }_M=s_{\min }/(G+\zeta )$ and $L_M=s_{\max }/\zeta$.

Proof. 

Each diagonal entry of the lagged ${\widehat{v}}_{t-1}$ is a convex combination of past squared gradient components and hence lies in $[0,G^2]$; therefore $\sqrt{{\widehat{v}}_{t-1,i}}\in [0,G]$ and $1/(\sqrt{{\widehat{v}}_{t-1,i}}+\zeta )\in [1/(G+\zeta ),1/\zeta ]$, giving the bound on $P_t$ for $t\ge 2$. Since $S_t$ is diagonal with entries in $[s_{\min },s_{\max }]$ and commutes with the diagonal $P_t$, the product $M_t=P_t{S}_t$ is diagonal with entries in $[s_{\min }/(G+\zeta ),s_{\max }/\zeta ]$, which is the stated two-sided bound. The first-step convention $P_1=I$ lies in the stated interval whenever $G+\zeta \ge 1$, and otherwise the interval is enlarged to contain 1, which changes only the constants. □

Proposition 1 

(Descent direction). Let f be differentiable and let $M\succ 0$ be symmetric positive definite. For any θ with $\nabla f\left(\theta \right)\ne 0$, the direction $d=-M\nabla f\left(\theta \right)$ satisfies

$$\nabla f{\left(\theta \right)}^{\top }d=-\nabla f{\left(\theta \right)}^{\top }M\nabla f\left(\theta \right)\le -{\lambda }_{\min }\left(M\right){∥\nabla f\left(\theta \right)∥}^2<0,$$

(15)

so d is a strict descent direction. In particular, for the CBP operator $M_t=P_t{S}_t$ of Lemma 1, $-M_t\nabla f\left({\theta }_t\right)$ is a descent direction for the reparameterized objective at every iteration.

Proof. 

M is symmetric positive definite, so $\nabla f^{\top }M\nabla f\ge {\lambda }_{\min }\left(M\right){∥\nabla f∥}^2>0$ whenever $\nabla f\ne 0$; negating gives the claim. By Lemma 1, $M_t$ is positive definite with ${\lambda }_{\min }\left(M_t\right)\ge {\mu }_M>0$. Because M2 is an exact reparameterization, the centered gradient $g_t^{\mathrm{c}}$ is the true gradient of the reparameterized loss $\tilde{f}$, and the conclusion applies to $\tilde{f}$, whose stationary points coincide with those of f. □

Proposition 1 is deliberately elementary, but it is the load-bearing observation of the paper: Any positive per-layer scaling (M1) and any positive diagonal preconditioner (M3) preserve the descent property, so the freedom to choose them is constrained only by how much they improve conditioning, not by correctness. M2 contributes for a different reason: It changes the parameterization rather than the gradient operator, and its effect is analyzed next.

5.2. M1 Makes the Per-Layer Gradient Second Moment Depth-Independent

The equalization step is analyzed through the standard signal-propagation model for the backward pass [4,10,39]. This model tracks how the second moment of the error signal evolves layer by layer under random weights, and it is the standard tool for predicting vanishing and exploding gradients. The assumptions below are exact at initialization and hold approximately during training; they permit a closed-form expression for the per-layer gradient energy, from which one reads off precisely what the M1 scaling cancels.

Assumption 1.

At the point of evaluation, (i) the entries of $W^{\left(l\right)}$ are i.i.d. with mean zero and variance ${\sigma }_l^2$; (ii) the backpropagated error ${\delta }^{(l+1)}$ is independent of $W^{(l+1)}$ and of the activation-derivative mask ${\varphi }^{\prime }\left(z^{\left(l\right)}\right)$; and (iii) the activation derivatives satisfy $\mathbb{E}\left[{\varphi }^{\prime }{\left(z^{\left(l\right)}\right)}^2\right]={\chi }_l$ coordinatewise; and (iv) at each layer the error energy ${∥{\delta }^{\left(l\right)}∥}^2$ and the incoming activation energy ${∥a^{(l-1)}∥}^2$ are uncorrelated, which lets the per-block gradient energy factor as a product of the two.

Proposition 2 

(Geometric depth dependence and its removal). Under Assumption 1, the error second moment obeys the recursion

$$\mathbb{E}{∥{\delta }^{\left(l\right)}∥}^2={\kappa }_l\mathbb{E}{∥{\delta }^{(l+1)}∥}^2,{\kappa }_l=n_l{\sigma }_{l+1}^2{\chi }_l,$$

(16)

so that $\mathbb{E}{∥{\delta }^{\left(l\right)}∥}^2=\left({\prod }_{j=l}^{L-1}{\kappa }_j\right)\mathbb{E}{∥{\delta }^{\left(L\right)}∥}^2$, which is geometric in depth. Consequently, the per-layer gradient block has second moment

$$\mathbb{E}{∥g^{\left(l\right)}∥}_F^2=\mathbb{E}{∥{\delta }^{\left(l\right)}∥}^2\mathbb{E}{∥a^{(l-1)}∥}^2,$$

(17)

which inherits the same geometric factor (using Assumption 1(iv)). The equalized blocks ${\tilde{g}}^{\left(l\right)}=s_l{g}^{\left(l\right)}$ with $s_l=c/\sqrt{\mathbb{E}{∥g^{\left(l\right)}∥}_F^2}$ satisfy

$$\mathbb{E}{∥{\tilde{g}}^{\left(l\right)}∥}_F^2=c^2\mathrm{for}\mathrm{every}l,$$

(18)

i.e., the per-layer second moment is constant across depth.

Proof. 

From (3), ${\delta }^{\left(l\right)}={\left(W^{(l+1)}\right)}^{\top }{\delta }^{(l+1)}\odot {\varphi }^{\prime }\left(z^{\left(l\right)}\right)$. Taking squared norms and expectations, and using independence (Assumption 1(ii)),

$$\mathbb{E}{∥{\delta }^{\left(l\right)}∥}^2=\mathbb{E}\left[{\displaystyle \sum _{i=1}^{n_l}}{\varphi }^{\prime }{\left(z_i^{\left(l\right)}\right)}^2{\left({\left(W^{(l+1)}\right)}^{\top }{\delta }^{(l+1)}\right)}_i^2\right]={\chi }_l{\displaystyle \sum _{i=1}^{n_l}}\mathbb{E}\left[{\left({\left(W^{(l+1)}\right)}^{\top }{\delta }^{(l+1)}\right)}_i^2\right].$$

(19)

For a fixed ${\delta }^{(l+1)}$, each coordinate of ${\left(W^{(l+1)}\right)}^{\top }{\delta }^{(l+1)}$ is a sum of $n_{l+1}$ independent mean-zero terms with variance ${\sigma }_{l+1}^2{\left({\delta }_j^{(l+1)}\right)}^2$, so its second moment is ${\sigma }_{l+1}^2{∥{\delta }^{(l+1)}∥}^2$. Summing over the $n_l$ coordinates and taking the outer expectation gives $\mathbb{E}{∥{\delta }^{\left(l\right)}∥}^2={\chi }_l{n}_l{\sigma }_{l+1}^2\mathbb{E}{∥{\delta }^{(l+1)}∥}^2$, which is (16) with ${\kappa }_l=n_l{\sigma }_{l+1}^2{\chi }_l$; unrolling the recursion gives the stated product. Equation (17) follows from (4), $g^{\left(l\right)}={\delta }^{\left(l\right)}{\left(a^{(l-1)}\right)}^{\top }$, and ${∥u{v}^{\top }∥}_F^2={∥u∥}^2{∥v∥}^2$ together with the assumed independence of ${∥{\delta }^{\left(l\right)}∥}^2$ and ${∥a^{(l-1)}∥}^2$. Finally, with $s_l=c/\sqrt{\mathbb{E}{∥g^{\left(l\right)}∥}_F^2}$ one obtains $\mathbb{E}{∥{\tilde{g}}^{\left(l\right)}∥}_F^2=s_l^2\mathbb{E}{∥g^{\left(l\right)}∥}_F^2=c^2$, independent of l. □

Remark 1. 

The product ${\prod }_j{\kappa }_j$ is exactly the quantity that initialization schemes try to set to one [4,5]. Those schemes fix the scale once, at initialization, under idealized assumptions; M1 instead estimates $\mathbb{E}{∥g^{\left(l\right)}∥}_F^2$ online and equalizes it throughout training, so it adapts as ${\sigma }_l$ and ${\chi }_l$ drift. The clipping in (6) guards against the degenerate case $q_l\to 0$. The consequence of this cancellation for the convergence rate, and not only the per-layer scale, is made precise in Section 5.5 (Theorem 4).

5.3. M2 Reduces the Conditioning of the Per-Layer Curvature

The Gauss–Newton (and, under the usual identification, Fisher) curvature block for $W^{\left(l\right)}$ factorizes approximately as a Kronecker product $A\otimes B$, where the input factor $A=\mathbb{E}\left[a{a}^{\top }\right]$ with $a:=a^{(l-1)}$ captures the activation statistics and the output factor B captures the backpropagated-error statistics [8]. The analysis below concerns the input factor A only and holds the output factor B fixed; since the eigenvalues of $A\otimes B$ are products ${\lambda }_i\left(A\right){\lambda }_j\left(B\right)$, a multiplicative change in the conditioning of A carries over to the block when B is unchanged, which is the sense in which the result below is to be read. Writing $\mu =\mathbb{E}\left[a\right]$ and $\mathrm{\Sigma }=\mathrm{Cov}\left(a\right)=\mathbb{E}\left[a{a}^{\top }\right]-\mu {\mu }^{\top }$, the rank-one decomposition reads $A=\mathrm{\Sigma }+\mu {\mu }^{\top }$. Centering (M2) replaces A by $\mathrm{\Sigma }$, removing the term $\mu {\mu }^{\top }$. The next proposition quantifies the benefit of the regime that matters for rectified activations, where the (nonnegative) mean aligns with the dominant principal direction.

Proposition 3 

(Mean removal improves conditioning). Let $\mathrm{\Sigma }\succ 0$ have eigenpairs $({\lambda }_i,v_i)$ with ${\lambda }_1\ge \cdots \ge {\lambda }_n>0$, and let $\mu \ne 0$.

(i) 

(General: The mean inflates the top eigenvalue.) For $A=\mathrm{\Sigma }+\mu {\mu }^{\top }$, the Rayleigh quotient at $v_1$ gives

$${\lambda }_{\max }\left(A\right)\ge {\lambda }_1+{\left(v_1^{\top }\mu \right)}^2\ge {\lambda }_1,$$

(20)

with equality on the left iff $\mu \perp v_1$. Removing $\mu {\mu }^{\top }$ therefore never increases, and strictly decreases (unless $\mu \perp v_1$), the largest eigenvalue. The effect on the full condition number, however, is not determined by this alone: adding a rank-one positive term also raises the interior eigenvalues (${\lambda }_i\left(\mathrm{\Sigma }\right)\le {\lambda }_i\left(A\right)\le {\lambda }_{i-1}\left(\mathrm{\Sigma }\right)$ for $i\ge 2$ by Weyl interlacing), so for a general μ centering can in principle move κ in either direction. The next case shows the regime in which the direction is unambiguous.

(ii) 

(Aligned mean.) If μ is parallel to the top eigenvector $v_1$, then A and Σ share all eigenvectors, ${\lambda }_{\max }\left(A\right)={\lambda }_1+{∥\mu ∥}^2$ and ${\lambda }_{\min }\left(A\right)={\lambda }_n$, so

$$\kappa \left(A\right)=\kappa \left(\mathrm{\Sigma }\right)+{\displaystyle \frac{{∥\mu ∥}^2}{{\lambda }_n}}\ge \kappa \left(\mathrm{\Sigma }\right).$$

(21)

The excess ${∥\mu ∥}^2/{\lambda }_n$ is large precisely when the mean energy dominates the smallest activation variance.

Thus, in the aligned regime, removing $\mu {\mu }^{\top }$ (i.e., applying M2) strictly decreases the condition number whenever $∥\mu ∥>0$; in general, it strictly decreases the top eigenvalue, while its net effect on κ depends on how μ aligns with the spectrum. The aligned regime is the empirically relevant one for rectified activations: In the measurements reported below, the mean is almost collinear with the top eigenvector (median $\cos \angle (\mu ,v_1)\approx 0.99$, Section 6.2).

Proof. 

(i) By the Rayleigh quotient, ${\lambda }_{\max }\left(A\right)\ge v_1^{\top }A{v}_1={\lambda }_1+{\left(v_1^{\top }\mu \right)}^2$, with equality on the left iff the Rayleigh quotient is maximized at $v_1$, i.e., iff $v_1^{\top }\mu =0$. Since $A=\mathrm{\Sigma }+\mu {\mu }^{\top }$ is a rank-one positive-semidefinite perturbation of $\mathrm{\Sigma }$, Weyl’s interlacing inequalities give ${\lambda }_i\left(\mathrm{\Sigma }\right)\le {\lambda }_i\left(A\right)\le {\lambda }_{i-1}\left(\mathrm{\Sigma }\right)$ for $i\ge 2$, so the interior and smallest eigenvalues can only rise; hence, the change in $\kappa ={\lambda }_{\max }/{\lambda }_{\min }$ is not sign-definite in general. (ii) If $\mu =\alpha v_1$ then $\mu {\mu }^{\top }={\alpha }^2{v}_1{v}_1^{\top }$ is diagonal in the eigenbasis of $\mathrm{\Sigma }$; thus, A has eigenvalues ${\lambda }_1+{\alpha }^2$ (along $v_1$) and ${\lambda }_2,\dots ,{\lambda }_n$ unchanged. Since ${\alpha }^2={∥\mu ∥}^2$, ${\lambda }_{\max }\left(A\right)={\lambda }_1+{∥\mu ∥}^2$ and ${\lambda }_{\min }\left(A\right)={\lambda }_n$, giving (21); this reduces to $\kappa \left(\mathrm{\Sigma }\right)$ when $\mu =0$ and exceeds it when $∥\mu ∥>0$. □

Remark 2 

(Why M2 and M3 are complementary). For rectified activations $a\ge 0$, the mean μ is a nonnegative vector whose direction, in the experiments reported below, is almost collinear with the leading principal component of the activation covariance ($\cos \angle (\mu ,v_1)\approx 0.99$); this is the aligned regime of Proposition 3(ii), where the harm from the mean term is greatest. The input factor is also strongly rank-deficient in practice (${\lambda }_{\min }\approx 0$; see Section 6.2), so the undamped condition number is effectively infinite for both the raw and the centered factor. The quantity the update actually faces is the damped condition number ${\kappa }_{\zeta }\left(A\right)=({\lambda }_{\max }\left(A\right)+\zeta )/({\lambda }_{\min }\left(A\right)+\zeta )$, whose denominator is set by the damping ζ of M3 rather than by ${\lambda }_{\min }$. This makes the two modifications complementary: M2 attacks the numerator by removing the rank-one mean term that inflates ${\lambda }_{\max }$, while M3 regularizes the near-null space that M2 cannot touch. Section 6.2 reports ${\lambda }_{\max }$, ${\lambda }_{\min }$, the alignment, and ${\kappa }_{\zeta }$ directly, so that the mechanism, rather than an artifact of the metric, is visible.

5.4. Convergence of the Bounded Preconditioned Scheme

Consider the update ${\theta }_{t+1}={\theta }_t-\eta M_t\nabla f\left({\theta }_t\right)$ with a bounded positive-definite preconditioner, which by Lemma 1 and Proposition 1 is exactly the structure CBP produces. The objective f is assumed ℓ-smooth: $∥\nabla f\left(\theta \right)-\nabla f\left({\theta }^{\prime }\right)∥\le \ell ∥\theta -{\theta }^{\prime }∥$. Here, f denotes the training objective in the (M2-)centered parameterization, so that $\nabla f$ is the centered gradient on which $M_t$ acts, consistent with Proposition 1; because M2 is an exact reparameterization, f and the original objective share the same minimizers and optimal value.

Two structural properties of Algorithm 1, together with one modeling assumption, let the analysis treat a single fixed objective with a preconditioner sequence that is well behaved. The theorems in this subsection are stated and proved for the momentum-free case ${\beta }_1=0$; heavy-ball momentum is used in the algorithm but is excluded from the guarantees below, with the extension sketched in Appendix B.

The first structural property is measurability. By construction, the scales $s_l$ and the preconditioner $P_t$ applied at step t are functions of the statistics stored at the end of step $t-1$ (Algorithm 1, with $s_l=1$ and $P_1=I$ at the first step): the current gradient enters $q_l$ and $v_t$ only after the step has been taken. The operator $M_t=P_t{S}_t$ is therefore measurable with respect to the past iterates and independent of the current minibatch noise, which is precisely what the stochastic analysis of Theorem 3 requires; no assumption is needed for this. The second structural property is the bias transport (10): A mean update composed with the transport is a change of coordinates under which the loss value is invariant, $f_{{\mu }_{t+1}}\left({\theta }_{t+1}^{\prime }\right)=f_{{\mu }_t}\left({\theta }_{t+1}\right)$, so the per-step descent inequalities telescope across mean updates without a drift term; the gradient norms appearing in the rates are then measured in the coordinates in force at the corresponding step, which are uniformly equivalent whenever the activation means stay bounded.

Assumption 2

(Slowly varying statistics). The EMA statistics that define the modifications, namely, the activation means ${\mu }^{\left(l\right)}$ (M2), the block scales $s_l$ (M1), and the second moments $v_t$ (M3), evolve on a slower timescale than the parameters so that over the horizon of the analysis the smoothness constant ℓ of the centered objective and the bound G on the equalized gradients can be taken as uniform in t.

Theorem 1 

(Nonconvex stationarity). Let f be ℓ-smooth and bounded below by $f^{★}$, and suppose ${\mu }_{M}I⪯M_t⪯L_{M}I$ for all t. With the constant step size $\eta ={\mu }_M/(\ell L_M^2)$,

$${\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=0}^{T-1}}{∥\nabla f\left({\theta }_t\right)∥}^2\le {\displaystyle \frac{2\ell L_M^2\left(f\left({\theta }_0\right)-f^{★}\right)}{{\mu }_M^{2}T}}=O\left({\displaystyle \frac{1}{T}}\right).$$

(22)

Proof. 

By ℓ-smoothness and the update rule, with $g_t=\nabla f\left({\theta }_t\right)$,

$$\begin{array}{cc}\hfill f\left({\theta }_{t+1}\right)& \le f\left({\theta }_t\right)+g_t^{\top }({\theta }_{t+1}-{\theta }_t)+\frac{\ell }{2}{∥{\theta }_{t+1}-{\theta }_t∥}^2\hfill \\ \hfill & =f\left({\theta }_t\right)-\eta g_t^{\top }{M}_t{g}_t+\frac{\ell {\eta }^2}{2}{∥M_t{g}_t∥}^2\hfill \\ \hfill & \le f\left({\theta }_t\right)-\eta {\mu }_M{∥g_t∥}^2+\frac{\ell {\eta }^2{L}_M^2}{2}{∥g_t∥}^2=f\left({\theta }_t\right)-\left(\eta {\mu }_M-\frac{\ell {\eta }^2{L}_M^2}{2}\right){∥g_t∥}^2,\hfill \end{array}$$

(23)

where the bounds $g_t^{\top }{M}_t{g}_t\ge {\mu }_M{∥g_t∥}^2$ and $∥M_t{g}_t∥\le L_M∥g_t∥$ were used. Substituting $\eta ={\mu }_M/(\ell L_M^2)$ gives a per-step decrease of $c_0{∥g_t∥}^2$ with $c_0={\mu }_M^2/(2\ell L_M^2)>0$. Telescoping from 0 to $T-1$ and using $f\left({\theta }_T\right)\ge f^{★}$ yields $c_0{\sum }_t{∥g_t∥}^2\le f\left({\theta }_0\right)-f^{★}$, which rearranges to the claim. □

Theorem 2 

(Linear rate under Polyak–Łojasiewicz). In addition to the hypotheses of Theorem 1, suppose f satisfies the PL inequality $\frac{1}{2}{∥\nabla f\left(\theta \right)∥}^2\ge {\mu }_{\mathrm{PL}}(f\left(\theta \right)-f^{★})$ for some ${\mu }_{\mathrm{PL}}>0$. Then, with $\eta ={\mu }_M/(\ell L_M^2)$ and $c_0={\mu }_M^2/(2\ell L_M^2)$,

$$f\left({\theta }_{t+1}\right)-f^{★}\le \left(1-2c_0{\mu }_{\mathrm{PL}}\right)\left(f\left({\theta }_t\right)-f^{★}\right),$$

(24)

so $f\left({\theta }_t\right)\to f^{★}$ linearly provided $2c_0{\mu }_{\mathrm{PL}}<1$.

Proof. 

From the proof of Theorem 1, $f\left({\theta }_{t+1}\right)\le f\left({\theta }_t\right)-c_0{∥g_t∥}^2$. The PL inequality gives ${∥g_t∥}^2\ge 2{\mu }_{\mathrm{PL}}(f\left({\theta }_t\right)-f^{★})$, so $f\left({\theta }_{t+1}\right)-f^{★}\le (f\left({\theta }_t\right)-f^{★})-2c_0{\mu }_{\mathrm{PL}}(f\left({\theta }_t\right)-f^{★})$, which is the stated contraction. □

Remark 3. 

Over-parameterized networks are known to satisfy a local PL^★ condition in neighborhoods of the interpolation manifold rather than a global PL inequality [33]; Theorem 2 should accordingly be read as a local linear rate that applies once iterates enter such a region, which is the regime in which deep networks are typically trained. The contraction factor improves as $c_0$ grows, i.e., as the preconditioner becomes better conditioned ($L_M/{\mu }_M$ closer to one), which is precisely what M1 and M2 promote; the practical caveat that the clip and damping constants make $L_M/{\mu }_M$ large is discussed in Section 7.

Theorem 3 

(Stochastic convergence). Let f be ℓ-smooth and bounded below, and let $g_t$ be an unbiased stochastic gradient with $\mathbb{E}[g_t\mid {\theta }_t]=\nabla f\left({\theta }_t\right)$ and $\mathbb{E}[{∥g_t-\nabla f\left({\theta }_t\right)∥}^2\mid {\theta }_t]\le {\sigma }^2$. By the lagged construction of Algorithm 1, $M_t$ is measurable with respect to the past and hence independent of the current sample noise; suppose also ${\mu }_{M}I⪯M_t⪯L_{M}I$. Then, for constant $\eta \le {\mu }_M/(\ell L_M^2)$,

$${\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=0}^{T-1}}\mathbb{E}{∥\nabla f\left({\theta }_t\right)∥}^2\le {\displaystyle \frac{2\left(f\left({\theta }_0\right)-f^{★}\right)}{\eta {\mu }_{M}T}}+{\displaystyle \frac{\eta \ell L_M^2{\sigma }^2}{{\mu }_M}},$$

(25)

and choosing $\eta =\min \left\{{\mu }_M/(\ell L_M^2),\sqrt{2(f\left({\theta }_0\right)-f^{★})/(\ell L_M^2{\sigma }^{2}T)}\right\}$ gives a rate of $O(1/\sqrt{T})$.

Proof. 

Taking the conditional expectation of the ℓ-smoothness bound,

$$\begin{array}{cc}\hfill \mathbb{E}[f\left({\theta }_{t+1}\right)\mid {\theta }_t]& \le f\left({\theta }_t\right)-\eta \nabla f{\left({\theta }_t\right)}^{\top }{M}_t\nabla f\left({\theta }_t\right)+\frac{\ell {\eta }^2}{2}\mathbb{E}{∥M_t{g}_t∥}^2,\hfill \end{array}$$

(26)

setting $M_t$ as independent of the noise, so $\mathbb{E}[M_t{g}_t\mid {\theta }_t]=M_t\nabla f\left({\theta }_t\right)$. The first inner-product term is $\le -\eta {\mu }_M{∥\nabla f\left({\theta }_t\right)∥}^2$, and $\mathbb{E}{∥M_t{g}_t∥}^2\le L_M^2\mathbb{E}{∥g_t∥}^2\le L_M^2({∥\nabla f\left({\theta }_t\right)∥}^2+{\sigma }^2)$. Hence,

$$\mathbb{E}[f\left({\theta }_{t+1}\right)\mid {\theta }_t]\le f\left({\theta }_t\right)-\left(\eta {\mu }_M-\frac{\ell {\eta }^2{L}_M^2}{2}\right){∥\nabla f\left({\theta }_t\right)∥}^2+\frac{\ell {\eta }^2{L}_M^2}{2}{\sigma }^2.$$

(27)

For $\eta \le {\mu }_M/(\ell L_M^2)$ one has $\eta {\mu }_M-\ell {\eta }^2{L}_M^2/2\ge \eta {\mu }_M/2$. Taking total expectations, telescoping, and dividing by $\eta {\mu }_{M}T/2$ yields the displayed bound, the stated choice of $\eta$ balances the two terms to give $O(1/\sqrt{T})$. □

These three inherited theorems are not abstract; they apply to Algorithm 1 directly. Lemma 1 certifies the explicit bounds ${\mu }_M=s_{\min }/(G+\zeta )$ and $L_M=s_{\max }/\zeta$ for the CBP operator, which are exactly the quantities the theorems require. The clipping in M1 and the damping in M3 are therefore not cosmetic safeguards but the precise ingredients that keep the preconditioner bounded and positive definite, and hence keep the guarantees nonvacuous. Removing either one would allow ${\mu }_M\to 0$ or $L_M\to \infty$, at which point the step-size prescription and the rates degrade, which is the analytical reason both appear in the algorithm.

5.5. A Depth-Uniform Rate for the Equalization Step

The three theorems above treat $M_t$ as a single positive-definite operator and control its effect through the global eigenvalue ratio $L_M/{\mu }_M$, which Lemma 1 bounds in the worst case by $s_{\max }(G+\zeta )/\left(s_{\min }\zeta \right)$. That ratio ignores the structure of M1; the scaling $S_t$ is block diagonal, one scalar per layer, and it is built to cancel exactly the geometric depth factor that Proposition 2 identifies. Proposition 2 shows that the per-layer gradient energy carries the product ${\prod }_j{\kappa }_j$; the result below shows that M1 removes this factor from the convergence rate, not just from the per-layer scale. This is the one place where the analysis goes beyond the inherited guarantees and isolates a property those borrowed results do not provide, and it concerns the modification (M1) that is otherwise the least standard of the three.

Suppose the objective is block-wise smooth and strongly convex along the L weight blocks; writing $g^{\left(l\right)}={\nabla }_{{\theta }_l}f$, assume the Hessian decomposes as

$${\nabla }^{2}f\left(\theta \right)=D\left(\theta \right)+E\left(\theta \right), D=blockdiag\left({\nabla }_{{\theta }_l}^{2}f\right),{\mu }_{l}I⪯{\nabla }_{{\theta }_l}^{2}f⪯{\ell }_{l}I,$$

(28)

where E collects the off-block coupling. Theorem 4 below is proved for the exactly block-diagonal case $E=0$; Proposition 4 then quantifies how much off-block coupling the rate tolerates and how the coupling enters the constant. This is the standard setting of block-coordinate analyses and is consistent with the pronounced block heterogeneity of the Hessian documented empirically for deep networks and transformers [13]. Let $S=\mathrm{blockdiag}\left(s_{l}I\right)$ be the equalization map of M1 (taking $P=I$ and ${\beta }_1=0$ to isolate M1), and define the equalized constants

$$\overline{L}=\underset{l}{\max }{s}_l{\ell }_l,\overline{m}=\underset{l}{\min }{s}_l{\mu }_l.$$

(29)

Theorem 4 

(Depth-uniform linear rate). Under (28) with $E=0$ (exactly block-diagonal Hessian), the equalized iteration ${\theta }_{t+1}={\theta }_t-\eta S{g}_t$ with step $\eta =1/\overline{L}$ satisfies

$$f\left({\theta }_{t+1}\right)-f^{★}\le \left(1-\frac{\overline{m}}{\overline{L}}\right)\left(f\left({\theta }_t\right)-f^{★}\right),$$

(30)

so ε-suboptimality is reached in $O\left((\overline{L}/\overline{m})\log (1/\epsilon )\right)$ iterations. For the curvature-equalizing choice $s_l\propto 1/{\ell }_l$ one has $\overline{L}/\overline{m}={\max }_l({\ell }_l/{\mu }_l)$, the worst per-block condition number, which carries no depth factor. A single step size ($S=I$) instead gives $\overline{L}/\overline{m}={\max }_l{\ell }_l/{\min }_l{\mu }_l$, which under Proposition 2 scales with the geometric product ${\prod }_j{\kappa }_j$ and is therefore generically exponential in depth.

Proof. 

The change of variables $\widehat{\theta }=S^{-1/2}\theta$ turns the iteration into ordinary gradient descent on $\widehat{f}\left(\widehat{\theta }\right)=f\left(S^{1/2}\widehat{\theta }\right)$; indeed $\nabla \widehat{f}=S^{1/2}\nabla f$, so ${\widehat{\theta }}_{t+1}={\widehat{\theta }}_t-\eta \nabla \widehat{f}$ is equivalent to ${\theta }_{t+1}={\theta }_t-\eta S\nabla f$. The Hessian of $\widehat{f}$ is $S^{1/2}{\nabla }^{2}f{S}^{1/2}$, whose block l is $s_l{\nabla }_{{\theta }_l}^{2}f$ with eigenvalues in $[s_l{\mu }_l,s_l{\ell }_l]\subseteq [\overline{m},\overline{L}]$. Hence, $\widehat{f}$ is $\overline{L}$-smooth and $\overline{m}$-strongly convex, and gradient descent with $\eta =1/\overline{L}$ contracts the suboptimality by $1-\overline{m}/\overline{L}$ per step, a textbook fact; the objective value is unchanged by the reparameterization. With $s_l\propto 1/{\ell }_l$ the products $s_l{\ell }_l$ are constant, so $\overline{L}$ equals that constant and $\overline{m}\propto {\min }_l{\mu }_l/{\ell }_l$, giving $\overline{L}/\overline{m}={\max }_l{\ell }_l/{\mu }_l$. With $S=I$, $\overline{L}/\overline{m}={\max }_l{\ell }_l/{\min }_l{\mu }_l$. □

The next result removes the idealization $E=0$ and makes precise how much off-block coupling the depth-uniform rate tolerates; what must be small is the coupling, as seen after equalization, and it enters the rate constant multiplicatively.

Proposition 4 

(Tolerance to off-block coupling). Assume (28), and let

$$e_S=\underset{\theta }{sup}∥S^{1/2}E\left(\theta \right)S^{1/2}∥,$$

(31)

the operator norm of the equalized coupling. If $e_S\le {\epsilon }_c\overline{m}$ for some ${\epsilon }_c<1$, then the equalized iteration ${\theta }_{t+1}={\theta }_t-\eta S{g}_t$ with $\eta =1/(\overline{L}+e_S)$ satisfies

$$f\left({\theta }_{t+1}\right)-f^{★}\le \left(1-{\displaystyle \frac{\overline{m}-e_S}{\overline{L}+e_S}}\right)\left(f\left({\theta }_t\right)-f^{★}\right),{\displaystyle \frac{\overline{L}+e_S}{\overline{m}-e_S}}\le {\displaystyle \frac{\overline{L}/\overline{m}+{\epsilon }_c}{1-{\epsilon }_c}}.$$

(32)

In particular, for the curvature-equalizing choice $s_l\propto 1/{\ell }_l$, the rate constant is at most $({\kappa }_{\mathrm{blk}}+{\epsilon }_c)/(1-{\epsilon }_c)$ with ${\kappa }_{\mathrm{blk}}={\max }_l{\ell }_l/{\mu }_l$; the conclusion of Theorem 4 survives off-block coupling verbatim up to the factor ${(1-{\epsilon }_c)}^{-1}$, and remains depth-uniform whenever ${\epsilon }_c$ is bounded uniformly in L.

Proof. 

As in Theorem 4, the change of variables $\widehat{\theta }=S^{-1/2}\theta$ maps the iteration to gradient descent on $\widehat{f}$ with Hessian $S^{1/2}(D+E)S^{1/2}$. The block-diagonal part has eigenvalues in $[\overline{m},\overline{L}]$, and adding the symmetric perturbation $S^{1/2}E{S}^{1/2}$ moves each eigenvalue by at most $e_S$ (Weyl), so $(\overline{m}-e_S)I⪯{\nabla }^2\widehat{f}⪯(\overline{L}+e_S)I$. With $e_S\le {\epsilon }_c\overline{m}<\overline{m}$, $\widehat{f}$ is strongly convex, and gradient descent with $\eta =1/(\overline{L}+e_S)$ contracts the suboptimality by $1-(\overline{m}-e_S)/(\overline{L}+e_S)$ per step. Finally, $(\overline{L}+e_S)/(\overline{m}-e_S)\le (\overline{L}+{\epsilon }_c\overline{m})/\left((1-{\epsilon }_c)\overline{m}\right)=(\overline{L}/\overline{m}+{\epsilon }_c)/(1-{\epsilon }_c)$. □

Two qualifications keep the claim honest. First, M1 equalizes the gradient energy $\mathbb{E}{∥g^{\left(l\right)}∥}_F^2$, not the block curvature ${\ell }_l$ directly; the two targets coincide only when the per-block displacement from the optimum is comparable across blocks, which holds in the near-isotropic regime at initialization that Proposition 2 describes. In general they differ by a factor $\omega ={\max }_l\left(s_l{\ell }_l\right)/{\min }_l\left(s_l{\ell }_l\right)\ge 1$, and the rate holds with $\overline{L}/\overline{m}$ replaced by $\omega ·{\max }_l({\ell }_l/{\mu }_l)$; the claim is only that equalization removes the depth-exponential part of the global condition number, leaving the factor $\omega$ and the worst per-block conditioning, both of which are normally far smaller. The mechanism behind the approximation can be stated explicitly. In a block-quadratic neighborhood of a minimizer, $g^{\left(l\right)}\approx {\nabla }_{{\theta }_l}^{2}f({\theta }_l-{\theta }_l^{★})$, so the block energy scales as $\mathbb{E}{∥g^{\left(l\right)}∥}_F^2\approx {\ell }_l^2{r}_l^2$ with $r_l$ the root-mean-square displacement of block l from its optimum. The norm equalizer $s_l\propto q_l^{-1/2}$ therefore behaves as $s_l\propto 1/\left({\ell }_l{r}_l\right)$, which reproduces the curvature equalizer $s_l\propto 1/{\ell }_l$ exactly when the displacements $r_l$ are comparable across blocks, and the factor $\omega$ equals the spread ${\max }_l{r}_l/{\min }_l{r}_l$ of those displacements otherwise. Away from this regime, $\omega$ must be measured rather than assumed small: the depth study of Section 6.5 reports the empirical $\omega$ directly, and it stays below a factor of a few in the settings tested there; verifying its smallness across a wider range of regimes is part of the empirical program of Section 8.2. Second, M1 uses an exponential moving average of the energy rather than its exact value, so $\omega$ also absorbs the estimation error, which Assumption 2 controls because the statistic varies slowly relative to the iterates. The depth-uniformity claim is therefore a statement about the rate constant on a block-structured model, not a claim that arbitrary deep networks converge at a depth-independent rate.

The next model problem shows that the separation is genuine and exponential in depth, not merely a loosening of worst-case constants.

Proposition 5 

(Exponential depth separation on a model problem). Let $f\left(\theta \right)=\frac{1}{2}{\sum }_{l=1}^L{\ell }_l{\theta }_l^2$ with ${\ell }_l={\gamma }^l$ for some $\gamma >1$: a block-diagonal quadratic whose block curvatures grow geometrically with depth, in the manner predicted by Proposition 2. To reach $∥{\theta }_T∥\le \epsilon ∥{\theta }_0∥$,

(i) 

gradient descent with any single step size requires $T_{\mathrm{GD}}=\mathrm{\Omega }\left({\gamma }^L\log (1/\epsilon )\right)$ iterations, while

(ii) 

the curvature-equalized iteration ${\theta }_l\leftarrow {\theta }_l-\eta {\ell }_l^{-1}{g}^{\left(l\right)}$, which is M1 with $s_l={\ell }_l^{-1}$ (equivalently the energy equalizer $s_l=c/\sqrt{q_l}$ when the blocks start at comparable scale), requires $T_{\mathrm{eq}}=O\left(\log (1/\epsilon )\right)$ iterations, independent of L.

The iteration counts thus separate by a factor ${\gamma }^L$, which is exponential in depth.

Proof. 

The blocks evolve independently. (i) Stability of ${\theta }_l\leftarrow (1-\eta {\ell }_l){\theta }_l$ for every block requires $\eta \le 2/{\max }_l{\ell }_l=2{\gamma }^{-L}$. The slowest block $l=1$ then contracts by $|1-\eta {\ell }_1|\ge 1-2{\gamma }^{1-L}$, so reaching $\epsilon$ needs $T\ge \log (1/\epsilon )/\left(-\log (1-2{\gamma }^{1-L})\right)=\mathrm{\Omega }\left({\gamma }^{L-1}\log (1/\epsilon )\right)$. (ii) With $s_l={\ell }_l^{-1}$ every block becomes ${\theta }_l\leftarrow (1-\eta ){\theta }_l$ for a common $\eta <1$, contracting at the depth-independent rate $1-\eta$, so $T=O\left(\log \right(1/\epsilon \left)\right)$. □

Theorem 4 analyzes the unconstrained equalizer. The implemented M1 clips each scale into a fixed range, ${\widehat{s}}_l=clip(s_l^{★},s_{\min },s_{\max })$ with $s_l^{★}\propto 1/{\ell }_l$ the curvature-equalizing target, because Lemma 1 needs the range bounded to keep the preconditioner uniformly positive definite. The clip is therefore not incidental; it is the price of the boundedness guarantee, and the question is how much depth robustness it survives. Write the two structural ratios that govern the answer,

$${\kappa }_{★}=\frac{{\max }_l{\ell }_l}{{\min }_l{\ell }_l}(\mathrm{the}\mathrm{curvature}\mathrm{spread},\propto {\prod }_j{\kappa }_j),R=\frac{s_{\max }}{s_{\min }}\left(\mathrm{the}\mathrm{clip}\mathrm{range}\right),$$

(33)

and let ${\kappa }_{\mathrm{blk}}={\max }_l{\ell }_l/{\mu }_l$ be the worst per-block condition number.

Lemma 2 

(The clip cannot over-correct). For scales restricted to $[s_{\min },s_{\max }]$, the equalized block spread $\mathrm{\Phi }=\left({\max }_l{\widehat{s}}_l{\ell }_l\right)/\left({\min }_l{\widehat{s}}_l{\ell }_l\right)$ obeys $\mathrm{\Phi }\ge {\kappa }_{★}/R$ for every choice of centering, with the matching bound $\mathrm{\Phi }\le \max (1,{\kappa }_{★}/R)$ attained by centering $s_l^{★}$ inside the range. The same statement holds verbatim with the curvatures ${\ell }_l$ replaced by the gradient-block norms $∥g^{\left(l\right)}∥$ that M1 actually measures: the equalizer reduces the block-norm disparity by at most the factor R.

Proof. 

Since ${\widehat{s}}_l\in [s_{\min },s_{\max }]$, the block of largest curvature has ${\widehat{s}}_l{\ell }_l\ge s_{\min }{\ell }_{\max }$ and the block of smallest curvature has ${\widehat{s}}_l{\ell }_l\le s_{\max }{\ell }_{\min }$, so $\mathrm{\Phi }\ge s_{\min }{\ell }_{\max }/\left(s_{\max }{\ell }_{\min }\right)={\kappa }_{★}/R$ regardless of how the scales are centered. For the upper bound, ${\widehat{s}}_l{\ell }_l=clip(\tau ,s_{\min }{\ell }_l,s_{\max }{\ell }_l)$ for the centering constant $\tau$; if ${\kappa }_{★}\le R$ the intervals share a common point, and $\tau$ can be placed in $[s_{\min }{\ell }_{\max },s_{\max }{\ell }_{\min }]$, leaving every block unclipped with ${\widehat{s}}_l{\ell }_l=\tau$ and $\mathrm{\Phi }=1$; if ${\kappa }_{★}>R$ the extreme blocks clip and $\mathrm{\Phi }=s_{\min }{\ell }_{\max }/\left(s_{\max }{\ell }_{\min }\right)={\kappa }_{★}/R$. The norm version is identical with ${\ell }_l\mapsto ∥g^{\left(l\right)}∥$. □

Theorem 5 

(Rate of the clipped equalizer). Under the block-wise smoothness of Theorem 4, the clipped iteration ${\theta }_{t+1}={\theta }_t-\eta \widehat{S}{g}_t$ with $\eta =1/\overline{L}$ converges linearly with

$${\displaystyle \frac{\overline{L}}{\overline{m}}}\le {\kappa }_{\mathrm{blk}}·\max \left(1,{\displaystyle \frac{{\kappa }_{★}}{R}}\right).$$

(34)

Two regimes follow. While the curvature spread stays within the clip range, ${\kappa }_{★}\le R$, the bound is ${\kappa }_{\mathrm{blk}}$: the clipped method is exactly as depth-uniform as the idealized one of Theorem 4. Once ${\kappa }_{★}>R$, the depth-dependent factor reappears but divided by the clip range, ${\kappa }_{\mathrm{blk}}{\kappa }_{★}/R$, so the clipped equalizer improves the condition number by the constant factor R at every depth and never by more.

Proof. 

As in Theorem 4, ${\widehat{S}}^{1/2}$ reparameterizes the iteration into gradient descent with smoothness $\overline{L}={\max }_l{\widehat{s}}_l{\ell }_l$ and strong convexity $\overline{m}={\min }_l{\widehat{s}}_l{\mu }_l$, giving the linear rate $1-\overline{m}/\overline{L}$. Bounding $\overline{m}\ge \left({\min }_l{\widehat{s}}_l{\ell }_l\right)/{\kappa }_{\mathrm{blk}}$ yields $\overline{L}/\overline{m}\le {\kappa }_{\mathrm{blk}}\mathrm{\Phi }$, and Lemma 2 supplies $\mathrm{\Phi }\le \max (1,{\kappa }_{★}/R)$. □

Corollary 1 

(Depth-robustness that survives the clip). If the per-layer curvature growth is geometric, ${\kappa }_{★}={\overline{\kappa }}^L$ with $\overline{\kappa }>1$, the clipped equalizer is depth-uniform for all depths up to

$$L^{★}={\displaystyle \frac{\log R}{\log \overline{\kappa }}},$$

(35)

and for $L>L^{★}$ retains a constant factor-R reduction of the condition number relative to a single step size. Depth-uniformity is thus not lost gradually, but at a sharp, clip-determined depth $L^{★}$, after which a bounded speedup persists at all depths.

This sharpens the earlier informal reading of the clip from a limitation into a quantitative prediction, and it concerns the operator that is actually run rather than its idealization. With the implemented range $[s_{\min },s_{\max }]=[0.1,4]$, so $R=40$, and the per-layer growth $\overline{\kappa }\approx e^{0.25}\approx 1.28$ measured in Section 6.5, the predicted crossover is $L^{★}\approx \log 40/0.25\approx 15$ hidden layers; the equalized disparity reported there stays at one through depth twelve and departs only beyond it, matching the prediction. The residual gap between the measured equalized disparity and the value one in the transition region around $L^{★}$ is the energy-versus-curvature discrepancy already quantified by the factor $\omega$ after Theorem 4, not a failure of the clip analysis.

The reading is that M1 contributes precisely what the inherited theorems cannot express: It converts a condition number that grows with depth into one set by the worst single layer, up to the clip-determined depth $L^{★}$ and, beyond it, by the worst single-layer times a depth factor, reduced by the fixed clip range. This prediction is tested directly in Section 6.5. On the block-structured model problem, the equalized scheme converges in a constant number of iterations at every depth, while single-step-size descent grows geometrically (Figure 5), and in networks, the per-layer disparity the rate depends on is confirmed to grow geometrically with depth (Figure 6). It is also the formal counterpart of the empirical finding that equalization removes the depth-dependent scale disparity rather than lowering the error floor; a rate constant that does not degrade with depth is what this robustness means quantitatively, with the training-level boundary of that statement measured directly in Section 6.5.

5.6. Computational and Memory Overhead

Proposition 6 

(Overhead). Per iteration, CBP adds $O(P+\mathcal{A})$ arithmetic operations and $O(P+\mathcal{A})$ memory beyond plain backpropagation, where P is the number of parameters and $\mathcal{A}$ the number of activations. This is the same asymptotic order as one backward pass.

Proof. 

M1 computes L block norms and an EMA, costing $O\left(P\right)$ for the norms and $O\left(L\right)$ for the EMA; M2 maintains running means, costing $O\left(\mathcal{A}\right)$; M3 maintains an EMA second moment and forms the elementwise quotient, costing $O\left(P\right)$. The dominant cost of a backward pass is the matrix products, already $\mathrm{\Omega }\left(P\right)$ for dense layers, so the added work does not change the asymptotic order. The extra state is the mean buffers ($O\left(\mathcal{A}\right)$), the second-moment buffer ($O\left(P\right)$), and the per-layer scales ($O\left(L\right)$), totaling $O(P+\mathcal{A})$. □

Table 3 compares CBP with common baselines. Its footprint matches Adam plus a set of activation-mean buffers, which are typically much smaller than the parameters themselves for fully connected and convolutional layers. The asymptotic statement is borne out, though not negligible, in practice: On the 10-layer width-48 network used below, a CBP step takes about $2.2$ times as long as a bare SGD step and about $1.6$ times as long as an Adam step in the pure-NumPy reference implementation (averaged over 400 steps). The constant is inflated at this small scale because the extra exponential moving averages and block-norm computations are not amortized against large matrix products; as width grows and the matmuls dominate, the relative overhead falls toward unity.

Table 3. Per-iteration overhead beyond a forward/backward pass. P is the number of parameters; $\mathcal{A}$ the number of activations (summed widths); L the depth.

Method	Extra Compute	Extra State	Positive-Definite by Construction
SGD	n/a	n/a	yes (identity)
SGD + momentum	$O\left(P\right)$	$O\left(P\right)$	yes (identity)
Adam	$O\left(P\right)$	$2P$	yes ($ϵ$-floor)
CBP (M1 + M2 + M3)	$O(P+\mathcal{A})$	$2P+\mathcal{A}+L$	yes (clip + damping)

Due to the fact that modern training runs in vectorized frameworks rather than in NumPy, the overhead is also measured in PyTorch [40] (Table 4): per-iteration wall-clock time of SGD, Adam, and the full CBP (including centering and the bias transport) on fully connected networks of width 256, 1024, and 4096 and on the eight-layer residual convolutional network used in Section 6.6, on the fixed CPU hardware described in Appendix C. Two facts emerge. First, the overhead relative to Adam falls as the matrix products grow, from $2.1\times$ at width 256 to $1.6\times$ at width 4096, and is $1.7\times$ on the convolutional network, where the convolutions dominate; this is the amortization the asymptotic analysis predicts. Second, the optimizer state of CBP equals Adam’s two buffers per parameter plus the activation-mean buffers and L scalars, so its measured memory footprint is indistinguishable from Adam’s at these scales. The reported constants come from a straightforward per-tensor implementation; the per-element operations that dominate the residual overhead are exactly those that kernel fusion optimizes, so the constants on GPU/TPU hardware, which remain unmeasured here, should be read as open rather than implied by these CPU numbers.

Table 4. Measured per-iteration wall-clock time in PyTorch (CPU, two threads, batch 128, mean over 60 timed iterations after warm-up) and optimizer state. “×Adam” is the ratio of the CBP time to the Adam time at the same architecture.

Architecture	Params	SGD (ms)	Adam (ms)	CBP (ms)	×Adam
MLP, width 256	0.40 M	2.2	3.8	7.8	2.1
MLP, width 1024	4.0 M	23.2	36.5	69.5	1.9
MLP, width 4096	53.6 M	281	654	1061	1.6
ResNet-8 (CIFAR)	78 K	147	146	252	1.7

6. Experiments

The experiments are built to test mechanisms rather than to crown a champion and are designed to be honest about what does and does not follow from the data. Each study targets one part of the analysis: Section 6.1 tests the norm-equalization of Proposition 2; Section 6.2 tests the conditioning result of Proposition 3, including the alignment assumption it relies on; Section 6.3 reports a head-to-head training comparison with the published ablation; Section 6.4 adds a common-base factorial ablation over all combinations of M1, M2, and M3 together with a sensitivity analysis of every CBP hyperparameter; Section 6.5 confirms the depth-uniform rate of Theorem 4 on the model problem it analyzes and in the disparity that drives it, and adds a network-level training experiment that delineates the scope of the claim; and Section 6.6 evaluates the method on two standard real-world benchmarks (MNIST and CIFAR-10), including a residual convolutional network, with wall-clock time and memory measured in PyTorch. A preview of the conclusion: the reliable, low-variance effects are about optimization (training loss and convergence speed), with centering doing most of the work; generalization is comparable across methods, and the depth-uniform rate is confirmed on the model problem, while its network counterpart is bounded by the equalization clip at extreme depth.

The synthetic studies use data generated from fixed seeds, so every number below is reproducible from the accompanying code. The task is a multi-arm noisy spiral in ${\mathbb{R}}^2$ (a standard hard-to-separate benchmark), standardized to zero mean and unit variance and split $60/20/20$ into training, validation, and test sets. Learning rates are chosen on the validation split and every number is reported on the held-out test split; the selected rate is listed in each table. Networks are fully connected multilayer perceptrons of width 48 (width 40 for the conditioning study), with softmax outputs and cross-entropy loss. Adaptive methods use the usual defaults (${\beta }_1=0.9$, ${\beta }_2=0.999$); for CBP the damping is $\zeta ={10}^{-3}$, the EMA decay is $\rho =0.99$, and the clip is $s_l\in [0.1,4]$. The noise level of the spiral is fixed per study (it is not tuned per method) and is listed in Appendix C. The baselines are SGD, SGD with momentum, Adam [14], AdamW [15], the layerwise trust-ratio methods LARS [29] and LAMB [30], and Adam+Center (Adam on the M2-centered reparameterization), the last serving as a controlled baseline that isolates mean removal; it captures only that component of normalization and is not a full batch- or layer-norm layer. CBP is ablated into its single modifications (M1, M2, M3), the pair M1+M2, and the full method. Because these published single-modification ablations attach to different base update rules (M1 and M2 to an SGD-style step, M3 and the full method to the Adam-style preconditioned step), they are read as indicative of each modification’s contribution; the fully orthogonal common-base factorial that disentangles the base rule from the modifications is reported separately in Section 6.4.

The statistical protocol was strengthened for this revision. Every training configuration of the head-to-head and factorial studies is run over ten seeds (fifteen for the disparity sweep, five for the sensitivity grid) and reported as mean ± standard deviation, with 95% bootstrap confidence intervals for the primary quantities and two-sided paired t-tests (checked against Wilcoxon signed-rank tests) against the Adam reference, pairing by seed; p-values are reported uncorrected alongside effect sizes so the reader can apply any preferred correction. All methods share one common learning-rate grid $\{0.001, 0.003, 0.01, 0.03, 0.1, 0.3\}$, so every method receives an identical tuning budget; the full validation profile over the grid is reported in Appendix C (Table A2), which makes the sensitivity of the conclusions to the selected search range directly visible. A learning rate is discarded only if it produces a non-finite loss on at least one seed; this rule is applied identically to every method, and the grid profiles in Table A2 show that for every method the validation loss is already deteriorating at the last finite rate before a discarded one, so the discard rule removes rates that were not competitive rather than clipping any method’s optimum.

6.1. M1 Equalizes Per-Layer Gradient Norms

A 24-hidden-layer tanh network is initialized in a saturating regime that induces vanishing gradients: weights at $1.2$ times the Xavier scale with randomly offset biases (standard deviation $0.25$), so that the bias offsets push a fraction of the units toward their saturating range, the backward derivative attenuation compounds with depth, and the forward signal keeps a mean-driven floor. The Frobenius norm of each layer’s gradient block is measured on a full batch. This experiment characterizes the disparity at initialization, where the signal-propagation model of Proposition 2 applies most cleanly; the runtime version of M1 re-estimates the block norms online as training proceeds. Averaged over ten seeds, the raw block norms span an order of magnitude, with a largest-to-smallest ratio of $16\pm 10$ and the earliest layers receiving the weakest signal, the geometric decay predicted by Proposition 2 and shown in Figure 2. A single application of the M1 scaling (6) compresses this ratio to $2.1\pm 1.0$. It does not reach exactly 1 because the clip $s_l\in [0.1, 4]$ caps how much any one block may be rescaled, so layers that would need more than a fourfold boost remain slightly under-corrected; this is the deliberate price of the bounded scaling that Lemma 1 relies on. Even so, an order-of-magnitude disparity is removed, which is what lets a single global learning rate train all depths at a comparable effective rate.

Figure 2. Per-layer weight-gradient block norms in a 24-hidden-layer tanh network at initialization in the saturating offset-bias regime (log scale; median over ten seeds with shaded interquartile bands). Standard backpropagation (grey) decays geometrically toward the input, spanning a $16\pm 10\times$ range across seeds; layerwise equalization (M1, red) compresses the spread to $2.1\pm 1.0\times$, the residual being a consequence of the bounded clip $s_l\in [0.1, 4]$.

6.2. M2 Reduces Curvature Conditioning

An 8-hidden-layer ReLU network is trained briefly, and then, for each hidden layer, the spectra of the uncentered input factor $A=\mathbb{E}\left[a{a}^{\top }\right]$ and the centered factor $\mathrm{\Sigma }=\mathrm{Cov}\left(a\right)$ are compared, and aggregated over ten seeds. This is a static spectral diagnostic of the effect centering would have on the curvature seen at a given point, rather than a measurement along a CBP trajectory. Table 5 reports the mean norm $∥\mu ∥$, the alignment $\cos \angle (\mu ,v_1)$ between the mean and the top eigenvector, the top eigenvalue, and the damped condition number ${\kappa }_{\zeta }$ ($\zeta ={10}^{-3}$). Two facts stand out. First, the ReLU mean is almost perfectly aligned with the leading eigenvector (median $\cos \approx 0.99$), so the situation is squarely the aligned regime of Proposition 3(ii) where centering is guaranteed to help. Second, the mean inflates the top eigenvalue substantially, and centering removes it, cutting ${\lambda }_{\max }$ by a median factor of $3.4$. The smallest eigenvalue is essentially zero for both factors (the activation Gram matrix is strongly rank-deficient), so the undamped condition number is ill-defined; the damped ${\kappa }_{\zeta }$, which is the quantity the M3 update actually faces, improves by the same median factor of $3.4$ because its denominator is set by the damping rather than by ${\lambda }_{\min }$. This is exactly the complementary picture of Proposition 3’s remark: M2 lowers the top of the spectrum, M3 lifts its floor. Figure 3 illustrates the activation centering (M2) and curvature conditioning across the hidden layers of an 8-layer ReLU network.

Table 5. Effect of activation centering (M2) on the per-layer input curvature factor of an 8-layer ReLU network (medians over ten seeds). $∥\mu ∥$ is the mean norm, cos the alignment of $\mu$ with the top eigenvector, ${\lambda }_{\max }$ the top eigenvalue, and ${\kappa }_{\zeta }$ the damped condition number with absolute $\zeta ={10}^{-3}$ (in units of ${10}^3$). The smallest eigenvalue is $\approx 0$ for both factors, so ${\kappa }_{\zeta }\approx ({\lambda }_{\max }+\zeta )/\zeta$; its reduction is, by construction, the same factor as the ${\lambda }_{\max }$ reduction rather than independent evidence, and is shown only to make explicit the magnitude the damped update faces. “raw” uses $A=\mathbb{E}\left[a{a}^{\top }\right]$, “cen” uses $\mathrm{\Sigma }=\mathrm{Cov}\left(a\right)$.

l	$∥\mathit{\mu }∥$	$\cos \mathit{\angle }(\mathit{\mu },{\mathit{v}}_1)$	${\mathit{\lambda }}_{\max }^{\mathbf{raw}}$	${\mathit{\lambda }}_{\max }^{\mathbf{cen}}$	${\mathit{\kappa }}_{\mathit{\zeta }}^{\mathbf{raw}}/{10}^3$	${\mathit{\kappa }}_{\mathit{\zeta }}^{\mathbf{cen}}/{10}^3$
1	3.49	0.93	17.42	11.69	17.4	11.7
2	4.47	0.98	26.63	12.71	26.6	12.7
3	6.02	0.99	48.67	14.62	48.7	14.6
4	7.54	1.00	74.53	22.02	74.5	22.0
5	9.14	1.00	111.14	31.70	111.1	31.7
6	11.44	1.00	163.70	39.79	163.7	39.8
7	13.34	1.00	235.16	66.28	235.2	66.3
8	16.91	0.99	348.44	89.32	348.4	89.3
median ratio	n/a	n/a	$3.4\times$		$3.4\times$

Figure 3. Activation centering (M2) and curvature conditioning across the hidden layers of an 8-layer ReLU network (medians over ten seeds, shaded interquartile bands). (a) The nonnegative mean inflates the top eigenvalue of the input factor; centering removes the near-aligned rank-one mean term. (b) The damped condition number ($\zeta ={10}^{-3}$) is correspondingly reduced (on a log scale).

6.3. Head-to-Head Training and Ablations

A 10-hidden-layer ReLU network is trained on a six-arm spiral; final performance and a convergence-speed proxy, the number of iterations to drive a smoothed training loss below $0.30$, are reported in Table 6; learning curves are in Figure 4. On this moderate task, every method ends within a narrow test-accuracy band ($0.86$ to $0.88$), which is what one should expect and what large benchmarks predict [3]: nobody wins on generalization here. The interesting structure is in optimization. The configurations that combine centering with the damped preconditioner reach the lowest training loss and converge fastest: Adam + Center at $0.164\pm 0.010$ and the full CBP at $0.170\pm 0.015$, against $0.201\pm 0.025$ for Adam and $0.245\pm 0.039$ for SGD, with the loss threshold reached in $234\pm 7$ and $241\pm 9$ iterations against $326\pm 22$ for Adam and $896\pm 67$ for SGD. With ten seeds, these gaps support formal testing. Against the Adam reference, the full CBP improves the final training loss with paired $p=0.003$ (Cohen’s $d=1.3$ on the per-seed differences) and the convergence speed with $p<{10}^{-5}$ ($d=3.2$); Adam + Center shows the same pattern ($p=0.002$ and $p<{10}^{-6}$). The two leaders are statistically indistinguishable from each other on training loss (paired $p=0.058$), while Adam + Center is about three percent faster to the threshold ($p=0.001$), a small but resolvable gap. The reading is the one the ablation already suggested: Centering combined with a damped diagonal preconditioner is what produces the speed and the low floor, and the full CBP matches that behavior while adding the equalization step, whose contribution is the depth mechanism probed in Section 6.5 rather than further gains on this ten-layer task, and the boundedness guarantee of Lemma 1. The trust-ratio baselines behave as their design predicts: LAMB tracks Adam closely on this small-batch task, and LARS trails the adaptive methods, consistent with both being designed for large-batch stability rather than small-batch speed. The published ablation reading is unchanged: centering (M2) is the single modification that most improves optimization, the equalization (M1) and damping (M3) contribute stability and speed rather than a lower floor, and the full method inherits the fast convergence of the preconditioner together with the low loss of centering. The flip side is visible in the test-loss column, where the most aggressive optimizers overfit slightly; this is the expected gap between optimizing the training objective, which is what the analysis addresses, and generalizing, which it does not. The convergence proxy is a coarse summary: it depends on the smoothing constant and the threshold $0.30$, so it is read only as a robust ordering of speed, not as a precise count. The differences in final test accuracy ($0.86$ to $0.88$) are within the reported seed variability and are not interpreted as a ranking; only the training-loss and convergence gaps exceed the seed standard deviations and survive the paired tests above.

Table 6. Training results on the six-arm spiral with a 10-layer ReLU network, mean ± standard deviation over ten seeds, with 95% bootstrap confidence intervals for the training loss and the convergence proxy. “iters@$0.30$” is the number of iterations to first drive the smoothed training loss below 0.30 (lower is faster). The learning rate is selected on the validation split from the common grid and shown as “l.r.”. The best entry of each column is in bold together with any entry statistically indistinguishable from it under a two-sided paired t-test at the 5% level; test accuracies are statistically indistinguishable across methods and are not marked.

Method	l.r.	Train Loss	CI95	Test Loss	Test Acc.	iters@0.30	CI95
SGD	0.03	0.245 ± 0.039	[0.223, 0.269]	0.368	0.862 ± 0.016	896 ± 67	[857, 935]
SGD + momentum	0.001	0.222 ± 0.033	[0.205, 0.244]	0.337	0.879 ± 0.011	897 ± 100	[840, 956]
Adam	0.001	0.201 ± 0.025	[0.187, 0.216]	0.387	0.871 ± 0.010	326 ± 22	[313, 339]
AdamW	0.001	0.201 ± 0.021	[0.189, 0.213]	0.390	0.870 ± 0.012	324 ± 24	[310, 338]
LARS	0.001	0.231 ± 0.016	[0.221, 0.240]	0.364	0.868 ± 0.012	726 ± 55	[696, 760]
LAMB	0.001	0.196 ± 0.014	[0.188, 0.205]	0.376	0.875 ± 0.008	371 ± 32	[351, 389]
Adam + Center	0.001	0.164 ± 0.010	[0.158, 0.169]	0.400	0.872 ± 0.008	234 ± 7	[230, 238]
CBP-M1	0.03	0.233 ± 0.040	[0.215, 0.260]	0.361	0.870 ± 0.013	721 ± 83	[672, 770]
CBP-M2	0.01	0.186 ± 0.014	[0.180, 0.195]	0.304	0.884 ± 0.005	535 ± 63	[498, 572]
CBP-M3	0.001	0.206 ± 0.037	[0.188, 0.231]	0.367	0.871 ± 0.016	342 ± 28	[326, 358]
CBP-M1M2	0.01	0.183 ± 0.014	[0.176, 0.192]	0.308	0.884 ± 0.005	504 ± 63	[468, 542]
CBP (full)	0.001	0.170 ± 0.015	[0.163, 0.180]	0.387	0.875 ± 0.010	241 ± 9	[236, 247]

Figure 4. Training dynamics on the six-arm spiral with a 10-layer ReLU network. Curves show the median across 10 seeds, with shaded interquartile bands. (a) Smoothed mini-batch training loss (log scale); (b) test accuracy. All methods are competitive on accuracy; the methods that combine centering with a damped preconditioner (Adam + Center and the full CBP) drive the training loss lower and faster.

6.4. Common-Base Factorial Ablation and Hyperparameter Sensitivity

The published ablation of Table 6 attaches M1 and M2 to an SGD-style step and M3 to an Adam-style step, so base rule and modification are confounded. This subsection removes the confound with a factorial design and then asks how sensitive the full method is to each of its hyperparameters.

The factorial holds the base update fixed for all eight combinations of the three modifications: bias-corrected heavy-ball momentum with ${\beta }_1=0.9$, to which M1 adds block equalization, M2 adds the centered reparameterization, and M3 adds the damped diagonal preconditioner. Each cell is tuned on the common learning-rate grid and run over ten seeds, exactly as in Section 6.3. Table 7 reports the result. The reading is clean and confirms the division of labor inferred earlier. Centering is the component that lowers the loss floor: adding M2 to any combination that lacks it improves the final training loss (from $0.223$ to $0.177$ on the bare base, from $0.206$ to $0.170$ on the preconditioned base). Preconditioning is the component that buys speed: adding M3 cuts the iterations to threshold by half or more wherever it is added (from 858 to 342 on the bare base, from 554 to 241 once centering is present). Equalization is close to neutral on this ten-layer He-initialized task ($0.223$ versus $0.213$ alone; 241 iterations with or without it in the full method), which is exactly what the analysis predicts: with homogeneous initialization there is little cross-layer disparity to remove at depth ten, and M1’s contribution appears in the depth studies of Section 6.5 rather than here. No combination shows a negative interaction; the full method inherits the best floor and the best speed simultaneously, and test accuracies stay within the same narrow band as before.

Table 7. Common-base factorial ablation over all combinations of M1 (equalization), M2 (centering), and M3 (damped preconditioning), on the shared bias-corrected momentum base (${\beta }_1=0.9$): mean ± standard deviation over ten seeds, learning rate selected per cell from the common grid.

Combination	l.r.	Train Loss	Test Acc.	iters@0.30
base	0.01	0.223 ± 0.020	0.875 ± 0.011	858 ± 121
M1	0.01	0.213 ± 0.023	0.884 ± 0.008	692 ± 73
M2	0.01	0.177 ± 0.006	0.885 ± 0.003	554 ± 72
M3	0.001	0.206 ± 0.037	0.871 ± 0.016	342 ± 28
M1 + M2	0.01	0.173 ± 0.006	0.888 ± 0.004	523 ± 69
M1 + M3	0.001	0.207 ± 0.028	0.870 ± 0.015	340 ± 25
M2 + M3	0.001	0.170 ± 0.014	0.873 ± 0.006	241 ± 9
M1 + M2 + M3	0.001	0.170 ± 0.015	0.875 ± 0.010	241 ± 9

The sensitivity analysis varies one hyperparameter of the full method at a time around the defaults ($\zeta ={10}^{-3}$, $\rho =0.99$, $[s_{\min },s_{\max }]=[0.1, 4]$, ${\beta }_1=0.9$, ${\beta }_2=0.999$), holding the selected learning rate fixed and running five seeds per setting. Table 8 reports final training loss and iterations to threshold. Three observations summarize it: First, no setting in the tested ranges diverges, and apart from removing momentum entirely (${\beta }_1=0$, which raises the floor by about $0.03$), no setting shifts the final loss by more than a few thousandths, so the defaults sit on a broad plateau. Second, the damping $\zeta$ is the only hyperparameter with a pronounced effect, and it acts on speed rather than on the floor—at $\zeta ={10}^{-1}$ the preconditioner is heavily damped toward a constant rescaling and the iterations to threshold rise from 245 to 578, recovering SGD-momentum-like behavior, while $\zeta \in [{10}^{-4},{10}^{-2}]$ is essentially flat. This matches the role the theory assigns it: $\zeta$ sets the floor of $P_t$, and only an overly large value suppresses the adaptivity. Third, the clip range barely matters on this task, from the tight $[0.25,2]$ to the loose $[0.01,100]$, because the He-initialized ten-layer network exhibits little cross-layer disparity for M1 to correct; the clip becomes consequential only in the deep heterogeneous regime of Section 6.5, where its range R sets the crossover depth of Corollary 1. The EMA decay $\rho$ trades tracking speed against estimator variance with a mild optimum near the default, and the momentum constants behave as in Adam-family methods, with ${\beta }_1=0.95$ slightly lowering the floor at the cost of slightly slower threshold crossing. Across diverse data distributions the picture could differ, and the depth studies probe the one axis (cross-layer heterogeneity) where the defaults are known to bind.

Table 8. One-factor-at-a-time sensitivity of the full CBP around its defaults (marked with ^*), five seeds per setting, learning rate fixed at the selected $0.001$: final training loss and iterations to threshold. No tested setting diverges.

Factor	Value	Train Loss	iters@0.30
damping $\zeta$	${10}^{-4}$	0.168 ± 0.008	244
	${10}^{-3}$ *	0.171 ± 0.008	245
	${10}^{-2}$	0.171 ± 0.013	284
	${10}^{-1}$	0.175 ± 0.001	578
EMA decay $\rho$	0.9	0.164 ± 0.011	215
	0.99 *	0.171 ± 0.008	245
	0.999	0.170 ± 0.011	312
clip $[s_{\min },s_{\max }]$	$[0.25,2]$	0.171 ± 0.011	245
	$[0.1,4]$ *	0.171 ± 0.008	245
	$[0.02,20]$	0.173 ± 0.012	245
	$[0.01,100]$	0.173 ± 0.012	245
momentum ${\beta }_1$	0	0.198 ± 0.023	259
	0.5	0.175 ± 0.013	237
	0.9 *	0.171 ± 0.008	245
	0.95	0.159 ± 0.007	258
second moment ${\beta }_2$	0.99	0.174 ± 0.006	244
	0.999 *	0.171 ± 0.008	245
	0.9999	0.175 ± 0.007	245

6.5. Confirming the Depth-Uniform Rate

Theorem 4 makes a falsifiable prediction: Equalization should convert a depth-dependent convergence rate into a depth-independent one. This is tested in three complementary ways. The first is a direct test on the block-structured model the theorem analyzes, where the curvature is made to grow geometrically with depth by construction so that the prediction can be checked without confounds. The second measures, in actual networks, the per-layer disparity that determines the rate, asking whether it grows with depth and whether M1 removes it. The third, added in this revision, trains those networks to a fixed target and asks how far removing the disparity carries end-to-end training, which delineates the scope of the claim and points to its natural extension.

The model problem is exactly that of Proposition 5: $f\left(\theta \right)=\frac{1}{2}{\sum }_{l=1}^L{\gamma }^l{\theta }_l^2$ with $\gamma =1.5$, so the block curvatures span a factor ${\gamma }^L$ that grows with depth. Figure 5 reports the iterations needed to reach $∥\theta ∥\le {10}^{-3}∥{\theta }_0∥$ as the depth L increases. Single-step-size gradient descent rises geometrically, from 6 iterations at $L=2$ to 2416 at $L=16$; a straight-line fit on the logarithmic axis gives a growth exponent of $0.420$ per layer, matching the predicted $\log \gamma =0.405$ to within the fit error. The curvature-equalized iteration is flat at 10 iterations for every depth. This is the depth-uniform rate of Theorem 4 confirmed exactly, with no seed noise; the separation between the two is the factor ${\gamma }^L$ that Proposition 5 predicts.

Figure 5. Direct confirmation of the depth-uniform rate (Theorem 4, Proposition 5) on the block-diagonal quadratic $f=\frac{1}{2}{\sum }_l{\gamma }^l{\theta }_l^2$ whose curvature grows geometrically with depth. Iterations to reach $∥\theta ∥\le {10}^{-3}∥{\theta }_0∥$ versus depth L (log scale). Single-step-size gradient descent grows geometrically (fitted exponent $0.420$ per layer, matching $\log \gamma =0.405$), while curvature-equalized descent is constant at 10 iterations for every depth.

The model confirms the consequence; the next experiment confirms that its premise holds in networks. The quantity that sets the per-block conditioning is the spread of gradient-block norms, ${\max }_l∥g^{\left(l\right)}∥/{\min }_l∥g^{\left(l\right)}∥$, which Proposition 2 predicts that it grows geometrically with depth. A tanh network is initialized so that the forward signal stays bounded while the per-layer gains vary geometrically (a heterogeneous-curvature regime that keeps the network trainable, unlike a strictly sub-critical initialization), and the disparity is measured over 15 seeds for depths from 4 to 24 (Table 9, Figure 6). The raw disparity grows from $4.0\times$ at depth 4 to $797\times$ at depth 24; a log-linear fit gives a slope of $0.25$ per layer, with the shallow end dominated by a depth-independent contribution of the narrow input and output layers and the geometric factor taking over beyond depth about twelve, confirming the depth dependence predicted by the analysis. Equalization (M1) lowers the disparity at every depth. Theorem 5 and Corollary 1 make this quantitative; the clipped equalizer is depth-uniform until the raw disparity reaches the clip range $R=s_{\max }/s_{\min }=40$, which for the fitted per-layer growth $\overline{\kappa }\approx e^{0.25}\approx 1.28$ occurs at the predicted crossover depth $L^{★}=\log R/\log \overline{\kappa }\approx 15$, and beyond it retains a constant factor-R reduction. The data follow this prediction: the equalized disparity sits at $1.0$ through depth 12, the raw disparity crosses $R=40$ between depths 16 and 20, bracketing the predicted crossover (the vertical marker in Figure 6), and only beyond it does the equalized disparity climb (to a residual $93\times$ at depth 24, against $797\times$ unequalized, a reduction of about $8.6\times$ that is bounded by R as Lemma 2 requires). The within-range equalized disparity, which exact curvature equalization would drive to one, settles between $1.0$ and $1.7$; this residual is a direct empirical estimate of the energy-versus-curvature factor $\omega$ of Theorem 4, the sole quantity by which M1’s energy target departs from exact curvature equalization, and it stays small (a factor below two within the clip range), which is the regime the theorem’s caveat assumes. M1 therefore removes a bounded factor of the depth-induced disparity, set by its clip range, rather than eliminating it at every depth; lifting that limit, at the cost of the boundedness guarantee of Lemma 1, is the explicit trade-off discussed in Section 7.

Table 9. Depth scaling of the rate-determining block-norm disparity ${\max }_l∥g^{\left(l\right)}∥/{\min }_l∥g^{\left(l\right)}∥$ in a tanh network with heterogeneous per-layer curvature (medians over 15 seeds; the reduction factor is the ratio of the two medians). The raw disparity grows geometrically with depth, as Proposition 2 predicts; M1 removes it fully while it is within the clip range $s_{\max }/s_{\min }=40$ and is clip-limited beyond.

Depth	4	8	12	16	20	24
Raw disparity	4.0	4.3	5.4	13.5	46.4	797
With M1 (equalized)	1.0	1.0	1.0	1.7	6.8	93.0
Reduction factor	4.0	4.3	5.4	7.9	6.9	8.6

Figure 6. Depth scaling of the per-layer gradient-block disparity in a tanh network with heterogeneous curvature (median over 15 seeds, shaded interquartile range). The unequalized disparity grows geometrically with depth (confirming the premise of Theorem 4); M1 reduces it at every depth, fully within its clip range and clip-limited beyond. The dashed horizontal line is the clip range $R=40$, and the vertical marker is the crossover depth $L^{★}=\log R/\log \overline{\kappa }\approx 15$ predicted by Corollary 1; the raw disparity reaches R close to that depth, where the equalized disparity begins to depart from depth-uniformity.

A third part of the study, added in this revision, asks the question the first two deliberately leave open: How far does the gradient-scale mechanism carry end-to-end training in nonlinear networks? The same heterogeneous tanh networks are trained to a fixed training-loss target at depths 4 to 20, comparing plain SGD against SGD with M1, with the learning rate selected per depth and per method on the validation split from a six-point grid shifted one step upward from the common grid ($\{0.003,\dots ,1.0\}$ because the shallow single-step-size runs select large rates), five seeds per cell (Figure 7). The outcome delineates the mechanism’s scope precisely. Within the clip range, the two methods reach the target in statistically comparable iteration counts (for example, medians of 77 versus 63 at depth 12), which is what the analysis predicts once both arms are tuned; the disparity M1 removes is the same disparity that per-depth tuning absorbs, so the two routes to the compensation coincide. Beyond depth 16 the iteration counts of both methods rise together (medians of 380 versus 419 at depth 20), and the governing factor there is no longer gradient scale but the compounding saturation of the forward signal, which a rescaling of the backward pass leaves untouched by design. The experiment thereby identifies exactly which ingredient a depth-robust end-to-end method must add: a treatment of the forward signal, for instance coupling equalization with saturation-aware initialization or normalization, which Section 8.2 takes up as future work. Consistent with this scope, and pending that extension, the paper claims no end-to-end depth-scaling advantage.

Figure 7. Network-level depth-training comparison (Study D3): Iterations to drive the smoothed training loss below $0.20$ in heterogeneous tanh networks, for plain SGD and SGD with M1, learning rate tuned per depth and per method (median over five seeds, shaded interquartile range, log scale). The two methods are statistically comparable at every depth, and beyond depth 16 the iteration counts of both rise together, governed by the saturation of the forward signal rather than by the gradient-scale disparity that M1 removes; the study thereby delineates the regime in which the mechanism of Theorem 4 operates and motivates the forward-signal extension of Section 8.2.

Taken together, the three experiments confirm the theorem on its own terms and delineate its practical scope. On the model the theorem analyzes, equalization makes the rate exactly depth-independent (Figure 5). In networks, the depth-growing disparity the theorem assumes is real and geometric (Figure 6), and M1 removes it up to a clip-bounded factor. At the training level, the gradient-scale mechanism operates alongside a forward-signal effect that the present method deliberately leaves for future work (Figure 7), which sharpens the scope of the claim. What is not claimed, pending that extension, is an end-to-end depth-scaling advantage in generalization or training; the concrete program toward one, namely lifting the clip and coupling equalization with a forward-signal treatment, is laid out in Section 8.2.

6.6. Real-Data Studies: MNIST and CIFAR-10

The synthetic studies isolate mechanisms; this subsection asks whether the optimization effects survive contact with standard real-world data, a non-fully-connected architecture, and a mainstream framework. The scale is deliberately modest—small networks trained for a few epochs on fixed CPU hardware (Appendix C)—because the question is transfer of the mechanism, not leaderboard performance; the large-scale regimes that remain untested are stated in Section 7. All runs use PyTorch [40], with CBP implemented as a stand-alone optimizer plus centered layer modules realizing (8) and the transport (10).

The first study trains a four-hidden-layer width-256 ReLU MLP on MNIST [41] (batch 256, four epochs, five seeds per method, learning rate selected per method on a held-out 5000-example validation split from the common grid $\{3\times {10}^{-4},{10}^{-3}$, $3\times {10}^{-3}$, ${10}^{-2}$, $3\times {10}^{-2}$, ${10}^{-1}\}$, one selection epoch per rate, identical budget for every method). The baselines are SGD with momentum, Adam, AdamW, LAMB, and Adam + Center. Table 10 reports the outcome, which reproduces the synthetic pattern on real data. Test accuracies sit in a narrow band ($0.969$ to $0.975$), so once again nobody wins on generalization. On optimization, CBP reaches the lowest final training loss ($0.040\pm 0.008$ against $0.047\pm 0.006$ for Adam) and crosses the training-loss threshold of $0.10$ in the fewest iterations ($318\pm 21$ against $455\pm 9$ for Adam and $393\pm 27$ for Adam + Center), a reduction of roughly thirty percent relative to Adam and nineteen percent relative to the next-fastest baseline. For CBP, the two rates above the selected one diverged and were discarded under the common rule; the selected rate is the optimum of the finite profile, but a finer grid between the selected and discarded rates was not explored. LAMB, designed for large-batch training, again trails at this batch size.

Table 10. MNIST, four-hidden-layer width-256 ReLU MLP, batch 256, four epochs: mean ± standard deviation over five seeds. “iters@0.10” is the number of iterations to drive the smoothed training loss below 0.10.

Method	l.r.	Final Train Loss	Test Acc.	iters@0.10
SGD + momentum	0.1	0.049 ± 0.009	0.9730 ± 0.0022	462 ± 14
Adam	0.003	0.047 ± 0.006	0.9746 ± 0.0025	455 ± 9
AdamW	0.003	0.047 ± 0.008	0.9741 ± 0.0025	461 ± 13
LAMB	0.01	0.079 ± 0.014	0.9689 ± 0.0055	684 ± 31
Adam + Center	0.003	0.047 ± 0.005	0.9729 ± 0.0023	393 ± 27
CBP (full)	0.003	0.040 ± 0.008	0.9734 ± 0.0028	318 ± 21

The second study moves to CIFAR-10 [42] and a small residual convolutional network in the style of [43]: an eight-layer ResNet with widths 16–32–64, one basic block per stage and identity or projection shortcuts ($77,706$ parameters), trained for four epochs at batch 256 over three seeds, with an equal three-point learning-rate budget per method. For CBP, every convolution is a centered convolution (per-channel input means, with the transport applied to the effective bias), which extends M2 beyond the fully connected setting of the synthetic studies. Alongside SGD with momentum, Adam, AdamW, and CBP on the normalization-free network, a batch-normalized variant [44] of the same architecture trained with AdamW is included, so the method can be compared against a normalization-equipped network rather than only against optimizers (Table 11). Two things should be said before the numbers. The budget is short by design (about 700 iterations), so every method is far from convergence; in this regime, optimization speed translates directly into accuracy within the budget, and the accuracy ordering must be read as a statement about optimization, not about generalization at convergence. With three seeds, the orderings are indicative: The paired comparisons below do not reach the 5% level and are reported as such. With that scope fixed, the pattern from the synthetic and MNIST studies persists. CBP reaches the lowest final training loss ($1.048\pm 0.027$ against $1.188\pm 0.073$ for AdamW and $1.109\pm 0.154$ for the batch-normalized variant) and the highest within-budget test accuracy ($0.617\pm 0.010$ against $0.572\pm 0.021$ and $0.598\pm 0.028$; paired $p=0.12$ to $0.14$ against AdamW). The batch-normalized variant is the fastest in the first two epochs, which is normalization doing its usual early-stabilization work, and the centered, equalized, damped update overtakes it by the end of the budget. The comparison with the batch-normalized network is the informative one; M2’s mean removal captures part of what normalization provides, and on this task the full CBP on the normalization-free network matches or exceeds the batch-normalized baseline within the budget, while carrying the boundedness guarantee that normalization does not. Whether CBP adds anything on top of normalization, the question Section 8.2 raises, is not answered by this comparison and remains open.

Table 11. CIFAR-10, eight-layer residual network (16–32–64, $77,706$ parameters), batch 256, four epochs, mean ± standard deviation over three seeds. “AdamW + BN” is the same architecture with batch normalization, trained with AdamW; all other rows use the normalization-free network. “epoch-2 loss” and “final loss” are the training loss after two and four epochs; with three seeds, the orderings are indicative rather than statistically conclusive.

Method	l.r.	Epoch-2 Loss	Final Train Loss	Test Acc.
SGD + momentum	0.03	1.719 ± 0.016	1.475 ± 0.073	0.463 ± 0.033
Adam	0.003	1.461 ± 0.005	1.205 ± 0.050	0.565 ± 0.015
AdamW	0.003	1.468 ± 0.020	1.188 ± 0.073	0.572 ± 0.021
AdamW + BN	0.001	1.175 ± 0.006	1.109 ± 0.154	0.598 ± 0.028
CBP (full)	0.003	1.353 ± 0.010	1.048 ± 0.027	0.617 ± 0.010

Practical training cost, the question wall-clock numbers answer, combines the per-iteration overhead of Table 4 with the iteration counts above. One measurement condition differs between the two: Table 4 is timed at batch 128, whereas the training runs here use batch 256. At the training batch size of 256, the measured per-iteration ratio of CBP to Adam is $2.1$ on the MNIST MLP, matching the table, and $1.3$ on the residual network, below the table’s $1.7$; the larger batch enlarges the convolutional matrix products over which CBP’s per-element overhead is amortized while leaving the fully connected ratio essentially unchanged at this width. On MNIST, the per-iteration cost offsets currently CBP’s thirty percent iteration reduction, so time-to-threshold is about $1.5\times$ Adam’s; on CIFAR-10, where the convolutions amortize the overhead, CBP reaches AdamW’s four-epoch training loss in about three epochs, making wall-clock-to-matched-loss roughly even. No wall-clock advantage is claimed at these scales. The iteration advantage becomes a time advantage exactly where the per-iteration ratio falls below the iteration savings, and the measured trends move in that direction as width and convolutional content grow; the residual overhead is concentrated in the per-element operations that kernel fusion targets, so pursuing the crossover on fused GPU implementations is a concrete direction for future work rather than a limitation of the mechanism itself. Memory behaves as Proposition 6 states: CBP stores Adam’s two moment buffers plus the activation-mean buffers and L scalar scales, and the measured peak resident memory of the training runs is indistinguishable between CBP and Adam at these scales, being dominated by the data and activations. These are CPU measurements on a fixed machine; GPU constants involve kernel fusion and memory bandwidth effects that are not probed here and are left as future work, as Section 7 states.

6.7. Summary of Findings

Read together, the studies support a narrower and more honest claim than a blanket superiority result. M1 removes the depth-dependent gradient-scale disparity, up to the residual left by its bounded clip (Figure 2). M2 removes a near-perfectly aligned rank-one mean term ($\cos \approx 0.99$) and cuts the top curvature eigenvalue by about $3.4\times$ (Table 5), and it is the single modification most responsible for the lowest training loss (Table 6). M3 supplies the damping that keeps the preconditioner positive definite and bounded, and the common-base factorial (Table 7) shows the division of labor cleanly: Centering lowers the floor, preconditioning buys the speed, the two compose without negative interaction, and equalization is neutral at moderate homogeneous depth because its contribution is the depth mechanism probed separately. The sensitivity analysis (Table 8) shows the defaults sit on a broad plateau, with only an overly large damping degrading speed. Generalization is comparable across methods wherever training runs to completion, on the spiral studies and on MNIST alike; on the deliberately short CIFAR-10 budget, the accuracy ordering reflects within-budget optimization rather than generalization, and the optimization-level effects (lowest training loss, fastest threshold crossing for the centering-plus-preconditioning methods) persist on real data and a residual convolutional architecture at a measured per-iteration overhead between about $1.3$ and $2.1$ relative to Adam, falling as the matrix products grow (Section 6.6). The depth-uniform rate predicted for M1 is confirmed exactly on the block-structured model problem (Figure 5), and the geometric growth of the rate-determining disparity is confirmed in networks (Figure 6), with the practical reduction bounded by the equalization clip at extreme depth and with the training-level scope of the claim measured directly and its extension charted as future work (Figure 7). In short, the benefits that can be stood behind are about optimization geometry, exactly the property the analysis establishes, and not one of them requires architectural change or explicit curvature estimation.

7. Discussion

This section interprets the results, situates the three modifications against normalization and adaptive methods, and states the limitations of both the theory and the experiments together with the directions they open. The following subsections address, in turn, the scope of the guarantees, the relationship to normalization and adaptive methods, the gap between the guarantees and their constants, the limitations of the study and the outlook they define, and a connection to ill-conditioned scientific models.

7.1. Scope of the Guarantees

The cleanest message is a correctness-by-construction one: because M1 is a positive block-scalar and M3 is a positive diagonal, their composition is a bounded positive-definite preconditioner (Lemma 1), and M2 is an exact reparameterization, so the whole pipeline inherits standard descent and convergence guarantees (Theorems 1–3) without any new assumptions about the network. What the analysis does not buy is a generalization guarantee; like most optimization results, those reported here speak to how fast and how stably the training objective is reduced, not to test error. The experiments are consistent with this distinction; the measurable advantages of CBP are in optimization quality, namely the lowest training loss and the fastest convergence, and not in generalization, which is comparable across methods wherever training runs to completion. On the deliberately short CIFAR-10 budget, the accuracy differences are the optimization advantage expressed within a fixed budget, not a generalization claim.

7.2. Relationship to Normalization and to Adaptive Methods

M2 is, in effect, the mean-removal half of batch and layer normalization, distilled to its conditioning role and applied as a reparameterization rather than as a layer; the Adam + Center baseline makes this explicit, and its behavior is much like CBP’s centering ablation is reassuring rather than surprising. It also inherits the familiar train/test subtlety of running statistics: as the centering means are exponential moving averages of the training activations, the function evaluated at test time depends on those frozen statistics, exactly as for the running mean of batch normalization. M3 is an Adam-style scaler reinterpreted as a damped Fisher diagonal, with the caveat that the empirical–Fisher diagonal is only a coarse proxy for the true curvature. The less standard ingredient is M1’s cross-layer equalization, which is orthogonal to the in-time adaptivity of Adam and to the global clipping of [32]. Recent explanations of why adaptive methods help cite block-wise curvature heterogeneity [13,23]; M1 addresses the across-block scale component of that heterogeneity and is complementary to the within-block scale that M3 captures. The revised CIFAR-10 study makes the relationship to normalization concrete in one direction; within a short budget, the full CBP on a normalization-free residual network matches or exceeds the same architecture equipped with batch normalization and trained with AdamW (Table 11), which supports reading M2 as capturing a meaningful part of normalization’s conditioning role. Whether CBP adds anything on top of normalization is a separate question that Section 8.2 leaves open. Theorem 4 gives the formal counterpart of this view: Removing the across-block scale spread turns a condition number that grows with depth into one set by the worst single layer, which is the property the heterogeneity literature observes empirically but does not state as a rate. The factorial ablation makes the division of labor explicit: Centering carries the lower loss floor, the damped preconditioner carries the speed, and equalization contributes the removal of the depth-dependent scale disparity rather than further gains at moderate homogeneous depth.

7.3. The Price of the Boundedness Guarantee

There is an honest tension between the clean guarantees and their constants. The bounds of Lemma 1 are ${\mu }_M=s_{\min }/(G+\zeta )$ and $L_M=s_{\max }/\zeta$, so with the settings used here ($s_{\min }=0.1$, $s_{\max }=4$, $\zeta ={10}^{-3}$) the preconditioner’s own condition number $L_M/{\mu }_M$ is on the order of ${10}^4$, and the worst-case rate constants of Theorems 1–3, which scale like ${(L_M/{\mu }_M)}^2$, are correspondingly loose. In other words, the clipping and damping that guarantee a bounded positive-definite preconditioner are exactly what make the guaranteed rate pessimistic. The right reading is qualitative; the theorems certify that CBP is a legitimate descent method with the standard forms of convergence, and the experiments, not the constants, speak to how fast it actually goes.

The clip, however, is not only a source of loose constants; its effect on the depth-uniform rate can be characterized exactly, and this is where the analysis of the implemented operator pays off. Theorem 5 shows that the clipped equalizer keeps the idealized depth-uniformity of Theorem 4 verbatim while the curvature spread ${\kappa }_{★}$ stays below the clip range $R=s_{\max }/s_{\min }$, and that beyond this point the depth factor returns but divided by R, so the condition number is improved by the fixed factor R at every depth and never more. Corollary 1 turns this into a sharp crossover depth $L^{★}=\log R/\log \overline{\kappa }$; depth-uniformity is not lost gradually but at the depth where the geometric disparity first exceeds the clip range. This is the exact sense in which M1, as implemented, “removes a bounded factor”: the factor is R, and widening $[s_{\min },s_{\max }]$ as the boundedness margin of Lemma 1 allows pushes $L^{★}$ deeper at the cost of looser rate constants. Tightening or scheduling the constants, for instance by adapting $s_{\min },s_{\max },\zeta$ to the observed spectrum, is therefore both a way to sharpen the bounds and a way to extend the depth over which equalization remains uniform.

7.4. Limitations and Outlook

Several limitations are worth stating plainly, each paired with the direction it opens. First, the signal-propagation analysis (Proposition 2) relies on independence assumptions that hold at initialization and only approximately during training; the experiments, run on trained networks, validate the qualitative prediction but not the exact constants. Second, the smoothness and boundedness premises of the convergence theorems are idealizations: ReLU networks are only piecewise smooth, so the ℓ-smoothness assumption should be read as holding on the regions the iterates traverse rather than globally. Third, Proposition 3 establishes a strict improvement only in the aligned-mean regime; that regime holds strongly here ($\cos \approx 0.99$), but it need not for every architecture, which is why the effect is measured directly. Fourth, the reliable effects reported are about optimization; no generalization gain is observed, and although the depth-uniform rate is confirmed exactly on the block-structured model problem, and the disparity that drives it is confirmed to grow geometrically in networks, the practical clipped equalizer removes a bounded, clip-determined factor of that disparity at extreme depth. Safely widening that factor is part of the program of Section 8.2, and no end-to-end depth-scaling advantage is claimed in the meantime. Fifth, although the revised study now includes two standard real-world benchmarks and a residual convolutional network (Section 6.6), those experiments are deliberately modest in scale: a small MLP on MNIST and an eight-layer residual network on CIFAR-10, trained for a few epochs on CPU hardware. They establish that the optimization effects verified on synthetic data survive contact with real data and a non-fully-connected architecture, and the conclusions are scoped to the regimes tested; ImageNet-class vision models, Transformer language models, long training schedules, and GPU/TPU execution are left for future work, and extending the evaluation to them is the natural next step, which the $O(P+\mathcal{A})$ overhead makes practical, with the qualification that for layer types whose activation count exceeds the parameter count, the activation term, not the parameter term, dominates. The decay rates ${\beta }_1,{\beta }_2,\rho$, the clip the one-factor-at-a-time sensitivity covers range, and the damping analysis of Section 6.4, which finds a broad plateau around the defaults; setting them adaptively is a further opportunity taken up in Section 8.2.

A few methodological caveats deserve equal candor. The head-to-head and factorial studies are run over ten seeds with bootstrap confidence intervals and paired tests, which is enough to resolve the training-loss and convergence gaps formally; the bolded entries in Table 6 carry that statistical reading, and the remaining gaps that do not reach significance are stated as such. Ten seeds remain a modest sample, so the reported p-values should be read as evidence of robust ordering rather than as precise tail probabilities. The depth study leans on deliberately controlled setups; the model problem of Study D1 confirms Proposition 5 by direct simulation, which validates the algebra and the implementation rather than supplying evidence independent of the proof, and the geometric disparity of Study D2 is induced by the “hetero” initialization chosen precisely to decouple curvature growth from a vanishing forward signal, so it demonstrates the mechanism in a regime engineered to expose it rather than one observed in a default network; the network-level training comparison of Study D3 removes one layer of this artificiality by measuring iterations-to-target in actual training runs, and extending the measurement to naturally arising heterogeneity, beyond the engineered initialization it inherits, is a natural follow-up study. The conditioning study reports both ${\lambda }_{\max }$ and ${\kappa }_{\zeta }$, but because ${\lambda }_{\min }\approx 0$ their reductions are the same number and should be counted as one effect, not two (Table 5 states this). The wall-clock overhead is now measured both in the pure-NumPy reference and in PyTorch on an MLP width sweep and the CIFAR-10 residual network (Section 5.6 and Section 6.6); all timings are CPU timings on the fixed hardware listed in Appendix C, and GPU/TPU measurements, where kernel fusion and memory bandwidth reshape the constants in the method’s favor or against it, are a natural next measurement. Finally, momentum is present in Algorithm 1, but the convergence proofs analyze the ${\beta }_1=0$ case, with the momentum extension sketched in Appendix B and its full development left as future work, so the stated rates are guaranteed for the non-momentum version.

7.5. Connections to Ill-Conditioned Scientific Models

The pathologies CBP targets are not specific to classification. Physics-informed neural networks, for instance, are notoriously hard to train precisely because their composite loss produces gradients whose per-term scales differ by orders of magnitude and whose curvature is severely ill-conditioned [35,37]; remedies there often amount to per-term gradient reweighting and adaptive loss balancing [36]. Read in the present framework, those remedies are instances of the same equalization principle as M1, applied across loss terms rather than across layers. This suggests that the three-step view transfers to settings where the “layers” are replaced by heterogeneous loss components, and that a unified, provably positive-definite reweighting could replace the hand-tuned schemes currently used.

8. Conclusions

This paper set out to show that backpropagation, viewed as a pipeline of separable steps rather than as a single indivisible operation, admits small and individually justifiable repairs that compose into a method with provable guarantees. This section summarizes what was established and then outlines the directions the work opens up.

8.1. Summary of Contributions

This paper revisited backpropagation as a three-step pipeline (error propagation, gradient assembly, and update) and proposed one small, analyzable modification at each step: layerwise gradient-norm equalization, activation-centering reparameterization, and damped diagonal preconditioning. The modifications compose into a single method, Conditioned Backpropagation, whose induced operator is a bounded positive-definite preconditioner, a combination of composability and joint guarantee that, to the best of the author’s knowledge, no prior method provides. This structure alone guarantees a valid descent direction and the standard nonconvex, PL, and stochastic convergence rates; under the signal-propagation and aligned-mean assumptions made precise in the analysis, it additionally yields a depth-independent per-layer gradient scale and reduced curvature conditioning for rectified activations. Going beyond these inherited results, the equalization step is shown to admit a depth-uniform linear rate on block-structured objectives, and the clipped equalizer that is actually run is analyzed directly; it stays depth-uniform up to a sharp clip-determined depth and retains a constant-factor improvement beyond it, all at the same asymptotic cost as ordinary backpropagation. Controlled experiments, run over ten or more seeds with confidence intervals and paired tests, confirm each mechanism: Equalization flattens the per-layer gradient profile, centering removes a near-aligned rank-one mean term and is the modification most responsible for the lowest training loss, the damped preconditioner supplies the convergence speed, and the configurations combining the two, including the full method, converge fastest among the methods compared, a division of labor that a common-base factorial ablation over all eight combinations makes explicit. The optimization effects persist on MNIST and CIFAR-10, the latter with a small residual convolutional network, at a measured per-iteration overhead below about twice that of Adam on the hardware tested. The predicted depth-uniform rate is confirmed exactly on the block-structured model problem, and the per-layer disparity that drives it is confirmed to grow geometrically in networks. The scope of the evidence is stated just as plainly: Generalization is comparable across methods wherever training runs to completion, the practical clipped equalizer removes a bounded, clip-determined factor of that disparity at extreme depth, and a training-level depth study delineates the complementary forward-signal component that future work must address before an end-to-end depth advantage can be pursued, so none is claimed here. The broader message is methodological: Treating backpropagation as a sequence of separable steps, each amenable to a small correctness-preserving repair, is a productive way to turn optimization folklore into statements one can prove and then test.

8.2. Future Work

Several directions follow naturally from the three-step view, and it is convenient to group them by the step they touch. At the update-construction step, the equalization target c and the clip range $[s_{\min },s_{\max }]$ are currently fixed constants; making them schedule-free or learning-rate-free, in the spirit of recent parameter-free methods [20,21], would remove the last manually chosen scales and let the preconditioner adapt its own bounds. The damping $\zeta$ and the decay rates ${\beta }_1,{\beta }_2,\rho$ could likewise be inferred from the observed curvature rather than held fixed; the sensitivity analysis of Section 6.4 shows the defaults sit on a broad plateau, so the payoff of adapting them is likely to appear first in the regimes where the damping demonstrably binds.

At the gradient-assembly step, the diagonal preconditioner could be replaced by a low-rank-plus-diagonal correction that interpolates between CBP and Kronecker-factored curvature methods [8]. As the descent guarantee of Proposition 1 requires only that the correction be positive definite, such an extension would retain the convergence theory unchanged, while capturing the off-diagonal curvature that the present diagonal surrogate ignores. Quantifying the trade-off between the richer correction and its added cost is an open and concrete question.

Beyond the architectures studied here, the most consequential next step remains empirical. The revised evaluation verifies the optimization effects on MNIST and CIFAR-10 with a small residual convolutional network; scaling that evidence to ImageNet-class vision models, attention architectures, long schedules, and GPU execution would test whether the benefits survive at production scale and whether the depth-uniform rate of Theorem 4, confirmed here exactly on a block-structured model and in premise on networks, can be made to translate into depth-robust training of real networks, which the training-level study of Section 6.5 shows requires addressing the saturating forward signal alongside the gradient-scale disparity once the equalization clip is relaxed. The disparity sweep showed the practical limit clearly: The clip $[s_{\min },s_{\max }]$ caps the removable disparity at $R=s_{\max }/s_{\min }$, so at extreme depth, geometric disparity outpaces it. By Corollary 1 the clip-determined crossover sits at $L^{★}=\log R/\log \overline{\kappa }$, so widening R as the boundedness margin of Lemma 1 allows, or replacing the energy target with a curvature-aware one (which would shrink the factor $\omega$), pushes $L^{★}$ deeper and lets M1 track the geometric growth further while preserving positive definiteness; making that trade-off between a wider clip, looser rate constants, and a deeper crossover precise is a concrete open question. The $O(P+\mathcal{A})$ overhead established in Proposition 6 makes such an evaluation practical. Given that centering carried most of the loss-floor benefit, and that the CIFAR-10 study already compares CBP against a batch-normalized network in one direction, a useful early experiment is the converse: whether CBP adds measurably to networks that already include normalization layers.

Finally, the equalization principle behind M1 is not limited to layers. The same idea applies across heterogeneous loss terms, such as those that arise in physics-informed and multi-objective training [35,36,37], where per-term gradients can differ in scale by orders of magnitude. A unified, provably positive-definite reweighting could there replace the hand-tuned balancing schemes currently in use. This broader program, namely turning each step of a learning algorithm into an object that can be analyzed and safely modified, is regarded as the most promising direction the present work points toward.

Appendix A.1. Forward Second Moment

Let ${\nu }_l:=\frac{1}{n_l}\mathbb{E}{∥a^{\left(l\right)}∥}^2$ denote the per-coordinate forward activation energy at layer l. From $z^{\left(l\right)}=W^{\left(l\right)}{a}^{(l-1)}+b^{\left(l\right)}$ with i.i.d. mean-zero weights of variance ${\sigma }_l^2$ and (for this calculation) negligible bias, each preactivation coordinate is a sum of $n_{l-1}$ independent terms, so

$$\mathbb{E}\left[{\left(z_i^{\left(l\right)}\right)}^2\right]=n_{l-1}{\sigma }_l^2{\nu }_{l-1}.$$

(A1)

Passing through the activation, ${\nu }_l=\mathbb{E}\left[\varphi {\left(z^{\left(l\right)}\right)}^2\right]$. For a homogeneous activation such as ReLU, $\mathbb{E}\left[\varphi {\left(z\right)}^2\right]=\frac{1}{2}\mathbb{E}\left[z^2\right]$ for symmetric z, giving the familiar He scaling ${\sigma }_l^2=2/n_{l-1}$ as the fixed point ${\nu }_l={\nu }_{l-1}$. For tanh near the origin $\varphi \left(z\right)\approx z$, so ${\nu }_l\approx n_{l-1}{\sigma }_l^2{\nu }_{l-1}$, and the Xavier choice ${\sigma }_l^2=1/n_{l-1}$ keeps ${\nu }_l$ approximately constant until saturation sets in. The key point is that ${\nu }_l$ obeys a multiplicative recursion whose per-layer factor is $n_{l-1}{\sigma }_l^2$ times an activation-dependent constant.

Appendix A.2. Backward Second Moment

Recall from the proof of Proposition 2 that

$$\mathbb{E}{∥{\delta }^{\left(l\right)}∥}^2={\chi }_l{n}_l{\sigma }_{l+1}^2\mathbb{E}{∥{\delta }^{(l+1)}∥}^2=:{\kappa }_l\mathbb{E}{∥{\delta }^{(l+1)}∥}^2,$$

(A2)

where ${\chi }_l=\mathbb{E}\left[{\varphi }^{\prime }{\left(z^{\left(l\right)}\right)}^2\right]$. For ReLU, ${\varphi }^{\prime }\in \{0,1\}$ and ${\chi }_l=Pr[z^{\left(l\right)}>0]\approx \frac{1}{2}$; for tanh, ${\chi }_l=\mathbb{E}\left[{(1-{tanh}^{2}z)}^2\right]\le 1$ and shrinks sharply once units saturate, which is the mechanism that the saturating offset-bias regime of Figure 2 exploits to make a deep tanh stack vanish toward the input. Unrolling (A2),

$$\mathbb{E}{∥{\delta }^{\left(l\right)}∥}^2=\left({\displaystyle \prod _{j=l}^{L-1}}{\kappa }_j\right)\mathbb{E}{∥{\delta }^{\left(L\right)}∥}^2,$$

(A3)

which is the geometric depth dependence. The product ${\prod }_j{\kappa }_j$ equals one only on a measure-zero set of $({\sigma }_j,{\chi }_j)$; outside this, the early-layer error is exponentially smaller or larger than the late-layer error.

Appendix A.3. Per-Block Gradient Second Moment and the Equalization Target

Combining the forward and backward energies through $g^{\left(l\right)}={\delta }^{\left(l\right)}{\left(a^{(l-1)}\right)}^{\top }$ and ${∥u{v}^{\top }∥}_F^2={∥u∥}^2{∥v∥}^2$,

$$\mathbb{E}{∥g^{\left(l\right)}∥}_F^2=\mathbb{E}{∥{\delta }^{\left(l\right)}∥}^2·\mathbb{E}{∥a^{(l-1)}∥}^2=\left({\displaystyle \prod _{j=l}^{L-1}}{\kappa }_j\right)\mathbb{E}{∥{\delta }^{\left(L\right)}∥}^2·n_{l-1}{\nu }_{l-1},$$

(A4)

where the assumed independence of the error and incoming activation energies has been used. Equation (A4) shows that the per-block gradient energy is the product of a backward geometric factor and a forward energy term. The online estimate $q_l$ in M1 is precisely a running estimate of the left-hand side, and the scale $s_l=c/\sqrt{q_l}$ cancels both factors, leaving $\mathbb{E}{∥{\tilde{g}}^{\left(l\right)}∥}_F^2=c^2$ for every l. This is why a single post-hoc rescaling (as measured in Figure 2) already flattens the profile, and why the online version tracks it as the statistics drift during training.

Appendix B.1. Exact Positive-Definiteness of the Composite Operator

Lemma A1. 

Let $S=blockdiag(s_1{I}_{d_1},\dots ,s_L{I}_{d_L})$ with $s_l>0$ and let $P=diag\left(p\right)$ with $p_i>0$. Then S and P commute, and $M=PS$ is symmetric positive definite with ${\lambda }_{\min }\left(M\right)\ge \left({\min }_i{p}_i\right)\left({\min }_l{s}_l\right)$ and ${\lambda }_{\max }\left(M\right)\le \left({\max }_i{p}_i\right)\left({\max }_l{s}_l\right)$.

Proof. 

S is diagonal (a piecewise-constant diagonal); hence, $PS=SP$ is diagonal with entries $p_i{s}_{l\left(i\right)}>0$, where $l\left(i\right)$ is the block containing coordinate i. A diagonal matrix with positive entries is symmetric positive definite, and its extreme eigenvalues are the extreme diagonal entries, giving the stated bounds. □

Lemma A1 is the precise statement underlying Lemma 1 and Proposition 1; the M1 and M3 factors never interact destructively, and the damping/clipping bounds transfer directly to $M_t$.

Appendix B.2. Heavy-Ball Momentum

The convergence theorems were stated for ${\beta }_1=0$. The following sketch indicates why momentum does not break the guarantee. Write the momentum iterate as ${\theta }_{t+1}={\theta }_t-\eta P_t{\widehat{m}}_t$ with ${\widehat{m}}_t=(1-{\beta }_1){\sum }_{k\le t}{\beta }_1^{t-k}{\tilde{g}}_k/(1-{\beta }_1^t)$, a convex combination of past equalized gradients. Standard heavy-ball analyses for ℓ-smooth nonconvex objectives (e.g., via the Lyapunov function $V_t=f\left({\theta }_t\right)+\frac{c_1}{2}{∥{\theta }_t-{\theta }_{t-1}∥}^2$) yield the same $O(1/T)$ stationarity rate provided the effective preconditioner stays within $[{\mu }_M,L_M]$, which Lemma 1 guarantees, and provided $\eta$ is reduced by a constant factor depending on ${\beta }_1$. The bias correction $1/(1-{\beta }_1^t)$ only affects the first few iterations and is bounded by $1/(1-{\beta }_1)$. The qualitative guarantees are therefore retained, while momentum contributes the usual variance reduction in the stochastic setting of Theorem 3.

Appendix B.3. Explicit Step-Size Balancing in the Stochastic Rate

In Theorem 3, writing $D_0=f\left({\theta }_0\right)-f^{★}$ and $C=\ell L_M^2/{\mu }_M$, the bound reads $\frac{1}{T}{\sum }_t\mathbb{E}{∥\nabla f\left({\theta }_t\right)∥}^2\le 2D_0/\left(\eta {\mu }_{M}T\right)+\eta C{\sigma }^2$. Minimizing the right-hand side over $\eta >0$ gives the optimal ${\eta }^{★}=\sqrt{2D_0/\left({\mu }_{M}C{\sigma }^{2}T\right)}$ and value $2\sqrt{2D_{0}C{\sigma }^2/\left({\mu }_{M}T\right)}=O(1/\sqrt{T})$, valid whenever ${\eta }^{★}\le {\mu }_M/(\ell L_M^2)$, i.e., once T is large enough. For smaller T the deterministic cap $\eta ={\mu }_M/(\ell L_M^2)$ is active, and the rate is the $O(1/T)$ optimization term of Theorem 1.