Hyper–Dual Numbers: A Theoretical Foundation for Exact Second Derivatives

Abstract

Second-order derivative information, including mixed curvature, is central to Newton and trust-region optimization, uncertainty quantification, and simulation-based design. Classical finite differences (FD) remain popular but require delicate step-size tuning and can suffer from cancelation and noise amplification. Complex-step differentiation offers machine-precision gradients without subtractive cancelation, yet many second-derivative complex-step formulas reintroduce differencing. Hyper-dual numbers provide an algebraically principled alternative: by lifting real code to a four-component commutative nilpotent algebra, one obtains exact first and mixed second derivatives from a single evaluation, without finite differencing. This article gives a consolidated theoretical and experimental foundation for hyper-dual numbers. We formalize the algebra, prove exact Taylor truncation at second order, derive coefficient–extraction formulas for gradients and Hessians, and state assumptions for exactness, including limitations at non-smooth points and the need to branch on real parts. We present implementation patterns and language skeletons (C++, Python 3.11.5, MATLAB R2023b), and we provide fair numerical comparisons with FD, complex-step, and AD baselines. Stability tests under additive noise and ill-conditioning, together with runtime and memory profiling, demonstrate that hyper-dual coefficients are robust and reproducible in floating-point arithmetic, particularly for black-box codes where Hessian information is needed but differencing is fragile.

1. Introduction

Accurate and robust second-order derivatives are central to Newton-type optimization, model calibration, sensitivity analysis, and uncertainty quantification. In these contexts, the Hessian matrix—or selected second-order entries—governs convergence of Newton and quasi-Newton methods, informs trust-region or line-search strategies, and quantifies curvature in parameter spaces.

Classical finite-difference (FD) formulas approximate derivatives via Taylor expansions but suffer from two competing error sources: truncation error for large step sizes and subtractive cancelation for very small steps in floating-point arithmetic. Selecting a globally robust step size is notoriously problem-dependent, especially in the presence of measurement noise or ill-conditioned models. Complex-step differentiation addresses cancelation for first derivatives by embedding a purely imaginary perturbation and reading the imaginary component without differencing. However, standard recipes for second derivatives reintroduce differencing or multi-step identities, thereby re-exposing the method to subtraction and step sensitivity.

Hyper-dual numbers offer a principled remedy. By introducing two commuting nilpotent directions, one can encode both first and second derivatives as non-real coefficients in a four-component algebra. The Taylor lift of a $C^2$ function in this algebra truncates exactly at second order, so that the mixed nilpotent coefficient corresponds directly to the second derivative. The resulting method provides second derivatives to working precision without differencing and without step search, while simultaneously returning first derivatives.

Beyond classical finite differences, the complex-step technique provides machine-precision first derivatives without subtractive cancelation, yet standard second-derivative recipes typically reintroduce differencing or multi-step identities; see Squire and Trapp [1] for the seminal note, Fornberg [2] on FD weights and truncation error, and Higham [3] and Goldberg [4] for floating-point effects. Curvature is essential for Newton-type optimization and sensitivity analysis [5], while adjoint (reverse-mode) approaches remain the workhorse for large-scale gradient evaluation in simulation-based design [6]. Hyper-dual numbers complement these tools by delivering exact second derivatives without step tuning or differencing. In the broader landscape, algorithmic differentiation (AD) texts—Griewank and Walther [7] and Naumann [8]—and survey/review articles—Martins and Hwang [9], Baydin et al. [10], and Kim [11]—frame where hyper-dual numbers fit relative to forward, reverse, and adjoint approaches.

Compared with earlier expositions of hyper-dual numbers, we pursue the following two goals:

• Assumption transparency and algebraic clarity. We make the assumptions required for exactness explicit (smoothness, control-flow branching policies, floating-point limitations, and behavior at non-smooth points). We prove the exact second-order Taylor lift and multivariate chain rule in a concise, implementation-oriented style.
• Comprehensive experimental assessment. We supplement algebraic results with numerical stability tests under noise and ill-conditioning, runtime and memory profiling, and sensitivity analyses in FD step size and variable scaling. A curated set of benchmark problems—from low-dimensional analytic tests to classical optimization functions and higher-dimensional examples—illustrates accuracy and robustness. We also include minimal C++, Python 3.11.5, and MATLAB R2023b skeletons for rapid prototyping.

To keep the exposition cohesive, we consolidate the material into a small number of sections. Algebraic constructions and assumptions appear in Section 2 and Section 3, practical algorithms and implementation patterns in Section 4, numerical experiments and stability tests in Section 5, and cost/stability/comparative analysis in Section 6. Section 7 discusses extensions and integration with AD frameworks, and Section 8 concludes. Appendix A, Appendix B and Appendix C provide language-specific skeletons.

2. Hyper-Dual Algebra and Assumptions for Exactness

2.1. From Generalized Complex Numbers to Hyper-Duals

Numbers of the form $a+bE$ with multiplication

$$(a+bE)(c+dE)=ac+(ad+bc)E+bd{E}^2$$

unify several familiar algebras: complex numbers ($E^2=-1$), double numbers ($E^2=+1$), and dual numbers ($E^2=0$). In particular, dual numbers $a+b\epsilon$ with ${\epsilon }^2=0$ encode exact first derivatives: The coefficient of $\epsilon$ in $f(x+h\epsilon )$ equals $h{f}^{\prime }\left(x\right)$ for $C^1$ functions f.

Quaternions enlarge the space of imaginary directions but remain noncommutative, and the fact that squares of pure imaginary quaternions are real prevents the clean isolation of second-order information in a purely non-real component. What is needed instead are two commuting nilpotent directions whose product is nonzero but whose squares vanish.

2.2. Definition of the Hyper-Dual Algebra

Introduce commuting nilpotent units ${\epsilon }_1,{\epsilon }_2$ with

$${\epsilon }_1^2={\epsilon }_2^2={\left({\epsilon }_1{\epsilon }_2\right)}^2=0,{\epsilon }_1{\epsilon }_2={\epsilon }_2{\epsilon }_1\ne 0.$$

(1)

A hyper-dual number is an element

$$a=a_0+a_1{\epsilon }_1+a_2{\epsilon }_2+a_{12}{\epsilon }_1{\epsilon }_2,a_0,a_1,a_2,a_{12}\in \mathbb{R},$$

(2)

and the set of such elements forms a commutative, associative, unital algebra over $\mathbb{R}$, denoted $\mathbb{H}\mathbb{D}$.

Definition 1 

(Coefficient readout and real part). For $a\in \mathbb{H}\mathbb{D}$ written as above, we denote by ${\left[a\right]}_{{\epsilon }_1}$, ${\left[a\right]}_{{\epsilon }_2}$, and ${\left[a\right]}_{{\epsilon }_1{\epsilon }_2}$ the coefficients of ${\epsilon }_1$, ${\epsilon }_2$, and ${\epsilon }_1{\epsilon }_2$, respectively, and write

$$\mathrm{Re}\left(a\right):=a_0$$

for the real part. In code, control-flow branches are evaluated on $\mathrm{Re}\left(a\right)$ to preserve the real-valued path.

Arithmetic in $\mathbb{H}\mathbb{D}$ follows from distributivity and (1). If $a=a_0+a_1{\epsilon }_1+a_2{\epsilon }_2+a_{12}{\epsilon }_1{\epsilon }_2$ and $b=b_0+b_1{\epsilon }_1+b_2{\epsilon }_2+b_{12}{\epsilon }_1{\epsilon }_2$, then

$$\begin{array}{cc}\hfill a+b& =(a_0+b_0)+(a_1+b_1){\epsilon }_1+(a_2+b_2){\epsilon }_2+(a_{12}+b_{12}){\epsilon }_1{\epsilon }_2,\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill ab& =a_0{b}_0+(a_0{b}_1+a_1{b}_0){\epsilon }_1+(a_0{b}_2+a_2{b}_0){\epsilon }_2\hfill \\ \hfill & +(a_0{b}_{12}+a_1{b}_2+a_2{b}_1+a_{12}{b}_0){\epsilon }_1{\epsilon }_2.\hfill \end{array}$$

(4)

If $a_0\ne 0$, the inverse $a^{-1}$ exists and can be written explicitly.

Proposition 1 

(Inverse in $\mathbb{H}\mathbb{D}$). Let $a\in \mathbb{H}\mathbb{D}$ with $a_0\ne 0$. Then

$$a^{-1}={\displaystyle \frac{1}{a_0}}-{\displaystyle \frac{a_1}{a_0^2}}{\epsilon }_1-{\displaystyle \frac{a_2}{a_0^2}}{\epsilon }_2+\left({\displaystyle \frac{2a_1{a}_2}{a_0^3}}-{\displaystyle \frac{a_{12}}{a_0^2}}\right){\epsilon }_1{\epsilon }_2,$$

(5)

and the usual quotient rule holds under smooth lifting of scalar functions.

2.3. Assumptions for Exactness and Limitations at Non-Smooth Points

The algebraic statements in this paper are exact. To translate them into reproducible numerical claims in floating-point arithmetic, we make the following assumptions explicit and use them consistently throughout:

(A1)

Smoothness.We assume that $f:{\mathbb{R}}^n\to \mathbb{R}$ is $C^2$ in a neighborhood of the evaluation point. This guarantees the existence of mixed partial derivatives and Schwarz symmetry of the Hessian.

(A2)

Real-path control flow. All branching in code is evaluated on real parts, i.e., conditionals use $\mathrm{Re}\left(x\right)$ only. This ensures that the hyper-dual execution follows the same path as the real-valued code.

(A3)

Floating-point arithmetic. Numerical claims are qualified by floating-point arithmetic: hyper-dual coefficients are returned to working precision and are ultimately limited by the conditioning of the real evaluation and the underlying hardware/BLAS stack.

(A4)

Non-smooth points. At non-smooth points (e.g., absolute values, maximum/minimum, limiters), classical second derivatives may not exist or may depend on the approach direction. In such cases, hyper-dual coefficients represent the derivatives of the implemented procedure under (A2), not classical derivatives of an idealized continuous model. Smoothing or regularization should be applied when classical second derivatives are required.

A Practical Smoothing Remark

When a model contains non-smooth primitives such as $\left|x\right|$ or $max(a,b)$, hyper-dual coefficients reflect directional derivatives of the implemented code under the real-path policy (A2), which may deviate from classical second derivatives. In practice, a mild, smooth surrogate often restores a well-defined Hessian while preserving the intended physics. Two standard examples are

$$\begin{array}{cc}\hfill \left|x\right|& \approx {\varphi }_{\delta }\left(x\right):=\sqrt{x^2+{\delta }^2},\delta >0,\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill max(a,b)& \approx {\psi }_{\kappa }(a,b):={\displaystyle \frac{1}{\kappa }}log\left(e^{\kappa a}+e^{\kappa b}\right),\kappa \gg 1.\hfill \end{array}$$

(7)

Both ${\varphi }_{\delta }$ and ${\psi }_{\kappa }$ are $C^{\infty }$ and converge pointwise to the original non-smooth operators as $\delta \to 0$ or $\kappa \to \infty$, respectively. Hyper-dual evaluations of the smoothed code then return the exact Hessian of the surrogate, which is typically the quantity required in Newton-type solvers, sensitivity analysis, or curvature-based uncertainty quantification.

Assumption (A1) delineates the class of functions for which hyper-dual numbers return second derivatives corresponding to the Hessian of f. Assumptions (A2)–(A4) connect the algebra to real code, clarifying that deviations from theoretical exactness may arise from control-flow policies, non-smooth modeling choices, and floating-point effects rather than from the hyper-dual algebra itself.

Our algebraic construction is consistent with forward-mode AD via dual numbers and its object-oriented implementations; see Neidinger [12] for a clear tutorial. For engineering optimization contexts that motivated generalized numbers, including hyper-duals, see Fike and Alonso [13] and the follow-up implementation notes.

3. Exact Taylor Lift and Extraction of Derivatives

The exact truncation at bi-degree (1, 1) parallels Taylor-arithmetic viewpoints in AD; Griewank and Walther [7] and implementations that emphasize no differencing for coefficient extraction. For domain applications that require second-order sensitivity but maintain real-path control flow, Rehner et al. [14] generalize (hyper-)dual numbers to higher orders and vector-valued settings.

3.1. Scalar Case

Let $f:\mathbb{R}\to \mathbb{R}$ be $C^2$ near x and consider the hyper-dual perturbation

$$d=h_1{\epsilon }_1+h_2{\epsilon }_2,h_1,h_2\in \mathbb{R}.$$

Because all monomials of total degree $\ge 3$ in ${\epsilon }_1,{\epsilon }_2$ vanish, the Taylor expansion of f in $\mathbb{H}\mathbb{D}$ truncates exactly at degree two.

Theorem 1 

(Exact second-order lift in $\mathbb{H}\mathbb{D}$). Let $f\in C^2\left(\mathbb{R}\right)$ near x. Then

$$f(x+d)=f\left(x\right)+h_1{f}^{\prime }\left(x\right){\epsilon }_1+h_2{f}^{\prime }\left(x\right){\epsilon }_2+h_1{h}_2{f}^{″}\left(x\right){\epsilon }_1{\epsilon }_2.$$

(8)

Consequently,

$${\displaystyle \frac{{\left[f(x+d)\right]}_{{\epsilon }_1}}{h_1}}=f^{\prime }\left(x\right),{\displaystyle \frac{{\left[f(x+d)\right]}_{{\epsilon }_2}}{h_2}}=f^{\prime }\left(x\right),{\displaystyle \frac{{\left[f(x+d)\right]}_{{\epsilon }_1{\epsilon }_2}}{h_1{h}_2}}=f^{″}\left(x\right).$$

(9)

In floating-point arithmetic, these identities hold to working precision modulo roundoff and the conditioning of the real evaluation.

Proof. 

Write the classical Taylor series to order two and observe that all terms of degree $\ge 3$ vanish because every monomial contains either ${\epsilon }_1^2$, ${\epsilon }_2^2$, or ${\left({\epsilon }_1{\epsilon }_2\right)}^2$, which are zero by (1). Terms linear in d produce the ${\epsilon }_1$ and ${\epsilon }_2$ coefficients, and the unique degree-two mixed term yields the ${\epsilon }_1{\epsilon }_2$ coefficient.    □

3.2. Multivariate Case and Hessian Extraction

For $f:{\mathbb{R}}^n\to \mathbb{R}$ and standard basis vectors $e_i,e_j$, consider

$$x^{★}=x+h_1{\epsilon }_1{e}_i+h_2{\epsilon }_2{e}_j.$$

(10)

The multivariate Taylor expansion yields

$$f\left(x^{★}\right)=f\left(x\right)+h_1{\partial }_{x_i}f\left(x\right){\epsilon }_1+h_2{\partial }_{x_j}f\left(x\right){\epsilon }_2+h_1{h}_2{\partial }_{x_i{x}_j}^{2}f\left(x\right){\epsilon }_1{\epsilon }_2.$$

(11)

Thus a single hyper-dual evaluation recovers two first derivatives and one mixed second derivative,

$${\displaystyle \frac{{\left[f\left(x^{★}\right)\right]}_{{\epsilon }_1}}{h_1}}={\partial }_{x_i}f\left(x\right),{\displaystyle \frac{{\left[f\left(x^{★}\right)\right]}_{{\epsilon }_2}}{h_2}}={\partial }_{x_j}f\left(x\right),{\displaystyle \frac{{\left[f\left(x^{★}\right)\right]}_{{\epsilon }_1{\epsilon }_2}}{h_1{h}_2}}={\partial }_{x_i{x}_j}^{2}f\left(x\right).$$

(12)

Repeating this procedure for $1\le i\le j\le n$ and exploiting symmetry builds the full Hessian in $n(n+1)/2$ hyper-dual evaluations. Vector-mode implementations can propagate multiple directions concurrently, further reducing the required number of passes at a modest memory cost.

3.3. Chain Rule and Compositions

Let $g:{\mathbb{R}}^n\to {\mathbb{R}}^m$ and $f:{\mathbb{R}}^m\to \mathbb{R}$ be $C^2$ near x, and let $v,w\in {\mathbb{R}}^n$. Set

$$x^{★}=x+h_1{\epsilon }_{1}v+h_2{\epsilon }_{2}w$$

and define $F=f\circ g$. A direct calculation yields a mixed-direction chain rule for the second derivative of F.

Theorem 2 

(Mixed-direction chain rule in $\mathbb{H}\mathbb{D}$). Under assumptions (A1)–(A3), one has

$$\begin{array}{ccc}\hfill {\displaystyle \frac{{\left[(f\circ g)\left(x^{★}\right)\right]}_{{\epsilon }_1{\epsilon }_2}}{h_1{h}_2}}& =& v^{\top }{\nabla }^2(f\circ g)\left(x\right)w\hfill \\ \hfill & =& {\left(J_{g}v\right)}^{\top }{\nabla }^{2}f\left(g\left(x\right)\right)\left(J_{g}w\right)+\sum _{a=1}^m{\partial }_{a}f\left(g\left(x\right)\right)v^{\top }{\nabla }^2{g}_a\left(x\right)w,\hfill \end{array}$$

(13)

where $J_g$ is the Jacobian of g and $g_a$ denotes the a-th component of g.

Proof. 

Write

$$g(x+h_1{\epsilon }_{1}v+h_2{\epsilon }_{2}w)=g\left(x\right)+\Delta$$

with

$$\Delta =J_g\left(x\right)(h_1{\epsilon }_{1}v+h_2{\epsilon }_{2}w)+{\displaystyle \frac{1}{2}}{h}_1^2{\epsilon }_1^2{\nabla }^{2}g\left[[v,v]\right]+{\displaystyle \frac{1}{2}}{h}_2^2{\epsilon }_2^2{\nabla }^{2}g\left[[w,w]\right]+h_1{h}_2{\epsilon }_1{\epsilon }_2\Gamma ,$$

where the bilinear form $\Gamma$ has a-th component $v^{\top }{\nabla }^2{g}_a\left(x\right)w$. Nilpotency kills all ${\epsilon }_1^2$ and ${\epsilon }_2^2$ terms, so

$$\Delta =h_1{\epsilon }_1{J}_{g}v+h_2{\epsilon }_2{J}_{g}w+h_1{h}_2{\epsilon }_1{\epsilon }_2\Gamma .$$

Now lift f at $y=g\left(x\right)$:

$$f(y+\Delta )=f\left(y\right)+\nabla f{\left(y\right)}^{\top }\Delta +\frac{1}{2}{\Delta }^{\top }{\nabla }^{2}f\left(y\right)\Delta ,$$

and retain only coefficients of ${\epsilon }_1$, ${\epsilon }_2$, and ${\epsilon }_1{\epsilon }_2$. The linear terms give the first two identities. The mixed coefficient equals ${\left(J_{g}v\right)}^{\top }{\nabla }^{2}f\left(y\right)\left(J_{g}w\right)+\nabla f{\left(y\right)}^{\top }\Gamma$, which is exactly the stated chain rule for the mixed second derivative.    □

This identity confirms that hyper-dual numbers faithfully represent the Hessian of composed maps, provided that the real-path control flow is respected.

When only Hessian-vector products are required, reverse-on-forward strategies or the classic Pearlmutter [15] operator yield $Hv$ at a cost comparable to one gradient, complementing the hyper-dual readout used here for explicit Hessian entries.

Remark 1 

(Nonsmooth points and branch sensitivity). If f is not $C^2$ (e.g., contains $\left|x\right|$, max, or limiters), the second derivative may not exist or depend on the chosen real-path branch. Hyper-dual coefficients then reflect the implemented model under the stated branching policy, and users should smooth or regularize as appropriate.

Lemma 1 

(Schwarz symmetry via commuting nilpotents). If $f\in C^2\left({\mathbb{R}}^n\right)$, then

$${\left[f(x+h_1{\epsilon }_1{e}_i+h_2{\epsilon }_2{e}_j)\right]}_{{\epsilon }_1{\epsilon }_2}/\left(h_1{h}_2\right)={\partial }_{ij}^{2}f\left(x\right)={\partial }_{ji}^{2}f\left(x\right)$$

and is independent of assigning directions to ${\epsilon }_1,{\epsilon }_2$.

Proof. 

The lift is unique up to the commutation of ${\epsilon }_1$ and ${\epsilon }_2$, and the mixed coefficient encodes the bilinear form associated with the Hessian. Since $f\in C^2$, Schwarz’s theorem gives ${\partial }_{x_i{x}_j}^{2}f={\partial }_{x_j{x}_i}^{2}f$, hence the stated equality.    □

Corollary 1 

(Affine reparametrization). Let $\psi \left(x\right)=Ax+b$ with $A\in {\mathbb{R}}^{m\times n}$, $b\in {\mathbb{R}}^m$, and $F\left(x\right)=f\left(\psi \right(x\left)\right)$ for $f\in C^2\left({\mathbb{R}}^m\right)$. Then

$${\displaystyle \frac{{\left[F(x+h_1{\epsilon }_{1}v+h_2{\epsilon }_{2}w)\right]}_{{\epsilon }_1{\epsilon }_2}}{h_1{h}_2}}=v^{\top }\left(A^{\top }{\nabla }^{2}f(Ax+b)A\right)w,$$

i.e., the mixed coefficient transforms by the standard Hessian congruence under affine maps.

Proof. 

Apply Theorem 2 with $g=\psi$. Since ${\nabla }^2{\psi }_a\equiv 0$, only the first term

$${\left(J_{\psi }v\right)}^{\top }{\nabla }^{2}f\left(J_{\psi }w\right)={\left(Av\right)}^{\top }{\nabla }^{2}f(Ax+b)\left(Aw\right)$$

remains.    □

4. Implementation Patterns and Algorithms

4.1. Smooth Lifts of Elementary Functions

Analytic functions lift to $\mathbb{H}\mathbb{D}$ via their Taylor series. For $a=a_0+a_1{\epsilon }_1+a_2{\epsilon }_2+a_{12}{\epsilon }_1{\epsilon }_2$, one obtains, for example,

$$\begin{array}{cc}\hfill exp\left(a\right)& =exp\left(a_0\right)+a_{1}exp\left(a_0\right){\epsilon }_1+a_{2}exp\left(a_0\right){\epsilon }_2+exp\left(a_0\right)\left(a_{12}+a_1{a}_2\right){\epsilon }_1{\epsilon }_2,\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill sin\left(a\right)& =sin\left(a_0\right)+a_{1}cos\left(a_0\right){\epsilon }_1+a_{2}cos\left(a_0\right){\epsilon }_2+\left(a_{12}cos{a}_0-a_1{a}_{2}sin{a}_0\right){\epsilon }_1{\epsilon }_2,\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill cos\left(a\right)& =cos\left(a_0\right)-a_{1}sin\left(a_0\right){\epsilon }_1-a_{2}sin\left(a_0\right){\epsilon }_2-\left(a_{12}sin{a}_0+a_1{a}_{2}cos{a}_0\right){\epsilon }_1{\epsilon }_2,\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill log\left(a\right)& =log\left(a_0\right)+{\displaystyle \frac{a_1}{a_0}}{\epsilon }_1+{\displaystyle \frac{a_2}{a_0}}{\epsilon }_2+\left({\displaystyle \frac{a_{12}}{a_0}}-{\displaystyle \frac{a_1{a}_2}{a_0^2}}\right){\epsilon }_1{\epsilon }_2,a_0>0.\hfill \end{array}$$

(17)

These formulas can be used to overload elementary functions for $\mathbb{H}\mathbb{D}$ in C++, Python 3.11.5, or MATLAB R2023b (see Appendix A, Appendix B and Appendix C). Correct lifts of special functions such as tanh, $\sqrt{·}$, or Bessel functions follow the same pattern.

4.2. Hessian Extraction Algorithm

The following pseudocode summarizes a standard pattern for assembling a dense Hessian using hyper-dual numbers. A “real-converge, hyper-dual sweep” pattern is recommended when f embeds an iterative solver.

Algorithm 1 provides a minimal building block for sparse Hessians, Hessian-vector products, or curvature in selected coordinate pairs, reducing the number of hyper-dual evaluations when full dense Hessians are unnecessary.

In practice, $h_1$ and $h_2$ can be taken as 1 after nondimensionalizing variables, since the algebra is exact. Reporting the chosen $h_1,h_2$ nevertheless helps document the numeric ranges of coefficients.

4.3. Branching, Non-Smooth Operators, and Reproducibility

As emphasized in (A2), branch conditions should be evaluated on real parts only. Non-smooth operators (e.g., absolute value, hard limiters) require particular care: hyper-dual coefficients then reflect the differentiability properties of the discrete model. A practical reproducibility checklist includes the following:

• description of any non-smooth operators and the smoothing/regularization policies used.
• The scaling and units of variables and the nondimensionalization adopted.
• details of the linear algebra stack (factorizations, tolerances, BLAS/LAPACK backends).
• Seeds and distributions for randomized tests.

Implementation Cross-Reference

A unified MATLAB class implementing the hyper-dual algebra, real-path branching rule, and coefficient readout used throughout Section 5 is provided in Appendix C. This skeleton realizes Algorithms 1 and 2 in a single program class, directly supporting reproducibility and rapid prototyping across benchmarks.

Algorithm 1: Selected mixed second derivative or Hessian–vector product.

Require:  $f:{\mathbb{R}}^n\to \mathbb{R}$, point x, directions $v,w\in {\mathbb{R}}^n$ Ensure:  Mixed second directional derivative $v^{\top }{\nabla }^{2}f\left(x\right)w$ 1: Form a hyper–dual seed $X=x+{\epsilon }_{1}v+{\epsilon }_{2}w$ 2: Evaluate $Y=f\left(X\right)\in \mathbb{HD}$ 3: return  ${\left[Y\right]}_{{\epsilon }_1{\epsilon }_2}$

Algorithm 2: Dense Hessian via hyper-dual numbers.

Require: Function f: ${\mathbb{R}}^n\to \mathbb{R}$, point $x\in {\mathbb{R}}^n$, step sizes h1, h2  1: Initialize $H\in {\mathbb{R}}^{n\times n}$ to zero  2: for 1 ≤ i ≤ n do  3:  for i ≤ j ≤ n do  4:   Lift x to $X\in \mathbb{H}{\mathbb{D}}^n$ with real parts xk and nilpotent parts $$X_k=\{\begin{array}{ll}{x}_k+h_1{\epsilon }_1,& k=i,\\ x_k+h_2{\epsilon }_2,& k=j,\\ x_k,& \mathrm{otherwise};\end{array}$$  5:  Evaluate Y = $f(X)\in \mathbb{H}\mathbb{D}$  6:  Set Hij = Hji = ${\left[Y\right]}_{{\epsilon }_1{\epsilon }_2}/h_1{h}_2$  7: end for  8: end for  9: return H

5. Numerical Experiments

This section reorganizes and enlarges the example material into a coherent set of test problems. We proceed from analytic tests with known Hessians, through classical optimization benchmarks, to higher-dimensional and stiff problems. Each example compares hyper-dual results with FD and complex-step methods where applicable.

In this section we consider five representative classes of test problems, summarized in

Table 1. Overview of numerical examples in Section 5. Each example is chosen to highlight a specific aspect of accuracy, robustness, or scalability under the assumptions in Section 2.3.

Example Class	Dimension n	Analytic Hessian?	Purpose
Analytic scalar/bivariate tests	$1,2$	Yes	Validate exact coefficient readout; step invariance
Rosenbrock/Beale/Himmelblau	2	Yes	Coupling, nonlinearity, benchmark comparability
Quadratic-oscillatory high-n	10–200	Yes	Scaling and ill-conditioning sensitivity
Stiff solver/structured residual	problem-dependent	Partial/reference	Real-path branching and black-box robustness
Noise & step-sensitivity sweeps	1 (representative)	Yes	Quantify FD/CS U-shape vs. HD stability

, each chosen to emphasize a specific aspect of accuracy, robustness, or scalability under the assumptions of Section 2.3.

5.1. Analytic Test Functions

We begin with analytic functions whose Hessians are available in closed form. These tests isolate truncation versus roundoff and verify the step-invariance of hyper-dual coefficients under (A1)–(A3) before moving to coupled or ill-conditioned models.

Polynomial and Trigonometric Combinations

Consider first $f\left(x\right)={sin}^{3}x$ on $\mathbb{R}$. The exact second derivative is

$$f^{″}\left(x\right)=6sinx-9{sin}^{3}x.$$

Let $t_0=x+h_1{\epsilon }_1+h_2{\epsilon }_2$, $t_1=sin{t}_0$, $t_2=t_1^3$. A short calculation yields

$$\begin{array}{cc}\hfill t_2& ={sin}^{3}x+3h_{1}cosx{sin}^{2}x{\epsilon }_1+3h_{2}cosx{sin}^{2}x{\epsilon }_2\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill & -{\displaystyle \frac{3}{4}}{h}_1{h}_2\left(sinx-3sin3x\right){\epsilon }_1{\epsilon }_2,\hfill \end{array}$$

(19)

hence ${\left[t_2\right]}_{{\epsilon }_1{\epsilon }_2}/\left(h_1{h}_2\right)=f^{″}\left(x\right)$ to working precision.

Evaluating f at $x+h{\epsilon }_1+h{\epsilon }_2$ and collecting the ${\epsilon }_1{\epsilon }_2$ coefficient recovers $f^{″}\left(x\right)$ to machine precision across a wide range of x and h. Central FD approximations exhibit the expected $O\left(h^2\right)$ convergence followed by roundoff-dominated stagnation, while hyper-dual errors remain essentially flat at the level of machine precision.

As a two-variable test, consider

$$f(x,y)=exp\left(xy\right)+sin(x+y),$$

whose gradient and Hessian can be written in closed form. For $(x,y)\in {\mathbb{R}}^2$ one has

$$\nabla f=\left[\begin{array}{c}y{e}^{xy}+cos(x+y)\\ x{e}^{xy}+cos(x+y)\end{array}\right],$$

$${\nabla }^{2}f=\left[\begin{array}{cc}{y}^2{e}^{xy}-sin(x+y)& e^{xy}+xy{e}^{xy}-sin(x+y)\\ e^{xy}+xy{e}^{xy}-sin(x+y)& x^2{e}^{xy}-sin(x+y)\end{array}\right].$$

Evaluating f once at $x^{★}=(x,y)+h_1{\epsilon }_1{e}_i+h_2{\epsilon }_2{e}_j$ yields

$${\left[f\left(x^{★}\right)\right]}_{{\epsilon }_1}=h_1{\partial }_{x_i}f(x,y),{\left[f\left(x^{★}\right)\right]}_{{\epsilon }_2}=h_2{\partial }_{x_j}f(x,y),$$

and

$${\left[f\left(x^{★}\right)\right]}_{{\epsilon }_1{\epsilon }_2}=h_1{h}_2{\partial }_{x_i{x}_j}^{2}f(x,y).$$

For instance, choosing $(i,j)=(1,2)$ returns the mixed entry ${\partial }_{xy}^{2}f=e^{xy}+xy{e}^{xy}-sin(x+y)$ directly as the ${\epsilon }_1{\epsilon }_2$ coefficient divided by $h_1{h}_2$.

Hyper-dual evaluations at $(x,y)+h{\epsilon }_1{e}_i+h{\epsilon }_2{e}_j$ yield all second-order entries, which match analytic values up to machine precision. This example exercises both multiplicative coupling and trigonometric behavior.

Remark 2 

(Vector-mode implementations). The identities above extend verbatim when several ε-directions are propagated concurrently to assemble multiple rows/columns of the Hessian per pass. Memory increases linearly with the number of directions, while the arithmetic remains local and branch semantics are preserved by the real-part order.

5.2. Optimization Benchmarks: Rosenbrock, Beale, and Himmelblau

Classical optimization test functions exhibit narrow valleys, multiple minima, and strong variable coupling. Their Hessians govern Newton-type convergence, making them a canonical stress test for mixed second derivatives and method robustness.

To reflect typical optimization workloads, we include classical test functions

$$\begin{array}{cc}\hfill R(x,y)& =100{(y-x^2)}^2+{(1-x)}^2\left(\mathrm{Rosenbrock}\right),\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill B(x,y)& ={(1.5-x+xy)}^2+{(2.25-x+x{y}^2)}^2+{(2.625-x+x{y}^3)}^2\left(\mathrm{Beale}\right),\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill H(x,y)& ={(x^2+y-11)}^2+{(x+y^2-7)}^2\left(\mathrm{Himmelblau}\right).\hfill \end{array}$$

(22)

For each function and a set of representative points (including known minima and curved valley regions), we compare the following:

• Analytic Hessians.
• Hyper-dual Hessians.
• Central FD Hessians (with tuned step sizes).
• Complex-step second-derivative formulas where applicable.

Across all cases, hyper-dual results agree with analytic Hessians to within a small multiple of machine precision, while FD approximations require problem-specific step tuning to avoid either truncation error or cancelation. In regions of high curvature, FD errors are particularly sensitive to step choice, whereas hyper-dual coefficients remain stable.

5.3. Higher-Dimensional and Stiff Problems

Practical simulation objectives are often high-dimensional and ill-conditioned or contain stiff solver components. This subsection evaluates whether hyper-dual exactness and stability persist when curvature scales vary strongly across coordinates.

To assess scalability, we consider higher-dimensional functions such as

$$f\left(x\right)={\displaystyle \frac{1}{2}}{x}^{\top }Qx+c^{\top }x+\sum _{k=1}^{n}sin\left({\alpha }_k^{\top }x\right),$$

where Q is symmetric positive-definite, the ${\alpha }_k$ are random direction vectors, and n ranges from moderate to large (e.g., n = 10, 50, 100, 200). The true Hessian is Q plus a rank-n trigonometric contribution. Hyper-dual Hessians match this structure exactly and yield accurate entries without step search.

As a representative stiff problem, one may embed hyper-dual numbers into a time-integration code for a stiff ODE and extract sensitivities of a final-time quantity of interest with respect to parameters. Provided that the underlying integrator uses real-part branching only, hyper-dual coefficients deliver second derivatives while the stiffness is handled by the real solver.

5.3.1. Structured PDE–Residual Benchmark (Toy Poisson Model)

To mimic black-box simulation structure, we consider a discrete Poisson residual $G(u;x)=Au-b\left(x\right)$ where A is an SPD stiffness matrix, and $b\left(x\right)$ depends smoothly on parameters $x\in {\mathbb{R}}^n$. The quantity of interest is

$$F\left(x\right)={\displaystyle \frac{1}{2}}{\parallel u\left(x\right)-u_{\mathrm{obs}}\parallel }^2,\mathrm{where}u\left(x\right)\mathrm{solves}G(u;x)=0.$$

Following the real-converge, hyper-dual sweep strategy, we first solve for the real state $u\left(x\right)$ to tolerance, then re-evaluate the residual solve with hyper-dual seeds in x while enforcing real-part branching in the linear/nonlinear solver. The ${\epsilon }_1{\epsilon }_2$ coefficient yields curvature of F with respect to x without differencing across solver iterations, providing a realistic stress test for ill-conditioning and control-flow sensitivity.

5.3.2. Neural-Network Loss Benchmark (Small MLP)

As a second structured example, we compute curvature of a small multilayer perceptron (MLP) training loss. Let $\theta \in {\mathbb{R}}^n$ collect all weights and biases, and define

$$L\left(\theta \right)={\displaystyle \frac{1}{m}}\sum _{i=1}^m\mathcal{l}\left({\mathrm{MLP}}_{\theta }\left(x_i\right),y_i\right),$$

with smooth activation (e.g., tanh) to satisfy (A1). Hyper-dual lifting of $\theta$ yields exact mixed second derivatives of L at a given training iterate in $n(n+1)/2$ evaluations, providing a high-dimensional, highly coupled benchmark where FD step tuning is particularly fragile.

5.4. Numerical Stability Under Noise and Ill-Conditioning

Table 2. Noise robustness and step-size sensitivity for second-derivative estimation. We report the mean relative error ($\overline{\mathrm{RelErr}}$) and standard deviation ($\mathrm{Std}$) over $N_{\mathrm{MC}}$ Monte Carlo trials for the scalar test $f\left(x\right)=exp\left(x\right)sinx$ at $x=x_0$. Additive noise is modeled by $\tilde{f}\left(x\right)=f\left(x\right)+\eta$, $\eta \sim \mathcal{N}(0,{\sigma }^2)$. FD uses the central second-difference formula, CS uses a standard second-derivative complex-step identity (which reintroduces differencing), and HD reads the ${\epsilon }_1{\epsilon }_2$ coefficient from a single hyper-dual evaluation.

Noise Level	Step h	FD (Central 2nd)		CS (2nd)		HD (Coef-Based)
$\mathbf{\sigma }$	(Sweep)	$\overline{\mathrm{RelErr}}$	$\mathrm{Std}$	$\overline{\mathrm{RelErr}}$	$\mathrm{Std}$	$\overline{\mathrm{RelErr}}$	$\mathrm{Std}$
${10}^{-8}$	${10}^{-1}$	$3.2\times {10}^{-4}$	$1.1\times {10}^{-6}$	$8.4\times {10}^{-5}$	$3.0\times {10}^{-6}$	$1.1\times {10}^{-8}$	$0.7\times {10}^{-8}$
	${10}^{-3}$	$1.6\times {10}^{-2}$	$1.0\times {10}^{-2}$	$5.3\times {10}^{-3}$	$3.4\times {10}^{-3}$	$1.2\times {10}^{-8}$	$0.8\times {10}^{-8}$
	${10}^{-6}$	$1.2\times {10}^3$	$9.0\times {10}^2$	$4.1\times {10}^2$	$3.0\times {10}^2$	$1.2\times {10}^{-8}$	$0.8\times {10}^{-8}$
${10}^{-6}$	${10}^{-1}$	$4.4\times {10}^{-4}$	$1.0\times {10}^{-4}$	$1.3\times {10}^{-4}$	$4.1\times {10}^{-5}$	$1.2\times {10}^{-6}$	$0.8\times {10}^{-6}$
	${10}^{-3}$	$8.1\times {10}^{-1}$	$6.2\times {10}^{-1}$	$3.1\times {10}^{-1}$	$2.2\times {10}^{-1}$	$1.2\times {10}^{-6}$	$0.8\times {10}^{-6}$
	${10}^{-6}$	$2.0\times {10}^5$	$1.5\times {10}^5$	$6.2\times {10}^4$	$4.4\times {10}^4$	$1.3\times {10}^{-6}$	$0.9\times {10}^{-6}$

Protocol. $N_{\mathrm{MC}}=200$ trials, pseudo-random seed fixed to 2025, $x_0=0.7$, and relative error defined as $\mathrm{RelErr}=|{\widehat{f}}^{″}-f^{″}|/max(1,\left|f^{″}\right|)$. The data illustrate the characteristic U-shaped FD/CS sensitivity and the near h-invariance of hyper-dual coefficients under noise, consistent with the stability discussion in Section 5.4.

summarizes the noise robustness and step-size sensitivity of the three second-derivative estimators (FD, CS, and HD) for the scalar test function $f\left(x\right)=exp\left(x\right)sinx$ at $x_0$.

5.4.1. Experimental Protocol

Unless stated otherwise, stability results use a fixed evaluation point $x=x_0$, Gaussian additive noise $\eta \sim \mathcal{N}(0,{\sigma }^2)$ with $\sigma \in \left\{{\mathit{10}}^{-\mathit{8}},{\mathit{10}}^{-\mathit{6}},{\mathit{10}}^{-\mathit{4}},{\mathit{10}}^{-\mathit{2}}\right\}$, and $N_{\mathrm{MC}}=\mathit{500}$ independent trials with a fixed pseudo-random seed. Errors are reported as relative errors $\mathrm{RelErr}=|{\widehat{f}}^{″}-f^{″}|/max(1,\left|f^{″}\right|)$, together with $\mathrm{Std}$ over trials. These settings are chosen to mirror typical measurement/Monte Carlo uncertainty while preserving reproducibility.

5.4.2. Additive Noise in Function Values

Let $\tilde{f}\left(x\right)=f\left(x\right)+\eta$, where $\eta$ is a mean-zero random perturbation with variance ${\sigma }^2$ representing measurement or Monte Carlo noise. For a scalar test function (e.g., the polynomial-trigonometric example above), we compare the following:

• Central FD approximations to $f^{″}\left(x\right)$ using $\tilde{f}$.
• Hyper-dual extraction of $f^{″}\left(x\right)$ using a single evaluation of $\tilde{f}$ in $\mathbb{H}\mathbb{D}$.

FD formulas amplify noise differently depending on h: for small h, subtraction of nearly equal noisy values leads to large variance in the derivative; for large h, truncation error dominates. Hyper-dual coefficients, by contrast, avoid explicit differencing and therefore inherit noise only through the real evaluation itself. In numerical experiments, the variance of hyper-dual second-derivative estimates remains essentially independent of h, whereas FD estimates exhibit a characteristic U-shaped error curve as a function of h.

5.4.3. Complex-Step Under Noise

While complex-step differentiation is cancelation-free for first derivatives, standard second-derivative complex-step identities rely on a differencing structure. Consequently, under the additive noise model $\tilde{f}=f+\eta$, the variance of CS second-derivative estimates scales similarly to central FD, with amplification proportional to $\sigma /h^2$. This explains why CS exhibits the same qualitative U-shaped error-versus-h pattern in noisy settings, whereas the hyper-dual estimate remains nearly h-invariant once the real path is fixed.

Figure 1 displays the mean relative error of FD, CS, and HD second-derivative estimates as a function of the step size h under additive Gaussian noise. Panel (a) corresponds to $\sigma ={10}^{-8}$ and panel (b) to $\sigma ={10}^{-6}$.

5.4.4. Ill-Conditioned Problems and Scaling

To probe ill-conditioning, we consider functions whose Hessians have widely separated eigenvalues. Hyper-dual evaluations reproduce the ill-conditioned Hessian exactly (up to roundoff), while FD formulas struggle to resolve small curvature directions in the presence of large ones. We additionally perform scaling experiments: variables are rescaled by diagonal matrices, and FD step sizes are either kept fixed or adapted. Hyper-dual coefficients are invariant under such rescalings (modulo floating-point effects), while FD approximations are highly sensitive to both variable scaling and step-size heuristics.

These tests confirm that the algebraic advantages of hyper-dual numbers translate into practical robustness when function values are noisy or when problems are ill-conditioned.

6. Cost, Floating-Point Effects, and Comparative Analysis

We emphasize ratios and scaling trends rather than absolute timings, since wall-clock performance varies with compilers, BLAS/LAPACK backends, and hardware. The guiding theme is that hyper-duals trade modest per-call arithmetic overhead for a large reduction in expensive target evaluations while avoiding differencing instabilities.

6.1. Operation Counts, Runtime, and Memory Profiling

6.1.1. Operation Counts

A hyper-dual scalar stores four real coefficients; addition is componentwise, while multiplication expands to a small fixed number of real flops. For f: ${\mathbb{R}}^n\to \mathbb{R}$, dense Hessian assembly needs $n(n+1)/2$ hyper-dual evaluations, versus ${(n+1)}^2$ forward FD and $2n(n+2)$ central FD evaluations. Hence, when each f evaluation is costly (e.g., embedded PDE/CFD solves), the reduced call count dominates overall time.

6.1.2. Runtime and Memory Profiling

Per-scalar hyper-dual arithmetic is slower than real arithmetic, but the overhead is bounded by a modest constant factor. Empirically, the reduced evaluation count often yields net speedups, especially in simulation-driven optimization where f calls are the bottleneck. Memory overhead is predictable and linear: arrays require four times real storage, with similar access patterns.

6.1.3. Integration Pattern (“Real Converge, HD Sweep”)

To isolate solver cost from differentiation cost, we recommend (i) converge the state using purely real arithmetic, then (ii) perform one hyper-dual sweep of $n(n+1)/2$ calls to assemble ${\nabla }^{2}f$. This preserves the exact second-derivative readout while keeping iterative loops inexpensive.

6.1.4. Algebraic Foundation (Concise)

Hyper-duals live in

$$\mathbb{HD}=\mathbb{R}[{\epsilon }_1,{\epsilon }_2]/({\epsilon }_1^2,{\epsilon }_2^2,{\left({\epsilon }_1{\epsilon }_2\right)}^2),$$

so any lifted $C^2$ function truncates exactly at bi-degree $(1,1)$:

$$f(x+{\epsilon }_1{h}_{1}v+{\epsilon }_2{h}_{2}w)=f\left(x\right)+h_1{\partial }_{v}f{\epsilon }_1+h_2{\partial }_{w}f{\epsilon }_2+h_1{h}_2{\partial }_{vw}f{\epsilon }_1{\epsilon }_2.$$

Because lifting is an algebra homomorphism, compositions automatically satisfy the chain rule. This is the core reason mixed second derivatives can be read off from the ${\epsilon }_1{\epsilon }_2$ coefficient without differencing.

6.1.5. Implicit Maps

For $G(x,y)=0$ defining $y=\varphi \left(x\right)$ with ${\partial }_{y}G\ne 0$, lifting x to hyper-dual form, and propagating through a real-path Newton/fixed-point solve yields ${\varphi }^{\prime }\left(x_0\right)$ from ${\epsilon }_{1,2}$ coefficients and ${\varphi }^{″}\left(x_0\right)$ from ${\epsilon }_1{\epsilon }_2$. This enables curvature extraction in implicitly defined models (e.g., shock-angle or constraint solves) in one stabilized pass.

6.1.6. Wall-Clock Scaling and Crossover Regimes

To make scalability transparent beyond operation counts, we distinguish two regimes: (i) cheap objectives, where runtime is dominated by scalar arithmetic, and (ii) expensive objectives, where runtime is dominated by the real evaluation path (e.g., PDE/CFD solves). In regime (ii), the reduction from ${(n+1)}^2$ (FD) to $n(n+1)/2$ (HD) evaluations yields a clear wall-clock advantage, and a crossover dimension $n_{★}$ is typically observed where HD becomes faster end-to-end despite per-call overhead.

6.2. Floating-Point Effects and Deviation from Theoretical Exactness

6.2.1. Numerical Deviation Sources

Hyper-dual differentiation is symbolically exact; finite-precision errors arise only from (i) roundoff in the real evaluation path, (ii) roundoff accumulation in coefficient arithmetic, and (iii) ill-conditioning of f at the evaluation point. Crucially, hyper-duals introduce no difference quotients, so they do not add subtractive cancelation beyond what already exists in the real code.

Figure 2 reports measured wall-clock time versus n in both regimes to illustrate this crossover.

6.2.2. Quantifying Floating-Point Deviation

To quantify departures from theoretical exactness, we sweep the conditioning of the real path and the arithmetic precision. Specifically, for quadratic-oscillatory benchmarks with SPD Q, we vary $\kappa \left(Q\right)={\lambda }_{max}/{\lambda }_{min}$ over several orders of magnitude, and measure the relative error between analytic and hyper-dual Hessians in both float32 and float64.

Table 3. Hessian accuracy versus conditioning and precision for FD, CS, and HD. Entries report the relative Frobenius error $\parallel H_{\mathrm{approx}}-H_{\mathrm{exact}}{\parallel }_F/{\parallel H_{\mathrm{exact}}\parallel }_F$ for a quadratic-oscillatory benchmark with varying condition number $\kappa \left(Q\right)$.

$\mathit{\kappa }\left(\mathit{Q}\right)$	FD		CS		HD
	float32	float64	float32	float64	float32	float64
${10}^2$	$3.5\times {10}^{-3}$	$7.0\times {10}^{-5}$	$8.0\times {10}^{-4}$	$1.5\times {10}^{-5}$	$4.0\times {10}^{-7}$	$2.0\times {10}^{-14}$
${10}^4$	$1.2\times {10}^{-2}$	$4.0\times {10}^{-4}$	$3.5\times {10}^{-3}$	$9.0\times {10}^{-5}$	$6.0\times {10}^{-7}$	$4.0\times {10}^{-14}$
${10}^6$	$4.5\times {10}^{-2}$	$1.7\times {10}^{-3}$	$1.2\times {10}^{-2}$	$3.8\times {10}^{-4}$	$1.1\times {10}^{-6}$	$1.0\times {10}^{-13}$
${10}^8$	$1.8\times {10}^{-1}$	$7.0\times {10}^{-3}$	$4.0\times {10}^{-2}$	$1.6\times {10}^{-3}$	$3.0\times {10}^{-6}$	$3.0\times {10}^{-13}$

summarizes the resulting accuracy trends, confirming that hyper-dual errors track the conditioning of the real evaluation rather than any step-selection artifact.

Figure 3 summarizes evaluation counts versus dimension and target Hessian density;

Table 4. Arithmetic/memory footprint for hyper-dual scalars (dense).

Quantity	Real	Hyper-Dual
Scalar storage	1	4
Addition flops	1	4 (componentwise)
Multiplication flops	1	≈9 mult. + 5 add.
Array of size N	N reals	$4N$ reals

records arithmetic/memory footprints.

6.2.3. Scaling and Nondimensionalization

To keep coefficient magnitudes well inside floating-point ranges, scale inputs so typical nondimensional variables are $O\left(1\right)$. This improves numeric robustness of $(a_1,a_2,a_{12})$ without affecting algebraic exactness.

6.2.4. h-Sweep Sanity Check

On smooth benchmarks, the hyper-dual second derivative remains essentially flat near machine precision across wide h ranges once the real-path is resolved, whereas FD shows the classical truncation-to-roundoff transition. This behavior confirms that accuracy is governed by real-path conditioning, not by step selection.

6.3. Sensitivity to FD Step Size and Variable Scaling

6.3.1. FD Step-Size Policy Used Throughout

All FD baselines use the central second-difference formula with a uniform diagonal step policy

$$h_i=\eta max(1,\left|x_i\right|),$$

where $\eta$ is swept on a logarithmic grid $\eta \in \{{10}^{p_1},{10}^{p_2},\dots ,{10}^{p_k}\}$. This standard heuristic equalizes absolute and relative perturbations across variables and is commonly recommended in numerical differentiation practice. Reporting the full $\eta$-sweep reveals the truncation-to-roundoff transition and provides a fair, reproducible comparison against hyper-dual and complex-step references.

We assess robustness by sweeping FD steps over many orders of magnitude and by rescaling variables diagonally. FD accuracy depends delicately on both h and units; a step size that is optimal in one scale regime can fail in another. Hyper-dual values provide a stable reference across these sweeps, enabling clear attribution of FD error to step and scaling choices.

6.3.2. Verification Examples

Rosenbrock.

$$R(x,y)=100{(y-x^2)}^2+{(1-x)}^2,{\nabla }^{2}R=\left(\begin{array}{cc}1200x^2-400y+2& -400x\\ -400x& 200\end{array}\right).$$

One hyper-dual call at $(x+{\epsilon }_{1}h,y+{\epsilon }_{2}h)$ recovers ${\partial }_{xy}R=-400x$ exactly from the ${\epsilon }_1{\epsilon }_2$ coefficient.

Beale and Himmelblau.

$$B(x,y)={(1.5-x+xy)}^2+{(2.25-x+x{y}^2)}^2+{(2.625-x+x{y}^3)}^2,$$

$$H(x,y)={(x^2+y-11)}^2+{(x+y^2-7)}^2.$$

Across multiple points and minima, hyper-dual mixed derivatives match analytic references, while FD varies with h and scaling.

Controlled polynomial. For $p(x,y)=x^3+2x^{2}y+x{y}^2+y^3$, ${\partial }_{xy}p=4x+4y$ is recovered identically.

6.3.3. Noise Sensitivity

Because FD relies on differencing nearby values, it amplifies additive evaluation noise. Hyper-dual coefficients do not, making them preferable for noisy surrogates or ill-conditioned simulations.

6.3.4. Reporting Checklist (Condensed)

State nondimensional scales; list hardware, compiler, and linear algebra backends; identify any nonsmooth operators or smoothing policies; provide analytic baselines and error norms; and document random seeds and which kernels remained real versus hyper-dual.

6.4. Method Selection in Practice

6.4.1. Positioning

Complex-step gradients are attractive for first derivatives, but second-order formulas often combine multiple evaluations and may reintroduce subtraction. Hyper-duals avoid explicit differencing and supply mixed terms directly, with predictable overhead.

6.4.2. Practical Guidance

• Gradients only, very high n: complex-step or reverse AD.
• Dense Hessians or selected entries in legacy codes: hyper-duals.
• PDE/CFD optimization: reverse AD for gradients + hyper-dual sweep for curvature at convergence.
• Nonsmooth models: use smoothing or interpret coefficients diagnostically at kinks.

6.4.3. Implementation Pattern and Hessian Extraction

Wrap only assembly layers with hyper-dual types, keeping heavy solvers real. Mixed entries follow from a single evaluation:

function Hessian_HD(f, x):

    n = length(x); H = zeros(n,n)

    for j in 1..n:

        for i in 1..j:

            xHD = lift_to_HD(x)

            xHD[i] += eps1*h; xHD[j] += eps2*h

            yHD = f(xHD)

            hij = coeff_eps1eps2(yHD)/(h*h)

            H[i,j] = hij; H[j,i] = hij

    return H

6.4.4. Pitfalls

Branch only on real parts; avoid or smooth $\left|x\right|$, max/min, limiters at kinks; and ensure lifted special functions respect bi-degree $(1,1)$ truncation. If third-party black-box kernels break dual semantics internally, confine hyper-dual propagation to the surrounding real assembly.

7. Extensions and Outlook

7.1. Higher-Order Derivatives

Hyper-dual numbers can be generalized to higher-order nilpotent algebras that encode third and higher derivatives via additional commuting nilpotent directions or via truncated polynomial algebras. Such constructions retain many of the algebraic advantages of hyper-dual numbers but increase the number of coefficients per scalar. The design of efficient higher-order algebras that remain numerically stable and implementation-friendly is an active topic of research.

Sketch: Third Derivatives via Tri-Dual Extension

A direct higher-order extension introduces three commuting nilpotent units ${\epsilon }_1,{\epsilon }_2,{\epsilon }_3$ with ${\epsilon }_i^2=0$, encoding third mixed derivatives in the ${\epsilon }_1{\epsilon }_2{\epsilon }_3$ coefficient. For $f\in C^3$ and seed $x^{★}=x+{\epsilon }_1{h}_{1}v+{\epsilon }_2{h}_{2}w+{\epsilon }_3{h}_{3}z$, the Taylor lift truncates exactly at tri-degree $(1,1,1)$, yielding

$${\displaystyle \frac{{\left[f\left(x^{★}\right)\right]}_{{\epsilon }_1{\epsilon }_2{\epsilon }_3}}{h_1{h}_2{h}_3}}={\partial }_{vwz}^{3}f\left(x\right).$$

Although the coefficient count grows combinatorially, this construction remains purely algebraic and can be implemented by extending the scalar type in Appendix C. In many applications, mixed third derivatives are required only in a few directions; a direction-seeded tri-dual sweep then remains computationally feasible.

7.2. Vector-Valued Functions and Jacobians

For vector-valued maps $f:{\mathbb{R}}^n\to {\mathbb{R}}^m$, hyper-dual numbers can be used to obtain Jacobians and Hessian-vector products. In particular, propagating multiple directions in vector-mode implementations allows one to recover several columns of the Jacobian or Hessian in a single pass. Coupling hyper-dual arithmetic with reverse-mode AD (“forward-on-reverse” strategies) yields efficient Hessian-vector computations in high dimensions.

7.3. Integration with AD Frameworks

Hyper-dual numbers complement rather than replace existing AD tools. They are particularly attractive when the following apply:

• Reverse-mode pipelines are impractical to retrofit into legacy codes.
• Only modest numbers of variables or Hessian entries are required.
• Reproducibility and robustness under noise are prioritized over absolute speed.

In AD frameworks, hyper-dual types can serve as local scalar types in tape-based computations, enabling hybrid differentiation strategies.

Discussion Bridge

Section 6.4 and Section 7 collectively emphasize that hyper-dual numbers are not a competitor to complex-step or AD, but a complementary tool that reliably supplies second-order information when differencing is fragile or retrofitting reverse-mode is impractical. With these practical and extensibility considerations in place, we now summarize the main contributions and implications.

8. Conclusions

We have provided a consolidated theoretical and experimental account of hyper-dual numbers for exact second derivatives. Building on a commutative, four-component nilpotent algebra, we proved the exact truncation of the Taylor lift at second order, derived coefficient-extraction formulas for gradients and Hessians, and stated clearly the assumptions under which these results hold, including the behavior at non-smooth points and the necessity of consistent real-part branching.

Practical Takeaways

For readers seeking immediate guidance, the main lessons of this study are as follows:

• Hyper-dual numbers deliver mixed second derivatives to working precision without step tuning, provided $f\in C^2$ and real-path branching is enforced.
• In noisy or ill-conditioned regimes, FD and CS second derivatives inherit $\sigma /h^2$ amplification, whereas hyper-dual coefficients remain essentially h-invariant and scale primarily with the real-path conditioning.
• For dense Hessians (or selected mixed entries) in legacy or black-box codes, hyper-dual sweeps offer a robust, reproducible alternative to differencing.
• When only gradients are needed at very large n, complex-step or reverse AD remains preferable; hyper-duals are most beneficial once curvature is required.
• At non-smooth points, hyper-dual outputs diagnose the discrete model’s directional behavior; smoothing surrogates restore classical Hessians if needed.

On the practical side, we presented implementation patterns and language-specific skeletons (C++, Python, and MATLAB) for rapid prototyping, reorganized and expanded numerical examples to include analytic test functions, classical optimization benchmarks, and higher-dimensional problems, and carried out stability tests under noise and ill-conditioning. Runtime and memory profiling, together with sensitivity tests in FD step size and variable scaling, highlighted the reproducible robustness of hyper-dual coefficients relative to FD and complex-step methods and clarified when evaluation-count savings outweigh per-scalar algebraic overhead.

From a methodological standpoint, hyper-dual numbers offer an algebraically principled and computationally reliable route to second-order information in black-box codes. When coupled with real-path convergence and careful API boundaries—keeping heavy linear-algebra kernels in reals while wrapping only residual/functional assembly in hyper-duals—they reduce the trial-and-error tuning common in FD workflows. In this sense, hyper-duals naturally complement complex-step differentiation for gradients and reverse-mode/adjoint AD for large-scale problems: complex-step remains attractive for first derivatives and reverse AD for very high-dimensional gradients, while hyper-duals provide differencing-free, machine-precision Hessians or selected mixed entries with minimal refactoring of legacy code.

Our contribution is a synthesis and systematization of the hyper-dual approach: we collected the core algebraic facts, made assumptions explicit, and provided implementation recipes and fair cost/accuracy comparisons under clearly stated conditions. These results connect and extend early accounts that popularized hyper-duals for exact second derivatives [13,16], and they align with recent demonstrations of hyper-dual formalization and applications beyond aerospace and optimization, including thermodynamics [14] and formal methods/verification [17]. For a broader applied, textbook-level perspective on derivative-based design workflows, we refer to Martins and Ning [18].

Overall, hyper-dual numbers combine the numerical stability associated with complex-step differentiation and the composability of dual-number lifting, exposing both first- and second-order sensitivities in a single evaluation. We expect them to be particularly useful in optimization, uncertainty quantification, simulation-based design, and other settings where curvature information is valuable but differencing is fragile. Promising future directions include higher-order generalizations, GPU-friendly and vector-mode implementations, and further domain-specific deployments in aerodynamics, materials modeling, and machine-learning optimization.