An Analysis of Vectorised Automatic Differentiation for Statistical Applications

Abstract

Automatic differentiation (AD) is a general method for computing exact derivatives in complex sensitivity analyses and optimisation tasks, particularly when closed-form solutions are unavailable and traditional analytical or numerical methods fall short. This paper introduces a vectorised formulation of AD grounded in matrix calculus. It aligns naturally with the matrix-oriented style prevalent in statistics, supports convenient implementations, and takes advantage of sparse matrix representation and other high-level optimisation techniques that are not available in the scalar counterpart. Our formulation is well-suited to high-dimensional statistical applications, where finite differences (FD) scale poorly due to the need to repeat computations for each input dimension, resulting in significant overhead, and is advantageous in simulation-intensive settings—such as Markov Chain Monte Carlo (MCMC)-based inference—where FD requires repeated sampling and multiple function evaluations, while AD can compute exact derivatives in a single pass, substantially reducing computational cost. Numerical studies are presented to demonstrate the efficacy and speed of the proposed AD method compared with FD schemes.

1. Introduction

Automatic differentiation (AD) has become a foundational tool in modern statistical computing, enabling efficient and exact gradient computation in a wide range of applications—from parameter estimation [1,2] and sensitivity analysis [3,4,5] to simulation-based methods such as variational inference and Markov-Chain Monte Carlo (MCMC) inference [6,7,8,9,10,11]. Its widespread adoption is evident in major software ecosystems such as PyTorch [12], TensorFlow [13], Stan [14], and Julia [7], where AD powers both machine learning workflows and traditional statistical methods.

AD works by transforming a program that computes the value of a function into one that also computes its derivatives by systematically applying the chain rule to elementary operations. This allows AD to compute derivatives with machine-level precision and minimal overhead, avoiding truncation and round-off errors and eliminating the need for repeated function evaluations, a known bottleneck in numerical differentiation, especially in high-dimensional problems [8,15,16]. While symbolic differentiation can provide exact derivatives, it requires closed-form expressions and cannot handle procedural logic (e.g., if-else statements and for-loops) or stochastic elements such as random number generation or Monte Carlo simulations. AD bridges this gap, offering a robust and general-purpose solution for derivative computation.

Early implementations of AD relied on operator overloading [17] and source code translation [18] techniques that, while powerful, had notable limitations. Operator overloading incurs significant runtime overhead and is inherently local, recording operations as they occur without access to global program structure or opportunities for optimisation. In contrast, compiler-based systems transform entire programs automatically but often make it difficult to selectively extract intermediate values for debugging or inspection. Modern AD frameworks improve on these predecessors by adopting either eager execution (as in PyTorch 1 and 2) or static computational graphs (as in TensorFlow 1). These approaches offer greater flexibility, traceability, and support for modular development and deep introspection. However, the eager mode requires users to structure their code in specific ways; for example, making explicit calls such as backward() and zero_grad() in PyTorch can appear rigid and error-prone. Moreover, because it operates step-by-step, the eager approach often fails to exploit the broader structure of computations, such as block matrix operations, thereby missing optimisation opportunities. Conversely, while static graph systems are more declarative and amenable to global analysis, they can struggle with dynamic control flow and runtime-dependent logic. In both paradigms, the need to conform to AD-specific programming idioms often shifts user attention away from the statistical problem itself and toward the mechanics of the AD system.

In this work, we present a vectorised formulation of AD grounded in the matrix calculus of [19], designed to align more naturally with the matrix-oriented style prevalent in the field of statistics and the statistical programming language R. Our approach mirrors the derivation style of analytical work, enabling clearer and more intuitive implementations. It also exposes opportunities for high-level optimisation, including the use of sparse matrix representations and block-wise computations, features often inaccessible in bottom-up, scalar-based AD systems. This formulation supports transparent complexity analysis and efficient implementations, particularly in settings involving Kronecker products. It also enables fully automatic workflows, akin to source code translation techniques, while preserving the ability to inspect intermediate variables as in eager execution and graph-based systems.

We introduce a complete set of matrix calculus rules for building an AD system tailored to statistical applications, including operations for random variable simulations and structural transformations, many of which are undocumented in the existing literature. We also introduce the sparse representation of transformation matrices and discuss a range of optimisation techniques applied to the AD system to achieve significant performance gains in practice, which we demonstrate through comparisons with finite differences (FD). As an illustration, we apply the proposed methods using a real data example: a factor model estimated using simulated maximum likelihood, a setting commonly encountered when modelling dependence structures in complex data. The numerical results confirm the computational advantage of the proposed vectorised AD, particularly for simulation-intensive functions where FD incurs unnecessary repeated calculations.

The remainder of the paper is organised as follows: Section 2 introduces the core mechanism of the AD system and presents the full set of matrix calculus rules. Section 3 discusses optimisation strategies and evaluates computational performance, while Section 4 details the application. All code listings are provided in the Appendix B.

2. Materials and Methods

2.1. AD via Vectorisation

2.1.1. From Vector Calculus to Matrix Calculus via Vectorisation

Our AD formulation builds on a set of vector calculus rules rather than elementary scalar calculus [19]. Before presenting the full framework, we introduce three key definitions: Definition 1 defines the derivative of a vector-valued function; Definition 2 introduces the vectorisation operator; and Definition 3 combines the first two to define the derivative of a matrix-valued function.

Definition 1.

Suppose $\mathbf{x}\in {\mathbb{R}}^n$ and $\mathbf{f}:{\mathbb{R}}^n\to {\mathbb{R}}^m$, then Jacobian matrix J of $\mathbf{f}$ is a $m\times n$ matrix $\frac{\partial \mathbf{f}}{\partial \mathbf{x}}$ with $(i,j)$ entry given by:

$$\frac{\partial f_i}{\partial x_j},i=1,2,\dots ,m;j=1,2,\dots ,n$$

where $f_i$ and $x_j$ are components of $\mathbf{f}$ and $\mathbf{x}$.

Definition 2.

Let A be an $m\times n$ matrix and $a_j$ its j-th column. Then $\mathit{vec}A$ is the $mn\times 1$ column vector (i.e., $\mathit{vec}A$ stacks the columns of A):

$${[a_{11}{a}_{21}\dots a_{m1}{a}_{12}{a}_{22}\dots a_{m2}\dots a_{1n}{a}_{2n}\dots a_{mn}]}^T.$$

Note that $\mathrm{vec}ABC=(C^T\otimes A)\mathrm{vec}B$, where A, B, and C are three matrices with appropriate dimensions such that the matrix product $ABC$ is well-defined; $C^T$ denotes the transpose of C, and $C^T\otimes A$ denotes the Kronecker product of $C^T$ and A.

Definition 3.

Let $F:{\mathbb{R}}^{n\times q}\to {\mathbb{R}}^{m\times p}$ be a real matrix function, the Jacobian matrix of F at X is defined to be the $mp\times nq$ matrix:

$$DF(X):=\frac{\partial \mathit{vec}F(X)}{\partial {(\mathit{vec}X)}^T}$$

For notational convenience, we write $\partial \mathrm{vec}X$ as $dX$. In the definition above, the numerator is always treated as a column vector and the denominator as a row vector. This allows us to write $DF(X)$ as $\frac{dF(X)}{dX}$ without ambiguity, rather than the more cumbersome $\frac{dF(X)}{{(dX)}^T}$. Indeed, since $\frac{\partial \mathrm{vec}X}{\partial {(\mathrm{vec}X)}^T}=I$, the identity matrix of dimension $mp$, accepting $dX$ as ${(dX)}^T$ in the denominator amounts to a notational simplification:

$$\frac{dF(X)}{dX}=\frac{\partial \mathrm{vec}F(X)}{\partial \mathrm{vec}X}·I=\frac{\partial \mathrm{vec}F(X)}{\partial \mathrm{vec}X}·\frac{\partial \mathrm{vec}X}{\partial {(\mathrm{vec}X)}^T}=\frac{\partial \mathrm{vec}F(X)}{\partial {(\mathrm{vec}X)}^T},$$

aligning with the definition above.

A key advantage of Definition 3 is that it allows higher-order matrix derivatives to remain within the familiar matrix framework, rather than escalating into high-order tensors, e.g., the Hessian of F at X is also a matrix. This simplifies notation and facilitates the use of matrix algebra to exploit structure in Jacobian and Hessian matrices, making the formulation more efficient. For further discussion and critique of alternative matrix derivative conventions—including the numerator and denominator layouts—see [19].

Once derivatives are defined for the basic operations, they can be propagated through a computation using the matrix-based chain rule. Suppose that at a certain stage, we have already computed matrices A and B, along with their derivatives $DA(X)$ and $DB(X)$, with respect to some input X. The next step of the computation involves evaluating a new matrix $C=F(A,B,X)$, where F is differentiable in all parameters. Using the chain rule in vectorised form, the derivative of C with respect to X is given by:

$$DC(X)=\frac{\partial \mathrm{vec}F}{\partial \mathrm{vec}X}(A,B,X)+\frac{\partial \mathrm{vec}F}{\partial \mathrm{vec}A}(A,B,X)DA(X)+\frac{\partial \mathrm{vec}F}{\partial \mathrm{vec}B}(A,B,X)DB(X).$$

(1)

In our formulation, any computation that can be decomposed into a sequence of basic matrix operations—each admitting a tractable and well-defined derivative—can be differentiated efficiently using the chain rule. These operations include (i) basic matrix arithmetic such as addition, subtraction, product, inverse, and Kronecker product; (ii) element-wise arithmetic such as Hadamard product/division and element-wise univariate differentiable transformations; (iii) scalar matrix arithmetic such as scalar matrix addition, subsection, multiplication, and division; (iv) structural transformations such as extracting elements and rearranging or combining matrices; and (v) operations on matrices such as Cholesky decomposition, column/row sum, cross-products, transposition of cross-products, determinants, and traces. They will be presented in Section 2.1.3.

2.1.2. Dual Construction

We present an implementation of an AD system that can differentiate any multivariate matrix polynomial to illustrate the underlying logic of our AD formulation. Let A, B, and C be $n\times n$ matrices and $I_n$ the $n\times n$ diagonal (identity) matrix, and consider the following two matrix calculus rules:

$$\begin{array}{cccc}\hfill C& =A+B\hfill & \hfill \Rightarrow dC& =dA+dB,\hfill \end{array}$$

(2)

$$\begin{array}{cccc}\hfill C& =AB\hfill & \hfill \Rightarrow dC& =(B^T\otimes I_n)dA+(I_n\otimes A)dB.\hfill \end{array}$$

(3)

To implement these rules, first, we attach to each matrix a dual component that stores the derivative with respect to some other parameters (i.e., let $A_{dual}=\left(A,\frac{dA}{d\ast }\right),B_{dual}=\left(B,\frac{dB}{d\ast }\right)$) and refer to them as the dual matrices. For example, if the parameters are the entries of A and B, then $\frac{dA}{d\ast }=\left[\frac{dA}{d\left[A\right|B]}\right]=\left[I_{n^2},0_{n^2}\right]$; similarly $\frac{dB}{d\ast }=\left[\frac{dB}{d\left[A\right|B]}\right]=\left[0_{n^2},I_{n^2}\right]$. We can then define the arithmetic for dual matrices using (2) and (3):

$$\begin{array}{cc}\hfill A_{dual}+B_{dual}& =\left(A,\frac{dA}{d\ast }\right)+\left(B,\frac{dB}{d\ast }\right)=\left(A+B,\frac{dA}{d\ast }+\frac{dB}{d\ast }\right)\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill A_{dual}·B_{dual}& =\left(A,\frac{dA}{d\ast }\right)·\left(B,\frac{dB}{d\ast }\right)=\left(AB,(B^T\otimes I_n)\frac{dA}{d\ast }+(I_n\otimes A)\frac{dB}{d\ast }\right)\hfill \end{array}$$

(5)

and program them as shown in Listing 1.

These 16 lines of code define an AD system that can handle the class of multivariate matrix polynomials formed by addition and multiplication. For example, the derivative of the function $f(A,B)=A(AB+B^2)+B$ is simply the one-line

df <- function(A, B) A %times% ((A %times% B) %plus% (B %times% B)) %plus% B

rather than a tedious program associated with the analytical derivative

$$\begin{array}{cc}\hfill df(A,B)=[{(AB+B^2)}^T\otimes I_n& +(I_n\otimes A)(B^T\otimes I_n)]dA\hfill \\ & +\left[{(I_n\otimes A)}^2+(I_n\otimes A)(B^T\otimes I_n+I_n\otimes B)+I_{n^2}\right]dB.\hfill \end{array}$$

(6)

Readers encountering AD for the first time may be surprised that the program df above appears to compute f itself, rather than its derivative—which is precisely what makes AD so appealing. Given the addition and multiplication operators defined for dual matrices, any function constructed using these operators will automatically have its derivative computed. Specifically, derivatives are evaluated on the fly each time %times% or %plus% is called. The final output of df is a dual matrix, where the first component is the result $f(A,B)$, and the second component is the derivative of f at $(A,B)$. This approach abstracts away the derivative calculation, allowing users to obtain derivatives automatically once the function f is implemented. To complete the system, subtraction and inverse operations are also required (also 16 lines). To main the flow, we list them under the Appendix B, along with a complete working example.

Listing 1. Implementation of the sum and product matrix calculus rules in R.

‘%plus%‘ <- function(A_dual, B_dual) {   A <- A_dual$X;  dA <- A_dual$dX;   B <- B_dual$X;  dB <- B_dual$dX;   list(X = A + B, dX = dA + dB) }   ‘%times%‘ <- function(A_dual, B_dual) {   A <- A_dual$X;  dA <- A_dual$dX;   B <- B_dual$X;  dB <- B_dual$dX;   list(     X = A %∗% B,     dX = (t(B) %x% I(nrow(A))) %∗% dA + (I(ncol(B)) %x% A) %∗% dB   ) }   I <- diag   # function to create diagonal matrices

2.1.3. A Layered Approach to Construction

In the previous section, we showed that formulating AD with dual matrices closely mirrors the underlying analytic derivation. Once the calculus rules for dual matrices are established, building an AD system becomes relatively straightforward. In this section, we present the full set of matrix calculus rules. The rules are grouped by type and presented in the order they would typically be implemented in practice. Illustrative statistical applications are provided in the Appendix.

In the following discussion, the derivative of any matrix is assumed to be w.r.t. some input z with d parameters. Hence, if A is a $(m\times n)$ matrix, then $\frac{dA}{dz}$ is a $(mn\times d)$ Jacobian matrix, and this shall be written as $dA$ for the sake of convenience.

2.1.4. Notation

The symbols $I_{.},K_{.},E_{.},D_{.},1_{.}$ are reserved for the follow special matrices:

• $I_n$ is the $n\times n$ identity matrix.
• $I_{nq}$ is the $n\times q$ matrix where the entries on the diagonal are all ones, and the entries off the diagonal are all zeros.
• $K_{nq}$ is the $nq\times nq$ commutation matrix. We also define $K_n:=K_{nn}$.
• $E_n$ is the $\frac{n(n+1)}{2}\times n^2$ elimination matrix.
• $1_{nq}$ is the $n\times q$ matrix of ones.

(Definitions of the commutation and elimination matrices can be found in Section 2.2.4 and Section 2.2.5, respectively.)

Let A be a $m\times n$ matrix. We denote

• the $(i,j)$-entry of A is denoted by $A_{ij}$ or $A_{i,j}$,
• the i-th row of A is denoted by $A_{i\ast }$ or $A_{i,\ast }$,
• the j-th column of A is denoted by $A_{\ast j}$ or $A_{\ast ,j}$.

We define $v_A(i,j)$ and $v_A^{-1}(k)$ such that they satisfy the relations

$$\begin{array}{cc}\hfill (i,j)\text{-}\mathrm{entry}\mathrm{of}A& \iff v_A(i,j)\text{-}\mathrm{th}\mathrm{entry}\mathrm{of}\mathrm{vec}(A)\hfill \\ \hfill v_A^{-1}(k)\text{-}\mathrm{entry}\mathrm{of}A& \iff k\text{-}\mathrm{th}\mathrm{entry}\mathrm{of}\mathrm{vec}(A)\hfill \end{array}$$

and they are given by the formulas

$$v_A(i,j)=i+(j-1)m\mathrm{and}{v}_A^{-1}(k)=\left([(k-1)modm]+1,⌈\frac{k}{m}⌉\right).$$

(7)

This indices-conversion function is needed because the derivative of the $(i,j)$-entry of A is stored in the $v_A(i,j)$-th row of $dA$, and vice versa.

2.1.5. Matrix Arithmetic

We present the matrix calculus rules associated with basic matrix arithmetic:

• Addition: Let A and B be $m\times n$ matrices, then $d(A+B)=dA+dB$.
• Subtraction: Let A and B be $m\times n$ matrices, then $d(A-B)=dA-dB$.
• Product: Let A and B be $m\times n$ and $n\times k$ matrices, then

  $$d(AB)=(B^T\otimes I_m)dA+(I_k\otimes A)dB.$$
• Inverse: Let A be a $n\times n$ matrix, then $d{A}^{-1}=-(A^{-1}\otimes A^{-1})dA$.
• Kronecker-product: Let A and B be $m\times n$ and $p\times q$ matrices, then

  $$d(A\otimes B)=(I_n\otimes K_{qm}\otimes I_p)\left[\left(I_{mn}\otimes \mathrm{vec}(B)\right)dA+\left(\mathrm{vec}(A)\otimes I_{pq}\right)dB\right].$$
• Transpose: Let A be a $m\times n$ matrix, $d{A}^T=K_{mn}dA$.

We now examine the computational advantages of AD relative to FD through a complexity analysis of basic matrix operations. Our discussion is restricted to central (finite) differencing instead of forward differencing, since the former has superior accuracy ([20], pp. 378–379) and is generally preferred over the latter. Applying central differencing to a function f incurs a cost of evaluating f multiplied by twice the dimension of the inputs. (For ease of discussion, we assume that conditional branches, if they exist, have the same order of computational complexity.) Applying AD incurs a cost of evaluating f and the derivative $df$ as given by the calculus rules.

For matrix arithmetic, suppose A and B are $n\times n$ matrices, so the dimension of the inputs is $d=2n^2$ for matrix additions, subtractions, products, and Kronecker products, and $d=n^2$ for matrix inversions. The number of operations associated with applying the central differencing to $f(A,B):{\mathbb{R}}^d\to {\mathbb{R}}^k$ is:

$$\begin{array}{cc}\hfill & 2d\mathrm{perturbations}(\mathrm{additions}/\mathrm{subtractions})\mathrm{of}\mathrm{the}\mathrm{input}\hfill \\ \hfill & +2d\mathrm{evaluations}\mathrm{of}f\mathrm{with}\mathrm{the}\mathrm{perturbed}\mathrm{input}\hfill \\ \hfill & +d\mathrm{times}2k\mathrm{operations}\mathrm{for}\mathrm{finite}\text{-}\mathrm{differencing}\mathrm{the}\mathrm{output}(\mathrm{subtractions}\mathrm{and}\mathrm{divisions})\hfill \\ \hfill & =2d·(1+\cos \mathrm{t}(f)+k).\hfill \end{array}$$

Assuming standard matrix multiplication, the computational costs of addition, subtraction, multiplication, and the Kronecker product are $n^2$, $n^2$, $2n^3-n^2$, and $n^4$, respectively. Since we are only interested in the leading terms, these simplify to $n^2$, $n^2$, $2n^3$, and $n^4$. The number of operations for matrix inversion using the LU decomposition followed by the inversion of triangular matrices is around $2n^3$ [21]. It then follows that applying finite-differencing would require (in the leading term) $8n^4$ operations for addition and subtraction, $8n^5$ operations for product, $4n^5$ operations for matrix inversion, and $8n^6$ operations for the Kronecker product.

For AD, the number of operations works out to be $n^2+2n^4$ for addition and subtraction, $(2n^3-n^2)+(8n^5-2n^4)$ for multiplication, $2n^3+(8n^5-2n^4)$ for matrix inversion, and $n^4+6n^6$ for the Kronecker product. The results are summarised in

Table 1. Comparison of the number of operations (in terms of the leading order) needed in finite differencing and AD.

Operations	Central Differencing	AD
Addition	$8n^4$	$2n^4$
Subtraction	$8n^4$	$2n^4$
Multiplication	$8n^5$	$8n^5$
Inversion	$4n^5$	$8n^5$
Kronecker product	$8n^6$	$6n^6$

. Note that when a Kronecker product is post-multiplied by a matrix, there is a shortcut to avoid the explicit computation of the Kronecker product. The details will be given in Section 2.2.7. From Table 1, we observe that finite differencing and AD have the same complexity for the operations listed in the table, but AD generally has equal or better leading coefficients, except for the matrix inversion.

2.1.6. Element-Wise Arithmetic

We now present matrix calculus rules for element-wise operations. These rules follow directly from applying scalar calculus to each entry independently. The cases of addition and subtraction are identical to those covered in the previous section on standard matrix arithmetic. Let A, B, and C be $m\times n$ matrices, and $\mathrm{diag}(\mathbf{v})$ be the square matrix in which the vector $\mathbf{v}$ is placed on the diagonal.

• Hadamard product:

  $$\begin{array}{cc}& C_{ij}=A_{ij}{B}_{ij},i=1,2,\dots ,m,j=1,2,\dots ,n\hfill \\ \hfill \Rightarrow & {(dC)}_{k,\ast }=B_{v_B^{-1}(k)}{(dA)}_{k,\ast }+A_{v_A^{-1}(k)}{(dB)}_{k,\ast },k=1,\dots ,mn\hfill \end{array}$$

  Alternatively,

  $$\begin{array}{c}\hfill C=A\odot B\Rightarrow dC=\mathrm{diag}(\mathrm{vec}(B))dA+\mathrm{diag}(\mathrm{vec}(A))dB\end{array}.$$
• Hadamard division:

  $$\begin{array}{cc}& C_{ij}=A_{ij}/B_{ij},i=1,2,\dots ,m,j=1,2,\dots ,n\hfill \\ \hfill \Rightarrow & {(dC)}_{k,\ast }=B_{v_B^{-1}(k)}^{-1}{(dA)}_{k,\ast }-A_{v_B^{-1}(k)}{B}_{v_B^{-1}(k)}^{-2}{(dB)}_{k,\ast },i=1,\dots ,mn\hfill \end{array}$$

  Alternatively,

  $$\begin{array}{c}\hfill C=A\oslash B\Rightarrow dC=\mathrm{diag}(\mathrm{vec}(B^{\circ -1}))dA-\mathrm{diag}(\mathrm{vec}(A\odot B^{\circ -2}))dB\end{array}.$$
• Univariate differentiable function f:

  $$\begin{array}{cc}& C_{ij}=f(A_{ij}),i=1,2,\dots ,m,j=1,2,\dots ,n\hfill \\ \hfill \Rightarrow & {(dC)}_{k,\ast }=f^{\prime }(A_{v_A^{-1}(k)}){(dA)}_{k,\ast },i=1,\dots ,mn\hfill \end{array}$$

  Alternatively,

  $$C=\stackrel{\circ }{f}(A)\Rightarrow dC=\mathrm{diag}\left(\mathrm{vec}\left(\stackrel{\circ }{f^{\prime }}(A)\right)\right)dA,$$

  where $\stackrel{\circ }{f}(A)$ denotes applying f element-wise to A. Note that f may also be a function that is differentiable almost everywhere, e.g., $f(x)=\left|x\right|$ is differentiable everywhere except at $x=0$. When the derivative is evaluated at the non-differentiable locations, it is common to use a subgradient [8]—in this case, any value in the interval $[-1,1]$—or simply to assume a value such as 0, effectively treating these points as having no impact on the result.

2.1.7. Scalar-Matrix Arithmetic

Let A be a $m\times n$ matrix and c be a scalar. Then the differentials $d(c\mathrm{Op}A)$ and $d(A\mathrm{Op}c)$, where $\mathrm{Op}\in \{+,-,\ast ,/\}$, can be computed by lifting the scalar c to a matrix of the same dimension as A, via multiplication with a matrix of ones, $1_{mn}$. This allows the scalar-matrix operation to be treated as an element-wise operation. Importantly, this lifting is a conceptual construct, and the implementation need not construct a new matrix $k·1_{mn}$. Instead, the operation can be performed element-wise directly. For instance, the product rule $d(cA)$ becomes

$$\begin{array}{ccc}\hfill B=cA& \Rightarrow B_{ij}=c{A}_{ij},\hfill & \hfill i=1,2,\dots ,m,j=1,2,\dots ,n;\\ \hfill & \Rightarrow {(dB)}_{k,\ast }=A_{v_A^{-1}(k)}dc+c{(dA)}_{k,\ast },\hfill & \hfill k=1,2,\dots ,mn.\end{array}$$

2.1.8. Structural Transformation

We now present calculus rules for structural transformations—operations that extract, rearrange, or combine matrix entries without performing any arithmetic computations. Because these transformations are primarily operational rather than analytical, they seldom appear in formal derivations and are often left undocumented in standard references.

• Transpose: Let A be a $m\times n$ matrix, $d{A}^T=K_{mn}dA$.
• Row binding: Let $A,B$ be $m\times n,k\times n$ matrices, $\mathrm{rowBind}(A,B):=\left[\begin{array}{c}A\\ B\end{array}\right]$, then

  $$C=\mathrm{rowBind}(A,B)\Rightarrow {(dC)}_{k,\ast }=1_{r\le m}·{(dA)}_{v_A(r,c),\ast }+1_{r>m}·{(dB)}_{v_B(r-m,c),\ast }$$

  where $(r,c)=v_C^{-1}(k)$, $k=1,2,\dots ,(m+k)n$.
• Column binding: Let $A,B$ be $m\times n,m\times k$ matrices, $\mathrm{colBind}(A,B):=\left[AB\right]$, then

  $$C:=\mathrm{colBind}(A,B)\Rightarrow dC=\mathrm{rowBind}(dA,dB).$$
• Subsetting: Let A be a $m\times n$ matrix,
  1. Index extraction: $A_{ij}$ for fixed $i,j$$\Rightarrow d{A}_{ij}={(dA)}_{v_A(i,j),\ast }$
  2. Row extraction: $A_{i\ast }$ for fixed i$\Rightarrow d{A}_{i\ast }={(dA)}_{S,\ast }$, where $S=\{i,i+m,\dots$, $i+(n-1)m\}$
  3. Column extraction: $A_{\ast j}$ for fixed j $\Rightarrow d{A}_{\ast j}={(dA)}_{S,}$, where $S=\{(j-1)m+1$, $(j-1)m+2,\dots ,(j-1)m+n\}$
  4. Diagonal extraction: ${\left[A_{ii}\right]}_{i=1,2,\dots ,n}$ (column vector) $\Rightarrow d{\left[A_{ii}\right]}_{i=1,2,\dots ,n}={(dA)}_{S,}$, where $S=\{1,1+(m+1),1+2(m+1)+\dots ,1+(min(n,m)-1)(m+1)\}$.
• Vectorisation: Let A be a $n\times n$ matrix, $d\mathrm{vec}(A)=dA$
• Half-vectorisation: Let A be a $n\times n$ matrix, $d\mathrm{vech}(A)={(dA)}_{S,},S=\{v_A(i,j),i\ge j\}$.
  Note that S follows the order as the column-major order of A, i.e.,

  $$S=\{v_A(1,1),v_A(2,1),v_A(3,1),\dots ,v_A(2,2),v_A(2,3),\dots ,v_A(n,n),\}.$$
• Diagonal expansion: Let $\mathbf{v}$ be a $n\times 1$ vector, then $diag(\mathbf{v})$ is defined to be the $n\times n$ matrix with $\mathbf{v}$ on the diagonal. If $B=\mathrm{diag}(\mathbf{v})$, then for $k=1,2,\dots ,n^2$,

  $${(dB)}_{k,\ast }={(dA)}_{kmod(n+1),\ast }·1_{k\in S}+\mathbf{0}·1_{k\notin S},$$

  where $S=\{1,n+2,2n+3+\dots ,n^2\}$.

2.1.9. Operations on Matrices

• Cholesky Decomposition: Let A be a $n\times n$ matrix, $A=L{L}^T$ be the Cholesky decomposition and $L=Chol(A)$, then

  $$\begin{array}{c}\hfill dL=D_n^{\ast }{\left[E_n(I_{n^2}+K_n)(L\otimes I_n)D_n^{\ast }\right]}^{-1}{E}_{n}dA\end{array}$$

  where $D_n^{\ast }=E_n^T$ is the duplication matrix for triangular matrices.

Let A be a $m\times n$ matrix.

• Column-sum:
  $$\begin{array}{cc}& \mathrm{colSum}(A):=\sum _i{A}_{i,\ast }\hfill \\ \hfill \Rightarrow & {(d\mathrm{colSum}(A))}_{k,\ast }=\sum _{i=(k-1)m+1,(k-1)m+2,\dots ,km}{(dA)}_{i,\ast },k=1,\dots ,n\hfill \end{array}$$
• Row-sum:
  $$\begin{array}{cc}& \mathrm{rowSum}(A):=\sum _j{A}_{\ast ,j}\hfill \\ \hfill \Rightarrow & {(d\mathrm{rowSum}(A))}_{k,\ast }=\sum _{i=k,k+m,\dots ,k+(n-1)m}{(dA)}_{i,\ast },k=1,\dots ,m\hfill \end{array}$$
• Sum: $\mathrm{sum}(A):={\sum }_{i,j}{A}_{ij}\Rightarrow d\mathrm{sum}(A)=\mathrm{colSum}(dA)$
• Cross-product:
  $$\mathrm{crossprod}(A):=A^{T}A\Rightarrow d\mathrm{crossprod}(A)=(I_{q^2}+K_{qq})(I_q\otimes A^T).$$
• Transpose of cross-product:
  $$\mathrm{tcrossprod}(A):=A{A}^T\Rightarrow d\mathrm{tcrossprod}(A)=(I_{n^2}+K_{nn})(A_n\otimes I_n).$$
  Alternatively, both ‘crossprod’ and ‘tcrossprod’ can be implemented directly as is, since they are composed of the multiplication and transpose operations defined previously.

Let A be a $n\times n$ matrix.

• Determinant: $ddet(A)=det(A)·\mathrm{vec}{(A^{-T})}^T·dA$
• Trace: $d\mathrm{tr}(A)=\mathrm{vec}{(I_n)}^{T}dA$. Alternatively, it can be implemented by composing sum and diagonal extraction defined previously.

2.1.10. Random Variables

Given a probability space $(\mathsf{\Omega },\mathcal{F},P)$, a random variable X is a $\mathcal{F}$-measurable function mapping $\mathsf{\Omega }$ to $\mathbb{R}$. The formalism suggests that in the process of simulating a random variate, the randomness can always be isolated, and it is possible to differentiate (in the pathwise sense) the random variables $X\sim F_X(x;\mathit{\alpha })$ w.r.t. the parameters $\mathit{\alpha }$ when the derivative exists. In the simplest case of normal random variables, the parameters and the randomness can be separated as follows:

$$Z\sim N(\mu ,\sigma )\Rightarrow Z=\mu +\sigma Z_0,Z_0\sim N(0,1)\Rightarrow dZ=d\mu +d\sigma ·Z_0$$

As Z depends smoothly on the parameters $\mu$ and $\sigma$, the derivatives w.r.t. these parameters are well defined. This is commonly referred to as the reparametrisation trick [22].

When explicit separation cannot be done, we utilise the inverse transform method. Suppose $Z\sim F_Z(z;\theta )$ where $F_Z(z;\theta )$ is the cumulative density function of Z, assumed to be invertiable, and $\theta$ is the parameter. Then Z can be simulated using the inverse transform method $Z=F_Z^{-1}(U;\theta )$, $U\sim U[0,1]$. It then follows that if $F_Z^{-1}(·;\theta )$ is differentiable in $\theta$, then the derivative of a random sample is well-defined. This applies to, for instance, the Exponential, Weibull, Rayleigh, log-Cauchy, and log-Logistic distributions.

In the most general case where Z is high-dimensional and $F_Z^{-1}$ may not be known, we rely on the class of isoprobabilistic transformations $T(x_1,\dots ,x_k;\alpha )$, which transforms an absolutely continuous k-variate distribution $F(x_1,\dots ,x_k;\alpha )$ into the uniform distribution on the k-dimensional hyper-cube [23]. This gives an explicit formula to find the derivative of a random vector:

$$\frac{\partial X}{\partial \alpha }=-{\left(\frac{\partial T(X,\alpha )}{\partial X}\right)}^{-1}\frac{\partial T(X,\alpha )}{\partial \alpha }$$

assuming $det\left(\frac{\partial T(X,\mathit{\alpha })}{\partial X}\right)\ne 0$ so that the inverse exists.

For clarity, let us consider a 1-dimensional example. Suppose we have a random variable $X\sim F_X(x;\mathit{\alpha })$, where $F_X$ is invertible, then an isoprobabilistic transformation $T(X,\mathit{\alpha })$ would simply be $F_X(X,\mathit{\alpha })$ as $F_X(X,\mathit{\alpha })$ is distributed uniformly. Hence, it follows that

$$\frac{\partial X}{\partial \mathit{\alpha }}=-f_X{(X,\mathit{\alpha })}^{-1}\frac{\partial F_X(X,\mathit{\alpha })}{\partial \mathit{\alpha }}.$$

It is easy to check via elementary means that this is indeed correct. Starting with the identity $F_X(F_X^{-1}(U,\mathit{\alpha }),\mathit{\alpha })=U$ and applying implicit differentiation, we have

$$\begin{array}{cc}\hfill & \frac{\partial F_X(F_X^{-1}(U,\mathit{\alpha }),\mathit{\alpha })}{\partial \mathit{\alpha }}=\mathbf{0}\hfill \\ \hfill & \Rightarrow \frac{\partial F_X}{\partial X}(F_X^{-1}(U,\mathit{\alpha }),\mathit{\alpha })·\frac{\partial F_X^{-1}(U,\mathit{\alpha })}{d\mathit{\alpha }}+\frac{\partial F_X}{\partial \mathit{\alpha }}(F_X^{-1}(U,\mathit{\alpha }),\mathit{\alpha })=\mathbf{0}\hfill \\ \hfill & \Rightarrow \frac{\partial F_X^{-1}(U,\mathit{\alpha })}{d\mathit{\alpha }}=-\frac{\frac{\partial F_X}{\partial \mathit{\alpha }}(F_X^{-1}(U,\mathit{\alpha }),\mathit{\alpha })}{f_X(F_X^{-1}(U,\mathit{\alpha }),\mathit{\alpha })}\hfill \\ \hfill & \Rightarrow \frac{\partial X}{\partial \mathit{\alpha }}=-f_X{(X,\mathit{\alpha })}^{-1}\frac{\partial F_X(X,\mathit{\alpha })}{\partial \mathit{\alpha }}.\hfill \end{array}$$

Some specific one-dimensional cases include the gamma, inverse-gamma, chi-squared, Dirichlet, Wishart, and inverse-Wishart distributions.

2.2. Optimising AD Implementation

In the previous section, we introduced the vectorised AD formulation along with the full set of matrix calculus rules to support the dual construction. In this section, we explore several implementation strategies aimed at optimising execution. Benchmarking is carried out in R using the ADtools package available on CRAN ([24]) and GitHub ([25]). While the exact performance gains presented here (and in Section 3) are environment-specific, the optimisation principles are broadly applicable, and improvements can be expected in other environments.

2.2.1. Memoisation

Memoisation (or tabulation) is a technique for non-invasively attaching a cache to a function, allowing it to store and reuse previously computed results for repeated inputs [26]. It can greatly accelerate tasks such as constructing large structured matrices and provides a convenient way to organise computations.

The technique works by checking whether a given input has already been evaluated. If so, the cached result is returned; if not, the computation is performed, and the result is stored for future use.

Table 2. Speed comparison of the diagonal matrix function with and without memoisation. mem_diag is memorised. The best results in each column are highlighted in bold.

Function	First Time	Second Time	Average over 100 Executions
diag(5000)	97.87 ms	127.7 ms	117.3 ms
mem_diag(5000)	101.8 ms	0.132 ms	0.959 ms

shows a speed comparison of the built-in R function diag, with and without memoisation. An illustrative 12-line implementation is included in the Appendix B.

2.2.2. Sparse Matrix Representation

For efficient implementation, all special matrices are constructed and stored using sparse representations, which improve both computational and memory efficiency. A sparse matrix is typically represented as a list of triples, where each triple $(i,j,v)$ records the value v at position $(i,j)$ in the matrix.

2.2.3. The Diagonal Matrix $D_n$

An $n\times n$ diagonal matrix $D_n$ is represented as $\{(k,k,v_k),k=1,2,\dots ,n\}$, where $v_k$ is the kth diagonal entry of $D_n$. This representation takes $O(n)$ storage space and incurs an $O(n^2)$ computation cost when multiplied by a $n\times n$ dense matrix. A speed comparison of the diagonal matrix function with dense and sparse representations is provided in

Table 3. Speed comparison of the diagonal matrix function with dense and sparse representations. Time is averaged over 100 executions. The best results in each column are highlighted in bold.

Tasks	n = 1000	n = 2000	n = 3000	n = 4000	n = 5000
Creation—Dense	2.436 ms	17.39 ms	50.23 ms	63.86 ms	303.7 ms
Creation—Sparse	0.080 ms	0.091 ms	0.110 ms	0.077 ms	0.089 ms
Multiplication—Dense	19.40 ms	111.6 ms	404.4 ms	1.041 s	1.576 s
Multiplication—Sparse	17.42 ms	66.50 ms	236.6 ms	432.3 ms	668.2 ms

. In the table, “Dense” uses the R function diag, and “Sparse” uses the R function ADtools::diagonal. Using sparse representation, a substantial increase in speed was observed for large matrices.

2.2.4. The Commutation Matrix $K_{nq}$

The $nq\times nq$ matrix $K_{nq}$ is a commutation matrix if $\mathrm{vec}(A^T)=K_{nq}\mathrm{vec}(A)$ for any $n\times q$ matrix A [27]. From (7), we have

$$\begin{array}{cc}\hfill k\mathrm{th}\mathrm{entry}\mathrm{of}\mathrm{vec}(A)& =v_A^{-1}(k)\mathrm{th}\mathrm{entry}\mathrm{of}A=(a,b)\mathrm{entry}\mathrm{of}A\hfill \\ & =(b,a)\mathrm{entry}\mathrm{of}{A}^T=v_{A^T}(b,a)\mathrm{th}\mathrm{entry}\mathrm{of}\mathrm{vec}(A^T)\hfill \\ & =[b+(a-1)q]\mathrm{th}\mathrm{entry}\mathrm{of}\mathrm{vec}(A^T)\hfill \end{array}$$

where $a=\left[(k-1)\mathrm{mod}n\right]+1,b=⌈\frac{k}{n}⌉$. As $K_{nq}$ is the matrix that maps $\mathrm{vec}(A)$ to $\mathrm{vec}(A^T)$, and we have derived that the kth entry of $\mathrm{vec}(A)$ needs to map to $⌈\frac{k}{n}⌉+\left[(k-1)\mathrm{mod}n\right]q$-th entry of $\mathrm{vec}(A^T)$, it follows that $K_{nq}$ is a matrix having value 1 at position $\left(⌈\frac{k}{n}⌉+\left[(k-1)\mathrm{mod}n\right]q,k\right),k=1,2,\dots ,nq$. Therefore, in the sparse representation, we have

$$K_{nq}=\left\{\left(⌈\frac{k}{n}⌉+\left[(k-1)\mathrm{mod}n\right]q,k,1\right),k=1,2,\dots ,nq\right\}.$$

(8)

It is worth noting that since the commutation matrix simply reorders the entries of $\mathrm{vec}(A)$, one can implement a function that directly remaps the indices as specified in (8), rather than explicitly constructing the matrix and performing the associated matrix multiplication.

Table 4. Speed comparison of the commutation matrix function with dense and sparse representations. Time is averaged over 100 executions. The best results in each column are highlighted in bold.

Tasks	n = q = 10	n = q = 20	n = q =30	n = q =40
Creation—Dense	10.77 ms	708.4 ms	18.09 s	120.4 s
Creation—Sparse	0.646 ms	0.667 ms	1.553 ms	1.595 ms
Multiplication—Dense	0.727 ms	41.54 ms	477.3 ms	2.821 s
Multiplication—Sparse	0.189 ms	2.388 ms	11.48 ms	71.69 ms

compares the performance of commutation matrix functions implemented using dense and sparse representations. The “Dense” implementation relies on the R function matrixcalc::elimination.matrix, while the “Sparse” version uses ADtools::elimination_matrix. A substantial improvement in speed is observed with the sparse approach.

2.2.5. The Elimination Matrix $E_n$

Let E (and $E_n$) denote the elimination matrix and D the duplication matrix. These matrices are defined by the identities they satisfy:

$$\begin{array}{cc}\hfill \mathrm{vech}(A)& =E\mathrm{vec}(A)\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill D\mathrm{vech}(A)& =\mathrm{vec}(A),\mathrm{for}\mathrm{symmetric}\mathrm{matrix}A\hfill \end{array}$$

(10)

where $\mathrm{vech}(.)$ is the half-vectorisation operator (vectorising the lower-triangular part of a square matrix). The names of the special matrices come from the fact that D duplicates entries to turn a half vector into a full vector, and E eliminates entries to turn a full vector into a half vector. Note that if A is an $n\times n$ matrix, then $\mathrm{vec}(A)$ has a length $n^2$ and $\mathrm{vech}(A)$ has a length of $\frac{n(n+1)}{2}$. Hence, D has a dimension of $n^2\times \frac{n(n+1)}{2}$, and E has a dimension of $\frac{n(n+1)}{2}\times n^2$.

Now we derive the sparse representation of the elimination matrix. First, for an $n\times n$ matrix A, we define $h_A(i,j)$ and $h_A^{-1}(k)$ such that they satisfy the relations.

$$\begin{array}{cc}\hfill (i,j)\text{-}\mathrm{entry}\mathrm{of}A& \iff h_A(i,j)\text{-}\mathrm{th}\mathrm{entry}\mathrm{of}\mathrm{vech}(A)\hfill \\ \hfill h_A^{-1}(k)\text{-}\mathrm{entry}\mathrm{of}A& \iff k\text{-}\mathrm{th}\mathrm{entry}\mathrm{of}\mathrm{vech}(A),\hfill \end{array}$$

where $i\ge j$, $h_A^{-1}(k)$ must be in the lower triangular part of A. The functions $h_A(.,.)$ and $h_A^{-1}(.)$ are given by the formulae:

$$h_A(i,j)=i+(j-1)n-\frac{j(j-1)}{2}\mathrm{and}{h}_A^{-1}(k)=(a,b),$$

(11)

where $a=k+\frac{b(b-1)}{2}-(b-1)n$, $b=f(k,n,1)$ and

$$f(k,n,c)=\left\{\begin{array}{cc}f(k-n,n-1,c+1)& \mathrm{if}k>n,\\ c& \mathrm{otherwise}.\end{array}\right.$$

$h_A(.,.)$ and $h_A^{-1}(.)$ are needed to convert back and forth between the matrix and the half-vector representations. It then follows that:

$$\begin{array}{cc}\hfill k\mathrm{th}\mathrm{entry}\mathrm{of}\mathrm{vech}(A)& =h_A^{-1}(k)\mathrm{entry}\mathrm{of}A=v_A(h_A^{-1}(k))\mathrm{th}\mathrm{entry}\mathrm{of}\mathrm{vec}(A)\hfill \\ & =v_A\left(k+\frac{b(b-1)}{2}-(b-1)n,b\right)\mathrm{th}\mathrm{entry}\mathrm{of}\mathrm{vec}(A)\hfill \\ & =k+\frac{b(b-1)}{2}\mathrm{th}\mathrm{entry}\mathrm{of}\mathrm{vec}(A),\hfill \end{array}$$

where $b=f(k,n,1)$. Since by definition $E_n$ maps $\mathrm{vec}(A)$ to $\mathrm{vech}(A)$, and we have shown $k+\frac{b(b-1)}{2}\mathrm{th}\mathrm{entry}\mathrm{of}\mathrm{vec}(A)$ is mapped to $k\mathrm{th}\mathrm{entry}\mathrm{of}\mathrm{vech}(A)$, so $E_n$ is a matrix having a value of 1 at the position $\left(k,k+\frac{b(b-1)}{2}\right)$. Hence, the sparse representation of the elimination matrix is given by:

$$E_n=\left\{\left(k,k+\frac{b(b-1)}{2},1\right),k=1,2,\dots ,\frac{n(n+1)}{2}\right\}.$$

In the actual implementation, b does not need to be computed recursively; it can be obtained directly using the closed-form expression $b=⌈(n+0.5)-\sqrt{{(n+0.5)}^2-2k)}⌉$. A performance comparison of the elimination matrix function using dense and sparse representations is shown in

Table 5. Speed comparison of the elimination matrix function with dense and sparse representations. Time is averaged over 100 executions. The best results in each column are highlighted in bold.

Tasks	n = 10	n = 20	n = 30	n = 40	n = 50
Creation—Dense	4.981 ms	113.1 ms	939.9 ms	4.735 s	20.97 s
Creation—Sparse	0.823 ms	0.816 ms	0.823 ms	0.816 ms	0.891 ms
Multiplication—Dense	0.412 ms	21.98 ms	237.0 ms	1.490 s	5.715 s
Multiplication—Sparse	0.115 ms	1.723 ms	6.720 ms	22.45 ms	49.16 ms

. In the table, “Dense” uses the R function matrixcalc::commutation.matrix, and “Sparse” uses the R function ADtools::commutation_matrix. The results demonstrate a clear speed advantage for the sparse implementation, with performance gains increasing as matrix size grows.

We also define the half-duplication matrix $D^{\ast }$ for lower-triangular matrix A to be the matrix that satisfies $D^{\ast }\mathrm{vech}(A)=\mathrm{vec}(A)$, where A is a lower-triangular matrix. For any lower-triangular matrix A, we have $D^{\ast }\mathrm{vech}(A)=E^T\mathrm{vech}(A)$.

2.2.6. Matrix Chain Multiplication

In the implementation of vectorised AD, the derivative computation frequently involves sequences of matrix multiplications, as shown in Equation (1). While mathematically straightforward, these operations can become computational bottlenecks in high-dimensional settings. A key yet often overlooked aspect is that matrix multiplication, although associative in output—i.e., $(AB)C=A(BC)$—is not associative in computational cost. To illustrate, consider $n\times n$ matrices $A,B$ and a $n\times 1$ vector x. Evaluating $(AB)x$ requires explicitly forming the intermediate matrix $AB$, resulting in a complexity $O(n^3)$. In contrast, computing $A(Bx)$ avoids this and reduces the cost to $O(n^2)$. This simple example highlights a crucial insight: although the result is invariant to the order of operations, the efficiency is not. In large-scale computations, suboptimal ordering—such as naive left-to-right evaluation—can be unnecessarily costly. In simple cases where the dimensions of the matrices are known in advance, an optimal order can be enforced manually. However, in many applications, matrix dimensions are unknown until runtime, making it impossible to prespecify the optimal multiplication order. This leads to the matrix chain multiplication problem.

Matrix chain multiplication is an optimisation problem concerned with multiplying a chain of matrices $A_1·A_2·\dots ·A_m$ using the least number of arithmetic operations. This is traditionally solved using dynamic programming with the complexity $O(2^m)$, which can be reduced to $O(m^3)$ when the memoisation technique is employed. Ref. [28] provides an algorithm that solves the problem with a $O(mlogm)$ complexity. However, given that in many applications the length of the matrix chain rarely goes beyond $m=5$, it is usually sufficient to consider the simpler dynamic programming solution as follows:

Let $m(i,j)$ be the minimal number of arithmetic operations needed to multiply out a chain of matrices $A_i·A_{i+1}·\dots ·A_j$, and suppose for any i, $A_i$ has dimension $d_i\times d_{i+1}$. Our goal is to find $m(1,n)$. The recursive formula is given by [29]:

$$\begin{array}{c}\hfill m(i,j)=\left\{\begin{array}{cc}0\hfill & \hfill \mathrm{if}i=j,\\ \underset{i\le k<j}{min}\left\{m(i,k)+m(k+1,j)+d_i·d_{k+1}·d_{j+1}\right\}\hfill & \hfill \mathrm{if}i<j\end{array}\right.\end{array}$$

The above only provides the optimal number of arithmetic operations. To obtain the order of multiplication, we define the split point of the matrix chain $A_i·A_{i+1}·\dots ·A_j$ as:

$$s(i,j)=\arg {\min }_{i\le k<j}\left\{m(i,k)+m(k+1,j)+d_i·d_{k+1}·d_{j+1}\right\}.$$

For example, if $s(1,4)=2$, then the matrix chain $A_1{A}_2{A}_3{A}_4$ should be split after index 2 in the order of $(A_1{A}_2)(A_3{A}_4)$, whereas if $s(1,4)=3$, then the matrix chain is ordered as $(A_1{A}_2{A}_3)(A_4)$, after which one inspects $s(1,3)$ to decide the full order.

Table 6. Comparing multiplications of a chain of matrices in the naive and optimal orders. Figures represent the mean and standard deviation (in bracket) of the speed-up over 1000 simulations.

Length of Chain	2	3	4	5
Speed-up, $t_{naive}/t_{optimal}$	1.04 (0.322)	1.47 (0.564)	1.64 (0.688)	1.56 (0.637)

shows the increase in speed gained by switching from a naive (left-to-right) order to an optimal order. The comparison was conducted using 1000 simulations. The length of the chain was sampled from the set $\{2,3,4,5\}$, with probabilities proportional to $\{\frac{1}{2},\frac{1}{3},\frac{1}{4},\frac{1}{5}\}$, and matrix sizes were sampled from the discrete uniform distribution $U[10,200]$. Note that there is no speed-up by multiplying two matrices because there is only one possible order (the extra 0.04 is merely statistical noise). For a matrix chain with a length of three to five, the average speed-up was about 1.5 times. The speed-up distributions conditioning on the length of the chain were all positively skewed and had positive excess kurtosis (i.e., fat tails).

2.2.7. Kronecker Products

Among the basic matrix operations, the Kronecker product is one of the most computationally expensive. In general, computing the Kronecker product of an $(m\times n)$ matrix and a $p\times q$ matrix has a complexity of $O(mnpq)$. For simplicity, assume $m=n=p=q$; then the complexity becomes $O(n^4)$. In the context of Jacobian matrix computations, it is rare to compute a standalone Kronecker product. Instead, it typically appears as part of a larger expression—for example, in forms such as $X(B\otimes A)$ and $(B\otimes A)Z$, where $A,B$ are of size $n\times n$ and $X,Z$ are of size $m\times n^2,n^2\times m$.

If one first computes the Kronecker product explicitly and then multiplies it by the remaining matrix, the total complexity is $O(n^4+n^{4}m)=O(n^{4}m)$. However, by exploiting structural properties and avoiding explicit computation of the Kronecker product, the same result can be obtained in $O(n^{3}m)$ operations. We now show that this reduced complexity holds in the general case as well:

Proposition 1.

Suppose $A_1,A_2,\dots ,A_p$ are $n\times n$ matrices ($p\ge 2$), and $X,Z$ are $m\times n^p$ and $n^p\times m$ matrices, respectively. Then $X(A_1\otimes A_2\otimes \dots \otimes A_p)$ and $(A_1\otimes A_2\otimes \dots \otimes A_p)Z$ can be computed in $O(n^{p+1}m)$ operations instead of $O(n^{2p}m)$ operations, as observed in the naive order.

The proposition suggests that unless the Kronecker product itself is of interest, one should never compute it explicitly when it comes to multiplication because the algorithm (presented after the proof) always performs $(p-1)$ orders faster. The proposition also holds for cases in which $A_1,\dots ,A_p$ have arbitrary sizes, but we do not state the proposition in that form because it obscures the complexity improvement and logic of the proof. Nevertheless, the algorithm provided later does support the most general case.

Proof. 

We begin with the base case $(B\otimes A)Z$. Let $b_{i,j}$ be the $(i,j)$ element of B and $Z_i$ be the ith block row of Z (which is of size $n\times m$), then

$$\begin{array}{cc}\hfill (B\otimes A)Z& =\left[\begin{array}{cccc}{b}_{1,1}A& b_{1,2}A& \dots & b_{1,n}A\\ b_{2,1}A& b_{2,2}A& \dots & b_{2,n}A\\ ⋮& ⋮& \ddots & ⋮\\ b_{n,1}A& b_{n,2}A& \dots & b_{n,n}A\end{array}\right]·\left[\begin{array}{ccc}\overline{ }& Z_1& \overline{ }\\ \overline{ }& Z_2& \overline{ }\\ ⋮& ⋮& ⋮\\ \overline{ }& Z_n& \overline{ }\end{array}\right]\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & =\left[\begin{array}{ccc}\overline{ }& \sum _{j=1}^n{b}_{1,j}A{Z}_j& \overline{ }\\ \overline{ }& \sum _{j=1}^n{b}_{2,j}A{Z}_j& \overline{ }\\ ⋮& ⋮& ⋮\\ \overline{ }& \sum _{j=1}^n{b}_{n,j}A{Z}_j& \overline{ }\end{array}\right]\stackrel{denote}{:=}{\left[\sum _{j=1}^n{b}_{k,j}A{Z}_j\right]}_{k=1\dots n}\hfill \end{array}$$

(13)

For the kth block-row, we have ${\sum }_{j=1}^n{b}_{k,j}A{Z}_j=A{\sum }_{j=1}^n{b}_{k,j}{Z}_j$. Together with the multiplication by A, the sum can be computed in $O(n^{2}m)$ operations, and because there are n block rows, the overall complexity is $O(n^{3}m)$.

We now proceed to the general case by abstracting the component that makes the above work and applying it recursively to the chain of Kronecker products. If we define two binary operations ⊡ and ⊛ such that:

$$A⊡V=A⊡{\left[V_k\right]}_{k=1\dots n}\stackrel{def}{:=}{\left[A{V}_k\right]}_{k=1\dots n}\mathrm{and}$$

$$B⊛Z=B⊛{\left[Z_k\right]}_{k=1\dots n}\stackrel{def}{:=}{\left[\sum _{j=1}^n{b}_{k,j}{Z}_j\right]}_{k=1\dots n},$$

(14)

then $(B\otimes A)Z=A⊡(B⊛Z)={\left[A{(B⊛Z)}_k\right]}_{k=1\dots n}$. The key idea behind the two new binary operations is that they define block-wise matrix multiplication. Suppose both V and Z can be split into n by 1 blocks. The first binary operation $A⊡V$ defines the block-wise (pre-)multiplication, where each block of V is pre-multiplied by A, with A having the number of columns matching the number of rows of a block. The second binary operation ⊛ defines the block-wise matrix multiplication. This operation produces a matrix of $n\times 1$ blocks, where the i-th block is given by ${\sum }_{j=1}^n{b}_{i,j}{Z}_j$, naturally extending the usual matrix multiplication ${\sum }_{j=1}^n{b}_{i,j}{c}_j$, where $c_j$ is the jth entry of a column vector c. The last term in (14) is also denoted as ${\left[{(B⊛Z)}_k\right]}_{k=1\dots n}$ such that ${(B⊛Z)}_k$ corresponds to the kth block-row ${\sum }_{j=1}^n{b}_{k,j}{Z}_j$.

The new binary operations allow us to evaluate the expression in a different order and avoid the Kronecker product in the process. As a result, it takes fewer arithmetic operations to evaluate the expression, as we have seen in the base case. Next, it follows that

$$\begin{array}{cc}\hfill (C\otimes B\otimes A)Z& =(C\otimes (B\otimes A))Z=(B\otimes A)⊡(C⊛Z)\hfill \\ \hfill & ={\left[(B\otimes A){(C⊛Z)}_{k_C}\right]}_{k_C=1\dots n}\hfill \\ \hfill & ={\left[A⊡(B⊛{(C⊛Z)}_{k_C})\right]}_{k_C=1\dots n}\hfill \\ \hfill & ={\left[{\left[A{(B⊛{(C⊛Z)}_{k_C})}_{k_B}\right]}_{k_B=1\dots n}\right]}_{k_C=1\dots n},\hfill \end{array}$$

and the general case is given by

$$\begin{array}{cc}\hfill & (A_1\otimes A_2\otimes \dots \otimes A_p)Z\hfill \\ \hfill & =\underset{p-1}{\underbrace{\left[\right[\cdots [}}{A}_p{(A_{p-1}⊛{(A_{p-2}⊛(\cdots ⊛{(A_1⊛Z)}_{k_1}\cdots ))}_{k_{p-2}})}_{k_{p-1}}\underset{p-1}{\underbrace{{]}_{k_{p-1}=1\dots n}\cdots {]}_{k_2=1\dots n}{]}_{k_1=1\dots n}}}.\hfill \end{array}$$

Intuitively, every time we use the new binary operations to avoid a Kronecker product, the complexity is reduced by one in order, and given that there are $(p-1)$ of them, we expect to see the total complexity to be $O(n^{2p-(p-1)}m)=O(n^{p+1}m)$. We now present the formal proof by induction.

Let $S(p)$ be the statement that $(A_p\otimes A_{p-1}\otimes \dots \otimes A_1)Z^{(p)}$ has complexity $O(n^{p+1}m)$, where $A_1,\dots ,A_p$ are $n\times n$ matrices, and $Z^{(p)}$ denotes a $n^p\times m$ matrix. We have shown in (12) and (13) that $S(2)$ is true. Now suppose $S(k)$ is true and consider $S(k+1)$,

$$\begin{array}{cc}\hfill & (A_{p+1}\otimes A_p\otimes \dots \otimes A_1)Z\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill & =(A_p\otimes \dots \otimes A_1)⊡(A_{p+1}⊛Z^{(p+1)})\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & ={\left[(A_p\otimes \dots \otimes A_1){(A_{p+1}⊛Z^{(p+1)})}_k\right]}_{k=1\dots n}\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & ={\left[(A_p\otimes \dots \otimes A_1)Z_k^{(p)}\right]}_{k=1\dots n}\hfill \end{array}$$

(18)

In (17), computing ${(A_{p+1}⊛Z^{(p+1)})}_k$ requires $O(n^{p+1}m)$ operations, resulting in a matrix $Z_k^{(p)}$ of size $n^p\times m$. Next, in (18), the expression inside the square bracket, by hypothesis, has a complexity of $O(n^{p+1}m)$, and the complexity accumulated so far remains as $O(n^{p+1}m)$. Finally, given that there are n block-rows, the overall complexity is $O(n^{p+2}m)$. This proves the inductive step, and by induction, $S(n)$ is true for $n\ge 2$. This completes the proof for the case $(A_1\otimes A_2\otimes \dots \otimes A_p)Z$ in Proposition 1.

For the other case, $X(A_1\otimes A_2\otimes \dots \otimes A_p)$, because we are premultiplying the chain of Kronecker products, we work with blocks of columns instead of blocks of rows. Denote

$$X=\left[\begin{array}{cccc}|& |& \dots & |\\ X_1& X_2& \dots & X_n\\ |& |& \dots & |\end{array}\right]$$

by ${\left[X_k\right]}_{k=1\dots n}^c$ (where c stands for columns). Then the two corresponding binary operators ${⊡}^c$ and ${⊛}^c$ are defined as:

$$\begin{array}{cc}\hfill V{⊡}^{c}A& ={\left[V_k\right]}_{k=1\dots n}^c{⊡}^{c}A\stackrel{def}{:=}{\left[V_{k}A\right]}_{k=1\dots n}^c,\mathrm{and}\hfill \\ \hfill Z{⊛}^{c}B& ={\left[Z_k\right]}_{k=1\dots n}^c{⊛}^{c}B\stackrel{def}{:=}{\left[\sum _{i=1}^n{b}_{ik}{Z}_i\right]}_{k=1\dots n}.\hfill \end{array}$$

Now, it follows that $X(B\otimes A)=(X{⊛}^{c}B){⊡}^{c}A$, and the remainder of the proof proceeds the same as in the other case. □

In

Table 7. Speed-up achieved by evaluating $(B\otimes A)Z$ and $X(B\otimes A)$ without explicitly calculating the Kronecker product. The speed-up is computed using $t_{explicit}/t_{implicit}$. The number of simulations is 1000.

Tasks/Percentiles	0%	25%	50%	75%	100%	Mean
$(B\otimes A)Z$	0.42	9.43	14.02	19.99	57.48	15.82
$X(B\otimes A)$	2.65	10.82	15.44	21.34	54.30	16.81

, we compare the performance of evaluating $(B\otimes A)Z$ and $X(B\otimes A)$ with and without explicitly computing the Kronecker product. We conducted 1000 simulations, and in each simulation, the number of rows of $X,A,B$, and the number of columns of $A,B,Z$ were sampled from $\zeta$, where $\zeta \sim U[10,50]$. Obviously, the number of columns of X needs to match the number of rows of $B\otimes A$ (= the number of rows of B × the number of rows of A), and likewise for the number of rows of Z. The speed-up was computed using $t_{explicit}/t_{implicit}$, which denotes the time needed to evaluate the full expression using explicit and implicit Kronecker products, respectively. We note that in the case of $(B\otimes A)Z$, there were two speed-up “outliers”: one at 0.42 and, the other at 0.94. All the rest were above 1. The median speed-up was about 15X, and the mean speed-up was about 16X, favouring the implicit evaluation.

2.2.8. Kronecker Product: More Special Cases

We identified common special cases of the Kronecker product and represented those using the new binary operators to reduce the computation cost further. In particular, we examine the four cases: $A(B\otimes I)$, $A(I\otimes C)$, $(B\otimes I)D$, and $(I\otimes C)D$. These are chosen because they arise naturally in common operations such as the product rule $d(AB)=(B^T\otimes I_n)dA+(I_n\otimes A)dB$ and the transpose product rule $d(A{A}^T)=(I\otimes A+A\otimes I)$. Moreover, computing a Kronecker product (explicitly) with an identity matrix merely makes copies of the original matrix and arranging them in a particular way, hence yielding potential savings in time and memory use (despite that the complexity order remaining the same).

Note that $I⊡V={\left[I{V}_k\right]}_{k=1\dots n}={\left[V_k\right]}_{k=1\dots n}=V$ and $I⊛Z={\left[Z_k\right]}_{k=1\dots n}=Z$, so

• $(B\otimes I)D=I⊡(B⊛D)=B⊛D$
• $(I\otimes C)D=C⊡(I⊛D)=C⊡D$
  Similarly, $V{⊡}^{c}I={\left[V_{k}I\right]}_{k=1\dots n}={\left[V_k\right]}_{k=1\dots n}=V$ and $Z{⊛}^{c}I={\left[Z_k\right]}_{k=1\dots n}=Z$, so
• $A(B\otimes I)=(A{⊛}^{c}B){⊡}^{c}I=A{⊛}^{c}B$
• $A(I\otimes C)=(A{⊛}^{c}I){⊡}^{c}C=A{⊡}^{c}C$

In

Table 8. Speed-up achieved evaluating $(B\otimes I)D$, $(I\otimes C)D$, $A(B\otimes I)$ and $A(I\otimes C)$ without explicitly computing the Kronecker product. The speed-up is computed using $t_{optimised}/t_{naive}$. The number of simulations is 1000.

Tasks/Percentiles	0%	25%	50%	75%	100%	Mean
$(B\otimes I)D$	1.51	5.02	7.79	12.23	51.33	9.48
$(I\otimes C)D$	1.95	5.06	7.83	11.62	79.33	9.48
$A(B\otimes I)$	2.3	5.88	8.54	12.9	68.36	10.25
$A(I\otimes C)$	2.22	5.13	8.25	12.71	61.62	9.98

, we present the speed-up using the optimised implementation over the naive implementation. We conducted 1000 simulations, and in each simulation, the number of rows of $A,B,C,I$ and the number of columns of $B,C,D,I$ were sampled from $\zeta$, where $\zeta \sim U[10,50]$. The number of columns of A and the number of rows of D were specified such that the multiplication makes sense (and the choice is unique). The speed-up was computed using $t_{optimised}/t_{naive}$, which denotes the evaluation time corresponding to the optimised and the naive implementations, respectively. We observe that the median speed-up is about 8×, While the mean speed-up is about 10×, this aligns with our theoretical result that the Kronecker product should not be evaluated explicitly unless the product itself is of interest.

3. Results

This section provides some computational examples to demonstrate the speed and efficacy of our proposed methods. We first benchmark our method against the traditional numerical derivative under basic matrix operations and the computation of a covariance matrix’s log determinant. We then demonstrate our derivative computation’s effectiveness within a large stochastic optimisation scheme, i.e., simulated maximum likelihood estimation (SMLE) of a stochastic factor model.

3.1. Basic Operations

We benchmarked the performance of AD against that of FD using the basic arithmetic operations addition: subtraction, multiplication, matrix inversion, and Kronecker product. The results are presented in

Table 9. Benchmarking of AD against central FD in terms of basic arithmetic operations. Faster times are in bold.

Estimation Time (in ms) for Standard Tasks Under AD and FD by Matrix Size
Tasks	AD	FD	AD	FD	AD	FD	AD	FD	AD	FD
	n = 10		n = 20		n = 30		n = 40		n = 50
Addition	4.61	22.98	5.64	138.17	16.15	435.70	73.76	1136.07	146.96	2746.58
Subtraction	3.92	21.91	5.42	107.32	19.33	427.66	77.41	1236.07	167.06	2970.38
Multiplication	13.60	23.40	20.48	141.94	38.36	580.23	139.87	1732.45	245.83	4251.67
Inverse	2.46	20.94	8.09	93.98	29.20	334.35	156.82	945.50	444.41	1839.47
	n = 5		n = 10		n = 15		n = 20		n = 25
Kronecker	15.1	9.3	54.2	181.7	351.9	1756.4	1478.6	7492.1	5530.5	31,950.7

. The time figures in the table represent the averages of 100 executions. Overall, AD performed much faster compared with the FD.

3.2. Dynamic Factor Model Inference

Numerical assessments are often required in both classical and Bayesian statistical inference. Below, we illustrate the benefits of applying AD to derivative computation for the maximum likelihood estimation (MLE) of factor models when the analytical expression of the likelihood is not available and numerical assessment of derivatives is required. We show substantial computational gains using both simulated and real data. Readers interested in the use of AD in the context of Bayesian sensitivity analysis are referred to [30].

Factor models have been widely used in many areas, including psychology, bioinformatics, economics, and finance, to model the dependence structure of high-dimensional data. Different specifications of the factor model have been widely discussed in the literature ([31,32]). We follow [32] and consider a variation of the factor model in which the analytic expression of the derivative of the log-likelihood is intractable. Let ${\mathbf{y}}_t$ denote the $n\times 1$ vector of observations at time t where $t=1,\dots ,T$, and let ${\mathbf{f}}_t$ represent a $k\times 1$ column vector of latent factors. Then, the k-factor model with student-t noise is specified as:

$${\mathbf{y}}_t=\beta +\mathbb{A}{\mathbf{f}}_t+{ϵ}_t,$$

(19)

where $\beta$ is the $n\times 1$ column vector of intercepts and $\mathbb{A}$ is the $n\times k$ loading matrix. The factors are assumed to be normally distributed, ${\mathbf{f}}_t\sim N(\mathbf{0},\mathsf{\Omega })$, where $\mathsf{\Omega }$ is $k\times k$, and they are independent from innovations that are multivariate-t distributed ${ϵ}_t\sim t_{\nu }(\mathbf{0},\mathsf{\Sigma })$, where $\mathsf{\Sigma }$ is $n\times n$. For the purpose of identification, we require $n\ge 2k+1$ and assume that $\mathbb{A}$ is lower triangular with diagonal entries all equal to 1 ([32]). This particular specification is commonly used in financial econometrics.

For maximum likelihood inference of such a model, the likelihood function can be maximised directly via Monte Carlo simulation. Let $\mathbf{Y}={({\mathbf{y}}_1,{\mathbf{y}}_2,..{\mathbf{y}}_T)}^{\prime }$ be the observations and $\mathit{\theta }={[{\beta }^{\prime },\mathrm{vech}{(\mathbb{A})}^{\prime },\mathrm{vech}(\mathsf{\Omega }),\mathrm{vech}(\mathsf{\Sigma })]}^{\prime }$ be the parameters. Under our setting, the likelihood function $g(y_t;\mathit{\theta })$ does not have an analytical expression, and it needs to be evaluated using numerical methods. Specifically, we can write ${\mathbf{f}}_t={\mathsf{\Omega }}^{1/2}{\mathbf{Z}}_t$ where ${\mathbf{Z}}_t\sim N(\mathbf{0},I_k)$. Therefore, it follows that:

$$g(y_t;\mathit{\theta })={\mathbb{E}}_{\mathbf{Z}}\left[t_{\nu }(y_t|\beta +\mathbb{A}{\mathsf{\Omega }}^{1/2}{\mathbf{Z}}_t,\mathsf{\Omega })\right],$$

where $t_{\nu }(·|\mu ,\mathsf{\Sigma })$ denotes the probability density function of the multivariate-t distribution. In the rest of this section, we focus on a case in which both $\mathsf{\Sigma }$ and $\mathsf{\Omega }$ are diagonal matrices: $\mathsf{\Sigma }=\mathrm{diag}({\sigma }_1^2,\dots ,{\sigma }_n^2)$ and $\mathsf{\Omega }=\mathrm{diag}({\omega }_1^2,\dots ,{\omega }_k^2)$.

For the implementation of the simulated maximum likelihood approach, we use the Ada-Delta variation of the stochastic gradient descent algorithm [33] for the estimation (with hyperparameters $(\gamma ,\eta ,ϵ)=(0.9,0.01,{10}^{-8})$). We first considered a simulated dataset of 1000 observations, each with 10 measurements, where the dimensions of the hidden factors were 3 and the entries in the factor loading matrix $\mathbb{A}$ were sampled from the standard normal distribution. The entries of the diagonal covariance matrices of the factors and the innovations were sampled from $U(1,5)$ and $U(0.5,1)$, respectively. The estimates converged at around 2000 iterations.

The first column of results in

Table 10. Run-time comparison between SMLE analysis of the factor model using either AD or (central) FD in the stochastic gradient computations. The per-iteration summaries are based on 100 evaluations under the simulated data example. The best performance in each column is highlighted in bold.

Run-Time Comparison of Simulated MLE
	Total Run-Time		Per Iteration Run-Time (Simulated Data)
	Simulated Data	Real Data	min	lq	Mean	Median	uq	max
AD	4.36 h	2.08 h	7.23 s	7.41 s	7.52 s	7.47 s	7.56 s	8.37 s
FD	12.04 h	6.10 h	20.35 s	20.51 s	20.76 s	20.60 s	20.69 s	26.82 s

reports the total run times with derivatives computed using AD and, for comparison, FD, for the simulated data example.

Under AD, the estimation took 4.36 h, compared with over 12 h using FD for the required derivative calculation. This reduction in run-time by almost 66% is confirmed by the remaining results in Table 10, which provide a more detailed run-time comparison between the two implementations, confirming the patterns from the overall run-time. The improvement was consistent for both per-iteration run-time and total run-time in the case of simulated data and for the total run-time for simulated to real data.

The real data set contained data on currency exchange rates. The sample included 1045 observations of daily returns of nine international currency exchange rates relative to the United States dollar from January 2007 to December 2010. We applied a factor model to the log of the exchange rate returns (i.e., $y_{it}=100log(p_{i,t}/p_{i,t-1})$, where $p_{it}$ denotes the daily closing spot rate for currency i at time t. The nine selected currencies were the Australian dollar (AUD), Canadian dollar (CAD), Euro (EUR), Japanese yen (JPY), Swiss franc (CHF), British pound (GBP), South Korean won (KRW), New Zealand dollar (NZD), and New Taiwan dollar (TWD), representing the most heavily traded currencies over the period. The estimates converged at around 1000 iterations. As reported in the second column of results in Table 10, we again observed a substantial reduction in estimation time, with the run time under an AD-based derivative computation being a third of the time required by the FD-based computation.

4. Conclusions

This paper presents a vectorised formulation of AD grounded in matrix calculus, tailored for statistical applications that involve high-dimensional inputs and simulation-intensive computations. The proposed approach uses a compact set of matrix calculus rules to enable efficient and automatic derivative computation and introduces optimisation techniques, such as memoisation, sparse matrix representation, matrix chain multiplication, and implicit Kronecker product, to improve the efficiency of the AD implementation.

Compared to other AD approaches, our formulation aligns more naturally with the matrix-oriented notation commonly used in statistics and econometrics. It supports fully automatic workflows similar to source code transformation methods while also providing direct access to intermediate variables for inspection and analysis. Unlike imperative AD frameworks, it does not require users to adopt AD-specific programming idioms. In addition, the approach enables high-level optimisation by explicitly making use of matrix structure and the order of matrix multiplications, which are typically inaccessible in scalar-based or imperative implementations.

Despite its advantages, our formulation of AD introduces computational overhead, which can make it less efficient than FD for small-scale or low-dimensional problems. In such cases, the simplicity of FD often results in faster performance (but at the cost of lower accuracy). The benefits of our approach become more apparent in high-dimensional settings, where its scalability and accuracy outweigh the initial overhead, as we showed in the numerical study. A further limitation arises when computing second-order derivatives such as the Hessian. Under our vectorised approach, this requires dual numbers to carry second-order matrix derivatives, which can quickly exceed the memory capacity of typical personal computers. For a function mapping ${\mathbb{R}}^m$ to ${\mathbb{R}}^n$, the Jacobian has dimension $n\times m$, while the Hessian grows to $nm\times m$. In contrast, FD computes these matrices entry by entry by perturbing the function input one coordinate at a time; although slower, this method avoids out-of-memory issues. Similar memory constraints may also occur when handling extremely large Jacobian matrices.

In view of the modern trend toward increasingly large and high-dimensional datasets, the performance overhead of AD in small-scale settings is becoming less of a practical concern. As data and models continue to grow in complexity, the scalability advantages of AD are expected to outweigh its initial costs in a broader range of applications. For the memory demands associated with higher-order derivatives, potential solutions include distributed computing, on-disk array storage, and blocked algorithms that process derivative computations in smaller, memory-efficient segments. These strategies offer promising avenues for extending the applicability of our vectorised AD approach to even larger and more demanding statistical problems.