Measurement Uncertainty Evaluation: Differences Between Virtual Experiments and the Standardized Approach

Abstract

Virtual experiments (VE) can be used to assess the measurement uncertainty of complex measurements. The typical calculation procedure implemented in such a VE, called VE-DA in this paper, is based on a Monte Carlo method involving simulating possible measurement errors and possible measurement data based on extensive modeling of the measurement instrument, followed by applying a data analysis function (DA) to evaluate the measurement data. This procedure is similar to the propagation of distributions using a Monte Carlo method (PoD) procedure presented in the written standard JCGM-101, in which the Monte Carlo method is applied to an explicit mathematical model for the measurand involving simulating and applying possible corrections to the observed measurement data. However, in this paper, we show that the uncertainty provided by the VE-DA procedure can be both larger and smaller than the uncertainty evaluated based on applying the PoD to the correct measurement model, when available. This is important to realize by users of the VE-DA procedure when claiming conformity of an uncertainty evaluation with JCGM-101.

1. Introduction

Virtual experiments (VEs) and digital twins (DTs) have attracted substantial attention in the metrological community [1,2,3,4,5], also illustrated by the fact that virtual metrology is listed as a key area in the strategic research agenda of EURAMET’s European Metrology Network Mathmet [6]. Uncertainty evaluation involving VEs and DTs is an important area of interest. Recent papers considered uncertainty evaluation for the applications of, e.g., coordinate measuring machines [7,8,9,10,11,12], tilted wave interferometry [13,14], flow measurements [15,16] and scatterometry [17,18]. Some more theoretically oriented papers considered the exact replication of JCGM-101 [19] distributions by means of using samples of VEs [20,21,22]. In [23] various possible procedures for uncertainty evaluation based on VEs were discussed and assessed using computer experiments involving a Virtual Coordinate Measuring Machine (VCMM). It was found that resulting uncertainties and coverage intervals could be significantly different for measurands that have a complex dependency on their input quantities, like the peak-to-valley value as a measure of the circularity of an approximately circular shape. Bayesian methods applied to VEs are also regularly studied [24,25], in particular because Bayesian methods naturally allow for uncertainty quantification of the estimated parameters. Comparing different ways of evaluating measurement uncertainty has been conducted in many different contexts. See, e.g., [26] for an example involving the analysis of heteroscedastic measurement data.

The problem addressed by this paper is the question of how a regularly employed uncertainty calculation procedure [9] in the context of VEs compares with the procedure based on the propagation of distributions (PoD), as presented in JCGM-101. This comparison will be performed in detail for a simple example model, which will be formally presented in Section 2. It is important to understand possible differences between these procedures, as the first procedure is often used in practice, while claiming consistency with the second procedure, i.e., with JCGM-101. These differences do not seem to have been studied in the past.

In Section 2, we present these procedures in detail. To obtain a better understanding of possible differences, we approximate the variances in the distributions resulting from both procedures using the law of propagation of uncertainty from JCGM-100 [27] and we give an analytical formula indicating how much the calculated uncertainties by both procedures can be different in specific adverse circumstances. In Section 3, we present some concrete numerical examples using the full Monte Carlo procedures, in order to confirm the intuition gained from the analytical formula. We also present an example with a simplified VCMM. In Section 4, we discuss the results and our conclusions are presented in Section 5.

2. Method

A VE models how an $n_x$-dimensional measurement data vector $\mathbf{x}$ is generated in an experiment, given an $n_y$-dimensional vector $\mathbf{y}$ containing the values of the (possibly) multi-dimensional measurand, an $n_z$-dimensional vector $\mathbf{z}$ containing influence quantities not varying over repeated measurements, and an $n_e$-dimensional vector $\mathbf{e}$ containing influence quantities that vary between different measurements. This leads to the VE function g:

$$\begin{array}{cc}\hfill \mathbf{x}& =g(\mathbf{y},\mathbf{z},\mathbf{e}).\hfill \end{array}$$

(1)

Typically, $\mathbf{e}$ affects the measurement data in the form of $n_x$-dimensional additive measurement noise, leading to the function $\tilde{g}$ not depending on $\mathbf{e}$:

$$\begin{array}{cc}\hfill \mathbf{x}& =\tilde{g}(\mathbf{y},\mathbf{z})+\mathbf{e}.\hfill \end{array}$$

(2)

If $\tilde{g}$ is once continuously differentiable with respect to $\mathbf{y}$, the VE can be expanded in powers of $\mathbf{y}$ as

$$\begin{array}{cc}\hfill \mathbf{x}& =\tilde{g}(\mathbf{y},\mathbf{z})+\mathbf{e}=\mathbf{a}\left(\mathbf{z}\right)+B\left(\mathbf{z}\right)\mathbf{y}+\mathbf{e}+{\mathcal{O}}_{\mathbf{z}}{(\parallel \mathbf{y}\parallel }^2),\hfill \end{array}$$

(3)

where ${\mathcal{O}}_{\mathbf{z}}$ denotes the Landau big O symbol, where the implied bounding constant can still depend on $\mathbf{z}$. We will, therefore, focus our study on a first-order VE of the form

$$\begin{array}{cc}\hfill \mathbf{x}& =\mathbf{a}\left(\mathbf{z}\right)+B\left(\mathbf{z}\right)\mathbf{y}+\mathbf{e}.\hfill \end{array}$$

(4)

This class of functions is regularly studied, e.g., in [28]. Furthermore, many models for the measurand can be represented as the partial inverse of this function, e.g., in calibration or correction models [29].

We assume Gaussian noise with zero mean $\widehat{\mathbf{e}}=\mathbf{0}$ and covariance matrix $V_{\mathbf{e}}$, and that an estimate $\widehat{\mathbf{z}}$ of the other influence quantities is given, as well as its covariance matrix $V_{\mathbf{z}}$. Given a data sample $\mathbf{x}$, the central question is to determine $\mathbf{y}$ and its covariance matrix $V_{\mathbf{y}}$.

We will now discuss two common procedures for evaluating $V_{\mathbf{y}}$. The first is the procedure from JCGM-101 based on the propagation of distributions (PoD), the second procedure relies on creating multiple data samples using the VE and applying a data analysis function (DA) to the sampled data (VE-DA). To clarify analytically what is happening in these sampling-based procedures, we use the law of the propagation of uncertainty (LPU) from JCGM-100 to approximate the variance in the distributions, which can be used to construct an approximate distribution for the measurand when a normal distribution is assumed.

2.1. Propagation of Distributions

The Monte Carlo method-based PoD procedure from JCGM-101 prescribes that the VE-equation (4) is first transformed into a measurement function f expressing the measurand as a function of all other variables. This can readily be conducted in our case, leading to

$$\begin{array}{cc}\hfill \mathbf{y}& =f(\mathbf{x},\mathbf{z},\mathbf{e})=B^{+}\left(\mathbf{z}\right)(\mathbf{x}-\mathbf{e}-\mathbf{a}\left(\mathbf{z}\right)),\hfill \end{array}$$

(5)

where $B^{+}\left(\mathbf{z}\right)={\left(B^{\mathrm{T}}\left(\mathbf{z}\right)B\left(\mathbf{z}\right)\right)}^{-1}{B}^{\mathrm{T}}\left(\mathbf{z}\right)$ denotes the left Moore–Penrose inverse of the matrix $B\left(\mathbf{z}\right)$ and gives the least-squares solution $\widehat{\mathbf{y}}$ for $\mathbf{y}$ in (4) given measured data $\mathbf{x}$ and estimates $\widehat{\mathbf{z}}$ for $\mathbf{z}$ and $\widehat{\mathbf{e}}$ for $\mathbf{e}$.

In the PoD procedure, the estimate $\widehat{\mathbf{y}}$ and covariance matrix $V_{\mathbf{y}}$ are calculated by repeatedly creating samples ${\mathbf{z}}^{\left(j\right)}$ and ${\mathbf{e}}^{\left(j\right)}$ from the state-of-knowledge probability distributions characterizing the input quantities $\mathbf{z}$ and $\mathbf{e}$, respectively. The function f from Equation (5) is applied to the measured data $\mathbf{x}$ and each sample pair $({\mathbf{z}}^{\left(j\right)},{\mathbf{e}}^{\left(j\right)})$, resulting in samples ${\mathbf{y}}^{\left(j\right)}$. These samples ${\mathbf{y}}^{\left(j\right)}$ can be used to estimate the value of the measurand by means of the mean $\widehat{\mathbf{y}}$ of the samples, and the standard uncertainty of the measurand by means of the empirical covariance matrix $V_{\mathbf{y}}$ of the samples. While the PoD procedure outlined here might deviate from the description of the Monte Carlo method presented in JCGM-101, as we have introduced a separate variable $\mathbf{e}$ for the measurement noise affecting the measured data $\mathbf{x}$, it is equivalent, since we assume a normal distribution for $\mathbf{e}$ with known covariance $V_{\mathbf{e}}$ and a single measured value for $\mathbf{x}$.

Using the LPU, a covariance matrix $V_{\mathbf{y}}^{\prime }$ approximating $V_{\mathbf{y}}$ can be computed by means of

$$\begin{array}{cc}\hfill V_{\mathbf{y}}^{\prime }& ={\displaystyle \frac{\partial f}{\partial \mathbf{z}}}{V}_{\mathbf{z}}{\left({\displaystyle \frac{\partial f}{\partial \mathbf{z}}}\right)}^{\mathrm{T}}+{\displaystyle \frac{\partial f}{\partial \mathbf{e}}}{V}_{\mathbf{e}}{\left({\displaystyle \frac{\partial f}{\partial \mathbf{e}}}\right)}^{\mathrm{T}},\hfill \end{array}$$

(6)

where $\mathbf{z}$ and $\mathbf{e}$ are assumed independent, and where $\mathbf{x}$ is treated as known, as the measurement noise is separately modeled by $\mathbf{e}$.

To be able to apply the LPU, the derivative $J_{B^{+}}={\displaystyle \frac{\mathrm{d}{B}^{+}}{dz}}$ of the matrix $B^{+}\left(\mathbf{z}\right)$ is needed, which is given by [30]:

$$\begin{array}{cc}\hfill J_{B^{+}}& =D_1+D_2\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill D_1& =-B^{+}{J}_B{B}^{+}\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em} D_2& =B^{+}{B}^{+\mathrm{T}}{J}_B^{\mathrm{T}}(I-B{B}^{+})\hfill \end{array}$$

(9)

where $J_B={\displaystyle \frac{\mathrm{d}B}{\mathrm{d}\mathbf{z}}}$. Note that $J_B$ and $J_{B^{+}}$ are 3-dimensional tensors when each of $\mathbf{x}$, $\mathbf{y}$ and $\mathbf{z}$ have more than one dimension and the operations in (8) and (9) correspond to tensor contractions over the appropriate dimensions. In most of the subsequent examples $\mathbf{z}$ is one-dimensional; therefore, we will not focus on this aspect in the following equations.

Writing

$$\begin{array}{cc}\hfill \tilde{\mathbf{x}}& =\mathbf{x}-\widehat{\mathbf{e}}-\mathbf{a}\left(\widehat{\mathbf{z}}\right)\hfill \end{array}$$

(10)

and $J_{\mathbf{a}}={\displaystyle \frac{\mathrm{d}\mathbf{a}}{\mathrm{d}\mathbf{z}}}$, the application of the LPU and the product rule of differentiation yields for the approximation $V_y^{\left(\mathrm{PoD}’\right)}$ of $V_y^{\left(\mathrm{PoD}\right)}$

$$\begin{array}{cc}\hfill V_y^{\left(\mathrm{PoD}’\right)}& =B^{+}(V_{\mathbf{e}}+J_{\mathbf{a}}{V}_{\mathbf{z}}{J}_{\mathbf{a}}^{\mathrm{T}})B^{+\mathrm{T}}+\left(J_{B^{+}}\tilde{\mathbf{x}}\right)V_{\mathbf{z}}{\left(J_{B^{+}}\tilde{\mathbf{x}}\right)}^{\mathrm{T}}\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=B^{+}{V}_{\mathbf{e}}{B}^{+\mathrm{T}}+B^{+}{J}_a{V}_z{J}_a^{\mathrm{T}}{B}^{+\mathrm{T}}+D_1\tilde{\mathbf{x}}{V}_{\mathbf{z}}{\tilde{\mathbf{x}}}^{\mathrm{T}}{D}_1^{\mathrm{T}}+D_2\tilde{\mathbf{x}}{V}_{\mathbf{z}}{\tilde{\mathbf{x}}}^{\mathrm{T}}{D}_2^{\mathrm{T}}+2D_1\tilde{\mathbf{x}}{V}_{\mathbf{z}}{\tilde{\mathbf{x}}}^{\mathrm{T}}{D}_2^{\mathrm{T}}.\hfill \end{array}$$

(12)

All matrices are evaluated at the estimate $\widehat{\mathbf{z}}$. To simplify the notation, we have removed this dependence on $\mathbf{z}$ from the notation.

2.2. VE-Based Sampling Procedure

In the procedure VE-DA a data analysis function $f_0$ must be available to compute an estimate of the measurand based on the measurement data only:

$$\begin{array}{c}\hfill \mathbf{y}=f_0\left(\mathbf{x}\right).\end{array}$$

(13)

If a full explicit measurement model f as in Equation (5) is available, then $f_0$ can be derived from f by fixing the inputs $\mathbf{z}$ and $\mathbf{e}$ to some estimates $\widehat{\mathbf{z}}$ and $\widehat{\mathbf{e}}$ (most often $\widehat{\mathbf{e}}=\mathbf{0}$).

An estimate ${\widehat{\mathbf{y}}}_0$ of the measurand is fixed, which can be based on the measurement data, e.g., ${\widehat{\mathbf{y}}}_0=f_0\left(\mathbf{x}\right)$ or on a nominal value for the measurand. Once ${\widehat{\mathbf{y}}}_0$ is chosen, repeatedly samples ${\mathbf{z}}^{\left(j\right)}$ and ${\mathbf{e}}^{\left(j\right)}$ are drawn from the probability distributions characterizing the input quantities $\mathbf{z}$ and $\mathbf{e}$, respectively. These samples are used to generate simulated data samples ${\mathbf{x}}^{\left(j\right)}$ using the VE from (1), or in our case, from (4). Next, the function $f_0$ is applied to the simulated data ${\mathbf{x}}^{\left(j\right)}$, yielding samples ${\mathbf{y}}^{\left(j\right)}$. Statistics like the covariance matrix $V_{\mathbf{y}}$ can be computed from these samples. In the case $f_0$ is unbiased, the mean $\overline{{\mathbf{y}}^{\left(j\right)}}$ of the samples ${\mathbf{y}}^{\left(j\right)}$ will be close to ${\widehat{\mathbf{y}}}_0$. In other cases, $\overline{{\mathbf{y}}^{\left(j\right)}}$ may be simply disposed of, or it can be used for a bias correction [23]. The reported value of the measurand is typically $f_0\left(\mathbf{x}\right)$ and not $\overline{{\mathbf{y}}^{\left(j\right)}}$, though this choice is irrelevant as regards the uncertainty evaluation and will not be discussed any further in this paper.

The procedure VE-DA finds application in VCMMs [9], with which simulated measurement data are repeatedly generated, to which the DA is then subsequently applied. This DA may consist of, e.g., fitting a geometrical element like a circle to the simulated data, and the diameter may be the parameter of interest. As fixed ${\widehat{\mathbf{y}}}_0$ either the nominal values for the measurand or an estimate based on the measured coordinates are typically used. On top of this geometrical shape, measurement errors due to CMM and object imperfections are simulated, resulting in simulated measurement data in the form of simulated point clouds.

The main discrepancy between the VE-DA and PoD procedures is that in VE-DA, possible measurement errors resulting in possible measurement data are simulated based on the parameter $\mathbf{z}$, whereafter a relatively simple data analysis function $\mathbf{y}=f_0\left(\mathbf{x}\right)$ that is independent of $\mathbf{z}$, is applied to evaluate the measurement data. In the PoD procedure, an explicit mathematical model for the measurand of the form $\mathbf{y}=f(\mathbf{x},\mathbf{z},\mathbf{e})$ is created first by inverting the function $\mathbf{x}=g(\mathbf{y},\mathbf{z},\mathbf{e})$ that models the data generation process. Hereafter, all uncertain influence factors as modeled by $\mathbf{z}$ and $\mathbf{e}$ are simulated and applied as possible corrections to the observed measurement data.

We again calculate the LPU-based covariance matrix $V_{\mathbf{y}}^{(\mathrm{VE}\text{-}\mathrm{DA}’)}$ approximating the covariance matrix $V_{\mathbf{y}}^{\left(\mathrm{VE}\text{-}\mathrm{DA}\right)}$ that would be obtained using the Monte Carlo method-based VE-DA procedure. In a first step, we assess the covariance matrix of the data $V_{\mathbf{x}}$, which, using $\widehat{\mathbf{y}}=B^{+}\tilde{\mathbf{x}}$, is given by

$$\begin{array}{cc}\hfill V_{\mathbf{x}}& =J_a{V}_{\mathbf{z}}{J}_a^{\mathrm{T}}+J_B\widehat{\mathbf{y}}{V}_{\mathbf{z}}{\left(J_B\widehat{\mathbf{y}}\right)}^{\mathrm{T}}+V_{\mathbf{e}}.\hfill \end{array}$$

(14)

This covariance matrix is then propagated through the function

$$\begin{array}{cc}\hfill f_0\left(\mathbf{x}\right)& =f(\mathbf{x},\widehat{\mathbf{z}},\widehat{\mathbf{e}})=B^{+}\left(\widehat{\mathbf{z}}\right)(\mathbf{x}-\widehat{\mathbf{e}}-\mathbf{a}\left(\widehat{\mathbf{z}}\right))\hfill \end{array}$$

(15)

yielding

$$\begin{array}{cc}\hfill V_{\mathbf{y}}^{(\mathrm{VE}\text{-}\mathrm{DA}’)}& =B^{+}{V}_{\mathbf{x}}{B}^{+\mathrm{T}}\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} =B^{+}(V_{\mathbf{e}}+J_a{V}_{\mathbf{z}}{J}_a^{\mathrm{T}}+J_B{B}^{+}\tilde{\mathbf{x}}{V}_{\mathbf{z}}{\left(J_B{B}^{+}\tilde{\mathbf{x}}\right)}^{\mathrm{T}})B^{+\mathrm{T}}\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} =B^{+}{V}_{\mathbf{e}}{B}^{+\mathrm{T}}+B^{+}{J}_{\mathbf{a}}{V}_{\mathbf{z}}{J}_{\mathbf{a}}^{\mathrm{T}}{B}^{+\mathrm{T}}+D_1\tilde{\mathbf{x}}{V}_{\mathbf{z}}{\tilde{\mathbf{x}}}^{\mathrm{T}}{D}_1^{\mathrm{T}}.\hfill \end{array}$$

(18)

Comparing Equations (18) and (12) we observe that Equation (12) has two additional terms involving $D_2$, which do not necessarily sum up to zero, and therefore, the VE-DA procedure does not always give identical results as the PoD procedure from JCGM-101. The reason is that the impact of the uncertainty of $\mathbf{z}$ on $B\left(\mathbf{z}\right)$ and $B^{+}\left(\mathbf{z}\right)$ is not evaluated in an equivalent way. The matrix $D_2$ is related to the null-space of B. In the first order, uncertainties lying in this null-space are not taken into account by the VE-DA procedure, whereas they are taken into account by the PoD procedure. If B was fixed, then any noise orthogonal to B would not matter, as it would not affect the calculated value for $\mathbf{y}$. However, due to the uncertainty in $\mathbf{z}$, a perturbed matrix $B\left(\mathbf{z}\right)$ may actually start containing components aligning with the noise, which results in additional uncertainty for $\mathbf{y}$. This part of the uncertainty is neglected by the VE-DA procedure. Note that if ${\displaystyle \frac{\mathrm{d}B}{\mathrm{d}\mathbf{z}}}=\mathbf{0}$ then both procedures yield the same result, as this would result in $D_1=D_2=\mathbf{0}$.

2.3. Ratio Between VE-DA and PoD Uncertainties

In this section, we analyze how large or small the ratio of the standard uncertainties calculated according to the different procedures for a single measurand y can become. We define the ratio

$$\begin{array}{cc}\hfill r& ={\displaystyle \frac{u_y^{\left(\mathrm{VE}\text{-}\mathrm{DA}\right)}}{u_y^{(\mathrm{PoD})}}}\hfill \end{array}$$

(19)

where $u_y^{\left(\mathrm{PoD}\right)}=\sqrt{V_{\mathbf{y}}^{\left(\mathrm{PoD}\right)}}$ denotes the standard uncertainty of the scalar measurand y using the PoD procedure, and similarly $u_y^{\left(\mathrm{VE}\text{-}\mathrm{DA}\right)}=\sqrt{V_{\mathbf{y}}^{\left(\mathrm{VE}\text{-}\mathrm{DA}\right)}}$ for the VE-DA procedure. We assume that the estimate $\widehat{\mathbf{z}}$ corresponds to the true value of $\mathbf{z}$, such that we can write $\mathbf{x}=\mathbf{a}\left(\widehat{\mathbf{z}}\right)+B\left(\widehat{\mathbf{z}}\right)\mathbf{y}+\mathbf{e}$ and that we do not mix in this analysis the effects of an incorrect estimate of $\mathbf{z}$ with the effect of using a different procedure for uncertainty evaluation. To approximate the ratio r we will analytically study the ratio $r^{\prime }$

$$\begin{array}{cc}\hfill r^{\prime }& ={\displaystyle \frac{u_y^{\left(\mathrm{VE}\text{-}\mathrm{DA}’\right)}}{u_y^{(\mathrm{PoD}’)}}}\hfill \end{array}$$

(20)

which is based on the LPU approximation of the variances resulting from applying VE-DA and PoD. We call the standard deviations of these approximations $u_y^{(\mathrm{VE}\text{-}\mathrm{DA}’)}$ and $u_y^{\left(\mathrm{PoD}’\right)}$. The analytical approximations are used to find models for which the VE-DA and PoD procedures may give substantially different results. Therefore, the approximation error is not a primary concern, as long as the results based on the LPU help to identify interesting cases for which the procedures yield different results. Nevertheless, we have excluded pathological examples corresponding to cases in which $B\left(z\right)$ is almost singular and for which the resulting PoD distribution had a very long tail, in contrast to the VE-DA solution.

From Equations (12) and (18) we see that the difference in the covariance matrices $V_{\mathbf{y}}^{(\mathrm{VE}\text{-}\mathrm{DA}’)}$ and $V_{\mathbf{y}}^{(\mathrm{PoD}’)}$ is equal to

$$\begin{array}{c}\hfill D_2\tilde{\mathbf{x}}{V}_{\mathbf{z}}{\tilde{\mathbf{x}}}^{\mathrm{T}}{D}_2^{\mathrm{T}}+2D_1\tilde{\mathbf{x}}{V}_{\mathbf{z}}{\tilde{\mathbf{x}}}^{\mathrm{T}}{D}_2^{\mathrm{T}}.\end{array}$$

(21)

and that it is governed by the matrix-vector products $D_1\tilde{\mathbf{x}}$ and $D_2\tilde{\mathbf{x}}$. Two extreme cases for the value of the difference shown in Equation (21) are given by $D_1\tilde{\mathbf{x}}=-D_2\tilde{\mathbf{x}}$, in which the VE-DA uncertainty is larger than the PoD uncertainty, and $D_1\tilde{\mathbf{x}}=\mathbf{0}$ and $D_2\tilde{\mathbf{x}}\ne \mathbf{0}$, in which the VE-DA uncertainty is smaller than the PoD uncertainty. Furthermore, we choose $\mathbf{a}\left(z\right)\equiv \mathbf{0}$ to get rid of the term $J_{\mathbf{a}}{V}_{\mathbf{z}}{J}_{\mathbf{a}}$ that would draw the ratio (20) towards unity.

We now assume that both $\mathbf{y}=y$ and $\mathbf{z}=z$ are one-dimensional, whereas $\mathbf{x}$ remains $n_x$-dimensional. By choosing the physical units of y and z and/ or scaling and/ or translating y and z appropriately, we can arrange

$$\begin{array}{cc}\hfill y^{\left(\mathrm{true}\right)}& =1\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em} \widehat{z}& =z^{\left(\mathrm{true}\right)}=0.\hfill \end{array}$$

(23)

Let $V_{\mathbf{z}}=u_z^2$. By appropriate scaling of $\mathbf{x}$ we can arrange that $\parallel B\left(\widehat{z}\right)\parallel =1$, where $B=B\left(\widehat{z}\right)$ is actually a vector in the case of a single-dimensional measurand. To alleviate the notation, we will remove $\widehat{z}$ from quantities depending on z when they are evaluated at $\widehat{z}$. We obtain $B^{\mathrm{T}}B=1$ and $B^{+}=B^{\mathrm{T}}$. By an appropriate choice of the basis vectors of the $n_x$-dimensional space, we can, without loss of generality, assume that

$$\begin{array}{cc}\hfill B& ={(1,0,\dots ,0)}^{\mathrm{T}}\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill \hspace{1em}{J}_B& ={({\mu }_1,{\mu }_2,0,\dots ,0)}^{\mathrm{T}}\hfill \end{array}$$

(25)

for some constants ${\mu }_1$ and ${\mu }_2$. Note however that in this paper, we only intend to present some examples of cases in which uncertainty evaluation methods start to differ, so we do not strictly need to cover all cases. For the extreme cases, we choose

$$\begin{array}{cc}\hfill \mathbf{e}& ={({\lambda }_1,{\lambda }_2,0,\dots ,0)}^{\mathrm{T}},\hfill \end{array}$$

(26)

to have only noise components in the relevant directions, while keeping the norm $\parallel \mathbf{e}\parallel$ as small as possible. Using these definitions, we find that

$$\begin{array}{cc}\hfill \tilde{\mathbf{x}}& ={(y+{\lambda }_1,{\lambda }_2,0,\dots ,0)}^{\mathrm{T}}\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill \hspace{1em}{D}_1\tilde{\mathbf{x}}& =-B^{+}{J}_B{B}^{+}\tilde{\mathbf{x}}=-{\mu }_1(y+{\lambda }_1)\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill \hspace{1em} D_2\tilde{\mathbf{x}}& =B^{+}{B}^{+\mathrm{T}}{J}_B^{\mathrm{T}}(I-B{B}^{+})\tilde{\mathbf{x}}={\mu }_2{\lambda }_2.\hfill \end{array}$$

(29)

Plugging Equations (28) and (29) into (12) and (18) and the resulting expressions into Equation (20) leads to the ratio

$$\begin{array}{cc}\hfill r^{\prime }& ={\displaystyle \frac{\sqrt{u_e^2+{\left({\mu }_1(y+{\lambda }_1)\right)}^2{u}_z^2}}{\sqrt{u_e^2+{({\mu }_1(y+{\lambda }_1)-{\mu }_2{\lambda }_2)}^2{u}_z^2}}}.\hfill \end{array}$$

(30)

2.3.1. VE-DA Uncertainty Larger than PoD Uncertainty

In order to maximize $r^{\prime }$ in Equation (30), corresponding to $D_1\tilde{\mathbf{x}}=-D_2\tilde{\mathbf{x}}$, we can choose as model parameters

$$\begin{array}{cc}\hfill {\lambda }_1& =0,{\lambda }_2=k_e{u}_e,{\mu }_1={\displaystyle \frac{{\mu }_2{\lambda }_2}{y_{\mathrm{true}}}}.\hfill \end{array}$$

(31)

where both components of $\mathbf{e}$ have standard uncertainty $u_e$, and $k_e$ denotes an arbitrary factor. Assuming that the estimate $\widehat{y}$ for the measurand y equals $y_{\mathrm{true}}$, this leads to the ratio

$$\begin{array}{cc}\hfill r^{\prime }& ={\displaystyle \frac{\sqrt{u_e^2+{\mu }_1^2{y}^2{u}_z^2}}{u_e}}=\sqrt{1+{\left({\mu }_2{k}_e{u}_z\right)}^2}.\hfill \end{array}$$

(32)

It can be seen that the ratio (32) is always larger than 1 and increases when either the uncertainty in z becomes large or the impact of the additive effect of z orthogonal to B in the model for the measurand increases, i.e., ${\mu }_2$; choosing ${\mu }_2=1$, $u_z=1$ and $u_e=2$ leads to $r^{\prime }=\sqrt{2}$. Note that by an appropriate choice of the unit for the z-values, either ${\mu }_2$ or $u_z$ can be assumed to be equal to 1, as long as ${\mu }_2\ne 0$ and $u_z\ne 0$. Numerical simulations show that the ratio $\sqrt{2}$ is approximately attainable for the Monte Carlo method-based procedures VE-DA and PoD, from which we conclude that the approximation $r^{\prime }$ to r is appropriate for such cases. For larger values of ${\mu }_2$ or $u_z$ the approximation $r^{\prime }$ to r becomes poor and cannot be relied on. In such cases, the obtained distribution for y using the PoD procedure becomes highly skewed with a long tail.

2.3.2. VE-DA Uncertainty Smaller than PoD Uncertainty

We now turn to the opposite case in which we aim to minimize $r^{\prime }$ from Equation (30) corresponding to $D_1\tilde{\mathbf{x}}=\mathbf{0}$ and large $\parallel D_2\tilde{\mathbf{x}}\parallel$. Setting the model parameters to

$$\begin{array}{cc}\hfill {\lambda }_1& =0,{\lambda }_2=k_e{u}_e,{\mu }_1=0\hfill \end{array}$$

(33)

we find the ratio

$$\begin{array}{cc}\hfill r^{\prime }& ={\displaystyle \frac{u_e}{\sqrt{u_e^2+{\left({\mu }_2{k}_e{u}_e{u}_z\right)}^2}}}={\displaystyle \frac{1}{\sqrt{1+{\left({\mu }_2{k}_e{u}_z\right)}^2}}}.\hfill \end{array}$$

(34)

The ratio in (34) is actually the inverse of the ratio in (32), the only difference between the two cases being the choice for the value of ${\mu }_1$. Causes for a ratio significantly different from unity are a large derivative in the direction orthogonal to B (i.e., large ${\mu }_2$), a large uncertainty $u_z$ or a large realization of the noise vector $\mathbf{e}$ (i.e., large $k_e$), whereby the noise vector should be orthogonal to B. Simulations show that, in particular, by assuming a very large $k_e$, one can obtain a very small $r^{\prime }$ and r.

The mathematical intuition for achieving $D_1\tilde{\mathbf{x}}=0$ and large $\parallel D_2\tilde{\mathbf{x}}\parallel$ is as follows. The matrix $D_1$ will vanish if $J_B$ is orthogonal to $B^{+}$. The matrix $D_2$ equals the product of essentially three factors: the orthogonal projection $(I-B{B}^{\mathrm{T}})$, the factor $J_B^{\mathrm{T}}$ and the factor $\left(B^{+}{B}^{+\mathrm{T}}\right)$. As $\tilde{\mathbf{x}}=B\mathbf{y}+\mathbf{e}$ (as $\widehat{\mathbf{z}}=\mathbf{z}$ and $\widehat{\mathbf{e}}=\mathbf{0}$) and $(I-B{B}^{\mathrm{T}})$ is the orthogonal projection to the space orthogonal to the column space of B, the product $(I-B{B}^{\mathrm{T}})\tilde{\mathbf{x}}$ has the largest Euclidian norm for vectors of fixed length if $\mathbf{e}$ is orthogonal to the columns of B and will then be equal to $\mathbf{e}$. The product $J_B^{\mathrm{T}}\mathbf{e}$ has the largest absolute value for vectors of fixed norm if $J_B$ is parallel to $\mathbf{e}$. Finally, $\left(B^{+}{B}^{+\mathrm{T}}\right)={\left(B^{\mathrm{T}}B\right)}^{-1}$ is only a one-dimensional scalar if $\mathbf{y}$ is one-dimensional.

3. Numerical Simulations

In this section, we present some numerical simulations illustrating the difference in calculated uncertainty using the VE-DA and PoD procedures. We start with five synthetic cases with the identifiers (IDs) ‘Lin-Hetero-Large’, ‘Lin-Hetero-Small’, ‘Lin-Homo-Large’, ‘Lin-Homo-Small’ and ‘Trigon’ which were constructed to clearly demonstrate the possibility of obtaining different uncertainties as presented in the last section. The sixth case with ID ‘Circle’ is a more realistic example based on coordinate metrology and a VCMM.

In the Monte Carlo method-based calculations, we used ${10}^6$ repetitions for the numerical values presented in

Table 1. Comparison of uncertainty evaluation results. In the synthetic cases presented in the top part of the table, $\mathbf{x}$, y and z have arbitrary units indicated by [a.u.].

Case ID	${\mathit{u}}_{{\mathit{e}}_1}$ [a.u.]	${\mathit{u}}_{{\mathit{e}}_2}$ [a.u.]	${\mathit{u}}_{\mathit{z}}$ [a.u.]	$\mathit{u}\left({\widehat{\mathit{y}}}^{\left(\mathbf{VE}\text{-}\mathbf{DA}\right)}\right)$ [a.u.]	$\mathit{u}\left({\widehat{\mathit{y}}}^{\left(\mathbf{PoD}\right)}\right)$ [a.u.]	r -
Lin-Hetero-Large	0.02	0.1	0.1	0.036	0.027	1.35
Lin-Hetero-Small	0.02	0.1	0.1	0.020	0.039	0.51
Lin-Homo-Large	0.2	0.2	0.2	0.117	0.112	1.04
Lin-Homo-Small	0.2	0.2	0.2	0.100	0.121	0.83
Trigon	0.5	0.5	0.5	0.523	0.847	0.62
Case ID	${\mathit{u}}_{\mathit{e}}$ [$\mathsf{\mu }$m]	${\mathit{u}}_{{\mathit{s}}_{\mathit{p}}},{\mathit{u}}_{{\mathit{s}}_{\mathit{q}}}$ [$\mathsf{\mu }$m/m]	${\mathit{u}}_{{\mathit{s}}_{\mathit{pq}}}$ [mrad]	$\mathit{u}\left({\widehat{\mathit{r}}}_{\mathit{0}}^{\left(\mathbf{VE}\mathbf{\text{-}}\mathbf{DA}\right)}\right)$ [$\mathsf{\mu }$m/m]	$\mathit{u}\left({\widehat{\mathit{r}}}_{\mathit{0}}^{\left(\mathbf{PoD}\right)}\right)$ [$\mathsf{\mu }$m/m]	$\mathit{r}$ -
Circle	0.5	0.1	0.1	3.5	3.5	1.00

, and ${10}^5$ repetitions for the generation of the histograms due to calculation time. We assumed a one dimensional measurand and an $n_x=2$ dimensional data-space for the x-values with $B\left(\widehat{z}\right)={(1,0)}^{\mathrm{T}}$ such that $(I-B{B}^{+})\ne \mathbf{0}$. All cases used the following values: $y^{\left(\mathrm{true}\right)}=1$ (by an appropriate scaling of the y-axis), $\widehat{z}=0$ (by an appropriate translation of the z-axis), ${\mu }_2=1$ (by an appropriate scaling of the z-axis) and $k_e=3$ (a concrete realization of the noise vector equal to 3 times the standard deviation). The values ${\lambda }_1$, ${\lambda }_2$ and ${\mu }_1$ were chosen according to Equation (31) for the cases ‘Lin-Hetero-Large’ and ‘Lin-Homo-Large’, resulting in a VE-DA uncertainty larger than the PoD uncertainty, whereas they were chosen according to Equation (33) for the cases ‘Lin-Hetero-Small’, ‘Lin-Homo-Small’ and ’Trigon’, resulting in a VE-DA uncertainty smaller than the PoD uncertainty. The value $u_z$ was chosen as large as possible while ensuring that, in particular, the PoD procedure still resulted in an approximately normal distribution for y, and in a similar way $u_e$ was selected. In all cases, the noise components $e_1$ and $e_2$ of the noise vector $\mathbf{e}$ were independent. In the cases ‘Lin-Hetero-Large’ and ‘Lin-Hetero-Small’, the noise was heteroscedastic (i.e., different variance), whereas in the cases ‘Lin-Homo-Large’, ‘Lin-Homo-Small’ and ‘Trigon’ the noise was homoscedastic (i.e, same variance). In the cases where the ID starts with ‘Lin’ we considered a linear function of the form

$$\begin{array}{cc}\hfill B\left(z\right)& =B\left(\widehat{z}\right)+z{J}_B\hfill \end{array}$$

(35)

with $B\left(\widehat{z}\right)$ from Equation (24) and $J_B$ from Equation (25), whereas in the case ‘Trigon’ the function

$$\begin{array}{cc}\hfill B\left(z\right)& ={(cos\left(z\right),sin\left(z\right))}^{\mathrm{T}}\hfill \end{array}$$

(36)

involving trigoniometric functions was used.

The case ‘Circle’ has a 3-dimensional vector $\mathbf{z}$ in contrast to the 1-dimensional $\mathbf{z}=z$ of the preceding cases. Here, we consider a more realistic situation corresponding to a least-squares circle fit for determining the mid-point $(p_0,q_0)$ and radius $r_0$ to four geometrical points with coordinates ${(p_i,q_i)}_{1\le i\le 4}$ with nominal measurement directions ${\varphi }_i$ parallel to the coordinate axes in the horizontal plane. For the uncertainty structure the simplified VCMM from [23] has been used. This problem can be cast in the form of the model of Equation (4) by expressing the measured data, the linear scale errors $s_p$ and $s_q$ and the squareness error $s_{pq}$ as

$$\begin{array}{cc}\hfill {\varphi }_i& \in \{0,\pi /2,\pi ,3\pi /2\}\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} x_i& =p_0+r_{0}cos\left({\varphi }_i\right)+e_i\mathrm{for} i \mathrm{u}\mathrm{n}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em} x_i& =q_0+r_{0}sin\left({\varphi }_i\right)+e_i\mathrm{f}\mathrm{o}\mathrm{r} i \mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\hfill \end{array}$$

(39)

and defining

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em} \mathbf{y}& ={(p_0,q_0,r_0)}^{\mathrm{T}}\hfill \end{array}$$

(40)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em} \mathbf{z}& ={(s_p,s_q,s_{pq})}^{\mathrm{T}}\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em} \widehat{\mathbf{z}}& ={(0,0,0)}^{\mathrm{T}}\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill \mathbf{a}\left(\mathbf{z}\right)& =\mathbf{0}\hfill \end{array}$$

(43)

$$\begin{array}{cc}\hfill \hspace{1em} B\left(\mathbf{z}\right)& =B_1\left(\mathbf{z}\right)B_0\hfill \end{array}$$

(44)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{B}_0& ={\left(\begin{array}{cccccccc}1& 0& 1& 0& 1& 0& 1& 0\\ 0& 1& 0& 1& 0& 1& 0& 1\\ 1& 0& 0& 1& -1& 0& 0& -1\end{array}\right)}^{\mathrm{T}}\hfill \end{array}$$

(45)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} B_1\left(\mathbf{z}\right)& =\left(\begin{array}{cccc}A\left(\mathbf{z}\right)& \mathbf{0}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& A\left(\mathbf{z}\right)& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& A\left(\mathbf{z}\right)& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{0}& A\left(\mathbf{z}\right)\end{array}\right)\hfill \end{array}$$

(46)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}A\left(\mathbf{z}\right)& =\left(\begin{array}{cc}1+s_p& s_{pq}\\ 0& 1+s_q\end{array}\right).\hfill \end{array}$$

(47)

This model of shape (4) can be used for simulating the generation of measurement data. For the inverse operation, i.e., for fitting the circle parameters, Equation (5) cannot directly applied in practice, as the angles ${\varphi }_i$ needed for the computation of $B\left(\widehat{\mathbf{z}}\right)$ are only approximately known when performing a measurement. Therefore, a general least-squares fitting routine for fitting circle parameters to data [31] is used, like what is conducted in practice.

A worst case situation in terms of the ratio r occurs when both the realization of the noise vector $\mathbf{e}$ and the derivative $J_B$ of $B\left(z\right)$ are parallel to each other and orthogonal to $B_0$, e.g., if they are both parallel to

$$\begin{array}{cc}\hfill \mathbf{q}& ={(1,0,-1,0,1,0,-1,0)}^{\mathrm{T}}.\hfill \end{array}$$

(48)

In Figure 1 a schematic overview of the situation is shown including a measurement noise vector aligning with $\mathbf{q}$. However, it turned out that the derivative $J_B$ for this practical case is not at all orthogonal to $B_0$, leading to virtually identical uncertainties for both uncertainty procedures. In the simulation we used $p_0=1$ $\mathsf{\mu }$m, $q_0=2$ $\mathsf{\mu }$m and $r_0=50$ mm.

A summary of the results of the simulations is shown in Table 1. The calculated estimates ${\widehat{y}}^{\left(\mathrm{PoD}\right)}$ and ${\widehat{y}}^{\left(\mathrm{VE}\text{-}\mathrm{DA}\right)}$ were both reasonably close to $y^{\left(\mathrm{true}\right)}$ in each case. We chose not to present them in Table 1 in order not to distract the attention from the calculated uncertainties, which is the focus of this study.

In the left parts of Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6 the distributions for the five synthetic cases are shown. Both the smooth Gaussian distribution using the LPU approximation as well as the stair-case plots resulting from the histograms based on the Monte Carlo samples are shown. In all cases, the approximation is reasonable to very good. If we were to allow more wild distributions or larger values of $k_e$, then the ratio r would attain more extreme values than shown in this paper. Table 1 presents the ratios and in the left part of Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6 the distributions are shown for particular values of the noise $\mathbf{e}$, which were chosen to obtain extreme values of the ratio. We also performed experiments in which we kept the model shape fixed, but drew random samples for both $\widehat{z}$ and $\mathbf{e}$ to obtain an impression of the distribution of the ratio in a random case, rather than in the worst case, although the model structure is still the worst case. In the right part of Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6 histograms are drawn for the resulting ratios based on 10,000 repetitions. For case ‘Circle’ the resulting histogram is entirely concentrated at 1 and, therefore, we did not present similar figures for this last case.

4. Discussion

From the results in Table 1 we conclude that significant differences can occur between the VE-DA and PoD procedures, with the ratio r ranging from 0.51 to 1.35 in the cases considered. The numerical experiments also made it clear that it is easier to find examples with the PoD uncertainty being larger than the VE-DA uncertainty than the opposite. Further increasing the factor $k_e$ is possible but unrealistic and not conducted in this paper. It would nevertheless show that both analysis procedures can, in theory, give even more different results. Increasing ${\mu }_2$ and/ or $u_z$ makes the distributions behave in a non-Gaussian way, resulting in ratios r quite different from $r^{\prime }$, though in some cases still very different from unity. The histograms show that also in the case of randomly generated noise and an estimate of the additional z-parameter not coinciding with its true values, the ratio can still deviate considerably from unity, as long as the VE model itself has the particular shape with $J_B$ approximately orthogonal to B. Also, the values shown in Table 1 are covered by the histogram, indicating that the presented numerical values and, in particular, the employed value of $k_e=3$ are realistic for each case. Figure 2b and Figure 4b show that ratios below unity are more likely, even if the model structure was constructed to obtain a ratio larger than one. In Figure 3b, Figure 5b, and Figure 6b, we do not observe the opposite effect: all ratios are at most equal to one in these histograms.

In general, we observe that using a Monte Carlo method for drawing samples from the distribution for z and for the measurement noise and using this in combination with a mathematical model for the measurement is not sufficient to obtain the same result as the PoD procedure from JCGM-101. The point in the procedure at which the measured data $\mathbf{x}$ is converted to a value $\mathbf{y}$ for the measurand matters. The VE-DA procedure neglects, in the first order, some of the uncertainty related to the null-space of B, caused by the absence of the factor $(I-B{B}^{+})$ in Equation (18). However, in practical cases, the factor $\sqrt{1+{\left({\mu }_2{k}_e{u}_z\right)}^2}$ will often be very close to unity, as the product ${\mu }_2{u}_z$ will generally be small.

One may ask the question what the ‘correct’ uncertainty is, which is actually a general topic for which a considerable body of literature exists, including diverging views on it [32,33,34,35]. From a normative viewpoint, that would be the one based on the PoD procedure, as this procedure is described in JCGM-101, which is also known as ISO/IEC Guide 98-3-1. From a Bayesian point of view, one may also opt for the PoD procedure, as the propagation of distributions procedure aligns with the idea of propagating state-of-knowledge distributions through a model, while the JCGM-101 approach is not always compatible with a complete Bayesian point of view. Classical statisticians may want to consider the success rates of calculated coverage intervals, which have also been studied inside the metrological community [18,36]. We assessed the success rates, but did not find significant differences on average. The success rates of 95% coverage intervals constructed for both procedures in all five cases, as determined for the 10,000 repetitions considered for each of the histograms, were close to 95%. However, for specific choices of the parameters, there were sometimes large differences, and therefore, simply preferring the method yielding on average the shortest intervals did not seem ideal either, in particular as worst-case parameter settings are of main interest to this paper. So, no clear best procedure was identified by this criterion.

In this paper, we considered a first-order model in both $\mathbf{y}$ and $\mathbf{z}$ and used a least-squares inversion. When considering stronger non-linear models and other types of inversion and data processing, like Chebyshev or minimum-zone fitting, larger differences may occur. However, such models are much harder to analyze analytically than the model of Equation (4) studied in this paper, though a numerical comparison can be performed as long as the VE model can be transformed to a measurement model expressing the measurand explicitly as a function of the input quantities. Note that in practice it is often easier to apply the VE-DA procedure, whereas it can be practically infeasible to properly invert the virtual experiment and to express the measurand directly as a function of the measured data and other influence quantities.

A limitation of this study is that we illustrated the difference between the procedures only for a few, mainly synthetic, cases, whereas one may wonder what the difference is for a large number of practical cases. The fact that we did not find an example from real-life measurements for which both procedures give different results also limits the direct impact of the study.

The broader implication of this study is that metrologists cannot apply the VE-DA procedure to evaluate the measurement uncertainty and claim that it is compliant with JCGM-101 without any further thoughts. It needs to be assessed if, for the employed measurement model, both procedures give identical results or not.

5. Conclusions

In this paper, we demonstrated that the VE-DA procedure, as typically used in VCMMs and other applications, is an approximation to the PoD procedure. For most models the results from the VE-DA procedure will have excellent correspondence with the PoD outcomes, but in some particularly adverse circumstances the results can significantly deviate. In this paper we presented synthetic examples in which the uncertainty calculated by VE-DA was either 35% larger or 49% smaller than as calculated using PoD. However, in a more realistic example, the results of both methods coincided. It remains a topic of future research to find a practical example in which both procedures yield different results, with the metrologist possibly not being aware of the difference.