What Is Special About the Kirkwood–Dirac Distributions? Only They Produce Natural Conditional Expectations

Abstract

Among the many quasiprobability representations of quantum mechanics, the family of Kirkwood–Dirac (KD) representations has come to the foreground in recent years. Each such KD representation is determined by the choice of two complementary complete sets of commuting observables $\widehat{A}$ and $\widehat{B}$ with respect to which it is Born-compatible, meaning that it correctly reproduces their Born probabilities for every state. In this paper, we identify what property uniquely characterizes the KD representations among all such $\widehat{A}$ and $\widehat{B}$ Born-compatible quasiprobability representations. For that purpose, we first define a natural notion of a quantum conditional expectation of an observable $\widehat{X}$, given an observable $\widehat{Y}$, in a state $\widehat{\rho }$, as a best estimator, and we show that it has the basic properties generally expected of a conditional expectation. We then show that only the KD representations provide a notion of conditional, expectation given $\widehat{B}$ (or given $\widehat{A}$) that coincides with the above quantum conditional expectation. As a byproduct of our analysis, we show a state-dependent no-go theorem. We prove that, if the quantum conditional expectation of an observable $\widehat{X}$, given an observable $\widehat{Y}$ in a state $\widehat{\rho }$ admits an anomalous value (meaning a value lying outside the interval $[x_{min},x_{max}]$), then there cannot exist a Born-compatible joint probability distribution $\mu (x,y)$ for $\widehat{X}$ and $\widehat{Y}$ in the state $\widehat{\rho }$ for which the associated conditional probability $\mu \left(x\right|y)$ yields a conditional expectation that coincides with the quantum conditional expectation. We further apply our findings to revisit a standard model for phase estimation in quantum metrology. We show in particular that, within the real sector of a given KD representation, the classical Fisher information of this phase estimation problem vanishes identically.

1. Introduction

Within quantum mechanics, it is not possible to properly define the joint probability distribution of two incompatible (i.e., non-commuting) observables in a coherent manner for all states [1,2,3,4]. This impossibility is one of the characteristic features of quantum mechanics, distinguishing it from classical mechanics. For the prototypical incompatible observables of position and momentum, Wigner famously found a way around this impossibility by associating to each quantum state a quasiprobability distribution for position and momentum, having the correct marginals, but that can take negative values [5]. A large variety of other quasiprobability distributions for position and momentum have subsequently been developed and have been proven useful in various fields of physics [6,7,8,9,10,11,12,13], even though not all these constructions yield joint quasiprobability distributions: their marginals do not always coincide with the Born-rule probabilities for position and momentum. Extensions of these constructions to abelian groups other than the space translations and their associated Heisenberg groups have also been developed [14,15,16,17,18,19,20,21,22,23,24,25]. Other approaches include coherent state constructions [26,27,28,29]. All these approaches fit in a general unifying setup using frames and dual frames on the quantum Hilbert space $\mathcal{H}$. This setup allows one to define a quasiprobability distribution $Q\left(\widehat{\rho }\right)$ for all quantum states $\widehat{\rho }$ and symbols $\tilde{Q}\left(\widehat{X}\right)$ for all operators $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$ [3,30], both defined on a space $\Lambda$, thus forming a quasiprobability representation of quantum mechanics (see Section 4 for details). Here, $\mathcal{L}\left(\mathcal{H}\right)$ is the set of linear operators on $\mathcal{H}$; we suppose throughout that $dim\mathcal{H}=d<+\infty$.

Among the countless quasiprobability representations of quantum mechanics so obtained feature the ones introduced by Dirac [8]. Suppose one is given two complete sets of commuting operators (CSCO) $\widehat{A}$ and $\widehat{B}$. Dirac then proposed, for each quantum state $\widehat{\rho }$, two joint quasiprobability distributions $Q^{\mathrm{KD},\ell }\left(\widehat{\rho }\right)$ and $Q^{\mathrm{KD},\mathrm{r}}\left(\widehat{\rho }\right)$ (both with the correct marginals, therefore) on the space $\sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$ where $\sigma \left(\widehat{A}\right)$ and $\sigma \left(\widehat{B}\right)$ denote the spectra of $\widehat{A}$ and $\widehat{B}$:

$$Q_{a,b}^{\mathrm{KD},\ell }\left(\widehat{\rho }\right)=\mathrm{Tr}\left({\widehat{\Pi }}_a^{\widehat{A}}\widehat{\rho }{\widehat{\Pi }}_b^{\widehat{B}}\right),Q_{a,b}^{\mathrm{KD},\mathrm{r}}\left(\widehat{\rho }\right)=\mathrm{Tr}\left({\widehat{\Pi }}_b^{\widehat{B}}\widehat{\rho }{\widehat{\Pi }}_a^{\widehat{A}}\right),$$

(1)

where ${\widehat{\Pi }}_a^{\widehat{A}}$ and ${\widehat{\Pi }}_b^{\widehat{B}}$ are the one-dimensional spectral projectors of $\widehat{A}$ onto the eigenspace of $a\in \sigma \left(\widehat{A}\right)$ and of $\widehat{B}$ onto the eigenspace of $b\in \sigma \left(\widehat{B}\right)$, respectively. Throughout, we will assume that $\widehat{A}$ and $\widehat{B}$ are complementary CSCO, meaning that ${\widehat{\Pi }}_a^{\widehat{A}}{\widehat{\Pi }}_b^{\widehat{B}}\ne 0$ for all $a,b$. The reason for the choice of superscript ℓ, for “left” and $\mathrm{r}$ for “right” will become clear below. $Q^{\mathrm{KD},\ell }$ and $Q^{\mathrm{KD},\mathrm{r}}$ are today referred to as the Kirkwood–Dirac (KD) distributions [8]: they do indeed generalize the construction of Kirkwood [7] for the special case of position and momentum. The naturally associated KD symbols ${\tilde{Q}}^{\mathrm{KD},\ell }\left(\widehat{X}\right)$ and ${\tilde{Q}}^{\mathrm{KD},\mathrm{r}}\left(\widehat{X}\right)$ for any operator $\widehat{X}$,

$${\tilde{Q}}_{a,b}^{\mathrm{KD},\ell }\left(\widehat{X}\right)={\displaystyle \frac{\mathrm{Tr}\left({\widehat{\Pi }}_a^{\widehat{A}}\widehat{X}{\widehat{\Pi }}_b^{\widehat{B}}\right)}{\mathrm{Tr}\left({\widehat{\Pi }}_a^{\widehat{A}}{\widehat{\Pi }}_b^{\widehat{B}}\right)}},{\tilde{Q}}_{a,b}^{\mathrm{KD},\mathrm{r}}\left(\widehat{X}\right)={\displaystyle \frac{\mathrm{Tr}\left({\widehat{\Pi }}_b^{\widehat{B}}\widehat{X}{\widehat{\Pi }}_a^{\widehat{A}}\right)}{\mathrm{Tr}\left({\widehat{\Pi }}_b^{\widehat{B}}{\widehat{\Pi }}_a^{\widehat{A}}\right)}},$$

(2)

correspond to a choice of ordering between the operators $\widehat{A}$ and $\widehat{B}$: $\widehat{A}$ before $\widehat{B}$ for ℓ and $\widehat{B}$ before $\widehat{A}$ for $\mathrm{r}$. Indeed, the KD symbol ${\tilde{Q}}^{\mathrm{KD},\ell }\left(\widehat{X}\right)$ of the operator $\widehat{X}=f\left(\widehat{A}\right)g\left(\widehat{B}\right)$ is readily verified to be the function $(a,b)\mapsto f\left(a\right)g\left(b\right)$, whereas the KD symbol ${\tilde{Q}}^{\mathrm{KD},\mathrm{r}}\left(\widehat{X}\right)$ of $\widehat{X}=g\left(\widehat{B}\right)f\left(\widehat{A}\right)$ is also $(a,b)\mapsto f\left(a\right)g\left(b\right)$. (See Section 4 for details.) The flexibility of this construction, applying as it does to general non-commuting observables $\widehat{A}$ and $\widehat{B}$, has proven to be an asset in an increasing number of applications in the last few years; we refer to the recent reviews [4,31] for details.

The question then arises: what singles out the Kirkwood–Dirac quasiprobability representations of quantum mechanics among all possible such representations? The central result of this paper is that they are, in a precise sense to be explained below (see Theorem 1), the only ones that are well-behaved with respect to a natural notion of conditional expectation in quantum mechanics that we introduce. Equivalently, the KD conditional expectation is the only one that can be naturally interpreted as a best estimator and that, as such, coincides with the conditional expectation proposed in weak value physics. We present several applications of these results. We establish a simple state-dependent no-go result for the existence of joint probabilities in quantum mechanics (Section 3.3). In Section 6.3, we show that in a standard model for phase estimation in quantum metrology, the Fisher information vanishes within the real sector of a given KD representation. This result provides a novel interpretation of the real sector of KD representations and allows us to show that the optimal measurement yielding the quantum Fisher information cannot be KD real (Proposition 7). We further explain to what extent KD-positive states (for which $Q_{a,b}^{\mathrm{KD}}\left(\widehat{\rho }\right)\ge 0$), can be considered “classical” by exhibiting some of their nonclassical features.

In the rest of this introduction, we outline the paper. To precisely state and then establish our results, we first need to revisit the notions of conditional expectation used in both probability theory (Section 2) and quantum mechanics (Section 3 and Section 4). We shall use two distinct approaches to the definition of quantum conditional expectation. One approach consists in adapting to quantum mechanics the standard conditional expectation ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)$ of a random variable X given a random variable Y, defined as the complex-valued function $f\left(Y\right)$ of Y that minimizes the mean squared error

$${\mathbb{E}}_{\mathbb{P}}{\left(\right|X-f\left(Y\right)|}^2):={\int }_{\Omega }{|X\left(\omega \right)-f\left(Y\left(\omega \right)\right)|}^2\mathrm{d}\mathbb{P}\left(\omega \right),$$

(3)

$\mathbb{P}$ being the underlying probability measure on the space $\Omega$ on which the random variables X and Y are defined. (See Section 2). In statistics, ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)$ is referred to as the best predictor or best estimator of X among all complex-valued functions of Y. The idea that underlies this definition is that the law of ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)$ provides some information about the law of X. For example, it is easily verified that their first moments agree:

$${\mathbb{E}}_{\mathbb{P}}\left({\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)\right)={\mathbb{E}}_{\mathbb{P}}\left(X\right).$$

(4)

This relation, known as the iterated expectation formula, expresses the fact that the best estimator ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)$ of X is “unbiased”. In addition, one has the following well-known relation between their variances, referred to as the law of total variance or the variance decomposition formula:

$${\Delta }^{2}X={\Delta }^2{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)+{\mathbb{E}}_{\mathbb{P}}\left(\right|X-{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y){|}^2)={\Delta }^2{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)+{\mathbb{E}}_{\mathbb{P}}\left({\Delta }_{\mathbb{P}}^2\left(X\right|Y)\right).$$

(5)

This relation decomposes the total variability of X in a term due to the variability of the conditional expectation ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)$, plus an error term evaluating the fluctuations of X around this conditional expectation. This error term equals the expected value of the conditional variance ${\Delta }_{\mathbb{P}}^2\left(X\right|Y)$ of X, given Y. In particular, the smaller is this mean squared error, the closer are their variances. In that case, the variability of X is considered to be well explained by that of ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)$.

In Section 3, we define a conditional expectation in the quantum mechanical context, using a similar minimization argument [32,33,34,35]. We will then show that identities (see Equations (68) and (69)) analogous to, but in important ways strikingly different from Equations (4) and (5) hold for this quantum conditional expectation. Let $\widehat{X}$ be an operator on a Hilbert space $\mathcal{H}$ and let $\widehat{Y}$ be a CSCO. Then, for any given mixed state $\widehat{\rho }$ (positive trace-class operator of trace 1), we consider the following expression,

$$\mathrm{Tr}\left(\widehat{\rho }{(\widehat{X}-f\left(\widehat{Y}\right))}^{†}(\widehat{X}-f\left(\widehat{Y}\right))\right)=\mathrm{Tr}\left((\widehat{X}-f\left(\widehat{Y}\right))\widehat{\rho }{(\widehat{X}-f\left(\widehat{Y}\right))}^{†}\right),$$

(6)

where f is complex valued, and which is a natural analog of Equation (3). We then define, following [34,35], the conditional expectation ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ as that operator function $f\left(\widehat{Y}\right)$ of $\widehat{Y}$ which minimizes the expression in Equation (6) (see Definition 2). As in classical probability theory, where the conditional expectation depends on the underlying probability $\mathbb{P}$, the quantum conditional probability ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ depends on the state $\widehat{\rho }$. It can be computed explicitly:

$${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})=\sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\langle {\phi }_y^{\widehat{Y}},\widehat{X}\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }}{\widehat{\Pi }}_y^{\widehat{Y}},$$

(7)

where, for technical reasons and to ensure uniqueness of the solution, $\widehat{\rho }$ is supposed to belong to

$$D_{\widehat{Y}}=\{\widehat{\rho }\in D\left(\mathcal{H}\right):\forall y\in \sigma \left(\widehat{Y}\right),\mathrm{Tr}\left(\widehat{\rho }{\widehat{\Pi }}_y^{\widehat{Y}}\right)\ne 0\}.$$

(8)

Note that ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ is not necessarily self-adjoint, even if $\widehat{X}$ is. This is the first marked difference from what happens in probability theory: if X is a real random variable, then the conditional expectation ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)$ is also real-valued. The physical meaning of the real and imaginary parts of ${\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right|\widehat{Y})$ will be further discussed in Section 3 and Section 6.

The coefficients in Equation (7) are so-called “weak values”. They were introduced in [36], where they were given an operational meaning through an experimental procedure referred to as a weak measurement, that involves the weak coupling of the system to a meter whose position and/or momentum can be measured. For completeness and the reader’s convenience, we describe this setup and the relevant analysis in some detail in Appendix A. The nature of this experimental procedure has subsequently suggested the interpretation of the weak values as conditional expectations [37,38]. This course of events may leave one with the impression that the notion of conditional expectation in quantum mechanics depends on the experimental setup associated with weak values. Our results in Equation (7) show that this is not the case. On the contrary, weak values and their interpretation in terms of quantum conditional expectations arise naturally, in an experiment-independent manner, from the definition of conditional expectations via a quadratic minimization problem analogous to the one used to define conditional expectations in probability theory. In particular, the conditional expectation of $\widehat{X}$ given $\widehat{Y}$ can be viewed as the “best estimator” of $\widehat{X}$ by a function of $\widehat{Y}$. Following this line of thought, the possibility of experimental determination of the coefficients of ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ in Equation (7) in a weak value experiments is then an a posteriori observation that provides an operational interpretation to the quantum conditional expectation.

Even if one does agree that defining conditional expectations via a minimization procedure is appealing and elegant, one may still wonder why one should choose to minimize a quadratic error. In order to give additional justification for this choice beyond the fact that it provides a natural analogue of the classical definition and the operational interpretation in terms of weak values just given, we provide, in Section 3.2, a characterization of the quantum conditional expectation in Equation (7): we show in Theorem 5 that, for $\widehat{\rho }\in D_{\widehat{Y}}$, the map $\widehat{X}\mapsto {\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ is uniquely characterized by the following two properties, naturally associated with a conditional expectation and which also characterize conditional expectations in probability theory:

(i)

Pull-out formula: ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(f\left(\widehat{Y}\right)\widehat{X}\right|\widehat{Y})=f\left(\widehat{Y}\right){\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$, for all functions $f\left(\widehat{Y}\right)$ of $\widehat{Y}$ and for all $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$;

(ii)

Iterated expectations: ${\mathbb{E}}_{\widehat{\rho }}\left({\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\right)={\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right)$.

In (ii), we wrote ${\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right):=\mathrm{Tr}\left(\widehat{X}\widehat{\rho }\right)$ to stress the similarity with the analogous properties that hold for the probabilistic notion of conditional expectation, with $\widehat{\rho }$ playing the rôle of the probability measure $\mathbb{P}$. Property (ii) expresses the fact that the best estimator ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ of $\widehat{X}$ is “unbiased” in the sense that it has the same expected value (in the state $\widehat{\rho }$) as $\widehat{X}$ itself. It is the analog of Equation (4) in probability theory.

Finally, in Proposition 2 we show a quantum version of the law of total variance

$$\begin{array}{ccc}\hfill {\Delta }_{\widehat{\rho }}^{\ell /\mathrm{r},2}\widehat{X}& =& {\Delta }_{\widehat{\rho }}^{\ell /\mathrm{r},2}{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})+{\mathbb{E}}_{\widehat{\rho }}\left({\Delta }_{\widehat{\rho }}^{\ell /r,2}\left(\widehat{X}\right|\widehat{Y})\right).\hfill \end{array}$$

(9)

Here, ${\Delta }_{\widehat{\rho }}^{\ell /r,2}\left(\widehat{X}\right|\widehat{Y})$ is the conditional variance of $\widehat{X}$, given $\widehat{Y}$, defined in Equation (66). This equation is clearly analogous to Equation (5). It shows that the variance of $\widehat{X}$ equals the sum the variance of ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ and of the expected value of the conditional variance of $\widehat{X}$, given $\widehat{Y}$. We use these results to derive a sharpened version of an additive uncertainty principle first proposed in [37], see Section 6.2.

Having stressed the analogies between the quantum and classical (by which we mean here probabilistic) conditional expectations, we then identify a number of marked differences between them in Section 3.3 and explain their physical interpretation. We show in particular a simple but effective no-go theorem for the existence of joint probabilities. Consider $\widehat{\rho },\widehat{X}$ and $\widehat{Y}$ such that the quantum conditional expectation has a value ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}=y)$ that falls outside the interval $[x_{min},x_{max}]$ (where $x_{min,max}$ are the extremal eigenvalues of $\widehat{X}$): such values are said to be anomalous. Then, there does not exist a joint probability distribution for $\widehat{X}$ and $\widehat{Y}$ with the right Born marginals in $\widehat{\rho }$, and such that the corresponding conditional expectation for $\widehat{X}$, given $\widehat{Y}$, constructed with Bayes’ rule, equals the quantum conditional expectation ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}=y)$. We refer to Lemma 3 for a precise statement. The strength of this result lies in the fact that it puts a requirement only on a fixed triplet $(\widehat{\rho },\widehat{X},\widehat{Y})$ to preclude the existence of a joint probability. It also gives a precise meaning to the statement that the presence of anomalous values of the quantum conditional expectation points to “nonclassicality”.

The superscript ℓ appearing in ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ stands for “left” because in property (i) above $f\left(\widehat{Y}\right)$ appears to the left of $\widehat{X}$. In fact, although the above definition of a quantum conditional expectation as a best estimator is very natural; the choice of Equation (6) as the quantum equivalent to Equation (3) of the quantity to be minimized surreptitiously hides a choice of operator ordering. This should be expected, since the passage from the classical observable ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)$ to a quantum equivalent involves a choice of quantization of observables, and as such can be expected to imply a choice of ordering. Indeed, an equally natural quantum analog of Equation (3) is

$$\mathrm{Tr}\left(\widehat{\rho }(\widehat{X}-f\left(\widehat{Y}\right)){(\widehat{X}-f\left(\widehat{Y}\right))}^{†}\right)=\mathrm{Tr}\left({(\widehat{X}-f\left(\widehat{Y}\right))}^{†}\widehat{\rho }(\widehat{X}-f\left(\widehat{Y}\right))\right).$$

(10)

Note that this quantity differs from Equation (6) only by the order in which $(\widehat{X}-f\left(\widehat{Y}\right))$ and ${(\widehat{X}-f\left(\widehat{Y}\right))}^{†}$ appear. The same minimization procedure as above then leads to a different conditional expectation, which we denote using ${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$. Again, the map $\widehat{X}\mapsto {\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ is uniquely characterized by property (ii) above as well as by the following version of (i):

$${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}f\left(\widehat{Y}\right)\right|\widehat{Y})=f\left(\widehat{Y}\right){\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{Y}),\forall f\left(\widehat{Y}\right),\forall \widehat{X}\in \mathcal{L}\left(\mathcal{H}\right),$$

(11)

in which $f\left(\widehat{Y}\right)$ appears to the right of $\widehat{X}$. The “left” and “right” conditional expectation are simply related to each other:

$${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})={\mathbb{E}}_{\widehat{\rho }}^{\ell }{\left({\widehat{X}}^{†}\right|\widehat{Y})}^{†}.$$

(12)

Contrary to the classical conditional expectation of a real random variable, which is real, the quantum conditional expectations ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ are not necessarily self-adjoint, even if $\widehat{X}$ is. When, on the other hand, either ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ or ${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ is self-adjoint, for some self-adjoint $\widehat{X}$, we have

$${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}{\left(\widehat{X}\right|\widehat{Y})}^{†}={\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})={\mathbb{E}}_{\widehat{\rho }}^{\ell }{\left(\widehat{X}\right|\widehat{Y})}^{†}.$$

(13)

In other words, in that case the choice of ordering plays no role and the “left” and “right” conditional expectations are identical.

There exists a second definition of conditional expectation that can be used in quantum mechanics, and that we revisit in Section 4. It starts from the observation that, in probability theory, the notion of conditional expectation ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)$ of a random variable X, given a random variable Y, both defined on an underlying probability space $\Omega$ with probability $\mathbb{P}$ (both assumed to be discrete), can be defined in terms of their conditional probability distribution $\mathbb{P}(X=x|Y=y)$, which in turn is defined in terms of their joint probability distribution $\mathbb{P}(X=x,Y=y)$. Since, as pointed out above, such joint probabilities do not exist in quantum mechanics for non-commuting observables, this definition cannot straightforwardly be adapted to the quantum context. One approach to resolve this issue consists in replacing probabilities by quasiprobabilities and then proceeding in analogy with the classical treatment [38,39]. As we explain in Section 4, this approach leads to a notion of conditional expectation ${\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{Y})$ as an operator that depends not only on $\widehat{\rho }$, but also on the quasiprobability representation used in its definition. In addition, as we will show, the conditional expectation ${\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{Y})$ so defined does not generally have all the natural properties one would expect from a conditional expectation. In particular, whereas it does always satisfy the law of iterated expectations (property (ii) above), it does not in general satisfy the pull-out property (property (i) above). This is not surprising since, as we saw, the pull-out property together with the law of iterated expectations does uniquely fix the definition of conditional expectation (Theorem 5).

After these preparatory developments, we turn in Section 5.2 and Section 5.3 to our principal result, which is the unique characterization of the KD quasiprobability representations, introduced in Section 5.1. We consider the set of all quasiprobability representations $(Q,\tilde{Q})$, defined on some finite set $\Lambda$ with $d^2$ elements, that are Born-compatible with two given CSCO $\widehat{A}$ and $\widehat{B}$. This means that the marginals of the quasiprobability distribution $Q\left(\widehat{\rho }\right)$ of any state $\widehat{\rho }$ with respect to the symbols $\tilde{Q}\left(\widehat{A}\right)$ and $\tilde{Q}\left(\widehat{B}\right)$ yield the correct quantum mechanical Born probabilities for these two observables (see Definition 3 for the precise definition). Depending on the goal pursued, the set $\Lambda$ is sometimes referred to as the “ontic space” or simply as the “phase space” of the quasiprobability representation. Our central result concerning the KD representations of quantum mechanics is then summed up in the following theorem:

Theorem 1.

Let $\widehat{A}$ and $\widehat{B}$ be complementary $\mathrm{CSCO}$. Let $(Q,\tilde{Q})$ be an $\widehat{A}$ and $\widehat{B}$-compatible quasiprobability representation of quantum mechanics defined on a set Λ ($\left|\Lambda \right|=d^2)$. Then the following are equivalent:

(i) 

$\forall \widehat{\rho }\in D_{\widehat{B}},\forall \widehat{X}\in \mathcal{L}\left(\mathcal{H}\right),{\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{B})={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{B})$

(ii) 

$\forall \widehat{\rho }\in D_{\widehat{A}},\forall \widehat{X}\in \mathcal{L}\left(\mathcal{H}\right),{\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{A})={\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{A})$

(iii) 

There exists a bijective map $\Phi :\Lambda \to \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$ such that for all $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$

$$\forall \widehat{\rho }\in D\left(\mathcal{H}\right),Q_{{\Phi }^{-1}(a,b)}\left(\widehat{\rho }\right)=Q_{a,b}^{\mathrm{KD},\ell }\left(\widehat{\rho }\right)\mathrm{and}\forall \widehat{X}\in \mathcal{L}\left(\mathcal{H}\right),{\tilde{Q}}_{{\Phi }^{-1}(a,b)}\left(\widehat{X}\right)={\tilde{Q}}_{a,b}^{\mathrm{KD},\ell }\left(\widehat{X}\right).$$

The theorem can be paraphrased by saying that of all $\widehat{A}$ and $\widehat{B}$ Born-compatible quasiprobability representations of quantum mechanics, only the left KD representation has conditional expectations that coincide with the best estimators ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{B})$ and ${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{A})$. Theorem 1 is a generalization of a result presented by the authors in [40]. In that work, the class of quasiprobability representations among which the KD representation is proven to be unique was considerably restricted by the a priori assumption that $\Lambda =\sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$. When using quasiprobability representations in the study of foundational issues such as contextuality and hidden variables, this is not a natural assumption, and thus it was lifted here.

We will provide two independent proofs of this result. One uses the characterization of ${\mathbb{E}}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ via the pull-out property (Theorem 7). The other shows it follows from a slightly stronger statement (Theorem 9) that is itself based on Theorem 8 which is of interest in its own right: it shows that, quite generally, $\widehat{A}$ and $\widehat{B}$ Born-compatible quasiprobability representations of quantum mechanics are always uniquely determined by their conditional expectations given $\widehat{A}$ or given $\widehat{B}$. The proof of Theorem 8 uses the techniques developed here with an argument found in [41], where a partial result similar to Theorem 9, but limited to the case where $\Lambda =\widehat{\sigma }\left(\widehat{A}\right)\times \widehat{\sigma }\left(\widehat{B}\right)$, was recently announced and partially proven.

As we saw, one of the striking differences between classical and quantum conditional expectations is that the latter can be non-self-adjoint. In addition, even if they are self-adjoint, they can take classically forbidden values (Lemma 3) and, in this sense, still signal typical quantum behavior such as, for example, weak value amplification or quantum advantage in quantum metrology [31,42,43,44]. In Section 6, we further analyze the meaning of the imaginary part of the quantum conditional expectation values and more specifically of its vanishing. For that purpose, we first recall the role played by the imaginary part of the quantum conditional expectation in a well-known problem of phase estimation in quantum mechanics. We consider a one-parameter family of states

$${\widehat{\rho }}_{\widehat{X}}\left(\theta \right)=exp(-i\theta \widehat{X})\widehat{\rho }exp\left(i\theta \widehat{X}\right),$$

(14)

where $\theta \in \mathbb{R}$ is referred to as the phase and $\widehat{X}$ is self-adjoint. The outcome probabilities of repeated measurements of a CSCO $\widehat{Y}$ in ${\widehat{\rho }}_{\widehat{X}}\left(\theta \right)$ are given by $p(y;\theta )=\mathrm{Tr}\left({\widehat{\Pi }}_y^{\widehat{Y}}{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)\right)$ ($y\in \sigma \left(\widehat{Y}\right)$), and we write ${\mathrm{I}}_{\mathrm{F}}(\widehat{Y};\theta )$ for the Fisher information of $p(y;\theta )$. We first then recall the well-known relation [37,45] between the above Fisher information ${\mathrm{I}}_{\mathrm{F}}(y;\theta )$ and the variance of the imaginary part of ${\mathbb{E}}_{{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$:

$${\mathrm{I}}_{\mathrm{F}}(\widehat{Y};\theta )=4\mathrm{Tr}\left({\left(\mathrm{Im}{\mathbb{E}}_{{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})\right)}^2{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)\right).$$

(15)

Since it is well known that, in order to be able to obtain a good estimate on $\theta$, one needs a large value of the Fisher information, the imaginary part of the conditional expectation ${\mathbb{E}}_{{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ contains information on the phase $\theta$. For that reason, we will say that a state is phase insensitive (for a given choice of $\widehat{X}$ and of $\widehat{Y}$) if its Fisher information ${\mathrm{I}}_{\mathrm{F}}(\widehat{Y};0)$ vanishes. To further justify this terminology, we give an operational interpretation of the vanishing of Fisher information in Appendix D. As pointed out above, in the context of the phase estimation problem considered, vanishing of the associated Fisher information is equivalent to saying that the conditional expectation ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ is self-adjoint. When this is the case, the probability distribution $p(y;\theta )$ cannot be used to efficiently estimate $\theta$. Using the quantum law of total variance (Equation (9)), we sharpen an additive uncertainty relation first established in [37] between the Fisher information $I_{\mathrm{F}}(\widehat{Y},0)$ and the variance of the real part of the conditional expectation, denoted by ${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})$. We show

$$\underset{\widehat{Y}}{min}{\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})\le {\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{L})\le {\Delta }_{\widehat{\rho }}^2\widehat{X}-{\displaystyle \frac{1}{4}}{I}_{\mathrm{QF}}\left(0\right)\le {\Delta }_{\widehat{\rho }}^2\widehat{X}=\underset{\widehat{Y}}{max}{\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y}).$$

(16)

Here, $I_{\mathrm{QF}}\left(0\right)$ is the quantum Fisher information of $\widehat{\rho }$ and $\widehat{X}$ and $\widehat{L}$ is the symmetric logarithmic derivative of ${\widehat{\rho }}_{\widehat{X}}\left(\theta \right)$. (See Appendix E for an introduction to the quantum Fisher information.) When $\widehat{\rho }$ is pure, one more precisely has

$$\underset{\widehat{Y}}{min}{\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})={\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{L})={\Delta }_{\widehat{\rho }}^2\widehat{X}-{\displaystyle \frac{1}{4}}{I}_{\mathrm{QF}}\left(0\right)=0,{\Delta }_{\widehat{\rho }}^2\widehat{X}=\underset{\widehat{Y}}{max}{\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y}),$$

(17)

where the maximum on the right is reached when $\widehat{\rho }$ is phase insensitive for the given choice of $\widehat{X}$ and $\widehat{Y}$. In other words, when $\widehat{Y}$ is chosen close to $\widehat{L}$, then the Fisher information $I_{\mathrm{F}}(\widehat{Y};0)$ is close to its maximal value $I_{\mathrm{QF}}$ and the variance of ${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})$ is close to its minimal value and vice versa.

In light of this analysis, it is of interest to know for which triplets $\widehat{\rho },\widehat{X},\widehat{Y}$, the Fisher information ${\mathrm{I}}_{\mathrm{F}}(\widehat{Y};0)$ vanishes; we call such triplets phase insensitive. For that purpose, we place ourselves in the framework of KD representations and exploit Theorem 1. We show that, if $\widehat{\rho }$ and $\widehat{X}$ have a real KD symbol with respect to some CSCO $\widehat{A}$ and $\widehat{B}=\widehat{Y}$, then ${\mathrm{I}}_{\mathrm{F}}(\widehat{Y};0)$ vanishes (Section 6.3); hence, $(\widehat{\rho },\widehat{X},\widehat{B})$ form a phase-insensitive triplet. This observation provides an interpretation of the “real sector” of a given KD representation and shows it is inefficient for the above phase estimation problem. Since recent work has shown that the set of all self-adjoint operators with a real KD symbol can in many cases be explicitly identified [23,24,46], these results also provide many examples of phase insensitive states. Finally, supposing that $\widehat{\rho }$ and $\widehat{X}$ are KD real, we investigate their associated quantum Fisher information. We show that the latter is non-vanishing if and only if $\widehat{\rho }$ and $\widehat{X}$ do not commute. It is well known that it is obtained by optimizing the choice of observable $\widehat{Y}$, taking it to be equal to the symmetric logarithmic derivative $\widehat{L}$ of ${\widehat{\rho }}_{\widehat{X}}\left(0\right)$. We then show that $\widehat{L}$ cannot be KD real when $\widehat{\rho }$ is pure (Proposition 7). In other words, if $\widehat{\rho }$ and $\widehat{X}$ are KD-real, then the optimal observable $\widehat{L}$ is not KD-real.

It should be noted that a triplet $(\widehat{\rho },\widehat{X},\widehat{B})$ can be phase-insensitive, with $\widehat{\rho }$ KD positive, while the conditional expectation ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{B})$ still allows for anomalous values. This shows once again that the word “classical” needs to be given a precise meaning when used.

We have completed this paper with a number of appendices with the goal to make it reasonably self-contained and to provide, for the reader’s convenience, proofs of important properties not necessarily easily accessible in the literature. In Appendix A, we provide a short but rather complete introduction to weak value theory and in particular to weak value measurement. We have included a discussion of the conditional expectations and variances of meter positions in the weak limit which connects with recent work [47]. In Appendix B, we briefly explain the link between our definition of conditional expectations and a definition used in the context of von Neumann algebras, restricted to our finite dimensional setup. Appendix C contains illustrative examples and counterexamples. Appendix D gives operational interpretations of the vanishing of the Fisher information for parameter estimation. Appendix E provides an introduction to the quantum Fisher information. Using the ideas developed in the paper, we give a simple proof of the result of Braunstein and Caves showing that the Helström–Holevo quantum Fisher information is obtained as the maximum over all possible measurements of the classical Fisher information associated to those measurements. In Appendix F, we provide a variational characterization of the KD distribution of a state.

It is a pleasure to dedicate this work to Jean-Pierre Gazeau on the occasion of his 80th birthday. Jean-Pierre has been for decades an efficient and enthusiastic advocate of phase space representations of quantum mechanics, coherent states and frames, and their applications in quantum theory and signal analysis. We hope he will find our contribution of interest. SDB is particularly grateful to Jean-Pierre for having welcomed him in the French scientific world over 35 years ago and for his continued support and friendship throughout all these years.

2. Classical Conditional Expectation and Conditional Variance

In this section, we collect some basic definitions and properties of the classical conditional expectation as used in probability theory. This allows us to fix notation and prepare for the quantum formulation.

Let $\Omega$ be a finite set. Let $X:\Omega \to \mathbb{C}$ be a complex random variable, and let $Y:\Omega \to {\mathbb{R}}^m$ be a real vector-valued random variable. We write $\mathrm{Ran}\left(X\right)$ for the range of X, meaning

$$\mathrm{Ran}\left(X\right)=\{x\in \mathbb{C}\mid \exists \omega \in \Omega ,X(\omega )=x\}=X(\Omega ),$$

(18)

and similarly for Y. We use $D_Y$ to denote the set of probability measures $\mathbb{P}$ on $\Omega$ such that for each $y\in \mathrm{Ran}\left(Y\right)$, $\mathbb{P}(Y=y)>0$. The space of complex random variables on $\Omega$ is denoted by $\mathcal{F}(\Omega ;\mathbb{C})$. We first recall the elementary definition of “conditional expectation” of a random variable.

Definition 1.

Let $\mathbb{P}\in D_Y$. For $y\in \mathrm{Ran}\left(Y\right)$ and $x\in \mathbb{C}$, the conditional probability that X takes the value x knowing $Y=y$ is given by

$$\mathbb{P}(X=x|Y=y)={\displaystyle \frac{\mathbb{P}(X=x,Y=y)}{\mathbb{P}(Y=y)}}.$$

(19)

The conditional expectation of X knowing that $Y=y$ is

$${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y=y)=\sum _{x\in \mathrm{Ran}\left(\mathrm{X}\right)}x\mathbb{P}(X=x|Y=y)=\sum _{x\in \mathrm{Ran}\left(\mathrm{X}\right)}x{\displaystyle \frac{\mathbb{P}(X=x,Y=y)}{\mathbb{P}(Y=y)}}.$$

(20)

We will, as usual, define the conditional expectation of X, knowing Y, denoted by ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)$, as the function

$${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)\left(y\right)={\mathbb{E}}_{\mathbb{P}}\left(X\right|Y=y).$$

(21)

Note that ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)$ is a random variable that is a function of Y. Also,

$${\mathbb{E}}_{\mathbb{P}}\left(f\left(Y\right)\right|Y)=f\left(Y\right)\mathrm{and}{\mathbb{E}}_{\mathbb{P}}\left({\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)\right)={\mathbb{E}}_{\mathbb{P}}\left(X\right),$$

(22)

as is readily verified. Finally, if X is a real random variable, then ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)$ is also real-valued. As we will see, this is a marked difference from what happens in quantum mechanics, where the conditional expectation of an observable can be non self-adjoint.

We note that this definition, and all the properties that follow in this section, are strongly dependent on the initially chosen probability $\mathbb{P}$ on $\Omega$. We indicate this dependence in the notation to anticipate the quantum conditional probability defined in the next section, which also depends on the $\widehat{\rho }$ state considered.

We now provide two distinct characterizations of ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)$ that will be essential in the following sections in order to define a quantum conditional expectation. For $\mathbb{P}\in D_Y$, we equip $\mathcal{F}(\Omega ;\mathbb{C})$ with the following sesquilinear form:

$${\langle X,X^{\prime }\rangle }_{\mathbb{P}}={\mathbb{E}}_{\mathbb{P}}\left(\overline{X}{X}^{\prime }\right)={\int }_{\Omega }\overline{X}\left(\omega \right)X^{\prime }\left(\omega \right)\mathrm{d}\mathbb{P}=\sum _{\omega \in \Omega }\overline{X}\left(\omega \right)X^{\prime }\left(\omega \right)\mathbb{P}\left(\left\{\omega \right\}\right).$$

(23)

Theorem 2.

Let $\mathbb{P}\in D_Y$. The conditional expectation of X knowing Y is the unique minimizer of

$$f\mapsto {\mathbb{E}}_{\mathbb{P}}{\left(\right|X-f\left(Y\right)|}^2{)=\langle X-f\left(Y\right),X-f\left(Y\right)\rangle }_{\mathbb{P}},$$

(24)

over the set of complex valued functions f defined on $\mathrm{Ran}\left(Y\right)$.

We use ${\mathcal{F}}_{\mathbb{C},Y}$ to denote the set

$${\mathcal{F}}_{\mathbb{C},Y}=\left\{f\left(Y\right)\mid f:\mathrm{Ran}\left(Y\right)\to \mathbb{C}\right\}.$$

(25)

${\mathcal{F}}_{\mathbb{C},Y}$ is thus the set of all complex random variables on $\Omega$ that are functions of Y. In other words, $Z:\Omega \to \mathbb{C}$ belongs to ${\mathcal{F}}_{\mathbb{C},Y}$ provided that Z is constant on all level sets of Y. This means that there exists a function $f:\mathrm{Ran}\left(Y\right)\to \mathbb{C}$ such that $Z=f\left(Y\right)$. Note that ${\mathcal{F}}_{\mathbb{C},Y}$ is a complex vector space (${\mathcal{F}}_{\mathbb{C},Y}\subset \mathcal{F}(\Omega ;\mathbb{C})$); it is in addition closed under multiplication of functions. In the usual language of Hilbert space theory, the minimizer of Equation (24)—which is the conditional expectation of X given Y—is the orthogonal projection of X onto ${\mathcal{F}}_{\mathbb{C},Y}$. In more advanced treatments of probability, where the space $\Omega$ is not finite and where the probability measures are general, this property provides a simple way to define the conditional expectation, since Definition 1 then does not necessarily make sense. We will see in the next section that this definition adapts nicely to quantum theory as well.

Proof. 

For $f:\mathrm{Ran}\left(Y\right)\to \mathbb{C}$, we have

$$f\left(y\right)=\sum _{y^{\prime }\in \mathrm{Ran}\left(Y\right)}f\left(y^{\prime }\right){\mathbb{1}}_{y^{\prime }}\left(y\right)$$

(26)

where ${\mathbb{1}}_{y^{\prime }}\left(y\right)$ is 1 if $y=y^{\prime }$ and 0 otherwise. One then computes

$$\begin{array}{cc}\hfill {\mathbb{E}}_{\mathbb{P}}{\left(\right|X-f\left(Y\right)|}^2)& ={\mathbb{E}}_{\mathbb{P}}{\left(\right|X|}^2)-2\sum _{y\in \mathrm{Ran}\left(Y\right)}\mathrm{Re}\left(\overline{f\left(y\right)}{\mathbb{E}}_{\mathbb{P}}\left(X{\mathbb{1}}_y\left(Y\right)\right)\right)+\sum _{y\in \mathrm{Ran}\left(Y\right)}{\left|f\left(y\right)\right|}^2\mathbb{P}(Y=y)\hfill \\ \hfill & ={\mathbb{E}}_{\mathbb{P}}{\left(\right|X|}^2)+\sum _{y\in \mathrm{Ran}\left(Y\right)}\mathbb{P}(Y=y)\left({\left|f\left(y\right)\right|}^2-2{\displaystyle \frac{\mathrm{Re}\left(\overline{f\left(y\right)}{\mathbb{E}}_{\mathbb{P}}\left(X{\mathbb{1}}_y\left(Y\right)\right)\right)}{\mathbb{P}(Y=y)}}\right)\hfill \\ \hfill & ={\mathbb{E}}_{\mathbb{P}}{\left(\right|X|}^2)-\sum _{y\in \mathrm{Ran}\left(Y\right)}{\displaystyle \frac{{\left|{\mathbb{E}}_{\mathbb{P}}\left(X{\mathbb{1}}_y\left(Y\right)\right)\right|}^2}{\mathbb{P}(Y=y)}}+\sum _{y\in \mathrm{Ran}\left(Y\right)}\mathbb{P}(Y=y){\left|f\left(y\right)-{\displaystyle \frac{{\mathbb{E}}_{\mathbb{P}}\left(X{\mathbb{1}}_y\left(Y\right)\right)}{\mathbb{P}(Y=y)}}\right|}^2.\hfill \end{array}$$

(27)

This quantity is minimal if and only if for all $y\in \mathrm{Ran}\left(Y\right)$,

$$f\left(y\right)={\displaystyle \frac{{\mathbb{E}}_{\mathbb{P}}\left(X{\mathbb{1}}_y\left(Y\right)\right)}{\mathbb{P}(Y=y)}}.$$

(28)

Moreover, we have that

$$\begin{array}{cc}\hfill {\mathbb{E}}_{\mathbb{P}}\left(X{\mathbb{1}}_y\left(Y\right)\right)& =\sum _{(x,y^{\prime })\in \mathrm{Ran}\left(X\right)\times \mathrm{Ran}\left(Y\right)}\mathbb{P}(X=x,Y=y^{\prime })x{\mathbb{1}}_y\left(y^{\prime }\right)\hfill \\ \hfill & =\sum _{x\in \mathrm{Ran}\left(X\right)}x\mathbb{P}(X=x,Y=y)\hfill \\ \hfill & =\mathbb{P}(Y=y){\mathbb{E}}_{\mathbb{P}}\left(X\right|Y=y).\hfill \end{array}$$

This concludes the proof. □

From Equation (22), we recall the law of iterated expectations,

$${\mathbb{E}}_{\mathbb{P}}\left({\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)\right)={\mathbb{E}}_{\mathbb{P}}\left(X\right)$$

(29)

and we note, for later reference, that Equation (27) can be rewritten in the form

$${\Delta }^{2}X={\Delta }^2{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)+{\mathbb{E}}_{\mathbb{P}}\left(\right|X-{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y){|}^2),$$

(30)

which is the law of total variance, where

$${\Delta }^{2}Z:={\mathbb{E}}_{\mathbb{P}}\left(\right|Z-{\mathbb{E}}_{\mathbb{P}}\left(Z\right){|}^2)$$

(31)

is the variance of the random variable $Z:\Omega \to \mathbb{C}$. In other words, X and its best estimator $\mathbb{E}\left(X\right|Y)$ have the same expected value and their variances differ by the minimal “cost”. Introducing the conditional variance

$${\Delta }_{\mathbb{P}}^2\left(X\right|Y)={\mathbb{E}}_{\mathbb{P}}\left(\right|X-{\mathbb{E}}_{\mathbb{P}}{\left(X\right|Y)|}^2\left|Y\right),$$

(32)

the law of total variance reads alternatively as

$${\Delta }^{2}X={\Delta }^2{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)+{\mathbb{E}}_{\mathbb{P}}\left({\Delta }_{\mathbb{P}}^2\left(X\right|Y)\right).$$

(33)

We note for later reference that

$${\Delta }_{\mathbb{P}}^2\left(X\right|Y)={\mathbb{E}}_{\mathbb{P}}{\left(\right|X|}^2)-|{\mathbb{E}}_{\mathbb{P}}{\left(X\right|Y)|}^2.$$

(34)

As we shall now prove, the conditional expectation is also characterized by two simple properties:

Theorem 3.

Let Y be a real vector-valued random variable on Ω and $\mathbb{P}\in D_Y$. Then, there exists a unique linear map ${\mathcal{E}}_{\mathrm{cond},Y}:\mathcal{F}(\Omega ;\mathbb{C})\to {\mathcal{F}}_{\mathbb{C},Y}\subset \mathcal{F}(\Omega ;\mathbb{C})$ that satisfies the following properties:

1. 

Pull-out formula. For any complex random variable X on Ω and any $f:\mathrm{Ran}\left(Y\right)\to \mathbb{C}$,

$${\mathcal{E}}_{\mathrm{cond},Y}\left(f\left(Y\right)X\right)=f\left(Y\right){\mathcal{E}}_{\mathrm{cond},Y}\left(X\right);$$

(35)

2. 

The law of iterated expectations. For any complex random variable X on Ω,

$${\mathbb{E}}_{\mathbb{P}}\left({\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)\right)={\mathbb{E}}_{\mathbb{P}}\left(X\right).$$

(36)

One has ${\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)={\mathbb{E}}_{\mathbb{P}}\left(X\right|Y).$

Proof. 

Its existence is clear, since it is easily verified that $X\mapsto {\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)$ satisfies the two desired properties. Uniqueness remains to be shown. Let ${\mathcal{E}}_{\mathrm{cond},Y}:X\mapsto {\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)\in {\mathcal{F}}_{\mathbb{C},Y}$ satisfy properties 1 and 2. Let X be a random variable. For any $f\left(Y\right)\in {\mathcal{F}}_{\mathbb{C},Y}$, we have

$$\begin{array}{cc}\hfill {\langle f\left(Y\right),X-{\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)\rangle }_{\mathbb{P}}& ={\mathbb{E}}_{\mathbb{P}}\left(\overline{f\left(Y\right)}(X-{\mathcal{E}}_{\mathrm{cond},Y}\left(X\right))\right)\hfill \\ \hfill & ={\mathbb{E}}_{\mathbb{P}}\left(\overline{f\left(Y\right)}X\right)-{\mathbb{E}}_{\mathbb{P}}\left({\mathcal{E}}_{\mathrm{cond},Y}\left(\overline{f\left(Y\right)}X\right)\right)\mathrm{by}\mathit{1}\hfill \\ \hfill & ={\mathbb{E}}_{\mathbb{P}}\left(\overline{f\left(Y\right)}X\right)-{\mathbb{E}}_{\mathbb{P}}\left(\overline{f\left(Y\right)}X\right)\mathrm{by}\mathit{2}\hfill \\ \hfill & =0.\hfill \end{array}$$

We therefore also have ${\langle f\left(Y\right),X-{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)\rangle }_{\mathbb{P}}=0$. It follows that

$$\begin{array}{cc}\hfill & {\langle {\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)-{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y),{\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)-{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)\rangle }_{\mathbb{P}}\hfill \\ \hfill =& {\langle {\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)-X+X-{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y),{\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)-{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)\rangle }_{\mathbb{P}}\hfill \\ \hfill =& -{\langle X-{\mathcal{E}}_{\mathrm{cond},Y}\left(X\right),{\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)-{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)\rangle }_{\mathbb{P}}+{\langle X-{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y),{\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)-{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)\rangle }_{\mathbb{P}}\hfill \\ \hfill =& 0\hfill \end{array}$$

since ${\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)-{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)\in {\mathcal{F}}_{\mathbb{C},Y}$. Therefore, ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)={\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)$ almost everywhere with respect to $\mathbb{P}$. Let $\omega \in \Omega$, then $Y\left(\omega \right)\in \mathrm{Ran}\left(Y\right)$ and thus, since $\mathbb{P}\in D_Y$, there exists ${\omega }^{\prime }\in \Omega$ such that $\mathbb{P}\left(\left\{{\omega }^{\prime }\right\}\right)>0$ and $Y\left(\omega \right)=Y\left({\omega }^{\prime }\right)$. The fact that ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)={\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)$ almost everywhere with respect to $\mathbb{P}$ then implies ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)\left({\omega }^{\prime }\right)={\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)\left({\omega }^{\prime }\right)$. Moreover, since they are both functions of Y we have ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)\left({\omega }^{\prime }\right)={\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)\left(\omega \right)$ and ${\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)\left({\omega }^{\prime }\right)={\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)\left(\omega \right)$. In the end, we have indeed ${\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)={\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)$ on $\Omega$. This concludes the proof. □

Let us point out that one can see quite directly that Equations (35) and (36) imply that $I-{\mathcal{E}}_{\mathrm{cond},Y}\left(I\right)$ is orthogonal to the algebra ${\mathcal{F}}_{\mathbb{C},Y}$, as well as belonging to ${\mathcal{F}}_{\mathbb{C},Y}$. This implies

$${\mathcal{E}}_{\mathrm{cond},Y}\left(I\right)=I,$$

(37)

where I is the constant function equal to 1 on $\Omega$. Using Equation (35) again, we obtain that

$${\mathcal{E}}_{\mathrm{cond},Y}\left(f\left(Y\right)\right)=f\left(Y\right).$$

(38)

These identities can also be retrieved from ${\mathcal{E}}_{\mathrm{cond},Y}\left(X\right)={\mathbb{E}}_{\mathbb{P}}\left(X\right|Y).$

We finish this section by showing that one can recover the joint probability distribution of the random variables X and Y using the expectation and conditional expectation. We will discuss a similar relation between joint quasiprobabilities and the conditional expectation in the quantum realm in Proposition 3.

Proposition 1.

For any $(x,y)\in \mathrm{Ran}\left(X\right)\times \mathrm{Ran}\left(\mathrm{Y}\right)$, we have

$$\mathbb{P}(X=x,Y=y)={\mathbb{E}}_{\mathbb{P}}\left[{\mathbb{1}}_y\left(Y\right){\mathbb{E}}_{\mathbb{P}}\left({\mathbb{1}}_x\left(X\right)\right|Y)\right].$$

(39)

Proof. 

We compute

$$\begin{array}{cc}\hfill {\mathbb{E}}_{\mathbb{P}}\left[{\mathbb{1}}_y\left(Y\right){\mathbb{E}}_{\mathbb{P}}\left({\mathbb{1}}_x\left(X\right)\right|Y)\right]& =\sum _{y^{\prime }\in \mathrm{Ran}\left(Y\right)}\mathbb{P}(Y=y^{\prime }){\mathbb{1}}_y\left(y^{\prime }\right){\mathbb{E}}_{\mathbb{P}}\left({\mathbb{1}}_x\left(X\right)\right|Y)\left(y^{\prime }\right)\hfill \\ \hfill & =\mathbb{P}(Y=y){\mathbb{E}}_{\mathbb{P}}\left({\mathbb{1}}_x\left(X\right)\right|Y)\left(y\right)\hfill \\ \hfill & =\mathbb{P}(Y=y)\sum _{x^{\prime }\in \mathrm{Ran}\left(X\right)}{\mathbb{1}}_x\left(x^{\prime }\right){\displaystyle \frac{\mathbb{P}(X=x^{\prime },y=y)}{\mathbb{P}(Y=y)}}\hfill \\ \hfill & =\mathbb{P}(X=x,Y=y).\hfill \end{array}$$

□

We finally point out that Y can also be taken complex vector-valued in what precedes. But, to stress the analogy with the quantum mechanical treatment, we have taken Y real vector-valued here.

3. Quantum Conditional Expectation and Variance via Minimization

In Section 3.1, we show how to adapt the classical definition of a conditional expectation based on Theorem 2 to the quantum context in order to define a quantum conditional expectation: see Theorem 4. This procedure defines the quantum conditional expectation as a “best estimator” of the operator $\widehat{X}$ by functions of the observable $\widehat{Y}$. Such definition based on a minimization procedure in analogy with the classical definition clearly depends on the quantity (the cost function) that is chosen to be minimized. We will, as in the classical context, use a quadratic minimizer and see that its choice is closely linked to the familiar “ordering problem” in quantization. Indeed, whereas for random variables X and Y, one has $f\left(Y\right)X=Xf\left(Y\right)$, this is no longer generally true when $\widehat{X}$ and $\widehat{Y}$ are operators on a Hilbert space. This observation naturally leads to two conditional expectations, a “left” one and a “right” one, among many other possibilities. In Section 3.2, we provide in Theorem 5 a characterization of the “left” and “right” quantum conditional expectations introduced that is analogous to its classical counterpart, Theorem 3 in that it relies on a “pull-out property”. In Section 3.3, we introduce the conditional variance of $\widehat{X}$ given $\widehat{Y}$, and we prove relations analogous to Equations (29) and (30), relating the mean and variance of the operator $\widehat{X}$ to those of its quantum conditional expectation and variance. We also discuss some of the marked differences between the classical and quantum notions, in particular with regard to pure states. Finally, in Section 3.4, we show that the self-adjoint part of the quantum conditional expectation can itself be interpreted as a best estimator of $\widehat{X}$ by self-adjoint functions of $\widehat{Y}$.

3.1. The Quantum Conditional Expectation: Definition

Let $\mathcal{H}$ be a Hilbert space of dimension $d<+\infty$. Let $\mathcal{L}\left(\mathcal{H}\right)$ be the space of operators on $\mathcal{H}$, and $D\left(\mathcal{H}\right)$ the set of density matrices (positive semi-definite operators of trace 1). We will first adapt the classical construction of the conditional expectation of Theorem 2 to the quantum context. For that purpose, we need to identify a suitable quantum analog of the sesquilinear form between classical observables in Equation (23). One may expect such analogs not to be uniquely determined since the sesquilinear form involves the product of two classical observables, and in general, quantizations of observables do not respect products: the quantization of the product is usually not the product of the quantizations due to ordering problems. We will see this phenomenon manifests itself here as well. One choice that presents itself naturally is to define, for any two $\widehat{X},{\widehat{X}}^{\prime }\in \mathcal{L}\left(\mathcal{H}\right)$, the sesquilinear form

$${\langle \widehat{X},{\widehat{X}}^{\prime }\rangle }_{\ell }:={\langle \widehat{X},{\widehat{X}}^{\prime }\rangle }_{\widehat{\rho },\ell }:=\mathrm{Tr}\left(\widehat{\rho }{\widehat{X}}^{†}{\widehat{X}}^{\prime }\right)=\mathrm{Tr}\left({\widehat{X}}^{\prime }\widehat{\rho }{\widehat{X}}^{†}\right),$$

(40)

where $\widehat{\rho }\in D\left(\mathcal{H}\right)$ is a fixed quantum state, which we will mostly drop from the notations (except when we will consider varying states in Section 6 below). Now, let $\widehat{Y}=({\widehat{Y}}_1,\dots {\widehat{Y}}_m)$ be a set of commuting observables on $\mathcal{H}$. We define the joint spectrum of $\widehat{Y}$ as

$$\sigma \left(\widehat{Y}\right)=\sigma \left({\widehat{Y}}_1\right)\times \dots \times \sigma \left({\widehat{Y}}_m\right),$$

where $\sigma \left({\widehat{Y}}_j\right)$ is the spectrum of ${\widehat{Y}}_j$. We will say $\widehat{Y}$ is complete when, for each $y=(y_1,\dots ,y_m)\in \sigma \left(\widehat{Y}\right)$, there exists a unique (up to an irrelevant global phase) normalized common eigenvector ${\phi }_y^{\widehat{Y}}$, for all ${\widehat{Y}}_j:$

$${\widehat{Y}}_j{\phi }_y^{\widehat{Y}}=y_j{\phi }_y^{\widehat{Y}}.$$

(41)

In other words, $\widehat{Y}$ is complete if and only if its joint spectrum is non-degenerate. Such a family is called a complete set of commuting observables (CSCO). While demanding that $\widehat{Y}$ is a family of commuting observables rather than an observable itself may seem superfluous, it is in fact of importance in many cases. A relevant example is found in spin systems, where $\mathcal{H}={\left({\mathbb{C}}^2\right)}^{\otimes m}$, and where the Pauli operators are given by ${\widehat{Y}}_j^{\widehat{\sigma }}=I\otimes \dots \otimes I\otimes \widehat{\sigma }\otimes I\otimes \dots \otimes I$. Here, $\widehat{\sigma }$ is a $2\times 2$ Pauli matrix, acting on the j-th sector.

In what follows, we shall write ${\widehat{\Pi }}_y^{\widehat{Y}}$ for the orthogonal projector onto ${\phi }_y^{\widehat{Y}}$. We consider the algebra of all functions of $\widehat{Y}$:

$${\mathcal{F}}_{\mathbb{C},\widehat{Y}}=\left\{f\left(\widehat{Y}\right)\right|f:\sigma \left(\widehat{Y}\right)\to \mathbb{C}\}.$$

(42)

Note that ${\mathcal{F}}_{\mathbb{C},\widehat{Y}}$ is a commutative *-algebra. We use ${\left({\mathcal{F}}_{\mathbb{C},\widehat{Y}}\right)}^{\prime }$ to denote the commutant of ${\mathcal{F}}_{\mathbb{C},\widehat{Y}}$. ${\mathcal{F}}_{\mathbb{C},\widehat{Y}}$ is maximal in the following sense:

Lemma 1.

${\left({\mathcal{F}}_{\mathbb{C},\widehat{Y}}\right)}^{\prime }={\mathcal{F}}_{\mathbb{C},\widehat{Y}}.$

Proof. 

It is clear that ${\mathcal{F}}_{\mathbb{C},\widehat{Y}}\subseteq {\left({\mathcal{F}}_{\mathbb{C},\widehat{Y}}\right)}^{\prime }$. Suppose therefore $\widehat{C}\in {\left({\mathcal{F}}_{\mathbb{C},\widehat{Y}}\right)}^{\prime }$. Then, $[\widehat{C},{\widehat{\Pi }}_y^{\widehat{Y}}]=0$, for all $y\in \sigma \left(\widehat{Y}\right)$. Hence, for all $y\in \sigma \left(\widehat{Y}\right)$,

$$\widehat{C}{\phi }_y^{\widehat{Y}}={\phi }_y^{\widehat{Y}}\mathrm{Tr}\left(\widehat{C}{\widehat{\Pi }}_y^{\widehat{Y}}\right).$$

(43)

Defining the function $g\left(y\right)=\mathrm{Tr}\left(\widehat{C}{\widehat{\Pi }}_y^{\widehat{Y}}\right)$, one concludes that $\widehat{C}=g\left(\widehat{Y}\right)$ and hence that $\widehat{C}\in {\mathcal{F}}_{\mathbb{C},\widehat{Y}}$. □

For a density matrix $\widehat{\rho }$, analogous to Theorem 2, one can then define (following [32,33,34,35] and generalized in [48,49]) a quantum conditional expectation of $\widehat{X}$, knowing $\widehat{Y}$, that we shall denote using ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ as the minimizer of the following quantity:

$$f\mapsto \mathrm{Tr}(\widehat{\rho }|\widehat{X}-f\left(\widehat{Y}\right){|}^2)={\langle \widehat{X}-f\left(\widehat{Y}\right),\widehat{X}-f\left(\widehat{Y}\right)\rangle }_{\ell },$$

(44)

over all $f:\sigma \left(\widehat{Y}\right)\to \mathbb{C}$. Here, for $Z\in \mathcal{L}\left(\mathcal{H}\right)$, we write $\left|Z\right|=\sqrt{Z^{†}Z}$. As in the classical case, we think of the right-hand side of this equation as a mean squared error cost. We shall discuss the existence and uniqueness of this minimizer below. We shall refer to ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ as the left quantum conditional expectation of $\widehat{X}$ knowing $\widehat{Y}$ in the state $\widehat{\rho }$, for reasons that will become clear shortly. There is a second, equally natural choice for the sesquilinear product, analogous to the inner product in Equation (23), namely

$${\langle \widehat{X},{\widehat{X}}^{\prime }\rangle }_{\mathrm{r}}=\mathrm{Tr}\left(\widehat{\rho }{\widehat{X}}^{\prime }{\widehat{X}}^{†}\right)=\mathrm{Tr}\left({\widehat{X}}^{†}\widehat{\rho }{\widehat{X}}^{\prime }\right).$$

(45)

Note that, as anticipated, the difference between Equations (40) and (45) lies indeed in the change of order of appearance of ${\widehat{X}}^{†}$ and ${\widehat{X}}^{\prime }$. We then define the right quantum conditional expectation of $\widehat{X}$, knowing $\widehat{Y}$, that we shall denote using ${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$, as the minimizer over complex-valued functions of

$$f\mapsto {\langle \widehat{X}-f\left(\widehat{Y}\right),\widehat{X}-f\left(\widehat{Y}\right)\rangle }_{\mathrm{r}}.$$

(46)

The existence and uniqueness of these conditional expectations is guaranteed by the following theorem, under a suitable hypothesis on $\widehat{\rho }$. We define

$$D_{\widehat{Y}}=\{\widehat{\rho }\in D\left(\mathcal{H}\right):\forall y\in \sigma \left(\widehat{Y}\right),\mathrm{Tr}\left(\widehat{\rho }{\widehat{\Pi }}_y^{\widehat{Y}}\right)\ne 0\}.$$

(47)

Theorem 4.

Assume that $\widehat{\rho }\in D_{\widehat{Y}}$. Then, the functions given in Equations (44) and (46) admit unique minimizers given by

$$f_0^{\ell }\left(y\right)={\displaystyle \frac{\langle {\phi }_y^{\widehat{Y}},\widehat{X}\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }},f_0^{\mathrm{r}}\left(y\right)={\displaystyle \frac{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }\widehat{X}{\phi }_y^{\widehat{Y}}\rangle }{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }},$$

(48)

respectively, for all $y\in \sigma \left(\widehat{Y}\right)$.

Proof. 

We only prove the “left” case, the “right” case being similar. We compute

$$\begin{array}{cc}\hfill {\langle \widehat{X}-f\left(\widehat{Y}\right),\widehat{X}-f\left(\widehat{Y}\right)\rangle }_{\ell }& =\mathrm{Tr}(\widehat{\rho }|\widehat{X}{|}^2)-\mathrm{Tr}\left(\widehat{\rho }{\widehat{X}}^{†}f\left(\widehat{Y}\right)\right)-\mathrm{Tr}\left(\widehat{\rho }f{\left(\widehat{Y}\right)}^{†}\widehat{X}\right)+\mathrm{Tr}(\widehat{\rho }|f\left(\widehat{Y}\right){|}^2)\hfill \\ \hfill & =\mathrm{Tr}(\widehat{\rho }|\widehat{X}{|}^2)+\sum _{y\in \sigma \left(\widehat{Y}\right)}\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle \left({\left|f\left(y\right)\right|}^2-2Re\left(\overline{f\left(y\right)}{\displaystyle \frac{\langle {\phi }_y^{\widehat{Y}},\widehat{X}\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }}\right)\right)\hfill \\ \hfill & =\mathrm{Tr}(\widehat{\rho }|\widehat{X}{|}^2)-\sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{|\langle {\phi }_y^{\widehat{Y}},\widehat{X}\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle {|}^2}{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }}\hfill \\ \hfill & +\sum _{y\in \sigma \left(\widehat{Y}\right)}\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle {\left|f\left(y\right)-{\displaystyle \frac{\langle {\phi }_y^{\widehat{Y}},\widehat{X}\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }}\right|}^2.\hfill \end{array}$$

(49)

Hence, this quantity is minimal if and only if

$$f\left(y\right)={\displaystyle \frac{\langle {\phi }_y^{\widehat{Y}},\widehat{X}\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }}$$

(50)

for all $y\in \sigma \left(\widehat{Y}\right)$. □

Definition 2.

We define the left quantum conditional expectation of the operator $\widehat{X}$ knowing $\widehat{Y}$ in the state $\widehat{\rho }\in D_{\widehat{Y}}$ as

$${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})=f_0^{\ell }\left(\widehat{Y}\right)=\sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\langle {\phi }_y^{\widehat{Y}},\widehat{X}\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }}{\widehat{\Pi }}_y^{\widehat{Y}},$$

(51)

where ${\widehat{\Pi }}_y$ is the orthogonal projector to the eigenvector ${\phi }_y^{\widehat{Y}}$ for $y\in \sigma \left(\widehat{Y}\right)$. Similarly, we define the right quantum conditional expectation of $\widehat{X}$ knowing $\widehat{Y}$ in the state $\widehat{\rho }$ as

$${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})=f_0^{\mathrm{r}}\left(\widehat{Y}\right)=\sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }\widehat{X}{\phi }_y^{\widehat{Y}}\rangle }{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }}{\widehat{\Pi }}_y^{\widehat{Y}}.$$

(52)

We will write ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y}=y)=\mathrm{Tr}\left({\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y}){\widehat{\Pi }}_y^{\widehat{Y}}\right)$.

It is readily verified that for any $f:\sigma \left(\widehat{Y}\right)\to \mathbb{C}$,

$${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(f\left(\widehat{Y}\right)\right|\widehat{Y})=f\left(\widehat{Y}\right).$$

(53)

Remark 1. 

$\left(i\right)$ If the state $\widehat{\rho }$ is not an element of $D_{\widehat{Y}}$, i.e., if there exists $y\in \sigma \left(\widehat{Y}\right)$ such that $\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle =0$, then the functional (44) still admits minimizers, but they are no longer unique. Since $\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle =0$ implies that ${\widehat{\rho }}^{1/2}{\phi }_y^{\widehat{Y}}=0$, and therefore $\widehat{\rho }{\phi }_y^{\widehat{Y}}=0$, the sums in (49) only extend over y’s with $\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle \ne 0$ and the minimizers are in fact given by

$$f_0\left(\widehat{Y}\right)=\sum _{\begin{array}{c}y\in \sigma \left(\widehat{Y}\right)\\ \langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle \ne 0\end{array}}{\displaystyle \frac{\langle {\phi }_y^{\widehat{Y}},\widehat{X}\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }}{\widehat{\Pi }}_y^{\widehat{Y}}+\sum _{\begin{array}{c}y\in \sigma \left(\widehat{Y}\right)\\ \langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle =0\end{array}}{g}_y{\widehat{\Pi }}_y^{\widehat{Y}},$$

(54)

where ${\left(g_y\right)}_{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle =0}\subset \mathbb{C}$ can be any complex numbers [33]. In the usual interpretation of quantum mechanics, the quantity $\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle$ represents the probability of the event “$\widehat{Y}$ takes the value y”. It follows in accordance with classical probability theory that the minimizers of the functional (44) are unique up to functions of $\widehat{Y}$ which are “$\widehat{\rho }$-almost everywhere 0”, such functions corresponding to the term ${\sum }_{\begin{array}{c}y\in \sigma \left(\widehat{Y}\right)\\ \langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle =0\end{array}}{g}_y{\widehat{\Pi }}_y^{\widehat{Y}}$. In this article, we only consider non-degenerate states, except for in Appendix E, where we discuss the connection with quantum Fisher information. Note that here by non-degenerate we mean states $\widehat{\rho }\in D_{\widehat{Y}}$, which does not necessarily imply that the state $\widehat{\rho }$ is positive definite. For example the set $D_{\widehat{Y}}$ contains pure states. This non-degeneracy condition allows nice characterizations of the conditional expectations. Moreover, it is not too restrictive, as shown by Lemma 2.

$\left(ii\right)$ The expressions ${\langle ·,·\rangle }_{\ell /\mathrm{r}}$ are not inner products because ${\langle \widehat{X},\widehat{X}\rangle }_{\ell /\mathrm{r}}$ may vanish even if $\widehat{X}\ne 0$. This will happen if $\widehat{\rho }$ has a non-vanishing kernel, foremost when the state $\widehat{\rho }$ is pure. One may however think of them as degenerate inner products or as quadratic cost functions. Note that this cost depends on the state $\widehat{\rho }$. From this point of view, one may think of the operator ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ as the operator function of the observables $\widehat{Y}$ that approximates $\widehat{X}$ at minimal cost. It is also clear that such minimizers necessarily depend on the choice of cost function, that in turn necessarily depends on the physics of the problem considered. In the next subsection, we show the “left” and “right” choices made here are natural in the sense that they can be uniquely characterized by a pull-out property analogous to the classical conditional expectation.

$\left(iii\right)$ In view of what precedes, the quantum conditional expectations ${\mathbb{E}}^{\ell ,\mathrm{r}}$ are orthogonal projections with respect to the two “inner products” induced by the state $\widehat{\rho }$ of the operator $\widehat{X}$ onto the algebra of functions of the operators $\widehat{Y}$.

$\left(iv\right)$ The conditional expectations ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ are not self-adjoint in general, even if we take $\widehat{X}$ to be self-adjoint. This is markedly different from the classical situation, where the conditional expectation of a real random variable X, given a real random variable Y, is always real. We will extensively elaborate on this point below. Note that

$${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})={\mathbb{E}}_{\widehat{\rho }}^{\ell }{\left({\widehat{X}}^{†}\right|\widehat{Y})}^{†}.$$

(55)

$\left(v\right)$ When $\widehat{X}$ is self-adjoint, the coefficients

$${\left({\displaystyle \frac{\langle {\phi }_y,\widehat{X}\widehat{\rho }{\phi }_y\rangle }{\langle {\phi }_y,\widehat{\rho }{\phi }_y\rangle }}\right)}_{y\in \sigma \left(\widehat{Y}\right)}$$

(56)

appearing in ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ are weak values with as the initial state $\widehat{\rho }$, a weak measurement of the observable $\widehat{X}$ and a post-selection ${\widehat{\Pi }}_y^{\widehat{Y}}$ for each $y\in \sigma \left(\widehat{Y}\right)$. This observation links the conditional expectation of $\widehat{X}$ to the physics of weak values, which has been studied extensively, see [31,45,50,51,52,53]. For the readers’ convenience, we provide, in Appendix A, a concise, complete, self-contained and mathematically clean introduction to weak value measurement, which is the procedure by which the above weak values can be measured. This procedure relies in particular on a postselection, which has historically provided the intuitive link between weak value measurement and conditioning. We refer to the appendix for details. The definition of quantum conditional expectation through minimization that we propose here allows to sharpen this link as we shall further discuss is Section 3.3 and Section 6.

3.2. Characterization of the Left/Right Quantum Conditional Expectations

Let us point out that there are many other quantum analogs of the inner product in Equation (3), among which one could for example consider

$${\langle \widehat{X},{\widehat{X}}^{\prime }\rangle }_{\alpha }=\alpha {\langle \widehat{X},{\widehat{X}}^{\prime }\rangle }_{\ell }+(1-\alpha ){\langle \widehat{X},\widehat{X^{\prime }}\rangle }_{\mathrm{r}}.$$

(57)

for $0\le \alpha \le 1$, interpolating between ${\langle \widehat{X},\widehat{X^{\prime }}\rangle }_{\ell }$ and ${\langle \widehat{X},\widehat{X^{\prime }}\rangle }_{\mathrm{r}}$. Minimizing ${\langle \widehat{X}-f\left(\widehat{Y}\right),\widehat{X}-f\left(\widehat{Y}\right)\rangle }_{\alpha }$ leads to a candidate notion of conditional expectation, whose explicit expression can be easily computed to be

$${\mathbb{E}}_{\widehat{\rho }}^{\alpha }\left(\widehat{X}\right|\widehat{Y})=\sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\langle {\phi }_y^{\widehat{Y}},(\alpha \widehat{X}\widehat{\rho }+(1-\alpha )\widehat{\rho }\widehat{X}){\phi }_y^{\widehat{Y}}\rangle }{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }}{\widehat{\Pi }}_y^{\widehat{Y}}.$$

(58)

Other, more elaborate choices have been considered in [49].

The following theorem shows that to obtain a conditional expectation that satisfies the properties of the classical conditional expectation given in Theorem 3, $\alpha$ has to be either 0 or 1.

Theorem 5.

Let $\widehat{\rho }\in D_{\widehat{Y}}$. For any $\#\in \{\ell ,\mathrm{r}\}$, there exist a unique map ${\mathcal{E}}_{\mathrm{cond},Y}^{\#}:\mathcal{L}\left(\mathcal{H}\right)\to {\mathcal{F}}_{\mathbb{C},\widehat{Y}}$ satisfying the following properties:

1. 

Pull-out formula. For any $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$ and any $f:\sigma \left(\widehat{Y}\right)\to \mathbb{C}$,

$$\begin{array}{cc}\hfill & \mathit{for}\#=\ell ,{\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(f\left(\widehat{Y}\right)\widehat{X}\right)=f\left(\widehat{Y}\right){\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)={\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)f\left(\widehat{Y}\right),\hfill \end{array}$$

(59)

$$\begin{array}{cc}\hfill & \mathit{for}\#=\mathrm{r},{\mathcal{E}}_{\mathrm{cond},Y}^{\mathrm{r}}\left(\widehat{X}f\left(\widehat{Y}\right)\right)=f\left(\widehat{Y}\right){\mathcal{E}}_{\mathrm{cond},Y}^{\mathrm{r}}\left(\widehat{X}\right)={\mathcal{E}}_{\mathrm{cond},Y}^{\mathrm{r}}\left(\widehat{X}\right)f\left(\widehat{Y}\right).\hfill \end{array}$$

(60)

2. 

Quantum law of iterated expectations. For any $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$,

$${\mathbb{E}}_{\widehat{\rho }}\left({\mathcal{E}}_{\mathrm{cond},Y}^{\#}\left(\widehat{X}\right)\right)={\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right).$$

(61)

This map is given by ${\mathcal{E}}_{\mathrm{cond},Y}^{\#}\left(\widehat{X}\right)={\mathbb{E}}_{\widehat{\rho }}^{\#}\left(\widehat{X}\right|\widehat{Y})$.

This theorem singles out two natural definitions of quantum conditional expectation through the conditions Equations (59) and (61) or Equations (60) and (61).

As in the classical case, the uniqueness in Theorem 5 proves that Equation (53) can be deduced from Equations (59)–(61). Again, one could have shown this fact directly.

Proof. 

Again, we write the proof for $\#=\ell$, the other case being similar. We begin by showing that the left quantum conditional expectation satisfies these two properties. For point 1, we compute

$$\begin{array}{cc}\hfill {\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(f\left(\widehat{Y}\right)\widehat{X}\right|\widehat{Y})& =\sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\langle {\phi }_y^{\widehat{Y}},f\left(\widehat{Y}\right)\widehat{X}\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }}{\widehat{\Pi }}_y\hfill \\ \hfill & =\sum _{y\in \sigma \left(\widehat{Y}\right)}f\left(y\right){\displaystyle \frac{\langle {\phi }_y^{\widehat{Y}},\widehat{X}\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }}{\widehat{\Pi }}_y\hfill \\ \hfill & =f\left(\widehat{Y}\right){\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})f\left(\widehat{Y}\right).\hfill \end{array}$$

For point 2, we compute

$$\begin{array}{cc}\hfill {\mathbb{E}}_{\widehat{\rho }}\left({\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\right)& =\mathrm{Tr}\left(\widehat{\rho }\sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\langle {\phi }_y^{\widehat{Y}},\widehat{X}\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }}{\widehat{\Pi }}_y^{\widehat{Y}}\right)\hfill \\ \hfill & =\sum _{y\in \sigma \left(\widehat{Y}\right)}\langle {\phi }_y^{\widehat{Y}},\widehat{X}\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle \hfill \\ \hfill & =\mathrm{Tr}\left(\widehat{X}\widehat{\rho }\right)\hfill \\ \hfill & ={\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right).\hfill \end{array}$$

Now, let ${\mathcal{E}}_{\mathrm{cond},Y}^{\ell }:\mathcal{L}\left(\mathcal{H}\right)\to {\mathcal{F}}_{\mathbb{C},\widehat{Y}}$ be a map that satisfies Equations (59) and (61). We will show that ${\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ for any $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$. Let $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$. For any $f\left(\widehat{Y}\right)\in {\mathcal{F}}_{\mathbb{C},\widehat{Y}}$, we have

$$\begin{array}{cc}\hfill {\langle f\left(\widehat{Y}\right),\widehat{X}-{\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)\rangle }_{\ell }& =\mathrm{Tr}\left(\widehat{\rho }f{\left(\widehat{Y}\right)}^{†}(\widehat{X}-{\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right))\right)\hfill \\ \hfill & =\mathrm{Tr}\left(\widehat{\rho }f{\left(\widehat{Y}\right)}^{†}\widehat{X}\right)-\mathrm{Tr}\left(\widehat{\rho }{\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(f{\left(\widehat{Y}\right)}^{†}\widehat{X}\right)\right)\mathrm{by}\mathrm{Equation}\left(59\right)\hfill \\ \hfill & =\mathrm{Tr}\left(\widehat{\rho }f{\left(\widehat{Y}\right)}^{†}\widehat{X}\right)-\mathrm{Tr}\left(\widehat{\rho }f{\left(\widehat{Y}\right)}^{†}\widehat{X}\right)\mathrm{by}\mathrm{Equation}\left(61\right)\hfill \\ \hfill & =0,\hfill \end{array}$$

and of course ${\langle f\left(\widehat{Y}\right),\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell }=0$ as well. It follows

$$\begin{array}{cc}\hfill & {\langle {\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}),{\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell }\hfill \\ \hfill =& {\langle {\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)-\widehat{X}+\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}),{\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell }\hfill \\ \hfill =& -{\langle \widehat{X}-{\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right),{\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell }+{\langle \widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}),{\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell }\hfill \\ \hfill =& 0\hfill \end{array}$$

since ${\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\in {\mathcal{F}}_{\mathbb{C},\widehat{Y}}$. Therefore, $\mathrm{Tr}\left(\widehat{\rho }\right|{\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}){|}^2)=0$. Since ${\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\in {\mathcal{F}}_{\mathbb{C},\widehat{Y}}$, there exists $C_{\widehat{\rho }}:\sigma \left(\widehat{Y}\right)\to \mathbb{C}$ such that ${\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})=C_{\widehat{\rho }}\left(\widehat{Y}\right)$. One gets

$$\begin{array}{c}\hfill 0=\mathrm{Tr}(\widehat{\rho }|C_{\widehat{\rho }}\left(\widehat{Y}\right){|}^2)=\sum _{y\in \sigma \left(\widehat{Y}\right)}{\left|C_{\widehat{\rho }}\left(y\right)\right|}^2\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle ,\end{array}$$

which implies, since $\widehat{\rho }\in D_{\widehat{Y}}$, that $C_{\widehat{\rho }}=0$, i.e.,${\mathcal{E}}_{\mathrm{cond},Y}^{\ell }\left(\widehat{X}\right)={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$. □

It is legitimate to ask if one can define a conditional expectation which satisfies both the left- and right-pullout properties, see Equations (59) and (60). This has been performed in the context of von Neumann algebras in [54]. In Appendix B, we briefly explain the link between that approach and ours.

The conditional expectation of $\widehat{X}$ knowing $\widehat{Y}$ is defined only for states $\widehat{\rho }\in D_{\widehat{Y}}$, to avoid dividing by 0. As we now prove in the following Lemma, this condition on the state $\widehat{\rho }$ is not too restrictive, as the set $D_{\widehat{Y}}$ is, in a topological sense, a large set (it is dense in the set of all density matrices).

Lemma 2.

Let $\widehat{Y}$ be a self-adjoint operator on $\mathcal{H}$. Then $D_{\widehat{Y}}$, defined in Equation (47) is dense in $D\left(\mathcal{H}\right)$ (for any norm topology).

Proof. 

Since we are working in finite dimensions, all norms are equivalent. We chose to work with the trace-class norm. Let ${\widehat{\rho }}_0\in D\left(\mathcal{H}\right)$. If ${\widehat{\rho }}_0\in D_{\widehat{Y}}$ there is nothing to do. Otherwise, let

$$I_0=\{y\in \sigma \left(\widehat{Y}\right):\mathrm{Tr}\left({\widehat{\rho }}_0{\widehat{\Pi }}_y^{\widehat{Y}}\right)=0\}.$$

We use $\left|I\right|$ to denote the cardinal of the finite set I. For $\epsilon >0$, let

$${\widehat{\rho }}_{\epsilon }={\displaystyle \frac{1}{1+\epsilon |I_0|}}\left({\widehat{\rho }}_0+\epsilon \sum _{y\in I_0}{\widehat{\Pi }}_y^{\widehat{Y}}\right)$$

Then, it is easily verified that ${\widehat{\rho }}_{\epsilon }\in D_{\widehat{Y}}$ for any $\epsilon >0$ and moreover

$$\begin{array}{cc}\hfill \mathrm{Tr}\left(\right|{\widehat{\rho }}_{\epsilon }-{\widehat{\rho }}_0\left|\right)& =\mathrm{Tr}\left(\left|{\widehat{\rho }}_0\left({\displaystyle \frac{1}{1+\epsilon |I_0|}}-1\right)+{\displaystyle \frac{\epsilon }{1+\epsilon |I_0|}}\sum _{y\in I_0}{\widehat{\Pi }}_y^{\widehat{Y}}\right|\right)\hfill \\ \hfill & \le \left|1-{\displaystyle \frac{1}{1+\epsilon |I_0|}}\right|+{\displaystyle \frac{\epsilon |I_0|}{1+\epsilon |I_0|}}\hfill \\ \hfill & \underset{\epsilon \to 0}{\to }0.\hfill \end{array}$$

This proves the lemma. □

Remark 2. 

$\left(i\right)$ If $\widehat{Y}$ is an incomplete set of commuting observables, the associated projectors ${\left({\widehat{\Pi }}_y^{\widehat{Y}}\right)}_{y\in \sigma \left(\widehat{Y}\right)}$ are not all of rank 1. However, the results in this section still adapt to this case. For $\widehat{\rho }\in D_{\widehat{Y}}$ and $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$, it is easily shown that the minimizers of Equation (48) become

$$\forall y\in \sigma \left(\widehat{Y}\right),f_0^{\ell }\left(y\right)={\displaystyle \frac{\mathrm{Tr}\left(\widehat{X}\widehat{\rho }{\widehat{\Pi }}_y^{\widehat{Y}}\right)}{\mathrm{Tr}\left(\widehat{\rho }{\widehat{\Pi }}_y^{\widehat{Y}}\right)}},f_0^{\mathrm{r}}\left(y\right)={\displaystyle \frac{\mathrm{Tr}\left(\widehat{\rho }\widehat{X}{\widehat{\Pi }}_y^{\widehat{Y}}\right)}{\mathrm{Tr}\left(\rho {\widehat{\Pi }}_y^{\widehat{Y}}\right)}},$$

(62)

and that one can define the left and right conditional expectations by

$${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})=\sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\mathrm{Tr}\left(\widehat{X}\widehat{\rho }{\widehat{\Pi }}_y\right)}{\mathrm{Tr}\left(\widehat{\rho }{\widehat{\Pi }}_y\right)}}{\widehat{\Pi }}_y,{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})=\sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\mathrm{Tr}\left(\widehat{\rho }\widehat{X}{\widehat{\Pi }}_y\right)}{\mathrm{Tr}\left(\widehat{\rho }{\widehat{\Pi }}_y\right)}}{\widehat{\Pi }}_y.$$

(63)

The characterization given in Theorem 5 remains true in this setting, as do the equivalences of Proposition A6 but not Equation (A44). In the following sections, we require the completeness of $\widehat{Y}$.

$\left(ii\right)$ We finally note that in the case of commuting operators $\widehat{X}$ and $\widehat{Y}$, the quantum conditional expectation reduces to the classical conditional expectation (1), when considering the two operators as classical random variables with joint probability $\mathbb{P}(\widehat{X}=x,\widehat{Y}=y)=\mathrm{Tr}\left({\widehat{\Pi }}_x{\widehat{\Pi }}_y\widehat{\rho }\right)$, the Born measure of the commuting system $(\widehat{X},\widehat{Y})$.

3.3. The Quantum Laws of Iterated Expectations and of Total Variance

The first two moments of the quantum conditional expectations ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ have properties that are reminiscent of those of the classical conditional expectation: the law of iterated expectations and the law of total variance. There are however some important differences between the classical and quantum versions that we shall point out. The results are summed up in Proposition 2 and in the discussion following it.

We introduce, for any operator $\widehat{C}\in \mathcal{L}\left(\mathcal{H}\right)$, the left and right variance of $\widehat{C}$ in the state $\widehat{\rho }$ by

$${\Delta }_{\widehat{\rho }}^{\ell ,2}\left(\widehat{C}\right)=\mathrm{Tr}\left(\widehat{\rho }{(\widehat{C}-{\mathbb{E}}_{\widehat{\rho }}\left(\widehat{C}\right))}^{†}(\widehat{C}-{\mathbb{E}}_{\widehat{\rho }}\left(\widehat{C}\right))\right),{\Delta }_{\widehat{\rho }}^{\mathrm{r},2}\left(\widehat{C}\right)=\mathrm{Tr}\left(\widehat{\rho }(\widehat{C}-{\mathbb{E}}_{\widehat{\rho }}\left(\widehat{C}\right)){(\widehat{C}-{\mathbb{E}}_{\widehat{\rho }}\left(\widehat{C}\right))}^{†}\right).$$

(64)

When ${\widehat{C}}^{†}=\widehat{C}$ or ${\widehat{C}}^{†}=-\widehat{C}$, we will write

$${\Delta }_{\widehat{\rho }}^2\widehat{C}={\Delta }_{\widehat{\rho }}^{\ell ,2}\left(\widehat{C}\right)={\Delta }_{\widehat{\rho }}^{\mathrm{r},2}\left(\widehat{C}\right).$$

(65)

We further introduce the left and right conditional variances of $\widehat{C}$, analogously to the classical definition:

$${\Delta }_{\widehat{\rho }}^{\ell ,2}\left(\widehat{C}\right|\widehat{Y})={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\right|\widehat{C}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{C}\right|\widehat{Y}){|}^2\left|\widehat{Y}\right),{\Delta }_{\widehat{\rho }}^{\mathrm{r},2}\left(\widehat{C}\right|\widehat{Y})={\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left((\widehat{C}-{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{C}\right|\widehat{Y})){(\widehat{C}-{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{C}\right|\widehat{Y}))}^{†}|\widehat{Y}\right).$$

(66)

Recall that, for any operator Z, one has $|\widehat{Z}{|}^2={\widehat{Z}}^{†}\widehat{Z}$. In addition, a simple computation allows establishing that the expected value of the conditional variance equals the error

$${\mathbb{E}}_{\widehat{\rho }}\left({\Delta }_{\widehat{\rho }}^{\ell /r,2}\left(\widehat{C}\right|\widehat{Y})\right)={\langle \widehat{C}-{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{C}\right|\widehat{Y}),\widehat{C}-{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{C}\right|\widehat{Y})\rangle }_{\ell /\mathrm{r}}.$$

(67)

The notations are cumbersome, but the treatment is similar for the “left” and “right” cases. Recall that, even if $\widehat{X}$ is self-adjoint, its conditional expectation may not be. The left/right conditional variances will generally then differ. The following proposition sums up the basic properties of the conditional expectation and the conditional variance.

Proposition 2.

If $\widehat{\rho }\in D_{\widehat{Y}}$, then for any $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$ the quantum law of iterated expectations holds:

$${\mathbb{E}}_{\widehat{\rho }}\left({\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})\right)={\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right).$$

(68)

The quantum law of total variance reads as

$$\begin{array}{ccc}\hfill {\Delta }_{\widehat{\rho }}^{\ell /\mathrm{r},2}\widehat{X}& =& {\Delta }_{\widehat{\rho }}^{\ell /\mathrm{r},2}{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})+{\langle \widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y}),\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell /\mathrm{r}}\hfill \end{array}$$

(69)

$$\begin{array}{ccc}& =& {\Delta }_{\widehat{\rho }}^{\ell /\mathrm{r},2}{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})+{\mathbb{E}}_{\widehat{\rho }}\left({\Delta }_{\widehat{\rho }}^{\ell /r,2}\left(\widehat{X}\right|\widehat{Y})\right).\hfill \end{array}$$

(70)

If $\widehat{\rho }\in D_{\widehat{Y}}$ is pure,

$${\Delta }_{\widehat{\rho }}^{\ell ,2}\left(\widehat{X}\right|\widehat{Y})=0$$

(71)

and thus,

$${\Delta }_{\widehat{\rho }}^{\ell /\mathrm{r},2}\widehat{X}={\Delta }_{\widehat{\rho }}^{\ell /\mathrm{r},2}{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y}).$$

(72)

The quantum law of iterated expectations, Equation (68), says that the best estimator of $\widehat{X}$ by a function of $\widehat{Y}$ has the same expected value as $\widehat{X}$ itself; it is, in this sense, unbiased, as the classical conditional expectation. Equation (69) is in turn a quantum analog of the classical law of total variance Equation (30). It provides a relation between the variance of $\widehat{X}$, the variance of the conditional expectation ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$, and the expected value of the conditional variance ${\Delta }^{\ell /\mathrm{r},2}\left(X\right|Y)$, in complete analogy with the classical situation. The error term in this relation (the second term on the right-hand side of Equation (69)) can, as in the classical case, be expressed as the expected value of the conditional variance: this is the content of Equation (67).

Proof. 

We give the proof for the left conditional expectation ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$. Note that Equation (68) is exactly Equation (61), and it has already been shown in the proof of Theorem 5.

To prove Equation (69), we note that, from Equation (49), we obtain

$$\begin{array}{ccc}\hfill \underset{f\in {\mathcal{F}}_{\mathbb{C},\widehat{Y}}}{min}{\langle (\widehat{X}-f\left(\widehat{Y}\right)),(\widehat{X}-f\left(\widehat{Y}\right))\rangle }_{\ell }& =& \mathrm{Tr}\left({\widehat{X}}^{†}\widehat{X}\widehat{\rho }\right)-\sum _y{\displaystyle \frac{|\langle {\phi }_y^{\widehat{Y}},\widehat{X}\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle {|}^2}{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }}\hfill \\ & =& {\langle \widehat{X},\widehat{X}\rangle }_{\ell }-{\langle {\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}),{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell }.\hfill \end{array}$$

(73)

Thus,

$${\langle \widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}),\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell }={\langle \widehat{X},\widehat{X}\rangle }_{\ell }-{\langle {\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}),{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell }.$$

(74)

We have that

$$\begin{array}{ccc}\hfill {\Delta }_{\widehat{\rho }}^{\ell ,2}\left(\widehat{X}\right)& =& \mathrm{Tr}\left(\widehat{\rho }{\widehat{X}}^{†}\widehat{X}\right)-\overline{{\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right)}\mathrm{Tr}\left(\widehat{\rho }\widehat{X}\right)-{\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right)\mathrm{Tr}\left(\widehat{\rho }{\widehat{X}}^{†}\right)+{\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right)\overline{{\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right)}\hfill \\ & =& {\langle \widehat{X},\widehat{X}\rangle }_{\ell }-{\left|{\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right)\right|}^2\hfill \\ & =& {\langle \widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}),\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell }+{\langle {\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}),{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell }-{\left|{\mathbb{E}}_{\widehat{\rho }}\left({\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\right)\right|}^2\hfill \\ & =& {\langle \widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}),\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell }+{\Delta }_{\widehat{\rho }}^{\ell ,2}\left({\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\right)\hfill \end{array}$$

(75)

where we used Equations (68) and (74) on the third line.

Equation (67) now follows:

$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\widehat{\rho }}\left({\Delta }^{\ell ,2}\left(\widehat{X}\right|\widehat{Y})\right)& =& {\mathbb{E}}_{\widehat{\rho }}\left({\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\right|\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}){|}^2\left|\widehat{Y}\right)\right)\hfill \\ & =& {\mathbb{E}}_{\widehat{\rho }}\left(|\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}){|}^2\right)\hfill \\ & =& \mathrm{Tr}\left(\rho |\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}){|}^2\right)={\langle \widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}),\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell }\hfill \end{array}$$

where we used the quantum law of iterated expectations.

We finally prove Equation (71). We have that

$$\begin{array}{ccc}\hfill {\Delta }_{\widehat{\rho }}^{\ell ,2}\left(\widehat{X}\right|\widehat{Y})& =& {\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\right|\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}){|}^2\left|\widehat{Y}\right)\hfill \\ & =& {\mathbb{E}}_{\widehat{\rho }}^{\ell }\left({\widehat{X}}^{†}\widehat{X}\right|\widehat{Y})-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left({\widehat{X}}^{†}{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\right|\widehat{Y})-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left({\mathbb{E}}_{\widehat{\rho }}^{\ell }{\left(\widehat{X}\right|\widehat{Y})}^{†}\widehat{X}\right|\widehat{Y})+{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\right|{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}){|}^2\left|\widehat{Y}\right)\hfill \\ & =& {\mathbb{E}}_{\widehat{\rho }}^{\ell }\left({\widehat{X}}^{†}\widehat{X}\right|\widehat{Y})-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left({\widehat{X}}^{†}{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\right|\widehat{Y}),\hfill \end{array}$$

(76)

We now compute

$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\widehat{\rho }}^{\ell }\left({\widehat{X}}^{†}{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\right|\widehat{Y})& =& \sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\mathrm{Tr}\left({\widehat{\Pi }}_y^{\widehat{Y}}{\widehat{X}}^{†}{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\widehat{\rho }\right)}{\mathrm{Tr}\left({\widehat{\Pi }}_y^{\widehat{Y}}\widehat{\rho }\right)}}{\widehat{\Pi }}_y^{\widehat{Y}}\hfill \\ & =& \sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{1}{\mathrm{Tr}\left({\widehat{\Pi }}_y^{\widehat{Y}}\widehat{\rho }\right)}}\left(\sum _{y^{\prime }\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\mathrm{Tr}\left({\widehat{\Pi }}_{y^{\prime }}^{\widehat{Y}}\widehat{X}\widehat{\rho }\right)\mathrm{Tr}\left({\widehat{\Pi }}_y^{\widehat{Y}}{\widehat{X}}^{†}{\widehat{\Pi }}_{y^{\prime }}^{\widehat{Y}}\widehat{\rho }\right)}{\mathrm{Tr}\left({\widehat{\Pi }}_{y^{\prime }}^{\widehat{Y}}\widehat{\rho }\right)}}\right){\widehat{\Pi }}_y^{\widehat{Y}}.\hfill \end{array}$$

As $\widehat{\rho }$ is a pure state, we note $\widehat{\rho }=\left|\psi \right.〉\psi$ , and we obtain that

$$\begin{array}{ccc}\hfill \sum _{y^{\prime }\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\mathrm{Tr}\left({\widehat{\Pi }}_{y^{\prime }}^{\widehat{Y}}\widehat{X}\widehat{\rho }\right)\mathrm{Tr}\left({\widehat{\Pi }}_y^{\widehat{Y}}{\widehat{X}}^{†}{\widehat{\Pi }}_{y^{\prime }}^{\widehat{Y}}\widehat{\rho }\right)}{\mathrm{Tr}\left({\widehat{\Pi }}_{y^{\prime }}^{\widehat{Y}}\widehat{\rho }\right)}}& =& \sum _{y^{\prime }\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\langle \psi ,{\phi }_{y^{\prime }}^{\widehat{Y}}\rangle \langle {\phi }_{y^{\prime }}^{\widehat{Y}},\widehat{X}\psi \rangle \langle \psi ,{\phi }_y^{\widehat{Y}}\rangle \langle {\phi }_y^{\widehat{Y}},{\widehat{X}}^{†}{\phi }_{y^{\prime }}^{\widehat{Y}}\rangle \langle {\phi }_{y^{\prime }}^{\widehat{Y}},\psi \rangle }{|\langle {\phi }_{y^{\prime }}^{\widehat{Y}},\psi \rangle {|}^2}}\hfill \\ & =& \sum _{y^{\prime }\in \sigma \left(\widehat{Y}\right)}\langle \psi ,{\phi }_y^{\widehat{Y}}\rangle \langle {\phi }_y^{\widehat{Y}},{\widehat{X}}^{†}{\phi }_{y^{\prime }}^{\widehat{Y}}\rangle \langle {\phi }_{y^{\prime }}^{\widehat{Y}},\widehat{X}\psi \rangle \hfill \\ & =& \langle \psi ,{\phi }_y^{\widehat{Y}}\rangle \langle {\phi }_y^{\widehat{Y}},{\widehat{X}}^{†}\widehat{X}\psi \rangle =\mathrm{Tr}\left({\widehat{\Pi }}_y^{\widehat{Y}}{\widehat{X}}^{†}\widehat{X}\widehat{\rho }\right).\hfill \end{array}$$

Consequently, we conclude that

$$\begin{array}{c}\hfill {\mathbb{E}}_{\widehat{\rho }}^{\ell }\left({\widehat{X}}^{†}{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\right|\widehat{Y})=\sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\mathrm{Tr}\left({\widehat{\Pi }}_y^{\widehat{Y}}{\widehat{X}}^{†}\widehat{X}\widehat{\rho }\right)}{\mathrm{Tr}\left({\widehat{\Pi }}_{y^{\prime }}^{\widehat{Y}}\widehat{\rho }\right)}}{\widehat{\Pi }}_y^{\widehat{Y}}={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left({\widehat{X}}^{†}\widehat{X}\right|\widehat{Y}).\end{array}$$

This proves Equation (71). Moreover, Equation (72) is a direct consequence of Equation (69), Equations (67) and (71). □

Having stressed the analogies between the classical and quantum conditional expectations and their basic properties, we now discuss four important differences between them.

First, classically, one has, if X is bounded ($X_{min}⩽X\left(\omega \right)⩽X_{max}$), that

$$X_{min}\le {\mathbb{E}}_{\mathbb{P}}\left(X\right|Y=y)\le X_{max}.$$

(77)

We have already pointed out that quantum mechanically, even if $\widehat{X}$ is self-adjoint, ${\mathbb{E}}_{\mathbb{P}}\left(\widehat{X}\right|\widehat{Y}=y)$ may be complex non-real. In addition, when it is indeed real, it may not be confined to lie between the minimal and maximal eigenvalue of $\widehat{X}$. One says a value ${\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right|\widehat{Y}=y)$ of the conditional expectation ${\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right|\widehat{Y})$ is anomalous if ${\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right|\widehat{Y}=y)\notin [x_{min},x_{max}]$, where $x_{min,max}$ are the extremal eigenvalues of $\widehat{X}$. The historic example of such anomalous values corresponds to a spin 1/2 [36]. Let $\mathcal{H}={\mathbb{C}}^2$, $\widehat{X}={\widehat{\sigma }}_z$, $\widehat{Y}={\widehat{\sigma }}_x$ and $\widehat{\rho }=|\alpha \rangle \langle \alpha |$, $|\alpha \rangle =cos\alpha |-1\rangle +sin\alpha |1\rangle$. Then,

$${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}=1)={\displaystyle \frac{sin\alpha -cos\alpha }{sin\alpha +cos\alpha }},{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}=-1)={\displaystyle \frac{sin\alpha +cos\alpha }{sin\alpha -cos\alpha }}.$$

(78)

Choosing $\alpha$ close to $\pi /4$, for example, the second term can be made arbitrary large, while the first one tends to 0. This phenomenon lies at the basis of weak value amplification; see [31] for a review on this topic. Note that, when $\alpha =\pi /4$, then $|\alpha \rangle$ does not belong to $D_{\widehat{Y}}$, since it is then an eigenvector of $\widehat{Y}={\widehat{\sigma }}_x$.

The existence of anomalous values is a strong indicator of typically quantum mechanical behavior as the following immediate consequence of Equation (77) shows.

Lemma 3.

Let $\widehat{X}$ and $\widehat{Y}$ be observables and $\widehat{\rho }\in D_{\widehat{Y}}$. Suppose there exists a probability $\mu :(x,y)\in \sigma \left(\widehat{X}\right)\times \sigma \left(\widehat{Y}\right)\mapsto \mu (x,y)\in [0,1]$ satisfying

(i) 

Born compatibility: for all $x\in \sigma \left(\widehat{X}\right)$, for all $y\in \sigma \left(\widehat{Y}\right)$,

$$\sum _{x\in \sigma \left(\widehat{X}\right)}\mu (x,y)=\mathrm{Tr}\left({\widehat{\Pi }}_y^{\widehat{Y}}\widehat{\rho }\right),\sum _{y\in \sigma \left(\widehat{Y}\right)}\mu (x,y)=\mathrm{Tr}\left({\widehat{\Pi }}_x^{\widehat{X}}\widehat{\rho }\right);$$

(79)

(ii) 

For all $y\in \sigma \left(\widehat{Y}\right)$, ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}=y)={\sum }_{x\in \sigma \left(\widehat{X}\right)}x{\displaystyle \frac{\mu (x,y)}{\left(\widehat{\rho }{\widehat{\Pi }}_y^{\widehat{Y}}\right)}}$.

Then,

$$x_{min}\le {\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}=y)\le x_{max}.$$

(80)

The same statement holds with ℓ replaced by $\mathrm{r}$. Condition (i) asserts that, for the given triple $(\widehat{X},\widehat{Y},\widehat{\rho })$, $\mu$ is a joint probability distribution on the joint spectrum of $\widehat{X}$ and $\widehat{Y}$ that has the correct quantum mechanical Born marginals in the state $\widehat{\rho }$ and $\widehat{X}$ and $\widehat{Y}$. Condition (ii) adds a condition on the natural associated conditional probability obtained from Bayes’ rule, $\mu \left(x\right|y)={\displaystyle \frac{\mu (x,y)}{\mathrm{Tr}\left(\widehat{\rho }\widehat{Y}\right)}}$: it must lead to a conditional expectation for $\widehat{X}$, given $\widehat{Y}$, that coincides with the quantum mechanical conditional expectation ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}=y)$. This then implies that the quantum mechanical conditional expectation cannot take on anomalous values. This lemma is of interest because its contrapositive provides a no-go theorem for the existence of joint probabilities: if, for a given triple $(\widehat{X},\widehat{Y},\widehat{\rho })$, the conditional expectation ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}=y)$ admits at least one anomalous value, then there does not exist a joint probability distribution satisfying (i) and (ii) of the lemma. The usual “no-go” theorems of this type, ruling out the existence of Born-compatible joint probability distributions in quantum mechanics, rely on assumptions that need to hold for all $\widehat{\rho }$, given a pair of observables $\widehat{X}$ and $\widehat{Y}$ or, alternatively, for all $\widehat{\rho }$ and all $\widehat{X}$ (see for example [1,4,30]). Here, we consider a single-state $\widehat{\rho }$ and add to the Born-compatibility of this state a second compatibility condition of the same type, still only for this unique state; indeed, (ii) requires equality between the “classical” conditional expectation of $\widehat{X}$ and the quantum one defined in the previous section. If the quantum conditional expectation admits at least one anomalous value (i.e., if Equation (80) is violated) then such a joint probability distribution cannot exist. It should be noted that the Born-compatibility condition in and by itself is very weak and can always be satisfied by taking, for example:

$$\mu (x,y)=\mathrm{Tr}\left(\widehat{\rho }{\widehat{\Pi }}_x^{\widehat{X}}\right)\mathrm{Tr}\left(\widehat{\rho }{\widehat{\Pi }}_y^{\widehat{Y}}\right).$$

However, this treats x and y as independent random variables, which is generally not coherent with the second condition. The current standard way to avoid this no-go theorem is to allow $\mu (x,y)$ to be complex-valued, i.e., a quasiprobability distribution rather than a probability distribution, as we will see in the next section.

To explain the second difference between the classical and quantum conditional expectations, we express Equation (69) in terms of the self-adjoint and the anti-self-adjoint part of ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$. For the self-adjoint part of ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$, we write

$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\widehat{\rho }}^{\ell ,\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})& :=& {\displaystyle \frac{1}{2}}\left({\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})+{\mathbb{E}}_{\widehat{\rho }}^{\ell }{\left(\widehat{X}\right|\widehat{Y})}^{†}\right)=\sum _{y\in \sigma \left(\widehat{Y}\right)}\mathrm{Re}\left({\displaystyle \frac{\langle {\phi }_y,\widehat{X}\widehat{\rho }{\phi }_y\rangle }{\langle {\phi }_y,\widehat{\rho }{\phi }_y\rangle }}\right){\widehat{\Pi }}_y\hfill \end{array}$$

(81)

$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r},\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})& :=& {\displaystyle \frac{1}{2}}\left({\mathbb{E}}_{\widehat{\rho }}^r\left(\widehat{X}\right|\widehat{Y})+{\mathbb{E}}_{\widehat{\rho }}^r{\left(\widehat{X}\right|\widehat{Y})}^{†}\right)=\sum _{y\in \sigma \left(\widehat{Y}\right)}\mathrm{Re}\left({\displaystyle \frac{\langle {\phi }_y,\widehat{\rho }\widehat{X}{\phi }_y\rangle }{\langle {\phi }_y,\widehat{\rho }{\phi }_y\rangle }}\right){\widehat{\Pi }}_y,\hfill \end{array}$$

(82)

for any $\widehat{\rho }\in D_{\widehat{Y}}$ and any $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$. Similarly, we introduce

$$\mathrm{Im}{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})={\displaystyle \frac{1}{2i}}\left({\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})-{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}{\left(\widehat{X}\right|\widehat{Y})}^{†}\right)$$

(83)

As the self-adjoint and anti-self-adjoint parts of ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ are both functions of $\widehat{Y}$, they commute. Then,

$${\Delta }_{\widehat{\rho }}^{\ell /\mathrm{r},2}{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})={\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r},\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})+{\Delta }_{\widehat{\rho }}^2\mathrm{Im}{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y}).$$

(84)

In conclusion, the law of total variance, Equation (69), shows that the variance of $\widehat{X}$ differs from the sum of the variances of the self-adjoint and anti-self-adjoint part of its conditional expectation given $\widehat{Y}$ by the mean quadratic error between the two. Note that, for self-adjoint $\widehat{X}$, the underlying chosen ordering is irrelevant:

$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\widehat{\rho }}^{\ell ,\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})={\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r},\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})& =& {\displaystyle \frac{1}{2}}\sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\langle {\phi }_y,\{\widehat{\rho },\widehat{X}\}{\phi }_y\rangle }{\langle {\phi }_y,\widehat{\rho }{\phi }_y\rangle }},\hfill \end{array}$$

(85)

$$\begin{array}{ccc}\hfill \mathrm{Im}{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})=-\mathrm{Im}{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})& =& {\displaystyle \frac{1}{2i}}\sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\langle {\phi }_y,[\widehat{\rho },\widehat{X}]{\phi }_y\rangle }{\langle {\phi }_y,\widehat{\rho }{\phi }_y\rangle }},\hfill \end{array}$$

(86)

where $\{·,·\}$ denotes the anti-commutator and $[·,·]$ the commutator. Thus, for self-adjoint $\widehat{X}$, we will write

$${\mathbb{E}}_{\rho }^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y}):={\mathbb{E}}_{\widehat{\rho }}^{\ell ,\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})={\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r},\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y}).$$

(87)

It should be noted that this equality need not be true if $\widehat{X}$ is not a self-adjoint operator. In particular, we have that, generally,

$${\mathbb{E}}_{\widehat{\rho }}^{\ell ,\mathrm{sa}}\left(f\left(\widehat{Y}\right)\widehat{X}\right|\widehat{Y})\ne {\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r},\mathrm{sa}}\left(f\left(\widehat{Y}\right)\widehat{X}\right|\widehat{Y})$$

for $f:\sigma \left(\widehat{Y}\right)\to \mathbb{R}$, even if $\widehat{X}$ is self-adjoint. The law of total variance, Equation (69), then becomes, for self-adjoint $\widehat{X}$,

$${\Delta }_{\widehat{\rho }}^2\widehat{X}={\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})+{\Delta }_{\widehat{\rho }}^2\left(\mathrm{Im}{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})\right)+{\langle (\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})),(\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y}))\rangle }_{\ell /\mathrm{r}}.$$

(88)

Comparing this to the classical equality Equation (30), one notices the extra term given by the variance of $\mathrm{Im}{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ which comes from the non-self-adjointness of the conditional expectation. This term depends on the commutator $[\widehat{\rho },\widehat{X}]$ and as such, its absence classically is expected. We will see in Section 6 that it can vanish even if $\widehat{\rho }$ and $\widehat{X}$ do not commute, and we will use results on the KD representations of quantum mechanics to show many examples where $\mathrm{Im}{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ vanishes. We also further explore there the link between $\mathrm{Im}{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ and an appropriately defined Fisher information associated to a parameter estimation protocol naturally associated with a triple $\widehat{\rho }$, $\widehat{X}$, and $\widehat{Y}$.

The third difference with the classical situation concerns what happens for pure states. In classical probability theory, the pure states are Dirac delta measures ${\delta }_{{\omega }_0}$, on $\Omega$, with ${\omega }_0\in \Omega$. If $X,Y$ are random variables, ${\mathbb{E}}_{{\delta }_{{\omega }_o}}\left(X\right|Y=y)=X\left({\omega }_0\right){\delta }_{Y\left({\omega }_0\right)}\left(y\right)$, and is therefore also a Dirac delta measure, on $\mathbb{R}$. As a result, the classical analogues to Equations (71) and (72) also hold, but with one major difference: since the Dirac delta measures are dispersion-free, both sides of this classical equivalent of Equation (72) then vanish. In the quantum case, pure states $\widehat{\rho }=|\psi \rangle \langle \psi |$ are never dispersion-free, and neither side vanishes as soon as $|\psi \rangle$ is not an eigenstate of $\widehat{X}$. So, in quantum theory, the error term in the law of total variation vanishes for pure states, and the variance of $\widehat{X}$ equals that of its conditional expectation ${\mathbb{E}}_{|\psi \rangle \langle \psi |}\left(\widehat{X}\right|\widehat{Y})$.

Fourth, and finally, we note that the well-known relation that holds in classical probability

$${\Delta }^2\left(X\right|Y)={\mathbb{E}}_{\mathbb{P}}\left({\left|X\right|}^2\right|Y)-{\left|{\mathbb{E}}_{\mathbb{P}}\left(X\right|Y)\right|}^2,$$

does not hold in the quantum case. Indeed, it follows from Equation (76) that, for the left quantum conditional expectation, for example:

$${\Delta }^{\ell ,2}\left(\widehat{X}\right|\widehat{Y})={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left({\widehat{X}}^{†}\widehat{X}\right|\widehat{Y})-{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left({\widehat{X}}^{†}{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\right|Y)\ne {\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\right|\widehat{X}{|}^2\left|Y\right)-{\left|{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\right|}^2.$$

One can also define an alternative conditional variance by the final expression above, specifically,

$${\tilde{\Delta }}_{\widehat{\rho }}^{2,\ell }\left(\widehat{X}\right|\widehat{Y}):={\mathbb{E}}_{\widehat{\rho }}^{\ell }(|\widehat{X}{|}^2|Y)-{\left|{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\right|}^2,$$

(89)

with an obvious right variant obtained by replacing $|\widehat{X}{|}^2={\widehat{X}}^{†}\widehat{X}$ by $\widehat{X}{\widehat{X}}^{†}$ (the two coincide if $\widehat{X}$ is self-adjoint). These will then still satisfy the law of total variance, in the form

$${\Delta }^{2,\ell }\left(\widehat{X}\right)={\Delta }^{2,\ell }\left({\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})\right)+{\mathbb{E}}_{\widehat{\rho }}\left({\tilde{\Delta }}^{2,\ell }\left(\widehat{X}\right|\widehat{Y})\right).$$

Another notion of conditional variance that has been proposed in the literature is the weak variance introduced in [47], which with our notations is defined as

$${\sigma }_{w,\widehat{\rho }}\left(\widehat{X}\right|\widehat{Y})={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left({\widehat{X}}^2\right|\widehat{Y})-{\mathbb{E}}_{\widehat{\rho }}^{\ell }{\left(\widehat{X}\right|\widehat{Y})}^2.$$

(90)

This weak variance can be given an operational meaning in the context of weak measurements (cf. Appendix A, in particular Formula (A33)) but does not satisfy the law of total variance, unlike (89) which can be given a similar operational significance: see Corollary A1. There exists in fact a whole range of plausible definitions of conditional variances, including variants which can be introduced in the context of quasiprobabilistic representations of quantum mechanics, whose relative merits still have to be investigated.

3.4. The Real Part of ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ as a Best Estimator

As one might expect, the left and right self-adjoint quantum conditional expectations ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r},\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})$ satisfy properties similar to the ones given in Theorem 5. We specify them for the left self-adjoint quantum conditional expectation. They also hold for the right self-adjoint one, with adapted ordering. We use ${\mathcal{F}}_{\mathbb{R},\widehat{Y}}$ to denote the set of self-adjoint functions of $\widehat{Y}$:

$${\mathcal{F}}_{\mathbb{R},\widehat{Y}}=\left\{f\left(\widehat{Y}\right)\mid f:\sigma \left(\widehat{Y}\right)\to \mathbb{R}\right\}.$$

(91)

Slightly adapting the proof of Theorem 5, one easily shows that for $\widehat{\rho }\in D_{\widehat{Y}}$, ${\mathbb{E}}_{\widehat{\rho }}^{\ell ,\mathrm{sa}}(·|\widehat{Y})$ is the unique map with values in the set ${\mathcal{F}}_{\mathbb{R},\widehat{Y}}$ with the two following properties:

1.

For any $f:\sigma \left(\widehat{Y}\right)\to \mathbb{R}$ and any $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$,

$${\mathbb{E}}_{\widehat{\rho }}^{\ell ,\mathrm{sa}}\left(f\left(\widehat{Y}\right)\widehat{X}\right|\widehat{Y})=f\left(\widehat{Y}\right){\mathbb{E}}_{\widehat{\rho }}^{\ell ,\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y}).$$

(92)

2.

For any $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$,

$${\mathbb{E}}_{\widehat{\rho }}\left({\mathbb{E}}_{\widehat{\rho }}^{\ell ,\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})\right)=\mathrm{Re}\left({\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right)\right).$$

(93)

Finally, it is also easily seen that this left/right conditional expectations can be obtained as the minimizer of the mean square error functional

$$f\mapsto {\langle \widehat{X}-f\left(\widehat{Y}\right),\widehat{X}-f\left(\widehat{Y}\right)\rangle }_{\ell /\mathrm{r}},$$

(94)

where the minimization is considered over the set of real valued functions $f:\sigma \left(\widehat{Y}\right)\to \mathbb{R}$. The self-adjoint left conditional expectation of $\widehat{X}$ is therefore the observable closest to $\widehat{X}$ (with respect to the above degenerate left inner product) that is a self-adjoint function of $\widehat{Y}$.

For self-adjoint $\widehat{X}$, this real minimization problem was previously addressed in [32,33,34,35,49], where the link between the minimizer, viewed as a conditional expectation, and the real part of weak values was also established. As we pointed out above, our analysis in Section 3.1 shows that, in fact, the conditional expectation ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ itself is a best estimator of $\widehat{X}$ (self-adjoint or not), when the minimization is performed over all complex valued operator functions of $\widehat{Y}$, and not only over the self-adjoint ones.

As in the proof of Proposition 2, one finds that for any $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$

$${\langle \widehat{X}-{\mathbb{E}}_{\rho }^{\ell /\mathrm{r},\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y}),\widehat{X}-{\mathbb{E}}_{\rho }^{\ell /\mathrm{r},\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell /\mathrm{r}}={\mathbb{E}}_{\widehat{\rho }}\left(\right|\widehat{X}{|}^2)-{\mathbb{E}}_{\widehat{\rho }}\left({\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r},\mathrm{sa}}{\left(\widehat{X}\right|\widehat{Y})}^2\right).$$

(95)

Furthermore, if $\widehat{X}$ is self-adjoint, we have ${\mathbb{E}}_{\widehat{\rho }}\left({\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})\right)=\mathrm{Re}\left({\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right)\right)={\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right)$. We deduce

$${\Delta }_{\widehat{\rho }}^2\widehat{X}={\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})+{\langle \widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y}),\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell /\mathrm{r}}.$$

(96)

Comparing this to Equation (69), we have that the error made when approximating the variance of $\widehat{X}$ by the variance of ${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})$ is greater than the error made when approximating the variance of $\widehat{X}$ by the variance of ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$. This is a consequence of the fact that the mean squared error functional is minimized over a larger class of functions in the latter case than in the former. More precisely, from Equation (88), we have

$${\langle \widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y}),\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})\rangle }_{\ell /\mathrm{r}}={\Delta }_{\widehat{\rho }}^2\left(\mathrm{Im}{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})\right)+{\langle (\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})),(\widehat{X}-{\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y}))\rangle }_{\ell /\mathrm{r}}.$$

4. Quantum Conditional Expectation via Quasiprobability Representation of Quantum Mechanics

As shown in the previous section, defining the quantum conditional expectation via a minimization problem as in Equation (44) or Equation (46) is a way of mimicking what happens classically, see Theorem 2. It also allowed us to compare the classical and quantum notions and to emphasize their differences. One can alternatively try to imitate the classical conditional expectation by using Definition 1. This definition uses the joint probability distribution of the two random variables under consideration, which is naturally defined in classical probability theory. In quantum mechanics, as recalled in the introduction, a notion of joint probability distribution does not exist for non-commuting observables, but one can try to use a quasiprobability distribution instead.

To that end, we first describe the class of quasiprobability representations of quantum mechanics that we consider. We will use the formalism of frames, as detailed in [3,30], to do so. Let $\mathcal{H}$ be a Hilbert space of dimension d, let $\Lambda$ be a finite set, with $\left|\Lambda \right|=d^2$, where $\left|A\right|$ is the cardinal of the finite set A. Let ${\left(S_{\lambda }\right)}_{\lambda \in \Lambda }$ be a basis of $\mathcal{L}\left(\mathcal{H}\right)$, satisfying

$$\sum _{\lambda }{S}_{\lambda }={\mathbb{I}}_d.$$

(97)

Let ${\left(T_{\lambda }\right)}_{\lambda \in \Lambda }$ denote its unique dual basis, which satisfies

$$\mathrm{Tr}\left(T_{\lambda }^{†}{S}_{{\lambda }^{\prime }}\right)={\delta }_{\lambda ,{\lambda }^{\prime }}.$$

(98)

Given an operator $\widehat{C}$ on $\mathcal{H}$, we define

$$Q_{\lambda }\left(\widehat{C}\right)=\mathrm{Tr}\left(\widehat{C}{\widehat{S}}_{\lambda }^{†}\right),{\tilde{Q}}_{\lambda }\left(\widehat{C}\right)=\mathrm{Tr}\left(\widehat{C}{T}_{\lambda }^{†}\right).$$

(99)

It follows that the maps $Q,\tilde{Q}:\mathcal{L}\left(\mathcal{H}\right)\to {\mathbb{C}}^{\Lambda }$ are bijective and that

$$\widehat{C}=\sum _{\lambda }{\tilde{Q}}_{\lambda }\left(\widehat{C}\right)S_{\lambda },\widehat{C}=\sum _{\lambda }{Q}_{\lambda }\left(\widehat{C}\right)T_{\lambda }.$$

(100)

Given $\widehat{C},\widehat{D}\in \mathcal{L}\left(\mathcal{H}\right)$, one then has

$$\mathrm{Tr}\widehat{C}=\sum _{\lambda }{Q}_{\lambda }\left(\widehat{C}\right),\mathrm{Tr}\left({\widehat{C}}^{†}\widehat{D}\right)=\sum _{\lambda }\overline{{\tilde{Q}}_{\lambda }\left(\widehat{C}\right)}{Q}_{\lambda }\left(\widehat{D}\right).$$

(101)

The pair $(Q,\tilde{Q})$, which is completely determined by the choice of $\Lambda$ and of the basis ${\left(S_{\lambda }\right)}_{\lambda }$, is referred to as a quasiprobability representation of quantum mechanics. It associates to each density matrix $\widehat{\rho }$ a quasiprobability $Q_{\lambda }\left(\widehat{\rho }\right)$ on $\Lambda$, which is a complex-valued function that satisfies

$$1=\sum _{\lambda }{Q}_{\lambda }\left(\widehat{\rho }\right),$$

(102)

and to each observable $\widehat{C}={\widehat{C}}^{†}$ its symbol ${\tilde{Q}}_{\lambda }\left(\widehat{C}\right)$, with

$$\mathrm{Tr}\left(\widehat{\rho }\widehat{C}\right)=\sum _{\lambda }{Q}_{\lambda }\left(\widehat{\rho }\right)\overline{{\tilde{Q}}_{\lambda }\left(\widehat{C}\right).}$$

(103)

Depending on context, one thinks of $\Lambda$ as an ontic space or as a classical phase space, on which the quantum states and observables are represented by quasiprobabilities and functions respectively. Many quasiprobability representations of quantum mechanics fitting in the above framework have been introduced and studied. A number of those [14,15,16,17,18,20,21] aim at defining quasiprobability representations for quantum systems on finite dimensional Hilbert space reproducing many of the known properties of the Wigner–Weyl–Moyal representation [5,9,12,13] associated with conjugate variables and the Fourier transform on $L^2\left({\mathbb{R}}^2\right)$. Extensions of such constructions to locally compact Abelian groups have been considered more recently in [22,25]. A more versatile family of quasiprobability representation is provided by the Kirkwood–Dirac representations: they are defined using two arbitrary observables $\widehat{A}$ and $\widehat{B}$ and we shall describe them in detail in the next section. Further examples of quasiprobability representations can be found in [30].

Definition 3.

Let $\widehat{Y}$ be a $\mathrm{CSCO}$ and let $(Q,\tilde{Q})$ be a quasiprobability representation of quantum mechanics. Writing ${\tilde{Q}}_{\lambda }\left(\widehat{Y}\right):=({\tilde{Q}}_{\lambda }\left({\widehat{Y}}_1\right),\dots ,{\tilde{Q}}_{\lambda }\left({\widehat{Y}}_m\right))$, we define, for all $y\in {\mathbb{C}}^m:$

$${\Lambda }_y^{\widehat{Y}}=\{\lambda \in \Lambda \mid {\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)=y\}.$$

We say the quasiprobability representation $(Q,\tilde{Q})$ of quantum mechanics is $\widehat{Y}$-compatible provided for all density matrices $\widehat{\rho }$, one has

$$\begin{array}{ccc}\hfill \sum _{\lambda \in {\Lambda }_y^{\widehat{Y}}}{Q}_{\lambda }\left(\widehat{\rho }\right)& =& \mathrm{Tr}\left(\widehat{\rho }{\widehat{\Pi }}_y^{\widehat{Y}}\right),\mathrm{when}y\in \sigma \left(\widehat{Y}\right),\hfill \end{array}$$

(104)

$$\begin{array}{ccc}& =& 0,\mathrm{when}y\notin \sigma \left(\widehat{Y}\right).\hfill \end{array}$$

(105)

If $Q_{\lambda }\left(\widehat{\rho }\right)$ is a probability on $\Lambda$ (meaning it is non-negative for all $\lambda$), then what this condition means is that the joint probability law of the vector-valued random variable $\tilde{Q}\left(\widehat{Y}\right)=(\tilde{Q}\left({\widehat{Y}}_1\right),\dots ,\tilde{Q}\left({\widehat{Y}}_m\right))\in {\left({\mathbb{C}}^m\right)}^{\Lambda }$ induced by the probability $Q_{\lambda }\left(\widehat{\rho }\right)$ on $\Lambda$ is identical to the quantum mechanical Born-probability law of $\widehat{Y}$ when the system is in the state $\widehat{\rho }$. More generally, to be $\widehat{Y}$-compatible, a quasiprobability representation must reproduce the correct Born-probabilities for the observable $\widehat{Y}$ and for all states $\widehat{\rho }$, even those for which the quasiprobability $Q_{\lambda }\left(\widehat{\rho }\right)$ takes on non-positive values and is therefore only a quasiprobability distribution.

The following lemma expresses the fact that, if $(Q,\tilde{Q})$ is $\widehat{Y}$-compatible, then it agrees naturally with the functional calculus of $\widehat{Y}$ in the sense that the symbol of $f\left(\widehat{Y}\right)$ equals $\lambda \mapsto f\left({\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)\right)$.

Lemma 4.

If $(Q,\tilde{Q})$ is a $\widehat{Y}$-compatible quasiprobability representation, then the family ${\left({\Lambda }_y^{\widehat{Y}}\right)}_{y\in \sigma \left(\widehat{Y}\right)}$ is a partition of Λ and

$$\forall f\left(\widehat{Y}\right)\in {\mathcal{F}}_{\mathbb{C},\widehat{Y}},\forall \lambda \in \Lambda ,{\tilde{Q}}_{\lambda }\left(f\left(\widehat{Y}\right)\right)=f\left({\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)\right).$$

(106)

The lemma states that the symbol of a function f of $\widehat{Y}$ equals the same function f of the symbol of $\widehat{Y}$. Note that, generally, if $\widehat{X}$ and $\widehat{Z}$ are operators on $\mathcal{H}$, then it is not true that $\tilde{Q}\left(\widehat{X}\widehat{Z}\right)=\tilde{Q}\left(\widehat{X}\right)\tilde{Q}\left(\widehat{Z}\right)$. When both $\widehat{X}$ and $\widehat{Z}$ are functions of $\widehat{Y}$, this is true, however, by the above lemma, for $\widehat{Y}$-compatible quasiprobability representations.

Proof. 

From Equation (104), one finds that for all $\widehat{\rho }$ and for all $y\in \sigma \left(\widehat{Y}\right)$

$$\mathrm{Tr}\left(\widehat{\rho }\sum _{\lambda \in {\Lambda }_y^{\widehat{Y}}}{S}_{\lambda }^{†}\right)=\mathrm{Tr}\left(\widehat{\rho }{\widehat{\Pi }}_y^{\widehat{Y}}\right).$$

(107)

Therefore,

$${\widehat{\Pi }}_y^{\widehat{Y}}=\sum _{\lambda \in {\Lambda }_y^{\widehat{Y}}}{S}_{\lambda }^{†}=\sum _{\lambda \in {\Lambda }_y^{\widehat{Y}}}{S}_{\lambda }.$$

(108)

Equation (100) then implies that

$$\tilde{Q}\left({\widehat{\Pi }}_y^{\widehat{Y}}\right)={\mathbb{1}}_{{\Lambda }_y^{\widehat{Y}}},$$

(109)

where ${\mathbb{1}}_E$ denotes the indicator function of a set E, and therefore

$$1=\tilde{Q}\left({\mathbb{I}}_d\right)=\sum _{y\in \sigma \left(\widehat{Y}\right)}\tilde{Q}\left({\widehat{\Pi }}_y^{\widehat{Y}}\right)=\sum _{y\in \sigma \left(\widehat{Y}\right)}{\mathbb{1}}_{{\Lambda }_y^{\widehat{Y}}}.$$

(110)

Moreover, it is clear that ${\Lambda }_y^{\widehat{Y}}\cap {\Lambda }_{y^{\prime }}^{\widehat{Y}}=\varnothing$ if $y\ne y^{\prime }$. Since none of the ${\Lambda }_y^{\widehat{Y}}$ are empty, by Equation (107), this implies that ${\left({\Lambda }_y^{\widehat{Y}}\right)}_{y\in \sigma \left(\widehat{Y}\right)}$ is a partition of $\Lambda$. Consequently, one has that

$$\tilde{Q}\left(\widehat{Y}\right)=\sum _{y\in \sigma \left(\widehat{Y}\right)}y{\mathbb{1}}_{{\Lambda }_y^{\widehat{Y}}}.$$

(111)

Finally,

$$\tilde{Q}\left(f\left(\widehat{Y}\right)\right)=\sum _{y\in \sigma \left(\widehat{Y}\right)}f\left(y\right)\tilde{Q}\left({\widehat{\Pi }}_y^{\widehat{Y}}\right)=\sum _{y\in \sigma \left(\widehat{Y}\right)}f\left(y\right){\mathbb{1}}_{{\Lambda }_y^{\widehat{Y}}}=f\left(\tilde{Q}\left(\widehat{Y}\right)\right).$$

(112)

□

Suppose now that we have a $(Q,\tilde{Q})$ that is $\widehat{Y}$-compatible. Given $\widehat{\rho }\in D_{\widehat{Y}}$, we then define the quasiprobability of $\lambda$, given $y\in \sigma \left(\widehat{Y}\right)$, as follows

$$\begin{array}{cccc}\hfill Q_{\lambda |y}\left(\widehat{\rho }\right)& =\hfill & \hfill {\displaystyle \frac{Q_{\lambda }\left(\widehat{\rho }\right)}{Q\left({\Lambda }_y^{\widehat{Y}}\right)}}={\displaystyle \frac{Q_{\lambda }\left(\widehat{\rho }\right)}{\langle {\phi }_y\left|\widehat{\rho }\right|{\phi }_y\rangle }}& \lambda \in {\Lambda }_y^{\widehat{Y}},\hfill \end{array}$$

(113)

$$\begin{array}{cccc}& =\hfill & \hfill 0& \lambda \notin {\Lambda }_y^{\widehat{Y}}.\hfill \end{array}$$

(114)

This allows us to define a notion of conditional expectation naturally associated to the quasiprobability representation $(Q,\tilde{Q})$, as follows.

Definition 4.

For $\widehat{Y}$-compatible $(Q,\tilde{Q})$ we define the Q-conditional expectation of $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$ knowing $\widehat{Y}$ in the state $\widehat{\rho }\in D_{\widehat{Y}}$ by

$${\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{Y})=\sum _{y\in \sigma \left(\widehat{Y}\right)}\sum _{\lambda \in {\Lambda }_y^{\widehat{Y}}}\overline{{\tilde{Q}}_{\lambda }\left({\widehat{X}}^{†}\right)}{Q}_{\lambda |y}\left(\widehat{\rho }\right){\widehat{\Pi }}_y^{\widehat{Y}}.$$

(115)

Note that, whenever $\widehat{\rho }$ is Q-positive, by which we mean that $Q_{\lambda }\left(\widehat{\rho }\right)\ge 0$ for all $\lambda$, $Q_{\lambda |y}\left(\widehat{\rho }\right)$ is equal to the conditional probability of $\lambda$, given that the random variable $\tilde{Q}\left(\widehat{Y}\right)$ takes the value y: $\tilde{Q}\left(\widehat{Y}\right)=y$. Nevertheless, even in that case, and even if $\widehat{X}$ is self-adjoint, ${\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{Y})$ is not necessarily self-adjoint. This will be the case, provided ${\tilde{Q}}_{\lambda }\left(\widehat{X}\right)$ is real.

The following result is now immediate:

Proposition 3.

The Q-conditional expectation has the following property: for any $\widehat{\rho }\in D_{\widehat{Y}}$ and any $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$,

$${\mathbb{E}}_{\widehat{\rho }}\left({\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{Y})\right)={\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right).$$

(116)

In particular, for any $\rho \in D_{\widehat{Y}}$ and any $\lambda \in \Lambda$

$$Q_{\lambda }\left(\widehat{\rho }\right)=\mathrm{Tr}\left(\widehat{\rho }{S}_{\lambda }^{†}\right)={\mathbb{E}}_{\widehat{\rho }}\left({\mathbb{E}}_{\widehat{\rho }}^Q\left(S_{\lambda }^{†}\right|\widehat{Y})\right).$$

(117)

We can now formulate our first main result. It provides a characterization of all quasiprobability distributions that are compatible with the projective measurement of a given CSCO $\widehat{Y}$ and the associated conditional expectation of which coincides with the one defined in terms of minimization, as in the previous section. Recall that $\lambda \mapsto {\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)$ sends $\Lambda$ onto the spectrum of $\widehat{Y}$, by Equation (111).

Theorem 6.

Let $(Q,\tilde{Q})$ be a $\widehat{Y}$-compatible quasiprobability representation of quantum mechanics on $\mathcal{H}$. Then, the following statements are equivalent:

(i) 

$\forall \widehat{X}\in \mathcal{L}\left(\mathcal{H}\right),\forall \widehat{\rho }\in D_{\widehat{Y}},{\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{Y})={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$;

(ii) 

$\forall \widehat{X}\in \mathcal{L}\left(\mathcal{H}\right),\forall \widehat{\rho }\in D_{\widehat{Y}},\forall f:\sigma \left(\widehat{Y}\right)\to \mathbb{C}$, ${\mathbb{E}}_{\widehat{\rho }}^Q\left(f\left(\widehat{Y}\right)\widehat{X}\right|\widehat{Y})=f\left(\widehat{Y}\right){\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{Y})$.

(iii) 

$\forall \lambda \in \Lambda ,\forall \widehat{\rho }\in D_{\widehat{Y}},S_{\lambda }=S_{\lambda }{\widehat{\Pi }}_y^{\widehat{Y}},T_{\lambda }=T_{\lambda }{\widehat{\Pi }}_y^{\widehat{Y}}$, with $y={\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)\in \sigma \left(\widehat{Y}\right)$.

Condition (iii) implies that both the frame operators $S_{\lambda }$ and their duals $T_{\lambda }$ are rank one operators, which is quite a restrictive condition. It is this property that will allow us to single out the KD distributions in Section 5.2 below.

Proof. 

The statement that (i) and (ii) are equivalent follows from Theorem 5 and Equation (116). We now prove that (i) implies (iii). From (i), one finds that, for all $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$, for all $y\in \sigma \left(\widehat{Y}\right)$ and $\widehat{\rho }\in D_{\widehat{Y}}$,

$$\sum _{\lambda \in {\Lambda }_y^{\widehat{Y}}}\overline{{\tilde{Q}}_{\lambda }\left({\widehat{X}}^{†}\right)}{Q}_{\lambda |y}\left(\widehat{\rho }\right)={\displaystyle \frac{\mathrm{Tr}\left(\widehat{\rho }{\widehat{\Pi }}_y^{\widehat{Y}}\widehat{X}\right)}{\mathrm{Tr}\left(\widehat{\rho }{\widehat{\Pi }}_y^{\widehat{Y}}\right)}}.$$

(118)

Inserting $\widehat{X}=S_{{\lambda }^{\prime }}^{†}$ and using that ${\tilde{Q}}_{\lambda }\left(S_{{\lambda }^{\prime }}\right)={\delta }_{\lambda ,{\lambda }^{\prime }}$, one finds that

$$\sum _{\lambda \in {\Lambda }_y^{\widehat{Y}}}{\delta }_{\lambda {\lambda }^{\prime }}{Q}_{\lambda }\left(\widehat{\rho }\right)=\mathrm{Tr}\left(\widehat{\rho }{\widehat{\Pi }}_y^{\widehat{Y}}{S}_{{\lambda }^{\prime }}^{†}\right).$$

(119)

Since both sides are continuous in $\widehat{\rho }$, this equality holds for all $\widehat{\rho }$ by Lemma 2 and hence, for all $y\in \sigma \left(\widehat{Y}\right),{\lambda }^{\prime }\in \Lambda$,

$$\sum _{\lambda \in {\Lambda }_y^{\widehat{Y}}}{\delta }_{\lambda {\lambda }^{\prime }}{S}_{\lambda }^{†}={\widehat{\Pi }}_y^{\widehat{Y}}{S}_{{\lambda }^{\prime }}^{†}.$$

(120)

We know from Lemma 4 that ${\tilde{Q}}_{{\lambda }^{\prime }}\left(\widehat{Y}\right)\in \sigma \left(\widehat{Y}\right)$. Consequently, Equation (120) implies that, if ${\lambda }^{\prime }\notin {\Lambda }_y^{\widehat{Y}}$, then $S_{{\lambda }^{\prime }}{\widehat{\Pi }}_y^{\widehat{Y}}=0$ and if ${\lambda }^{\prime }\in {\Lambda }_y^{\widehat{Y}}$, then $S_{{\lambda }^{\prime }}=S_{{\lambda }^{\prime }}{\widehat{\Pi }}_y^{\widehat{Y}}$. Since ${\sum }_{y\in \sigma \left(\widehat{Y}\right)}{\widehat{\Pi }}_y^{\widehat{Y}}={\mathbb{I}}_d$, it follows that, for all ${\lambda }^{\prime }\in \Lambda$

$$S_{{\lambda }^{\prime }}=\sum _{y\in \sigma \left(\widehat{Y}\right)}{S}_{{\lambda }^{\prime }}{\widehat{\Pi }}_y^{\widehat{Y}}=S_{{\lambda }^{\prime }}{\widehat{\Pi }}_{{\tilde{Q}}_{{\lambda }^{\prime }}\left(\widehat{Y}\right)}^{\widehat{Y}}+\sum _{y\in \sigma \left(\widehat{Y}\right)\setminus \left\{{\tilde{Q}}_{{\lambda }^{\prime }}\left(\widehat{Y}\right)\right\}}{S}_{{\lambda }^{\prime }}{\widehat{\Pi }}_y^{\widehat{Y}}=S_{{\lambda }^{\prime }}{\widehat{\Pi }}_{{\tilde{Q}}_{{\lambda }^{\prime }}\left(\widehat{Y}\right)}^{\widehat{Y}}.$$

(121)

Introducing, for $y\in \sigma \left(\widehat{Y}\right)$,

$${\mathcal{L}}_y=\mathrm{span}\{S_{\lambda }\mid \lambda \in {\Lambda }_y^{\widehat{Y}}\},$$

(122)

it follows that these spaces are orthogonal with respect to the Hilbert–Schmidt inner product,

$${\mathcal{L}}_y\perp {\mathcal{L}}_{y^{\prime }}\mathrm{if}y\ne y^{\prime }.$$

(123)

Since the ${\Lambda }_y^{\widehat{Y}}$ form a partition of $\Lambda$, one therefore concludes that

$$\mathcal{L}\left(\mathcal{H}\right)={\displaystyle \underset{y\in \sigma \left(\widehat{Y}\right)}{\overset{\perp }{⨁}}}{\mathcal{L}}_y,$$

(124)

where the sum is an orthogonal direct sum for the Hilbert–Schmidt inner product. It follows from this that

$$T_{\lambda }=T_{\lambda }{\widehat{\Pi }}_{{\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)}^{\widehat{Y}}.$$

(125)

Indeed, $T_{\lambda }$ is by definition orthogonal to all $S_{{\lambda }^{\prime }}$ with ${\lambda }^{\prime }\ne \lambda$. It is therefore orthogonal to all $S_{{\lambda }^{\prime }}$ with ${\lambda }^{\prime }$ such that ${\tilde{Q}}_{{\lambda }^{\prime }}\left(\widehat{Y}\right)\ne {\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)$. Hence $T_{\lambda }$ is orthogonal to each of the subspaces ${\mathcal{L}}_{y^{\prime }}$ with $y^{\prime }\ne {\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)$ and therefore

$$T_{\lambda }=\sum _{{\lambda }^{\prime }\in {\Lambda }_{{\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)}}{c}_{{\lambda }^{\prime }}{S}_{{\lambda }^{\prime }}=\sum _{{\lambda }^{\prime }\in {\Lambda }_{{\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)}}{c}_{{\lambda }^{\prime }}{S}_{{\lambda }^{\prime }}{\widehat{\Pi }}_{{\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)}^{\widehat{Y}}=T_{\lambda }{\widehat{\Pi }}_{{\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)}^{\widehat{Y}}.$$

(126)

This implies (iii).

We finally show that (iii) implies (ii). For that purpose, we compute

$$\begin{array}{ccc}\hfill {\tilde{Q}}_{\lambda }\left({\widehat{X}}^{†}f{\left(\widehat{Y}\right)}^{†}\right)& =& \mathrm{Tr}\left({\widehat{X}}^{†}f{\left(\widehat{Y}\right)}^{†}{T}_{\lambda }^{†}\right)\hfill \\ & =& \mathrm{Tr}\left({\widehat{X}}^{†}f{\left(\widehat{Y}\right)}^{†}{\widehat{\Pi }}_{{\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)}^{\widehat{Y}}{T}_{\lambda }^{†}\right)\hfill \\ & =& \overline{f\left({\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)\right)}\left({\widehat{X}}^{†}{T}_{\lambda }^{†}\right)\hfill \\ & =& \overline{f\left({\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)\right)}{\tilde{Q}}_{\lambda }\left({\widehat{X}}^{†}\right),\hfill \end{array}$$

(127)

where we used for the third line that $f{\left(\widehat{Y}\right)}^{†}{\widehat{\Pi }}_{{\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)}^{\widehat{Y}}=\overline{f\left({\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)\right)}{\widehat{\Pi }}_{{\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)}^{\widehat{Y}}$. One then computes

$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\widehat{\rho }}^Q\left(f\left(\widehat{Y}\right)\widehat{X}\right|\widehat{Y})& =& \sum _{y\in \sigma \left(\widehat{Y}\right)}\sum _{\lambda \in {\Lambda }_y^{\widehat{Y}}}\left(f\left({\tilde{Q}}_{\lambda }\left(\widehat{Y}\right)\right)\overline{{\tilde{Q}}_{\lambda }\left({\widehat{X}}^{†}\right)}{Q}_{\lambda |y}\left(\widehat{\rho }\right)\right){\widehat{\Pi }}_y^{\widehat{Y}}\hfill \\ & =& \sum _{y\in \sigma \left(\widehat{Y}\right)}f\left(y\right)\sum _{\lambda \in {\Lambda }_y^{\widehat{Y}}}\left(\overline{{\tilde{Q}}_{\lambda }\left({\widehat{X}}^{†}\right)}{Q}_{\lambda |y}\left(\widehat{\rho }\right)\right){\widehat{\Pi }}_y^{\widehat{Y}}\hfill \\ & =& f\left(\widehat{Y}\right){\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{Y}),\hfill \end{array}$$

which establishes (ii). □

5. Characterizing Quasiprobability Representations

In Section 4, we have associated a notion of conditional expectation to every quasiprobability representation $(Q,\tilde{Q})$ that is Born-compatible with a CSCO $\widehat{Y}$. In this section, we consider all quasiprobability representations that are compatible with two given complementary CSCO $\widehat{A}$ and $\widehat{B}$ (see Definition 5 for the notion of complementarity of CSCO).

In Section 5.1, we recall the definition of the Kirkwood–Dirac representations Born-compatible with $\widehat{A}$ and $\widehat{B}$. In Section 5.2, we show that those KD representations are the only quasiprobability representations for which the associated conditional expectation (given $\widehat{A}$ or given $\widehat{B}$) satisfies the “pull-out formula” (see Theorem 7). This structural result provides a unique characterization of the KD quasiprobability representations that allows us to show Theorem 1, which states that the (left/right) Kirkwood–Dirac representation is the only quasiprobability representation for which the associated conditional expectations (given $\widehat{A}$ or given $\widehat{B}$) agree with the (left/right) conditional expectations defined as best estimators in Section 3.

In Section 5.3, we show that all $\widehat{A}$ and $\widehat{B}$ Born-compatible quasiprobability representations with ontic space $\Lambda =\sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$ are completely determined by their associated conditional expectations (Theorem 9). This result provides a strengthening of Theorem 1 and allows for some further applications that we discuss.

Section 5.2 and Section 5.3 provide independent proofs of Theorem 1 and can be read in any order.

5.1. The Kirkwood–Dirac Distribution

The definition of a KD quasiprobability representation of quantum mechanics depends on the choice of two bases in the Hilbert space [4,7,8,23,24,31]. Let $\widehat{A}=({\widehat{A}}_1,\dots ,{\widehat{A}}_n)$ and $\widehat{B}=({\widehat{B}}_1,\dots ,{\widehat{B}}_m)$ be two CSCO. We write ${\left({\phi }_a^{\widehat{A}}\right)}_{a\in \sigma \left(\widehat{A}\right)},{\left({\phi }_b^{\widehat{B}}\right)}_{b\in \sigma \left(\widehat{B}\right)}$ for the corresponding eigenbases:

$${\widehat{A}}_i{\phi }_a^{\widehat{A}}=a_i{\phi }_a^{\widehat{A}},{\widehat{B}}_j{\phi }_b^{\widehat{B}}=b_j{\phi }_b^{\widehat{B}}.$$

(128)

We use $d_i^{\widehat{A}}$ and $d_j^{\widehat{B}}$ to denote the number of eigenvalues of ${\widehat{A}}_i$ and ${\widehat{B}}_j$ for each $i\in 〚1,n〛$, respectively, $j\in 〚1,m〛$. For any $p=(p_1,\dots ,p_n)\in {\mathbb{N}}^n$, we write

$${\widehat{A}}^p:={\widehat{A}}_1^{p_1}\dots {\widehat{A}}_n^{p_n},$$

(129)

and for any $a=(a_1,\dots ,a_n)\in \sigma \left(\widehat{A}\right)$, we write

$$a^p:=a_1^{p_1}\dots a_n^{p_n}.$$

(130)

We define ${\Gamma }^{\widehat{A}}$ as

$${\Gamma }^{\widehat{A}}=〚0,d_1^{\widehat{A}}-1〛\times \dots \times 〚0,d_n^{\widehat{A}}-1〛,$$

(131)

and ${\Gamma }^{\widehat{B}}$ is obtained analogously.

We consider, as in Equation (42), the maximal commutative algebras

$${\mathcal{F}}_{\mathbb{C},\widehat{A}}=\left\{f\left(\widehat{A}\right)\right|f:\sigma \left(\widehat{A}\right)\to \mathbb{C}\},{\mathcal{F}}_{\mathbb{C},\widehat{B}}=\left\{f\left(\widehat{B}\right)\right|f:\sigma \left(\widehat{B}\right)\to \mathbb{C}\}.$$

(132)

Several times, we will use the fact that, as we are working on a finite dimensional Hilbert space, any $f\left(\widehat{A}\right)\in {\mathcal{F}}_{\mathbb{C},\widehat{A}}$ can be written as a polynomial $f\left(\widehat{A}\right)={\sum }_{p\in {\Gamma }^{\widehat{A}}}{c}_p{\widehat{A}}^p$, where ${\left(c_p\right)}_{p\in {\Gamma }^{\widehat{A}}}\subset \mathbb{C}$. This is a direct consequence of the Lagrange interpolation. Of course, a similar statement holds for the elements of ${\mathcal{F}}_{\mathbb{C},\widehat{B}}$.

We have the following lemma:

Lemma 5.

Let $\widehat{A},\widehat{B}$ be two $\mathrm{CSCO}$, as above. Consider the statements

(i) 

$\forall (a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right),\langle {\phi }_a^{\widehat{A}},{\phi }_b^{\widehat{B}}\rangle \ne 0;$

(ii) 

${\mathrm{span}}_{\mathbb{C}}{\mathcal{F}}_{\mathbb{C},\widehat{A}}{\mathcal{F}}_{\mathbb{C},\widehat{B}}=\mathcal{L}\left(\mathcal{H}\right)={\mathrm{span}}_{\mathbb{C}}{\mathcal{F}}_{\mathbb{C},\widehat{B}}{\mathcal{F}}_{\mathbb{C},\widehat{A}}$;

(iii) 

${\mathcal{F}}_{\mathbb{C},\widehat{A}}\cap {\mathcal{F}}_{\mathbb{C},\widehat{B}}=\mathbb{C}{\mathbb{I}}_d.$

Then, $\left(i\right)\iff \left(ii\right)\Rightarrow \left(iii\right)$.

Proof. 

That $\left(i\right)$ implies $\left(ii\right)$ follows from the fact that ${\widehat{\Pi }}_a^{\widehat{A}}{\widehat{\Pi }}_b^{\widehat{B}}$, with $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$ form a basis of $\mathcal{L}\left(\mathcal{H}\right)$ and from the observation that ${\widehat{\Pi }}_a^{\widehat{A}}\in {\mathcal{F}}_{\mathbb{C},\widehat{A}}$, ${\widehat{\Pi }}_b^{\widehat{B}}\in {\mathcal{F}}_{\mathbb{C},\widehat{B}}$.

To see that $\left(ii\right)$ implies $\left(i\right)$, let us note that

$$f\left(\widehat{A}\right)g\left(\widehat{B}\right)=\sum _{(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)}f\left(a\right)g\left(b\right){\widehat{\Pi }}_a^{\widehat{A}}{\widehat{\Pi }}_b^{\widehat{B}}.$$

(133)

Hence,

$${\mathrm{span}}_{\mathbb{C}}{\mathcal{F}}_{\mathbb{C},\widehat{A}}{\mathcal{F}}_{\mathbb{C},\widehat{B}}={\mathrm{span}}_{\mathbb{C}}\{{\widehat{\Pi }}_a^{\widehat{A}}{\widehat{\Pi }}_b^{\widehat{B}}\mid \langle {\phi }_a^{\widehat{A}},{\phi }_b^{\widehat{B}}\rangle \ne 0\},$$

(134)

which implies the result.

We now show that $\left(i\right)$ implies $\left(iii\right)$. Let $f\left(\widehat{A}\right)\in {\mathcal{F}}_{\mathbb{C},\widehat{A}}$. We need to show that, if $f\left(\widehat{A}\right)\in {\mathcal{F}}_{\mathbb{C},\widehat{B}}$, then it is a multiple of the identity. But, since ${\mathcal{F}}_{\mathbb{C},\widehat{B}}={\left({\mathcal{F}}_{\mathbb{C},\widehat{B}}\right)}^{\prime }$, this means we need to show that if $f\left(\widehat{A}\right)\in {\left({\mathcal{F}}_{\mathbb{C},\widehat{B}}\right)}^{\prime }$, then it is a multiple of the identity. But, $f\left(\widehat{A}\right)\in {\left({\mathcal{F}}_{\mathbb{C},\widehat{B}}\right)}^{\prime }$ is equivalent to $[f\left(\widehat{A}\right),{\widehat{\Pi }}_b^{\widehat{B}}]=0$, for all $b\in \sigma \left(\widehat{B}\right)$, which is equivalent to

$$\forall a,a^{\prime }\in \sigma \left(\widehat{A}\right),\forall b\in \sigma \left(\widehat{B}\right),0=\langle {\phi }_a^{\widehat{A}},[f\left(\widehat{A}\right),{\widehat{\Pi }}_b^{\widehat{B}}]{\phi }_{a^{\prime }}^{\widehat{A}}\rangle =(f\left(a\right)-f\left(a^{\prime }\right))\langle {\phi }_a^{\widehat{A}},{\phi }_b^{\widehat{B}}\rangle \langle {\phi }_b^{\widehat{B}},{\phi }_{a^{\prime }}^{\widehat{A}}\rangle .$$

(135)

This shows that $f\left(\widehat{A}\right)=c{\mathbb{I}}_d$ for some $c\in \mathbb{C}$. □

We point out that it is easy to construct two bases in dimension 3 for which condition $\left(iii\right)$ of Lemma 5 is fulfilled but condition $\left(i\right)$ fails. This shows that those conditions are not equivalent.

Definition 5.

We say two complete sets of commuting observables $\widehat{A}, \widehat{B}$ are complementary if they satisfy any one of conditions (i) and (ii) in Lemma 5.

The term “complementary” is used for a variety of different notions in the literature [39,55,56,57]. In the context of finite dimensional Hilbert space, it is sometimes used as synonymous to “mutually unbiased”, in other words to $\left|\langle a|b\rangle \right|=d^{-1/2}$, for all $a\in \sigma \left(\widehat{A}\right),b\in \sigma \left(\widehat{B}\right)$. Our definition here is therefore a relaxation of the latter definition. We refer to [57] for the link of this definition to various notions of incompatibility.

We now introduce the left and right Kirkwood–Dirac distributions associated with two CSCO $\widehat{A}$ and $\widehat{B}$, assumed to be complementary. Two natural choices of Born-compatible frames for $\widehat{A}$ and $\widehat{B}$ are

$$S_{a,b}^{\ell }={\widehat{\Pi }}_a^{\widehat{A}}{\widehat{\Pi }}_b^{\widehat{B}}\mathrm{and}{S}_{a,b}^{\mathrm{r}}={\widehat{\Pi }}_b^{\widehat{B}}{\widehat{\Pi }}_a^{\widehat{A}}.$$

It is then easily shown that the dual frames are

$$T_{a,b}^{\ell }={\displaystyle \frac{1}{|\langle {\phi }_a^{\widehat{A}},{\phi }_b^{\widehat{B}}\rangle {|}^2}}{\widehat{\Pi }}_a^{\widehat{A}}{\widehat{\Pi }}_b^{\widehat{B}}\mathrm{and}{T}_{a,b}^{\mathrm{r}}={\displaystyle \frac{1}{|\langle {\phi }_a^{\widehat{A}},{\phi }_b^{\widehat{B}}\rangle {|}^2}}{\widehat{\Pi }}_b^{\widehat{B}}{\widehat{\Pi }}_a^{\widehat{A}}.$$

The associated quasiprobability representations are the left and right KD representations

$$\begin{array}{cc}\hfill & {\mathrm{Q}}_{a,b}^{\mathrm{KD},\ell }\left(\widehat{\rho }\right)=\langle {\phi }_b^{\widehat{B}},{\phi }_a^{\widehat{A}}\rangle \langle {\phi }_a^{\widehat{A}},\widehat{\rho }{\phi }_b^{\widehat{B}}\rangle ,{\tilde{\mathrm{Q}}}_{a,b}^{\mathrm{KD},\ell }\left(\widehat{X}\right)={\displaystyle \frac{1}{\langle {\phi }_a^{\widehat{A}},{\phi }_b^{\widehat{B}}\rangle }}\langle {\phi }_a^{\widehat{A}},\widehat{X}{\phi }_b^{\widehat{B}}\rangle \hfill \\ \hfill & {\mathrm{Q}}_{a,b}^{\mathrm{KD},\mathrm{r}}\left(\widehat{\rho }\right)=\langle {\phi }_a^{\widehat{A}},{\phi }_b^{\widehat{B}}\rangle \langle {\phi }_b^{\widehat{B}},\widehat{\rho }{\phi }_a^{\widehat{A}}\rangle ,{\tilde{\mathrm{Q}}}_{a,b}^{\mathrm{KD},\mathrm{r}}\left(\widehat{X}\right)={\displaystyle \frac{1}{\langle {\phi }_b^{\widehat{B}},{\phi }_a^{\widehat{A}}\rangle }}\langle {\phi }_b^{\widehat{B}},\widehat{X}{\phi }_a^{\widehat{A}}\rangle .\hfill \end{array}$$

For later reference, we note that, for all $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$, one has that

$$\overline{{\tilde{Q}}_{a,b}^{\mathrm{KD},\ell }\left(\widehat{X}\right)}={\tilde{Q}}_{a,b}^{\mathrm{KD},\mathrm{r}}\left({\widehat{X}}^{†}\right)$$

(136)

They are called the left and right Kirkwood–Dirac representations because of the following fact. For $f:\sigma \left(\widehat{A}\right)\to \mathbb{C}$ and $g:\sigma \left(\widehat{B}\right)\to \mathbb{C}$, one gets

$${\tilde{\mathrm{Q}}}_{a,b}^{\mathrm{KD},\ell }\left(f\left(\widehat{A}\right)g\left(\widehat{B}\right)\right)=f\left(a\right)g\left(b\right)$$

(137)

so the symbol is well-behaved when $f\left(\widehat{A}\right)$ is “to the left”, and

$${\tilde{\mathrm{Q}}}_{a,b}^{\mathrm{KD},\mathrm{r}}\left(g\left(\widehat{B}\right)f\left(\widehat{A}\right)\right)=g\left(b\right)f\left(a\right),$$

(138)

the symbol is well-behaved when $f\left(\widehat{A}\right)$ is “to the right”.

We now analyze the right and left KD-conditional expectations of arbitrary observables, given $\widehat{A}$ or $\widehat{B}$.

Lemma 6.

Given two complementary $\mathrm{CSCO}$$\widehat{A}$ and $\widehat{B}$, we have that, for every $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$,

$${\mathbb{E}}_{\widehat{\rho }}^{{\mathrm{Q}}^{\mathrm{KD},\ell }}\left(\widehat{X}\right|\widehat{B})={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{B}),{\mathbb{E}}_{\widehat{\rho }}^{{\mathrm{Q}}^{\mathrm{KD},\mathrm{r}}}\left(\widehat{X}\right|\widehat{B})={\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{B}).$$

(139)

and

$${\mathbb{E}}_{\widehat{\rho }}^{{\mathrm{Q}}^{\mathrm{KD},\ell }}\left(\widehat{X}\right|\widehat{A})={\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{A}),{\mathbb{E}}_{\widehat{\rho }}^{{\mathrm{Q}}^{\mathrm{KD},\mathrm{r}}}\left(\widehat{X}\right|\widehat{A})={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{A}).$$

(140)

Proof. 

Following Definition 4, one computes, for all $\widehat{\rho }\in D_{\widehat{B}}$ and any $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$,

$$\begin{array}{cc}\hfill {\mathbb{E}}_{\widehat{\rho }}^{{\mathrm{Q}}^{\mathrm{KD},\ell }}\left(\widehat{X}\right|\widehat{B})& =\sum _{(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)}\overline{{\displaystyle \frac{1}{\langle {\phi }_a^{\widehat{A}},{\phi }_b^{\widehat{B}}\rangle }}\langle {\phi }_a^{\widehat{A}},{\widehat{X}}^{†}{\phi }_b^{\widehat{B}}\rangle }{\displaystyle \frac{\langle {\phi }_b^{\widehat{B}},{\phi }_a^{\widehat{A}}\rangle \langle {\phi }_a^{\widehat{A}},\widehat{\rho }{\phi }_b^{\widehat{B}}\rangle }{\langle {\phi }_b^{\widehat{B}},\widehat{\rho }{\phi }_b^{\widehat{B}}\rangle }}{\widehat{\Pi }}_b^{\widehat{B}}\hfill \\ \hfill & =\sum _{(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)}{\displaystyle \frac{\langle {\phi }_b^{\widehat{B}},\widehat{X}{\phi }_a^{\widehat{A}}\rangle \langle {\phi }_a^{\widehat{A}},\widehat{\rho }{\phi }_b^{\widehat{B}}\rangle }{\langle {\phi }_b^{\widehat{B}},\widehat{\rho }{\phi }_b^{\widehat{B}}\rangle }}{\widehat{\Pi }}_b^{\widehat{B}}\hfill \\ \hfill & =\sum _{b\in \sigma \left(\widehat{B}\right)}{\displaystyle \frac{\langle {\phi }_b^{\widehat{B}},\widehat{X}\widehat{\rho }{\phi }_b^{\widehat{B}}\rangle }{\langle {\phi }_b^{\widehat{B}},\widehat{\rho }{\phi }_b^{\widehat{B}}\rangle }}{\widehat{\Pi }}_b^{\widehat{B}}.\hfill \end{array}$$

One recognizes the left conditional expectation given in Definition 2, proving the first equality in Equation (139). The other three equalities follow from a similar computation. □

In conclusion, both the left and right KD representations are not only $\widehat{A}$ and $\widehat{B}$ compatible, but in addition, the associated conditional expectations given either $\widehat{A}$ or $\widehat{B}$ both can be interpreted as best estimators with respect to either the right or left sesquilinear forms introduced in Section 3. In particular, their associated conditional expectation satisfy the pull-out formula, see the first point in Theorem 5.

5.2. Characterization of the Kirkwood–Dirac Quasiprobability Representations via the Pull-Out Property

We shall prove in this section that the KD representations are the only $\widehat{A}$ and $\widehat{B}$ Born-compatible quasiprobability representations that give rise to Q-conditional expectations satisfying the pull-out property: this is the content of Theorem 7.

The main technical result we need is contained in the following proposition, which is formulated for the left KD representation. An analogous statement holds for the right KD representation.

Proposition 4.

Let $\mathcal{H}$ be a d dimensional Hilbert space and let $(Q,\tilde{Q})$ be a quasiprobability representation of quantum mechanics on $\mathcal{H}$ over a space Λ with $\left|\Lambda \right|=d^2$. Suppose that there exists a $\mathrm{CSCO}$ $\widehat{B}$ such that

(i) 

$(Q,\tilde{Q})$ is $\widehat{B}$-compatible;

(ii) 

$\forall \widehat{X}\in \mathcal{L}\left(\mathcal{H}\right),\forall f\left(\widehat{B}\right)\in {\mathcal{F}}_{\mathbb{C},\widehat{B}},\forall \widehat{\rho }\in D_{\widehat{B}},{\mathbb{E}}_{\widehat{\rho }}^Q\left(f\left(\widehat{B}\right)\widehat{X}\right|\widehat{B})=f\left(\widehat{B}\right){\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{B})$;

Then, the following holds. If there exists a second $\mathrm{CSCO}$ $\widehat{A}$, complementary to $\widehat{B}$, and for which $(Q,\tilde{Q})$ is $\widehat{A}$-compatible, then there exists a bijective map $\Phi :\Lambda \to \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$ such that for all $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$

$$\forall \widehat{\rho }\in D\left(\mathcal{H}\right),Q_{{\Phi }^{-1}(a,b)}\left(\widehat{\rho }\right)=Q_{a,b}^{\mathrm{KD},\ell }\left(\widehat{\rho }\right)\mathrm{and}\forall \widehat{X}\in \mathcal{L}\left(\mathcal{H}\right),{\tilde{Q}}_{{\Phi }^{-1}(a,b)}\left(\widehat{X}\right)={\tilde{Q}}_{a,b}^{\mathrm{KD},\ell }\left(\widehat{X}\right).$$

We illustrate this proposition in Appendix C where we compare the conditional expectations arising from the Gross-Wigner and the KD quasiprobability representations for a qutrit system and show that they are different. We show in particular that the conditional expectation associated to the Gross–Wigner representation does not satisfy the pull-out formula.

The pull-out formula ($\left(ii\right)$ above) is an intrinsic condition on the Q-conditional expectation that can be weakened in the presence of two complementary CSCO, as shown in the following lemma:

Lemma 7.

Suppose that $\widehat{A}$ and $\widehat{B}$ are complementary CSCO on a Hilbert space $\mathcal{H}$ and that $(Q,\tilde{Q})$ is a $\widehat{B}$-compatible quasiprobability representation of quantum mechanics on $\mathcal{H}$. Then, the following statements are equivalent for all $\widehat{\rho }\in D_{\widehat{B}}$:

(i) 

$\forall p\in {\Gamma }^{\widehat{A}},\forall q\in {\Gamma }^{\widehat{B}},{\mathbb{E}}_{\widehat{\rho }}^Q\left({\widehat{B}}^q{\widehat{A}}^p\right|\widehat{B})={\widehat{B}}^q{\mathbb{E}}_{\widehat{\rho }}^Q\left({\widehat{A}}^p\right|\widehat{B})$;

(ii) 

$\forall \widehat{X}\in \mathcal{L}\left(\mathcal{H}\right),\forall f\left(\widehat{B}\right)\in {\mathcal{F}}_{\mathbb{C},\widehat{B}},{\mathbb{E}}_{\widehat{\rho }}^Q\left(f\left(\widehat{B}\right)\widehat{X}\right|\widehat{B})=f\left(\widehat{B}\right){\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{B})$;

Proof. 

We only need to prove that (i) implies (ii). By Lemma 5, one can assume that $\widehat{X}=g\left(\widehat{B}\right)h\left(\widehat{A}\right)$. By Lagrange interpolation and linearity, one can therefore assume $\widehat{X}=g\left(\widehat{B}\right){\widehat{A}}^p$, for $p\in {\Gamma }^{\widehat{A}}$. By Lagrange interpolation again, one has that $f\left(\widehat{B}\right)g\left(\widehat{B}\right)={\sum }_{q\in {\Gamma }^{\widehat{B}}}{c}_q{\widehat{B}}^q$. One has

$$\begin{array}{cc}\hfill {\mathbb{E}}_{\widehat{\rho }}^Q\left(f\left(\widehat{B}\right)\widehat{X}\right|\widehat{B})& ={\mathbb{E}}_{\widehat{\rho }}^Q\left(\sum _{q\in {\Gamma }^{\widehat{B}}}{c}_q{\widehat{B}}^q{A}^p|\widehat{B}\right)=\sum _{q\in {\Gamma }^{\widehat{B}}}{c}_q{\widehat{B}}^q{\mathbb{E}}_{\widehat{\rho }}^Q\left({\widehat{A}}^p\right|\widehat{B})\hfill \\ \hfill & =f\left(\widehat{B}\right)g\left(\widehat{B}\right){\mathbb{E}}_{\rho }\left({\widehat{A}}^p\right|\widehat{B})=f\left(\widehat{B}\right){\mathbb{E}}_{\rho }^Q\left(g\left(\widehat{B}\right){\widehat{A}}^p\right|\widehat{B})\hfill \\ \hfill & =f\left(\widehat{B}\right){\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{B}),\hfill \end{array}$$

where we used (i) twice and the fact that $g\left(\widehat{B}\right)\in {\mathcal{F}}_{\mathbb{C},\widehat{B}}$ can also be expressed as a polynomial. This proves the result. □

As a direct consequence of Proposition 4 and Lemma 7, we get the following characterization of the KD distribution, in terms of the pull-out formula with respect to powers of the CSCO $\widehat{A}$ and $\widehat{B}$:

Theorem 7.

Let $\mathcal{H}$ be a d dimensional Hilbert space and let $(Q,\tilde{Q})$ be a quasi-probability representation of quantum mechanics on $\mathcal{H}$ over a space Λ with $\left|\Lambda \right|=d^2$. Suppose that there exist $\widehat{A}$ and $\widehat{B}$ two complementary CSCO such that

(i) 

$(Q,\tilde{Q})$ is $\widehat{A}$ and $\widehat{B}$-compatible;

(ii) 

$\forall p\in {\Gamma }^{\widehat{A}},\forall q\in {\Gamma }^{\widehat{B}},\forall \widehat{\rho }\in D_{\widehat{B}},{\mathbb{E}}_{\widehat{\rho }}^Q\left({\widehat{B}}^q{\widehat{A}}^p\right|\widehat{B})={\widehat{B}}^q{\mathbb{E}}_{\widehat{\rho }}^Q\left({\widehat{A}}^p\right|\widehat{B})$;

Then, there exists a bijective map $\Phi :\Lambda \to \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$ such that for all $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$

$$\forall \widehat{\rho }\in D\left(\mathcal{H}\right),Q_{{\Phi }^{-1}(a,b)}\left(\widehat{\rho }\right)=Q_{a,b}^{\mathrm{KD},\ell }\left(\widehat{\rho }\right)\mathrm{and}\forall \widehat{X}\in \mathcal{L}\left(\mathcal{H}\right),{\tilde{Q}}_{{\Phi }^{-1}(a,b)}\left(\widehat{X}\right)={\tilde{Q}}_{a,b}^{\mathrm{KD},\ell }\left(\widehat{X}\right).$$

We shall now prove Proposition 4.

Proof of Proposition 4. 

As in the proof of Theorem 6, conditions (i) and (ii) of the proposition imply that, for all $\lambda \in \Lambda$ and for all $b\in \sigma \left(\widehat{B}\right)$, $S_{\lambda }=S_{\lambda }{\widehat{\Pi }}_{B\left(\lambda \right)}^{\widehat{B}}$, where we wrote $B\left(\lambda \right)={\tilde{Q}}_{\lambda }\left(\widehat{B}\right)$. Consequently, $S_{\lambda }{\widehat{\Pi }}_{b^{\prime }}^{\widehat{B}}=S_{\lambda }{\widehat{\Pi }}_{B\left(\lambda \right)}^{\widehat{B}}{\widehat{\Pi }}_{b^{\prime }}^{\widehat{B}}=0$, when $b^{\prime }\ne B\left(\lambda \right)$. Since $(Q,\tilde{Q})$ is both $\widehat{B}$ and $\widehat{A}$ compatible, and since $\widehat{A}$ and $\widehat{B}$ are complementary and complete, one concludes, via Theorem 6, for all $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$, that

$${\widehat{\Pi }}_b^{\widehat{B}}{\widehat{\Pi }}_a^{\widehat{A}}=\sum _{\lambda \in {\Lambda }_b^{\widehat{B}}}\sum _{{\lambda }^{\prime }\in {\Lambda }_a^{\widehat{A}}}{S}_{\lambda }{S}_{{\lambda }^{\prime }}^{†}=\sum _{\lambda \in {\Lambda }_b^{\widehat{B}}}\sum _{{\lambda }^{\prime }\in {\Lambda }_a^{\widehat{A}}}{S}_{\lambda }{\widehat{\Pi }}_b^{\widehat{B}}{S}_{{\lambda }^{\prime }}^{†}=\sum _{\lambda \in {\Lambda }_b^{\widehat{B}}}\sum _{{\lambda }^{\prime }\in {\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}}{S}_{\lambda }{\widehat{\Pi }}_b^{\widehat{B}}{S}_{{\lambda }^{\prime }}^{†}=\sum _{\lambda \in {\Lambda }_b^{\widehat{B}}}\sum _{{\lambda }^{\prime }\in {\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}}{S}_{\lambda }{S}_{{\lambda }^{\prime }}^{†}.$$

(141)

Since we assume that $\widehat{A}$ and $\widehat{B}$ are complementary, the left-hand side of this equality does not vanish for any $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$. Hence, there exists ${\lambda }^{\prime }\in {\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}$ such that ${\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}\ne \varnothing$. Consequently, writing $A\left(\lambda \right)={\tilde{Q}}_{\lambda }\left(\widehat{A}\right)$ as above, the map

$$\Phi :=(A,B):\lambda \in \Lambda \mapsto (A\left(\lambda \right),B\left(\lambda \right))\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$$

(142)

is surjective and since $\left|\Lambda \right|=d^2=\left|\sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)\right|$, it is a bijection. It follows that

$${\Lambda }_a^{\widehat{A}}=\bigcup _{b\in 〚1,d〛}{\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}},\left|{\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}\right|=1,\left|{\Lambda }_a^{\widehat{A}}\right|=d.$$

(143)

We can therefore identify $\Lambda$ with $\sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$, by considering $\left(A\right(\lambda ),B(\lambda \left)\right)$ as coordinates on $\Lambda$. We can now conclude as follows. From Equation (141), we find

$${\widehat{\Pi }}_b^{\widehat{B}}{\widehat{\Pi }}_a^{\widehat{A}}={\widehat{\Pi }}_b^{\widehat{B}}\left(\sum _{{\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}}{S}_{{\lambda }^{\prime }}^{†}\right)={\widehat{\Pi }}_b^{\widehat{B}}{S}_{\lambda (a,b)}^{†},$$

for the unique $\lambda (a,b)\in {\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}$. Since $\lambda (a,b)\in {\Lambda }_b^{\widehat{B}}$, Theorem 6 (iii) implies that $S_{\lambda (a,b)}=S_{\lambda (a,b)}{\widehat{\Pi }}_b^{\widehat{B}}$. This in turn implies that $S_{\lambda }={\widehat{\Pi }}_{A\left(\lambda \right)}^{\widehat{A}}{\widehat{\Pi }}_{B\left(\lambda \right)}^{\widehat{B}}$.

This concludes the proof with $\Phi \left(\lambda \right)=\left(A\right(\lambda ),B(\lambda \left)\right)$. □

This first characterization of the left KD distribution relies on an intrisic property of its quantum conditional expectation: among the $\widehat{A}$ and $\widehat{B}$ compatible quasiprobability representations of quantum mechanics, the only one for which the associated quantum conditional expectation satisfies a left pull-out property is the left Kirkwood–Dirac distribution.

We now prove Theorem 1: Lemma 6 proves that $\left(iii\right)$ implies $\left(i\right)$ and $\left(iii\right)$ implies $\left(ii\right)$, after direct computations.

We prove that $\left(i\right)$ implies $\left(iii\right)$: from Theorem 6, we know that $\left(i\right)$ of Theorem 1 implies that

$$\forall \widehat{X}\in \mathcal{L}\left(\mathcal{H}\right),\forall f\left(\widehat{B}\right)\in {\mathcal{F}}_{\mathbb{C},\widehat{B}},{\mathbb{E}}_{\widehat{\rho }}^Q\left(f\left(\widehat{B}\right)\widehat{X}\right|\widehat{B})=f\left(\widehat{B}\right){\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{B}).$$

Thus, as $(Q,\tilde{Q})$ satisfies a left pull out property (given just above) and is $\widehat{A}$ and $\widehat{B}$ compatible, $(Q,\tilde{Q})$ satisfies the hypothesis of Proposition 4. This proves $\left(iii\right)$. The fact that $\left(ii\right)$ implies $\left(iii\right)$ also follows from the same reasoning.

5.3. Characterization of Quasiprobability Representations via Their Quantum Conditional Expectations

In this subsection, we show that $\widehat{A}$ and $\widehat{B}$ Born-compatible quasiprobability representations are uniquely determined by their associated conditional expectations. To do so, we combine the framework we developed in Section 4 and Section 5.1 and Section 5.2 with an idea first developed in [41].

Theorem 8.

Let $\widehat{A}$ and $\widehat{B}$ be two complementary CSCO. Let $(R,\tilde{R})$ be a $\widehat{A}$ and $\widehat{B}$-compatible quasiprobability representation of quantum mechanics on $\sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$. Let $(Q,\tilde{Q})$ be a $\widehat{A}$ and $\widehat{B}$-compatible quasiprobability representation of quantum mechanics on Λ, with $\left|\Lambda \right|=d^2$. Then, the following statements are equivalent:

(i) 

for all $\widehat{\rho }\in D_{\widehat{B}}$, for all $p\in {\Gamma }^{\widehat{A}}$,

$${\mathbb{E}}_{\widehat{\rho }}^Q\left({\widehat{A}}^p\right|\widehat{B})={\mathbb{E}}_{\widehat{\rho }}^R\left({\widehat{A}}^p\right|\widehat{B});$$

(ii) 

for all $\widehat{\rho }\in D_{\widehat{B}}$, for all $f\left(\widehat{A}\right)\in {\mathcal{F}}_{\mathbb{C},\widehat{A}}$, ${\mathbb{E}}_{\widehat{\rho }}^Q\left(f\left(\widehat{A}\right)\right|\widehat{B})={\mathbb{E}}_{\widehat{\rho }}^R\left(f\left(\widehat{A}\right)\right|\widehat{B})$;

(iii) 

for all $\widehat{\rho }\in D_{\widehat{B}}$, for all $(p,q)\in {\Gamma }^{\widehat{A}}\times {\Gamma }^{\widehat{B}}$,

$${\mathbb{E}}_{\widehat{\rho }}^Q\left({\widehat{A}}^p{\widehat{B}}^q\right|\widehat{B})={\mathbb{E}}_{\widehat{\rho }}^R\left({\widehat{A}}^p{\widehat{B}}^q\right|\widehat{B}).$$

(iv) 

for all $\widehat{\rho }\in D_{\widehat{B}}$, for all $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$,

$${\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{B})={\mathbb{E}}_{\widehat{\rho }}^R\left(\widehat{X}\right|\widehat{B}).$$

(v) 

there exists a bijective map $\Phi :\Lambda \to \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$ such that

$$\forall (a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right),R_{a,b}=Q_{{\Phi }^{-1}(a,b)};$$

Proof. 

Let us prove first that $\left(v\right)\Rightarrow \left(iv\right)$. Let $S^R$ and $S^Q$ be the respective frames associated to $(R,\tilde{R})$ and $(Q,\tilde{Q})$. The fact that $R_{a,b}=Q_{{\Phi }^{-1}(a,b)}$ for all $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$ implies that $S_{a,b}^R=S_{{\Phi }^{-1}(a,b)}^Q$ for all $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$. Moreover, from Equation (98), the symbols also satisfy ${\tilde{R}}_{a,b}={\tilde{Q}}_{{\Phi }^{-1}(a,b)}$ for all $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$. Recall from Lemma 4 that the fact that $(Q,\tilde{Q})$ is $\widehat{B}$-Born compatible implies that ${\left({\Lambda }_b^{\widehat{B}}\right)}_{b\in \sigma \left(\widehat{B}\right)}$ is a partition of $\Lambda$. In particular, we can define the projection ${\pi }_0:\Lambda \to \sigma \left(\widehat{B}\right)$ which verifies that for all $\lambda \in \Lambda$, $\lambda \in {\Lambda }_{{\pi }_0\left(\lambda \right)}^{\widehat{B}}$. For $b\in \sigma \left(\widehat{B}\right)$, we have, from $\widehat{B}$-Born compatibility of $(R,\tilde{R})$,

$$\begin{array}{cc}\hfill {\widehat{\Pi }}_b^{\widehat{B}}& =\sum _{a\in \sigma \left(\widehat{A}\right)}{S}_{a,b}^R=\sum _{a\in \sigma \left(\widehat{A}\right)}{S}_{{\Phi }^{-1}(a,b)}^Q=\sum _{\lambda \in {\Phi }^{-1}(\sigma \left(\widehat{A}\right)\times \left\{b\right\})}{S}_{\lambda }^Q.\hfill \end{array}$$

Moreover, from $\widehat{B}$-compatibility of $(Q,\tilde{Q})$,

$${\widehat{\Pi }}_b^{\widehat{B}}=\sum _{\lambda \in {\Lambda }_b^{\widehat{B}}}{S}_{\lambda }^Q.$$

Taking the Hilbert–Schmidt inner product on these two expressions with the dual frame $T^Q$ yields that $\Phi$ is, for any $b\in \sigma \left(\widehat{B}\right)$, a bijection between ${\Lambda }_b^{\widehat{B}}$ and $\sigma \left(\widehat{A}\right)\times \left\{b\right\}$. In particular, we have that ${\pi }_0\left({\Phi }^{-1}(a,b)\right)=b$ for all $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$. Let $\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$ and $\widehat{\rho }\in D_{\widehat{B}}$. One has

$$\begin{array}{cc}\hfill {\mathbb{E}}_{\widehat{\rho }}^R\left(\widehat{X}\right|\widehat{B})& =\sum _{(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)}\overline{{\tilde{R}}_{a,b}\left({\widehat{X}}^{†}\right)}{\displaystyle \frac{R_{a,b}\left(\widehat{\rho }\right)}{\langle {\phi }_b^{\widehat{B}},\widehat{\rho }{\phi }_b^{\widehat{B}}\rangle }}{\widehat{\Pi }}_b^{\widehat{B}}\hfill \\ \hfill & =\sum _{(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)}\overline{{\tilde{Q}}_{{\Phi }^{-1}(a,b)}\left({\widehat{X}}^{†}\right)}{\displaystyle \frac{Q_{{\Phi }^{-1}(a,b)}\left(\widehat{\rho }\right)}{\langle {\phi }_{{\pi }_0\left({\Phi }^{-1}(a,b)\right)}^{\widehat{B}},\widehat{\rho }{\phi }_{{\pi }_0\left({\Phi }^{-1}(a,b)\right)}^{\widehat{B}}\rangle }}{\widehat{\Pi }}_{{\pi }_0\left({\Phi }^{-1}(a,b)\right)}^{\widehat{B}}\hfill \\ \hfill & =\sum _{\lambda \in \Lambda }\overline{{\tilde{Q}}_{\lambda }\left({\widehat{X}}^{†}\right)}{\displaystyle \frac{Q_{\lambda }\left(\widehat{\rho }\right)}{\langle {\phi }_{{\pi }_0\left(\lambda \right)}^{\widehat{B}},\widehat{\rho }{\phi }_{{\pi }_0\left(\lambda \right)}^{\widehat{B}}\rangle }}{\widehat{\Pi }}_{{\pi }_0\left(\lambda \right)}^{\widehat{B}}\hfill \\ \hfill & =\sum _{b\in \sigma \left(\widehat{B}\right)}\sum _{\lambda \in {\Lambda }_b^{\widehat{B}}}\overline{{\tilde{Q}}_{\lambda }\left({\widehat{X}}^{†}\right)}{\displaystyle \frac{Q_{\lambda }\left(\widehat{\rho }\right)}{\langle {\phi }_b^{\widehat{B}},\widehat{\rho }{\phi }_b^{\widehat{B}}\rangle }}{\widehat{\Pi }}_b^{\widehat{B}}\hfill \\ \hfill & ={\mathbb{E}}_{\widehat{\rho }}^Q\left(\widehat{X}\right|\widehat{B}),\hfill \end{array}$$

proving $\left(iv\right)$.

It is clear that $\left(iv\right)\Rightarrow \left(iii\right)\Rightarrow \left(i\right)$. Moreover, as already mentioned, any $f\left(\widehat{A}\right)\in {\mathcal{F}}_{\mathbb{C},\widehat{A}}$ can be written as a polynomial $P_f\left(\widehat{A}\right)$ and thus, $\left(i\right)\iff \left(ii\right)$ by linearity. We now prove that $\left(i\right)\Rightarrow \left(v\right)$ to conclude the proof. We compute, for all $\widehat{\rho }\in D_{\widehat{B}}$, for all $p\in {\Gamma }^{\widehat{A}}$:

$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\widehat{\rho }}^Q\left({\widehat{A}}^p\right|\widehat{B})& =& \sum _{b\in \sigma \left(\widehat{B}\right)}\sum _{\lambda \in {\Lambda }_b^{\widehat{B}}}\overline{{\tilde{Q}}_{\lambda }\left({\left({\widehat{A}}^p\right)}^{†}\right)}{Q}_{\lambda |b}\left(\widehat{\rho }\right){\widehat{\Pi }}_b^{\widehat{B}}=\sum _{b\in \sigma \left(\widehat{B}\right)}\sum _{\lambda \in {\Lambda }_b^{\widehat{B}}}\overline{{\tilde{Q}}_{\lambda }\left({\widehat{A}}^p\right)}{Q}_{\lambda |b}\left(\widehat{\rho }\right){\widehat{\Pi }}_b^{\widehat{B}}\hfill \\ & =& \sum _{b\in \sigma \left(\widehat{B}\right)}\sum _{\lambda \in {\Lambda }_b^{\widehat{B}}}\overline{{\left({\tilde{Q}}_{\lambda }\left(\widehat{A}\right)\right)}^p}{Q}_{\lambda |b}\left(\widehat{\rho }\right){\widehat{\Pi }}_b^{\widehat{B}}=\sum _{b\in \sigma \left(\widehat{B}\right)}\sum _{a\in \sigma \left(\widehat{A}\right)}\sum _{\lambda \in {\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}}\overline{{\left({\tilde{Q}}_{\lambda }\left(\widehat{A}\right)\right)}^p}{Q}_{\lambda |b}\left(\widehat{\rho }\right){\widehat{\Pi }}_b^{\widehat{B}}\hfill \\ & =& \sum _{b\in \sigma \left(\widehat{B}\right)}\sum _{a\in \sigma \left(\widehat{A}\right)}\sum _{\lambda \in {\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}}\overline{a^p}{Q}_{\lambda |b}\left(\widehat{\rho }\right){\widehat{\Pi }}_b^{\widehat{B}}=\sum _{b\in \sigma \left(\widehat{B}\right)}\sum _{a\in \sigma \left(\widehat{A}\right)}{a}^p\sum _{\lambda \in {\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}}{Q}_{\lambda |b}\left(\widehat{\rho }\right){\widehat{\Pi }}_b^{\widehat{B}}.\hfill \end{array}$$

where ${\tilde{Q}}_{\lambda }\left({\widehat{A}}^p\right)={\tilde{Q}}_{\lambda }{\left(\widehat{A}\right)}^p$ follows directly from Lemma 4. We also have that

$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\widehat{\rho }}^R\left({\widehat{A}}^p\right|\widehat{B})& =& \sum _{b\in \sigma \left(\widehat{B}\right)}\sum _{a\in \sigma \left(\widehat{A}\right)}{a}^p{R}_{(a,b)|b}\left(\widehat{\rho }\right){\widehat{\Pi }}_b^{\widehat{B}}.\hfill \end{array}$$

Thus, for all polynomial P with for all $i\in 〚1,n〛{deg}_i\left(P\right)⩽d_i^{\widehat{A}}-1$ and all $\widehat{\rho }\in D_{\widehat{B}}$, we have that

$$\forall b\in \sigma \left(\widehat{B}\right),\sum _{a\in \sigma \left(\widehat{A}\right)}P\left(a\right)\sum _{\lambda \in {\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}}{Q}_{\lambda |b}\left(\widehat{\rho }\right)=\sum _{a\in \sigma \left(\widehat{A}\right)}P\left(a\right)R_{(a,b)|b}\left(\widehat{\rho }\right).$$

We fix $a^{\prime }\in \sigma \left(\widehat{A}\right)$. By Lagrange interpolation, there exists $P_{a^{\prime }}$ such that $P_{a^{\prime }}\left(a\right)={\delta }_{a,a^{\prime }}$ for all $a\in \sigma \left(\widehat{A}\right)$ and ${deg}_i\left(P\right)⩽d_i^{\widehat{A}}-1$ for all $i\in 〚1,n〛$. Thus, we have that, for all $\widehat{\rho }\in D_{\widehat{B}}$

$$\forall b\in \sigma \left(\widehat{B}\right),\sum _{\lambda \in {\Lambda }_{a^{\prime }}^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}}{Q}_{\lambda |b}\left(\widehat{\rho }\right)=R_{(a^{\prime },b)|b}\left(\widehat{\rho }\right).$$

Using Equations (99) and (113) multiplied by $\mathrm{Tr}\left(\widehat{\rho }{\widehat{\Pi }}_b^{\widehat{B}}\right)$, we obtain that, for all $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$ and for all $\widehat{\rho }\in D_{\widehat{B}}$

$$\mathrm{Tr}\left(\widehat{\rho }{\left(\sum _{\lambda \in {\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}}{S}_{\lambda }^Q-S_{a,b}^R\right)}^{†}\right)=0.$$

Consequently, we conclude that, for all $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$

$$\sum _{\lambda \in {\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}}{S}_{\lambda }^Q=S_{a,b}^R$$

(144)

As $S_{a,b}^R\ne 0$, we have that for all $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$, ${\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}\ne \varnothing$. Consequently, the map

$$\Phi :\left\{\begin{array}{ccc}\hfill \Lambda & \to & \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)\hfill \\ \hfill \lambda & \mapsto & (Q_{\lambda }\left(\widehat{A}\right),Q_{\lambda }\left(\widehat{B}\right))\hfill \end{array}\right.$$

is surjective and as $\left|\Lambda \right|=\left|\sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)\right|=d^2$, $\Phi$ is a bijection. In particular, $\left|{\Lambda }_a^{\widehat{A}}\cap {\Lambda }_b^{\widehat{B}}\right|=1$ and this element is ${\Phi }^{-1}(a,b)$ for all $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$. We finally obtain, by Equation (144), that for all $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$

$$S_{{\Phi }^{-1}(a,b)}^Q=S_{a,b}^R$$

(145)

proving $\left(v\right)$. □

This theorem shows that a quasiprobability representation of quantum mechanics on a Hilbert space $\mathcal{H}$ that is Born-compatible for two complementary CSCO $\widehat{A}$ and $\widehat{B}$, is completely determined by its associated quantum conditional expectations of ${\widehat{A}}^p$ given $\widehat{B}$ with $p\in {\Gamma }^{\widehat{A}}$. Combining Theorem 8 with Lemma 6, and setting $(R,\tilde{R})=(Q^{\mathrm{KD},\ell },{\tilde{Q}}^{\mathrm{KD},\ell })$, we immediately obtain the following theorem:

Theorem 9.

Let $\widehat{A}$ and $\widehat{B}$ be complementary $\mathit{CSCO}$. Let $(Q,\tilde{Q})$ be an $\widehat{A}$ and $\widehat{B}$-compatible quasiprobability representation of quantum mechanics defined on a set Λ ($\left|\Lambda \right|=d^2)$. Then, the following are equivalent:

(i) 

$\forall \widehat{\rho }\in D_{\widehat{B}},\forall p\in {\Gamma }^{\widehat{A}},{\mathbb{E}}_{\widehat{\rho }}^Q\left({\widehat{A}}^p\right|\widehat{B})={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left({\widehat{A}}^p\right|\widehat{B})$

(ii) 

$\forall \widehat{\rho }\in D_{\widehat{A}},\forall p\in {\Gamma }^{\widehat{B}},{\mathbb{E}}_{\widehat{\rho }}^Q\left({\widehat{B}}^p\right|\widehat{A})={\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left({\widehat{B}}^p\right|\widehat{A})$

(iii) 

There exists a bijective map $\Phi :\Lambda \to \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$ such that for all $(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$

$$\forall \widehat{\rho }\in D\left(\mathcal{H}\right),Q_{{\Phi }^{-1}(a,b)}\left(\widehat{\rho }\right)=Q_{a,b}^{\mathrm{KD},\ell }\left(\widehat{\rho }\right)\mathrm{and}\forall \widehat{X}\in \mathcal{L}\left(\mathcal{H}\right),{\tilde{Q}}_{{\Phi }^{-1}(a,b)}\left(\widehat{X}\right)={\tilde{Q}}_{a,b}^{\mathrm{KD},\ell }\left(\widehat{X}\right).$$

Theorem 9 provides a direct proof of Theorem 1 as it is a strenghtened version thereof: indeed, Theorem 9 relies on the hypothesis that the conditional expectations coincides on powers of $\widehat{A}$ or on powers of $\widehat{B}$ whereas Theorem 1 relies on the equality of the conditional expectation on all operators in $\mathcal{L}\left(\mathcal{H}\right).$

The left Kirkwood–Dirac distribution is therefore the only quasiprobability representation of quantum mechanics that is $\widehat{A}$ and $\widehat{B}$ compatible and for which the quantum conditional expectation of ${\widehat{A}}^p$ given $\widehat{B}$ is given by the “left weak-value” formalism, meaning by the values of ${\left({\displaystyle \frac{\mathrm{Tr}\left({\widehat{\Pi }}_b^{\widehat{B}}{\widehat{A}}^p\widehat{\rho }\right)}{\mathrm{Tr}\left({\widehat{\Pi }}_b^{\widehat{B}}\rho \right)}}\right)}_{b\in \sigma \left(\widehat{B}\right),p\in {\Gamma }^{\widehat{A}}}$. For the right Kirkwood–Dirac representation, it is characterized by the “right weak-value formalism”, meaning by the values of ${\left({\displaystyle \frac{\mathrm{Tr}\left(\widehat{\rho }{\widehat{A}}^p{\widehat{\Pi }}_b^{\widehat{B}}\right)}{\mathrm{Tr}\left({\widehat{\Pi }}_b^{\widehat{B}}\rho \right)}}\right)}_{b\in \sigma \left(\widehat{B}\right),p\in {\Gamma }^{\widehat{A}}}$.

Theorem 8 can further be used to characterize any fixed $\widehat{A}$ and $\widehat{B}$ Born-compatible quasiprobability representation of quantum mechanics. If one briefly goes to the infinite dimensional setting for illustrative purposes, the Wigner representation is then uniquely determined by the associated conditional expectations of ${\widehat{P}}^n$ given $\widehat{Q}$ that will not coincide with the ones arising from the weak value formalism. We illustrate this fact in Appendix C where we compare the KD and Wigner conditional expectations of ${\widehat{P}}^n$ knowing $\widehat{Q}$, for $n=1,2$, showing they are different when $n=2$. This also means that the conditional expectation given by the Wigner quasiprobability representation cannot be the one defined by minimization in Section 3.

6. Quantum Conditional Expectation and Parameter Estimation

In this section, we explore properties of the quantum conditional expectations ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ introduced in Section 3 in relation to the following well-known parameter estimation problem. Let $\widehat{\rho }$ be any quantum state and consider the state

$${\widehat{\rho }}_{\widehat{X}}\left(\theta \right):=e^{-i\theta \widehat{X}}\widehat{\rho }{e}^{i\theta \widehat{X}}$$

(146)

evolved under the unitary flow generated by some observable $\widehat{X}$. The problem is then to estimate the “phase” $\theta$ from the measurement of an appropriately chosen observable $\widehat{Y}$ in ${\widehat{\rho }}_{\widehat{X}}\left(\theta \right)$ [37,45,58,59,60].

We first show that the variance of the imaginary part of the above quantum conditional expectation is equal to the classical Fisher information associated with this phase estimation problem (Proposition 5). This relation was previously established and elaborated upon in [37,45] within the context of weak value physics. We then apply the results of Section 3.3, and in particular the law of iterated expectations and the law of total variance to relate the mean and the variance of $\widehat{X}$ in the state ${\widehat{\rho }}_{\widehat{X}}\left(\theta \right)$ to those of its best estimator ${\mathbb{E}}_{{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ and of its conditional variance ${\Delta }_{{\widehat{\rho }}_X\left(\theta \right)}^{\ell /\mathrm{r},2}\left(\widehat{X}\right|\widehat{Y})$ in the same state. This sharpens a result of [37] and corroborates the idea that ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ is a “best estimator” of $\widehat{X}$.

These developments allow us to introduce and then study, in Section 6.3, the notion of phase insensitive state, which is a state for which the above Fisher information vanishes. We then show that, modulo some technical assumptions, KD-real states are phase insensitive (Proposition 6). More precisely, we will observe that if the metrological protocol described above involves an observable $\widehat{X}$ and and a state $\widehat{\rho }$ that are KD-real, with $\widehat{Y}=\widehat{B}$ or $\widehat{Y}=\widehat{A}$, then their Fisher information ${\mathrm{I}}_{\mathrm{F}}(\widehat{Y};0)$ vanishes. Such states and observables are therefore not useful for this particular phase estimation protocol. But, as noted in Section 6.2, they can provide simultaneous information on the Born probability distributions of non-commuting observables $\widehat{X},\widehat{Y}$ through a weak measurement protocol. This result further suggests that, if $\widehat{\rho }$ and $\widehat{X}$ are KD real, and if the associated quantum Fisher information is non-zero, then the logarithmic derivative $\widehat{L}$ of $\widehat{\rho }$ cannot be KD real. We show this to be the case when $\widehat{\rho }$ is pure and provide two families of examples where $\widehat{\rho }$ is mixed and $\widehat{L}$ is indeed not KD real.

6.1. The Imaginary Part of ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$: Fisher Information

If the observable $\widehat{Y}$ is measured when the system is in the state ${\widehat{\rho }}_{\widehat{X}}\left(\theta \right)$, the outcome $y\in \sigma \left(\widehat{Y}\right)$ is observed with probability

$$p(y;\theta ):=\mathrm{Tr}\left({\widehat{\Pi }}_y^{\widehat{Y}}{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)\right).$$

(147)

The Fisher information [61,62,63,64] of this $\theta$-dependent probability is defined as the expected value of ${({\partial }_{\theta }ln\left(p(y;\theta )\right))}^2$:

$${\mathrm{I}}_{\mathrm{F}}(\widehat{Y};\theta ):=\sum _y{({\partial }_{\theta }ln\left(p(y;\theta )\right))}^{2}p(y;\theta ).$$

(148)

Note that this Fisher information depends not only on $\widehat{Y}$ and $\theta$, but also on $\widehat{\rho }$ and $\widehat{X}$, although we do not indicate this dependence in the notation. The Fisher information derives its importance principally from the famous Cramèr–Rao inequality which states that the variance $\mathbb{V}\left(\tilde{\theta }\right)$ of any unbiased estimator $\tilde{\theta }\left(\widehat{Y}\right)$ of $\theta$ is bounded from below by the inverse of the Fisher information:

$$\mathbb{V}\left(\tilde{\theta }\right)\ge {\left({\mathrm{I}}_{\mathrm{F}}(\widehat{Y};\theta )\right)}^{-1}.$$

(149)

The Fisher information ${\mathrm{I}}_{\mathrm{F}}(\widehat{Y};\theta )$ is related to the (imaginary part of) the conditional expectation ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$, as follows [37,45]:

Proposition 5.

Let $\widehat{Y}$ be a $\mathrm{CSCO}$, ${\widehat{X}}^{†}=\widehat{X}\in \mathcal{L}\left(\mathcal{H}\right)$ and suppose that ${\widehat{\rho }}_{\widehat{X}}\left(\theta \right)\in D_{\widehat{Y}}$. Then

$${\mathrm{I}}_{\mathrm{F}}(\widehat{Y};\theta )=4\mathrm{Tr}\left({\left(\mathrm{Im}{\mathbb{E}}_{{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})\right)}^2{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)\right).$$

(150)

Proof. 

A straightforward computation yields [37,45]

$${\partial }_{\theta }lnp(y;\theta )={\displaystyle \frac{-i\mathrm{Tr}\left({\widehat{\Pi }}_y^{\widehat{Y}}[\widehat{X},{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)]\right)}{\mathrm{Tr}\left({\widehat{\Pi }}_y^{\widehat{Y}}{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)\right)}}=2\mathrm{Im}\left({\displaystyle \frac{\mathrm{Tr}\left({\widehat{\Pi }}_y^{\widehat{Y}}\widehat{X}{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)\right)}{\mathrm{Tr}\left({\widehat{\Pi }}_y^{\widehat{Y}}{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)\right)}}\right),$$

(151)

where we used the self-adjointness of $\widehat{X}$. This is the imaginary part of the coefficient of ${\widehat{\Pi }}_y$ in ${\mathbb{E}}_{{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$, see Definition 2. This proves Equation (150). □

Note that ${\mathbb{E}}_{\widehat{\rho }}\left(\mathrm{Im}\left({\mathbb{E}}_{\widehat{\rho }}^{\ell /r}\left(\widehat{X}\right|\widehat{Y})\right)\right)=0$. Hence Equation (150) states that the classical Fisher information of the quantum mechanical Born probabilities $p(y;\theta )$ equals (up to a factor 4) the variance of the imaginary part of the (left/right) quantum conditional expectation. This observation provides an operational meaning to the imaginary part of the conditional expectation ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$. Since in classical probability theory the conditional expectation of a real random variable is always real, the presence of an imaginary part to ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ can be considered as a typical quantum feature. This raises the question precisely when $\mathrm{Im}{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ vanishes, depending on $\widehat{\rho },\widehat{X}$, and $\widehat{Y}$. From Equation (51) one obtains that

$$\mathrm{Im}{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})=-{\displaystyle \frac{1}{2}}\sum _{y\in \sigma \left(\widehat{Y}\right)}{\displaystyle \frac{\langle {\phi }_y^{\widehat{Y}},[\widehat{X},\widehat{\rho }]{\phi }_y^{\widehat{Y}}\rangle }{\langle {\phi }_y^{\widehat{Y}},\widehat{\rho }{\phi }_y^{\widehat{Y}}\rangle }}{\widehat{\Pi }}_y^{\widehat{Y}}=-\mathrm{Im}{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{Y}).$$

(152)

Hence, if

$$[\widehat{\rho },\widehat{X}]=0,\mathrm{or}[\widehat{\rho },{\widehat{\Pi }}_y^{\widehat{Y}}]=0,\mathrm{or}[{\widehat{\Pi }}_y^{\widehat{Y}},\widehat{X}]=0,$$

(153)

then $\mathrm{Im}{\mathbb{E}}^{\ell }/\mathrm{r}\left(\widehat{X}\right|\widehat{Y})=0$. This, in turn, implies that the Fisher information ${\mathrm{I}}_{\mathrm{F}}(\widehat{Y};0)$ vanishes. One may therefore conclude that the absence of incompatibility between $\widehat{\rho },\widehat{X}$ and $\widehat{Y}$—which is a form of “classicality”—is a sufficient condition for the absence of an imaginary part to the quantum conditional expectation and hence of the vanishing of the Fisher information ${\mathrm{I}}_{\mathrm{F}}(\widehat{Y};0)$. We will see below that it is not a necessary condition, as the Fisher information can vanish in cases where the commutators of Equation (153) are all non-zero. An example is given in Equation (78). Generally, self-adjointness of the conditional expectation ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y})$ does not preclude the manifestation of other nonclassical features associated to the triplet $(\widehat{\rho },\widehat{X},\widehat{Y})$: as we will see in the next section, anomalous values of ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{Y}=y)$ point to the possibility of weak value amplification, which is a typical quantum phenomenon as well.

In order to address the question under what circumstances the Fisher information vanishes, we introduce the notion of phase insensitive states:

Definition 6.

Given an observable $\widehat{X}$ and a $\mathrm{CSCO}$$\widehat{Y}$, we say that a state $\widehat{\rho }\in D_{\widehat{Y}}$ is phase insensitive for a measurement of $\widehat{Y}$ if ${\mathrm{I}}_{\mathrm{F}}(\widehat{Y};0)=0$.

The phase insensitivity of a state $\widehat{\rho }$ depends both on the generator $\widehat{X}$ associated to the phase and on the observable $\widehat{Y}$. We will simply say “$\widehat{\rho }$ is phase insensitive” when the choice of $\widehat{X}$ and $\widehat{Y}$ is clear from the context; otherwise, we will say the triplet $(\widehat{\rho },\widehat{X},\widehat{Y})$ is phase-insensitive. The operational meaning of phase insensitivity is discussed in some detail in Appendix D. We recall there that the Fisher information can be viewed as the squared relative rate of change of the probabilities $p(y;\theta )$, when the phase $\theta$ varies by a small amount $\delta \theta$. Comparing this quantity to the mean squared relative error ${\parallel \delta p\parallel }_{\mathrm{F}}^2$ of the experimentally determined probabilities, one obtains an estimate on the minimal variation $\delta {\theta }_{\mathrm{m}in}$ that is experimentally detectable:

$$\delta {\theta }^2\ge \delta {\theta }_{\mathrm{m}in}^2:={\displaystyle \frac{{\Vert \delta p\parallel }_{\mathrm{F}}^2}{I_{\mathrm{F}}(\widehat{Y};\theta )}}.$$

For small, and a fortiori, for vanishing Fisher information, this minimal phase variation becomes very large, thereby justifying the definition of phase insensitivity. We refer to Appendix D for more details. We will show how to identify phase insensitive triplets $(\widehat{\rho },\widehat{X},\widehat{Y})$ using the notion of KD-reality in Section 6.3, where we will also further explore the link with the quantum Fisher information.

6.2. An Additive Uncertainty Principle

As a direct application of the results of Section 3, and in particular of the quantum law of total variance derived there, we can now relate the variance of $\widehat{X}$ to the variances of the real and imaginary parts of its best estimator ${\mathbb{E}}_{{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)}^{\ell /r}\left(\widehat{X}\right|\widehat{Y})$ and to the expected value of its conditional variance ${\Delta }_{{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)}^{\ell /\mathrm{r},2}\left(\widehat{X}\right|\widehat{Y})$. In what follows, for $Z\in \mathcal{L}\left(\mathcal{H}\right)$, we write $\left|Z\right|=\sqrt{Z^{†}Z}$. Since ${\Delta }_{{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)}^2\widehat{X}={\Delta }_{\widehat{\rho }}^2\widehat{X}$ for any value of $\theta$, it follows from the law of total variance in Proposition 2 combined with Equation (88) and Proposition 5 that, for any $\widehat{X}={\widehat{X}}^{†}\in \mathcal{L}\left(\mathcal{H}\right)$ and $\mathrm{CSCO}$ $\widehat{Y}$,

$${\Delta }_{\widehat{\rho }}^2\widehat{X}={\Delta }_{{\widehat{\rho }}_X\left(\theta \right)}^2{\mathbb{E}}_{{\widehat{\rho }}_X\left(\theta \right)}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})+{\displaystyle \frac{1}{4}}{\mathrm{I}}_{\mathrm{F}}(\widehat{Y};\theta )+\mathrm{Tr}({\widehat{\rho }}_{\widehat{X}}\left(\theta \right)\mid \widehat{X}-{\mathbb{E}}_{{\widehat{\rho }}_X\left(\theta \right)}^{\ell }\left(\widehat{X}\right|\widehat{Y}){\mid }^2),$$

(154)

and

$${\Delta }_{\widehat{\rho }}^2\widehat{X}={\Delta }_{{\widehat{\rho }}_X\left(\theta \right)}^2{\mathbb{E}}_{{\widehat{\rho }}_X\left(\theta \right)}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})+{\displaystyle \frac{1}{4}}{\mathrm{I}}_{\mathrm{F}}(\widehat{Y};\theta )+\mathrm{Tr}({\widehat{\rho }}_{\widehat{X}}\left(\theta \right)\mid \widehat{X}-{\mathbb{E}}_{{\widehat{\rho }}_X\left(\theta \right)}^{\mathrm{r}}{\left(\widehat{X}\right|\widehat{Y})}^{†}{\mid }^2),$$

(155)

where we recall the notation ${\mathbb{E}}_{{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y}):={\mathbb{E}}_{{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)}^{\ell /\mathrm{r},\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})$. As we saw in Section 3.3, the error term (last term) can be understood as the expected value of the conditional variance of $\widehat{X}$. When $\widehat{\rho }$ is pure, this error term vanishes and one finds

$${\Delta }_{\widehat{\rho }}^2\widehat{X}={\Delta }_{{\widehat{\rho }}_X\left(\theta \right)}^2{\mathbb{E}}_{{\widehat{\rho }}_X\left(\theta \right)}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})+{\displaystyle \frac{1}{4}}{\mathrm{I}}_{\mathrm{F}}(\widehat{Y};\theta ).$$

(156)

We therefore have, for all $\widehat{\rho }$

$${\Delta }_{\widehat{\rho }}^2\widehat{X}\ge {\Delta }_{{\widehat{\rho }}_X\left(\theta \right)}^2{\mathbb{E}}_{{\widehat{\rho }}_X\left(\theta \right)}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})+{\displaystyle \frac{1}{4}}{\mathrm{I}}_{\mathrm{F}}(\widehat{Y};\theta ).$$

(157)

Equations (156) and (157) were first derived in [37] (at $\theta =0$), using a formulation in terms of weak values instead of in terms of conditional expectations. The identification of the error term in Equations (154) and (155), the concurrent link with the quantum law of total variance, and the interpretation of the error term as the expected value of a quantum conditional variance are, to the best of our knowledge, new here.

It is pointed out in [37] that one can think of Equation (156) (at $\theta =0$) as an (additive) uncertainty relation. Indeed, both terms in the right-hand side depend on $\widehat{Y}$, whereas the left-hand side does not. So if one term is small, the other must be large, and vice versa; this result has the following interpretation. If, for a given $\widehat{\rho }$ and $\widehat{X}$, $\widehat{Y}$ can be chosen such that the Fisher information is large, meaning close to (four times) the variance of $\widehat{X}$, then, according to the Cramer–Rao bound, the variance on the estimation of $\theta$ can be made small, meaning one can obtain a good estimate on the phase $\theta$ from measurements of $\widehat{Y}$ in the state ${\widehat{\rho }}_{\widehat{X}}\left(\theta \right)$. In that case, however, ${\mathbb{E}}_{{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})$ will have a small variance that will be far from the variance of $\widehat{X}$. So then the fluctuations in the conditional expectation ${\mathbb{E}}_{{\widehat{\rho }}_{\widehat{X}}\left(\theta \right)}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})$ do not provide much information on the probability distribution of $\widehat{X}$ for the state $\widehat{\rho }$. For pure states $\widehat{\rho }$, the extreme case occurs when $\widehat{Y}$ is taken to be equal to the symmetric logarithmic derivative $\widehat{L}$ of ${\widehat{\rho }}_{\widehat{X}\left(\theta \right)}$ at $\theta =0$, because then the Fisher information is maximal. This maximum Fisher information, which is the quantum Fisher information $I_{\mathrm{QF}}\left(0\right)$, then equals $4{\Delta }_{\widehat{\rho }}^2\widehat{X}$ (See Appendix E for details). On the other hand, if $\widehat{Y}$ is such that the Fisher information vanishes or is small, one can obtain only little information on the phase $\theta$ from measurements of $\widehat{Y}$ in the state ${\widehat{\rho }}_{\widehat{X}}\left(\theta \right)$. However, in that case, the variance of ${\mathbb{E}}_{{\widehat{\rho }}_X\left(\theta \right)}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})$ will be close or equal to the one of $\widehat{X}$. Specifically, when the Fisher information vanishes and $\widehat{\rho }$ is pure, the former is equal to the latter (see Equation (156)). In conclusion, if $\widehat{\rho }$ is pure, we find that

$$\underset{\widehat{Y}}{min}{\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})={\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{L})={\Delta }_{\widehat{\rho }}^2\widehat{X}-{\displaystyle \frac{1}{4}}{I}_{\mathrm{QF}}\left(0\right)=0,{\Delta }_{\widehat{\rho }}^2\widehat{X}=\underset{\widehat{Y}}{max}{\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y}),$$

(158)

where the maximum on the right is reached on phase insensitive triplets $(\widehat{\rho },\widehat{X},\widehat{Y})$. When $\widehat{\rho }$ is mixed, one has

$$\underset{\widehat{Y}}{min}{\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y})\le {\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{L})\le {\Delta }_{\widehat{\rho }}^2\widehat{X}-{\displaystyle \frac{1}{4}}{I}_{\mathrm{QF}}\left(0\right)\le {\Delta }_{\widehat{\rho }}^2\widehat{X}=\underset{\widehat{Y}}{max}{\Delta }_{\widehat{\rho }}^2{\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{Y}),$$

(159)

where the maximum is reached when $\widehat{Y}=\widehat{X}$, for example.

We note that the real and imaginary parts of the conditional expectations ${\mathbb{E}}_{\widehat{\rho }}^{\ell ,\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ can be experimentally accessed via a weak measurement procedure that we describe in Appendix A. We conclude that there is therefore a necessary trade-off between obtaining information on $\theta$ from $p(y;\theta )$, for which a large Fisher information is needed, or on the probability distribution of $\widehat{X}$ in $\widehat{\rho }$ through the weak measurement procedure of $\widehat{X}$, conditioned on $\widehat{Y}$, for which the Fisher information should optimally vanish, such that the triplet $(\widehat{\rho },\widehat{X},\widehat{Y})$ is phase insensitive. Phase insensitivity will be related to KD reality in Section 6.3.

We finally point out that, for all $\widehat{\rho }$ and $\theta$

$$0\le {\mathrm{I}}_{\mathrm{F}}(\widehat{Y};\theta )\le I_{\mathrm{QF}}\left(\theta \right)\le 4{\Delta }_{\widehat{\rho }}^{\ell ,2}\widehat{X}\le {(x_{max}-x_{min})}^2,$$

(160)

where $x_{max}$ and $x_{min}$ are the largest and smallest, eigenvalues of $\widehat{X}$, respectively, and we used Popoviciu’s inequality, which states that the variance of a random variable taking values in some bounded interval $[a,b]$ is bounded from above by ${\displaystyle \frac{1}{4}}{(b-a)}^2.$ For example, see [65], where this inequality is derived as a consequence of a stronger inequality which states that

$${\displaystyle \frac{1}{4}}{\mathrm{I}}_{\mathrm{F}}(\widehat{Y};\theta )\le {\Delta }_{\widehat{\rho }}^{\ell ,2}\widehat{X}\le (x_{max}-{\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right))({\mathbb{E}}_{\widehat{\rho }}\left(\widehat{X}\right)-x_{min}),$$

but for which one has to know the expected value of $\widehat{X}.$

6.3. An Application: KD Reality Implies Phase Insensitivity

In this subsection, we exploit the link between the KD representations of quantum mechanics and the conditional expectations ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{Y})$ (Theorem 1) with $\widehat{Y}=\widehat{A}$ or $\widehat{Y}=\widehat{B}$ in order to construct phase-insensitive triplets using the KD-real sector of a KD representation of quantum mechanics.

We have seen that, if any two of $\widehat{\rho },\widehat{X}$, and $\widehat{Y}$ are compatible (commute), then the associated Fisher information vanishes and hence $\widehat{\rho }$ is phase insensitive. However, these conditions are very strong, and in fact rather trivial. For example, if $[\widehat{\rho },\widehat{X}]=0$, then ${\widehat{\rho }}_{\widehat{X}}\left(\theta \right)=\widehat{\rho }$ is independent of $\theta$ and if $[\widehat{X},{\widehat{\Pi }}_y^{\widehat{Y}}]=0$, the two observables involved are compatible. We will now show that non-trivial examples of phase insensitivity arise naturally within the KD-real sector of KD representations.

As a first set of examples, consider the following situation. Let $\widehat{A}$ and $\widehat{B}$ be two complementary CSCO and suppose

$$\widehat{X}=f_1\left(\widehat{A}\right)+g_1\left(\widehat{B}\right),\widehat{\rho }=f_2\left(\widehat{A}\right)+g_2\left(\widehat{B}\right),$$

(161)

where $f_1,g_1$ are real-valued and $f_2,g_2$ are positive. Then

$$[\widehat{X},\widehat{\rho }]=[f_1\left(\widehat{A}\right),g_2\left(\widehat{B}\right)]+[g_1\left(\widehat{B}\right),f_2\left(\widehat{A}\right)].$$

(162)

Note that this commutator will not vanish if the functions $f_i,g_i$ are non-trivial. It nevertheless follows straightforwardly from Equation (152) that $\mathrm{Im}{\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{B})=0=\mathrm{Im}{\mathbb{E}}_{\widehat{\rho }}^r\left(\widehat{X}\right|\widehat{B})$ and hence that ${\mathrm{I}}_{\mathrm{F}}(\widehat{B};0)=0$ and, similarly, ${\mathrm{I}}_{\mathrm{F}}(\widehat{A};0)=0$. In the following, we shall extend this family of examples by using the KD representation of quantum mechanics associated to $\widehat{A}$ and $\widehat{B}$.

For that purpose, we define the space of left/right KD-real operators as

$$V_{\mathrm{KD},\mathrm{real}}^{\ell /\mathrm{r}}=\left\{\widehat{C}\in \mathcal{L}\left(\mathcal{H}\right)\mid \forall (a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right),{\tilde{Q}}_{a,b}^{\mathrm{KD},\ell /\mathrm{r}}\left(\widehat{C}\right)\in \mathbb{R}\right\}.$$

(163)

So, $\widehat{C}\in V_{\mathrm{KD},\mathrm{real}}^{\ell /\mathrm{r}}$ if and only if $\widehat{C}$ has a real left/right KD-symbol. In view of Equation (136), one has that $\widehat{C}\in V_{\mathrm{KD},\mathrm{real}}^{\ell }$ if and only if ${\widehat{C}}^{†}\in V_{\mathrm{KD},\mathrm{real}}^{\mathrm{r}}$. Consequently,

$$V_{\mathrm{KD},\mathrm{real}}^{\mathrm{sa}}=V_{\mathrm{KD},\mathrm{real}}^{\ell }\cap {\mathcal{L}}^{\mathrm{sa}}\left(\mathcal{H}\right)=V_{\mathrm{KD},\mathrm{real}}^{\mathrm{r}}\cap {\mathcal{L}}^{\mathrm{sa}}\left(\mathcal{H}\right).$$

(164)

In short, $V_{\mathrm{KD},\mathrm{real}}^{\mathrm{sa}}$ is the set of all observables with a real KD symbol. We will refer to $V_{\mathrm{KD},\mathrm{real}}^{\mathrm{sa}}$ as the KD-real sector of quantum mechanics on $\mathcal{H}$, and we will say that a state $\widehat{\rho }$ or an observable $\widehat{X}$ is KD real when it belongs to $V_{\mathrm{KD},\mathrm{real}}^{\mathrm{sa}}$.

Before turning to the study of KD-real pairs $(\widehat{\rho },\widehat{X})$, let us take a brief look at the special case where $\widehat{\rho }$ is KD positive and $\widehat{X}$ is KD real. Analogous to the discussion in Section 3.3, KD-positive states can be thought of as “classical”, at least to the extent that they allow for a joint probability distribution for $\widehat{A}$ and $\widehat{B}$. Moreover, we know from Theorem 1 that the associated conditional expectations ${\mathbb{E}}_{\widehat{\rho }}^{Q^{\mathrm{KD}}}\left(\widehat{X}\right|\widehat{B})$ equal ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{B})$, for all $\widehat{X}$ and can be thought as a classical conditional expectation (see Definition 4). When in addition $\widehat{X}=\widehat{A}$ and $\widehat{\rho }$ is KD positive, we are in the situation of Lemma 3: the KD distribution of $\widehat{\rho }$ is then a joint probability for $\widehat{A}$ and $\widehat{B}$ and satisfies (ii) of Lemma 3, such that in particular, ${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{A}\right|\widehat{B})$ does not have anomalous values. In addition, such positive states, when injected into a KD-positivity preserving quantum circuit, can be efficiently simulated classically [66,67,68]. Note however the following. Even if $\widehat{\rho }$ is KD-positive, and $\widehat{X}$ is KD-real, such that ${\mathbb{E}}_{\widehat{\rho }}^{Q^{\mathrm{KD}}}\left(\widehat{X}\right|\widehat{B})={\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{B})$ is self-adjoint, it only satisfies the operator bound

$$min{\tilde{Q}}^{\mathrm{KD}}\left(\widehat{X}\right)\le {\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{B})\le max{\tilde{Q}}^{\mathrm{KD}}\left(\widehat{X}\right);$$

(165)

it is therefore still possible for $E_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{B})$ to take on anomalous values, since nothing guarantees that $x_{min}\le \tilde{Q}\left(\widehat{X}\right)\le x_{max}$. This is then still a form of nonclassicality, despite the positivity of the KD distribution of $\widehat{\rho }$. In addition, there is, for general $\widehat{X}$ no straightforward notion of “marginals”. In conclusion, even if KD positive states have some classical features, they may still display quantum characteristics, depending on the problem considered.

We now turn to KD-real pairs $(\widehat{\rho },\widehat{X})$. Our first result, which is an immediate consequence of Lemma 6, shows that the KD-real states (and not only the KD-positive ones) are phase-insensitive provided $\widehat{X}$ is KD real and $\widehat{Y}=\widehat{B}$:

Proposition 6.

Let $\widehat{A},\widehat{B}$ be two complementary $\mathrm{CSCO}$ and let $Q^{\mathrm{KD},\ell /\mathrm{r}}$ be the corresponding left/right KD quasiprobability representations. Let $\widehat{X}={\widehat{X}}^{†}$ belong to $V_{\mathrm{KD},\mathrm{real}}^{\mathrm{sa}}$ and $\widehat{\rho }$ belong to $V_{\mathrm{KD},\mathrm{real}}^{\mathrm{sa}}\cap D_{\widehat{B}}$. Then, the left/right conditional expectations of $\widehat{X}$, given $\widehat{B}$, are self-adjoint such that

$${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{B})={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{B})={\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{B}).$$

(166)

Similarly, if $\widehat{\rho }$ belongs to $V_{\mathrm{KD},\mathrm{real}}^{\mathrm{sa}}\cap D_{\widehat{A}}$,

$${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa}}\left(\widehat{X}\right|\widehat{A})={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{A})={\mathbb{E}}_{\widehat{\rho }}^{\mathrm{r}}\left(\widehat{X}\right|\widehat{A}).$$

(167)

Consequently, $\widehat{\rho }$ is phase insensitive for the measurement of $\widehat{B}$ and for the measurement of $\widehat{A}$.

Proof. 

$\widehat{\rho }\in V_{\mathrm{KD},\mathrm{real}}^{\mathrm{sa}}$ implies that $Q_{a,b}^{\mathrm{KD},\ell }\left(\widehat{\rho }\right)$ and therefore also $Q_{a|b}^{\mathrm{KD},\ell }\left(\widehat{\rho }\right):=Q_{a,b}^{\mathrm{KD},\ell }\left(\widehat{\rho }\right)/\langle {\phi }_b^{\widehat{B}},\widehat{\rho }{\phi }_b^{\widehat{B}}\rangle$ are real. If $\widehat{X}={\widehat{X}}^{†}$ it follows from Lemma 6 that

$${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{B})=\sum _b\left(\sum _a\overline{{\tilde{Q}}_{a,b}^{\mathrm{KD},\ell }\left(\widehat{X}\right)}{Q}_{a|b}^{\mathrm{KD},\ell }\left(\widehat{\rho }\right)\right){\widehat{\Pi }}_b^{\widehat{B}}.$$

This operator is self-adjoint if $\widehat{X}$ has a real KD symbol. Moreover, ${\mathbb{E}}_{\widehat{\rho }}^r\left(\widehat{X}\right|\widehat{B})={\mathbb{E}}_{\widehat{\rho }}^{\ell }{\left({\widehat{X}}^{†}\right|\widehat{B})}^{†}={\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{B})$, proving Equation (166). Finally, Equation (150) implies that ${\mathrm{I}}_{\mathrm{F}}(\widehat{A};0)={\mathrm{I}}_{\mathrm{F}}(\widehat{B};0)=0$, meaning that ${\widehat{\rho }}_{\widehat{X}}\left(\theta \right)$ is phase insensitive for the measurement of $\widehat{B}$ and of $\widehat{A}$ at $\theta =0$. □

The proposition provides an interpretation of the KD-real sector, as follows. If $\widehat{\rho }$ and $\widehat{X}$ belong to the KD-real sector, then the conditional expectations ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{A})$ and ${\mathbb{E}}_{\widehat{\rho }}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{B})$ are self-adjoint, which implies that the Fisher information ${\mathrm{I}}_{\mathrm{F}}(\widehat{A};0)$ and ${\mathrm{I}}_{\mathrm{F}}(\widehat{B};0)$ vanishes such that the state $\widehat{\rho }$ is phase insensitive for measurements of $\widehat{A}$ or of $\widehat{B}$.

Consequently, in order to extract information about the phase $\theta$ from information about the family of states ${\widehat{\rho }}_{\widehat{X}}\left(\theta \right)$, one needs to perform a different measurement than the one of $\widehat{A}$ or of $\widehat{B}$. This result is of interest because the space $V_{\mathrm{KD},\mathrm{real}}^{\mathrm{sa}}$ can in a number of cases be described quite explicitly. As already pointed out above, the operators $f\left(\widehat{A}\right)+g\left(\widehat{B}\right)$, with $f:\sigma \left(\widehat{A}\right)\to \mathbb{R}$ and $g:\sigma \left(\widehat{B}\right)\to \mathbb{R}$, always belong to $V_{\mathrm{KD},\mathrm{real}}^{\mathrm{sa}}$. It was proven in [69] that all operators in $V_{\mathrm{KD},\mathrm{real}}^{\mathrm{sa}}$ are of this form with probability one when the eigenbases of $\widehat{A}$ and $\widehat{B}$ are chosen randomly from the uniform (Haar) distribution. In specific cases, the space $V_{\mathrm{KD},\mathrm{real}}^{\mathrm{sa}}$ can be considerably larger, such as for the KD representation associated to the natural position-momentum Heisenberg representation of qdits (with d not a prime number) and of finite and certain second countable LCA groups [23,24]. Let us finally note that the space $V_{\mathrm{KD},\mathrm{real}}^{\mathrm{sa}}$ is particularly simple when the transition matrix between $\widehat{A}$ and $\widehat{B}$ only has real entries: $V_{\mathrm{KD},\mathrm{real}}^{\mathrm{sa}}$ is then the space of all real symmetric matrices [46]. This, occurs, for example, for n-qubit systems [66]. In fact, we note in passing that, quite generally, if $\widehat{X}$ and $\widehat{\rho }$ both have real symmetric matrices in the $\widehat{Y}$ basis, then ${\mathrm{I}}_{\mathrm{F}}(\widehat{Y};0)=0$, always, as can be seen directly from Equation (152).

Suppose $\widehat{X}$ and $\widehat{\rho }$ are fixed and belong to the KD-real sector. To estimate $\theta$, a measurement different from $\widehat{A}$ or $\widehat{B}$ then has to be chosen. As is known, the measurement $\widehat{L}$ yielding the largest Fisher information, referred to as the quantum Fisher information,

$${\mathrm{I}}_{\mathrm{QF}}={\mathrm{I}}_{\mathrm{F}}(\widehat{L},0)=4\mathrm{Tr}\left({\left(\mathrm{Im}{\mathbb{E}}_{{\widehat{\rho }}_{\widehat{X}}\left(0\right)}^{\ell /\mathrm{r}}\left(\widehat{X}\right|\widehat{L})\right)}^2{\widehat{\rho }}_{\widehat{X}}\left(0\right)\right),$$

(168)

is provided by the symmetric logarithmic derivative $\widehat{L}$ of ${\widehat{\rho }}_{\widehat{X}}\left(\theta \right)$ at $\theta =0$. We refer to Appendix E for a self-contained discussion on the quantum Fisher information and its properties; see in particular Corollary A2. In view of what precedes, it is tempting to conjecture that, if $\widehat{\rho }$ and $\widehat{X}$ are both KD real, the optimal observable $\widehat{L}$ must not be KD-real itself, provided ${\mathrm{I}}_{\mathrm{QF}}$ does not vanish. For pure states, this conjecture is true, as proved in the proposition below. To that effect, we first prove the following lemma.

Lemma 8.

The following statements are equivalent:

1. 

$I_{\mathrm{QF}}\left(0\right)=0$,

2. 

$\widehat{\rho }$ commutes with $\widehat{X}$.

In other words, if $\widehat{\rho }$ and $\widehat{X}$ do not commute, then the triplet $(\widehat{\rho },\widehat{X},\widehat{L})$ is not phase insensitive.

Proof. 

We use $\widehat{L}$ to denote the symmetric logarithmic derivative of $\widehat{\rho }\left(\theta \right)={\widehat{\rho }}_{\widehat{X}}\left(\theta \right)$ at $\theta =0$. If $I_{\mathrm{QF}}\left(0\right)=0$ then by definition, we have that

$$I_{\mathrm{QF}}\left(0\right)=\mathrm{Tr}\left(\widehat{\rho }{\widehat{L}}^2\right)=0.$$

(169)

We thus obtain that $0=\mathrm{Tr}\left(\widehat{\rho }{\widehat{L}}^2\right)=\mathrm{Tr}\left({\left({\widehat{\rho }}^{1/2}\widehat{L}\right)}^{†}{\widehat{\rho }}^{1/2}\widehat{L}\right)$. Thus, ${\widehat{\rho }}^{1/2}\widehat{L}=0$, which implies $\widehat{\rho }\widehat{L}=0$. The same argument proves that $\widehat{L}\widehat{\rho }=0$. Consequently, we obtain that

$${\partial }_{\theta }\widehat{\rho }{\left(\theta \right)|}_{\theta =0}={\displaystyle \frac{1}{2}}\{\widehat{L},\widehat{\rho }\left(0\right)\}={\displaystyle \frac{1}{2}}(\widehat{L}\widehat{\rho }+\widehat{\rho }\widehat{L})=0$$

(170)

and by definition of $\widehat{\rho }\left(\theta \right)$, we obtain that $-i[\widehat{X},\widehat{\rho }]={\partial }_{\theta }\widehat{\rho }\left(\theta \right){|}_{\theta =0}=0$ and thus, $\widehat{X}$ and $\widehat{\rho }$ commute. Conversely, if $[\widehat{X},\widehat{\rho }]=0$ then we still have that

$$0=-i[\widehat{X},\widehat{\rho }]={\partial }_{\theta }\widehat{\rho }\left(\theta \right){|}_{\theta =0}={\displaystyle \frac{1}{2}}(\widehat{L}\widehat{\rho }+\widehat{\rho }\widehat{L})$$

(171)

and consequently,

$$I_{\mathrm{QF}}\left(0\right)=\mathrm{Tr}\left(\widehat{\rho }{\widehat{L}}^2\right)=\mathrm{Tr}\left({\displaystyle \frac{1}{2}}\left(\widehat{\rho }\widehat{L}+\widehat{L}\widehat{\rho }\right)\widehat{L}\right)=0,$$

(172)

ending the proof. □

Proposition 7.

Let $\widehat{A}$ and $\widehat{B}$ be two complementary CSCO. Suppose that $\widehat{\rho }=\left|\psi \right.〉\psi \in D_{\widehat{B}}$ is a left KD-real pure state and that $\widehat{X}={\widehat{X}}^{†}$ is a left KD-real observable such that $I_{\mathrm{QF}}\left(0\right)>0$ (or equivalently, that $\left|\psi \right.〉$ is not an eigenvector of $\widehat{X})$. Let $\widehat{L}$ be the symmetric logarithmic derivative of ${\widehat{\rho }}_X\left(\theta \right)$ at $\theta =0$.

Then, the conditional expectation of $\widehat{L}$ given $\widehat{B}$, ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{L}\right|\widehat{B})$, is non-zero and anti-Hermitian. In particular, $\widehat{L}$ is not left KD real.

Proof. 

Suppose, by contradiction, that ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{L}\right|\widehat{B})=0$. Then, as $\widehat{\rho }\in D_{\widehat{B}}$, we have that, for all $b\in \sigma \left(\widehat{B}\right)$:

$$\langle {\phi }_b^{\widehat{B}},\widehat{L}\psi \rangle =0,$$

meaning that $\widehat{L}\psi =0$ as $\widehat{B}$ is a CSCO. Moreover, by definition of $\widehat{L}$, we have that

$${\displaystyle \frac{1}{2}}\{\widehat{L},\widehat{\rho }\}={\displaystyle \frac{1}{2}}\{\widehat{L},{\widehat{\rho }}_{\widehat{X}}\left(0\right)\}={\partial }_{\theta }{\rho }_{\widehat{X}}\left(\theta \right){|}_{\theta =0}=-i[\widehat{X},\widehat{\rho }].$$

(173)

Applying this equality with $\widehat{\rho }=|\psi \rangle \langle \psi |$ we obtain

$$-i\left(\widehat{X}\psi -\langle \psi ,\widehat{X}\psi \rangle \psi \right)={\displaystyle \frac{1}{2}}\left(\widehat{L}\psi +\langle \psi ,\widehat{L}\psi \rangle \psi \right)=0,$$

as $\widehat{L}\psi =0$. This implies that

$$\widehat{X}\psi =\langle \psi ,\widehat{X}\psi \rangle \psi ,$$

proving that $\psi$ is an eigenvector of $\widehat{X}$, which is a contradiction. Thus, ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{L}\right|\widehat{B})\ne 0$.

According to Proposition 6, as $\widehat{\rho }$ and $\widehat{X}$ are left KD real, we have that ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{X}\right|\widehat{B})$ is self-adjoint, meaning that, for all $b\in \sigma \left(\widehat{B}\right)$:

$$\mathrm{Im}(\mathrm{Tr}({\widehat{\Pi }}_b^{\widehat{B}}\widehat{X}\widehat{\rho }))=0.$$

Moreover, from Equation (173), we compute that, for all $b\in \sigma \left(\widehat{B}\right)$:

$$\mathrm{Re}\left(\mathrm{Tr}\left({\widehat{\Pi }}_b^{\widehat{B}}\widehat{L}\widehat{\rho }\right)\right)=\mathrm{Im}\mathrm{Tr}({\widehat{\Pi }}_b^{\widehat{B}}\widehat{X}\widehat{\rho }).$$

This shows that ${\mathbb{E}}_{\widehat{\rho }}^{\mathrm{sa},\ell }\left(\widehat{L}\right|\widehat{B})=0$, proving that ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{L}\right|\widehat{B})$ is anti-hermitian.

Finally, as ${\mathbb{E}}_{\widehat{\rho }}^{\ell }\left(\widehat{L}\right|\widehat{B})$ is both non-zero and anti-hermitian, and as $\widehat{\rho }$ is left KD real, Proposition 6 implies that $\widehat{L}$ cannot be left KD real. This concludes the proof. □

We now show that this conjecture also holds for mixed states in two examples.

For any state $\widehat{\rho }$, there exists a CSCO $\widehat{A}$ such that $\widehat{\rho }=\rho \left(\widehat{A}\right)$ for some $\rho :\sigma \left(\widehat{A}\right)\to \mathbb{R}$. If $\widehat{\rho }$ has simple eigenvalues, one can take $\widehat{\rho }=\widehat{A}$. Let $\widehat{B}$ be a CSCO complementary to $\widehat{A}$. $\widehat{\rho }$ is then automatically KD real. For $a\in \sigma \left(\widehat{A}\right)$, we use ${\rho }_a$ to denote the eigenvalue of $\widehat{\rho }$ associated to the vector ${\phi }_a^{\widehat{A}}$. From Equation (A65), one easily computes that a logarithmic derivative of ${\widehat{\rho }}_{\widehat{X}}\left(\theta \right)$ at $\theta =0$, $\widehat{L}$, is given by

$${\widehat{L}}_{a,a^{\prime }}=\left\{\begin{array}{c}i{\displaystyle \frac{({\rho }_a-{\rho }_{a^{\prime }})}{{\rho }_a+{\rho }_{a^{\prime }}}}\langle {\phi }_a^{\widehat{A}},\widehat{X}{\phi }_{a^{\prime }}^{\widehat{A}}\rangle ,\mathrm{if}{\rho }_a+{\rho }_{a^{\prime }}\ne 0\hfill \\ 0\mathrm{otherwise},\hfill \end{array}\right.$$

where ${\widehat{L}}_{a,a^{\prime }}$ are the matrix coefficients of $\widehat{L}$ in the eigenbasis of $\widehat{A}$. The KD symbol of $\widehat{L}$ is then given by

$${\tilde{Q}}_{a,b}^{\mathrm{KD},\ell }\left(\widehat{L}\right)=i\sum _{\begin{array}{c}{a}^{\prime },b^{\prime }\\ {\rho }_a+{\rho }_{a^{\prime }}\ne 0\end{array}}{\displaystyle \frac{{\rho }_a-{\rho }_{a^{\prime }}}{{\rho }_a+{\rho }_{a^{\prime }}}}{\tilde{Q}}_{a,b^{\prime }}^{\mathrm{KD},\ell }\left(\widehat{X}\right){\displaystyle \frac{\langle {\phi }_a^{\widehat{A}},{\phi }_{b^{\prime }}^{\widehat{B}}\rangle }{\langle {\phi }_a^{\widehat{A}},{\phi }_b^{\widehat{B}}\rangle }}\langle {\phi }_{b^{\prime }}^{\widehat{B}},{\phi }_{a^{\prime }}^{\widehat{A}}\rangle \langle {\phi }_{a^{\prime }}^{\widehat{A}},{\phi }_b^{\widehat{B}}\rangle ,$$

(174)

$(a,b)\in \sigma \left(\widehat{A}\right)\times \sigma \left(\widehat{B}\right)$. From this expression we can give at least two situations in which the KD symbol of $\widehat{L}$ is purely imaginary:

1.

If the transition matrix between $\widehat{A}$ and $\widehat{B}$ has real coefficients and $\widehat{X}$ is KD real, then, the right-hand side of Equation (174) is purely imaginary. The condition that the transition matrix between $\widehat{A}$ and $\widehat{B}$ has real coefficients is for example fulfilled in systems of n qbits, where the transition matrix between $\widehat{A}$ and $\widehat{B}$ is given by the matrix of the Fourier transform on the abelian group ${\left(\mathbb{Z}/2\mathbb{Z}\right)}^n$ [23,66].

2.

If $\widehat{X}$ is of the form $\widehat{X}=f\left(\widehat{A}\right)+{\widehat{\Pi }}_{b_0}^{\widehat{B}}$, where $f:\sigma \left(\widehat{A}\right)\to \mathbb{R}$ and $b_0\in \sigma \left(\widehat{B}\right)$. It is clear in this case that $\widehat{X}$ belongs to the KD real sector. Moreover, it is easily computed in this case that for any $a\in \sigma \left(\widehat{A}\right)$,

$${\tilde{Q}}_{a,b_0}^{\mathrm{KD},\ell }\left(\widehat{L}\right)=i\sum _{\begin{array}{c}{a}^{\prime }\\ {\rho }_a+{\rho }_{a^{\prime }}\ne 0\end{array}}{\displaystyle \frac{{\rho }_a-{\rho }_{a^{\prime }}}{{\rho }_a+{\rho }_{a^{\prime }}}}|\langle {\phi }_{b_0}^{\widehat{B}},{\phi }_{a^{\prime }}^{\widehat{A}}\rangle {|}^2,$$

which is also purely imaginary.

One may conclude from this analysis that, in order to test the phase sensitivity of a KD-real state $\widehat{\rho }$ under the unitary flow generated by a KD-real observable $\widehat{X}$, one cannot limit oneself to measuring the observable $\widehat{B}$: indeed, under these hypotheses, the triplet $(\widehat{\rho },\widehat{X},\widehat{B})$ is phase-insensitive. This analysis does not rule out the possibility that some KD-real observable $\widehat{C}$ exists for which $I_{\mathrm{F}}(\widehat{C},0)>0$. Nevertheless, we proved that the optimal choice $\widehat{L}$ of measurement observable is not KD real, provided $\widehat{\rho }$ is pure. We leave the question if whether this remains true for all mixed states open. Another way to state the above result is to say that, if a pure state $\widehat{\rho }$ and $\widehat{X}$ do not commute, then there does not exist a KD representation defined by two CSCO $\widehat{A}$ and $\widehat{B}$ for which $\widehat{L}=g\left(\widehat{B}\right)$ and for which both $\widehat{\rho }$ and $\widehat{X}$ are KD real.

We end this section with a short remark on post-selected phase estimation, as studied in [42,43]. It is proven in those works that a quantum advantage can be obtained in that case, even though the state is KD real, but not KD positive. This does not contradict our findings for two reasons. First of all, the phase estimation problem considered there differs from the one considered here because of the post-selection process. As a result, the Fisher information is different as well. In addition, the relevant KD distribution used to obtain the results of [42,43] is different from the one considered here.

7. Discussion and Conclusions

There is elegance in defining concepts through optimization: the principles of least action and of maximal entropy are eminent examples. There is also convenience in defining concepts in quantum mechanics analogously to concepts in classical mechanics and probability theory. We have combined both approaches in order to define the notion of conditional expectation of an observable $\widehat{X}$ given an observable $\widehat{Y}$ in quantum mechanics: it is the best predictor of $\widehat{X}$ by a function of $\widehat{Y}$ in the sense that it is the function of $\widehat{Y}$ that minimizes a quadratic error in a given state $\widehat{\rho }$. In the first part of this paper, we have shown that, because of the non-commutativity between observables inherent in quantum mechanics, several different choices of quadratic error mimicking the classical choice exist, and lead to different definitions. We have explained how two of these choices are particularly natural. First, they are intimately linked to weak value physics, thereby sharpening the known links between weak values and quantum conditional expectations. Second, our approach has allowed us to establish that the quantum conditional expectations so defined are characterized fully by a “pull out” property, and obey a law of iterated expectations as well as a law of total variance, similar to the original notion in probability theory. In this manner, the definition of the quantum conditional expectation through minimization of a quadratic error provides a unified picture within which the essential properties of the quantum conditional expectation naturally emerge. In addition, by working in analogy with classical probability theory, this approach has allowed us to highlight precisely under what circumstances the quantum object differs from its classical counterpart: in other words, it sheds light on aspects of the classical–quantum boundary. Notably, the conditional expectation of an observable is not necessarily an observable, but can fail to be self-adjoint, a feature that has no equivalent in the classical theory. Its imaginary part is thus a typical quantum feature. Its variance provides an extra term in the law of the total variance, absent in the classical theory. This imaginary part has an interpretation as the Fisher information of a phase estimation problem associated to the triplet $\widehat{\rho }$, $\widehat{X}$, $\widehat{Y}$. We have also shown how the quantum conditional expectation can be used to establish a simple but efficient state-dependent no-go theorem for the existence of joint probabilities in quantum mechanics.

We conclude from this analysis that the proposed approach to quantum conditional expectations via minimization of a quadratic error provides a satisfying unifying picture that allows to analyze efficiently its properties and to pinpoint its similarities and—most importantly—its differences with the classical notion.

In the second part of the paper, we established a relation between these results and the theory of quasiprobability representations of quantum mechanics. The origin of the difficulties with the notion of conditional expectations in quantum mechanics is well known to be that joint probabilities of incompatible observables cannot be naturally defined. Quasiprobability representations of quantum mechanics circumvent this difficulty by allowing for non-positive and even non-real distributions to represent quantum states. This has led to a definition of joint quasiprobabilities and, through an analog of Bayes’ rule, to an alternative notion of conditional expectation from the one through minimization, that however depends very strongly on the choice of quasiprobability representation. Our main result in this paper is then that, among all possible quasiprobability representations that produce the correct Born probabilities associated to two chosen observables $\widehat{A}$ and $\widehat{B}$, only the Kirkwood–Dirac quasiprobability representations reproduce the above conditional expectations defined through miminization of a quadratic error. This provides therefore a defining feature for the Kirkwood–Dirac representations, singling them out among all Born-compatible quasiprobability representations.

In the last part of the paper, we have applied our insights to provide an interpretation of the KD-real sector of quantum mechanics in terms of phase-insensitivity.

Our work leaves a number of questions unanswered. First, whereas it is certainly elegant and natural to require a quantum conditional expectation to obey a “pull-out” formula, as we did, we did not find a direct operational meaning to such a pull-out formula. Such an interpretation would certainly strengthen the case for the approach promoted here. As we pointed out, other approaches exist; see, for example, [48,49,70], where no pull-out property is required and [54], where in a more algebraic context, on the contrary, both the left and right pull-out properties are required, thereby effectively restricting the states for which the notion makes sense. Note, however, that giving up the “pull-out” formula implies giving up the link with weak-value physics, as we proved. Second, although a quantum conditional variance appears naturally within our approach, we did not identify an operational meaning for this quantity, contrary to what is known for the conditional expectations, which are linked to weak value physics. Indeed, although the expected value of this conditional variance plays an analogous role to the so-called “unexplained variation” familiar from statistics, no such interpretation seems to arise in an evident manner in quantum mechanics. It would be clearly of interest to develop such an interpretation. In addition, the definition of the conditional variance does not seem to be unique, several possibilities presenting themselves, that appear natural from different points of view.

Finally, to avoid technical complications and to allow for mathematical precision, we have restricted ourselves in this paper to finite probability spaces $\Omega$ and to finite dimensional Hilbert spaces $\mathcal{H}$. While many of our results extend, at least on a formal level, to the infinite dimensional setting, technical difficulties arise that we have not tried to address; see [34] for quantum conditional expectations given a self-adjoint operator when the underlying state is pure.