Peelle's Pertinent Puzzle (PPP) was introduced in 1987 in the context of estimating fundamental parameters that arise in nuclear interaction experiments [1]. PPP is described briefly below and in more detail in [2]. When PPP occurs, generalized least squares (GLS) behaves reasonably under the model assumptions [2]. However, in PPP, GLS parameter estimates fall outside the range of the data, which has raised concerns that GLS is somehow flawed and has led to suggested alternatives to GLS estimators.

Others [1,3-9,12-15] have investigated alternatives to the GLS approach to PPP, but no numerical performance comparisons to other methods have been reported. And, one ad hoc approach involving simulated data realizations is statistically incomplete [8].

The following section includes additional background for PPP and the GLS approach. Section 3 discusses the ordinary sample mean as a perhaps naive alternative and shows that estimation error in the assumed covariance matrix cannot always be ignored. Sections 4 and 5 develop an alternative involving a modified form of approximate Bayesian computation (ABC). This ABC-based approach uses density estimation applied to realizations of simulated data that follow an error model appropriate for PPP and having various error distributions. Section 6 provides a second Bayesian alternative resulting from completion of the ad hoc approach by specifying a suitable prior probability density function (pdf). Section 7 is a summary.

A situation that leads to PPP is as follows. Two experiments aim to estimate a physical constant μ. Experiment one produces a measurement y₁, and experiment two produces y₂. Measurements y₁ and y₂ arise by imperfect conversion of underlying measurements m₁ and m₂. Suppose m₁ = μ + ∊_R1 and m₂ = μ + ∊_R2 where ∊_R1 is zero-mean random error in m₁ and similarly for ∊_R2. Burr et al. [2] prove that if y₁ =m₁ + ∊_S and y₂ = m₂ + α∊_S, where ∊_S ∼ N(0, σ_S) and α is any positive scale factor other than 1, the covariance matrix Σ of (y₁, y₂) can lie in the PPP region in which the GLS estimate of μ lies outside the range of y₁ and y₂. And, Burr et al. [2] give an example for which this error model is appropriate.

Let Σ be the 2-by-2 symmetric covariance matrix for y₁ and y₂ with diagonal entries σ 1 2, σ 2 2, and off-diagonal entry σ₁₂, which denote the variance of y₁, the variance of y₂, and the covariance of y₁ and y₂, respectively. For the case considered here and in Zhao and Perry [4] (1) ∑ = ( 0.1134 0.06 0.06 0.0504 )and we assume throughout that Σ exactly equals the numerical values given in Equation 1.

If the three errors ∊_R1, ∊_R2, and ∊_S have normal (Gaussian) distributions, then the joint pdf of (y₁, y₂) is also normal. If the three errors have non-normal distributions, then the joint pdf of (y₁, y₂) will not be normal and might be difficult to calculate analytically. Therefore, to handle non-normal distributions that might be difficult to calculate analytically, Sections 4 and 5 develop an estimator based on ABC. This ABC-based estimation can be applied with the error model just described resulting from the two experiments producing y₁, y₂ values with a covariance given by Equation 1, and can accommodate non-normal distributions.

It is well known that GLS applied to y₁ and y₂ results in the best linear unbiased estimate (BLUE) μ̂ of μ [16]. Here, “best” means minimum variance and unbiased means that on average (across hypothetical or real realizations of the same experiment), the estimate μ̂ will equal its true value μ. GLS properties are well known and are reviewed in [2]. The GLS estimate for μ arising from the generic model (2) ( y 1 y 2 ) = ( μ + e 1 μ + e 2 )with Σ = Cov(e₁, e₂) = Cov(y₁, y₂) is given by (3) μ ^ = c G t ∑ − 1 ( y 1 y 2 )where the scalar c = (G^tΣ⁻¹G)⁻¹. And the variance σ² of the GLS estimate μ̂ is given by (4) σ 2 = ( G t ∑ − 1 G ) − 1where the sensitivity (or design) matrix G is given by G^t = (1, 1). Putting the covariance of Equation 1 into Equations 3 and 4 and following the example in Burr et al. [2] with y₁ = 1.5 and y₂ = 1.0 gives μ̂ = c₁y₁ + c₂y₂ = 0.89 and σ = 0.22 where c₁ = −0.22 and c₂ = 1.22. Notice that c₁ + c₂ = 1 (so that μ̂ is unbiased) but also that c₁ < 0 and that μ̂ is smaller than each of the two measured values.

GLS estimation is guaranteed to produce the BLUE even if the underlying data are not normal. However, if the data are not normal, then the minimum variance unbiased estimator is not necessarily linear in the data. Also, unbiased estimation is not necessarily superior to biased estimation [17].

The main alternative to GLS estimation in this context is estimation that uses the pdf. When viewed as a function of the unknown model parameters (μ in our case) conditional on the observed data (y₁, y₂ in our case), the pdf is referred to as the likelihood. The maximum likelihood (ML) estimate therefore of course depends on the pdf for the errors. For example, the normal distributions are replaced with lognormal distributions, the ML estimate will change. Because the ML approach makes strong use of the assumed error distributions, the ML estimate is sensitive to the assumed error distribution. The ML method has desirable properties, including asymptotically minimum variance as the sample size increases. However, in our example, the sample size is tiny (two), so asymptotic results for ML estimates are not relevant. It still is possible that an ML estimator will be better for non-normal data than GLS. In this paper, “better” is defined as the mean squared error (MSE) of the estimator, which is well known to satisfy MSE = variance + bias². In some cases, biased estimators have lower MSE than unbiased estimators because the bias introduced is more than offset by a reduction in variance [17]. The other estimators considered here are two versions of Bayesian estimators, both of which rely on the likelihood as does ML.

In nearly all real situations, Σ is not known exactly, but must be estimated. Suppose the estimate Σ̂ = S, where S is the sample covariance matrix defined as ( n − 1 ) S = ∑ i = 1 n ( y 1 i − y ¯ 1 ) ( y 2 i − y ¯ 2 ) t (t denotes transpose) based on n pairs of y₁, y₂ values that are auxiliary data pairs separate from the y₁ = 1.5 and y₂ = 1.0 values. We will consider two examples.

Bayesian analysis requires a prior probability p(θ) for the parameters θ, and a likelihood f(D|θ) for the data given the parameter values. The result is a posterior probability π(θ|D) ∝ f(D|θ)p(θ) for the parameters θ. For many priors and likelihoods, calculating the posterior is difficult or impossible, but observations from the posterior can be obtained by Markov chain Monte Carlo (MCMC). MCMC is a Monte Carlo sampling scheme that accepts a candidate value θ′ with probability that depends on the relative probability of the current θ value and the candidate θ′ value. The goal of MCMC is to simulate many observations from the posterior, and use summary statistics such as posterior means and standard deviations to characterize the state of knowledge about the parameters. In our examples, θ is the scalar-valued mean which we denote throughout as μ.

ABC has arisen relatively recently in applications for which the likelihood f(D|θ) is intractable [22,23], but data can be simulated somehow, often relying on detailed simulations of physical processes. In our context, the simulation is very simple, based on realizations of the measurement error pdfs. And if some subset of the three underlying error distributions described in the background section are non-normal, leading to a complicated bivariate distribution for y₁, y₂ that could be difficult to derive analytically, then ABC is attractive. Predating ABC is a nonBayesian (“frequentist”) method referred to as “inference for implicit statistical models” [21,24] that was based on density estimation. We use the term ABC to include any Bayesian method in a situation for which an analytical likelihood is not available, but simulations can produce realizations from the likelihood.

In the common version of ABC, because data can be simulated from the pdf, candidate values θ′ can be accepted with a probability that depends on how close certain summary statistics such as the first few moments computed from the simulated data having parameter value θ′ are to the corresponding summary statistics from the real data. Some sources [24] reserve the term ABC for methods that rely on such summary statistics, so would not use the term ABC with density estimation. Our combination of MCMC with density estimation appears to be a new combination of ideas, though some might not refer to it as ABC. Regardless of the jargon, the method will be clearly delineated below.

As a simple example to motivate our modified form of ABC, suppose y₁, y₂, …, y_n are independently and identically distributed as N(μ, σ) and it is known that σ = 0.1. Introductory statistics texts usually prove that the sample mean y̅ is the ML estimator (MLE) for μ and more advanced texts investigate properties of MLEs. And, Equation (3) with a diagonal Σ (because in this example the y_i are independent) can be used to show that y̅ is the GLS estimator so is also the BLUE and because the data is normal in this simple example (but not in our extended PPP examples below), y̅ is also the MVUE. Specify the prior p(μ) to be N ( μ prior , σ prior 2 ). Then, because the normal prior for μ is the “conjugate” prior (meaning that the posterior distribution is the same type of distribution as the prior distribution) for the normal likelihood, the posterior π(θ|D) is N ( μ post , σ post 2 ), where μ post = 0.1 2 0.1 2 + σ prior 2 μ prior + σ prior 2 0.1 2 + σ prior 2 y ¯.

Our modified ABC that relies on density estimation has the following steps:

Determine a lower (L) and upper bound (U) for the parameter μ. In this example, having observed y₁, y₂, …, y_n1 and knowing σ = 0.1, we can assume μ lies between say L = y̅ − 5σ/n₁ and U = y̅ + 5σ/n₁.

Figure 1 illustrates the resulting log likelihood for μ = 1, μ = 1.1, n₁ = 10, and n₂ = 1000. Notice that the log likelihood peaks very near the true μ value in both cases.

To check our implementation, we specified the prior p(μ) to be N ( μ prior , σ prior 2 ). Then, because the normal prior for μ is the “conjugate” prior (meaning that the posterior distribution is the same type of distribution as the prior distribution) for the normal likelihood, the posterior π(θ|D) is N ( μ post , σ post 2 ). The ABC estimate of the posterior mean can then be compared to the mean of the exact posterior π(θ|D) to ensure proper implementation. In this case, our implementation of ABC also leads to μ̂ ≈ y̅ because we used σ = 0.1, σ_prior = 10, and n₁ = 10, so the posterior distribution has mean essentially equal to y̅. Specifically, in 100 simulations of the entire process of generating n₁ = 10 observations, and applying density estimation to estimate the likelihood using n₂ = 1000 observations resulted in no statistical difference (based on a paired t-test) between the 100 values of the exact Bayesian μ̂, the ABC estimate of μ̂, and μ̂ = y̅. Readers can duplicate our results using the R functions listed in the Appendix.

To our knowledge, this simple use of density estimation to substitute for having an analytical form for the likelihood has not been studied in the ABC literature. However, provided the data can be simulated sufficiently fast to acquire many observations (n₂ = 1000 was sufficiently large in this toy problem), and provided the dimension of the data is not too large, density estimation is feasible and effective. In the PPP case, we have only two variables, y₁ and y₂, so certainly 2-dimensional density estimation is feasible. We note here that summary statistics such as moments from the real and simulated data in the MCMC accept/reject steps in typical ABC implementations are fast to compute, but convergence criteria and measures of adequacy are still being developed [23].

This section again uses the slightly modified form of ABC that relies on density estimation rather than using summary statistics computed from the real and simulated data, but MCMC is applied in the context of PPP.

As mentioned in Section 2, suppose from two experiments m₁ = μ + ∊_R1 and m₂ = μ + ∊_R2, where ∊_R1 is zero-mean random error in m₁ and similarly for ∊_R2. Then if y₁ = m₁ + ∊_S and y₂ = m₂ + α∊_S, where ∊_S ∼ N(0, σ_S) and α is any positive scale factor other than 1, the covariance matrix Σ of (y₁, y₂) can lie in the PPP region [2].

To implement our version of ABC, many (y₁, y₂) realizations are simulated from each of many values of μ Given the observed (y₁, y₂) and its known covariance Σ, we can again set up good L and U values for the grid of candidate μ values.

A key advantage of ABC is that any probability distribution can be easily accommodated for any of ∊_R1, ∊_R2, and ∊_S. Here are two examples.

Reference [8] presented an ad hoc procedure to estimate μ as follows.

Using a model of the two experiments that produce y₁, y₂ such as that by [4] in Section 2, generate thousands of a y₁, y₂ pairs but regard them as being μ₁, μ₂ pairs. See below for “justification” in switching from y₁, y₂ to μ₁, μ₂. Because of the strong prior belief that μ₁ ≈ μ₂, restrict attention to bivariate points (y₁, y₂) lying very near the diagonal, defining μ̂ as the value that maximizes the estimated probability density of the scalar-values of either y₁ or y₂ (or both y₁ and y₂) satisfying |y₁ − y₂| < ∊ for some small ∊ near 0. See Figure 3. As an important aside, recall [2] showed that PPP cannot occur with this error model (Section 2). However, because of how Σ is estimated (using m₁ or m₂ to estimate μ_m), PPP appears to occur with this error model.

In one dimension, the analogous ad hoc procedure for a sample of size n from a normal distribution, is to assume μ ∼ N(y̅, σ²/n) in the case of unknown μ and known variance σ² with y ¯ = ∑ i = 1 n y i / n. The formally correct posterior for μ depends on the likelihood and on the prior, but for a normal likelihood y ∼ N(μ, σ²) and a normal prior μ ∼ N(μ_p, τ²), as the prior variance τ² → ∞, the prior becomes “flat,” and the ad hoc procedure is well known to be correct in an asymptotic sense.

Although this ad hoc approach in two dimensions is a recipe that can be followed to produce μ̂, it is statistically incomplete. It can be completed and justified by adopting a two dimensional prior for μ. That is, to regard y₁, y₂ pairs as being μ₁, μ₂ pairs requires that y₁, y₂ pairs be interpreted as being observations from the posterior distribution for μ₁, μ₂ in a Bayesian sense. Second, to formalize the notion of restricting attention to bivariate points (μ₁, μ₂) lying very near the diagonal, we must define a prior probability distribution. Specifically, assume (5) ( y 1 y 2 ) ∼ N ( μ , ∑ )where μ = ( μ 1 μ 2 ) and ∑ = ( σ 1 2 ρ σ 1 σ 2 ρ σ 1 σ 2 σ 2 2 )

Let the prior for μ in Equation 5 be (6) ( μ 1 μ 2 ) ∼ N ( μ prior , ∑ prior )where μ prior = ( μ p 1 μ p 2 ) and ∑ prior = ( σ p 1 2 ρ σ p 1 σ p 2 ρ σ p 1 σ p 2 σ p 2 2 ) .

The recipe to restrict attention to simulated values satisfying y₁ ≈ y₂ is ad hoc, but for a given choice of ∊ to define y₁ ≈ y₂ as satisfying |y₁ − y₂| < ∊, there is a defensible Bayesian posterior corresponding to a choice for Σ_prior. And, given (7) y = ( y 1 y 2 )the posterior distribution is (8) ( μ 1 μ 2 ) ∼ N ( μ post , ∑ post )where μ post = ( ∑ − 1 + ∑ prior − 1 ) − 1 ( ∑ − 1 y + ∑ prior − 1 μ prior )and (9) ∑ post = ( ∑ − 1 + ∑ prior − 1 ) − 1

For example, choosing (10) ∑ prior = ( 100 100 ( 1 − .01 j ) 100 ( 1 − .01 j ) 100 )results in (μ₁, μ₂) estimates of (1.49, 1.00), (1.08, 0.92), and (0.89, 0.89) for j = 1, 2, and 3 or larger, respectively, for any moderate value of μ_prior such as 0.1 to 10. The prior can be chosen such that the resulting estimate of μ₁ is within a small ∊ of the estimate of μ₂. Notice that the (0.89, 0.89) values for the (μ₁, μ₂) estimate agrees with the GLS method for j ≥ 3 in Equation 11 as claimed for the ad hoc approach of choosing (y₁, y₂) pairs along the diagonal [8].

This Bayesian approach revises the ad hoc recipe and is easily implemented using MCMC for any likelihood. It is sometimes difficult to calculate the likelihood for complicated combinations of distributions involving for example sums and products of random variables (which can arise in modifications of the prescription to generate y₁, y₂ pairs). An appeal of the modified ad hoc approach is that the likelihood does not need to be computed. Instead, one need only be able to generate many y₁, y₂ pairs. This is the same appeal of the ABC approach in our context.

To summarize this section, the ad hoc approach in [8] regards simulated (y₁, y₂) pairs from any distribution as providing an estimate of μ by specifying a small ∊ and computing the mean of all scalar-valued y₁ and y₂ satisfying |y₁ − y₂| < ∊ (or by finding the value of μ that maximizes the likelihood of the scalar data that arise from concatenating all y₁ and y₂ values that are similar). This approach has a corresponding formally correct Bayesian interpretation, but is easier to implement than the corresponding correct Bayesian implementation with a particular choice of the prior pdf.

For Example 5.1 in Section 5, for which the distributions for ∊_R1, ∊_R2, and ∊_S are normal, the estimates arising from the Bayesian posterior maximum probability and the posterior mean are 0.89 and 0.89, respectively, both the same as the GLS estimate.

For Example 6.2, although the RMSEs from low to high (low is good) are ABC, GLS, and alternate Bayes, for this example, we do not make any general performance claims ranking GLS, ABC, and the modified ad hoc (“alternate Bayes”) approaches. Instead, the goal is to present statistically defensible approaches and demonstrate that they do perform differently.

Figure 4 illustrates the ad hoc and revised procedures in the case of the normal distribution. The top plot shows the accepted (μ₁, μ₂) pairs in the metrop MCMC (in red), which tightly follow the diagonal line. The bottom plot shows the estimated density of the accepted pairs using the Bayesian strategy and using the ad hoc strategy. Notice that the two densities are not the same. However, in this case, both methods arrive at μ̂ = 0.89 as the estimate for μ.

There will almost always be estimation error in Σ̂, and often the measurement errors are non-normal. Therefore, we considered the following three topics: (1) alternatives to GLS when there is estimation error in Σ̂, (2) approximate Bayesian computation [22] to provide estimators other than GLS that use the estimated likelihood in the case of non-normal error models that make ML estimation difficult, and (3) completion of an incompletely specified ad hoc approach to deal with non-normal likelihoods [8].

Regarding (1), it was illustrated in the PPP case that weighted estimates do not always outperform equally-weighted estimates when there is estimation error in the weights. We have already noted that estimation of Σ in Equation 1 introduces apparent PPP when PPP does not actually occur.

Regarding (2), ABC based on density estimation to estimate the required likelihood was introduced. We showed that for some likelihoods, Bayesian estimation can outperform GLS, and that for some data realizations, the ABC estimator fell within the range of the simulated x₁, x₂ data pairs. Of course GLS provides a good estimate μ̂ in general because of its well-known BLUE property (and UMVUE if the data is normal) and in particular for the PPP problem if the covariance Σ is well known.

Regarding (3), ad hoc estimators based on incomplete statistical reasoning did produce values that lie within the data range [8]. We modified the incomplete approach by another use of Bayesian reasoning that involved a two-component prior. We concur that for some data realizations, the ad hoc estimator can lie within the data range, but in the examples considered, the RMSE of this estimator was slightly higher than the RMSE of the GLS or ABC estimators.