Maximum Penalized-Likelihood Structured Covariance Estimation for Imaging Extended Objects, with Application to Radio Astronomy

Abstract

Image formation in radio astronomy is often posed as a problem of constructing a nonnegative function from sparse samples of its Fourier transform. We explore an alternative approach that reformulates the problem in terms of estimating the entries of a diagonal covariance matrix from Gaussian data. Maximum-likelihood estimates of the covariance cannot be readily computed analytically; hence, we investigate an iterative algorithm originally proposed by Snyder, O’Sullivan, and Miller in the context of radar imaging. The resulting maximum-likelihood estimates tend to be unacceptably rough due to the ill-posed nature of the maximum-likelihood estimation of functions from limited data, so some kind of regularization is needed. We explore penalized likelihoods based on entropy functionals, a roughness penalty proposed by Silverman, and an information-theoretic formulation of Good’s roughness penalty crafted by O’Sullivan. We also investigate algorithm variations that perform a generic smoothing step at each iteration. The results illustrate that tuning parameters allow for a tradeoff between the noise and blurriness of the reconstruction.

1. Introduction

The insatiable thirst of astronomers for increasingly high-resolution images of the cosmos has led to ingenious developments in both instrumentation and the processes employed to extract information from the available raw data. In optical astronomy, system resolution is, at first glance, limited by atmospheric turbulence. Undaunted, researchers have tackled challenges with a variety of tactics, including flexible mirrors that can rapidly bend to compensate for the fluctuating atmosphere, and sophisticated signal processing algorithms for combining multiple short exposures into a single image. At the comparatively low frequencies of interest in radio astronomy, the resolution of a single radio telescope is significantly limited by the difficulty and extraordinary cost involved in constructing antennas with large dishes. In the 1940s, this hurdle was overcome by correlating the signals from multiple antennas to form an interferometer. Diverse data can be collected by mounting the antennas on tracks or exploiting the rotation of the earth to take interferometric measurements at different positions [1,2].

The literature on radio astronomy is vast and rich. We have found the collections of papers edited by Goldsmith [3] and Felli and Spencer [4], as well as the text by Thompson, Moran, and Swenson [5], to be particularly helpful. Since the genesis of interferometric radio astronomy, researchers have exploited the connection between the correlation measurements and the Fourier transform of the object being imaged. Basically, each correlation measurement gives an estimate of one point of that Fourier transform. The position of the point in Fourier space depends on the vector separation of the two antennas associated with that measurement. For instance, an array of 27 elements (unequally spaced to avoid redundant position differences) provides ${27}^2=729$ Fourier points. In changing the position of the array elements (either by physically moving the elements or by waiting for the Earth to rotate), additional points may be obtained, but it is almost impossible to obtain all the points needed to form a good image by direct Fourier inversion. Simply setting the missing points to zero and applying an inverse Fourier transform yields a blurry image with complicated sidelobe structures, often called a “dirty map”. In addition, negative pixel values are often obtained, even though the intensity should be nonnegative. Radio astronomers have relied on two main tools, the CLEAN algorithm [6,7,8] and maximum entropy [9,10], to produce useful “clean maps”. Although the underlying mapping between the correlation data and the intensity map is linear, the CLEAN and maximum entropy algorithms are both nonlinear forms of processing.

This paper explores a radically different approach based on statistical estimation theory. It does not require or explicitly exploit the Fourier relationships that have dominated radio interferometry for the past fifty years. The method can deal with any reasonable linear relationship between the astronomical source and the raw measurements; the fact that these relationships are complex exponentials is somewhat incidental. Under the statistical model, the pixels that form the sky emit white Gaussian processes. The antenna elements see a mixture of these processes through a linear transformation and additive receiver noise. The signals seen at the antenna elements are then another set of Gaussian process, with correlations between the components. One can then write down the likelihood of the data given unknown powers and maximize the likelihood with respect to unknown parameters. This turns out to be a challenging maximization that does not yield to a frontal assault; hence, we adopt an expectation–maximization (EM) algorithm originally proposed by Snyder, O’Sullivan, and Miller [11] in the different but related context of imaging diffuse radar targets. This is the approach taken by Knapp et al. for radio astronomical imaging using vector sensors in [12].

The CLEAN algorithm requires the astronomer to choose a variety of parameters; the results of this algorithm are often highly dependent on the skill of the practitioner in choosing these parameters. By contrast, in adopting the maximum-likelihood framework, one can “turn the crank” of the EM algorithm without having to tune any such operational parameters. Our approach also fully exploits all statistical knowledge about the data collection.

One of the advantages of maximum-entropy techniques in traditional radio astronomy formulations is that the entropy functional ensures nonnegative estimates. Our approach, which is congruent with that in [12], has this same advantage; nonnegativity is automatically guaranteed by the form of the EM iterations.

To our knowledge, Leshem and van der Veen [13], along with their colleague Boonstra [14], presented the earliest reported work on a maximum-likelihood approach to radio astronomy. They consider imaging in the presence of strong human-made radio interference, for instance, from communication satellites; hence, their primary interest lies in performing maximum-likelihood inference for specific point sources. This yields close ties to the direction-of-arrival literature [15]. They present a simplifying approximation to the log-likelihood and a coordinate descent algorithm that treats already discovered point sources as colored noise. This differs, in both goal and execution, from our EM algorithm for pixel-based imaging. In [16], a maximum-likelihood approach was used to remove unwanted radio sources.

Our work is closer in spirit to that of the pioneering research of Mhiri et al. [17] and Zawadzki et al. [18], which also employed maximum-likelihood estimation for pixel-based imaging. Our work differs fundamentally in that we treat the radiation from each sky pixel as a random process and estimate the parameters of those processes, instead of treating the radiation as a nonrandom parameter. In drawing an analogy with maximum-likelihhood narrowband direction finding, Refs. [17,18] may be thought of as analogous with the “deterministic signal model” of Section III.A in [19], and our work may be thought of as analogous with the “random signal model” of Section III.B in [19].

We employ a Gaussian model for additive noise, whereas [17] employed a compound model consisting of a Gaussian random variable multiplied by an inverse gamma random variable. The algorithms we describe, including various forms of regulation, ensure nonnegative solutions, so an explicit nonnegativity constraint (as described in the last paragraph of Section 4.1 in [17]) is not required. Each iteration of our unconstrained algorithm may be expressed in closed form, unlike Equations (11) and (19) in [17].

Section 2 presents our statistical formulation of the radio astronomy problem, and Section 3 presents the EM algorithm for computing estimates using this model, as well as some simulation results. These experiments illustrate the need for regularization; various possibilities are explored in Section 4. Section 5 suggests some avenues for future exploration.

2. Problem Formulation

Suppose that the sensor array has K elements. Since radio astronomy antennas are quite expensive, it is common to either mount the antennas on tracks, which allow them to be moved, or to simply let the earth rotate, which places them at a new effective look position. Suppose we take data at M such look positions and we collect N snapshots at each position.

Anticipating discretization for computer implementation, we suppose that the sky is made up of I pixels. We will use a single index $i=1,\dots ,I$ to go over the the entire two-dimensional pixel array. Let ${\varphi }_i$ and ${\theta }_i$ denote the angles that pixel i forms with respect to a line perpendicular to and crossing the center of the array. Assume each pixel i radiates a white, complex, zero-mean Gaussian process $c_i(n,m),n=1,\dots ,N,m=1,\dots ,M$ with unknown variance ${\sigma }_i^2$. Let ${\sigma }^2={[{\sigma }_1^2,\dots ,{\sigma }_I^2]}^T$ and $\Sigma =\mathrm{diag}\left({\sigma }^2\right)$.

If ${\varphi }_i$ and ${\theta }_i$ are small so that small-angle approximations to trigonometric functions apply, each sensor sees the signal from $c_i$ with an associated phase shift $exp\left\{j2\pi [x_k\left(m\right){\varphi }_i+y_k\left(m\right){\theta }_i]\right\}$, where $(x_k\left(m\right),y_k\left(m\right))$ are the coordinates of sensor k at look position m (see pp. 6–24 of [20]). We can write the data received at sensor k as

$$r_k(n,m)=\sum _{i=1}^I{c}_i(n,m)exp\left\{j2\pi [x_k\left(m\right){\varphi }_i+y_k\left(m\right){\theta }_i]\right\}+w_k(n,m),$$

(1)

where $w_k(n,m)$ is white, complex, zero-mean Gaussian receiver noise with variance $N_0$.

Let $\mathbf{r}={[r_1,\dots ,r_K]}^T$. For convenience, we form matrices of complex exponentials ${\Gamma }^H\left(m\right),m=1,\dots ,M$, and express (1) for all k as

$$\mathbf{r}(n,m)={\Gamma }^H\left(m\right)\mathbf{c}(n,m)+\mathbf{w}(n,m),$$

(2)

where $\mathbf{c}={[c_1,\dots ,c_I]}^T$ and $\mathbf{w}={[w_1,\dots ,w_K]}^T$ are independent and identically distributed as $CN(0,\Sigma )$ and $CN(0,N_0\mathbf{I})$, independent with respect to each other, and independent with respect to n and m. This is analogous to Equation (13) in [12].

Note that for all n, $\mathbf{r}(n,m)\sim N(0,{\mathbf{K}}_{\mathbf{r}}\left(m\right))$, where

$${\mathbf{K}}_{\mathbf{r}}\left(m\right)={\Gamma }^H\left(m\right)\Sigma \Gamma \left(m\right)+N_0\mathbf{I}.$$

(3)

Estimating the intensity map $\Sigma$ brings us to the structured covariance estimation waters first charted by Burg et al. [21]. The log-likelihood for the data is

$$L_{id}(\Sigma )=-N\sum _{m=1}^{M}lndet{\mathbf{K}}_{\mathbf{r}}\left(m\right)-\sum _{m=1}^M\sum _{n=1}^N{\mathbf{r}}^H(n,m){{\mathbf{K}}_{\mathbf{r}}}^{-1}\left(m\right)\mathbf{r}(n,m).$$

(4)

3. An Algorithm for Maximum-Likelihood Imaging

Maximum-likelihood imaging requires us to find the diagonal $\Sigma =diag\left({\sigma }^2\right)$ that maximizes (4). Since no simple solution is evident for this difficult optimization problem, we turn to the expectation–maximization algorithm. The algorithm explored here is a trivial extension of the EM algorithm designed by Snyder, O’Sullivan, and Miller [11] in the context of radar imaging. It is similar to formulations presented in [12,22,23,24,25,26,27] and Chapter 6 of the PhD thesis by Robey [28]. In the context of this EM algorithm [29], $\mathbf{r}(n,m)$ is called the incomplete data, and (4) is the incomplete data log-likelihood.

To formulate an EM algorithm, one postulates a set of complete data that, if available, would make the maximization problem tractable. Here, we take the complete data to be $\left\{\mathbf{c}\right(n,m),\mathbf{w}(n,m),n=1,\dots ,N,m=1,\dots ,M\}$, where $\mathbf{c}(n,m)\sim CN(0,\Sigma )$ and $\mathbf{w}(n,m)\sim CN(0,N_0\mathbf{I})$. The classic EM formulation requires that there be a many-to-one mapping from the complete data to the incomplete data; this is provided by (2). The complete data log-likelihood is

$$\begin{array}{ccc}\hfill L_{cd}(\Sigma )& =& -NMlndet\Sigma -\sum _{m=1}^M\sum _{n=1}^N{\mathbf{c}}^H(n,m){\Sigma }^{-1}\mathbf{c}(n,m)\hfill \\ & =& -NM\sum _{i=1}^{I}ln{\sigma }_i^2-\sum _{m=1}^M\sum _{n=1}^N\sum _{i=1}^I{\displaystyle \frac{|c_i{(n,m)|}^2}{{\sigma }_i^2}}.\hfill \end{array}$$

(5)

Let Q denote the expectation of the complete data log-likelihood given the incomplete data and the estimate from the previous iteration:

$$\begin{array}{ccc}\hfill Q[\Sigma |{\Sigma }^{\left(old\right)}]& \stackrel{\mathrm{df}}{=}& E\left[L_{cd}(\Sigma )\right|{\Sigma }^{\left(old\right)},\mathbf{r}]\hfill \\ & =& -NM\sum _{i=1}^{I}ln{\sigma }_i^2-N\sum _{m=1}^M\sum _{i=1}^I{\displaystyle \frac{E\left[\right|c_i\left(m\right){|}^2|{\Sigma }^{\left(old\right)},\mathbf{r}]}{{\sigma }_i^2}}.\hfill \end{array}$$

(6)

Notice that in the second term, since the N snapshots of ${\mathbf{c}}_i$ are independent, we have not notated the explicit dependence of ${\mathbf{c}}_i$ on n in the expectation.

At each iteration of the EM algorithm, we obtain the new estimate by maximizing Q:

$${\Sigma }^{\left(new\right)}=arg\underset{\Sigma }{max}Q[\Sigma |{\Sigma }^{\left(old\right)}].$$

(7)

The derivative of (6) with respect to ${\sigma }_i^2$, set to zero, is

$$-NM{\displaystyle \frac{1}{{\sigma }_i^2}}+N\sum _{m=1}^M{\displaystyle \frac{E\left[\right|c_i\left(m\right){|}^2|{\Sigma }^{\left(old\right)},\mathbf{r}]}{{\left({\sigma }_i^2\right)}^2}}=0,$$

(8)

which yields the simple update

$${\sigma }_i^{2\left(new\right)}={\displaystyle \frac{1}{M}}E\left[\right|c_i{\left(m\right)|}^2|{\Sigma }^{\left(old\right)},\mathbf{r}].$$

(9)

Computing the expectation in (9) is a standard problem addressed in estimation theory texts (see, for instance, Equation (7.112) on p. 303 of Scharf [30] or Equations (V.B.21) and (7.B.22) on p. 221 of Poor [31]). Its solution results in the explicit update

$$\begin{array}{ccc}\hfill {\sigma }_i^{2\left(new\right)}& =& {\sigma }_i^{2\left(old\right)}-{\displaystyle \frac{{\left[{\sigma }_i^{2\left(old\right)}\right]}^2}{M}}\sum _{m=1}^M[\Gamma \left(m\right){\mathbf{K}}^{-1}\left(m\right){\Gamma }^H\left(m\right)\hfill \\ & & -\Gamma \left(m\right){\mathbf{K}}^{-1}\left(m\right)\mathbf{S}\left(m\right){\mathbf{K}}^{-1}\left(m\right){\Gamma }^H{]}_{ii},\hfill \end{array}$$

(10)

where

$$\mathbf{K}\left(m\right)={\Gamma }^H\left(m\right){\Sigma }^{\left(old\right)}\Gamma \left(m\right)+N_0\mathbf{I}$$

(11)

and

$$\mathbf{S}\left(m\right)={\displaystyle \frac{1}{N}}\sum _{n=1}^N\mathbf{r}(n,m){\mathbf{r}}^H(n,m).$$

(12)

Notice that the data only enter into the inference via its empirical covariance $\mathbf{S}$. We write (10) in a way to show the symmetric structure of the matrix computations. In implementation, it is more efficient to compute the terms in the square brackets of (10) by calculating $\Xi \left(m\right)=\Gamma \left(m\right){\mathbf{K}}^{-1}\left(m\right)$ followed by $\Xi \left(m\right)[{\Gamma }^H\left(m\right)-\mathbf{S}\left(m\right){\Xi }^H\left(m\right)]$. Since only the diagonal terms are needed, the final step only requires I inner products.

Taking $N=1$ in (12) and $M=1$ in (10) yields the original algorithm [11] for forming radar images of diffuse targets. In [11], $\mathbf{c}$ is the reflectance of a radar scatterer, ${\sigma }^2$ is called its scattering function, and $\Gamma$ contains time-shifted and doppler-shifted versions of the transmitted waveform.

Simulations

Our simulations were meant to illustrate the overall operation of the EM algorithm, and were not intended to represent any particular real-world scenario. We assumed a 27-element array modeled after the Very Large Array (VLA) in New Mexico [32]. We chose the VLA for this example because it is well known, but the ideas in this paper could be applied to other array geometries. The array consists of three arms arranged in a Y pattern, with equal angular spacing between each arm. Each arm has nine elements. The distance of the nth antenna on each arm from the center of the array is proportional to $n^{1.716}$. The real VLA’s elements are mounted on tracks, so the overall size of the array can be changed. Here, we assume that the array was set at its largest configuration, in which the furthest elements on each arm are 21 km away from the center.

For simplicity, our simulated array departs from the real array in New Mexico in that we supposed the array can be rotated around its center to obtain different looks (or equivalently, the array is centered at one of the Earth’s poles). We supposed that $M=5$ different looks were taken, and that the array rotated 10 degrees between looks, for a total rotation of 40 degrees. Figure 1 shows the Fourier sampling pattern this generates in the traditional radio astronomy paradigm. At each look m, $N=500$ snapshots were taken to form the empirical covariance matrix $\mathbf{S}\left(m\right)$. Computations for all results in this paper were performed with MATLAB.

The top row of Figure 2 shows two ideal intensity functions used in our experiments. The bottom row shows the simple estimate

$${\widehat{\sigma }}_i^2=\mathrm{real}\left\{{\left[\sum _{m=1}^M\Gamma \left(m\right)\mathbf{S}\left(m\right){\Gamma }^H\left(m\right)\right]}_{ii}\right\},$$

(13)

which corresponds to the “dirty map” of traditional radio astronomy.

Figure 3 shows the results of EM iterations (10) on data simulated from these ideal intensity functions using the described setup, initialized with a uniform image. Although the results of the EM algorithm do not appear qualitatively much different than the dirty maps in terms of contrast, we see background artifacts being substantially reduced with further iterations.

4. Regularization Techniques

Although the estimates shown in Figure 3 are quite sharp, and the background chaff becomes less evident with increasing iterations, the reconstructed objects suffer from a spiky, noisy appearance. The roughness associated with increasing EM iterations is indicative of the ill-posed nature of the maximum-likelihood estimation of functions from limited data. In most numerical algorithms, accuracy improves as the discretization is refined. Here, the opposite is true; as the grid is refined, the problem becomes increasingly ill posed and the solutions increasingly ill behaved. This phenomenon, called dimensional instability by Tapia and Thompson [33], would manifest itself in any algorithm that maximizes the log-likelihood.

These issues have been extensively studied in the context of Poisson intensity estimation in applications such as medical imaging (see [34,35], and Chapter 3 in [36]). We propose adapting solutions previously applied to Poisson imaging to our radio astronomy problem. The methods explored in this paper are most appropriate for extended objects that do not consist of isolated point sources, such as galaxies and nebula; the need for such methods is outlined in the last paragraph of Section 4 in [12]. For fields of isolated stars, L1-norm regularization, as used in [17], may be more appropriate.

4.1. The Method of Sieves

One approach is to restrict the solution to lie in a restricted subset called a sieve [37]. One possibility is to require it to be a weighted sum of basis functions. Gaussian basis functions have been successful in emission tomography [35,38,39]. In the radio astronomy problem and the related radar imaging problem originally considered by Snyder, O’Sullivan, and Miller [11], the EM algorithm of Section 3 can be extended to incorporate such a sieve constraint. Moulin and co-workers [40,41] presented this extended algorithm (for N = 1, M = 1), along with specific examples employing B-splines for radar imaging. Radio astronomers employing the CLEAN algorithm typically smooth the result by convolving it with a so-called “clean beam”. This clean beam may provide a natural starting point for choosing the sieve in maximum-likelihood estimation. Wavelet thresholding techniques have also been explored for denoising speckled radar imagery [42]. We mention these possibilities for completeness but will not consider them further here.

4.2. Regularization via Penalties

Another approach to regularization is to subtract a penalty $\Phi (\Sigma )$, which discourages unacceptably rough estimates, and to maximize the penalized likelihood

$$P(\Sigma )=L_{id}(\Sigma )-\alpha \Phi (\Sigma ).$$

(14)

Bayesians may think of this as maximum-posterior estimation using a (usually improper) prior proportional to $exp[-\alpha \Phi (\Sigma \left)\right]$.

The EM algorithm can be easily extended to maximize this penalized likelihood. The function Q (6) is generalized to

$$\begin{array}{ccc}\hfill Q_P[\Sigma |{\Sigma }^{\left(old\right)}]& \stackrel{\mathrm{df}}{=}& E\left[L_{cd}(\Sigma )\right|{\Sigma }^{\left(old\right)},\mathbf{r}]-\alpha \Phi (\Sigma )\hfill \\ & =& -NM\sum _{i=1}^{I}ln{\sigma }_i^2-N\sum _{m=1}^M\sum _{i=1}^I{\displaystyle \frac{E\left[\right|c_i\left(m\right){|}^2|{\Sigma }^{\left(old\right)},\mathbf{r}]}{{\sigma }_i^2}}-\alpha \Phi (\Sigma ).\hfill \end{array}$$

(15)

To maximize (15), we can take derivatives (analogous to (8)) and solve the set of equations

$$-NM{\displaystyle \frac{1}{{\sigma }_i^2}}+N\sum _{m=1}^M{\displaystyle \frac{E\left[\right|c_i\left(m\right){|}^2|{\Sigma }^{\left(old\right)},\mathbf{r}]}{{\left({\sigma }_i^2\right)}^2}}-\alpha {\displaystyle \frac{\partial \Phi (\Sigma )}{\partial {\sigma }_i^2}}=0,$$

(16)

alternatively written as

$$-{\displaystyle \frac{1}{{\sigma }_i^2}}+{\displaystyle \frac{{\sigma }_i^{2\left(uc\right)}\left({\Sigma }^{\left(old\right)}\right)}{{\left({\sigma }_i^2\right)}^2}}-{\displaystyle \frac{\alpha }{NM}}{\displaystyle \frac{\partial \Phi (\Sigma )}{\partial {\sigma }_i^2}}=0,$$

(17)

where ${\sigma }^{2\left(uc\right)}\left({\Sigma }^{\left(old\right)}\right)$ is the result of the unconstrained update specified by (10). At each iteration, we take the updated value ${\Sigma }^{\left(new\right)}$ to be the $\Sigma$ that solves (17).

4.2.1. Entropy Functionals

Beginning with Frieden [43], numerous authors [44,45,46,47,48,49,50,51,52,53,54] have proposed regularizing a variety of inverse problems with nonnegativity constraints via the entropy functional

$${\Phi }_E(\Sigma )=\sum _i{\sigma }_i^{2}ln{\sigma }_i^2.$$

(18)

At each iteration, the new $\Sigma$ is found by finding the zero of

$$-{\sigma }_i^2+{\sigma }_i^{2\left(uc\right)}\left({\Sigma }^{\left(old\right)}\right)-{\displaystyle \frac{\alpha }{NM}}{\left({\sigma }_i^2\right)}^2(1+ln{\sigma }_i^2)$$

(19)

for each i. This is convenient because the solution is decoupled from pixel to pixel. Molina and Ripley [55] suggest that entropy “corresponds to a rather peculiar prior, since it depends only on the marginal distribution of grey levels and not on their spatial locations. It is thus surprising that maximum entropy solutions appear smooth in many published examples”. Donoho, Johnstone, Hoch, and Stern [56] offer a highly practical discussion on how entropy regularization operates in practice. They suggest that nonlinearities of the form induced by (19) encourage a “shrinking” of estimates toward a nominal value of $1/e$. Narayan and Nityanada [57] studied a general class of functions that include (18) as a special case and have similar effects on the reconstruction.

It is important to avoid a potential misunderstanding. Maximizing the penalized likelihood (14), using the entropy penalty (19) and the loglikelihood given by (4), is not equivalent to “maximum entropy” as traditionally practiced by radio astronomers. In traditional maximum entropy, one maximizes $-{\sum }_i{\sigma }_i^{2}ln{\sigma }_i^2$ subject to agreement with the correlation data. This agreement is most often measured using a least-squares criterion; this could be viewed as maximizing a penalized likelihood where the “likelihood” has an implied Gaussian form, with the correlation measurements assumed to be corrupted by additive white Gaussian noise, which is not at all like (4). The empirical correlation would be more accurately modeled as following a Wishart distribution [58].

Our experiments suggest that the entropy functional does not seem to be offer significant improvement over unconstrained estimates in this context, so we omit those results here.

4.2.2. Good’s Roughness

Good’s roughness penalty [59] was originally formulated for smoothing estimates in nonparametric probability density estimation; a thorough analysis in this context is given by Tapia and Thompson [33]. Following the suggestion of Snyder and Miller ([34], Section II.1), Good’s roughness was later applied to closely related problems of Poisson intensity estimation in PET [60], SPECT [61,62,63,64], and optical-sectioning microscopy [65]. It is applied to radar imaging in Section 5.2.2 of [66].

In the next few subsections, to conveniently express the penalties, we will index ${\sigma }^2$ using two spatial coordinates, as in ${\sigma }_{i_1,i_2}^2$, $i_1=1,\dots ,I_1$, $i_2=1,\dots ,I_2$, where $I=I_1{I}_2$. The penalties we now explore are most easily interpreted in their original continuous form, so we will also let ${\sigma }^2(·)$ or ${\sigma }^2(·,·)$ denote the continuous functions underlying the discrete representations. In one dimension, Good’s continuous first-order roughness penalty [59] may be written in several equivalent ways:

$$\begin{array}{ccc}\hfill 4\int {\left[{\displaystyle \frac{d}{dx}}\sqrt{{\sigma }^2\left(x\right)}\right]}^{2}dx& =& \int {\sigma }^2\left(x\right){\left[{\displaystyle \frac{d}{dx}}ln{\sigma }^2\left(x\right)\right]}^{2}dx\hfill \\ \hfill =\int {\displaystyle \frac{{\left({\sigma }^2\right)}^{\prime }\left(x\right)}{{\sigma }^2\left(x\right)}}dx& =& -\int {\sigma }^2\left(x\right){\displaystyle \frac{d^2}{d{x}^2}}ln{\sigma }^2\left(x\right)dx.\hfill \end{array}$$

(20)

The last equality in (20), which was established independently in [67,68], follows from integration by parts. Consider the rightmost expression. The penalty is straightforwardly extended to two dimensions (see pp. 155–156 of [36] or Section 3 of [60]):

$$-\int \int {\sigma }^2(x,y)\left({\displaystyle \frac{{\partial }^2}{\partial x^2}}+{\displaystyle \frac{{\partial }^2}{\partial y^2}}\right)ln{\sigma }^2(x,y)dxdy.$$

(21)

Discretizing (21) yields ${\Phi }_G\left(f\right)=$

$$-\sum _{i_1,i_2}{\sigma }_{i_1,i_2}^2[ln{\sigma }_{i_1+1,i_2}^2+ln{\sigma }_{i_1-1,i_2}^2+ln{\sigma }_{i,i_2+1}^2+ln{\sigma }_{i_1,i_2-1}^2-4ln{\sigma }_{i_1,i_2}^2].$$

(22)

O’Sullivan [67] noted that the discretized penalty (22) has an appealing information-theoretic interpretation in terms of I-divergences between neighboring pixel values.

At each iteration, the new $\Sigma$ may be found by solving a set of nonlinear difference equations:

$$\begin{array}{ccc}\hfill 0& =& -{\displaystyle \frac{1}{{\sigma }_{i_1,i_2}^2}}+{\displaystyle \frac{{\sigma }_{i_1,i_2}^{2\left(un\right)}\left({\Sigma }^{\left(old\right)}\right)}{{\left({\sigma }_{i_1,i_2}^2\right)}^2}}\hfill \\ & +& {\displaystyle \frac{\alpha }{NM}}[(ln{\sigma }_{i_1+1,i_2}^2+ln{\sigma }_{i_1-1,i_2}^2+ln{\sigma }_{i_1,i_2+1}^2+ln{\sigma }_{i_1,i_2-1}^2-4ln{\sigma }_{i_1,i_2}^2)\hfill \\ & & +{\displaystyle \frac{1}{{\sigma }_{i_1,i_2}^2}}({\sigma }_{i_1+1,i_2}^2+{\sigma }_{i_1-1,i_2}^2+{\sigma }_{i_1,i_2+1}^2+{\sigma }_{i_1,i_2-1}^2-4{\sigma }_{i_1,i_2}^2)].\hfill \end{array}$$

(23)

Figure 4 shows the results of 1000 iterations of the EM algorithm using Goods’s roughness penalty for $\alpha /\left(NM\right)=0.002$ and $\alpha /\left(NM\right)=0.005$.

4.2.3. Silverman’s Roughness

Inspired by the work of Good and Gaskins [59], Silverman [69] suggested alternative penalties that employ differential operators of the logarithm of the function. Like Good’s roughness, Silverman proposed his penalty in the context of density estimation. We consider the special case

$$\int \int {\left[\left({\displaystyle \frac{\partial }{\partial x}}+{\displaystyle \frac{\partial }{\partial y}}\right)ln{\sigma }^2(x,y)\right]}^{2}dxdy.$$

(24)

In the context of optical astronomy, Molina and Ripley [55] suggest smoothing the logarithm of the image, noticing that “astronomers tend to look at the raw data on a logarithmic scale (by choosing contour levels in a geometric progression) except when looking at details at near background level”. Section 5.2.3 in [66] considers the application of (24) to radar imaging.

Discretizing (24) yields

$${\Phi }_S\left(f\right)=\sum _{i_1,i_2}[{(ln{\sigma }_{i_1+1,i_2}^2-ln{\sigma }_{i_1,i_2}^2)}^2+{(ln{\sigma }_{i_1,i_2+1}^2-ln{\sigma }_{i_1,i_2}^2)}^2].$$

(25)

Using this penalty requires solving the following set of nonlinear equations at each iteration of the EM algorithm:

$$\begin{array}{ccc}\hfill 0& =& -{\displaystyle \frac{1}{{\sigma }_{i_1,i_2}^2}}+{\displaystyle \frac{{\sigma }_{i_1,i_2}^{2\left(un\right)}\left({\Sigma }^{\left(old\right)}\right)}{{\left({\sigma }_{i_1,i_2}^2\right)}^2}}\hfill \\ & +& {\displaystyle \frac{\alpha }{NM}}{\displaystyle \frac{2}{{\sigma }_{i_1,i_2}^2}}[ln{\sigma }_{i_1+1,i_2}^2+ln{\sigma }_{i_1-1,i_2}^2+ln{\sigma }_{i_1,i_2+1}^2+ln{\sigma }_{i_1,i_2-1}^2-4ln{\sigma }_{i_1,i_2}^2].\hfill \end{array}$$

(26)

Figure 5 shows the results of 1000 iterations of the EM algorithm using Silverman’s roughness penalty for $\alpha /\left(NM\right)=0.002$ and $\alpha /\left(NM\right)=0.005$.

4.2.4. General Markov Random Fields

When discretized, Good’s and Silverman’s roughness penalties can be thought of as inducing a prior with a nearest-neighbor Markov random field structure. A wide variety of such priors have been proposed for image reconstruction [70] that could be tried here. For instance, the use of the square in (20) and (24) results in reconstructions that, while less noisy, have smoothed edges. Employing powers less than two results in a penalty that tends to smooth continuous regions while better preserving edges. For simple penalties employing the derivative of the function (instead of the square root or the logarithm as above), this corresponds to a Markov random field with the popular generalized Gaussian distribution [71,72]. Good’s roughness and Silverman’s roughness can be generalized in a similar way. We leave this topic for future work.

4.3. Regularization via General Smoothing Steps

Notice that in the method of penalties, a simple modification of the EM algorithm can be used to produce the penalized likelihood estimates; the resulting algorithm happens to amount to nonlinearly smoothing the result of the maximization step at each iteration before returning to the expectation step. In the fields of emission tomography and stereology, Silverman et al. [73] suggested experimenting with different kinds of smoothing steps. For arbitrary choices of smoothing, the resulting expectation–maximization–smoothing (EMS) algorithm will not, in general, correspond to a particular penalized likelihood method. In addition, it is difficult to determine exactly what such algorithms converge to, if they converge at all. However, considering the success illustrated in [73] for Poisson intensity estimation, we are motivated to explore other kinds of smoothing in our radio astronomy application. One such example is shown in Figure 6, in which the smoothing step is a simple nearest-neighbor linear filter defined by

$$\begin{array}{ccc}\hfill {\left(Lf\right)}_{i_1,i_2}& =& f_{i_1,i_2}+\alpha [f_{i_1+1,i_2}+f_{i_1-1,i_2}+f_{i_1,i_2+1}+f_{i_1,i_2-1}]\hfill \\ & & +{\displaystyle \frac{\alpha }{\sqrt{2}}}[f_{i_1+1,i_2+1}+f_{i_1-1,i_2+1}+f_{i_1-1,i_2+1}+f_{i_1-1,i_2-1}],\hfill \end{array}$$

(27)

where $\alpha$ here is a parameter that controls the amount of smoothing.

Applying a linear smoothing step in the EM iteration for emission tomography has a natural relationship to minimizing a penalized likelihood in which the penalty is quadratic in the square root of the intensity (see [74] and Section 5 in [73]), yielding a rather unexpected connection to Good’s roughness of Section 4.2.2. Surprisingly, neither the authors nor the discussants in [73] refer to the work of Good and Gaskins or to its later application in emission tomography. The connection is explicitly noted by Eggermont and LaRiccia [75], however. Latham and Anderssen [76] provided some deep analysis of linear smoothing in the emission tomography iteration. We know of no equivalent existing analysis of adding a linear smoothing step to (10).

Eggermont and LaRiccia [75] proposed smoothing using nonlinear operation $\mathfrak{N}$, constructed from a linear smoother L according to $\mathfrak{N}\left\{f\right\}=exp\left(L\right\{lnf\left\}\right)$. When applied to density or Poisson intensity estimation, the resulting EMS algorithm maximizes a particular functional they call a “modified log-likelihood”. This contrasts with other EMS algorithms that do not necessarily correspond to minimizing some particular functional. To our knowledge, there is no parallel body of analysis of EMS algorithms for structured covariance estimation; it remains a wide open area for future work.

For another example, the left column of Figure 7 shows the results of an EMS algorithm that uses a 3 by 3 median filter in the smoothing step. Observe how the EMS algorithm with median smoothing maintains sharp edges and yields a cartoon-like reconstruction; also note that this is not equivalent to median filtering the raw ML estimate of the unconstrained EM algorithm, as shown in the right column of Figure 7.

5. Conclusions

The simulations presented in this paper offer preliminary insight into the behavior of structured covariance estimation techniques for forming radio astronomy images. Without regularization, the resulting estimates (as seen in Figure 3) are unacceptably rough. This is a result of the inherent ill-posed nature of the maximum-likelihood estimation of underlying continuous functions from limited data, and not a fault of the expectation–maximization algorithm itself. The results in Section 4.2.2, Section 4.2.3, and Section 4.3 illustrate that tuning parameters allow for a tradeoff between the noise and blurriness of the reconstruction. More detailed studies should be conducted, including studies with real data, which compare maximum penalized-likelihood estimates and EMS algorithm estimates with the results of traditional procedures such as the CLEAN algorithm and classical maximum-entropy algorithms.

Real radio astronomy measurements exhibit several phenomena that our statistical model currently does not account for. In some radio astronomy arrays, in order to reduce cost, the correlators are “hard-wired” in such a way that only certain correlation measurements are made; thus certain elements of $\mathbf{S}$ may be missing. Before correlation, the raw data are often significantly truncated, sometimes as coarsely as one bit, which has an unpleasant effect on the correlation measurement (see Sections 8.3 through 8.5 in [5]). It may be beneficial to explicitly model these effects and derive appropriate extensions to the EM algorithm presented here. There are also a host of calibration issues (particularly in Very Long Baseline Interferometry (VLBI), where the sensor array may span whole continents, [4]) that we have neglected that could also be incorporated into the model and the algorithm. Leshem and van der Veen [13] discuss these calibration issues from a maximum-likelihood viewpoint.

One of the main difficulties with EM algorithms in general is their slow convergence rates. A variety of EM variants have been proposed that boast faster convergence. For instance, the Space-Alternating Generalized EM (SAGE) algorithm of Fessler and Hero [77] updates the parameters in groups instead of all at once; each group has its own associated hidden data space, which would be a complete data space if the remaining parameters were known. Schulz has formulated several SAGE algorithms [78] for maximizing (4) that could be implemented for forming radio astronomy images. The one-dimensional experiments reported in [78] demonstrate the greater likelihood change per iteration of the SAGE algorithm; however, this particular implementation requires around three times the amount of computation per iteration as the original EM implementation. The details of such timing analyses are highly platform-dependent; one could imagine a specialized hardware architecture that could perform the SAGE iterations faster than the EM iterations, or vice versa. In any particular application, one should compare various implementations to find the algorithm with best performance on some given hardware architecture for that particular instance. SAGE algorithms are more convenient for maximizing penalized likelihoods using Markov random field-type penalties, because the SAGE recipe decouples the joint maximization step, which requires solving sets of equations like (23) and (26), into a series of single-parameter maximizations.

As Mhiri et al. pointed out in the right column of p. 3 in [17], their EM algorithm for nonrandom intensity estimation with a compound Gaussian noise model could be extended to other regularization techniques besides the L1-norm they used for point-like images. This could include the Goods/O’Sullivan or Silverman functionals explored in Section 4.2.2 and Section 4.2.3. In the other direction, for inspiration, L1-norm regularization could be used with our algorithm.

An appealing aspect of the statistical formulation is that it allows the prediction of estimator performance via Cramér–Rao bounds. In addition to giving scientists an understanding of the fundamental limits of the available instrumentation, it provides appealing criteria for making design decisions, such as the placement of receiver locations, in various applications. There has been a tremendous amount of work on studying such design choices [79,80] using other kinds of criteria. For large images, inverting the Fisher information can become cumbersome; Hero and Fessler [81] proposed an iterative algorithm for computing Cramér–Rao bounds on parameter subsets that avoids the explicit inversion of the Fisher information matrix via a complete/incomplete data formulation analogous to the EM algorithm.