Partial Exchangeability for Contingency Tables

Abstract

A parameter free version of classical models for contingency tables is developed along the lines of de Finetti’s notions of partial exchangeability.

1. Introduction

Consider cross-classified data: $X_1,X_2,\dots ,X_n$, where $X_a=(i_a,j_a)$, $i_a\in \left[I\right]$, $j_a\in \left[J\right]$ (for $\left[I\right]=\{1,2,\dots ,I\})$. Such data are often presented as an $I\times J$ contingency table $T=\left(t_{ij}\right)$ where $t_{ij}$ is the number of times $(i,j)$ happens. Suppose that $X_1,\dots ,X_n$ are exchangeable and extendible. Then, de Finetti’s theorem says:

Theorem 1.

For exchangeable ${\left\{X_i\right\}}_{i=1}^{\infty }$ taking values in $\left[I\right]\times \left[J\right]$

$$P[X_1=(i_1,j_1),\dots ,X_n=(i_n,j_n)]={\int }_{{\mathsf{\Delta }}_{I\times J}}\prod _{i,j}{p}_{ij}^{t_{ij}}\mu \left(dp\right),$$

where ${\mathsf{\Delta }}_{I\times J}=\{p_{ij}\ge 0,{\sum }_{i,j}{p}_{ij}=1\}$. The representing measure μ is unique.

A popular model for cross classified data is

$$p_{ij}={\theta }_i{\eta }_j.$$

Here is a Bayesian, parameter free, description.

Theorem 2.

For exchangeable ${\left\{X_i\right\}}_{i=1}^{\infty }$ taking values in $\left[I\right]\times \left[J\right]$, a necessary and sufficient condition for the mixing measure μ in Theorem 1 to be supported on ${\mathsf{\Delta }}_I\times {\mathsf{\Delta }}_J$ (with ${\mathsf{\Delta }}_I=\{p_1,\dots ,p_I:p_i\ge 0,{\sum }_i{p}_i=1\}$), so

$$P[X_1=(i_1,j_1),\dots ,X_n=(i_n,j_n)]={\int }_{{\mathsf{\Delta }}_I\times {\mathsf{\Delta }}_J}\prod {\theta }_i^{t_{i\ast }}{\eta }_j^{t_{\ast j}}\mu (d\theta ,d\eta ),$$

is that

$$\begin{array}{c}P[X_1=(i_1,j_1),X_2=(i_2,j_2),X_3=(i_3,j_3),\dots ,X_n=(i_n,j_n)]=\hfill \\ \hfill P[X_1=(i_1,j_2),X_2=(i_2,j_1),X_3=(i_3,j_3),\dots ,X_n=(i_n,j_n)].\end{array}$$

(1)

Condition (1) is to hold for any $n\ge 2$ and any $(i_a,j_a)$ $1\le a\le n$.

Proof. 

Condition (1) implies for all n and $h\ge 1$ (surpressing P a.s. throughout)

$$\begin{array}{c}P[X_1=(i_1,j_1),X_2=(i_2,j_2)|X_n=(i_n,j_n),\dots ,X_{n+h}=(i_{n+h},j_{n+h})]=\hfill \\ \hfill P[X_1=(i_1,j_2),X_2=(i_2,j_1)|X_n=(i_n,j_n),\dots ,X_{n+h}=(i_{n+h},j_{n+h})].\end{array}$$

(2)

Let $h\uparrow \infty$ and then $n\uparrow \infty$. Let $\mathcal{T}$ be the tail field of ${\left\{X_i\right\}}_{i=1}^{\infty }$. Then, Doob’s increasing and decreasing martingale theorems show

$$P[X_1=(i_1,j_1),X_2=(i_2,j_2)|\mathcal{T}]=P[X_1=(i_1,j_2),X_2=(i_2,j_1)|\mathcal{T}].$$

However, a standard form of de Finetti’s theorem says that, given $\mathcal{T}$, the ${\left\{X_i\right\}}_{i=1}^{\infty }$ are i.i.d. with $P[X_1=(i,j)]=p_{ij}$. Thus

$$p_{ij}{p}_{i^{\prime }{j}^{\prime }}=p_{i{j}^{\prime }}{p}_{i^{\prime }j}\mathrm{for}\mathrm{all}i,i^{\prime },j,j^{\prime }.$$

(3)

Finally, observe that (3) implies (writing $p_{i\ast }:={\sum }_j{p}_{ij}$, $p_{\ast j}:={\sum }_i{p}_{ij}$)

$$p_{i\ast }{p}_{\ast j}=\sum _{h,l}{p}_{ih}{p}_{lj}=\sum _{hl}{p}_{ij}{p}_{hl}=p_{ij}.$$

□

We remark the following points.

1

If $X_i=(Y_i,Z_i)$ condition (2) is equivalent to

$$\mathcal{L}((Y_1,Z_1),(Y_2,Z_2),\dots ,(Y_n,Z_n))=\mathcal{L}((Y_1,Z_{\sigma \left(1\right)}),\dots ,(Y_n,Z_{\sigma \left(n\right)}))$$

for all n and $\sigma \in S_n$ ($S_n$ is the symmetric group over $1,2,\dots ,n$). Since ${\left\{(Y_i,Z_i)\right\}}_{i=1}^n$ are exchangeable this is equivalent to saying the law is invariant under $S_n\times S_n$.

2

The mixing measure $\mu (d\theta ,d\eta )$ allows general dependence between the row parameters $\theta$ and column parameters $\eta$. Classical Bayesian analysis of contingency tables often chooses $\mu$ so that $\theta$ and $\eta$ are independent. A parameter free version is that under P, the row sums $t_{i\ast }$ and column sums $t_{\ast j}$ are independent. It is natural to weaken this to “close to independent” along the lines of [1] or [2]. See also [3].

3

Theorems 1 and 2 have been stated for discrete state spaces. By a standard discretization argument, they hold for quite general spaces. For example:

Theorem 3.

Let $X_i=(Y_i,Z_i)$ be exchangeable with $Y_i\in \mathcal{Y}$, $Z_i\in \mathcal{Z}$, complete separable metric spaces, $1\le i<\infty$. Suppose

$$\begin{array}{c}P[X_1\in (A_1,B_1),X_2\in (A_2,B_2),\dots ,X_n\in (A_n,B_n)]=\hfill \\ \hfill P[X_1\in (A_1,B_2),X_2\in (A_2,B_1),\dots ,X_n\in (A_n,B_n)]\end{array}$$

for all measurable $A_i,B_i$ and all n. Then,

$$P(X_1\in (A_1,B_1),\dots ,X_n\in (A_n,B_n))={\int }_{\mathcal{P}\left(\mathcal{Y}\right)\times \mathcal{P}\left(\mathcal{Z}\right)}\prod _1^n\theta \left(A_i\right)\eta \left(B_i\right)\mu (d\theta ,d\eta ),$$

with $\mathcal{P}\left(\mathcal{Y}\right),\mathcal{P}\left(\mathcal{Z}\right)$ the probabilities on the Borel sets of $\mathcal{Y},\mathcal{Z}$. The mixing measure μ is unique.

4.

Theorem 2 is closely related to de Finetti’s work in [1,4].

5.

De Finetti’s law of large numbers holds as well, in Theorem 3

$$\frac{1}{n}\sum {\delta }_{X_i}(A\times B)\to \mu (\theta \left(A\right),\eta \left(B\right)).$$

One object of this paper is to develop similar parameter free de Finetti theorems for widely used log-linear models for discrete data. Section 2 begins by relating this to an ongoing conversation with Eugenio Regazzini. Section 3 provides needed background on discrete exponential families and algebraic statistics. Section 4 and Section 5 apply those tools to give de Finetti style partially exchangeable theorems for some widely used hierarchical and graphical models for contingency tables. Section 6 shows how these exponential family tools can be used for other Bayesian tasks: building “de Finetti priors” for “almost exchangeability” and running the “exchange” algorithm for doubly intractable Bayesian computation. Some philosophy and open problems are in the final section.

2. Some History

I was lucky enough to be able to speak at Eugenio Regazzini’s 60^TH birthday celebration, in Milan, in 2006. My talk began this way:

≪ Hello, my name is Persi and I have a problem. ≫

For those of you not aware of the many “10 step-programs” (alcoholics anonymous, gamblers anonymous, …) they all begin this way, with the participants admitting to having a problem. In my case the problem was this:

(a)

After 50 years of thinking about it, I think that the subjectivist approach to probability, induction and statistics is the only thing that works;

(b)

At the same time, I have done a lot of work inventing and analyzing various schemes for generating random samples for things like contingency tables with given row and column sums; graphs with given degree sequences; …; Markov Chain Monte Carlo. These are used for things like permutation tests and Fisher’s exact test.

There is a lot of nice mathematics and hard work in (b) but such tests violate the likelihood principle and lead to poor scientific practice. Hence my problem (I still have it): (a) and (b) are incompatible.

There has been some progress. I now see how some of the tools developed for (b) can be usefully employed for natural tasks suggested by (a). Not so many people care about such inferential questions in these ’big data’ days. However, there are also lots of small datasets where the inferential details matter. There are still useful questions for people like Eugenio (and me).

3. Background on Exponential Families and Algebraic Statistics

The following development is closely based on [5], which should be considered for examples, proofs and more details.

Let $\mathcal{X}$ be a finite set. Consider the exponential family:

$$p_{\theta }\left(x\right)=\frac{1}{Z\left(\theta \right)}{e}^{\theta ·T\left(x\right)}\theta \in {\mathbb{R}}^d,x\in \mathcal{X}.$$

(4)

Here, $Z\left(\theta \right)$ is a normalizing constant and $T:\mathcal{X}\to {\mathbb{N}}^d-\left\{0\right\}$. If $X_1,X_2,\dots ,X_n$ are independent and identically distributed from (4), the statistic $t=T\left(X_1\right)+\cdots +T\left(X_n\right)$ is sufficient for $\theta$. Let

$${\mathcal{Y}}_t=\{(x_1,\dots ,x_n):T\left(x_1\right)+\cdots +T\left(x_n\right)=t\}.$$

Under (4), the distribution of $X_1,\dots ,X_n$ given t is uniform on ${\mathcal{Y}}_t$. It is usual to write

$$t=\sum _{i=1}^{n}T\left(X_i\right)=\sum _{\mathcal{X}}\sigma \left(x\right)T\left(x\right)\mathrm{with}\sigma \left(x\right)=\#\{i:T\left(X_i\right)=T\left(x\right)\}.$$

Let

$${\mathcal{F}}_t=\{f:\mathcal{X}\to \mathbb{N}:\sum f\left(x\right)T\left(x\right)=t\}.$$

Example 1.

For contingency tables $\mathcal{X}=\left\{\right(i,j):1\le i\le I,1\le j\le J\}.$ The usual model for independence has $T(i,j)\in {\mathbb{N}}^{I+J}$ a vector of length $I+J$ with two non zero entries equal 1. The 1’s in $T(i,j)$ are in the $i^{th}$ place and position j of the last j places. The sufficient statistic t contains the row and column sums of the contingency table associated to the first n observations. The set ${\mathcal{F}}_t$ is the set of an $I\times J$ tables with these row and column sums.

A Markov chain on this ${\mathcal{F}}_t$ can be based on the following moves: pick $i\ne i^{\prime }$, $j\ne j^{\prime }$ and change the entries in the current f by adding $\pm 1$ in pattern

$$\begin{array}{ccc}& j& j^{\prime }\\ i& +& -\\ i^{\prime }& -& +\end{array}or\begin{array}{cc}-& +\\ +& -\end{array}$$

This does not change the row sums and it does not change the column sums. If told to go negative, just pick new $i,i^{\prime },j,j^{\prime }$. This gives a connected, aperiodic Markov chain on ${\mathcal{F}}_t$ with a uniform stationary distribution. See [6].

Returning to the general case, an analog of $\begin{array}{cc}+& -\\ -& +\end{array}$ moves is given by the following:

Definition 1

(Markov basis). A Markov basis is a set of functions $f_1,f_2,\dots ,f_L$ from $\mathcal{X}$ to $\mathbb{Z}$ such that

$$\sum _{\mathcal{X}}{f}_i\left(x\right)T\left(x\right)=01\le i\le L$$

(5)

and that for any t and $f,f^{\prime }\in {\mathcal{F}}_t$ there are $(t_1,f_{i_1}),\dots ,(t_A,f_{i_A})$ with $t_i=\pm 1$, such that

$$f^{\prime }=f+\sum _{j=1}^A{t}_j{f}_{i_j}andf+\sum _{j=1}^a{t}_j{f}_{i_j}\ge 0,\mathrm{for}1\le a\le A.$$

(6)

This allows the construction of a Markov chain on ${\mathcal{F}}_t$: from f, pick $I\in \{1,2,\dots ,L\}$ and $t=\pm 1$ at random and consider $f+t{f}_I$. If this is positive, move there. If not, stay at f. Assumptions (5) and (6) ensure that this Markov chain is symmetric and ergodic with a uniform stationary distribution. Below, I will use a Markov basis to formulate a de Finetti theorem to characterize mixtures of the model (4).

One of the main contributions of [5] is a method of effectively constructing Markov bases using polynomial algebra. For each $x\in \mathcal{X}$, introduce an indeterminate, also called x. Consider the ring of polynomials $k\left[\mathcal{X}\right]$ in these indeterminates where k is a field, e.g., the complex numbers. A function $g:\mathcal{X}\to \mathbb{N}$ is represented as a monomial ${\mathcal{X}}^g={\prod }_{\mathcal{X}}{x}^{g\left(x\right)}$. The function $T:\mathcal{X}\to {\mathbb{N}}^d$ gives a homomorphism

$$\begin{array}{cc}\hfill {\phi }_T:k\left[\mathcal{X}\right]& ⟶k[t_1,\dots ,t_d]\hfill \\ \hfill x& ⟼t_1^{T_1\left(x\right)}{t}_2^{T_2\left(x\right)}\cdots t_d^{T_d\left(x\right)},\hfill \end{array}$$

extended linearly and multiplicatively (${\phi }_T(x+y)={\phi }_T\left(x\right)+{\phi }_T\left(y\right)$ and ${\phi }_T\left(x^2\right)={\phi }_T{\left(x\right)}^2$ and so on). The basic object of interest is the kernel of ${\phi }_T$:

$$I_T=\{p\in k\left[\mathcal{X}\right]:{\phi }_T\left(p\right)=0\}.$$

This is an ideal in $k\left[\mathcal{X}\right]$. A key result of [5] is that a generating set for $I_T$ is equivalent to a Markov basis. To state this, observe that any $f:\mathcal{X}\to \mathbb{Z}$ can be written $f=f_{+}-f_{-}$ with $f_{+}\left(x\right)=max(f\left(x\right),0)$ and $f_{-}\left(x\right)=max(-f\left(x\right),0)$. Observe $\sum f\left(x\right)T\left(x\right)=0$ iff ${\mathcal{X}}^{f_{+}}-{\mathcal{X}}^{f_{-}}\in I_T$. The key result is

Theorem 4.

A collection of functions $f_1,f_2,\dots ,f_L$ is a Markov basis if and only if the set

$${\mathcal{X}}^{f_{i+}}-{\mathcal{X}}^{f_{i-}}1\le i\le L$$

generates the ideal $I_T$.

Now, the Hilbert Basis Theorem shows that ideals in $k\left[\mathcal{X}\right]$ have finite bases and modern computer algebra packages give an effective way of finding bases.

I do not want (or need) to develop this further. See [5] or the book by Sullivant [7] or Aoki et al. [8]. There is even a Journal of Algebraic Statistics.

I hope that the above gives a flavor for what I mean by “working in (b) is hard honest work”. Most of the applications are for standard frequentist tasks. In the following sections, I will give Bayesian applications.

4. Log Linear Model for Contingency Tables

Log linear models for multiway contingency tables are a healthy part of the modern statistics. The index set is $\mathcal{X}={\prod }_{\gamma \in \mathsf{\Gamma }}{I}_{\gamma }$ with $\mathsf{\Gamma }$ indexing categories and $I_{\gamma }$ the levels of $\gamma$. Let $p\left(x\right)$ be the probability of falling into cell $x\in \mathcal{X}$. A log linear model can be specified by writing:

$$logp\left(x\right)=\sum _{a\subseteq \mathsf{\Gamma }}{\phi }_a\left(x\right).$$

The sum ranges over subsets a of $\mathsf{\Gamma }$ and ${\phi }_a\left(x\right)$ means a function that only depends on x through the coordinates in a. Thus, ${\phi }_{\varnothing }\left(x\right)$ is a constant and ${\phi }_{\mathsf{\Gamma }}\left(x\right)$ is allowed to depend on all coordinates. Specifying ${\phi }_a=0$ for some class of sets a determines a model. Background and extensive references are in [9]. If the a with ${\phi }_a\ne 0$ permitted form a simplicial complex $\mathcal{C}$ (so $a\in \mathcal{C}$ and $\varnothing \ne a^{\prime }\subseteq a\Rightarrow a^{\prime }\in \mathcal{C}$) the model is called hierarchical. If $\mathcal{C}$ consists of the cliques in a graph, the model is called graphical. If the graph is chordal (every cycle of length $\ge 4$ contains a chord) the graphical model is called decomposable.

Example 2

(3 way contingency tables). The graphical models for three way tables are:

The simplest hierarchical model that is not graphical is No Three Way Interaction Model.

This can be specified by saying ’the odds rate of any pair of variables does not depend on the third’. Thus,

$$\frac{p_{ijk}{p}_{i^{\prime }{j}^{\prime }k}}{p_{i{j}^{\prime }k}{p}_{i^{\prime }jk}}\mathrm{is}\mathrm{constant}\mathrm{in}k\mathrm{for}\mathrm{fixed}i,i^{\prime },j,j^{\prime }.$$

(7)

As one motivation, recall that for two variables, the independence model is specified by

$$p_{ij}={\theta }_i{\eta }_j.$$

For three variables, suppose there are parameters ${\theta }_{ij},{\eta }_{jk},{\psi }_{ik}$ satisfying:

$$p_{ijk}={\theta }_{ij}{\eta }_{jk}{\psi }_{ik}\mathrm{for}\mathrm{all}i,j,k.$$

(8)

It is easy to see that (8) entails (7) hence ’no three way interaction’. Cross multiplying (7) entails

$$p_{ijk}{p}_{i^{\prime }{j}^{\prime }k}{p}_{i{j}^{\prime }{k}^{\prime }}{p}_{i^{\prime }j{k}^{\prime }}=p_{ij{k}^{\prime }}{p}_{i^{\prime }{j}^{\prime }{k}^{\prime }}{p}_{i{j}^{\prime }k}{p}_{i^{\prime }jk}.$$

(9)

This is the form we will work with for the de Finetti theorems below.

For background, history and examples (and some nice theorems) see ([10], Section 8.2), [11,12], Simpsons ’paradox’ [13] is based on understanding the no three way interaction model. Further discussion is in Section 5 below.

5. From Markov Bases to de Finetti Theorems

Suppose $\mathcal{X}$ is a finite set, $T:\mathcal{X}\to {\mathbb{N}}^d-\left\{0\right\}$ is a statistic and ${\left\{f_i\right\}}_{i=1}^L$ is a Markov basis as in Section 3. The following development shows how to translate this into de Finetti theorems for the contingency table examples of Section 4. The first argument abstracts the argument used for Theorem 2 above.

Lemma 1

(Key Lemma). Let $\mathcal{X}$ be a finite set and ${\left\{X_i\right\}}_{i=1}^{\infty }$ an exchangeable sequence of $\mathcal{X}$-valued random variables. Suppose for all $n>m$

$$\begin{array}{c}P[X_1=x_1,\dots ,X_m=x_m,X_{m+1}=x_{m+1},\dots ,X_n=x_n]=\hfill \\ \hfill P[X_1=y_1,\dots ,X_m=y_m,X_{m+1}=x_{m+1},\dots ,X_n=x_n].\end{array}$$

(10)

In (10), $x_1,\dots ,x_m,y_1,\dots ,y_m$ are fixed and $x_{m+1},\dots ,x_n$ are arbitrary. Then, if $\mathcal{T}$ is the tail field of ${\left\{X_i\right\}}_{i=1}^{\infty }$ and $p\left(x\right)=P[X_1=x|\mathcal{T}]$,

$$\prod _{i=1}^{m}p\left(x_i\right)=\prod _{i=1}^{m}p\left(y_i\right).$$

(11)

Proof. 

From (10) and exchangeability

$$\begin{array}{c}P[X_1=x_1,\dots ,X_m=x_m,X_{n+1}=x_{n+1},\dots ,X_{n+h}=x_{n+h}]=\hfill \\ \hfill P[X_1=y_1,\dots ,X_m=y_m,X_{n+1}=x_{n+1},\dots ,X_{n+h}=x_{n+h}]\end{array}$$

so

$$\begin{array}{c}P[X_1=x_1,\dots ,X_m=x_m|X_{n+1}=x_{n+1},\dots ,X_{n+h}=x_{n+h}]=\hfill \\ \hfill P[X_1=y_1,\dots ,X_m=y_m|X_{n+1}=x_{n+1},\dots ,X_{n+h}=x_{n+h}].\end{array}$$

Let $h\uparrow \infty$ and then $n\uparrow \infty$, use Doob’s upward and then downward martingale convergence theorems to see:

$$P[X_1=x_1,\dots ,X_m=x_m|\mathcal{T}]=P[X_1=y_1,\dots ,X_m=y_m|\mathcal{T}].$$

Now, de Finetti’s theorem implies (11). □

Remark 1.

The Key Lemma shows that the $p\left(x\right)$ satisfy certain relations. Using choices of $\left\{x_i\right\},\left\{y_i\right\}$ derived from a Markov basis will show that $p\left(x\right)$ satisfy the required independence properties. Suppose that ${\sum }_{\mathcal{X}}f\left(x\right)T\left(x\right)=0$, ${\sum }_{\mathcal{X}}f\left(x\right)=0$ and $f\in \{0,\pm 1\}$. Let $S_{+}=\{x:f\left(x\right)=1\}$, $S_{-}=\{y:f\left(y\right)=-1\}$. Say $|S_{+}|=|S_{-}|=m$. Enumerate $S_{+}=\{x_1,\dots ,x_m\}$, $S_{-}=\{y_1,\dots ,y_m\}$. Assumptions (10) and conclusion (11) will give our theorems.

Example 3

(Independence in a two way table). Let $\mathcal{X}=\left[I\right]\times \left[J\right]$. A minimal basis for the independence model is given by $f_{i,j,i^{\prime },j^{\prime }}$:

$$\begin{array}{ccc}& j& j^{\prime }\\ i& +& -\\ i^{\prime }& -& +\end{array}(allotherentries=0).$$

The condition of the Key Lemma becomes:

$$\begin{array}{c}P[X_1=(i,j),X_2=(i^{\prime },j^{\prime }),X_3=(i_3,j_3),\dots ,X_n=(i_n,j_n)]=\hfill \\ \hfill P[X_1=(i,j^{\prime }),X_2=(i^{\prime },j),X_3=(i_3,j_3),\dots ,X_n=(i_n,j_n)].\end{array}$$

Passing to the limit gives

$$p_{ij}{p}_{i^{\prime }{j}^{\prime }}=p_{i{j}^{\prime }}{p}_{i^{\prime }j}$$

and so

$$p_{i\ast }{p}_{\ast j}=\sum _{i^{\prime }{j}^{\prime }}{p}_{i{j}^{\prime }}{p}_{i^{\prime }j}=p_{ij}.$$

This is precisely Theorem 2 of the Introduction. □

Example 4

(Complete independence in a three way table). The sufficient statistics are $T_{i\ast \ast },T_{\ast j\ast },T_{\ast \ast k}$. From [5], there are two kinds of moves in a minimal basis. Up to symmetries, these are:

Class I	Class II
$\begin{array}{ccc}& j& j^{\prime }\\ i& +& -\\ i^{\prime }& -& +\end{array}\begin{array}{ccc}& & \\ & & \\ & \end{array}\begin{array}{ccc}& & \\ & & \\ & \end{array}$	$\begin{array}{ccc}& j& j^{\prime }\\ i& +& -\\ & \end{array}\begin{array}{ccc}& j& j^{\prime }\\ & & \\ i^{\prime }& -& +\end{array}\begin{array}{ccc}& & \\ & & \\ & \end{array}$

Passing to the limit, this entails:

$$p_{ijk}{p}_{i{j}^{\prime }k}=p_{i{j}^{\prime }k}{p}_{i^{\prime }jk}and{p}_{ijk}{p}_{i^{\prime }{j}^{\prime }{k}^{\prime }}=p_{i{j}^{\prime }k}{p}_{ij{k}^{\prime }}.$$

These may be said as ’the product of any $p_{ijk},p_{i^{\prime }jk}$ remains unchanged if the middle coordinates are exchanged’. By symmetry, this remains true if the two first or last coordinates are exchanged. As above, this entails

$$p_{i\ast \ast }{p}_{\ast j\ast }{p}_{\ast \ast k}=p_{ijk}.$$

These observations can be rephrased into a statement that looks more similar to the classical de Finetti theorem; using symmetry:

Theorem 5.

Let ${\left\{X_i\right\}}_{i=1}^{\infty }$ be exchangeable, taking values in $\left[I\right]\times \left[J\right]\times \left[K\right]$. Then

$$\begin{array}{c}P[X_1=(i_1,j_1,k_1),\dots ,X_n=(i_n,j_n,k_n)]=\hfill \\ \hfill P[X_1=(\sigma \left(i_1\right),\zeta \left(j_1\right),\eta \left(k_1\right)),\dots ,X_n=(\sigma \left(i_n\right),\zeta \left(j_n\right),\eta \left(k_n\right))]\end{array}$$

for all n, ${\left\{(i_a,j_a,k_a)\right\}}_{a=1}^n$ and $(\sigma ,\zeta ,\eta )\in S_I\times S_J\times S_K$ is necessary and sufficient for there to exist a unique μ on ${\mathsf{\Delta }}_I\times {\mathsf{\Delta }}_J\times {\mathsf{\Delta }}_K$ with

$$P[X_a=(i_a,j_a,k_a),1\le a\le n]={\int }_{{\mathsf{\Delta }}_I\times {\mathsf{\Delta }}_J\times {\mathsf{\Delta }}_K}\prod _{a=1}^n{p}_{i_a}{q}_{j_a}{r}_{k_a}\mu (dp,dq,dr).$$

□

Example 5

(One variable independent of the other two). Suppose, without loss, that the graph is

Identify the pairs $(j,k)$ with $\{1,2,\dots ,L\}$ with $L=JK$. The problem reduces to Example 4. A minimal basis consists of (again, up to relabeling)

$$\begin{array}{ccc}& l& l^{\prime }\\ i& +& -\\ i^{\prime }& -& +\end{array}$$

We may conclude

Theorem 6.

Let ${\left\{X_i\right\}}_{i=1}^{\infty }$ be exchangeable, taking values in $\left[I\right]\times \left[J\right]\times \left[K\right]$. Then

$$\begin{array}{c}P[X_1=(i_1,j_1,k_1),\dots ,X_n=(i_n,j_n,k_n)]=\hfill \\ \hfill P[X_1=(\sigma \left(i_1\right),\zeta (j_1,k_1)),\dots ,X_n=(\sigma \left(i_n\right),\zeta (j_n,k_n))]\end{array}$$

for all n, ${\left\{(i_a,j_a,k_a)\right\}}_{a=1}^n$ and $(\sigma ,\zeta )\in S_I\times S_{J\times K}$ is necessary and sufficient for there to exist a unique μ on ${\mathsf{\Delta }}_I\times {\mathsf{\Delta }}_{JK}$ with

$$P[X_a=(i_a,j_a,k_a),1\le a\le n]={\int }_{{\mathsf{\Delta }}_I\times {\mathsf{\Delta }}_{JK}}\prod _{a=1}^n{p}_a{q}_a\mu (dp,dq).$$

□

Example 6

(Conditional independence). Suppose variable i and j are conditionally independent given k.

Rewrite the parameter condition of section four as

$$p_{\ast \ast k}{p}_{ijk}=p_{i\ast k}{p}_{\ast jk}foralli,j,k$$

The sufficient statistics are ${\left\{T_{i\ast k}\right\}}_{i,k},{\left\{T_{\ast jk}\right\}}_{jk}$. From [5], a minimal generating set is

$$\begin{array}{ccc}& j_k& j_k^{\prime }\\ i_k& +& -\\ i_k^{\prime }& -& +\end{array}K\times \frac{I(I-1)}{2}\times \frac{J(J-1)}{2}movesinall.$$

From this, the Key Lemma shows (for all $i,j,k$)

$$p_{ijk}{p}_{i^{\prime }{j}^{\prime }k}=p_{i{j}^{\prime }k}{p}_{i^{\prime }jk}.$$

This entails:

$$p_{i\ast k}{p}_{\ast jk}=\sum _{i^{\prime },j^{\prime }}{p}_{i{j}^{\prime }k}{p}_{i^{\prime }jk}=\sum _{i^{\prime }{j}^{\prime }}{p}_{ijk}{p}_{i^{\prime }{j}^{\prime }k}=p_{ijk}{p}_{\ast \ast k}.$$

Again, phrasing the condition (10) in terms of symmetry.

Theorem 7.

Let ${\left\{X_i\right\}}_{i=1}^{\infty }$ be exchangeable, taking values in $\left[I\right]\times \left[J\right]\times \left[K\right]$. Then,

$$\begin{array}{c}P[X_1=(i_i,j_i,k_i),\dots ,X_n=(i_n,j_n,k_n)]=\hfill \\ \hfill P[X_1=({\sigma }^{k_1}\left(i_1\right),{\zeta }^{k_1}\left(j_1\right),k_1),\dots ,X_n=({\sigma }^{k_n}\left(i_n\right),{\zeta }^{k_n}\left(j_n\right),k_n)]\end{array}$$

(12)

for all n, ${\left\{(i_a,j_a,k_a)\right\}}_{a=1}^n$ and ${\sigma }^k,{\zeta }^k\in S_I\times S_J$, $1\le k\le K$, is necessary and sufficient for there to exist a unique family $\mu \times {\prod }_{b=1}^k{\mu }_{b,r}$ on ${\mathsf{\Delta }}_K\times {({\mathsf{\Delta }}_I\times {\mathsf{\Delta }}_J)}^K$

$$\begin{array}{c}P[X_a=(i_a,j_a,k_a),1\le a\le n]=\hfill \\ \hfill {\int }_{{\mathsf{\Delta }}_K\times {({\mathsf{\Delta }}_I\times {\mathsf{\Delta }}_J)}^K}\prod _{a=1}^n{r}_{k_a}{p}_{i_a}^{k_a}{q}_{j_a}^{k_a}\prod _{b=1}^k{\mu }_{b,r}\left(p^{i_b}{q}^{i_b}\right)\mu \left(dr\right).\end{array}$$

(13)

Both (12) and (13) have a simple interpretation. For (12), ${\left\{X_i\right\}}_{i=1}^n$ are exchangeable 3-vectors. For any k and specified sequence of values ${\left\{(i_a,j_a,k)\right\}}_{a=1}^n$ the chance of observing these values is unchanged under permuting the $(i_a,j_a,k)$, by permutations ${\sigma }^k\in S_I,{\zeta }^k\in S_J$. Here ${\sigma }^k,{\zeta }^k$ are allowed to depend on k.

On the right of (13), the mixing measure may be understood as follows. There is a probability $\mu$ on ${\mathsf{\Delta }}_K$. Pick $r=(r_1,\dots ,r_k)\in {\mathsf{\Delta }}_K$. Given this r, pick $(p^k,q^k)$ from ${\mu }_{k,r}$ on the $k^{th}$ copy of ${\mathsf{\Delta }}_I\times {\mathsf{\Delta }}_J$. These choices are allowed to depend on r but are independent, conditional on r, $1\le k\le K$.

All of this simply says that, conditional on the tail field,

$$P[X_a=(i,j,k)|\mathcal{T}]=P[X_a=(i,\ast ,k)|\mathcal{T})P(X_a=(\ast ,j,k)|\mathcal{T}].$$

The first two coordinates are conditionally independent given the third.

Example 7

(No three way interaction). The model is described in Section 4. The sufficient statistics are $\left\{T_{ij\ast }\right\},\left\{T_{i\ast k}\right\},\left\{T_{\ast jk}\right\}.$ Minimal Markov bases have proved intractable. See [5] or [8]. For any fixed $I,J,K$, the computer can produce a Markov basis but these can have a huge number of terms. See [7,8] and their references for a surprisingly rich development.

There is a pleasant surprise. Markov bases are required to connect the associated Markov chain. There is a natural subset, the first moves anyone considers, and and these are enough for a satisfactory de Finetti theorem (!).

Described informally, for an $I\times J\times K$ array, pick a pair of parallel planes, say the $k,k^{\prime }$ planes in the three dimensional array, and consider moves depicted as

$$\begin{array}{c}\begin{array}{ccc}& j& j^{\prime }\\ i& +& -\\ i^{\prime }& -& +\end{array}\begin{array}{ccc}& j& j^{\prime }\\ i& +& -\\ i^{\prime }& -& +\end{array}\\ k{k}^{\prime }\end{array}$$

These moves preserve all line sums (the sufficient statistics). They arenotsufficient to connect any two datasets with the same sufficient statistics. Using the prescription in the Key Lemma, suppose:

$$\begin{array}{c}P[X_1=(i,j,k),X_2=(i^{\prime },j^{\prime },k),X_3=(i,j^{\prime },k^{\prime }),X_4=(i^{\prime },j,k^{\prime }),\hfill \\ X_a=(i_a,j_a,k_a)5\le a\le n]=\\ P[X_1=(i,j^{\prime },k),X_2=(i^{\prime },j,k),X_3=(i,j,k^{\prime }),X_4=(i^{\prime },j^{\prime },k^{\prime }),\\ \hfill X_a=(i_a,j_a,k_a)5\le a\le n].\end{array}$$

(14)

Passing to the limit gives

$$p_{ijk}{p}_{i^{\prime }{j}^{\prime }k}{p}_{i{j}^{\prime }{k}^{\prime }}{p}_{i^{\prime }j{k}^{\prime }}=p_{i{j}^{\prime }k}{p}_{i^{\prime }jk}{p}_{ij{k}^{\prime }}{p}_{i^{\prime }{j}^{\prime }{k}^{\prime }}.$$

(15)

This is exactly the no three way interaction condition. Or, equivalently:

$$\frac{p_{ijk}{p}_{i^{\prime }{j}^{\prime }k}}{p_{i{j}^{\prime }k}{p}_{i^{\prime }jk}}=\frac{p_{ij{k}^{\prime }}{p}_{i^{\prime }{j}^{\prime }{k}^{\prime }}}{p_{i{j}^{\prime }{k}^{\prime }}{p}_{i^{\prime }j{k}^{\prime }}}.$$

The odds ratios are constant on the $k^{th}$ and $k^{\prime th}$ planes (of course, they depend on $i,j,i^{\prime },j^{\prime }$). These considerations imply:

Theorem 8.

Let ${\left\{X_i\right\}}_{i=1}^{\infty }$ be exchangeable, taking values in $\left[I\right]\times \left[J\right]\times \left[K\right]$. Then, condition (14) is necessary and sufficient for the existence of a unique probability μ on ${\mathsf{\Delta }}_{IJK}$, supported on the no three way interaction variety (15) satisfying

$$P[X_a=(i_a,j_a,k_a),1\le a\le n]={\int }_{{\mathsf{\Delta }}_{IJK}}\prod p_{ijk}^{{\eta }_{ijk}}\mu \left(d{p}_{ijk}\right).$$

We remark on the following points.

• It follows from theorems in [12] and [11] that, if all $p_{ijk}>0$, condition (15) is equivalent to the unique representation,

  $$p_{ijk}=r{\alpha }_{jk}{\beta }_{ki}{\gamma }_{ij},$$

  (16)

  where $r,\alpha ,\beta ,\gamma$ have positive entries and satisfy

  $$\sum _k{\alpha }_{jk}=\sum _i{\beta }_{ki}=\sum _j{\gamma }_{ij}=1\mathrm{for}\mathrm{all}i,j,k$$

  and

  $$r\sum _{i,j,k}{\alpha }_{jk}{\beta }_{ki}{\gamma }_{ij}=1.$$
  The integral representation in the theorem can be stated in this parametrization. The condition $p_{ijk}>0$ is equivalent to $P(X_1=(i,j,k))>0$ on observables.
• Condition (14) does not have an obvious symmetry interpretation.
• Conditions (14) and (15) are stated via varying the third variable when $i,j,i^{\prime },j^{\prime }$ are fixed. Because of (16), if they hold in this form, they hold for any two variables fixed as the third varies.
• It is possible to go on, but, as John Darroch put it, ’the extensions to higher order interactions… are not likely to be of practical interest’. The most natural development—the generalization to decomposable models—is being developed by Paula Gablenz.
• There are many extensions of the Key Lemma above. These allow a similar development for more general log linear models and exponential families.

6. Discussion and Conclusions

The tools of algebraic statistics have been harnessed above to develop partial exchangeability for standard contingency table models. I have used them for two further Bayesian tasks: approximate exchangeability and the problem of ’doubly intractable priors’. As both are developed in papers, I will be brief.

Approximate exchangeability.Consider n men and m women along with a binary outcome. If the men are judged exchangeable (for fixed outcomes for the women) and vice versa, and, if both sequences are extendable, de Finetti [1] shows that there is a unique prior on the unit square ${[0,1]}^2$ such that, for any outcomes $t_1,\dots ,t_n,{\sigma }_1,\dots ,{\sigma }_m$ in $\{0,1\}$

$$\begin{array}{c}P[X_1=t_1,\dots ,X_n=t_n,Y_1={\sigma }_1,\dots ,Y_m={\sigma }_m]=\hfill \\ \hfill {\int }_{{[0,1]}^2}{p}^S{(1-p)}^{n-S}{\theta }^T{(1-\theta )}^{m-T}\mu (dp,d\theta ),\end{array}$$

with $S={\sum }_{i=1}^n{t}_i$, $T={\sum }_{j=1}^m{\sigma }_j$.

If, for the outcome of interest, $\{X_i,Y_j\}$ were almost fully exchangeable (so the men/ women difference is judged practically irrelevant) the prior $\mu$ would be concentrated near the diagonal of ${[0,1]}^2$. De Finetti suggested implementing this by considering priors of the form

$$\mu (dp,d\theta )=Z^{-1}{e}^{-A{(p-\theta )}^2}dpd\theta$$

for A large.

In joint work with Sergio Bacallado and Susan Holmes [3], multivariate versions of such priors are developed. These are required to concentrate near sub-manifolds of cubes or products of simplicies; think about ‘approximate no three way interaction’. We used the tools of algebraic statistics to suggest appropriate many variable polynomials which vanish on submanifold of interest. Many ad hoc choices were involved. Sampling from such priors or posteriors is a fresh research area. See [2,14,15].

Doubly intractable priors. Consider an exponential family as in Section 3:

$$p_{\theta }\left(x\right)=\frac{1}{Z\left(\theta \right)}{e}^{\theta ·T\left(x\right)}.$$

Here $x\in \mathcal{X}$ a finite set, $T:\mathcal{X}\to {\mathbb{R}}^d$ and $\theta \in {\mathbb{R}}^d$. In many real examples, the normalizing constant $Z\left(\theta \right)$ will be unknown and unknowable. For a Bayesian treatment, let $\mathsf{\Pi }\left(d\theta \right)$ be a prior distribution on ${\mathbb{R}}^d$. For example, the conjugate prior.

If $X_1,X_2,\dots ,X_n$ is as i.i.d. sample from $p_{\theta }$, T is a sufficient statistic and the posterior has the form

$$Z{\left(Z^{-1}\left(\theta \right)\right)}^n{e}^{\theta F}\mathsf{\Pi }\left(d\theta \right),$$

with $F={\sum }_{i=1}^{n}T\left(X_i\right)$ and Z another normalizing constant. The problem is that $Z^{-1}\left(\theta \right)$ depends on $\theta$ and is unknown!

The exchange algorithm and many variants offer a useful solution. See [16,17].

In practical implementations, there is an intermediary step requiring a sample form $p_{{\theta }^{\prime }}^T$, the measure induced by $p_{\theta }^n$ under ${\sum }_i^{n}T\left(x_i\right):{\mathcal{X}}^n\to \mathbb{R}$. This is a discrete sampling task and Markov basis techniques have been proved useful. See [16].

A philosophical comment. The task undertaken above, finding believable Bayesian interpretations for widely used log linear models, goes somewhat against the grain of standard statistical practice. I do not think anyone takes a reasonably complex, high dimensional hierarchical model seriously. They are mostly used as a part of exploratory data analysis; this is not to deny their usefulness. Making any sense of a high dimensional dataset is a difficult task. Practitioners search through huge collections of models in an automated way. Usually, any reflection suggests the underlying data is nothing like a sample from a well specified population. Nonetheless, models are compared using product likelihood criteria. It is a far far cry from being based on anyone’s reasoned opinion.

I have written elsewhere about finding Bayesian justification for important statistical tasks such as graphical methods or exploratory data analysis [18]. These seem like tasks similar to ’how do you form a prior’. Different from the focus of even the most liberal Bayesian thinking.

The sufficiency approach. There is a different approach to extending de Finetti’s theorem. This uses ‘sufficiency’. Consider exchangeable ${\left\{X_i\right\}}_{i=1}^{\infty }$. For each n, suppose $T_n:{\mathcal{X}}^n\to \mathcal{Y}$ is a function. The $\left\{T_n\right\}$ have to fit together according to simple rules satisfied in all of the examples above. Call $\left\{X_i\right\}$ partially exchangeable with respect to $T_n$ if $P[X_1=x_1,\dots ,X_n=x_n|T_n=t_n]$ is uniform. Then, Diaconis and Freedman [19] show that a version of de Finetti’s theorem holds. The law of $\left\{X_i\right\}$ is a mixture of i.i.d. laws indexed by extremal laws. In dozens of examples, these extremal laws can be identified with standard exponential families. This last step remains to be carried out in the generality of Section 3 above. What is required is a version of the Koopman–Pitman–Darmois theorem for discrete random variables. This is developed in [19] when $\mathcal{X}\subseteq \mathbb{N}$ and $T_n(X_1,\dots ,X_n)=X_1+\cdots +X_n$. Passing to interpretation, this version of partial exchangeability has the following form:

$$\begin{array}{c}\mathrm{if}{T}_n(x_1,\dots ,x_n)=T_n(y_1,\dots ,y_n),\hfill \\ \hfill \mathrm{then}P[X_1=x_1,\dots ,X_n=x_n]=P[X_1=y_1,\dots ,X_n=y_n].\end{array}$$

This is neat mathematics (and allows a very general theoretical development). However, it does not seem as easy to think about in natural examples. Exchangeability via symmetry is much easier. The development above is a half-way house between symmetry and sufficiency. A close relative of the sufficiency approach is the topic of ‘extremal models’ as developed by Martin-Löf and Lauritzen. See [20] and its references. Moreover, Refs. [21,22] are recent extensions aimed at contingency tables.

Classical Bayesian contingency table analysis. There is a healthy development of parametric analysis for the examples of Section 5. This is based on natural conjugate priors. It includes nice theory and R packages to actually carry out calculations in real problems. Three papers that I like are [23,24,25,26]. The many wonderful contributions by I.J. Good are still very much worth consulting. See [27] for a survey. Section 5 provides ‘observable characterizations’ of the models. The problem of providing ‘observable characterizations’ of the associated conjugate priors (along the lines of [28]) remains open.