Successive Standardization of Rectangular Arrays

Abstract

In this note we illustrate and develop further with mathematics and examples, the work on successive standardization (or normalization) that is studied earlier by the same authors in [1,2]. Thus, we deal with successive iterations applied to rectangular arrays of numbers, where to avoid technical difficulties an array has at least three rows and at least three columns. Without loss, an iteration begins with operations on columns: first subtract the mean of each column; then divide by its standard deviation. The iteration continues with the same two operations done successively for rows. These four operations applied in sequence completes one iteration. One then iterates again, and again, and again, … In [1] it was argued that if arrays are made up of real numbers, then the set for which convergence of these successive iterations fails has Lebesgue measure 0. The limiting array has row and column means 0, row and column standard deviations 1. A basic result on convergence given in [1] is true, though the argument in [1] is faulty. The result is stated in the form of a theorem here, and the argument for the theorem is correct. Moreover, many graphics given in [1] suggest that except for a set of entries of any array with Lebesgue measure 0, convergence is very rapid, eventually exponentially fast in the number of iterations. Because we learned this set of rules from Bradley Efron, we call it “Efron’s algorithm". More importantly, the rapidity of convergence is illustrated by numerical examples.

1. Introduction

Research summarized in this paper is an extension of that reported in [1] and a conference proceeding [2], and concerns successive standardization (or normalization) of large rectangular arrays $\mathbf{X}$ of real numbers, such as arise in gene expression, from protein chips, or in the earth and environmental sciences. The basic message here is that convergence that holds on all but a set of measure 0 in the paper by Olshen and Rajaratnam ([1] is shown here to be exponentially fast in a sense we make precise. The basic result in [1], though true, is not argued correctly in [1]. The gap is filled here. Typically there is one column per subject; rows correspond to “genes" or perhaps gene fragments (including those that owe to different splicing of “the same" genes, or proteins). Typically, though not always, columns divide naturally into two groups: “affected" or not. Two-sample testing of rows that correspond to “affected" versus other individuals or samples is then carried out simultaneously for each row of the array. Corrections for multiple comparisons may be very simple, or might perhaps allow for “false discovery."

As was noted in [1], data may still suffer from problems that have nothing to do with differences between groups of subjects or differences between “genes" or groups of them. There may be variation in background, perhaps also in “primers." Thus, variability across subjects might be unrelated to status. In comparing two random vectors that may have been measured in different scales, one puts observations “on the same footing" by subtracting each vector’s mean and dividing by its standard deviation. Thereby, empirical covariances are changed to empirical correlations, and comparisons proceed. But how does one do this in the rectangular arrays described earlier? An algorithm by which such “regularization" is accomplished was described to us by colleague Bradley Efron; so we call the full algorithm Efron’s algorithm. We shall use the terms successive “normalization" or successive “standardization" interchangeably (and also point out that Bradley Efron considers the the latter term a better description of the algorithm). To avoid technical problems that are described in [1] and repeated here, we assume there are at least three rows and at least three columns to the array. It is immaterial to convergence, though not to limiting values, whether we begin regularization to be described by row or by column. In order that we fix an algorithm, we begin by column in the computations though by row in the mathematics. Thus, we first mean polish the column, then standard deviation polish the column; next we mean polish the row, and standard deviation polish the row. The process is then repeated. By “mean polish" of, say, a column, we mean subtract the mean value for that column from every entry. By “standard deviation” polish the column, we mean divide each number by the standard deviation of the numbers in that column. Definitions for “row" are entirely analogous.

In [1] convergence is studied with entries in the rectangular array with I rows and J columns viewed as elements of ${\mathbb{R}}^{IJ}$, Euclidean IJ space. Convergence holds for all but a Borel subset of ${\mathbb{R}}^{IJ}$ of Lebesgue measure 0. The limiting vector has all row and column means 0, all row and column standard deviations 1. We emphasize that to show convergence on a Lebesgue set of full measure, it is enough to find a probability mutually absolutely continuous with respect to Lebesgue measure for which convergence is established with probability 1.

2. Preliminaries

We begin precise definitions:

$${\overline{X}}_{i·}^{\left(0\right)}=\frac{1}{J}\sum _{j=1}^J{X}_{ij}$$

$${\left(S_i^{\left(0\right)}\right)}^2=\frac{1}{J}\sum _{j=1}^J{(X_{ij}-{\overline{X}}_{i·}^{\left(0\right)})}^2=(\frac{1}{J}\sum _{j=1}^J{X}_{ij}^2-J{\left({\overline{X}}_{i·}^{\left(0\right)}\right)}^2)$$

$${\mathbf{X}}^{\left(1\right)}=\left\{X_{ij}^{\left(1\right)}\right\},\mathrm{where}{X}_{ij}^{\left(1\right)}=\frac{X_{ij}-{\overline{X}}_{i·}^{\left(0\right)}}{S_i^{\left(0\right)}}$$

By analogy, set

$${\mathbf{X}}^{\left(2\right)}=\left\{X_{ij}^{\left(2\right)}\right\}$$

where

$$X_{ij}^{\left(2\right)}=\frac{X_{ij}^{\left(1\right)}-{\overline{X}}_{·j}^{\left(1\right)}}{S_j^{\left(1\right)}}\mathrm{and}{\overline{X}}_{·j}^{\left(1\right)}=\frac{1}{I}\sum _{i=1}^I{X}_{ij}^{\left(1\right)}$$

Now,

$${\left(S_j^{\left(1\right)}\right)}^2=\frac{1}{I}\sum _{i=1}^I{(X_{ij}^{\left(1\right)}-{\overline{X}}_{·j}^{\left(1\right)})}^2$$

and analogously define ${\left(S_j^{\left(n\right)}\right)}^2$ for $n=2,3,....$ By induction on dimension applied to regular conditional distributions, ${\left(S_i^{\left(0\right)}\right)}^2>0$ a.s.

As explained in [1] the $2\times 2$ case illustrates the need to work with dimensions greater than or equal to 3. Consider the following arbitrary $2\times 2$ matrix

$$\mathbf{X}=\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]$$

If $a<b$ and $c<d$, then

$${\mathbf{X}}^{\left(1\right)}=\left[\begin{array}{cc}-1& 1\\ -1& 1\end{array}\right]$$

so ${\left(S_j^{\left(1\right)}\right)}^2=0$

If $a,b,c,d$ are, say, iid and have symmetric distributions, then

$$P({\left(S_j^{\left(1\right)}\right)}^2=0)=\frac{1}{2}$$

A moment’s reflection shows that if $\mathbf{X}$ is $I\times 2$, with I odd, then after each row normalization, each column has an odd number of entries, each entry being $+1$ or $-1$. However, each row has exactly one $+1$ and one $-1$. Thus ${(S_i^{\left(n\right)}-S_j^{(n+1)})}^2\to 0$ is impossible. With $min(I,J)\ge 3$, both tend to 1 a.s. as $n\nearrow \infty$. The fact that $\left\{X_{ij}\right\}$ iid implies in particular that $\mathbf{X}$ is row and column exchangeable. Thus, if

$$\mathbf{X}=[{\mathbf{x}}_{1,c},...{\mathbf{x}}_{J,c}]$$

where each column is $I\times 1$, and $\pi$ is a permutation of the integers $\{1,...,J\}$, then $\mathbf{X}\sim [{\mathbf{x}}_{\pi \left(1\right),c},...,{\mathbf{x}}_{\pi \left(J\right),c}]$. The analogous holds for rows, where

$$\mathbf{X}=\left[\begin{array}{c}{\mathbf{x}}_{1,r}\\ ⋮\\ {\mathbf{x}}_{I,r}\end{array}\right]$$

where ${\mathbf{x}}_{i,r}$ is $1\times J$.

3. Background and Motivation

A first step in the argument of [1] is to note that Lebesgue measure on the Borel subsets of ${\mathbb{R}}^{IJ}$ is mutually absolutely continuous with respect to IJ product Gaussian measure, each coordinate being standard Gaussian. Thereby, the distinction between measure and topology is blurred; arguments of [1] as corrected here. Having translated a problem concerning Lebesgue measure to one concerning Gaussian measure, one cannot help note from graphs in [1] Figure 1, Figure 2, Figure 3, Figure 6 and Figure 7 that suggest with entries of $\mathbf{X}$ chosen independently from a common absolutely continuous distribution, and as led to these figures, almost surely ultimately, convergence is at least exponentially fast. In these graphs, the ordinate is always the difference of logarithms (the base does not matter) of squared Frobenius norm of the difference between current iteration and the immediately previous iteration; always abscissa indexes iteration. One purpose of this paper is to demonstrate the ultimately almost sure rapidity of convergence. Readers will note we assume that coordinates are independent and identically distributed, with standard normal distributions. This assumption is used explicitly though unnecessarily in [1], but only implicitly here, and then only to the extent that arguments here depend on those in [1]. Obviously, this Gaussian assumption is sufficient. It is pretty obviously not necessary.

For the sake of motivation we illustrate the patterns of convergence from successive normalizations on $5\times 5$ matrices. We first describe the details of the successive normalization in a manner very similar to [1]. Consider a simple $5\times 5$ matrix with entries generated from a normal distribution with a given mean and standard deviation. In our case we take the mean to be 2 and variance to be 4, though the specific values the mean and variance parameter take do not matter. We first standardize the initial matrix ${\mathbf{X}}^{\left(\mathbf{0}\right)}$ at the level of each by row, i.e., first subtracting the row mean from each entry and then dividing each entry in a given row by its row standard deviation. The matrix is then standardized at the level of each column, i.e., by first subtracting the column mean from each entry and then by dividing each entry by the respective column standard deviation. Row mean and standard deviation polishing followed by column mean and standard deviation polishing is defined as one iteration in the process of attempting to row and column standardize the matrix. The resulting matrix is defined as ${\mathbf{X}}^{\left(\mathbf{1}\right)}$. Now the same process is repeated with ${\mathbf{X}}^{\left(\mathbf{1}\right)}$ and repeated until successive renormalization eventually yields a row and column standardized matrix. The successive normalizations are repeated until “convergence” which for our purposes is defined as the difference in the squared Frobenius norm between two consecutive iterations being less than ${10}^{-8}$.

Figure 1. Examples of patterns of convergence.

Figure 1. Examples of patterns of convergence.

Figure 2. Examples of patterns of convergence.

Figure 2. Examples of patterns of convergence.

The figures (see Figure 1 and Figure 2) are plots of the log of the ratios of the squared Frobenius norms of the differences between consecutive iterates. In particular, they capture the type of convergence patterns that are observed in the $5\times 5$ case from different starting values. In [1] it was proved that regardless of the starting value (provided the dimensions are at least 3) the process of successive normalization always converges, in the sense that it leads to a doubly standardized matrix. In addition, it was noted that the convergence is very rapid. We can see empirically from Figure 1, Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11 that eventually the log of the successive squared differences tends to decay in a straight line, i.e., the rate of convergence is perhaps exponential. This phenomenon of linear decay between successive iterations in the log scale is observed in all the diagrams. Hence, one is led naturally to ask whether this is always true in theory, and if so under what conditions. The rate of convergence of successive normalization and related questions are addressed in this paper.

Figure 3. Two examples of simultaneous normalization: these illustrate that simultaneous normalization does not lead to convergence.

Figure 3. Two examples of simultaneous normalization: these illustrate that simultaneous normalization does not lead to convergence.

Figure 4. (Left) Example of simultaneous normalization, (Right) Same example zoomed in.

Figure 4. (Left) Example of simultaneous normalization, (Right) Same example zoomed in.

A natural question to ask is whether the convergence phenomenon observed will still occur if simultaneous normalization is undertaken as compared to successive normalization. In other words the row and column mean polishing and row and column standard deviation polishing are all done at once:

$$X_{ij}^{t+1}=\frac{X_{ij}^t-{\overline{X}}_{i.}^t-{\overline{X}}_{.j}^t}{s_{i.}{s}_{.j}}$$

In this case the simultaneous normalization algorithm does not converge. In fact it is shown to diverge. Figure 3, Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8 illustrate this phenomenon.

Figure 5. (Left) Example of simultaneous normalization, (Right) Same example zoomed in.

Figure 5. (Left) Example of simultaneous normalization, (Right) Same example zoomed in.

Figure 6. (Left) Example of simultaneous normalization, (Right) Same example zoomed in.

Figure 6. (Left) Example of simultaneous normalization, (Right) Same example zoomed in.

Figure 7. Two further examples of simultaneous normalization: these illustrate that simultaneous normalization does not lead to convergence.

Figure 7. Two further examples of simultaneous normalization: these illustrate that simultaneous normalization does not lead to convergence.

Figure 8. Example of simultaneous normalization

Figure 8. Example of simultaneous normalization

4. Convergence and Rates

Theorem 4.1 of [1] is false. A claimed backwards martingale is NOT. Fortunately, all that seems damaged by the mistake is pride. Much is true. To establish what this is requires notation.

$\mathbf{X}$ denotes an $I\times J$ matrix with values $x\in {\Re }^{IJ}$. We take coordinates $X_{ij}\left(x\right)={\mathbf{X}}_{ij}$ to be iid $N(0,1)$. As in [1,2], $3\le min(I,J)\le max(I,J)<\infty$. As before,

$$\begin{array}{ccc}\hfill {\left(S_i^{\left(0\right)}\right)}^2& =& \frac{1}{J}\sum _{j=1}^J{(X_{ij}-{\overline{X}}_{i·}^{\left(0\right)})}^2\hfill \end{array}$$

(1)

$$\begin{array}{ccc}& =& \frac{1}{J}\sum _{j=1}^J{\left(X_{ij}\right)}^2-\frac{2}{J}\sum _{j=1}^J{X}_{ij}{\overline{X}}_{i·}^{\left(0\right)}+{\left({\overline{X}}_{i·}^{\left(0\right)}\right)}^2\hfill \end{array}$$

(2)

$$\begin{array}{ccc}& =& \frac{1}{J}\sum _{j=1}^J{\left(X_{ij}\right)}^2-2{\overline{X}}_{·j}{\overline{X}}_{i·}+{\left({\overline{X}}_{i·}\right)}^2.\hfill \end{array}$$

(3)

${\mathbf{X}}^{\left(1\right)}=\left[X_{ij}^{\left(1\right)}\right]$.

In view of [1], almost surely ${\left(S_i^{\left(0\right)}\right)}^2>0$. By analogy, set ${\mathbf{X}}^{\left(2\right)}=\left[X_{ij}^{\left(2\right)}\right]$, where $X_{ij}^{\left(2\right)}=(X_{ij}^{\left(1\right)}-{\overline{X}}_{·j}^{\left(1\right)})/S_j^{\left(1\right)}$. As in [1], ${\left(S_j^{\left(1\right)}\right)}^2>0$. In general, for m odd, $X_{ij}^{\left(m\right)}=(X_{ij}^{(m-1)}-{\overline{X}}_{i·}^{(m-1)})/S_i^{(m-1)}$.

Likewise, for m even $X_{ij}^{\left(m\right)}=(X_{ij}^{(m-1)}-{\overline{X}}_{·j}^{(m-1)})/S_j^{(m-1)}$. Without loss, we take ${\left(S_i^{\left(l\right)}\right)}^2$ and ${\left(S_j^{\left(l\right)}\right)}^2$ to be positive for all $(i,j,l)$.

In various places we make implicit use of a theorem of Skorokhod, the Heine–Borel Theorem, and this obvious fact. Suppose we are given a sequence of random variables $\mathbf{Y}=(Y_1,Y_2,...)$ and a condition ${\mathcal{C}}_0$ that depends only on their finite-dimensional distributions. If we wish to make a conclusion ${\mathcal{C}}_1$ concerning $\mathbf{Y}$, then it is enough to find one probability space that supports $\mathbf{Y}$ with given finite dimensional distributions for which ${\mathcal{C}}_0$ implies ${\mathcal{C}}_1$. The theorem of Skorokhod mentioned (see pages 6–8 of [3]) is to the effect that if the underlying probability space is (for measure theoretic purposes) the real line with measures given by, say, distribution functions $\mathbf{F}=(F_1,F_2,...)$; and if $\mathbf{F}$ is compact with respect to weak convergence, where $F_{q_l}$ converges to G in distribution; then if each $F_i$ is absolutely continuous with respect to some H (which itself is absolutely continuous), and $Y_i\left(x\right)=F_i\left(x\right)$, then $Y_{q_l}$ converges H-almost surely to a random variable Y.

Off of a set of probability 0, ${\sum }_{i,j}{\left(X_{ij}^{\left(q\right)}\right)}^2=IJ$ for all $q\ge 1$. We assume that $\mathbf{x}$ lies outside this “cursed” set. This is key to convergence.

Almost surely, ${\left(S_j^{(2m-1)}\right)}^2$ has positive limsup as m increases without bound.

Let $A_j=\{{\overline{lim}}_m{\left(S_j^{(2m-1)}\right)}^2=0\}$.

$P\left(A_j\right)=0$$j=1,2,...J$

Since the entries of $\mathbf{X}$ are independent, $\mathbf{X}$ is row and column exchangeable. This property is inherited by ${\mathbf{X}}^{\left(q\right)}$ for every q. Because all entries (for $q\ge 1$) are a.e. bounded uniformly in $\mathit{q}$, $E\left\{X_{ij}^{\left(q\right)}\right\}$ and $E\left\{{\left(X_{ij}^{\left(q\right)}\right)}^2\right\}$ exist and are finite (with fixed bound that applies to all q). Exchangeability implies that all $E\left\{X_{ij}^{\left(q\right)}\right\}=0$, and all $E\left\{{\left(X_{ij}^{\left(q\right)}\right)}^2\right\}=1$. Bounded convergence implies that if ${\left(S_j^{(2m-1)}\right)}^2$ tends to 0 along a subsequence as m increases, not only is the limit bounded as a function of $\mathbf{x}$, but also the limit random variable has expectation 0. Necessarily every almost sure subsequential limit in m of the random variables ${\overline{X}}_{·j}^{(2m-1)}$ has mean 0. Likewise, every almost sure subsequential limit in m of the random variables ${\left(X_{ij}^{(2m-1)}\right)}^2$ has expectation 1. All are bounded as functions of $\mathbf{x}$. One consequence of these matters is that $P\left(A_j\right)=0$.

The only subsequential almost sure limits of $\{{\left(S_j^{(2m-1)}\right)}^2:j=1,2,...\}$ and of $\{{\left(S_j^{\left(2m\right)}\right)}^2:j=1,2,...\}$ have expectation 1. Fix a $j,1\le j\le J$. Let $E=E\left(j\right)=\{i:$ for some $q_l,P(\overline{lim}|X_{ij}^{\left(q_l\right)}|>0)>0\}$. Column exchangeability implies that for any pair $(i,i^{\prime }),i\in E\left(j\right)$ iff $i^{\prime }\in E\left(j\right)$. The first sentence entails that $E\left(j\right)$ is not empty. Therefore, $E\left(j\right)=\{1,...,I\}$.

Let $i_0\ne i_1$. There is a subsequence of $\left\{q_l\right\}$—for simplicity write it as $\left\{q_l\right\}$—along which almost surely

$$\left(a\right)\underset{q_l}{lim}{X}_{i_{o}j}^{(2_{q_l}-1)}\mathrm{and}\underset{q_l}{lim}{X}_{i_{0}j}^{\left(2_{q_l}\right)}\mathrm{both}\mathrm{exist};$$

$$\left(b\right)\underset{q_l}{lim}{X}_{i_{1}j}^{(2_{q_l}-1)}\mathrm{and}\underset{q_l}{lim}{X}_{i_{1}j}^{\left(2_{q_l}\right)}\mathrm{both}\mathrm{exist};\mathrm{and}$$

If $P\left({lim}_{q_l}\right|X_{i_{o}j}^{\left(2_{q_l}\right)}-X_{i_{1}j}^{\left(2_{q_l}\right)}|=0)=1$ then exchangeability implies that ${\left(S_j^{(2_{q_l}-1)}\right)}^2\to 0$. So without loss of generality, if $E=\left\{\overline{{lim}_{q_l}}\right|X_{i_{o}j}^{\left(2_{q_l}\right)}-X_{i_{1}j}^{\left(2_{q_l}\right)}|>0\}$ then $P\left(E\right)>0$. Write

$$(X_{i_{o}j}^{\left(2_{q_l}\right)}-X_{i_{o}j}^{(2_{q_l}-1)})-(X_{i_{1}j}^{\left(2_{q_l}\right)}-X_{i_{1}j}^{(2_{q_l}-1)})=(X_{i_{o}j}^{(2_{q_l}-1)}-X_{i_{1}j}^{(2_{q_l}-1)})(S_j^{(2_{q_l}-1)}-1)/S_j^{(2_{q_l}-1)}.$$

Since ${\left(X_{i_{o}j}^{\left(2_{q_l}\right)}\right)}^2-{\left(X_{i_{o}j}^{(2_{q_l}-1)}\right)}^2\to 0$ a.s.((a) and the expectations of both are 1), and likewise with $i_o$ replaced by $i_1$, ((b), etc..), and because $x^2-y^2=(x-y)(x+y)$, the first expression of the immediately previous display tends to 0 on E. So, too, does the second expression. This is possible only if $S_j^{(2_{q_l}-1)}\to 1$ on E (we take the positive square root). On $E^c$, $S_j^{(2_{q_l}-1)}\to 0$. Since $E\left[S_j^{(2_{q_l}-1)}\right]\to 1$, $P\left(E^c\right)=0$. Further,

$$(X_{i_{o}j}^{\left(2_{q_l}\right)}-X_{i_{o}j}^{(2_{q_l}-1)})=\frac{X_{i_{o}j}^{(2_{q_l}-1)}(S_j^{(2_{q_l}-1)}-1)+{\overline{X}}_{·j}^{(2_{q_l}-1)}}{S_j^{(2_{q_l}-1)}}$$

As a corollary, one sees now that ${\overline{X}}_{·j}^{(2_{q_l}-1)}\to 0$ a.s. Since the original $\left\{q_l\right\}$ could be taken to be an arbitrary subsequence of $\left\{q\right\}$, we conclude that convergence of row and column means to 0 and convergence of row and column standard deviations to 1 takes place everywhere except on a set of Lebesgue measure 0.

Theorem 4.1 Efron’s algorithm converges almost surely for $\mathbf{X}$ on a Borel set of entries with complement a set of Lebesgue measure 0.

We turn now to a study of rates of convergence. To begin, define the following

$$\mathbf{Z}=\underset{m}{lim}{\mathbf{X}}^{\left(m\right)}\mathrm{a}.\mathrm{e}.\mathrm{and}{\lambda }_{ij}^{(2m-1)}=X_{ij}^{(2m-1)}-Z_{ij}.$$

Now let ${\lambda }_j={max}_i\left|{\lambda }_{ij}^{(2m-1)}\right|.$ Almost everywhere convergence and the fact that for each row and each column not every $Z_{ij}$ can be of the same sign enable us to conclude that for m large enough

$$|{\overline{X}}_{·j}^{(2m-1)}|\le \frac{(I-1)}{I}{\lambda }_j^{(2m-1)}$$

Remember that for $j=1,...,J\ge 3$, $\frac{1}{I}{\sum }_{i=1}^I{\left(Z_{ij}\right)}^2=1$, and analogously for $i=1,...,I\ge 3$.

Now write

$$\begin{array}{ccc}\hfill \frac{1}{I}\sum _{i=1}^I{(X_{ij}^{(2m-1)}-{\overline{X}}_{·j})}^2-\frac{1}{I}\sum _{i=1}^I{(Z_{ij}-0)}^2& =& a^2-b^2\hfill \\ & =& (a+b)(a-b),\mathrm{so}\hfill \\ \hfill a-1& =& [(a^2-b^2)/a+b]-1\hfill \end{array}$$

We know that for all $(i,j),{\overline{Z}}_{i·}={\overline{Z}}_{·j}=0$ a.e. To continue, we compute that

$$|X_{ij}^{\left(2m\right)}-X_{ij}^{(2m-1)}|=|X_{ij}^{(2m-1)}(S_j^{(2m-1)}-1)+{\overline{X}}_{·j}^{(2m-1)}|/S_j^{(2m-1)},$$

where $S_j^{(2m-1)}$ is the positive square root of ${\left(S_j^{(sm-1)}\right)}^2$.

Now write

$$\begin{array}{ccc}\hfill {(X_{ij}^{(2m-1)}-{\overline{X}}_{·j}^{(2m-1)})}^2& =& {[(X_{ij}^{(2m-1)}-Z_{ij})+(Z_{ij}-{\overline{Z}}_{·j})+({\overline{Z}}_{·j}-{\overline{X}}_{·j})]}^2\hfill \\ & =& [{(X_{ij}^{(2m-1)}-Z_{ij})}^2+{(Z_{ij}-{\overline{Z}}_{·j})}^2+{({\overline{Z}}_{·j}-{\overline{X}}^{(2m-1)})}^2\hfill \\ & +& 2(X_{ij}^{(2m-1)}-Z_{ij})(Z_{ij}-{\overline{Z}}_{·j})\hfill \\ & +& 2(X_{ij}^{(2m-1)}-Z_{ij})({\overline{Z}}_{·j}-{\overline{X}}_{·j}^{(2m-1)})\hfill \\ & +& 2(Z_{ij}-{\overline{Z}}_{·j})({\overline{Z}}_{·j}-{\overline{X}}_{·j}^{(2m-1)})\hfill \end{array}$$

One argues that

$$\frac{1}{I}\sum _{i=1}^I{(X_{ij}^{(2m-1)}-{\overline{X}}_{·j}^{(2m-1)})}^2\le D{\lambda }_j^2+\frac{2}{I}\sum _{(i=1)}^I(X_{ij}^{(2m-1)}-Z_{ij})(Z_{ij}-{\overline{Z}}_{·j})$$

where $D=D(I,J)<\infty .$ A key observation is that for every m, there exists $j=j\left(m\right)$ for which $\left\{{\lambda }_{ij}^{(2m-1)}\right\}$ are not of the same or all of the opposite sign as $\{Z_{ij}:i=1,...,I\}$, which, as was noted, are not, themselves of the same sign. Argue analogously regarding $\left\{{\lambda }_{ij}^{\left(2m\right)}\right\}$. Refer now to Figure 9. Our arguments show the correctness of the concentric circles in ${\mathbb{R}}^{IJ}$ on which ${\mathbf{X}}^{\left(\mathbf{n}\right)},{\mathbf{X}}^{(\mathbf{n}+\mathbf{1})},{\mathbf{X}}^{(\mathbf{n}+\mathbf{2})},\cdots$ ultimately lie. We conclude that to suitable approximation, the successive iterates lie on such circles with radii in geometric ratio. That ratio is uniformly $<1$, but is not arbitrarily close to 0. However Figure 9 buries a key idea in our study of convergence are rates. Thus, write ${\theta }^{\left(n\right)}$ for the angle between $X^{\left(n\right)}$ and Z. Obviously, the squared Frobenius norm $\parallel {\mathbf{X}}^{\left(n\right)}{-\mathbf{Z}\parallel }_F^2$ can be expressed as

$$\parallel {\mathbf{X}}^{\left(n\right)}{-\mathbf{Z}\parallel }_F^2=\parallel {\mathbf{X}}^{\left(n\right)}{\parallel }_F^2+{\parallel \mathbf{Z}\parallel }_F^2-(2cos\left({\theta }^{\left(n\right)}\right)\parallel {\mathbf{X}}^{\left(n\right)}{\parallel }_F{\parallel \mathbf{Z}\parallel }_F$$

Now rewrite $cos\left({\theta }^{\left(n\right)}\right)=1-\frac{1}{2IJ}(\parallel {\mathbf{X}}^{\left(n\right)}-\mathbf{Z}{\parallel }_F^2)$. Therefore, convergence of ${\mathbf{X}}^{\left(n\right)}$ to $\mathbf{Z}$ implies that ${\theta }^{\left(n\right)}$ can be taken arbitrarily close to 0. This is another way of saying that for each n, $\parallel {\mathbf{X}}^{\left(n\right)}{\parallel }_F^2=IJ$. Failure of this condition is why “simultaneous normalization" fails.

Figure 9. For n large enough $log\{{\parallel {\mathbf{X}}^{(n+2)}-{\mathbf{X}}^{(n+1)}\parallel }_F^2/{{\parallel \mathbf{X}}^{(n+1)}-{\mathbf{X}}^{\left(n\right)}\parallel }_F^2\}<1$ uniformly.

Figure 9. For n large enough $log\{{\parallel {\mathbf{X}}^{(n+2)}-{\mathbf{X}}^{(n+1)}\parallel }_F^2/{{\parallel \mathbf{X}}^{(n+1)}-{\mathbf{X}}^{\left(n\right)}\parallel }_F^2\}<1$ uniformly.

5. Illustrations of Convergence

We now illustrate the rapidity of convergence in the $3\times 3$ case to give a geometric perspective. First note that in the $3\times 3$ case the set of fixed points are characterized by 3 unique values. For instance, the following doubly normalized matrix which arises a result of successive normalization has only three unique elements.

$$\begin{array}{ccc}\hfill {\mathbf{X}}^{\left(\mathit{final}\right)}& =& \left[\begin{array}{ccc}-1.4137& 0.7407& 0.6730\\ 0.7407& 0.6730& -1.4137\\ 0.6730& -1.4137& 0.7407\end{array}\right]\hfill \end{array}$$

(4)

Hence we can use the first column of the limit matrix to represent the fixed point arising from successive normalization. Hence the curve of fixed points can be generated by applying the successive normalization process to random starting values. Figure 10 and Figure 11 below illustrates the curve that characterizes the set of fixed points for the $3\times 3$ matrix case when this numerical exercise is implemented. The origin is marked in black. The latter 3 subfigures superimpose the unit circle on the diagram in order to illustrate that the set of fixed points represent a “ring" around the unit sphere. Figure 12 considers different starting values and their respective paths of convergence to the “ring" of fixed points. Figure 13, Figure 14 and Figure 15 present 3 individual illustrations of different starting values and their respective paths of convergence to the “ring" of fixed points. For each example, a magnification or close-up of the path is provided. It is clear from these diagrams that the algorithm “accelerates" or “speeds up" as the sequence nears its limit.

Figure 10. (Left) Set of fixed points, (Right) Set of fixed points + unit sphere.

Figure 10. (Left) Set of fixed points, (Right) Set of fixed points + unit sphere.

Figure 11. Set of fixed points + unit sphere from different perspectives.

Figure 11. Set of fixed points + unit sphere from different perspectives.

Figure 12. Illustration of convergence to fixed points from multiple starts.

Figure 12. Illustration of convergence to fixed points from multiple starts.

Figure 13. (Left) Example of convergence to set of fixed points, (Right) Close-up of Example.

Figure 13. (Left) Example of convergence to set of fixed points, (Right) Close-up of Example.

Figure 14. (Left) Example of convergence to set of fixed points, (Right) Close-up of Example.

Figure 14. (Left) Example of convergence to set of fixed points, (Right) Close-up of Example.

6. Topics for Future Study

There are at least several directions for future research that continues to build from [1,2] and from material presented here. For one, Adam Olshen notes that often in applications, and for many reasons, $\mathbf{X}$ may not be exactly rectangular. Thus, some initial coordinates may be missing. By arguments not reported here we have shown that so long as “missingness" is independent of values that would be reported for a full $\mathbf{X}$, and no matter the realization there are almost surely genuine data for at least three rows for each column and at least three columns for each row, and so long as standard deviations are defined with “correct:" divisors, then convergence on all but a set of initial values of measure 0 is plausible. The process of rendering rows and columns with means 0 and standard deviations 1 is potentially a powerful “hammer" in search of a biological “nail". Just as studying data by (scale-free) correlations rather than in original given scales can lend insight, so, too, can the process of successive normalization described here.

Figure 15. (Left) Example of convergence to set of fixed points, (Right) Close-up of Example.

Figure 15. (Left) Example of convergence to set of fixed points, (Right) Close-up of Example.