Learning Coefficient of Vandermonde Matrix-Type Singularities in Model Selection

Abstract

In recent years, selecting appropriate learning models has become more important with the increased need to analyze learning systems, and many model selection methods have been developed. The learning coefficient in Bayesian estimation, which serves to measure the learning efficiency in singular learning models, has an important role in several information criteria. The learning coefficient in regular models is known as the dimension of the parameter space over two, while that in singular models is smaller and varies in learning models. The learning coefficient is known mathematically as the log canonical threshold. In this paper, we provide a new rational blowing-up method for obtaining these coefficients. In the application to Vandermonde matrix-type singularities, we show the efficiency of such methods.

1. Introduction

In recent studies, real data associated with, for example, image or speech recognition, psychology, and economics, have been analyzed by learning systems. Hence, many learning models have been proposed, and thus, the need for appropriate model selection methods has increased.

In this section, we first introduce the widely-applicable information criterion (WAIC) [1,2,3,4,5,6,7] and cross-validation in Bayesian model selection.

Let $q\left(x\right)$ be a true probability density function of variables, $x\in {\mathbb{R}}^N$, and let $x^n:={\left\{x_i\right\}}_{i=1}^n$ be n training samples selected from $q\left(x\right)$ independently and identically. Consider a learning model that is written in probabilistic form as $p\left(x\right|w)$, where $w\in W\subset {\mathbb{R}}^d$ is a parameter.

Suppose that the purpose of the learning system is to estimate the unknown true density function $q\left(x\right)$ from $x^n$ using $p\left(x\right|w)$ in Bayesian estimation. Let $\psi \left(w\right)$ be an a priori probability density function on parameter set W and $p\left(w\right|x^n)$ be the a posteriori probability density function:

$$p\left(w\right|x^n)=\frac{1}{Z_n\left(\beta \right)}\psi \left(w\right)\prod _{i=1}^{n}p{\left(x_i\right|w)}^{\beta },$$

where:

$$Z_n\left(\beta \right)={\int }_W\psi \left(w\right)\prod _{i=1}^{n}p{\left(x_i\right|w)}^{\beta }\mathrm{d}w,$$

for inverse temperature $\beta$. We typically set $\beta =1$. Define:

$$E_w^{\beta }\left[f\left(w\right)\right]=\frac{\int dwf\left(w\right)\psi \left(w\right){\prod }_{i=1}^{n}p{\left(x_i\right|w)}^{\beta }}{\int dw\psi \left(w\right){\prod }_{i=1}^{n}p{\left(x_i\right|w)}^{\beta }},$$

and:

$$V_w^{\beta }\left[f\left(w\right)\right]=E_w^{\beta }[f{\left(w\right)}^2]-E_w^{\beta }{\left[f\left(w\right)\right]}^2.$$

We then have predictive density function $p\left(x\right|x^n)=E_w^{\beta }\left[p\left(x\right|w)\right]$, which is the average inference of the Bayesian density function.

We next introduce Kullback function $K\left(q\right|\left|p\right)$ and empirical Kullback function $K_n\left(q\right|\left|p\right)$ for density functions $p\left(x\right),q\left(x\right)$:

$$K\left(q\right|\left|p\right)=\int q\left(x\right)log\frac{q\left(x\right)}{p\left(x\right)}dx,$$

$$K_n\left(q\right|\left|p\right)=\frac{1}{n}\sum _{i=1}^{n}log\frac{q\left(x_i\right)}{p\left(x_i\right)}.$$

Function $K\left(p\right|\left|q\right)$, which always has a non-negative value and satisfies $K\left(q\right|\left|p\right)=0$, if and only if $q\left(x\right)=p\left(x\right)$, is a pseudo-distance between density functions $p\left(x\right),q\left(x\right)$. We define Bayes training loss $T_n$ and Bayes generalization loss $G_n$ as follows:

$$T_n=-\frac{1}{n}\sum _{i=1}^{n}logp\left(x_i\right|x^n),$$

and:

$$G_n=-\int q\left(x\right)logp\left(x\right|x^n)dx.$$

Additionally, we define Bayesian generalization error $B_g$ and Bayesian training error $B_t$ as follows:

$$B_g=K(q\left(x\right)\parallel p\left(x\right|x^n))$$

and

$$B_t=K_n(q\left(x\right)\parallel p\left(x\right|x^n)).$$

Then, we have:

$$\begin{array}{ccc}\hfill E\left[T_n\right]& =& G_n,\hfill \\ \hfill B_g& =& G_n-S,\hfill \\ \hfill B_t& =& T_n-S_n\hfill \end{array}$$

for average entropy $S=-\int q\left(x\right)logq\left(x\right)dx$ and empirical entropy $S_n=-\frac{1}{n}{\sum }_{i=1}^{n}logq\left(x_i\right)$ of the true density function. Value $B_g$ describes how precisely the predictive function approximates the true density function.

We define $x^n\backslash x_i=\{x_1,\dots ,x_{i-1},x_{i+1},\dots ,x_n\}$. The WAIC is denoted by:

$$W_n=T_n+\frac{\beta }{n}\sum _{i=1}^n{V}_w^{\beta }[logp\left(x_i\right|w)]$$

and the cross-validation loss is denoted by:

$$C_n=-\frac{1}{n}\sum _{i=1}^{n}logp(x_i|x^n\backslash x_i)$$

for $n\ge 2$.

Watanabe [2,3,6,7] proved the following four relations:

$$\begin{array}{ccc}\hfill E\left[G_n\right]& =& L\left(w_0\right)+{\displaystyle \frac{1}{n\beta }}\left(\lambda +{\displaystyle \frac{\beta -1}{\beta }}\nu \right)+o({\displaystyle \frac{1}{n\beta }}),\hfill \\ \hfill E\left[T_n\right]& =& L\left(w_0\right)+{\displaystyle \frac{1}{n\beta }}\left(\lambda -{\displaystyle \frac{\beta +1}{\beta }}\nu \right)+o({\displaystyle \frac{1}{n\beta }}),\hfill \\ \hfill E\left[W_n\right]& =& L\left(w_0\right)+{\displaystyle \frac{1}{n\beta }}\left(\lambda +{\displaystyle \frac{\beta -1}{\beta }}\nu \right)+o({\displaystyle \frac{1}{n\beta }}),\hfill \\ \hfill E\left[C_n\right]& =& L\left(w_0\right)+{\displaystyle \frac{1}{n\beta }}\left(\lambda +{\displaystyle \frac{\beta -1}{\beta }}\nu \right)+o({\displaystyle \frac{1}{n\beta }})\hfill \end{array}$$

for learning coefficient $\lambda \in \mathbb{Q}$ and singular fluctuation $\nu \in \mathbb{R}$, where $L\left(w\right)=-E_x[logp\left(x\right|w)]$ and $w_0\in W_0=\{w_0\in W\parallel L\left(w_0\right)={min}_{w\in W}L\left(w\right)\}$. For singular models, the set $W_0$ is typically not a singleton set. Nevertheless, the WAIC and the cross-validation can estimate the Bayesian generalization error without any knowledge of the true probability density function.

These values are calculated from training samples $x_i$ using learning model p. In real applications or experiments, we typically do not know the true distribution, but only the values of the training errors. Our purpose is to show that these methods are effective. We can select a suitable model from several statistical models by observing these values.

In this paper, we consider the value $\lambda$, which is equal to the log canonical threshold introduced in Definition 1. This coefficient is not needed to evaluate the WAIC and the cross-validation in practice, while the learning coefficients from our recent results have been used very effectively by Drton and Plummer [8] for model selection using a method called the “singular Bayesian information criterion (sBIC)”. The method sBIC even works using the bounds of learning coefficients.

It is known that $\lambda =\nu =d/2$ holds, where d is the dimension of the parameter space for regular models. Value $\lambda$ is obtained by a blowing-up process, which is the main tool in the desingularization of an algebraic variety. The following theorem is an analytic version of Hironaka’s theorem [9] used by Atiyah [10].

Theorem 1

(Desingularization [9]). Let f be an analytic function in a neighborhood of $w^{*}=(w_1^{*},\cdots ,w_d^{*})\in {\mathbf{R}}^d$ with $f\left(w^{*}\right)=0$. There exists an open set $U\ni w^{*}$, an analytic manifold M, and a proper analytic map μ from M to U such that: (1) $\mu :M-\mathcal{E}\to U-f^{-1}\left(0\right)$ is an isomorphism, where $\mathcal{E}={\mu }^{-1}\left(f^{-1}\left(0\right)\right)$, and: (2) for each $u\in M$, there is a local analytic coordinate system $(u_1,\cdots ,u_n)$ such that $f\left(\mu \left(u\right)\right)=\pm u_1^{s_1}{u}_2^{s_2}\cdots u_n^{s_n}$, where $s_1,\cdots ,s_n$ are non-negative integers.

The theorem establishes the existence of the desingularization map; however, generally, it is still difficult to obtain such maps for Kullback functions because the singularities of these functions are very complicated. From learning coefficient $\lambda$ and its order $\theta$, value $\nu$ is obtained theoretically as follows: Let $\xi \left(u\right)$ be an empirical process defined on the manifold obtained by a resolution of singularities and ${\sum }_{u^{*}}$ denote the sum of local coordinates that attain the minimum $\lambda$ and maximum $\theta$. We then have:

$${\displaystyle \nu =\frac{1}{2}{E}_{\xi }\frac{{\int }_0^{\infty }dt{\sum }_{u^{*}}\int du\xi \left(u\right)t^{\lambda -1/2}{e}^{-\beta t+\beta \sqrt{t}\xi \left(u\right)}}{{\int }_0^{\infty }dt{\sum }_{u^{*}}\int du{t}^{\lambda -1/2}{e}^{-\beta t+\beta \sqrt{t}\xi \left(u\right)}},}$$

(1)

where $\xi \left(u\right)$ is a random variable of a Gaussian process with mean zero, and covariance $E_{\xi }\left[\xi \left(w\right)\xi \left(u\right)\right]=E_X\left[a(x,w)a(x,u)\right]$ for the analytic function $a(x,w)$ obtained by the resolution of singularities using $log\left(q\right(x)/p(x\left|w\right))$.

Our purpose in this paper is to obtain $\lambda$. In recent studies, we determined the learning coefficients for reduced rank regression [11], the three-layered neural network with one input unit and one output unit [12,13], normal mixture models with a dimension of one [14], and the restricted Boltzmann machine [15]. Additionally, Rusakov and Geiger [16,17] and Zwiernik [18], respectively, obtained the learning coefficients for naive Bayesian networks and directed tree models with hidden variables. Drton et al. [19] considered these coefficients of the Gaussian latent tree and forest models.

The papers [20,21] derived bounds on the learning coefficients for Vandermonde matrix-type singularities and explicit values under some conditions.

The remainder of this paper is structured as follows: In Section 2, we introduce log canonical thresholds in algebraic geometry. In Section 3, we summarize key theorems for obtaining learning coefficients for learning theory. In Section 4, we present our main results. We consider the log canonical thresholds of Vandermonde matrix-type singularities (Definition 3). We present our conclusions in Section 5.

2. Log Canonical Threshold

Definition 1.

Let f be an analytic function in neighborhood U of $w^{*}$. Let ψ be a $C^{\infty }$ function with a compact support. Define log canonical threshold ${\lambda }_{w^{*}}(f,\psi )$ as the largest pole of ${\int }_U{\left|f\right|}^{2z}\psi dw$ over $\mathbb{C}$ or ${\int }_U{\left|f\right|}^z\psi dw$ over $\mathbb{R}$. Additionally, define ${\theta }_{w^{*}}(f,\psi )$ by its order. If $\psi \left(w^{*}\right)\ne 0$, then we define ${\lambda }_{w^{*}}\left(f\right)={\lambda }_{w^{*}}(f,\psi )$ and ${\theta }_{w^{*}}\left(f\right)={\theta }_{w^{*}}(f,\psi )$ because the log canonical threshold and its order are independent of ψ.

Applying Hironaka’s Theorem 1 to function $f\left(w\right)$, we have the proper analytic map $\mu$ from manifold M to neighborhood U of $w^{*}$ that satisfies Hironaka’s Theorems (1) and (2). Then, integration ${\int }_U{\left|f\right|}^z\psi \left(w\right)dw$ is equal to ${\int }_M{|f\circ \mu |}^z\psi \left(\mu \left(u\right)\right){\mu }^{\prime }\left(u\right)du$, which is the sum of ${\int }_{U_M}|u_1^{s_1}{u}_2^{s_2}\cdots u_d^{s_d}{|}^z\psi \left(\mu \left(u\right)\right)\left|{\mu }^{\prime }\left(u\right)\right|\mathrm{d}u$, where $(u_1,\cdots ,u_d)$ is a local analytic coordinate system on $U_M\subset M$. Therefore, the poles can be obtained. Note that for each $w^{*}$ with $f\left(w^{*}\right)\ne 0$, there exists neighborhood U such that $f\left(w\right)\ne 0$ for all $w\in U$. Thus, ${\int }_U{\left|f\right|}^z\psi \left(w\right)dw$ has no poles. The learning coefficient is the log canonical threshold of the Kullback function (relative entropy) over the real field.

3. Main Theorems

In this section, several theorems are introduced for obtaining real log canonical thresholds over the real field. Theorem 2 (the method for determining the deepest singular point), Theorem 3 (the method to add variables), and Theorem 4 (the rational blowing-up method) are very helpful for obtaining the log canonical threshold. These theorems over the real field are useful for reducing the number of changes of coordinates via blow-ups.

We denote constants, such as $a^{*}$, $b^{*}$, and $w^{*}$, by suffix ∗. Define the norm of a matrix $C=\left(c_{ij}\right)$ as $\left|\right|C\left|\right|=\sqrt{{\sum }_{i,j}{\left|c_{ij}\right|}^2}$. Set ${\mathbb{N}}_{+0}=\mathbb{N}\cup \left\{0\right\}$.

Lemma 1

([14,22,23]). Let U be a neighborhood of $w^{*}\in {\mathbb{R}}^d$. Consider the ring of analytic functions on U. Let $\mathcal{J}$ be the ideal generated by $f_1,\dots ,f_n$, which are analytic functions defined on U. $\left(1\right)$ If $g_1^2+\dots +g_m^2\le f_1^2+\cdots +f_n^2$, then ${\lambda }_{w^{*}}(g_1^2+\cdots +g_m^2)\le {\lambda }_{w^{*}}(f_1^2+\cdots +f_n^2).$

$\left(2\right)$ If $g_1,\dots ,g_m\in \mathcal{J}$, then ${\lambda }_{w^{*}}(g_1^2+\cdots +g_m^2)\le {\lambda }_{w^{*}}(f_1^2+\cdots +f_n^2).$ In particular, if $g_1,\dots ,g_m$ generate ideal $\mathcal{J}$, then ${\lambda }_{w^{*}}(f_1^2+\cdots +f_n^2)={\lambda }_{w^{*}}(g_1^2+\cdots +g_m^2).$

The following lemma is also used in the proofs.

Lemma 2

([15]). Let $\mathcal{J},{\mathcal{J}}^{\prime }$ be the ideals generated by $f_1\left(w\right),\dots ,f_n\left(w\right)$ and $g_1\left(w^{\prime }\right),\dots ,g_m\left(w^{\prime }\right)$, respectively. If w and $w^{\prime }$ are different variables, then:

$${\lambda }_{(w^{*},w^{\prime *})}(f_1^2+\cdots +f_n^2+g_1^2+\cdots +g_m^2)={\lambda }_{w^{*}}(f_1^2+\cdots +f_n^2)+{\lambda }_{w^{\prime *}}(g_1^2+\cdots +g_m^2).$$

Theorem 2

(Method for determining the deepest singular point [21]). Let $f_1(w_1,\dots ,w_d)$, …, $f_m(w_1,\dots ,w_d)$ be homogeneous functions of $w_1,\cdots ,w_j$$(j\le d)$. Furthermore, let ψ be a $C^{\infty }$ function such that $\psi (0,\cdots ,0,w_{j+1}^{*},\cdots ,w_d^{*})\ge \psi (w_1^{*},\cdots ,w_d^{*})$ and ${\psi }_w$ is a homogeneous function of $w_1,\cdots ,w_j$ in a small neighborhood of $(0,\cdots ,0,w_{j+1}^{*},\cdots ,w_d^{*})$. Then, we have:

$${\lambda }_{(0,\cdots ,0,w_{j+1}^{*},\cdots ,w_d^{*})}(f_1^2+\cdots +f_m^2,\psi )\le {\lambda }_{(w_1^{*},\cdots ,w_j^{*},w_{j+1}^{*},\cdots ,w_d^{*})}(f_1^2+\cdots +f_m^2,\psi )$$

Theorem 3

(Method to add variables [21]). Let $f_1(w_1,\dots ,w_d)$, …, $f_m(w_1,\dots ,w_d)$ be homogeneous functions of $w_1,\cdots ,w_d$. Set $f_1^{\prime }(w_2,\dots ,w_d)=f_1(1,w_2,\dots ,w_d)$, …, $f_m^{\prime }(w_2,\dots ,w_d)=f_m(1,w_2,\dots ,w_d)$. If $w_1^{*}\ne 0$, then we have:

$${\lambda }_{(w_1^{*},\cdots ,w_d^{*})}(f_1^2+\cdots +f_m^2)={\lambda }_{(w_2^{*}/w_1^{*},\cdots ,w_d^{*}/w_1^{*})}(f_1^{\prime 2}+\cdots +f_m^{\prime 2}).$$

Resolutions of singularities are obtained by constructing the blow-up along the smooth submanifold. In this paper, we use the blow-up method along some singular varieties as explained below for obtaining log canonical thresholds.

Theorem 4 (Rational blow-up process).

Let $m\le d$ and $({\alpha }_1,\dots ,{\alpha }_m)\in {\mathbf{R}}^m,{\alpha }_1,\dots ,{\alpha }_m>0$. Consider the set $U=\{w=(w_1,\dots ,w_d)\in {\mathbf{R}}^d|0\le w_1,\dots ,w_d\le 1\}$.

We set $g_i\left(u_i\right)=(g_{1i}\left(u_i\right),\dots ,g_{di}\left(u_i\right))$ for $g_{ii}\left(u_i\right)=u_{ii}$, $g_{ji}\left(u_i\right)=u_{ji}{u}_{ii}^{{\alpha }_i/{\alpha }_j}(1\le j\le m,j\ne i)$, $g_{ji}\left(u_i\right)=u_{ji}(1\le j\le d)$.

Let $f,$ ψ be analytic functions defined on U and $f_i(u_{1i},\cdots ,u_{di})=f\left(g_i\left(u_i\right)\right)$. Then, we have:

$$\underset{1\le i\le m}{min}\{{\lambda }_0(f_i\left(u_i\right),u_{ii}^{{\sum }_{j=1}^m{\alpha }_i/{\alpha }_j-1}\psi \left(g_i\left(u_i\right)\right))\}={\lambda }_0(f,\psi ).$$

Proof. 

The proof of this theorem uses a resolution of singularities along a smooth submanifold.

Set $w_i^{\prime }=w_i^{1/{\alpha }_i}$, and construct the blow-up along the submanifold $\{w_1^{\prime }=\cdots =w_m^{\prime }=0\}$. Then, we have for $1\le i\le m$, $w_i^{\prime }=v$, $w_j^{\prime }=v{w}_j^{″}$$(1\le j\le m,j\ne i)$, and $w_j^{\prime }=w_j^{″}$$(m+1\le j\le d)$.

Consider the set $U_i=\{u_i=(u_{1i},\dots ,u_{di})\in {\mathbf{R}}^d|0\le u_{1i},\dots ,u_{di}\le 1\}$ for $1\le i\le m$.

Set $u_{ii}=v^{{\alpha }_i},u_{ji}={w_j^{″}}^{{\alpha }_j}(j\ne i)$, then we have $w_i=u_{ii}$, $w_j=u_{ii}^{{\alpha }_j/{\alpha }_i}{u}_{ji}$$(1\le j\le m,j\ne i)$, and $w_j=u_{ji}(m+1\le j\le d)$.

The Jacobian $\partial w/\partial u_i$ is ${\prod }_{1\le j\le m,j\ne i}{u}_{ii}^{{\alpha }_j/{\alpha }_i}$. □

The theorem is simple; however, it is a useful tool for obtaining log canonical thresholds.

4. Main Results

In this section, we apply the theorems in Section 3 to Vandermonde matrix-type singularities, which are generic and essential in learning theory. Their associated log canonical thresholds provide the learning coefficients of, for example, three-layered neural networks in Section 4.1, normal mixture models in Section 4.2, and mixtures of binomial distributions [24].

4.1. Three-Layered Neural Network

Consider the three-layered neural network with N input units, H hidden units, and M output units, which is trained for estimating the true distribution with r hidden units. Denote an input value by $z=\left(z_j\right)\in {\mathbb{R}}^N$ with a probability density function $q\left(z\right)$. Then, an output value $y=\left(y_k\right)\in {\mathbb{R}}^M$ of the three layered neural network is given by $y_k=f_k(z,w)+\left(\mathrm{noise}\right),$ where $w={\{a_{ki},b_{ij}\}}_{1\le i\le H}$ and:

$$f_k(z,w)=\sum _{i=1}^H{a}_{ki}tanh(\sum _{j=1}^N{b}_{ij}{z}_j).$$

Consider a statistical model:

$$p\left(y\right|z,w)=\frac{1}{{\left(2\pi \right)}^{M/2}}exp(-\frac{1}{2}\left|\right|y-{f(z,w)\left|\right|}^2),$$

and $p(z,y|w)=p(y|z,w)q\left(z\right)$. Assume that the true distribution:

$$p\left(y\right|z,w_t^{*})=\frac{1}{{\left(2\pi \right)}^{M/2}}exp(-\frac{1}{2}\left|\right|y-f(z,w_t^{*}){\left|\right|}^2),$$

is included in the learning model, where $w_t^{*}={\{a_{k,H+i}^{*},b_{H+i,j}^{*}\}}_{1\le i\le r}$ and $f_k(z,w_t^{*})={\sum }_{i=1}^r(-a_{k,H+i}^{*})tanh({\sum }_{j=1}^N{b}_{H+i,j}^{*}{x}_j).$

We have:

$$p(z,y|w)=\frac{1}{{\left(2\pi \right)}^{M/2}}exp(-\frac{1}{2}\left|\right|y-{f(z,w)\left|\right|}^2)q\left(z\right),$$

and the notation $(z,y)$ for the three-layered neural network corresponds to x in Section 1 and Section 2.

4.2. Normal Mixture Models

We consider a normal mixture model [14] with identity matrix variances:

$$p\left(x\right|w)=\frac{1}{{\left(2\pi \right)}^{N/2}}\sum _{i=1}^H{a}_{i}exp(-\frac{{\sum }_{j=1}^N{(z_j-b_{ij})}^2}{2}),$$

where $w={\{a_i,b_{ij}\}}_{1\le i\le H}$ and ${\sum }_{i=1}^H{a}_i=1$, $a_i\ge 0$.

Set the true distribution by:

$$p\left(x\right|w_t^{*})=\frac{1}{{\left(2\pi \right)}^{N/2}}\sum _{i=H+1}^{H+r}(-a_i^{*})exp(-\frac{{\sum }_{j=1}^N{(z_j-b_{ij}^{*})}^2}{2}),$$

where $w_t^{*}={\{a_i^{*},b_{ij}^{*}\}}_{H+1\le i\le H+r}$ and ${\sum }_{i=H+1}^{H+r}{a}_i^{*}=-1$, $a_i^{*}<0$. (In order to simplify the following, we use the values $a_i^{*}<0$, not $a_i^{*}>0$.)

4.3. Vandermonde Matrix-Type Singularities

Definition 2.

Fix $Q\in \mathbb{N}$. Define: ${[b_1^{*},b_2^{*},\cdots ,b_N^{*}]}_Q={\gamma }_i(0,\cdots ,0,b_i^{*},\cdots ,b_N^{*})$ if $b_1^{*}=\cdots =b_{i-1}^{*}=0$, $b_i^{*}\ne 0$, and ${\gamma }_i=\left\{\begin{array}{cc}1\hfill & \mathit{if}Q\mathit{is}\mathit{odd},\hfill \\ \mathrm{sign}\left(b_i^{*}\right)\hfill & \mathit{if}Q\mathit{is}\mathit{even}.\hfill \end{array}\right.$

For simplicity, we use the notation $w={\{a_{ki},b_{ij}\}}_{1\le i\le H}$ instead of $w={\{a_{ki},b_{ij}\}}_{1\le k\le M,1\le i\le H,1\le j\le N}$ because we always have $1\le k\le M$ and $1\le j\le N$ in this section.

Definition 3.

Fix $Q\in \mathbb{N}$ and $m\in {\mathbb{N}}_{+0}$.

Let $A=\left(\begin{array}{cccccc}{a}_{11}\hfill & \cdots \hfill & a_{1H}\hfill & a_{1,H+1}^{*}\hfill & \dots \hfill & a_{1,H+r}^{*}\hfill \\ a_{21}\hfill & \cdots \hfill & a_{2H}\hfill & a_{2,H+1}^{*}\hfill & \dots \hfill & a_{2,H+r}^{*}\hfill \\ & ⋮\hfill & & & ⋮\hfill & \\ a_{M1}\hfill & \cdots \hfill & a_{MH}\hfill & a_{M,H+1}^{*}\hfill & \dots \hfill & a_{M,H+r}^{*}\hfill \end{array}\right)$,

$$B_I={(\prod _{j=1}^N{b}_{1j}^{{\ell }_j},\prod _{j=1}^N{b}_{2j}^{{\ell }_j},\cdots ,\prod _{j=1}^N{b}_{Hj}^{{\ell }_j},\prod _{j=1}^N{b_{H+1,j}^{*}}^{{\ell }_j},\cdots ,\prod _{j=1}^N{b_{H+r,j}^{*}}^{{\ell }_j})}^t,$$

for $I=({\ell }_1,\dots ,{\ell }_N)\in {{\mathbb{N}}_{+0}}^N$, and:

$$B={\left(B_I\right)}_{{\ell }_1+\cdots +{\ell }_N=Qn+m,n\ge 0}=(B_{(m,0,\cdots ,0)},B_{(m-1,1,\cdots ,0)},\cdots ,B_{(0,0,\cdots ,m)},B_{(m+Q,0,\cdots ,0)},\cdots )$$

(t denotes the transpose).

$a_{ki}$ and $b_{ij}$ $(1\le k\le M,1\le i\le H,1\le j\le N)$ are the variables in a neighborhood of $a_{ki}^{*}$ and $b_{ij}^{*}$, where $a_{ki}^{*}$ and $b_{ij}^{*}$ are fixed constants.

Let $\mathcal{J}$ be the ideal generated by the elements of $AB$.

We call singularities of $\mathcal{J}$ Vandermonde matrix-type singularities.

To simplify, we usually assume that:

$${(a_{1,H+j}^{*},a_{2,H+j}^{*},\cdots ,a_{M,H+j}^{*})}^t\ne 0,(b_{H+j,1}^{*},b_{H+j,2}^{*},\cdots ,b_{H+j,N}^{*})\ne 0,$$

for $1\le j\le r$ and:

$${[b_{H+j,1}^{*},b_{H+j,2}^{*},\cdots ,b_{H+j,N}^{*}]}_Q\ne {[b_{H+j^{\prime },1}^{*},b_{H+j^{\prime },2}^{*},\cdots ,b_{H+j^{\prime },N}^{*}]}_Q,$$

for $j\ne j^{\prime }$.

Example 1.

If $m=N=M=r=1$, $Q=2$, $H=3$, then we have $A=\left(\begin{array}{cccc}{a}_{11}\hfill & a_{12}\hfill & a_{13}\hfill & a_{14}^{*}\hfill \end{array}\right)$, $B=\left(\begin{array}{cccc}{b}_{11}\hfill & b_{11}^3\hfill & b_{11}^5\hfill & b_{11}^7\hfill \\ b_{21}\hfill & b_{21}^3\hfill & b_{11}^5\hfill & b_{21}^7\hfill \\ b_{31}\hfill & b_{31}^3\hfill & b_{11}^5\hfill & b_{31}^7\hfill \\ b_{41}^{*}\hfill & {b_{41}^{*}}^3\hfill & {b_{41}^{*}}^5\hfill & {b_{41}^{*}}^7\hfill \end{array}\right)$. These matrices $A,B$ correspond to the three-layered neural network:

$$p\left(y\right|z,w)=\frac{1}{{\left(2\pi \right)}^{1/2}}exp(-\frac{1}{2}{(y-a_{11}tanh\left(b_{11}z\right)-a_{12}tanh\left(b_{21}z\right)-a_{13}tanh\left(b_{31}z\right))}^2),$$

and the true distribution:

$$p\left(y\right|x,w_t^{*})=\frac{1}{{\left(2\pi \right)}^{1/2}}exp(-\frac{1}{2}{(y+a_{14}^{*}tanh\left(b_{41}^{*}z\right))}^2).$$

Example 2.

If $Q=r=M=1$, $H=2,N=2$, then we have $A=\left(\begin{array}{ccc}{a}_{11}\hfill & a_{12}\hfill & a_{13}^{*}\hfill \end{array}\right)$, $B=\left(\begin{array}{ccccccccc}{b}_{11}\hfill & b_{12}\hfill & b_{11}^2\hfill & b_{11}{b}_{12}\hfill & b_{12}^2\hfill & b_{11}^3\hfill & b_{11}{b}_{12}^2\hfill & b_{11}^2{b}_{12}\hfill & b_{12}^3\hfill \\ b_{21}\hfill & b_{22}\hfill & b_{21}^2\hfill & b_{21}{b}_{22}\hfill & b_{22}^2\hfill & b_{21}^3\hfill & b_{21}{b}_{22}^2\hfill & b_{21}^2{b}_{22}\hfill & b_{22}^3\hfill \\ b_{31}^{*}\hfill & b_{32}^{*}\hfill & {b_{31}^{*}}^2\hfill & b_{31}^{*}{b}_{32}^{*}\hfill & {b_{32}^{*}}^2\hfill & {b_{31}^{*}}^3\hfill & b_{31}^{*}{b_{32}^{*}}^2\hfill & {b_{31}^{*}}^2{b}_{32}^{*}\hfill & {b_{32}^{*}}^3\hfill \end{array}\right).$

If $a_{13}^{*}=-1$, these matrices $A,B$ correspond to a normal mixture model with identity matrix variances:

$$p\left(x\right|w)=\frac{a_{11}}{2\pi }exp(-\frac{{(z_1-b_{11})}^2+{(z_2-b_{12})}^2}{2})+\frac{a_{12}}{2\pi }exp(-\frac{{(z_1-b_{21})}^2+{(z_2-b_{22})}^2}{2}),$$

${\sum }_{i=1}^2{a}_{1i}=1$, $a_{1i}\ge 0$, and the true distribution is:

$$p\left(x\right|w_t^{*})=\frac{1}{2\pi }(-a_{13}^{*})exp(-\frac{{(z_1-b_{31}^{*})}^2+{(z_2-b_{32}^{*})}^2}{2}),-a_{13}^{*}=1.$$

In this paper, we denote:

$$\begin{array}{c}{A}_{M,H}=\left(\begin{array}{cccc}{a}_{11}\hfill & a_{12}\hfill & \cdots \hfill & a_{1H}\hfill \\ a_{21}\hfill & a_{22}\hfill & \cdots \hfill & a_{2H}\hfill \\ & & ⋮\hfill & \\ a_{M1}\hfill & a_{M2}\hfill & \cdots \hfill & a_{MH}\hfill \end{array}\right),B_{H,N,I}=\left(\begin{array}{c}\prod _{j=1}^N{b_{1j}}^{{\ell }_j}\hfill \\ \prod _{j=1}^N{b_{2j}}^{{\ell }_j}\hfill \\ ⋮\hfill \\ \prod _{j=1}^N{b_{Hj}}^{{\ell }_j}\hfill \end{array}\right)and:\hfill \end{array}$$

$B_{H,N}^{\left(Q\right)}={\left(B_{H,N,I}\right)}_{{\ell }_1+\dots +{\ell }_N=Qn+1,0\le n\le H-1}$.

Furthermore, we denote: ${\mathbf{a}}^{*}=\left(\begin{array}{c}{a}_{1,H+1}^{*}\hfill \\ ⋮\hfill \\ a_{M,H+1}^{*}\hfill \end{array}\right)$ and:

$$\begin{array}{c}(A_{M,H},{\mathbf{a}}^{*})=\left(\begin{array}{ccccc}{a}_{11}\hfill & a_{12}\hfill & \cdots \hfill & a_{1H}\hfill & a_{1,H+1}^{*}\hfill \\ a_{21}\hfill & a_{22}\hfill & \cdots \hfill & a_{2H}\hfill & a_{2,H+1}^{*}\hfill \\ & & ⋮\hfill & & \\ a_{M1}\hfill & a_{M2}\hfill & \cdots \hfill & a_{MH}\hfill & a_{M,H+1}^{*}\hfill \end{array}\right)\hfill \end{array}$$

Theorem 5

([14]). Consider sufficiently small neighborhood U of:

$$w^{*}={\{a_{ki}^{*},b_{ij}^{*}\}}_{1\le i\le H}$$

and variables $w={\{a_{ki},b_{ij}\}}_{1\le i\le H}$ in set U. Set $(b_{01}^{**},b_{02}^{**},\cdots ,b_{0N}^{**})=(0,\dots ,0)$. Let each $(b_{11}^{**},b_{12}^{**},\cdots ,b_{1N}^{**})$, …, $(b_{r^{\prime }1}^{**},b_{r^{\prime }2}^{**},\cdots ,b_{r^{\prime }N}^{**})$ be a different real vector in:

$${[b_{i1}^{*},b_{i2}^{*},\cdots ,b_{iN}^{*}]}_Q\ne 0,for i=1,\dots ,H+r;$$

that is,

$$\{(b_{11}^{**},\cdots ,b_{1N}^{**}),\dots ,(b_{r^{\prime }1}^{**},\cdots ,b_{r^{\prime }N}^{**});{[b_{i1}^{*},\cdots ,b_{iN}^{*}]}_Q\ne 0,i=1,\dots ,H+r\}.$$

Then, $r^{\prime }$ is uniquely determined, and $r^{\prime }\ge r$ by the assumption in Definition 3. Set $(b_{i1}^{**},\cdots ,b_{iN}^{**})={[b_{H+i,1}^{*},\cdots ,b_{H+i,N}^{*}]}_Q,$ for $1\le i\le r$. Assume that ${[b_{i1}^{*},\cdots ,b_{iN}^{*}]}_Q=\left\{\begin{array}{cc}0,& 1\le i\le H_0\hfill \\ (b_{11}^{**},\cdots ,b_{1N}^{**}),& H_0+1\le i\le H_0+H_1,\hfill \\ (b_{21}^{**},\cdots ,b_{2N}^{**}),& H_0+H_1+1\le i\le H_0+H_1+H_2,\hfill \\ ⋮\\ (b_{r^{\prime }1}^{**},\cdots ,b_{r^{\prime }N}^{**}),& H_0+\cdots +H_{r^{\prime }-1}+1\le i\le H_0+\cdots +H_{r^{\prime }},\hfill \end{array}\right.$

and $H_0+\cdots +H_{r^{\prime }}=H$. We then have:

$${\lambda }_{w^{*}}{\left(\right|\left|AB\right||}^2)=\frac{M{r}^{\prime }}{2}+{\lambda }_{{w_1^{\left(0\right)}}^{*}}(||A_{M,H_0}{B}_{H_0,N}^{\left(Q\right)}{\left|\right|}^2)$$

$$+\sum _{\alpha =1}^r{\lambda }_{{w_1^{\left(\alpha \right)}}^{*}}\left(\right||(A_{M,H_{\alpha }-1},{{\mathbf{a}}^{\left(\alpha \right)}}^{*})B_{H_{\alpha },N}^{\left(1\right)}{\left|\right|}^2)+\sum _{\alpha =r+1}^{r^{\prime }}{\lambda }_{{w_1^{\left(\alpha \right)}}^{*}}\left(\right||A_{M,H_{\alpha }-1}{B}_{H_{\alpha }-1,N}^{\left(1\right)}{\left|\right|}^2),$$

where ${w_1^{\left(0\right)}}^{*}={\{a_{k,i}^{*},0\}}_{1\le i\le H_{\alpha }},$

${w_1^{\left(\alpha \right)}}^{*}={\{a_{k,H_0+\cdots +H_{\alpha -1}+i}^{*},0\}}_{2\le i\le H_{\alpha }}$ and ${{\mathbf{a}}^{\left(\alpha \right)}}^{*}=\left(\begin{array}{c}{a}_{1,H+\alpha }^{*}\hfill \\ ⋮\hfill \\ a_{M,H+\alpha }^{*}\hfill \end{array}\right)$ for $\alpha \ge 1$.

Theorem 6.

We use the same notation as in Theorem 5. Set $\lambda ={\lambda }_0(||A_{M,H}{B}_{H,N}^{\left(Q\right)}{\left|\right|}^2)$.

We have the following:

1. 

$H=1$. $\lambda =min\{{\displaystyle \frac{M}{2}},{\displaystyle \frac{N}{2}}\}$.

2. 

$H=2$. $\lambda =min\{{\displaystyle \frac{\beta N+(2-\beta )M}{2}}$, $\beta =0,1,2$, ${\displaystyle \frac{2N+Q(N-1+M)}{2Q+2}}\}$.

3. 

$H=3$.

$\lambda =min\{{\displaystyle \frac{\beta N+(3-\beta )M}{2}}$, $\beta =0,1,2,3$,

${\displaystyle \frac{\beta N+(3-\beta )M+Q\left(\alpha \right(N+\alpha -\beta )+(3-\alpha \left)M\right)}{2(Q+1)}},$$\alpha =1,\cdots ,\beta -1$, $\beta =2,3$,

${\displaystyle \frac{3N+Q(3N-3+3M)}{2(2Q+1)}}\}$

4. 

$H=4$.

$\lambda =min\{{\displaystyle \frac{\beta N+(4-\beta )M}{2}}$, $\beta =0,1,2,3,4$,

${\displaystyle \frac{\beta N+(4-\beta )M+Q\left(\alpha \right(N+\alpha -\beta )+(4-\alpha \left)M\right)}{2(Q+1)}},$$\alpha =1,\cdots ,\beta -1$, $\beta =2,3,4$,

${\displaystyle \frac{4N+Q(\alpha N-\alpha -1+(8-\alpha \left)M\right)}{2(2Q+1)}},$$\alpha =2,3,4$, ${\displaystyle \frac{4N+Q(5N-5+3M)}{2(2Q+1)}},$

${\displaystyle \frac{3N+M+Q(\alpha N-\alpha +(8-\alpha \left)M\right)}{2(2Q+1)}},$$\alpha =2,3$,

${\displaystyle \frac{4N+Q(6N-6+6M)}{2(3Q+1)}}\}.$

Its proof appears in Appendix A.

In paper [22], we had exact values for $N=1$:

$${\lambda }_0\left(\right||A_{M,H}{B}_{H,1}^{\left(Q\right)}{\left|\right|}^2)=\frac{MQk(k+1)+2H}{4(1+kQ)}$$

where: $k=max\{i\in \mathbb{Z}:2H\ge M(i(i-1)Q+2i\left)\right\},$ and we had:

$$\theta =\left\{\begin{array}{cc}1,\hfill & \mathrm{if}2H>M\left(k\right(k-1)Q+2k)\hfill \\ 2,\hfill & \mathrm{if}2H=M\left(k\right(k-1)Q+2k)\hfill \end{array}\right.$$

5. Conclusions

In this paper, we proposed a new method of “rational blowing-up” (Theorem 4), and we applied the method to Vandermonde matrix-type singularities and demonstrated its effectiveness. Theorem 6 determines the explicit values of log canonical thresholds for $H=1,2,3,4$. Our future research aim is to improve our methods and obtain explicit values for the general model.

These theoretical values introduce a mathematical measure of preciseness for numerical calculations in information criteria in Section 1. Furthermore, our theoretical results will be helpful in numerical experiments such as the Markov chain Monte Carlo (MCMC). In the papers [25,26], the mathematical foundation for analyzing and developing the precision of the MCMC method was constructed by using the theoretical values of marginal likelihoods.

We will also consider these applications in the future.