Clustering Validation Inference

Abstract

Clustering validation is applied to evaluate the quality of classifications. This step is crucial for unsupervised machine learning. A plethora of methods exist for this purpose; however, a common drawback is that statistical inference is not possible. In this study, we construct a density function for the cluster number. For this purpose, we use smooth techniques. Then, we apply non-negative matrix factorization using the Kullback–Leibler divergence. Employing a unique linearly independent uncorrelated observational variable hypothesis, we construct a sequence by varying the dimension of the span space of the factorization only using analytical techniques. The expectation of the limit of this sequence follows a gamma probability density function. Then, identifying the dimension of the factorization of the space span with clusters, we transform the estimation of the suitable dimension of the factorization into a probabilistic estimate of the number of clusters. This approach is an internal validation method that is suitable for numerical and categorical multivariate data and independent of the clustering technique. Our main achievement is a predictive clustering validation model with graphical abilities. It provides results in terms of credibility, thus making it possible to compare results such as expert judgment on a quantitative basis.

1. Introduction

Clustering is a technique attributed to MacQueen et al. [1], who posed the problem of partitioning a set of observations into disjoint subsets such that the within-group variance is as small as possible, and they proposed the benchmark k-means method [1]. Currently, there is a well-established corpus in the statistics and machine learning (ML) fields that involves the use of extended and well-known methods with the purpose of a formal study of algorithms and methods for grouping or clustering objects according to measured or perceived intrinsic characteristics or similarities [2]. The vertiginous growth experienced in such fields in recent decades, particularly in terms of concepts and computational techniques, is reflected in several excellent books published in this field. One of them, providing detailed explanations of the most relevant contributions, is [3], which clearly reflects the difficulty of conducting simple classifications of existing techniques, as there exists an overlapping of methods aggravated with the introduction of eigenanalysis criteria.

An alternative to partitioning methods can be designed through admitting that, in the presence of uncertainty in observations, they may belong to more than one category (to a greater or lesser extent) and by using supporting generative models. These approaches are based on determining suitable mixtures of components in terms of number and parameters, and they are denoted by $f(\mathbf{x};\theta )=\sum {\alpha }_j{g}_j(x;{\theta }_j)$, where g is a density with parameter $\theta \in \Theta$, and $\alpha$ denotes the mixing weights assigned to each density. Two interpretations of this formulation exist: the first supposes that each entity belongs to each cluster with a different probability [4], while the second concept classifies the observations that most likely belong to each distribution [5].

If the data frame of observations can be handled as a matrix $\mathbf{X}$, its transformation to the probability space yields a stochastic matrix or probabilistic image of the data matrix, thus making non-negative matrix factorization (NMF) techniques especially apt for handling probabilistic structures. Many authors have attributed the introduction of these techniques to [6], while others attribute their introduction to [7]—a study clearly centered on ML. We prefer to attribute their introduction to Chen [8], who was influential in establishing conditions for the existence of stochastic matrices [9]. The main idea is to factorize the stochastic matrix as the product of the other two matrices and of non-negative inputs. This factorization depends on the dimension of the space span (the dimension of the factorization space), which is not determined a priori. Then, the problem of parameter estimation is translated to determining the number of suitable mixtures, or the space span of the factorization, converting the problem to the unsupervised classification of the parameter estimation that assigns probabilities of membership to each item in each cluster. Pioneering applications of clustering using NMF techniques have been demonstrated in studies on the classification of textures [10], bioinformatics [11], instrument classification [12], facial recognition [13], and spectroscopy [14].

Independent of the classification method, an important step is determining the quality of the classification. In this context, the number of clusters is one of the most relevant aspects. This problem, called clustering validation, is of active and increasing interest in this research field, and it is critical for unsupervised cases. Additionally, many validation methods exist. A drawback shared by many validation methods is that the use of one or more criteria may inadvertently satisfy different algorithms. Thus, clustering is a problem in which precise quantification is often not possible because of its unsupervised nature [3] (pg. 22). The practical consequence is that many existing validation procedures seem to be less stringent with respect to the clustering methods that they validate. From a theoretical viewpoint, the relationships between many of these criteria have not yet been well established. In contrast, qualitative methods exist—usually based on graphical criteria—that provide good estimates but suffer from drawbacks in terms of providing quantitative results [15]. Moreover, for large datasets, the presence of noise can be significant, and making estimations and results that are formally correct can be unrealistic.

A line of work that does not share these problems is based on the idea of stability. Stability is an internal measure that evaluates the results for a choice of the number of clusters. The main idea is that a dataset is a sample of a statistical distribution. The sample is stable if the underlying distribution is the same for different samples obtained from the data [16]. A review work that provides insight into these methods is [17]. A more recent work to determine the partition from the underlying distribution is [18]. This work redefines the concept of clustering as the partitioning of data into groups so that the partition is stable, proposing the stadion index. In essence, this index maximizes a function of the stability between clusters and minimizes it within cluster.

In our approach, we derive a probability density function (pdf) to assess the degree of validity for classifications. The first key step is the formulation of the problem in light of the NMF. The second key step is to obtain a sequence of traces by varying the space span of the NMF. Then, the expectation of the limit behavior is given as a pdf.

While the term inference is not vague, it has been given different definitions. It is extensively used in logic, referring to the correct result of reasoning with several propositions. In statistics, it refers to the branch... which deals with the generalizations from samples to the population parameters, being the relative frequency interpretation in the Bayesian context, or the interpretation of probability in a subjective sense, which may estimate the probability of an event on the basis of our experience or credibility [19]. In the ML context, according to the dictionary of Google (https://developers.google.com/machine-learning/glossary?hl=es-419#i, URL accessed 15 September 2023 ), it is the process of making predictions by applying a trained model to unlabeled examples. This definition implies a simple descriptive statistical framework, restricting the clustering validation problem to the best data structure description in order to handle the data while sacrificing predictions. In this manuscript, the definition of inference is related to inferential statistics.

The remainder of this manuscript is structured as follows: Studies covering similar topics are presented in Section 2. Basic formulas that are well known and extended for use with NMF practitioners are presented in Section 3. In this section, we also propose an error bound for the case of low-rank approximations. The manipulations necessary to obtain the traces and the underlying hypothesis (linear independence of observational variables) are explained in Section 3.4, as well as the main result; that is, the sequence of NMF traces follows a gamma pdf when the dimension of the space span varies, and it is the posterior of a Poisson distribution. We provide some indications for the application of this result for clustering validation in Section 4. In Section 5, we illustrate how the pdf operates in the context of several data configurations. We discuss several gaps and open questions in Section 6 before presenting our conclusions.

A key contribution of this manuscript is the generalization of NMF to real input matrices achieved through transformations to probabilistic space, which can always be justified in the presence of data uncertainty. However, the most important contribution of this work is the provision of a probabilistic validation criterion that allows for credibility to be assigned to clustering results independently of the clustering method. This result enables the construction of credible or acceptance regions, thereby allowing for a comparison of hypotheses or acceptance intervals to be carried out.

The existence of a pdf extends the validation problem to the evaluation on a quantitative basis of criteria other than purely numerical ones, including expert criteria. In the case of serious discrepancies, it allows justifying the re-analysis of the problem with more relevant data. Furthermore, in the case of big data, it may be more appropriate to provide a confidence interval than a single value for classification.

2. Related Work

The problem of clustering validation is as old as the techniques of clustering. Although there are no unified criteria to determine whether it is a geometric problem related to partitioning a set based on similarities or a problem of label assignment, this conceptual nuance does not affect the practical results. Validation requires that several properties (e.g., sensibility, cluster number impact, invariance) be determined in order to establish the capability of a cluster method for various data structures. There exist many studies that have justified these criteria, such as [3] (chap. 23) and [20].

One of the first statistical attempts to use multivariate techniques to determine the number of clusters was detailed in [21]. At this time, the use of statistical concepts to construct validation indices was quasi-mandatory [22]. A later work by the same author pointed out the effects of sample size, data dimension, and cluster spread [23]. These ideas are currently relevant, considering the increasing size of datasets. In [24], the difference between graphical methods and those that postulate the existence of an underlying parametric statistical model was pointed out. This idea is relevant when using the likelihood ratio test, which is the starting point for many fuzzy and probabilistic validation criteria. Furthermore, the current classification of validation methods was introduced in this work. One such criterion is the classical silhouette index, which evaluates the within and between variances of the groups into which the data have been divided for the selection of the cluster number [25]. Further studies focusing on this viewpoint of statistical techniques for clustering validation include Har [26], which introduced an indicator function for observations based on kernelization and used the null hypothesis as a classifier.

In Smyth [27], likelihood cross-validation was used to infer information on the number of model components. At this time, and as a consequence of the expansion of the Internet, other problems appeared regarding what constitutes a good measure of similarity. In Halkidi, a detailed survey analyzing the problem of data similarity was provided. This study raised the problem of how to measure the goodness of fit of the data relative to the groups that use concepts of compactness and separation. This work serves to create a consensus according to the types of indices: external validation (the availability of additional information, such as a subset of labeled data) and internal validation (the case in which only information on the data is available) [28]. This work also provided many examples. A classical index derived from these ideas is the gap statistic, which compares the within-cluster dispersion to the expectation of a reference distribution [29].

Using a different approach, the similarity between clusters can be evaluated using the ${\chi }^2$ statistic between probabilistic classifications [30]. In Brun, a model for error prediction based on the correlation error rate vs. several validity measures was provided. The Kendall rank correlation index can be used to quantify the degree of similarity between the validation indices and the classification errors [31]. By introducing an index and assuming that each cluster is generated by a parametric distribution, the minimum can be taken as the validation index [32]. The effect of clusters with different sizes and densities was studied in [33].

More recent works include [34], which, within the scope of astronomical observations and under the hypothesis of normality and the existence of a correlation, presented an algorithm in which the posterior distribution of the correlations follows a gamma pdf. A quantitative discriminant method using the elbow point for the determination of the optimal cluster number was presented in [35]. In [36], a purely algebraic approach was presented, in which elements are clustered according to their co-linearity. A review centered on the impact and importance of clustering validation in the context of the recent growth of bioinformatics was presented in [37], with the evaluation of the number of clusters being a more recent development [38].

3. Probabilistic Space Domain Transformation, Parametrization, and Non-Negative Factorization Sequence Traces

The transformation of data to the probabilistic space allows one to search for differences or similarities in terms of probabilities. This transformation is justified when uncertainty exists in the data estimation. Several techniques that involve transformation based on a kernel are extensively used in the ML context [3] (p. 10). Additionally, well-established methods for smoothing represent a commonly employed solution to obtain good results.

One of the relevant issues in the statistics literature is the determination of the underlying pdf of a sample. In the multivariate case, the data frame containing this sample is the matrix $\mathbf{X}\in {\mathbb{R}}^{m\times n}$, in which m observations are formulated as i ($i=1,\dots ,m$) rows obtained under the j ($j=1,\dots ,n$) observational criteria. Thus, an observation under the $j^{\mathrm{th}}$ criterion is the column vector ${\mathbf{x}}_j$, while the row vectors provide a description of an item or observation. We represent these observations as $i^{\prime }$ as $x_{i^{\prime }1},\dots ,x_{i^{\prime }n}$. Additionally, we assume the linear independence of the columns of $\mathbf{X}$.

3.1. Probabilistic Image of a Real-Valued Data Matrix

3.1.1. Real Field Domain

The transformation to the probabilistic space for the case in which the data frame of observations can be written as the matrix $\mathbf{X}\in {\mathbb{R}}^{m\times n}$ requires the determination of the underlying pdfs $f_1,\dots ,f_n$, such that

$$\begin{array}{cc}\hfill {\left[\mathbf{X}\right]}_{ij}& =\left[\begin{array}{ccc}{f}_1^{-1}\left(x_1\right)& \dots & f_n^{-1}\left(x_1\right)\\ ⋮& \ddots & ⋮\\ f_1^{-1}\left(x_m\right)& \dots & f_n^{-1}\left(x_m\right)\end{array}\right]\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill & =\left[f_1^{-1}\left(\mathbf{x}\right)|\dots |f_n^{-1}\left(\mathbf{x}\right)\right].\hfill \end{array}$$

(2)

Each column vector is associated with a density corresponding to the observational criteria used to generate the data, which are unknown in the general case.

The estimate of each density in (2) using a smoothing technique requires the j density to be approximated by p mixtures of a kernel density function (kdf), providing an estimate ${\widehat{f}}_j$ that approximates $f_j$ as

$$\begin{array}{c}\hfill {\widehat{f}}_j(x;h)={\displaystyle \frac{1}{p}}\sum _p{K}_h\left(x-x_p;h\right),\end{array}$$

(3)

where $K_h$ denotes the kdf. This adjustment involves the choice of the shape of the kdf and the selection of the smoothing parameter h. This point is controversial, as a small value provides a good estimation for each point but increases the variance, while large values reduce the variance but increase the bias and obscure the underlying data structure. An extended criterion for its selection involves the estimation of the mean integrated standard error (MISE), defined as

$$\begin{array}{c}\hfill \mathrm{MISE}\left(\widehat{f}\right)=\mathbb{E}{\left\{{\int }_{\mathcal{D}}(\widehat{f}-f)dx\right\}}^2,\end{array}$$

(4)

where $\widehat{f}$ is the MISE estimate of f.

After juxtaposing the densities ${\widehat{f}}_j(\mathbf{x};h)$, the result is the following stochastic column matrix:

$$\begin{array}{c}\hfill {\left[\tilde{\mathbf{Y}}\right]}_{ij}=\left[{\widehat{f}}_1(\mathbf{x};h_1)|\dots |{\widehat{f}}_n(\mathbf{x};h_n)\right]\left(\mathrm{s}.\mathrm{t}.\parallel {\widehat{f}}_j(\mathbf{x};h_j){\parallel }_1=1\mathrm{for}\mathrm{all}j=1,\dots ,n\right),\end{array}$$

(5)

where ${\parallel ·\parallel }_1$ is the $L_1$ norm in the Shalten sense.

This case is univariate, while the multivariate case involves considering the $x_{ij}$ entries of (1) as $x_{ij}=f_j^{-1}\left(x_i\right)$. Smoothing with the kdf is carried out:

$$\begin{array}{c}\hfill \widehat{f}(\mathbf{X};{\mathbf{H}}_{\mathcal{K}})={\displaystyle \frac{1}{p}}\sum _p{K}_{{\mathbf{H}}_{\mathcal{K}}}({\mathbf{x}}_p-\mathbf{x}),\end{array}$$

(6)

where ${\mathbf{H}}_{\mathcal{K}}$ is the matrix of smooth parameters, usually written as $\mathbf{H}$, in which we add the sub-index $\mathcal{K}$ referring to the kernel in order to avoid confusion with the matrix $\mathbf{H}$ used in the NMF context. Additionally, ${\mathbf{x}}_p$ is a row vector in (1), and $\mathbf{x}$ represents the other row vectors weighted by the kdf. ${\mathbf{H}}_{\mathcal{K}}$ is a positive semi-definite matrix containing the smooth parameters:

$$\begin{array}{c}\hfill {\mathbf{H}}_{\mathcal{K}}=\left[{\mathbf{h}}_1,\dots ,{\mathbf{h}}_n\right].\end{array}$$

(7)

Moreover, the probabilistic image is

$$\begin{array}{c}\hfill {\left[\mathbf{Y}\right]}_{ij}=f^{-1}(\mathbf{X};{\mathbf{H}}_{\mathcal{K}}).\end{array}$$

(8)

The relationship between (5) and (8) is determined by

$$\begin{array}{c}\hfill {\left[\tilde{\mathbf{Y}}\right]}_{ij}{\mathbf{D}}_C={\left[\mathbf{Y}\right]}_{ij},\end{array}$$

(9)

where ${\mathbf{D}}_C$ satisfies $\parallel {\mathbf{D}}_C{\parallel }_1=1$. Thus, pseudo-inverses are taken:

$$\begin{array}{c}\hfill \mathbf{I}{\mathbf{D}}_C={\left({\left[\tilde{\mathbf{Y}}\right]}_{ij}^{\prime }{\left[\tilde{\mathbf{Y}}\right]}_{ij}\right)}^{--1}\left({\left[\tilde{\mathbf{Y}}\right]}_{ij}^{\prime }{\left[\mathbf{Y}\right]}_{ij}\right),\end{array}$$

(10)

where $\mathbf{I}$ is an identity matrix of suitable dimension, and

$$\begin{array}{c}\hfill {\mathbf{D}}_C=\mathrm{diag}\left[{\displaystyle \frac{1}{m}}\right](\mathrm{s}.\mathrm{t}.{\mathbf{D}}_C\in {\mathbb{R}}^{m\times m}).\end{array}$$

(11)

More details on smoothing techniques have been presented in [39], which contains many examples and the R code. The multivariate case has been explained in depth in [40]. Additionally, the probabilistic sense of the matrices in (5) and (8) can be formally justified by considering the set $\mathcal{B}=\left\{x_i\right\}$ (or $\left\{x_{ij}\right\}$), which takes values in all possible outcomes of the set $\Omega$. Thus, the problem can be stated in terms of the triplet $(\Omega ,\mathcal{B},P)$, with P being a measure or probability and $\mathcal{B}$ being a Borel $\sigma$-algebra. The map of the inverse image of $\mathcal{B}$ is $F^{-1}\left(\mathcal{B}\right)=\{\omega \mathrm{s}.\mathrm{t}.F\left(\omega \right)\in \mathcal{B}\}(\omega \in \Omega )$. The Borel sets define probabilities $P(\mathbf{x}\in \mathcal{B})=P\left(F^{-1}\left(\mathcal{B}\right)\right)$ with distributions $P(\mathbf{x}\le x)$ and $P\left({\mathbf{x}}_j\right)=1$; this allows for the estimation of the density $f_j$ (or f) associated with the distribution P, which generates the data [41] (Chap. 1).

3.1.2. The Particular Case of the Positive Integer Domain

A classical probabilistic transformation widely used in many contexts consists of estimating the relative frequencies of a data frame $N(d_i,w_j)$, obtained from a corpus of $d_i(i=1,\dots ,m)$ documents when crossed with a thesaurus of $w_j(j=1,\dots ,n)$ words [42,43]. This approach has been extended to many other cases in which the data domain takes values in the positive integers [44], many of which have been explained in our recent work [45]. Then, the relative frequencies are the following probabilities:

$$\begin{array}{c}\hfill P(d_i,w_j)={\displaystyle \frac{N(d_i,w_j)}{{\sum }_j{\sum }_{i}N(d_i,w_j)}},\end{array}$$

(12)

which are estimated via joint probability $P(d_i,w_j)$.

The Bayes rule provides the following:

$$\begin{array}{c}\hfill P(d_i,w_j)=P\left(d_i\right|w_j)P\left(w_j\right)(\mathrm{with}\sum _{i}P\left(d_i\right|w_j)=1\mathrm{for}\mathrm{all}j).\end{array}$$

(13)

The introduction of matrix notation requires the identification of $\mathbf{N}\sim P(d_i,w_j)$ and $\tilde{\mathbf{N}}\sim P\left(d_i\right|w_j)$. Each column of $\tilde{\mathbf{N}}$ has an $L_1$ norm of 1. In this case, we have the following:

$$\begin{array}{c}\hfill {\left[\mathbf{N}\right]}_{ij}={\left[\tilde{\mathbf{N}}\right]}_{ij}{\mathbf{D}}_C,\end{array}$$

(14)

with ${\mathbf{D}}_C$ as defined in (11).

Furthermore, if ${\mathbf{D}}_N=\mathrm{diag}{[1/{\sum }_{j}N(d_i,w_j)]}_{j=1}^n$, the transformation (12) can be written as

$$\begin{array}{c}\hfill {\displaystyle \frac{N(d_i,w_j)}{{\sum }_j{\sum }_{i}N(d_i,w_j)}}=N(d_i,w_j){\mathbf{D}}_C{\mathbf{D}}_N.\end{array}$$

(15)

Transformation (12) is based on Laplace’s probability definition, and its use is limited to the case of count matrices or contingency tables. This restriction greatly limits the use of NMF techniques.

3.1.3. Equivalence

The probabilistic transformations given by (9) and (14) for the case of positive integer and real-entry matrices are equivalent, and equality between both results can be obtained if the entries of $\tilde{\mathbf{N}}$ and $\tilde{\mathbf{Y}}$ are the same. This result can be obtained by introducing a triangular kdf in (3) and a suitable smoothing parameter h.

The triangular kdf is based on the triangular function, as shown in Figure 1. It was investigated in [46,47], in which it was stated that it corresponds to a discrete pdf, while it was found to be continuous in [41] (Chap. 13). This question depends on the conditions of the definition of the variable domain and its support.

The triangular kernel is introduced as follows [46,47]:

$$\begin{array}{c}\hfill K(x;h)=\left\{\begin{array}{cc}{\displaystyle \frac{h-|x-x_i|}{h^2}}\hfill & (\mathrm{if}{x}_i-h\le x\le x_i+h)(h>0)\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right..\end{array}$$

(16)

Writing the difference $|x-x_i|=x_i+\left|h\right|x_i$ for $h=1$ the grid (i.e., the values at which the density is estimated) $x_i=(x_1,\dots x_m)$, the interval $[x-h,x+h]$ contains a unique point. Then, we have the following:

$$\begin{array}{c}\hfill K\left(x\right)=x_i,\end{array}$$

(17)

with the density estimate

$$\begin{array}{c}\hfill \widehat{f}(x_i;h)={\displaystyle \frac{x_i}{n_{ri}}},\end{array}$$

(18)

where $n_{ri}$ is the number of observations at $x_i$. To obtain the probabilistic image, this procedure must be carried out for all the columns of (1).

The multivariate case requires the consideration of a multivariate kernel in the form of $K(\mathbf{X};{\mathbf{H}}_{\mathcal{K}})$. According to our literature search, the d-dimensional triangular kernel has not been defined; although we think that it will not be difficult to obtain, it remains an unresolved issue. We point out that once (18) is obtained, the analogous multivariate case introducing the diagonal matrix ${\mathbf{D}}_C$ (as defined in (11)) can be immediately obtained. This matrix corresponds to a uniform distribution, with equal weights assigned to all observational variables. This result demonstrates that the Laplace rule is a particular case of the wider transformation to the probabilistic space.

We call the stochastic matrices $\mathbf{Y}$ and $\tilde{\mathbf{Y}}$ the probabilistic image and column probabilistic image, respectively. Moreover, they are the transformations to the probabilistic space or simply probabilistic transformations of the data.

3.2. Parametrization

To achieve the factorization of (8) while maintaining a probabilistic sense, one must impose the NMF on $\mathbf{Y}$ to obtain the matrices $\mathbf{W}$ and $\mathbf{H}$ as the following product:

$$\begin{array}{cc}\hfill {\left[\widehat{\mathbf{Y}}\right]}_{ij}& ={\left[\mathbf{W}\right]}_{ik}{\left[\mathbf{H}\right]}_{kj}\left(\mathrm{for}\parallel [\mathbf{W}\mathbf{H}{\parallel }_1=1\mathrm{with}\mathbf{W},\mathbf{H}\ge 0\mathrm{for}\mathrm{all}k=1,2,\dots \right)\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill & \approx {\left[\mathbf{Y}\right]}_{ij},\hfill \end{array}$$

(20)

which minimizes an objective or cost function. (The standard NMF formulation is usually stated as the product $\mathbf{Y}=\mathbf{WH}+\mathbf{E}$, being the objective to minimize $\mathbf{E}$ [48] (p. 8). In this work we have preferred to ignore the matrix $\mathbf{E}$ and formulate it strictly as an approximation problem.) This factorization always exists for non-negative real matrices [8], and it has a probabilistic sense if the normalization conditions of (19) are fulfilled.

The Kullback–Leibler (KL) divergence is chosen as the following objective function [49]:

$$\begin{array}{cc}\hfill D_{KL}\left({\left[\mathbf{Y}\right]}_{ij}\parallel {\left[\mathbf{WH}\right]}_{ij}\right)& =\sum _j\sum _i{\left[\mathbf{Y}\right]}_{ij}\odot log{\displaystyle \frac{{\left[\mathbf{Y}\right]}_{ij}}{{\displaystyle {\left[\mathbf{W}\mathbf{H}\right]}_{ij}}}}\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill & =\sum _j\sum _i\left({\left[\mathbf{Y}\right]}_{ij}\odot log{\left[\mathbf{Y}\right]}_{ij}-{\left[\mathbf{Y}\right]}_{ij}\odot log{\left[\mathbf{W}\mathbf{H}\right]}_{ij}\right),\hfill \end{array}$$

(22)

where ⊙ is the Hadamard (or element-wise) product, and the matrix fraction is the quotient for equal sub-index entries.

Then, (21) is minimized according to the Karush–Kuhn–Tucker conditions [48] (p. 141):

$$\begin{array}{cc}\hfill {\left[\mathbf{W}\right]}_{ik}\odot {\nabla }_{\mathbf{W}}{D}_{KL}& =0\left(\mathrm{for}{\nabla }_{\mathbf{W}}{D}_{KL}\left({\left[\mathbf{Y}\right]}_{ij}\parallel {\left[\mathbf{WH}\right]}_{ij}\right)\ge 0\mathrm{with}\mathbf{W}\ge 0\right)\hfill \end{array}$$

(23)

and

$$\begin{array}{cc}\hfill {\left[\mathbf{H}\right]}_{kj}\odot {\nabla }_{\mathbf{H}}{D}_{KL}& =0\left(\mathrm{for}{\nabla }_{\mathbf{H}}{D}_{KL}\left({\left[\mathbf{Y}\right]}_{ij}\parallel {\left[\mathbf{WH}\right]}_{ij}\right)\ge 0\mathrm{with}\mathbf{H}\ge 0\right),\hfill \end{array}$$

(24)

which provides the following solutions [50]:

$$\begin{array}{cc}\hfill {\left[\mathbf{W}\right]}_{ik}& \leftarrow {\left[\mathbf{W}\right]}_{ik}\odot \left({\displaystyle \frac{{\left[\mathbf{Y}\right]}_{ij}}{{\displaystyle {\left[\mathbf{W}\mathbf{H}\right]}_{ij}}}}{\left[\mathbf{H}\right]}_{kj}^{\prime }\right)\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill {\mathbf{H}]}_{kj}& \leftarrow {\left[\mathbf{H}\right]}_{kj}\odot \left({\left[\mathbf{W}\right]}_{ik}^{\prime }{\displaystyle \frac{{\left[\mathbf{Y}\right]}_{ij}}{{\displaystyle {\left[\mathbf{W}\mathbf{H}\right]}_{ij}}}}\right).\hfill \end{array}$$

(26)

This procedure is a particular case of NMF in which divergences are used, which are known as iterative updates or multiplicative iterative algorithms [48] (Chap. 3). In [51], it was demonstrated that this procedure maintains the normalization conditions in the iterative process.

To adjust the product (19) or approximation (20), it is necessary to select a value for k, initialize the matrices $\mathbf{W}$ and $\mathbf{H}$, and then apply an iterative process (switching between (25) and (26)) until a condition is reached where the approximation degree of (19) is achieved.

This procedure of approximating $\mathbf{WH}$ to $\mathbf{Y}$ in order to maintain matrix normalization involves an approximation of $\widehat{\mathbf{Y}}$ to $\mathbf{Y}$, which are densities. Thus, it is a sufficient (not minimal) statistic—roughly speaking, the density contains the full sample’s information. Therefore, the products $\{\mathbf{W},\mathbf{H}\}$ are distributional parameters with no distributional assumption, constituting a non-parametric approach (in the sense that no hypothesis is made for the parameters). We also refer to the columns of $\mathbf{W}$ (or $\tilde{\mathbf{W}}$) and the rows of $\mathbf{H}$ (or $\tilde{\mathbf{H}}$) as components.

3.3. Convergence of Solutions

The approximation of (19) requires the following:

$$\begin{array}{c}\hfill \parallel {\left[\widehat{\mathbf{Y}}\right]}_{ij}-{\left[\mathbf{Y}\right]}_{ij}{\parallel }_1\le ϵ(ϵ>0).\end{array}$$

(27)

Thus, convergence in the iterative process of approximating $\widehat{\mathbf{Y}}\to \mathbf{Y}$ must be proven.

Usually, these proofs are based on the same strategy as the Expectation-Maximization (EM) algorithm, which always converges [52]. Basically, they consist of postulating the existence of a function G accomplishing $G(h,h^{\prime })\ge F\left(h\right)$ and $G(h,h)=F\left(h\right)$, which leads to the sequence $h^{p+1}=arg\min G(h,h^{\prime })$. This procedure is similar to that detailed in [7].

Based on this proof, it is usually stated that factorization (19) with the KL divergence converges to the matrix $\mathbf{Y}$ through a simple observation of the expansion of the KL divergence of (22): While the first term is a constant, the second is the log-likelihood. A more general conclusion has been derived by [53], showing the asymptotic properties of this solution.

An alternative naive proof can be obtained by taking into account the fact that the divergence (22) takes positive values (divergences are non-negative). Thus, ${\parallel \mathbf{Y}\parallel }_1>{\parallel \mathbf{WH}\parallel }_1$. Then, for matrices satisfying this condition, Formulas (25) and (26) can be rewritten for the $p^{\mathrm{th}}$ iteration as follows:

$$\begin{array}{c}\hfill {\mathbf{W}}^{(p+1)}={\mathbf{W}}^{\left(p\right)}\odot {\mathbf{W}}^{\left(a\right)},\end{array}$$

(28)

$$\begin{array}{c}\hfill {\mathbf{H}}^{(p+1)}={\mathbf{H}}^{\left(p\right)}\odot {\mathbf{H}}^{\left(a\right)},\end{array}$$

(29)

interpreting ${\mathbf{W}}^{\left(a\right)}$ and ${\mathbf{H}}^{\left(a\right)}$ as actualization weight matrices.

As all the entries of the matrices ${\mathbf{W}}^{\left(a\right)}$ and ${\mathbf{H}}^{\left(a\right)}$ take values in $[0,1]$, it holds that ${\mathbf{W}}^{(p+1)}<{\mathbf{W}}^{\left(p\right)}$ (also for the matrices in (29)). Then, the product of the matrices $\mathbf{W}$ and $\mathbf{H}$ is monotonically decreasing, and the difference (27) also decreases.

A consequence of convergence is an approximation error bound. This requires the consideration of two cases. First, for $k\ge \min (m,n)$, the convergence $\widehat{\mathbf{Y}}\to \mathbf{Y}$ is well known (a formal proof can be found in [50]).

The case $k<\min (m,n)$ is known as low-rank approximation, which has particular interest in practice. It poses several problems, as reported in the literature. In the context of the probabilistic latent semantic analysis (PLSA), it has been pointed out that the convergence limit does not necessarily occur at a global optimum, thus providing sub-optimal results [54].

Our proposal to establish a convergence limit is similar to that used by Schmidt for the approximation theorem [55].

Theorem 1 below was formulated according to the singular value decomposition (SVD) theorem for real-valued matrices [56] (p. 275).

Theorem 1.

Let $\mathbf{X}\in {\mathbb{R}}^{m\times n}$. Then, orthogonal (or unitary) matrices $\mathbf{U}\in {\Re }^{m\times m}$ and $\mathbf{V}\in {\Re }^{n\times n}$ exist such that

$$\begin{array}{c}\hfill \mathbf{X}=\mathbf{U}\Sigma {\mathbf{V}}^{\prime }\Sigma =\left[\begin{array}{cc}{\Sigma }_1& \mathbf{0}\\ \mathbf{0}& \mathbf{0}\end{array}\right],\end{array}$$

where ${\Sigma }_1=\mathrm{diag}({\sigma }_1,\dots ,{\sigma }_r)$ with diagonal entries

$$\begin{array}{c}\hfill {\sigma }_1\ge ,\dots ,\ge {\sigma }_r>0r=\mathrm{rank}\left(\mathbf{A}\right).\end{array}$$

The approximation for $r^{\prime }<r$ is known as the low-rank SVD approximation. Additionally, it is necessary to consider that the spectral norm of a matrix is the sum of its eigenvalues (or trace). Thus, by writing the inequality

$$\begin{array}{c}\hfill {\parallel \mathbf{Y}-\mathbf{W}\mathbf{H}\parallel }_1\le \parallel \mathbf{Y}{\parallel }_1-\parallel \mathbf{W}\mathbf{H}{\parallel }_1\end{array}$$

(30)

and taking into account approximation (27) for the eigenvalues of the second term of (30),

$$\begin{array}{cc}\hfill ϵ& \le \left({\sigma }_1\left({\left[\mathbf{Y}\right]}_{ij}\right)+\dots +{\sigma }_r\left({\left[\mathbf{Y}\right]}_{ij}\right)\right)-\left({\sigma }_1\left({\left[\widehat{\mathbf{Y}}\right]}_{ij}\right)+\dots +{\sigma }_{r^{\prime }}\left({\left[\widehat{\mathbf{Y}}\right]}_{ij}\right)\right)\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill & =\left({\sigma }_1\left({\left[\mathbf{Y}\right]}_{ij}\right)-{\sigma }_1\left({\left[\widehat{\mathbf{Y}}\right]}_{ij}\right)\right)+\dots +\left({\sigma }_{r^{\prime }}\left({\left[\mathbf{Y}\right]}_{ij}\right)-{\sigma }_{r^{\prime }}\left({\left[\widehat{\mathbf{Y}}\right]}_{ij}\right)\right)+{\sigma }_{r^{\prime }+1}\left({\left[\mathbf{Y}\right]}_{ij}\right)+\dots +{\sigma }_r\left({\left[\mathbf{Y}\right]}_{ij}\right),\hfill \end{array}$$

(32)

as the common eigenvalues are the same [56] (p. 277):

$$\begin{array}{c}\hfill ϵ\le \sum _{r=r^{\prime }+1}^r{\sigma }_r\left({\left[\mathbf{Y}\right]}_{ij}\right),\end{array}$$

(33)

Formula (33) provides the error bound for the approximation case $k<\min (m,n)$.

Furthermore, in [43], it was demonstrated that

$$\begin{array}{c}\hfill {\left[\mathbf{WH}\right]}_{ij}={\displaystyle \frac{1}{k}}{\left[\tilde{\mathbf{W}}\right]}_{ik}{\left[\tilde{\mathbf{H}}\right]}_{kj}.\end{array}$$

(34)

3.4. Non-Negative Factorization Sequence Traces

In Formula (19), the trace

$$\begin{array}{cc}\hfill \mathrm{tr}\left(\widehat{{\mathbf{Y}}^{\prime }}\widehat{\mathbf{Y}}\right)& =\sum _k{\left(\mathrm{diag}{\left({\left[\mathbf{W}\right]}_{ik}{\left[\mathbf{H}\right]}_{kj}\right)}^{\prime }\left({\left[\mathbf{W}\right]}_{ik}{\left[\mathbf{H}\right]}_{kj}\right)\right)}^{1/2}\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill & =\parallel \mathrm{diag}{\left({\left({\left[\mathbf{W}\right]}_{ik}{\left[\mathbf{H}\right]}_{kj}\right)}^{\prime }\left({\left[\mathbf{W}\right]}_{ik}{\left[\mathbf{H}\right]}_{kj}\right)\right)}^{1/2}{\parallel }_1\hfill \end{array}$$

(36)

leads to a sequence with varying k values in the NMF factorization space span.

$$\begin{array}{cc}\hfill {\left\{z_{\left[k\right]}\right\}}_k& ={\left\{\mathrm{tr}\left({\left({\left[\mathbf{W}\right]}_{i1}{\left[\mathbf{H}\right]}_{1j}\right)}^{\prime }{(\left[\mathbf{W}\right]}_{i1}{\left[\mathbf{H}\right]}_{1j}\right)\right)}^{1/2},\mathrm{tr}{\left({\left({\left[\mathbf{W}\right]}_{i2}{\left[\mathbf{H}\right]}_{2j}\right)}^{\prime }\left({\left[\mathbf{W}\right]}_{i2}{\left[\mathbf{H}\right]}_{2j}\right)\right)}^{1/2},\dots \}\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill & =\left\{z_1,z_2,\dots \right\}.\hfill \end{array}$$

(38)

Here, the sub-index brackets indicate that terms are given in increasing order.

We can write z as

$$\begin{array}{c}\hfill z=\mathrm{tr}{\left(\left({\left[\mathbf{Y}\right]}_{ij}^{\prime }{\left[\mathbf{Y}\right]}_{ij}\right)\right)}^{1/2}.\end{array}$$

(39)

In Formulas (38) and (39), the quotient

$$\begin{array}{c}\hfill \tilde{z}={\left\{{\displaystyle \frac{z_{\left[k\right]}}{z}}\right\}}_k\end{array}$$

(40)

leads to a monotonically decreasing sequence. To achieve a reliable sequence, it is necessary to establish the same convergence condition for all of the terms. This requires consideration of the following cases.

If $k\ge \min (m,n)$ occurs, then $\widehat{\mathbf{Y}}\to \mathbf{Y}$ during the iteration process. By imposing the same approximation condition for $ϵ$ on all products of $\mathbf{WH}$ obtained with different values of k and introducing

$$\begin{array}{c}\hfill \parallel {\left[\mathbf{W}\right]}_{ik}{\left[\mathbf{H}\right]}_{kj}{\parallel }_1={\displaystyle \frac{1}{k}}\parallel {\left[\tilde{\mathbf{W}}\right]}_{ik}{\left[\tilde{\mathbf{H}}\right]}_{kj}{\parallel }_1,\end{array}$$

(41)

the inequality

$$\begin{array}{c}\hfill {\displaystyle \frac{1}{k^2}}\mathrm{tr}{\left({\left({\left[\tilde{\mathbf{W}}\right]}_{ik}{\left[\tilde{\mathbf{H}}\right]}_{kj}\right)}^{\prime }\left({\left[\tilde{\mathbf{W}}\right]}_{ik}{\left[\tilde{\mathbf{H}}\right]}_{kj}\right)\right)}^{1/2}>{\displaystyle \frac{1}{{\left(k^{\prime }\right)}^2}}\mathrm{tr}{\left({\left({\left[\tilde{\mathbf{W}}\right]}_{i{k}^{\prime }}{\left[\tilde{\mathbf{H}}\right]}_{k^{\prime }j}\right)}^{\prime }\left({\left[\tilde{\mathbf{W}}\right]}_{i{k}^{\prime }}{\left[\tilde{\mathbf{H}}\right]}_{k^{\prime }j}\right)\right)}^{1/2}\end{array}$$

(42)

holds only if $k^{\prime }>k$.

If $k<\min (m,n)$, then a convergence error bound given by (33) exists. Thus, it is necessary to impose a wider condition for the approximation of (27) (i.e., $ϵ=O(1/m)$), which is a jump in the empirical univariate distribution of the columns of $\tilde{\mathbf{Y}}$. As the convergence of the empirical distribution to each component (column) of $\tilde{\mathbf{Y}}$ is almost sure (a.s.), the difference is bounded. After taking the maximum difference, a decreasing behavior is observed when we carry out division by increasing values of k.

3.4.1. Trace Sequence Limit Behavior

For sequence (40) obtained from a full-rank matrix, the function

$$\begin{array}{c}\hfill \phi \left(z_{\left[k\right]}\right)={\left({\displaystyle \frac{z_{\left[k\right]}}{z}}\right)}^{-z}\end{array}$$

(43)

represents the similarity between z and $z_k$ in terms of the inverse of the (non-logarithmic) likelihood. This function can be written as

$$\begin{array}{c}\hfill \phi \left(z_{\left[k\right]}\right)={\left(1+{\displaystyle \frac{z_k-z}{z}}\right)}^{-z}.\end{array}$$

(44)

By introducing the following transformations

$$\begin{array}{cc}\hfill \lambda & =z_1-z\hfill \end{array}$$

(45)

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{\nu }}& ={\displaystyle \frac{z_k-z}{z_1-z}}(k\ne 1),\hfill \end{array}$$

(46)

with the Jacobian

$$\begin{array}{cc}\hfill \left|J\right|& =\det \left[\begin{array}{cc}{\displaystyle \frac{\partial z_1}{\partial \lambda }}& {\displaystyle \frac{\partial z_1}{\partial \nu }}\\ {\displaystyle \frac{\partial z_k}{\partial \lambda }}& {\displaystyle \frac{\partial z_k}{\partial \nu }}\end{array}\right]\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill & =\det \left[\begin{array}{cc}1& 0\\ 0& {\displaystyle \frac{\lambda }{{\nu }^2}}\end{array}\right]\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill & ={\displaystyle \frac{\lambda }{{\nu }^2}},\hfill \end{array}$$

(49)

we can express (43) as a function of the new variables as follows:

$$\begin{array}{cc}\hfill \phi (\lambda ,\nu )& ={\displaystyle \frac{\lambda }{{\nu }^2}}{\left(1+{\displaystyle \frac{\lambda }{z\nu }}\right)}^{-z}\left(1\le \nu <+\infty \right).\hfill \end{array}$$

(50)

Formula (45) is merely a displacement, and Relation (46) transforms the domain of $z_k$ to a set with lower bound 1 but no upper bound. This transformation does not change the dimension of the space, as $\nu$ depends on $\lambda$.

Hence,

$$\begin{array}{c}\hfill \phi (\lambda ,\nu )={\displaystyle \frac{z}{z-1}}{\displaystyle \frac{\partial }{\partial \nu }}{\left(1+{\displaystyle \frac{\lambda }{z\nu }}\right)}^{1-z}.\end{array}$$

(51)

As

$$\begin{array}{c}\hfill \left(1+{\displaystyle \frac{\lambda }{z\nu }}\right)\underset{\nu \to \infty }{\to }exp\left({\displaystyle \frac{\lambda }{z\nu }}\right),\end{array}$$

(52)

by substituting (52) into (51) and taking into account the fact that z is a constant, we obtain

$$\begin{array}{cc}\hfill \phi (\lambda ,\nu )& =c{\displaystyle \frac{\partial }{\partial \nu }}exp\left(-{\displaystyle \frac{\lambda }{c\nu }}\right)\left(\mathrm{s}.\mathrm{t}.c={\displaystyle \frac{z}{z-1}}\right).\hfill \end{array}$$

(53)

With the sole purpose of facilitating further calculations and avoiding an incomplete inverse gamma, we change the variables $y=\nu -1$, which ensures that the variation domain is $(0,+\infty )$, with no effect on the scale (the Jacobian is 1). Then, we take

$$\begin{array}{c}\hfill x={\displaystyle \frac{1}{cy}}.\end{array}$$

(54)

Function (53) can be rewritten as follows:

$$\begin{array}{c}\hfill \phi (\lambda ,x)={\displaystyle \frac{1}{c}}{\displaystyle \frac{\partial }{\partial x}}{\displaystyle \frac{{\partial }^2}{\partial {\lambda }^2}}{e}^{-\lambda x}.\end{array}$$

(55)

More generally,

$$\begin{array}{c}\hfill {\displaystyle \frac{{\partial }^{r-1}}{\partial x^{r-1}}}{\displaystyle \frac{{\partial }^{p-2}}{\partial {\lambda }^{p-2}}}\phi (\lambda ,x)={\displaystyle \frac{1}{c}}{\displaystyle \frac{{\partial }^r}{\partial x^r}}{\displaystyle \frac{{\partial }^p}{\partial {\lambda }^p}}{e}^{-\lambda x}(p\ge 2\mathrm{and}r\ge 1).\end{array}$$

(56)

3.4.2. Expectation of Trace Sequence Limit

To solve (56), we take into account (55). As exponential functions are sufficiently regular to interchange the derivative signs, considering Formula (55), we introduce

$$\begin{array}{c}\hfill \zeta (\lambda ,x)={\displaystyle \frac{{\partial }^r}{\partial x^r}}\phi (\lambda ,x).\end{array}$$

(57)

After taking a derivative, (57) becomes

$$\begin{array}{c}\hfill {\displaystyle \frac{{\partial }^{p-2}}{\partial {\lambda }^{p-2}}}\zeta (\lambda ,x)={\displaystyle \frac{\lambda }{c}}{(-1)}^p{x}^p{e}^{-\lambda x}.\end{array}$$

(58)

The Laplace transform of (58) is

$$\begin{array}{cc}\hfill {\zeta }_{(p-2)}\left(s\right)& ={\displaystyle \frac{\lambda }{c}}{(-1)}^p{\int }_0^{+\infty }{e}^{-sx}{x}^p{e}^{-\lambda x}dx(s>0)\hfill \end{array}$$

(59)

$$\begin{array}{cc}\hfill & ={\displaystyle \frac{\lambda }{c}}{(-1)}^p{\left({\displaystyle \frac{1}{s+\lambda }}\right)}^{p+1}.\hfill \end{array}$$

(60)

From (55), we find

$$\begin{array}{c}\hfill \zeta \left(s\right)={\displaystyle \frac{\lambda }{c}}\left({\displaystyle \frac{1}{s+\lambda }}\right).\end{array}$$

(61)

The relationship between (60) and (61) provides the following recursive formula:

$$\begin{array}{c}\hfill \zeta \left(s\right)={\displaystyle \frac{{(-1)}^{p+1}}{(p+1)!}}{\zeta }_{(p-2)}^{(p+1)}\left(s\right),\end{array}$$

(62)

where we indicate the order of the derivative in parentheses to avoid confusion with the exponents.

It follows that

$$\begin{array}{cc}\hfill {\int }_0^{+\infty }{e}^{-sx}\zeta (\lambda ,x)dx& ={\displaystyle \frac{{(-1)}^{p+1}}{(p+1)!}}{\int }_0^{+\infty }{e}^{-sx}{x}^p{e}^{-\lambda x}dx.\hfill \end{array}$$

(63)

Hence,

$$\begin{array}{cc}\hfill \zeta (x;p,\lambda )& ={\displaystyle \frac{{(-1)}^{p+1}}{(p+1)!}}{x}^p{e}^{-\lambda x}.\hfill \end{array}$$

(64)

By reversing the change given by (57) for $r=p+1$, we obtain

$$\begin{array}{cc}\hfill \phi (x;p,\lambda )& ={\displaystyle \frac{{\partial }^p}{\partial x^p}}\zeta (x;p,\lambda ).\hfill \end{array}$$

(65)

The negative signs cancel each other out, and we find

$$\begin{array}{cc}\hfill \phi (x;\alpha ,\lambda )& ={\displaystyle \frac{1}{\Gamma \left(\alpha \right)}}{\lambda }^{\alpha }{x}^{\alpha -1}{e}^{-\lambda x}(\alpha =p+1).\hfill \end{array}$$

(66)

Formula (66) is a solution of Equation (56), and this is the main result of this manuscript. As the Laplace transform can be viewed as an expectation ($E(exp-sx)=\zeta \left(s\right)$), this relation can be interpreted as the expectation of the limit function, obtained from the sequences of a non-negative matrix trace that follows a gamma pdf.

3.4.3. Gamma Parameter Selection

The adjustment of an unbiased (gamma) density for sequence (43) implicitly assumes

$$\begin{array}{c}\hfill \phi (x_m;\alpha ,\lambda )=\max c{\left({\displaystyle \frac{z_{\left[k\right]}}{z}}\right)}^{-z}\left(\mathrm{s}.\mathrm{t}.x_m=arg\underset{x}{\max }\phi (x;\alpha ,\lambda )\right),\end{array}$$

(67)

for c given by Formula (54) and imposes values for parameters $\alpha$ and $\lambda$. Additionally, $x_m$ can be obtained in closed form as

$$\begin{array}{c}\hfill x_m={\displaystyle \frac{\alpha -1}{\lambda }}.\end{array}$$

(68)

The classical transformation in (66) is introduced as follows:

$$\begin{array}{c}\hfill t=\lambda x\left(\mathrm{with}\left|{\displaystyle \frac{\partial x}{\partial t}}\right|={\displaystyle \frac{1}{\lambda }}\right),\end{array}$$

(69)

which leads to

$$\begin{array}{c}\hfill \phi (t;\alpha )={\displaystyle \frac{1}{\Gamma \left(\alpha \right)}}{t}^{\alpha -1}{e}^{-t}.\end{array}$$

(70)

This is a standard gamma density with the following maximum:

$$\begin{array}{c}\hfill arg\underset{t}{\max }\phi (t;\alpha )=\alpha -1\end{array}$$

(71)

and expectation $\alpha$.

Figure 2 shows the effects of the parameters on the gamma pdf.

If (66) and (70) must reproduce the same shape with $\lambda >0$, it is necessary to adjust the expectation of (68) to the maximum of (72). Then,

$$\begin{array}{c}\hfill \alpha ={\displaystyle \frac{\alpha -1}{\lambda }}\end{array}$$

(72)

and

$$\begin{array}{c}\hfill \lambda =1-{\displaystyle \frac{1}{\alpha }}.\end{array}$$

(73)

3.5. Relation between Solutions

Another solution for Equation (56) can be obtained by writing

$$\begin{array}{c}\hfill {\displaystyle \frac{{\partial }^r}{\partial x^r}}{\displaystyle \frac{{\partial }^2}{\partial {\lambda }^2}}\phi (\lambda ,x)={\displaystyle \frac{{(-1)}^r}{c}}{\lambda }^r{\displaystyle \frac{{\partial }^2}{\partial {\lambda }^2}}\phi (\lambda ,x)\end{array}$$

(74)

and denoting $\psi (\lambda ,x)={\partial }^2\phi (\lambda ,x)/\partial {\lambda }^2$. The relation between derivatives with respect to x is

$$\begin{array}{c}\hfill {\displaystyle \frac{{\partial }^r}{\partial x^r}}\psi (\lambda ,x)={\displaystyle \frac{{(-1)}^r}{c}}{\lambda }^r\psi (\zeta ,x).\end{array}$$

(75)

Following the same reasoning described in Section 3.4.2 and considering that ${\partial }^2\phi (\lambda ,x)/\partial {\lambda }^2=x^{2}exp(-\lambda x)$, the Laplace transforms of ${\psi }^{\left(p\right)}$ and $\psi$ are

$$\begin{array}{cc}\hfill {\psi }_{\left(r\right)}\left(s\right)& ={(-1)}^r{\displaystyle \frac{{\lambda }^r}{c}}{\int }_0^{+\infty }{e}^{-sx}{\psi }^{\left(r\right)}(\lambda ,x)dz\hfill \end{array}$$

(76)

$$\begin{array}{cc}\hfill & ={(-1)}^r{\displaystyle \frac{{\lambda }^r}{c}}{\left({\displaystyle \frac{1}{s+\lambda }}\right)}^{r+1}\hfill \end{array}$$

(77)

$$\begin{array}{cc}\hfill \psi \left(s\right)& ={\displaystyle \frac{1}{c}}{\int }_0^{+\infty }{e}^{-sx}\psi (\lambda ,x)dx\hfill \end{array}$$

(78)

$$\begin{array}{cc}\hfill & ={\displaystyle \frac{1}{c}}\left({\displaystyle \frac{1}{s+\lambda }}\right),\hfill \end{array}$$

(79)

respectively.

Comparing (77) and (79), we find

$$\begin{array}{c}\hfill \psi \left(s\right)={\displaystyle \frac{{(-1)}^r}{r!}}{\lambda }^r{\psi }^{\left(r\right)}\left(s\right),\end{array}$$

(80)

which leads to

$$\begin{array}{c}\hfill \psi (x;\lambda )={\displaystyle \frac{1}{r!}}{\lambda }^r{e}^{-\lambda x}.\end{array}$$

(81)

Equation (81) is a Poisson equation with parameter $\lambda$ ($\lambda >0$), and it also solves the differential Equation (56).

The pdfs that solve Equation (56) state the problem in a Bayesian context. The gamma and Poisson densities are conjugate. The interpretation in this context is immediate when comparing (66) and (81). For a value of $r=p+1$, we achieve equality by introducing a factor ${\prod }_{p}x$, and

$$\begin{array}{c}\hfill {\displaystyle \frac{1}{\Gamma \left(\alpha \right)}}{\lambda }^{\alpha }{x}^{\alpha -1}{e}^{-\lambda x}=\left(\prod _{p}x\right){\displaystyle \frac{1}{(p+1)!}}{\lambda }^{p+1}{e}^{-\lambda x}.\end{array}$$

(82)

The Bayesian theorem can be written as follows:

$$\begin{array}{c}\hfill P\left(\theta \right|x)=L(\theta \left)P\right(x\left|\theta \right),\end{array}$$

(83)

where $P\left(\theta \right|x)$ is the posterior factor, $L\left(\theta \right)$ is the likelihood factor, and $P\left(x\right|\theta )$ is the prior factor. We identify these factors as follows:

$$\begin{array}{cc}\hfill P\left(\theta \right|x)& ={\displaystyle \frac{1}{\Gamma \left(\alpha \right)}}{\lambda }^{\alpha }{x}^{\alpha -1}{e}^{-\lambda x}\hfill \end{array}$$

(84)

$$\begin{array}{cc}\hfill P\left(x\right|\theta )& ={\displaystyle \frac{1}{(p+1)!}}{\lambda }^{p+1}{e}^{-\lambda x}\hfill \end{array}$$

(85)

$$\begin{array}{cc}\hfill L\left(\theta \right)& =\prod _{p}x\hfill \end{array}$$

(86)

$$\begin{array}{cc}\hfill & =x^{\alpha -1},\hfill \end{array}$$

(87)

where $\theta =\{\alpha ,\lambda \}$ is the minimal sufficient statistic.

4. Moving to Clustering Validation

Transferring the previous results to the clustering validation problem requires some considerations. Clustering denotes the task of assigning each item of $\mathbf{X}$ to a subset $l_k\in \mathcal{Y}=\{l_1,\dots ,l_K\}$ or cluster based on the geometric or probabilistic properties of vectors ${\mathbf{x}}_i$. Formally, it is a map [4]:

$$\begin{array}{cc}\hfill h:& {\mathbf{x}}_{\mathbf{i}}⟶\mathcal{Y}({\mathbf{x}}_i\in \mathbf{X})\hfill \\ \hfill & h\left({\mathbf{x}}_i\right)\mapsto l_k,\hfill \end{array}$$

(88)

This map provides the assignment $\left\{({\mathbf{x}}_1,l_k),\dots ,({\mathbf{x}}_m,l_{k^{\prime }})\right\}$. The evaluation of the quality of this assignment in a probabilistic clustering model is stated in terms of the minimization of a loss function, usually the log-likelihood; in particular, the parameters that maximize the log-likelihood are those that provide the best clustering result. It can be immediately observed that (43) is the inverse of the non-logarithmic likelihood. From this point, we obtain the expectation of the limit, and the result is a pdf.

The existence of a pdf allows for the construction of acceptance regions, and its utilization—with or without disjoint subsets—allows for the division of the original dataset. This fact allows for the definition of acceptance regions for the distributional parameters. This problem, which is classic in the field of statistics, is usually known as hypothesis testing. It is stated as a set of parameter values $\theta \in {\Theta }_0$, for which certain values ${\Theta }_0$ constitute the null hypothesis or acceptance region. Its complementary in $\Theta$ is ${\Theta }_1$, constituting the rejection of the null hypothesis. In particular, the acceptance region $\mathcal{R}$ is

$$\begin{array}{cc}\hfill \mathcal{R}& =P(\theta \in {\Theta }_0)\hfill \end{array}$$

(89)

$$\begin{array}{cc}\hfill & =\left\{\theta \mathrm{s}.\mathrm{t}.\theta \in f_{{\theta }_0\in \Theta }(x;{\theta }_0)\right\}\hfill \end{array}$$

(90)

and, for a sample, dependence on the data $\theta =\theta \left(x\right)$ leads to

$$\begin{array}{c}\hfill \mathcal{C}=\left\{x\mathrm{s}.\mathrm{t}.f_{{\theta }_0\in \Theta }(x;{\theta }_0)\ge \alpha \right\},\end{array}$$

(91)

providing the probability $\alpha$ level confidence interval $({\theta }_1^{-1}\left(x\right),{\theta }_2^{-1}\left(x\right))$, with those values being the boundaries of $\mathcal{C}$. Then, the confidence level is

$$\begin{array}{c}\hfill 1-\beta ={\int }_{{\theta }_1^{-1}\left(x\right)}^{{\theta }_2^{-1}\left(x\right)}f(x;\theta )dx.\end{array}$$

(92)

In the literature, $\beta$ is usually referred to as $\alpha$. We use this notation to avoid confusion with the parameters of the gamma pdf. A detailed pedagogical exposition of these concepts, containing many examples, has been presented in [57]. This text deals with point estimation, the construction of hypothesis tests, and the construction of confidence intervals, emphasizing the equivalence between them in a standard statistical exposition.

When taking into account that only positive integers make sense in the clustering problem, Formula (66) should be rewritten as $\phi (x;\alpha ,\lambda )=\mathbf{J}/\Gamma \left(\alpha \right){\lambda }^{\alpha }{x}^{\alpha -1}{e}^{-\lambda x}$, with $\mathbf{J}$ being a suitable column matrix. We follow the statistical convention to omit the one-dimensional basis.

Moreover, it is important that the normalization of the data matrix must be carried out in the same space. As a consequence of smoothing via columns, the norm $\parallel \widehat{{\mathbf{Y}}^{\prime }}\widehat{\mathbf{Y}}\parallel =n_r$, where $n_r$ is the rank of the image of matrix $\mathbf{X}$ of (1). To achieve accurate results, it is necessary to transform to the same representation space and the following:

$$\begin{array}{c}\hfill \parallel {\mathbf{Y}}^{\prime }{\mathbf{Y}\parallel }_1={\displaystyle \frac{1}{n_r}}\parallel {\widehat{\mathbf{Y}}}^{\prime }\widehat{\mathbf{Y}}{\parallel }_1.\end{array}$$

(93)

This results in a re-estimation of sequence (38):

$$\begin{array}{c}\hfill z_{\left[k\right]}^{★}=\sqrt{n_r}{z}_{\left[k\right]}.\end{array}$$

(94)

This nuance has no theoretical effects since the limit of the sequence (43) is also a gamma, but it causes bias in the estimation of parameters.

On the other hand, the clustering validation problem makes sense for positive integers. The solution needs to be re-scaled for transformation to this space, for which we take the following:

$$\begin{array}{c}\hfill c_r={\displaystyle \frac{z}{1-z}}\sqrt{m{n}_r}.\end{array}$$

(95)

These results allow the evaluation of the number of clusters for any data frame that can be written in matrix form, with the only restriction being that the observational variables are linearly independent. This makes it possible to evaluate data structures that simultaneously contain continuous and categorical variables and do not require distributional assumptions or their independence on the parameters.

4.1. Computational Remarks

Several parameters affecting the reliability of the gamma density are required. To better clarify their effect, we divide the process into three phases. First, we $\left(i\right)$ obtain the sequence of traces, $\left(ii\right)$ evaluate the limit sequence, and $\left(iii\right)$ adjust the gamma density, for which it is necessary to reproduce the shape—especially the maximum and a sufficient number of values for the queue, which is the value of the ancillary statistic (k).

4.1.1. NMF Parameters

The first step is to obtain an NMF with random initialization (otherwise, it would not make sense), varying k from 1 to the ancillary statistic in increasing order. For each factorization, the sequence (43) is computed. This process is repeated, and the results are kept in a matrix. In this phase, the parameters of matrix $\mathbf{X}$ are also obtained, including the matrix dimension, the matrix rank, and z. Algorithm 1 provides details of this process.

As the computational cost required to obtain a reliable approximation of (19) is high, an alternative approach for the trace sequence is to relax the approximation degree and re-compute the terms with q random re-initializations of the matrices in formulas (25) and (26) (otherwise, the re-estimation would not make sense). Then, by selecting the statistic $z^{★}$ such that $E{z}^{★}=z$, we obtain an estimate sequence.

The convergence condition of (27) is somewhat complex in practice. A tight condition provides good values when $k>\min (m,n)$, but it causes problems if $k<\min (m,n)$, and the iterative process may not stop. Conversely, an overly wide condition provides no representative values. To achieve reasonable results, it is sufficient to take a value of $ϵ\sim O(1/m)$. Furthermore, deciding the rank of the matrix is not a trivial problem. This is a classical situation illustrating a well-solved problem in theory that faces problems in practice. This issue is intensified by transformations to the probabilistic space. We use SVD, selecting the significant eigenvalues for this situation.

Algorithm 1: Data Parameters and Trace Estimation.

4.1.2. Sequence of Traces

This step involves selecting an estimator for the matrix containing the trace sequences, which is calculated for each value of k. When the estimator is computed, we handle the results using local likelihood regression with the help of the $sm$ R package [58]. The selected smooth parameter is $h=1$, which has no effect on the data. It is also necessary to define a support (grid) that contains the positive integers valued at those same points, as detailed in Algorithm 2.

Algorithm 2: Trace Sequence.

4.1.3. Gamma Density Adjustment

In this step, we estimate the peak of (67) according to the criteria discussed in Section 3.4.3 with no further problems. Algorithm 3 details this simple procedure.

Algorithm 3: Gamma Parameters.

Input: S  1: $\alpha =\max \left(S\right)+1$  2: $\lambda =1-1/\alpha$ Output: $\alpha$,$\lambda$ (parameters of gamma pdf)

4.1.4. Overlapping

The application of the previous theoretical results for the clustering validation is based solely on the hypothesis of the linear independence of the observational variables. However, this situation may be very different from that which occurs in practical situations.

Assuming that the probabilistic image has been obtained from a matrix for which its columns are linearly independent (the information contained in the sample is preserved by omitting the linearly dependent observational variables), the existence of overlap plays an important role in obscuring the estimation of the parameters of (66), even in the case where the hypotheses required for its derivation are fulfilled. This is because the overlap between variables affects their number and the contained information. In the case of uncorrelated variables, this effect is dramatic. Let us consider the case of the mixture $f\left(x\right)=\lambda g_1\left(x\right)+(1-\lambda )g_2\left(x\right)$ with strong overlap. In this case, $g_2\left(x\right)\approx g_1\left(x\right)$; thus, $f\left(x\right)\approx g_1\left(x\right)$. This drawback can be corrected by introducing the overlapping effect as a factor.

Furthermore, in multivariate structures, the correction must consider the structure of the distribution of the information: it can be within variables, between variables, or both. This means that considering $\mathrm{sign}(\alpha -n_r)$, which provides information on the distributional behavior of the information (in the case where the number of non-overlapping informative variables is less than $\alpha$, the information is mainly contained within each variable; otherwise, the existence of more variables provides the main source of information between them). This suggests the introduction of the following factor:

$$\begin{array}{c}\hfill \delta =\left\{\begin{array}{cc}{n}_r{\vartheta }^{\mathrm{sign}(\alpha -n_r)}\hfill & \mathrm{if}\det \left(\mathbf{C}\right)=1\hfill \\ {\vartheta }^{\mathrm{sign}(\alpha -n_r)}\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(96)

where $\mathrm{sign}(\alpha -n_r)=1$ if $\alpha -n_r\ge 0$, $\mathrm{sign}(\alpha -n_r)=-1$ if $\alpha -n_r<0$, $\vartheta$ is the maximum overlapping, and $\det \left(\mathbf{C}\right)$ is the determinant of the correlation matrix. It is noted that $\det \left(\mathbf{C}\right)=\mathbf{1}$ indicates non-correlation, while smaller values correspond to correlation. This correction is also described in Algorithm 4.

Algorithm 4: Overlap and Correlation.

On the other hand, we indicate that this correction requires the choice of a value ${\vartheta }_0$ for the overlap parameter, and the choice of the distributional criteria is an open question.

5. Examples

The examples have the purpose of evaluating the pros and cons of the proposed method. Before performing experiments, we provide a classical toy example related to the classification of documents.

Example 1.

For a co-occurrence data frame $\mathbf{X}$ consisting of a corpus of seven ($d1$–$d7$) documents containing letters $\{a,b,c,d,e,f\}$, which we assimilate into words in a thesaurus, the probability image $\mathbf{Y}$ is as follows:

$$\begin{array}{c}\hfill \mathbf{X}=\begin{array}{cc}& \begin{array}{cccccc}a& b& c& d& e& f\end{array}\\ \begin{array}{c}d1\\ d2\\ d3\\ d4\\ d5\\ d6\\ d7\end{array}& \left(\begin{array}{cccccc}4& 3& 4& 0& 0& 0\\ 5& 3& 3& 0& 0& 0\\ 4& 3& 3& 0& 0& 0\\ 0& 0& 0& 9& 0& 0\\ 0& 0& 0& 8& 2& 0\\ 0& 0& 0& 0& 3& 4\\ 0& 0& 0& 0& 4& 3\end{array}\right)\end{array}\mathbf{Y}=\left[\begin{array}{cccccc}0.06& 0.05& 0.06& 0.00& 0.00& 0.00\\ 0.08& 0.05& 0.05& 0.00& 0.00& 0.00\\ 0.06& 0.05& 0.05& 0.00& 0.00& 0.00\\ 0.00& 0.00& 0.00& 0.14& 0.00& 0.00\\ 0.00& 0.00& 0.00& 0.12& 0.03& 0.00\\ 0.00& 0.00& 0.00& 0.00& 0.05& 0.06\\ 0.00& 0.00& 0.00& 0.00& 0.06& 0.05\end{array}\right]\end{array}$$

The objective is to find the suitable number of subjects that classifies the documents, with the number of subjects being unknown.

Simple visual inspection of the data frame $\mathbf{X}$ indicates three or four clusters. We first perform NMF clustering for the choices of $k=3$.

The results for $k=3$ are as follows:

$$\begin{array}{c}\hfill \mathbf{W}=\left[\begin{array}{ccc}0.34& 0.00& 0.00\\ 0.34& 0.00& 0.00\\ 0.31& 0.00& 0.00\\ 0.00& 0.00& 0.48\\ 0.00& 0.02& 0.52\\ 0.00& 0.49& 0.00\\ 0.00& 0.49& 0.00\end{array}\right]{\mathbf{W}}_{docs}=\left[\begin{array}{ccc}d2& -& -\\ d1& -& -\\ d3& -& -\\ -& -& d4\\ -& d5& d5\\ -& d6& -\\ -& d7& -\end{array}\right]{\mathbf{W}}_{prob}=\left[\begin{array}{ccc}d2& d6& d4\\ d1& d7& d5\\ d3& d5& -\\ -& -& -\\ -& -& -\\ -& -& -\\ -& -& -\end{array}\right]\end{array}$$

The matrix $\mathbf{W}$ represents the probabilities of documents vs. model components or clusters, and they are assimilated to the matrix’s columns. Once obtained, the qualitative matrix ${\mathbf{W}}_{docs}$ is a probabilistic classification per cluster. In each column, lines are the probabilities close to zero. This matrix is usually written as lists in decreasing order, and it is a probabilistic classification shown in matrix ${\mathbf{W}}_{prob}$.

Figure 3 shows the gamma pdf that provides credibility for the model components (or the number of clusters) used to achieve the NMF.

The parameters for the gamma pdf in Figure 3 are $\alpha =3$ and $\lambda =0.67$, providing an acceptance hypothesis test of $\beta =0.90$ for the confidence β-level acceptance interval of $[3,4]$.

From the matrix $\mathbf{W}$, it is possible to obtain a disjoint classification through the introduction of a Bayes classifier [43]. For the Bayes classifier ${\mathbf{w}}_i=ma{x}_{w_i\in W}\left(\mathbf{W}\right)$, the qualitative matrix containing the documents classified according its weight in the span space are

$$\begin{array}{c}\hfill {\mathbf{W}}_{Bayes}=\left[\begin{array}{ccc}d1& -& -\\ d2& -& -\\ d3& -& -\\ -& -& d4\\ -& -& d5\\ -& d6& -\\ -& d7& -\end{array}\right].\end{array}$$

This procedure illustrates the equivalence of NMF and k-means clustering. Despite NMF clustering being related to probabilistic and k-means clustering, the construction of the gamma pdf has no hypothesis that goes beyond the construction of a statistic, and it does not depend on the clustering method.

To examine the examples, we also compared the proposed criterion with indices that provide good results: the Dunn and silhouette indices and the gap statistic. These indices were chosen due to their widespread use in the related literature and the good results they provide: notably, these indices provide a single value. We took the maximum of (66) as $\kappa =arg\max f(x;\alpha ,\lambda )$ to make comparisons possible. We also considered the number of labels or distributions for each dataset or the distributions that generate them.

The Dunn index is an ancient internal evaluation index for identifying compact sets of clusters, with a small variance within them [59]. The silhouette index evaluates the compactness and separation of clusters [25]. The underlying idea of the gap statistic is to obtain the expected differences in the k clusters, with each of them having r elements; then, they are compared with a reference distribution [29]. These indices assume the optimization of a loss function related to the clustering method. We assume a k-means underlying partition schema.

We aid with the clValid-package of the R computing environment to obtain the Dunn and silhouette indices [60]. For the gap statistic, we use the factoextra package [61] with a brute force recursive procedure consisting of a loop varying the number of clusters.

We conducted experiments in two contexts. The first involved illustrative examples, and we selected four examples from well-known datasets in the UCI repository (https://archive.ics.uci.edu/datasets, accessed on 2 September 2023), which is currently of a reduced size. Using these examples, we illustrate the validation ability of the proposed criterion. Meanwhile, the other examples possessed more complex data structures (first used in [20], and it is available at http://cs.joensuu.fi/sipu/datasets, last accession 2 September 2023). We examined the validation criterion against overlapping, dimension variation, and different numbers of observations per cluster.

5.1. Simple Examples

The four datasets from the UCI repository are briefly described in

Table 1. Overview of the main features of selected datasets. The sub-categories of the dataset glass correspond to the presence of additional features. The size refers to the number of rows (m) and columns (n) of the data matrix. $n_r$ is the rank of the probabilistic image, $\det \left(\mathbf{C}\right)$ is the determinant of the correlation matrix, and $\vartheta$ is the maximum overlapping.

Dataset	Size ($\mathit{m}\times \mathit{n}$)	Labels	Sub-Categories	${\mathit{n}}_{\mathit{r}}$	$\det \left(\mathbf{C}\right)$	$\mathit{\vartheta }$
iris	150 × 5	3	no	2	0.42	0.66
seeds	210 × 8	3	no	2	0.67	0.11
ecoli	336 ×9	6	no	4	0.53	0.71
glass	214 × 19	7	yes	3	0.00	0.60

. The selection criterion was the non-existence of missing values in order to avoid the need for a pre-processing step.

The iris dataset is one of the most popular and best-studied datasets, and it has been used in many statistical studies. It contains three species of labeled flowers. Its original use can be attributed to Fisher. A discussion of how it was obtained and the conditions under which various authors attribute different results (2–3 categories, while some provide 4) can be found in [62]. The ecoli dataset was proposed to illustrate the unsupervised classification of biological functions [63]. The dataset seeds was first used to analyze several clustering algorithms on a real dataset of seed images [64]. The glass dataset allows for the study of the identification of seven types of glass and the effects of certain additives. Figure 4 provides a graphical description of the datasets.

Table 2. Results for Dunn (minimize), silhouette (maximize), and gap statistic (maximize), values of the non-parametric sequence of (69) (maximize), and gamma pdf versus the number of clusters (maximize). The dataset glass (a) omits thermal treatments, but they are included in (b). The maximum for the gap statistic in cases (a) and (b) was 14, which does not appear in the table. The gamma pdf and sequence of (43) for case (b) were 14, which also do not appear in the table.

Dataset	Validation Index	Clusters
		2	3	4	5	6	7	8	9	10
iris	Dunn	0.077	0.099	0.137	0.082	0.085	0.087	0.087	0.062	0.068
	silhouette	0.582	0.460	0.419	0.354	0.345	0.315	0.346	0.318	0.318
	gap	0.473	0.495	0.442	0.405	0.455	0.450	0.466	0.479	0.473
	sequence	0.131	0.225	0.191	0.129	0.094	0.070	0.055	0.043	0.340
	gamma	0.228	0.234	0.180	0.124	0.079	0.049	0.029	0.017	0.010
seeds	Dunn	0.055	0.086	0.049	0.056	0.044	0.078	0.084	0.089	0.089
	silhouette	0.466	0.404	0.321	0.321	0.275	0.253	0.284	0.244	0.250
	gap	0.312	0.415	0.395	0.367	0.374	0.373	0.361	0.361	0.356
	sequence	0.048	0.219	0.207	0.138	0.092	0.066	0.050	0.039	0.031
	gamma	0.152	0.184	0.167	0.135	0.103	0.075	0.053	0.037	0.025
ecoli	Dunn	0.080	0.087	0.041	0.065	0.072	0.071	0.053	0.086	0.085
	silhouette	0.347	0.396	0.338	0.341	0.249	0.244	0.223	0.242	0.221
	gap	0.553	0.639	0.621	0.633	0.635	0.637	0.621	0.637	0.628
	sequence	0.030	0.060	0.090	0.120	0.105	0.089	0.073	0.057	0.053
	gamma	0.006	0.044	0.100	0.142	0.156	0.146	0.121	0.093	0.067
glass (a)	Dunn	0.190	0.089	0.167	0.154	0.137	0.154	0.055	0.055	0.061
	silhouette	0.388	0.421	0.447	0.432	0.328	0.300	0.246	0.373	0.246
	gap	0.929	0.945	0.990	0.997	0.069	0.111	0.118	0.152	0.154
	sequence	0.025	0.051	0.076	0.102	0.127	0.152	0.138	0.124	0.110
	gamma	0.005	0.026	0.063	0.102	0.129	0.138	0.130	0.112	0.089
glass (b)	Dunn	0.142	0.085	0.133	0.132	0.131	0.054	0.054	0.054	0.039
	silhouette	0.448	0.367	0.391	0.298	0.329	0.218	0.243	0.342	0.227
	gap	0.957	1.009	1.000	1.053	1.069	1.085	1.090	1.094	1.300
	sequence	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.105	0.118
	gamma	0.000	0.000	0.000	0.000	0.001	0.003	0.008	0.019	0.034

shows the results obtained for the considered validation indices. It corresponds to $k=1,\dots ,10$ clusters. The value $\kappa$ refers to the $arg\max f(x;\alpha ,\lambda )$ of Formula (66).

The results for the glass (a) dataset assume the selection of the first $seven$ columns of the data frame. It was found that glass contains $seven$ classes of glasses, identifying them with the labels $1,\dots ,7$. In addition, there are additional treatments consisting of additives. Despite the quantities not being the same, it is easy to identify them (columns 8 and 9 of the data frame) as binary variables.

Table 3. Comparison of results. Overview of the results obtained with the used indices. Dataset glass (a) corresponds to the selection of the independent identically distributed (i.i.d.) variables, while dataset glass (b) introduces binary variables. These binary variables are captured by the gamma pdf and the gap statistic. $\kappa =arg\max f(x;\alpha ,\lambda )$.

Dataset	Labels	Dunn	Silhouette	Gap	$\mathit{\kappa }$
iris	3	4	2	3	3
seeds	3	9	2	3	3
ecoli	6	3	3	2	6
glass (a)	7	2	4	5	7
glass (b)	7	2	2	14	14

presents both results for this dataset, and in Figure 5, we present the case alone. All results obtained with our validation criteria were similar to those of the gap statistic; however, the Dunn and silhouette index and the gap statistic did not capture the presence of the binary variable. This illustrates, from a practical point of view, an advantage of our less restrictive assumption.

A comparison is presented in Table 3. Figure 5 shows the behavior of the pdf compared to the non-parametric curve obtained with Equation (43), in which the relation with both maximums can be seen.

Figure 6 shows the NMF clustering with the iris dataset. This dataset is useful to illustrate the independence of our validation criterion and the clustering method. To obtain the density of Equation (46), we set $p=3$ iterations in the switching process to adjust (34) and $q=20$ re-estimates, while to obtain the classification of Figure 6, we needed $O\left({10}^5\right)$ iterations. This shows that the underlying matrices of (40) are not necessary a plausible clustering and cannot be used to obtain the corresponding classification (except for the case where a good approximation of (30) is reached). This is due to the different speeds of convergence of the adjustment process to obtain the factorization and the sequence $z_{\left[k\right]}$.

5.2. More Complex Data Structures

To show the effects of overlapping, dimension variation, and different numbers of observations per cluster, we selected several synthetic datasets created in [20] to study validation criteria behavior (available at http://cs.joensuu.fi/sipu/datasets, last accession on 2 September 2023). These datasets were generated using a certain number of distributions, in which the overlaps and number of generating distributions vary.

The datasets include four types of artificial data configurations in which the overlapping effect increases (datasets $s1$, $s2$, $s3$, and $s4$), the number of clusters varies (datasets $a1$, $a2$, and $a3$), and the dimension varies (datasets $dim064$, $dim128$, $dim256$, $dim512$, and $dim1024$). These datasets are balanced (i.e., equal number of entities in each cluster). An $unbalanced$ dataset allows us to examine the effect of different numbers of observations per cluster.

To proceed with the examples, we selected the same parameters for each case: the number of component estimations $k=1,\dots ,120$, $q=200$ re-estimations, no condition on the degree of approximation, and $p=100$ iterations in the process of switching Equations (25) and (26). To determine the rank of the matrix, we considered the number of relevant eigenvalues of matrix $\mathbf{Y}$ with the help of the condition number (i.e., the quotient of the largest involved eigenvalue).

Running Algorithms 1–3, we obtain the parameter $\alpha$. These values are acceptable if the overlapping between observational variables does not exists. In the case of the existence of overlapping, it is necessary to adjust them according Formula (96), providing new values for ${\alpha }^{★}$ in

Table 4. Results of the execution of Algorithms 1–4. The values of $\alpha$ correspond to the output provided by Algorithm 3. The overlapping and correlation parameters ($\vartheta$ and $det\left(\mathbf{C}\right)$, respectively) are obtained from the probabilistic image $\mathbf{Y}$ in Algorithm 1. The elevation factor $\delta$ according Formula (94) provides the new parameter ${\alpha }^{★}$. For overlapping, we consider the values $\vartheta \ge 0.7$.

	a1	a2	a3	a4	s1	s2	s3	d32	d64	d128	d256	d512	d1024	unb.
$\alpha$	20	22	15	9	8	29	30	11	12	12	13	13	8	8
$n_r$	2	2	2	2	2	2	2	9	10	10	12	12	13	2
$\det \left(\mathbf{C}\right)$	0.998	0.982	0.886	0.995	0.853	1	1	0.356	0.427	0.686	0.558	0.453	0.011	0.982
$\vartheta$	0.74	0.70	0.78	0.89	0.34	0.40	0.81	0.69	0.74	0.74	0.84	0.81	0.84	0.17
${\alpha }^{★}$	15	15	12	8	8	29	48	16	16	16	16	16	16	8

.

Figure 7 shows the biplots of the data and the corresponding distributions for estimating the credibility of the number of clusters according to the proposed criteria.

5.3. Comparative

Comparisons illustrate the pros and cons of the proposed method versus the Dunn, silhouette, and gap statistic indices. In all cases, the most suitable value is chosen to minimize the corresponding loss function. The number of re-sampling steps in the bootstrap procedure used to estimate the reference density for determining the gap statistic was $B=100$. Those cases corresponded to the default parameters of the used R packages. Additionally, as in the previous examples, we took the value of the maximum of the gamma pdf for comparison with the other indices, with the results summarized in

Table 5. Comparative. The values m and n refer to the size of the data frame (rows and columns, respectively). D is the number of clusters used to generate the datasets (provided by Franti in [20]). Values of the silhouette index and gap statistic are given in the respective columns. $\kappa$ denotes the results for the maximum of the gamma pdf according to (66).

Data Set (Size)	D	Dunn	Silhouette	Gap	$\mathit{\kappa }$
a1 ($5000\times 2$)	15	15	15	19	15
a2 ($5000\times 2$)	15	15	15	18	15
a3 ($5000\times 2$)	15	52	17	17	12
a4 ($5000\times 2$)	15	68	16	15	8
s1 ($3000\times 2$9	25	21	20	53	8
s2 ($5250\times 2$)	35	44	35	37	29
s3 ($7500\times 2$)	50	65	50	57	48
dim32 ($1024\times 32$)	16	16	16	16	16
dim64 ($1024\times 64$)	16	16	16	16	15
dim128 ($1024\times 128$)	16	16	16	16	16
dim256 ($1024\times 256$)	16	16	16	16	16
dim512 ($1024\times 512$)	16	16	16	16	18
dim1024 ($1024\times 1024$)	16	16	16	16	16
unbalance ($6500\times 2$)	8	2	2	15	8

.

A quantitative comparison of the results can be carried out using the Adjusted Rand Index (ARI), defined as [65]

$$\begin{array}{c}\hfill \mathrm{ARI}={\displaystyle \frac{k-E\left(k\right)}{\max k-E\left(k\right)}},\end{array}$$

(97)

with k being the number of clusters and $E\left(k\right)$ denoting its expectation, which is a measure of the similarity of clustering results. To avoid negative values, and to be consistent with the $L_1$ metric in the context of non-negativity, we define the relative error in the $L_1$ space as:

$$\begin{array}{c}\hfill {\mathrm{Rel}}_{L_1}={\displaystyle \frac{|k-E(k\left)\right|}{\max k-\min k}}\end{array}$$

(98)

with $\max \left(k\right)$ and $\min \left(k\right)$ being the maximal and minimal optimal number of clusters provided by a method, respectively. This index provides values over the interval $[0,1]$, and values close to zero are preferable. The results are shown in

Table 6. Comparative of clustering validation methods. The $Re{l}_{L1}$ is computed according Formula (97) for Dunn, silhouette, gap, and $\kappa$. The $1-\beta$ value assumes that the expectation of the number of clusters is D, and it is computed for the cases such that $\kappa \ne D$. The acronym $ns$ means that the value of $1-\beta$ is not significant (exactly zero).

Dataset	$\mathit{E}\left(\mathit{x}\right)$	Dunn	Silhouette	Gap	$\mathit{\kappa }$	$1-\mathit{\beta }$
iris	3	0.500	0.500	0.000	0.000	$ns$
seeds	3	0.857	0.143	0.000	0.000	$ns$
ecoli	6	0.750	0.750	1.000	0.000	$ns$
glass (a)	7	1.000	0.600	0.400	0.000	$ns$
glass (b)	14	1.000	1.000	0.000	0.000	$ns$
a1	15	0.000	0.000	1.000	0.000	$ns$
a2	15	0.000	0.000	1.000	0.000	$ns$
a3	15	0.925	0.050	0.050	0.075	$0.10$
a4	15	0.883	0.017	0.000	0.117	$0.30$
s1	25	0.089	0.111	0.622	0.378	$O\left({10}^{-7}\right)$
s2	35	0.474	0.526	0.105	0.316	$0.09$
s3	50	0.082	0.000	0.412	0.118	$0.30$
dim32	16	0.000	0.000	0.000	0.000	$ns$
dim64	16	0.000	0.000	0.000	0.000	$ns$
dim128	16	0.000	0.000	0.000	0.000	$ns$
dim256	16	0.000	0.000	0.000	0.000	$ns$
dim512	16	0.000	0.000	0.000	0.000	$ns$
dim1024	16	0.000	0.000	0.000	0.000	$ns$
unbalance	8	0.462	0.462	0.538	0.000	$ns$

.

On the other hand, we provide values of the $1-\beta$ levels of $\kappa$, assuming that the reported number of densities (D) is the correct value. These examples show that the proposed validation criteria do not assume any partition or fuzzy scheme. It evaluates the number of clusters and provides no classification or clusterization.

It can be obeserved that the proposed procedure captured overlapping, providing underestimation in these cases and exhibiting stability when the dimension increased (compared with D). Therefore, the procedure works well in the unbalanced case. A discussion of the computational cost to obtain the NMF solutions is shown in [50], and some tricks are explained in [45].

Also, while the examples show the behavior of the proposed validation method and open the door to statistical inference, they do not constitute a numerical confirmation. Performing confirmatory numerical experiments would require a study that should include real-world examples including other recent validation indices, such as the Wemmert–Gancarski index [66], SpecialK [67], and Stadion [18], which should be used to establish more rich conclusions.

6. Discussion

Usually, the gamma density is obtained as a sum of exponentials [41] (p. 179). It is possible to prove that the independence of sums and sums of squares implies that the coefficient of variation follows this distribution [68,69], which is also true for the harmonic mean of the posterior distribution [70]. Recently, [71] focused on reducing the variance in such cases. Our approach aims to evaluate the expectation of a trace sequence in the probabilistic space. In fact, the Laplace transform confirms this interpretation. The same result can be reached by taking derivatives and carrying out normalization (56); however, we minimized the statistical assumptions and procedures needed to achieve this result. Additionally, we present an approach for applications in the context of clustering validations.

Our approach does not result in any hypotheses in terms of the space of parameters—that is, in neither the distribution nor the dimensionality of the parameters—and, therefore, constitutes a non-parametric approach. One of the advantages of such an approach is that it supports any type of variable. The unique hypothesis, in this sense, is the independence of observational variables. Conversely, the transformation to the probabilistic space and use of the KL divergence provides a maximum-likelihood estimate. However, some issues need to be highlighted.

First, it seems that the main drawback of this method results from the NMF approach being a slow iterative process with high computational costs. We alleviated this through the introduction of several random re-estimations. Another problem is that the gamma pdf is continuous on the compact set $[0,+\infty )$, while the clustering problem is appropriate only for non-negative integers. Additionally, the type of smoothing used in this method implicitly assumes Euclidean distances, while the approximation problem requires the $L_1$ norm. Although some works have focused on this issue [72,73], according to our literature search, such works have received little attention in recent years. We recall that this choice is critical to reproduce the data structure and is related to the variance.

Although we think that, from a practical viewpoint, the provided examples do not validate our clustering validation criteria or our attempts, they do provide a practical framework that illustrates several issues. Our proposal worked well for the UCI Machine Learning Repository datasets, but we also selected some of the synthetic datasets developed by Franti to illustrate certain problems. We made this choice as the effects of superposition, stability, and size could be controlled. In this sense, we believe that the greatest difficulties arise when attempting to establish conditions to adjust the model to cases in which compactness is high (datasets $a4$, $s1$, and $s1$).

We did not apply our proposed procedure to other data structures, such as those with well-defined geometric shapes, for which available methods such as CLARANS [74] offer good results and have been extended in the context of Big Data, although we think that this would allow for good exploration. However, according to the experiments of Section 5, our results are comparable to those provided by the selected indices.

Moreover, the provision of a pdf provides several advantages. In unsupervised environments, a graph can be easily interpreted by human operators without analytical skills, allowing them to incorporate other results in the analysis as part of their expert judgment. Expert judgment does not always coincide with the statistical results, leading to controversial situations. Additionally, expert judgment comes from many years of studying a discipline or practice, and it should not be misconsidered. We believe that controversial situations should be placed in the area of the selection of relevant observational variables [75].

Without entering into this discussion—which is philosophical and profound—we merely indicate that graphic visualization can provide relevant considerations for practical situations, and a contribution of the proposed method is the visualization of densities, thereby providing relevant graphical information.

Future research should include our proposal in a more exhaustive comparison, including artificial datasets that reproduce shapes. The effect of superposition is another window to explore.

7. Conclusions

Although inference on clustering is controversial, a pdf was built from the sequence of traces obtained with NMF techniques, the construction of which requires no assumptions beyond linearly independent uncorrelated observational variables. Thus, by transforming observations to the probabilistic space, with the expectation provided by the limit of the distribution on the sequence of traces, varying the dimensions of the space span provides a gamma density. This is the main result of this manuscript.

This result allows us to assign credence to clustering results regardless of the method used. To carry this out, we have established an error bound for the approximation error between the matrix of observations in the probabilistic space and the approximate factorization in the case of low-rank approximation.

Our proposal allows non-skilled humans to visualize the results in a fully unsupervised validation environment, achieved with a single plot of the adjusted gamma density. Additionally, in the context of Big Data and Computer Engineering, an interval of plausible estimations seems more advantageous. This result allows discussions in quantitative terms between different validation results. In practical situations, it allows the verification of whether the selection of observational or experimental criteria is correct.