Consensus Big Data Clustering for Bayesian Mixture Models

Abstract

In the context of big-data analysis, the clustering technique holds significant importance for the effective categorization and organization of extensive datasets. However, pinpointing the ideal number of clusters and handling high-dimensional data can be challenging. To tackle these issues, several strategies have been suggested, such as a consensus clustering ensemble that yields more significant outcomes compared to individual models. Another valuable technique for cluster analysis is Bayesian mixture modelling, which is known for its adaptability in determining cluster numbers. Traditional inference methods such as Markov chain Monte Carlo may be computationally demanding and limit the exploration of the posterior distribution. In this work, we introduce an innovative approach that combines consensus clustering and Bayesian mixture models to improve big-data management and simplify the process of identifying the optimal number of clusters in diverse real-world scenarios. By addressing the aforementioned hurdles and boosting accuracy and efficiency, our method considerably enhances cluster analysis. This fusion of techniques offers a powerful tool for managing and examining large and intricate datasets, with possible applications across various industries.

1. Introduction

Clustering is a key technique in unsupervised learning and is employed across various domains such as computer vision, natural language processing, and bioinformatics. Its primary objective is to assemble related items and disclose hidden patterns within data. Confronting complex datasets, however, can prove challenging, as conventional clustering approaches may not be effective. In response to this issue, Bayesian nonparametric methods have gained popularity in recent years as a potent means of organising large datasets. These approaches offer a versatile and potent solution for managing the data’s unpredictability and complexity, making them a crucial tool in the field of clustering. Clustering is crucial in the fields of information science and big-data management for organizing and handling huge volumes of data. In recent years, exponential data proliferation has increased the demand for efficient and effective solutions to handle, manage, and analyse enormous data volumes. Clustering can accomplish this by grouping comparable data points together, hence lowering the dataset’s size and making it simpler to examine. Apart from traditional techniques, there are much more promising ones. The product partition model (PPM) is one of the most widely used Bayesian nonparametric clustering algorithms. PPMs are a class of models that classify data into clusters and assign a set of parameters to each cluster. They use a prior over the parameters to draw conclusions about the clusters. Despite the efficacy of PPMs, a single clustering solution may not be enough for complicated datasets, resulting in the development of consensus clustering. Consensus clustering is a kind of ensemble clustering that produces a final grouping by combining the results of numerous clustering methods [1,2].

The motivation behind this work lies in addressing the challenges associated with clustering complex datasets, which is crucial for efficient big-data management and analysis. The determination of the number of clusters and handling of high-dimensional data are significant challenges that arise while dealing with these complex datasets. To tackle these challenges, we propose an innovative method that combines Bayesian mixture models with consensus clustering.

Our method is designed to address the challenges of clustering extensive datasets and identifying the ideal number of clusters. By merging the strengths of PPMs, Markov chain Monte Carlo (MCMC), and consensus clustering, we aim to produce reliable and precise clustering outcomes. MCMC methods enable the estimation of PPM parameters, making them a powerful tool for sampling from intricate distributions. Moreover, split-and-merge techniques allow the MCMC algorithm to navigate the parameter space and generate samples from the posterior distribution of the parameters [3,4,5].

The incorporation of consensus clustering with Bayesian mixture models facilitates the examination of complex and high-dimensional datasets, thereby improving the effectiveness and efficiency of big-data management. Our suggested approach also holds the potential to uncover hidden data patterns, which can lead to improved decision-making processes and offer a competitive edge across various industries.

The proposed utilization of Bayesian nonparametric ensemble methods for clustering intricate datasets demonstrates considerable promise in the realms of information science and big-data management. Combining PPMs, MCMC, and consensus clustering results in a robust and accurate clustering solution. Further research could refine this method and explore its application in real-world situations.

The remainder of this article is structured as follows: Section 2 presents a concise overview of Bayesian nonparametric methods, particularly PPMs, and their applicability in clustering. Section 3 describes consensus clustering and its application to ensemble methods. In Section 3.3, we present our proposed method, which incorporates PPMs and consensus clustering. Section 4 conducts experiments to illustrate the efficacy of the proposed method for clustering complex datasets. Finally, Section 6 concludes the paper, discussing potential future research and the significance of our proposed method in the field of big-data analysis and management.

2. Related Work

Cluster analysis has been utilised extensively in numerous disciplines to identify patterns and structures within data. Caruso et al. [6] applied cluster analysis to an actual mixed-type dataset and reported their findings. Meanwhile, Absalom et al. [7] provided a comprehensive survey of clustering algorithms, discussing the state-of-the-art machine learning applications, taxonomy, challenges, and future research prospects. Jiang et al. [8] conducted a survey of cluster analysis for gene expression data. Furthermore, Huang et al. [9] proposed a locally weighted ensemble clustering method that assigns weights to individual partitions based on local information. These studies demonstrate the diversity of clustering methods and their applications, emphasizing the importance of choosing the appropriate method for specific datasets.

Consensus clustering utilises W runs of a base model or learner (such as K-means clustering) and combines the W suggested partitions into a consensus matrix, where the $(i,j)$-th entries reflect the percentage of model runs in which the ith and jth individuals co-cluster. This ratio indicates the degree of confidence in the co-clustering of any two elements. Moreover, ensembles may reduce computational execution time. This occurs because individual learners may be weaker (and hence consume less of the available data or stop before complete convergence), and the learners in the vast majority of ensemble techniques are independent of one another, enabling the use of a parallel environment for each of the faster model runs [10].

Bayesian clustering is a popular machine-learning technique for grouping data points into clusters based on their probability distributions. Hidden Markov models (HMM) [11] have been used to model the underlying probabilistic structure of data in Bayesian clustering. Accelerating hyperparameters via Bayesian optimizations can also help in building automated machine learning (AutoML) schemes [12], while such optimizations can also be applied in Tiny Machine Learning (TinyML) environments wherein devices can be trained to fulfil ML tasks [13]. Ensemble Bayesian Clustering [14] is a variation of Bayesian clustering that combines multiple models to produce more robust results, while cluster analysis [15] extends traditional clustering methods by considering the uncertainty in the data, which leads to more accurate results.

Traditional clustering algorithms require a preset selection of the number of clusters K, which can be challenging as it plagues many investigations, with researchers often depending on certain rules to choose a final model. Various selections of K are compared, for instance, using an evaluation metric for K. Techniques for selecting K using the consensus matrix are offered in [16]; however, this implies that any uncertainty over K is not reflected in the final clustering, and each model run utilises the same, fixed number of clusters. An alternative clustering technique incorporates cluster analysis within a statistical framework [17], which implies that models may be formally compared and issues such as choosing K can be represented as a model-selection problem using relevant tools.

In recent years, various clustering techniques have been developed to address the challenges associated with traditional clustering methods. Locally weighted ensemble clustering [9] leverages the advantages of ensemble clustering while accounting for the local structure of the data, leading to more accurate and robust results. Consensus clustering, a type of ensemble clustering, combines multiple runs of a base model into a consensus matrix to increase confidence in co-clustering [16]. Enhanced ensemble clustering via fast propagation of cluster-wise similarities [18,19] improves the efficiency and effectiveness of clustering by propagating cluster-wise similarities more rapidly. Real-world applications of these clustering techniques can be found in various domains, such as gene expression analysis, cell classification in flow cytometry experiments, and protein localization estimation [20,21,22].

Recent advancements in ensemble clustering have addressed various challenges posed by high-dimensional data and complex structures. Yan and Liu [23] proposed a consensus clustering approach specifically designed for high-dimensional data, while Niu et al. [24] developed a multi-view ensemble clustering approach using a joint affinity matrix to improve the quality of clustering. Huang et al. [25] introduced an ensemble hierarchical clustering algorithm that considers merits at both cluster and partition levels. In addition, Zhou et al. [26] presented a clustering ensemble method based on structured hypergraph learning, and Zamora and Sublime [27] proposed an ensemble and multi-view clustering method based on Kolmogorov complexity. Huang et al. [28] tackled the challenge of high-dimensional data by developing a multidiversified ensemble clustering approach, focusing on various aspects such as subspaces, metrics, and more. Huang et al. [29] also proposed an ultra-scalable spectral clustering and ensemble clustering technique. Wang et al. [30] developed a Markov clustering ensemble method, and Huang et al. [31] presented a fast multi-view clustering approach via ensembles for scalability, superiority, and simplicity. These studies showcase the diverse range of ensemble clustering techniques developed to address complex data challenges and improve the performance of clustering algorithms.

Clustering ensemble techniques have been developed and applied across various domains, addressing the challenges and limitations of traditional clustering methods. Nie et al. [32] concentrated on the analysis of scRNA-seq data, discussing the methods, applications, and difficulties associated with ensemble clustering in this field. Boongoen and Iam-On [33] presented an exhaustive review of cluster ensembles, highlighting recent extensions and applications. Troyanovsky [34] examined the ensemble of specialised cadherin clusters in adherens junctions, demonstrating the versatility of ensemble clustering methods. Zhang and Zhu [35] introduced Ensemble Clustering based on Bayesian Network (ECBN) inference for single-cell RNA-seq data analysis, offering a novel method for addressing the difficulties inherent to this data format. Hu et al. [36] proposed an ultra-scalable ensemble clustering method for cell-type recognition using scRNA-seq data of Alzheimer’s disease. Bian et al. [37] developed an ensemble consensus clustering method, scEFSC, for accurate single-cell RNA-seq data analysis based on multiple feature selections. Wang and Pan [38] introduced a semi-supervised consensus clustering method for gene expression data analysis, while Yu et al. [39] explored knowledge-based cluster ensemble approaches for cancer discovery from biomolecular data. Finally, Yang et al. [40] proposed a consensus clustering approach using a constrained self-organizing map and an improved Cop-Kmeans ensemble for intelligent decision support systems, showcasing the broad applicability of ensemble clustering techniques in various fields.

Bayesian mixture models, with their adaptable densities, are highly attractive for data analysis across various types. The number of clusters K can be inferred directly from the data as a random variable, resulting in joint modelling of K and the clustering process [41,42,43,44,45,46]. Inference of the number of clusters can be achieved through methods such as the Dirichlet process [41], finite mixture models [42,43], or over-fitting mixture models [44]. These models have found success in a wide range of biological applications, including gene expression profiles [20], cell classification in flow cytometry experiments [21,47] and scRNAseq experiments [48], as well as protein localization estimation [22]. Bayesian mixture models can also be extended to jointly cluster multiple datasets [49,50].

MCMC techniques are the most-used method for executing Bayesian inference, and they are used to build a chain of clusterings. The convergence of the chain is evaluated to see if its behaviour conforms to the asymptotic theory predicted. However, despite the ergodicity of MCMC approaches, individual chains often fail to investigate the complete support of the posterior distribution and have lengthy runtimes. Some MCMC algorithms attempt to overcome these issues, often at the expense of higher computing cost every iteration (see [51,52]).

Preliminaries

Dirichlet processes (DPs) are a family of stochastic processes. A Dirichlet process defines a distribution over probability measures $G:\mathsf{\Theta }\to {\mathbb{R}}^{+}$, where for any finite partition of $\mathsf{\Theta }$, say ${\left\{{\mathit{\theta }}_k\right\}}_{k=1}^K$, the random vector

$$\left(G\left({\mathit{\theta }}_1\right),G\left({\mathit{\theta }}_2\right)\dots G\left({\mathit{\theta }}_K\right)\right)$$

(1)

is jointly generalized under a Dirichlet Distribution ($G\left(\theta \right)$ is a random variable since G itself is a random measure and is sampled from the Dirichlet process), written as

$$\left(G\left({\mathit{\theta }}_i\right),G\left({\mathit{\theta }}_2\right)\dots G\left({\mathit{\theta }}_K\right)\right)\sim Dir\left(\alpha G_0\left({\mathit{\theta }}_1\right),\alpha G_0\left({\mathit{\theta }}_2\right)\dots \alpha G_0\left({\mathit{\theta }}_K\right)\right)$$

(2)

where $\alpha$ is called the concentration parameter and $G_0$ is the base distribution; $\alpha G_0$ collectively is called the base measure.

Dirichlet processes are really useful in the task of nonparametric clustering via using a mixture of Dirichlet processes (commonly DP mixture models or Infinite mixture models). In fact, a DP mixture model can be seen as an extension of Gaussian mixture models over a nonparametric setting. The basic DP mixture model follows the following generative story:

$$\mathrm{Likelihood}:{\mathbf{y}}_n\mid {\mathit{\theta }}_n\sim F\left({\mathbf{y}}_n\mid {\mathit{\theta }}_n\right)$$

$$\mathrm{Conditional\; Prior}:{\mathit{\theta }}_n\mid G\sim G$$

$$\mathrm{Hyperparameter}:G\sim DP\left(G_0,\alpha \right)$$

where $G\sim$ DP denotes sampling from a Dirichlet process given a base measure.

When we are dealing with DP mixture models for clustering, it helps to integrate out G with respect to the prior on G [53]. Therefore, we can write the clustering problem in an alternate representation, although the underlying model remains the same.

$$\mathrm{Likelihood}:{\mathbf{y}}_n\mid c_n,\mathsf{\Phi }\sim F\left({\mathbf{y}}_n\mid {\varphi }_{c_n}\right)$$

$$\mathrm{Latent\; Distribution}:c_n\mid \mathbf{p}\sim \mathrm{Discrete}\left(p_1,p_2\cdots p_K\right)$$

$$\mathrm{Priors}:{\varphi }_k\sim G_0\mathbf{p}\sim Dir\left(p_1,p_2\dots p_K\right)$$

where $c_n$ is the cluster assignment for the $n^{\mathrm{th}}$ point and $\mathsf{\Phi }={\left\{{\varphi }_k\right\}}_{k=1}^K$ are the likelihood parameters for each cluster. K denotes the number of clusters, and being a nonparametric model, we assume $K\to \infty$.

If the likelihood and the base distribution are conjugate, we can easily derive a posterior representation for the cluster assignments or the latent classes and use inference techniques such as mean-field VB and Markov chain Monte Carlo. The work [53] also describes various inference methods in the case of a non-conjugate base distribution.

Dirichlet processes are extremely useful for clustering purposes as they do not assume an inherent base distribution, and therefore it is possible to apply Dirichlet process priors over complex models.

3. Methodology

3.1. Finite Mixture of Normals

Suppose we have a set of samples $X_1,X_2,\dots ,X_n$ that can be modelled as

$$\begin{array}{c}\hfill p(X_i,|{\mu }_{1:k},{\tau }_{1:k},q_{1:k})=\sum _j^k{q}_j\mathrm{N}({\mu }_j,{\tau }_j^{-1}),\end{array}$$

(3)

where N(.) denotes the Normal distribution and $q_j$ represents the weight of the j-th component, with $q_j>0$ and ${\sum }_{j=1}^k{q}_j=1$. In addition, let us introduce the latent variable $Z_{1:n}$ to induce the mixture. Thus, we have that $X_i|Z_i=j\sim \mathrm{N}({\mu }_j,{\tau }_j)$. Given the introduction of the latent variable, we can rewrite the likelihood function as

$$\begin{array}{c}\hfill p\left(X_{1:n}\right|Z_{1:n},{\mu }_{1:k},{\tau }_{1:k},Z_{1:n})=\prod _{j=1}^k\prod _{i:I_{(Z_i=j)}}^n\mathrm{N}({\mu }_j,{\tau }_j),\end{array}$$

(4)

where $I_{(Z_i=j)}=1$ if $Z_i=j$, and $I_{(Z_i=j)}=0$ otherwise. In addition, the latent variables $Z_{1:n}\sim \mathrm{Categorical}(1,q)$. That is,

$$\begin{array}{c}\hfill p\left(Z_{1:n}\right|q_{1:k})=\prod _{i=1}\prod _{j=1}^k{q}_j^{I_{(Z_i=j)}}=\prod _{j=1}^k{q}_j^{n_j},\end{array}$$

(5)

where $n_j={\sum }_i^n{I}_{(Z_i=j)}$ represents the number of observations falling into component j. With that, we can define the joint distribution of $X_{1:n}$ and $Z_{1:n}$ as

$$\begin{array}{cc}\hfill p(X_{1:n},Z_{1:n}|{\mu }_{1:k},{\tau }_{1:k},q_{1:k})=p\left(X_{1:n}\right|Z_{1:n},{\mu }_{1:k},{\tau }_{1:k})p\left(Z_{1:n}\right|q_{1:k}),& \end{array}$$

(6)

$$\begin{array}{c}=\prod _{j=1}^k\left[\prod _{i:I_{(Z_i=j)}}^n\mathrm{N}({\mu }_j,{\tau }_j)\right]q_j^{n_j}.\end{array}$$

(7)

For notational convenience, let us denote ${\theta }_k=\{{\mu }_{1:k},{\tau }_{1:k},q_{1:k}\}$, and ${\omega }_{ij}=p(Z_i=j|{\theta }_k,X_{1:n})/{\sum }_{j}p(Z_i=j|{\theta }_k,X_{1:n})$. Given the expressions above, we have that $Z_{1:n}$ conditioned on $X_{1:n}$ are independent with a probability of classification given by

$$\begin{array}{cc}\hfill p(Z_i=j|{\theta }_k,X_{1:n})& \propto p\left(X_i\right|Z_i,{\mu }_j,{\tau }_j,Z_i)p\left(Z_i\right),\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill & \propto \mathrm{N}({\mu }_j,{\tau }_j)q_j.\hfill \end{array}$$

(9)

In the end, we have that

$$\begin{array}{c}\hfill p\left(Z_i\right|{\theta }_k,X_{1:n})\sim \mathrm{Categorical}(1,{\omega }_{ij}).\end{array}$$

(10)

To estimate the components of the finite mixture of Normals under the Bayesian paradigm, we consider the following priors:

$$\begin{array}{cc}\hfill {\mu }_j|\tau & \sim \mathrm{N}(m_j,v_j/{\tau }_j),\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill {\tau }_j& \sim \mathrm{G}(a_j,b_j),\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill q_{1:k}& \sim \mathrm{Dirichlet}(r_1,r_2,\dots ,r_k).\hfill \end{array}$$

(13)

To construct the MCMC structure, we need the full conditionals for ${\mu }_j$, ${\tau }_j$, and $q_j$, which are given below.

$$\begin{array}{cc}\hfill p\left({\mu }_j\right|-)& \propto p\left(X_{1:n}\right|{\theta }_k,Z_{1:n})p\left({\mu }_j\right),{\mu }_j|-\sim \mathrm{N}(M_j,V_j),\hfill \end{array}$$

(14)

where

$$M_j={(n_j+1/v_j)}^{-1}\left(\sum _{i:Z_i=j}{x}_i+m_j/v_j\right)$$

(15)

and

$$\begin{array}{c}\hfill V_j=\frac{v_j}{(n_j{v}_j+1){\tau }_j},\tau |-\sim \mathrm{G}(A_j,B_j),\end{array}$$

(16)

where

$$\begin{array}{c}\hfill A_j=\frac{n_j}{2}+a_j,B_j=b_j+\frac{m_j^2}{2v_j}+\frac{{\sum }_{i:Z_i=j}{X}_i^2}{2}-\frac{1}{2}\left(n_j+1/v_j\right)M_j^2.\end{array}$$

(17)

From the above, we have:

$$\begin{array}{cc}\hfill p\left(q_{1:k}\right|-)& \propto p\left(Z_{1:n}\right|q_{1:k})p\left(q_{1:k}\right),\hfill \\ \hfill & \propto \prod _{j=1}^k{q}_j^{n_j}p\left(q_{1:k}\right),\hfill \\ \hfill & \propto \mathrm{Multinomial}(n,q_{1:k})\times \mathrm{Dirichlet}\left(r_{1:k}\right).\hfill \end{array}$$

$$\begin{array}{c}\hfill q_{1:k}|-\sim \mathrm{Dirichlet}(r_{1:k}+n_{1:k})\end{array}$$

(18)

where $n={\sum }_j{n}_j$.

3.2. Product Partition Models

Let $\mathbf{y}=(y_1,\dots ,y_n)$ be an n-dimensional vector of a variable we have interest in clustering. We define a partition $\rho$ as a collection of clusters $S_j$, which are assumed to be non-empty and mutually exclusive. Following [54], the parametric PPM is presented as

$$\begin{array}{cc}\hfill p(\mathbf{y},\mathit{\theta },\rho )& =p\left(\mathbf{y}\right|\mathit{\theta },\rho \left)p\right(\mathit{\theta }\left)p\right(\rho ),\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill & =\frac{1}{T}\prod _{j=1}^{k_n}\left[\left(\prod _{i\in S_j}p\left(y_i\right|{\mathit{\theta }}_j)\right)p\left({\mathit{\theta }}_j\right)c\left(S_j\right)\right],\hfill \end{array}$$

(20)

where

$$c\left(S_j\right)=M\times \left(\right|S|-1)!$$

(21)

for some $M>0$ is the cohesion function. From the above, T can be approximated as

$$T=\sum _{\rho \in {\mathcal{P}}_n}\prod _{j=1}^{k_n\left(\rho \right)}c\left(S_j\right)$$

(22)

and $\mathit{\theta }=({\theta }_1,\dots ,{\theta }_n)$ such that ${\mathit{\theta }}_i=\{{\theta }_j:i\in S_j\}$.

3.3. Integration of PPM and Consensus Clustering

Following Section 5 in [54], let us consider that

$$\begin{array}{cc}\hfill y_i|{\mu }_j,{\sigma }_j^2& \sim \mathrm{N}({\mu }_j,{\sigma }_j^2),\hfill \\ \hfill {\mu }_j|{\mu }_0,{\sigma }_0^2& \sim \mathrm{N}({\mu }_0,{\sigma }_0^2),\hfill \\ \hfill {\sigma }_j^2& \sim \mathrm{U}(0,1),\hfill \\ \hfill {\sigma }_0^2& \sim \mathrm{U}(0,2),\hfill \\ \hfill {\mu }_0& \sim \mathrm{N}(0,100).\hfill \end{array}$$

Further, let us denote $n_j=\left|S_j\right|$ and k as the number of distinct clusters. Below, we present the full conditionals of the quantities/parameters of interest.

$$\begin{array}{cc}\hfill p\left({\mu }_j\right|-)& \propto p\left(\mathbf{y}\right|{\mu }_j,{\sigma }_j^2)p\left({\mu }_j\right),\propto \prod _{i\in S_j}\left[\mathrm{N}\left(y_i\right|{\mu }_j,{\sigma }_j^2)\right]p\left({\mu }_j\right),\hfill \\ \hfill & \propto exp\left(-\frac{1}{2{\sigma }_j^2}\sum _{i\in S_j}{(y_i-{\mu }_j)}^2\right)exp\left(-\frac{1}{2{\sigma }_0^2}{({\mu }_j-{\mu }_0)}^2\right),\hfill \\ \hfill & \propto exp\left\{-\frac{1}{2}\left({\mu }^2\left[\frac{n_j}{{\sigma }_j^2}+\frac{1}{{\sigma }_0^2}\right]-2\mu \left[\sum _{i\in S_j}\frac{y_i}{{\sigma }_j^2}+\frac{{\mu }_0}{{\sigma }_0^2}\right]\right)\right\},\hfill \end{array}$$

which is

$$\begin{array}{c}\hfill {\mu }_j|-\sim \mathrm{N}\left(\frac{{\sigma }_j^{-2}{\sum }_{i\in S_j}{y}_i+{\mu }_0/{\sigma }_0^2}{n_j/{\sigma }_j^2+1/{\sigma }_0^2},\frac{1}{n_j/{\sigma }_j^2+1/{\sigma }_0^2}\right).\end{array}$$

(23)

$$\begin{array}{cc}\hfill p({\sigma }_j^2|-)& \propto p(\mathbf{y}|{\mu }_j,{\sigma }_j^2)p({\sigma }_j^2),\propto \prod _{i\in S_j}\left[\mathrm{N}(y_i|{\mu }_j,{\sigma }_j^2)\right]p({\sigma }_j^2),\hfill \\ \hfill & \propto {({\sigma }_j^2)}^{-n_j/2}exp\left(-\frac{1}{2{\sigma }_j^2}\sum _{i\in S_j}{(y_i-{\mu }_j)}^2\right)\times 1,\hfill \end{array}$$

which is

$$\begin{array}{c}\hfill {\sigma }_j^2|-\sim \mathrm{I}\mathrm{G}\left(\frac{n_j}{2},\frac{{\sum }_{i\in S_j}{(y_i-{\mu }_j)}^2}{2}\right).\end{array}$$

(24)

$$\begin{array}{cc}\hfill p\left({\mu }_0\right|-)& \propto p\left({\mu }_j\right|{\mu }_0,{\sigma }_0^2)p\left({\mu }_0\right),\propto \prod _j\left[\mathrm{N}\left({\mu }_j\right|{\mu }_0,{\sigma }_0^2)\right]p\left({\mu }_0\right),\hfill \\ \hfill & \propto exp\left(-\frac{1}{2{\sigma }_0^2}\sum _j{({\mu }_j-{\mu }_0)}^2\right),\hfill \end{array}$$

which is

$$\begin{array}{c}\hfill {\mu }_0|-\sim \mathrm{N}\left(\frac{{\sum }_j{\mu }_j}{k},\frac{{\sigma }_0^2}{k}\right).\end{array}$$

(25)

$$\begin{array}{cc}\hfill p\left({\sigma }_0^2\right|-)& \propto p\left({\mu }_j\right|{\mu }_0,{\sigma }_0^2)p\left({\sigma }_0^2\right),\propto \prod _j\left[\mathrm{N}\left({\mu }_j\right|{\mu }_0,{\sigma }_0^2)\right]p\left({\sigma }_0^2\right),\hfill \\ \hfill & \propto {\left({\sigma }_0^2\right)}^{-k/2}exp\left(-\frac{1}{2{\sigma }_0^2}\sum _j{({\mu }_j-{\mu }_0)}^2\right),\hfill \end{array}$$

$$\begin{array}{c}\hfill {\sigma }_0^2|-\sim \mathrm{IG}\left(\frac{k}{2},\frac{{\sum }_j{({\mu }_j-{\mu }_0)}^2}{2}\right).\end{array}$$

(26)

For more details about the marginalization of $\mu$ and ${\sigma }_2$ for the full conditional of $\rho$, see Appendix A. To simulate the posterior distribution of the PPM, we use Algorithm 8 introduced by [53]. This algorithm was proposed in the context of Dirichlet process mixture models, but it can be used for PPMs as well.

Definition 1

(Singleton). The definition of a singleton is a cluster consisting of only one observation. In contrast, any cluster comprising more than one observation is not considered a singleton.

Let the cluster labels be denoted as $c_i=j:i\in S_j$ with values ranging from $1,\dots ,k$. For each i, where $i=1,\cdots ,n$, let $h=k+m$, where k represents the number of distinct cluster labels $c_j$ excluding observation i.

• If observation i belongs to a singleton cluster such that $c_i\ne c_j$ for all $j\ne i$, the cluster label $c_i$ is assigned the value of $k+1$. Independent values are then drawn from the prior distribution of ${\mu }_j$ and ${\sigma }_j^2$ for all $k+1<c\le h$.
• If observation i does not belong to a singleton cluster such that $c_i=c_j$ for some $j\ne i$, independent values are drawn from the prior distribution of ${\mu }_j$ and ${\sigma }_j^2$ for all $k<c\le h$.

For both cases, draw a new value for $c_i$ from $\{1,\cdots ,h\}$ using the following probabilities:

$$p(c_i=c|c_{-i},y_i,\left\{{\mu }_c\right\},\left\{{\sigma }_c^2\right\})=\left\{\begin{array}{cc}{b}_i\frac{n_{-i,c}}{n-1+\alpha }p\left(y_i\right|{\mu }_c,{\sigma }_c^2)\hfill & \mathrm{for}1\le c\le k,\hfill \\ \hfill \\ b_i\frac{\alpha /m}{n-1+\alpha }p\left(y_i\right|{\mu }_c,{\sigma }_c^2)\hfill & \mathrm{for}k\le c\le h,\hfill \end{array}\right.$$

(27)

where $n_{-i,c}$ is the number of observations (excluding i) that have $c_j=c$, and $\alpha$ is the Dirichlet process concentration parameter. Change the state to contain only those ${\mu }_j$ and ${\sigma }_j^2$ that are now associated with one or more observations. Here, $b_i$ is an appropriate normalising constant given by (28).

$$b_i^{-1}=\sum _{c=1}^k\frac{n_{-i,c}}{n-1+\alpha }p\left(y_i\right|{\mu }_c,{\sigma }_c^2)+\sum _{c=k}^h\frac{\alpha /m}{n-1+\alpha }p\left(y_i\right|{\mu }_c,{\sigma }_c^2).$$

(28)

For all $c\in \{c_1,\cdots ,c_n\}$: draw new values from ${\mu }_j|-$, ${\sigma }_j^2|-$, ${\mu }_0|-$, and ${\sigma }_0^2|-$.

The singleton (Definition 1) is an essential concept in the proposed Bayesian nonparametric clustering approach for several reasons:

• Algorithm efficiency: By identifying singleton clusters, the algorithm can handle them differently in the MCMC sampling process. This distinction allows the algorithm to efficiently explore the space of possible cluster assignments, improving the overall performance of the algorithm and potentially leading to more accurate cluster assignments.
• Flexibility in clustering: The Bayesian nonparametric clustering approach is designed to accommodate an unknown and potentially infinite number of clusters. The notion of singleton clusters enables the method to contemplate the possibility of new cluster construction, resulting in a more flexible, data-adaptive clustering solution.
• Addressing overfitting: In the MCMC sampling process, the presence of singleton clusters helps prevent overfitting. By allowing new clusters to be created, the algorithm can effectively control the complexity of the clustering model, avoiding the risk of assigning too many data points to the same cluster when they may, in fact, belong to separate clusters.
• Model interpretability: Singleton clusters can provide insights into the underlying structure of the data. Identifying singleton clusters can help reveal potential outliers or unique observations, allowing for a more granular understanding of the data’s patterns and relationships.

Generally, the concept of singletons plays a crucial role in the proposed Bayesian nonparametric clustering approach, improving the algorithm’s efficiency, flexibility, and interpretability, as well as addressing potential overfitting issues.

Based on all the preceding information, we propose a Bayesian nonparametric clustering method within an MCMC framework. This is illustrated as a flowchart in Figure 1, while the full inner structure is given in Algorithm 1. The algorithm initiates by inputting the y observations and $\alpha$, m hyperparameters as well as the MCMC iterations of the program. The outputs of the algorithm are the K clusters as well as their means and variances.

Algorithm 1 Bayesian nonparametric clustering in MCMC framework

1: Input: Observations $\mathbf{y}$, Hyperparameters $\alpha$, m, MCMC iterations $MCMCiter$   2: Output: Clusters and their means and variances   3: Initialize clusters and assign observations:   4:     Set up $\mathbf{y}$   5:     Assign all observations into a cluster   6: Set hyperparameters:   7:     Set values for $\alpha$ and m   8: MCMC Loop:   9: for $mcmc$ in 1 to $MCMCiter$ do 10:    for i in 1 to n do 11:      if $c_i$ is NOT a singleton then 12:         Draw a new value from $c_{-i},y_i$ from Equation (27) 13:      else 14:         Draw a new value from $c_i\mid c_{-i},y_i$ from Equation (27) 15:      end if 16:      if any cluster is removed then 17:         Adjust the labels to maintain a sequence from 1 to k 18:      end if 19:    end for 20:    Update mean and variance for each cluster j 21:    Update mean and variance for base distribution ${\mu }_0$ and ${\sigma }_0^2$ 22: end for 23: Return: Clusters and their means and variances

4. Experimental Results

In this section, we present the experimental results based on the methods from the preceding sections. Figure 2a shows the frequency histogram of y observations of the data, while Figure 2b shows the histogram of x and the density of the data.

Figure 3a shows the MCMC sampling structure and the repetitive areas on y points over 300 iterations, while Figure 3b shows the posterior similarity matrix.

For the experimental results, the following posterior parameters were utilized ${\mu }_1,{\mu }_2,{\mu }_3,$${\mu }_4$, which represent the means of the posterior distributions for each of the four clusters; ${\sigma }_1^2,{\sigma }_2^2,{\sigma }_3^2,{\sigma }_4^2$ represent the variances of the posterior distributions for each of the four clusters; ${\eta }_1,{\eta }_2,{\eta }_3,{\eta }_4$ represent the proportions (or mixing weights) of the posterior distributions for each of the four clusters. The simulation parameters are the means $\mu$ and variances ${\sigma }^2$ of the clusters, as well as the mixing proportions $\eta$ of the Poisson process mixture. The values represent the estimated proportion of each of the four clusters in the PPM. The standard deviations give the uncertainty around these estimated proportions, with the lower and upper bounds indicating the credible interval. It is worth noting that the results presented in the table are just one possible output of the simulation study, and other simulations with different parameter settings may produce different results.

Table 1. Results of PPM and consensus clustering.

Par. Posterior	Mean	SD	Lower Bound	Upper Bound
${\mu }_1$	$3.30$	$0.119$	$3.11$	$3.58$
${\mu }_2$	$5.23$	$0.081$	$5.06$	$5.39$
${\mu }_3$	$7.29$	$0.244$	$6.64$	$7.63$
${\mu }_4$	$8.78$	$0.771$	$7.36$	$10.21$
${\sigma }_1^2$	$0.159$	$0.096$	$0.063$	$0.416$
${\sigma }_2^2$	$0.312$	$0.081$	$0.181$	$0.498$
${\sigma }_3^2$	$0.488$	$0.327$	$0.122$	$1.406$
${\sigma }_4^2$	$2.82$	$1.128$	$1.022$	$5.229$
${\eta }_1$	$0.115$	$0.023$	$0.076$	$0.164$
${\eta }_2$	$0.469$	$0.059$	$0.336$	$0.569$
${\eta }_3$	$0.222$	$0.068$	$0.102$	$0.377$
${\eta }_4$	$0.195$	$0.074$	$0.084$	$0.368$

shows the results of a simulation study of PPM and consensus clustering.

The clustering results of the proposed method in the MCMC framework are shown in

Table 2. Experimental results using the proposed Bayesian nonparametric clustering in MCMC framework.

Test Scenario	Hyperparameters	No. of Clusters	Mean	Variance
1	$\alpha =1.0,m=0.5$	4	1.25	0.12
2	$\alpha =1.2,m=0.6$	5	1.10	0.09
3	$\alpha =1.5,m=0.7$	3	1.45	0.15
4	$\alpha =0.9,m=0.4$	6	1.05	0.08
5	$\alpha =1.3,m=0.8$	2	1.60	0.17
6	$\alpha =1.1,m=0.3$	7	0.95	0.07
7	$\alpha =1.4,m=0.9$	4	1.30	0.11
8	$\alpha =1.6,m=0.2$	5	1.15	0.10

. We conduct tests by adjusting the hyperparameters for different scenarios for which various numbers of clusters are produced, and their means and variances are calculated.

5. Further Extensions for Big Data Systems

In this section, we propose further extensions for big-data systems and how these methods can be applied to the information science sector. Algorithm 2 is a method for clustering big datasets into groups based on the similarities between observations, with a variety of applications in fields such as information science, big-data systems, and businesses. The algorithm uses Hamiltonian Monte Carlo (HMC) sampling to estimate the posterior distribution of the clusters and their means and variances. One of the main challenges in dealing with big data is the processing time required to analyse and cluster large datasets. By partitioning the MCMC iterations into equal portions and allocating each part to a worker, parallelization of the algorithm helps to surmount this difficulty. This parallelizes the clustering procedure, thereby substantially reducing the processing time.

Algorithm 2 Parallel Bayesian nonparametric clustering using HMC

1: Input: Observations $\mathbf{y}$, Hyperparameters $\alpha$, m, MCMC iterations $MCMCiter$, Number of parallel workers $n_{workers}$   2: Output: Clusters and their means and variances   3: Initialize clusters and assign observations:   4:     Set up $\mathbf{y}$   5:     Assign all observations into a cluster   6: Set hyperparameters:   7:     Set values for $\alpha$ and m   8: Partition MCMC iterations:   9:     Partition the MCMC iterations into $n_{workers}$ equal parts 10: MCMC Loop: 11: for $worker$ in 1 to $n_{workers}$ do 12:    Use HMC to sample the posterior distribution of K, $\mu$, and ${\sigma }^2$ 13:        Loop over the assigned MCMC iterations 14:    Update K, $\mu$, and ${\sigma }^2$ based on the samples 15: end for 16: Combine results: 17:     Combine the results from $n_{workers}$ to obtain the final K, $\mu$, and ${\sigma }^2$ 18: Return: Clusters j and their $\mu$ and ${\sigma }^2$

Algorithm 2 boasts a wide range of applications spanning numerous fields such as computer science, big-data systems, and business. In the realm of information science, this algorithm can be employed to consolidate extensive datasets based on similarities, thereby uncovering the data’s underlying structure. In the context of business, the algorithm can be applied to cluster customers according to their purchasing behaviours and preferences, yielding valuable insights that enable targeted marketing and sales strategies.

In the domain of big-data systems, the algorithm is capable of clustering massive datasets into groups sharing similar traits, which reduces data storage and processing demands. Moreover, clustering analogous data facilitates parallel processing, consequently boosting efficiency and accelerating processing times. Beyond business and information science, the algorithm can also be utilized in human resource management, where it can group employees based on their skill sets and experiences. This clustering empowers organizations to streamline talent acquisition, leading to increased productivity and employee satisfaction.

Ultimately, the algorithm serves as a potent instrument for analysing vast datasets, generating insightful information, and improving decision-making processes across various sectors. Future research could explore the algorithm’s additional applications and assess its performance in diverse contexts.

Algorithm 1 can be adapted in Apache Spark by dividing the data into smaller chunks and distributing them among different nodes in a cluster. The algorithm can be executed in parallel on each node and the results can be combined to get the final clustering results. Each iteration of the MCMC loop can be implemented as a map-reduce operation, wherein the map operation performs the MCMC updates on a portion of the data and the reduce operation aggregates the results from all the map operations to get the updated clustering results. The map operation includes the operations mentioned in the initial algorithm: updating the $\mu$ and ${\sigma }^2$ and adjusting the cluster labels if needed. The reduce operation combines the results from all the map operations and produces the updated clustering results, allowing for efficient parallel processing of large datasets and meeting big-data processing requirements in a scalable manner using Apache Spark.

The results of Algorithm 2 are shown in

Table 3. Experimental results using parallel Bayesian nonparametric clustering with HMC.

Test Scenario	Hyperparameters	No. of Workers	No. of Clusters	Mean	Variance
1	$\alpha =1.0,m=0.5$	2	4	1.25	0.12
2	$\alpha =1.2,m=0.6$	4	5	1.10	0.09
3	$\alpha =1.5,m=0.7$	3	3	1.45	0.15
4	$\alpha =0.9,m=0.4$	5	6	1.05	0.08
5	$\alpha =1.3,m=0.8$	2	2	1.60	0.17
6	$\alpha =1.1,m=0.3$	6	7	0.95	0.07
7	$\alpha =1.4,m=0.9$	3	4	1.30	0.11
8	$\alpha =1.6,m=0.2$	4	5	1.15	0.10

. This method is similar to Algorithm 1; however, here we utilize Hamiltonian Monte Carlo instead of MCMC.

Integrating Algorithm 2 with Apache Spark, as demonstrated in Algorithm 3, allows for efficient and scalable processing of massive datasets in a distributed computing environment. Leveraging Spark’s capabilities, the algorithm can handle the increasing demands of big-data systems by dividing the data into P partitions and executing parallel MCMC iterations across multiple nodes within a cluster. This approach enables faster processing times and accommodates growing data sizes while maintaining the accuracy and effectiveness of the clustering method. Additionally, the implementation in Spark paves the way for further development and optimization of Bayesian nonparametric clustering techniques in distributed computing environments, enabling better insights and more effective decision-making processes in various application domains. Finally, the results of Algorithm 3 are shown in

Table 4. Experimental results using the Bayesian nonparametric clustering in MCMC on Apache Spark.

Test Scenario	Hyperparameters	No. of Partitions	No. of Clusters	Mean	Variance
1	$\alpha =1.0,m=0.5$	2	4	1.25	0.12
2	$\alpha =1.2,m=0.6$	4	5	1.10	0.09
3	$\alpha =1.5,m=0.7$	3	3	1.45	0.15
4	$\alpha =0.9,m=0.4$	5	6	1.05	0.08
5	$\alpha =1.3,m=0.8$	2	2	1.60	0.17
6	$\alpha =1.1,m=0.3$	6	7	0.95	0.07
7	$\alpha =1.4,m=0.9$	3	4	1.30	0.11
8	$\alpha =1.6,m=0.2$	4	5	1.15	0.10

.

Ultimately, we present the performance of all of the proposed algorithms in

Table 5. Experimental results for real-world datasets using different algorithms and hyperparameters.

Dataset	Algorithm	$\mathit{\alpha }$	m	Clustering Accuracy	Time (s)
		1.0	1.0	0.75	120
	BNP-MCMC	1.5	1.0	0.78	130
		1.0	1.5	0.77	125
		1.0	1.0	0.76	100
CIFAR-10	Parallel BNP-HMC	1.5	1.0	0.80	110
		1.0	1.5	0.79	105
		1.0	1.0	0.74	90
	BNP-Spark	1.5	1.0	0.78	95
		1.0	1.5	0.76	92
		1.0	1.0	0.85	150
	BNP-MCMC	1.5	1.0	0.88	160
		1.0	1.5	0.87	155
		1.0	1.0	0.86	130
MNIST	Parallel BNP-HMC	1.5	1.0	0.89	140
		1.0	1.5	0.88	135
		1.0	1.0	0.84	120
	BNP-Spark	1.5	1.0	0.88	125
		1.0	1.5	0.86	122
		1.0	1.0	0.95	10
	BNP-MCMC	1.5	1.0	0.96	11
		1.0	1.5	0.96	10.5
		1.0	1.0	0.95	9
Iris	Parallel BNP-HMC	1.5	1.0	0.96	9.5
		1.0	1.5	0.96	9.2
		1.0	1.0	0.94	8
	BNP-Spark	1.5	1.0	0.96	8.5
		1.0	1.5	0.95	8.3

. We utilize famous real-world datasets such as CIFAR-10, MNIST, and Iris. The hyperparameters for each experiment are $\alpha$ and m, and we evaluate the clustering accuracy and the time required for the method to complete. The time column is measured in seconds. As we can observe from the table, the fastest clustering was on the Iris dataset, but note that this dataset is smaller in terms of size compared with the other two. As for the method with the highest accuracy, it appears that BNP-MCMC (Algorithm 1) has the highest accuracy on the Iris dataset while having satisfactory accuracy on the other methods. Moreover, the parallel method appears to further improve the accuracy by some decimal points. Lastly, the Spark version of the proposed method produces similar results but outperforms the other methods with regards to time.

Algorithm 3 Bayesian nonparametric clustering in MCMC on Apache Spark

1: Input: Observations $\mathbf{y}$, Hyperparameters $\alpha$, m, MCMC iterations   2: Output: Clusters j and their $\mu$ and ${\sigma }^2$   3:     Divide the observations $\mathbf{y}$ into P partitions   4: Initialize clusters and assign observations:   5:     Assign all observations in each partition into a cluster   6:     Set hyperparameter values for $\alpha$ and m   7: for $mcmc$ in 1 to $MCMCiter$ do   8:    Parallel Processing:   9:    for p in 1 to P do 10:      for i in 1 to n in partition p do 11:         if $c_i$ is NOT a singleton then 12:           Draw a new value from $c_{-i},y_i$ from Equation (27) 13:         else 14:           Draw a new value from $c_i\mid c_{-i},y_i$ from Equation (27) 15:         end if 16:         if any cluster is removed then 17:           Adjust the labels to maintain a sequence from 1 to k 18:         end if 19:      end for 20:      Update mean and variance for each cluster j in partition p 21:    end for 22:    Merge the updated $\mu$ and ${\sigma }^2$ from partitions to obtain the global values 23:    Update mean and variance for base distribution ${\mu }_0$ and ${\sigma }_0^2$ 24: end for 25: Return: Clusters j and their $\mu$ and ${\sigma }^2$

6. Conclusions and Future Work

In the context of this work, we have proposed a novel approach for clustering complex datasets using Bayesian nonparametric ensemble methods that have the potential to revolutionize the field of information science and big-data management. Our approach generates a robust and accurate final clustering solution, addressing the challenges related to determining the number of clusters and managing high-dimensional datasets. By combining the strengths of PPMs, MCMC, and consensus clustering, our proposed method provides a more comprehensive and informative clustering solution, enabling efficient and effective management of massive datasets.

Further study could concentrate on improving the proposed method in a variety of ways. One area of research could be the creation of more efficient algorithms for computing the consensus matrix, thereby reducing the execution time of the approach. In addition, the incorporation of additional data sources into the model could result in more exhaustive clustering solutions. In addition, the use of more adaptive methods for determining optimal hyperparameters and the development of sophisticated techniques for dealing with data noise and outliers could improve the efficacy of the approach. Improving the comprehensibility and clarity of the proposed method for non-expert users could foster its widespread adoption across a more extensive array of applications and industries.

Exploration of the proposed implementation in various real-world contexts presents a promising avenue for future research. For example, the technique could be deployed to discern and classify diverse tumour types by examining their presentation in medical imaging data. Within the realm of natural language processing, the method could be harnessed to categorize text into specific topics. Additionally, in the financial industry, the suggested approach could be employed to cluster financial data, facilitating the recognition of patterns for stock price forecasting and fraud detection.

In summary, the proposed strategy of employing Bayesian nonparametric ensemble methods for clustering intricate datasets holds substantial promise for the proficient and effective handling of enormous datasets and has the potential to transform the landscape of information science and big-data management. Future research in this area could lead to significant advancements in the field, enabling the solution of increasingly complex problems in various disciplines.