Revisiting Possibilistic Fuzzy C-Means Clustering Using the Majorization-Minimization Method

Abstract

Possibilistic fuzzy c-means (PFCM) clustering is a kind of hybrid clustering method based on fuzzy c-means (FCM) and possibilistic c-means (PCM), which not only has the stability of FCM but also partly inherits the robustness of PCM. However, as an extension of FCM on the objective function, PFCM tends to find a suboptimal local minimum, which affects its performance. In this paper, we rederive PFCM using the majorization-minimization (MM) method, which is a new derivation approach not seen in other studies. In addition, we propose an effective optimization method to solve the above problem, called MMPFCM. Firstly, by eliminating the variable $V\in R^{p\times c}$, the original optimization problem is transformed into a simplified model with fewer variables but a proportional term. Therefore, we introduce a new intermediate variable $s\in R^c$ to convert the model with the proportional term into an easily solvable equivalent form. Subsequently, we design an iterative sub-problem using the MM method. The complexity analysis indicates that MMPFCM and PFCM share the same computational complexity. However, MMPFCM requires less memory per iteration. Extensive experiments, including objective function value comparison and clustering performance comparison, demonstrate that MMPFCM converges to a better local minimum compared to PFCM.

1. Introduction

Cluster analysis is one of the most important topics in machine learning [1], and it has been widely applied in various fields, such as data mining [2], image processing [3], and pattern recognition [4]. Clustering algorithms are a type of unsupervised learning method, aiming to partition datasets into multiple clusters based on similarity measures, ensuring that samples within the same cluster are similar [5,6].

Generally, clustering algorithms are divided into hard and soft clustering schemes [7]. The c-means algorithm [8] is the most widely used hard clustering method due to its speed and simplicity. However, it is sensitive to initial points and often falls short of finding an optimal solution. Xu et al. [9] explored an alternative to the c-means algorithm that retains its simplicity while mitigating its tendency to become trapped in local minima, called power k-means. The optimization objective of this method is a power mean function of the distances between sample points and cluster centers, leading to its non-convex and high-dimensional characteristics, which makes it difficult to solve directly. Therefore, the majorization-minimization (MM) method is used to derive a descent scheme. In this paper, the MM method will be employed for the first time to solve the possibilistic fuzzy c-means clustering problem.

As one of the most typical soft clustering methods, fuzzy c-means (FCM) [10,11] is a fuzzy extension method of c-means, which divides n data points into c clusters by the membership grade matrix U. Compared to c-means, FCM achieves better clustering performance on datasets with overlapped clusters. Due to its flexibility and robustness, FCM has been favored by researchers up to now. However, it also suffers from some drawbacks. The first is that this method is sensitive to initialization and may easily become trapped in local minima [12,13]. The iteratively re-weighted algorithm for FCM (IRWFCM) [14] was proposed to find better local minima by using the iteratively re-weighted method to solve an equivalent minimization problem of the original FCM problem. FCM with the entropy regularization method (ERFCM) [15] introduces a maximum entropy term to FCM, whose purpose is trading off fuzziness and compactness. Recently, the iteratively re-weighted method was also used to optimize the equivalent problem of ERFCM in [16]. IRWFCM and IRWERFCM both have smaller objective function values and better clustering performance with fewer iterations compared to the original optimization method.

Furthermore, FCM assigns memberships to $x_i$ based on the relative distance from $x_i$ to the c cluster centers. In noisy data environments, this can result in noise points that are far from the c clusters being assigned relatively high memberships, thereby not accurately reflecting the true clustering results [17]. To overcome this shortcoming, the possibilistic c-means (PCM) method was proposed by Krishnapuram and Keller [18], which relaxes the constraint ${\displaystyle \sum _{j=1}^c}{u}_{ij}=1\left(i=1,\cdots ,n\right)$ so that $u_{ij}$ better reflects what we expect for the typicality of the i-th sample to the j-th cluster. However, PCM still suffers from initialization sensitivity and is prone to obtaining a coincident clustering phenomenon [19]. Therefore, Pal et al. [20] proposed the possibilistic fuzzy c-means clustering (PFCM) algorithm, which is a kind of hybrid clustering method based on PCM and FCM, to solve the coincident clustering problem of PCM. This method not only has the stability of FCM but also partly inherits the robustness of PCM, making it widely researched and applied in many practical fields [21,22].

In contrast to traditional distance-based approaches like FCM, PCM, and PFCM, the generalized entropy-based PFCM (GEPFCM) algorithm [23] was proposed for clustering noisy data. GEPFCM utilizes distance-based functions to mitigate noise contributions on the cluster centers. An evaluation based on the distance between the actual and computed cluster centers reveals that the error of GEPFCM is approximately $80\%$ lower than that of PFCM. Moreover, PFCM encounters challenges when dealing with feature-imbalanced multidimensional data, often leading to misclassification, as it treats all features equally. Additionally, in multiclass datasets with strong noise injection, PFCM tends to produce significant deviations and overlapping centers due to difficulties in setting membership-weight parameters and the absence of between-class relationships in possibilistic memberships. To address these issues, the feature-weighted suppressed PFCM clustering (FW-S-PFCM) [24] method was introduced. This method incorporates a feature-weighted approach and a “suppressed competitive learning” strategy into the PFCM model, enhancing its capability to handle these two types of challenging data scenarios. Interested readers can find more PFCM-type algorithms in [25,26,27,28,29,30], and the relevant citations therein.

While recent advancements in PFCM methods have demonstrated enhanced performance in various aspects, such as handling complex data, distance, hyperparameters, and robustness, they do not address the problem of whether the method finds a better local optimal solution when it converges. However, as an extension of FCM on the objective function, PFCM inevitably inherits the shortcomings of FCM, such as numerous local minimums and susceptibility to falling into poor local minima. Therefore, studying how to avoid PFCM falling into poor local minima is a worthwhile endeavor. In this study, we aim to tackle this issue by first establishing an equivalent formulation of the original PFCM problem and then designing an iterative sub-problem using the MM method. In addition, we also propose an alternative derivation method for PFCM and use the MM method to prove the equivalence of these two methods. The main contributions of this paper are summarized as follows:

• We propose an alternative derivation method for PFCM, which begins by formulating an equivalent and simplified optimization problem, followed by solving it using the MM method. Finally, we demonstrate the equivalence of the new derivation method with PFCM.
• Due to the presence of a proportional term in the derived simplified optimization problem, we further transform it into an easily solvable equivalent form by introducing a new intermediate variable s. Then, the MM method is employed to design an iterative sub-problem. We refer to this method as MMPFCM.
• The complexity analysis indicates that MMPFCM and PFCM share the same computational complexity. However, MMPFCM utilizes the intermediate variable s of $c\times 1$ size instead of the variable V of $c\times p$ size to update U and T, resulting in smaller space complexity.
• It is theoretically proven that when the inner loop of MMPFCM is executed only once, MMPFCM degenerates to the original PFCM method.
• Experimental studies show that MMPFCM obtains better local minima compared to PFCM. In addition, compared with other state-of-the-art clustering methods, MMPFCM also shows its superiority.

The rest of this paper is organized as follows. Some preliminaries including notations, PFCM, and the MM method are given in Section 2. In Section 3, we rederive PFCM using the MM method, and another effective optimization method (MMPFCM) is presented in Section 4. A theoretical analysis is given in Section 5. In Section 6, the experimental results and discussions are reported. Section 7 concludes this paper.

2. Related Works

2.1. Notations

Let $X=\left[x_1,x_2,\cdots ,x_n\right]\in R^{p\times n}$ denote the data matrix, where $x_i\in R^p$ is the i-th sample for $i=1,2,\cdots ,n$, and $x_i^T$ denotes the transpose of vector $x_i$. Let $V=\left[v_1,v_2,\cdots ,v_c\right]\in R^{p\times c}$ denote the cluster center matrix, where $v_j\in R^p$ is the centroid of the cluster $C_j$ for $j=1,2,\cdots ,c$. The fuzzy membership matrix is denoted by $U={\left(u_{ij}\right)}_{n\times c}$, where $u_{ij}$ represents the fuzzy membership degree of the i-th sample to the j-th cluster. The possibilistic membership matrix is denoted by $T={\left(t_{ij}\right)}_{n\times c}$, where $t_{ij}$ represents the possibilistic membership degree of the i-th sample to the j-th cluster. $u_j$ and $t_j$ are the j-th columns of U and T, respectively. $u_j^{\left(k\right)}$ and $t_j^{\left(k\right)}$ denote $u_j$ and $t_j$ at the k-th iteration, respectively. In addition, e is a column vector with all elements equal to 1. $f\left(\theta \left|{\theta }_t\right.\right)$ represents the function f evaluated at $\theta$ given the condition or parameter value ${\theta }_t$.

2.2. Possibilistic Fuzzy C-Means Clustering

PFCM [20] is a kind of hybrid clustering method based on PCM and FCM, which contains both membership and typicality components. Its objective function is as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \underset{\left(U,T,V\right)}{min}{J}_{PFCM}\left(U,T,V\right)=& \sum _{i=1}^n\sum _{j=1}^c\left(a{u}_{ij}^m+b{t}_{ij}^q\right){∥x_i-v_j∥}_2^2+\sum _{j=1}^c{\gamma }_j\sum _{i=1}^n{\left(1-t_{ij}\right)}^q,\hfill \\ \hfill s.t.& \sum _{j=1}^c{u}_{ij}=1,0\le u_{ij},t_{ij}\le 1,\left(i=1,2,\cdots ,n\right).\hfill \end{array}\end{array}$$

(1)

Here, the membership weights $a\in \left(0,+\infty \right)$ and $b\in \left(0,+\infty \right)$ control the roles of the fuzzy membership degree and possibilistic membership degree. Furthermore, the exponential factors of fuzzy membership and possibilistic membership are $m\in \left(1,+\infty \right)$ and $q\in \left(1,+\infty \right)$, respectively. ${\gamma }_j$ is a penalty factor that is usually determined by [20]:

$${\gamma }_j=K\sum _{i=1}^n{u}_{ij}^m{∥x_i-v_j∥}^2/\sum _{i=1}^n{u}_{ij}^m,$$

where $K>0$, with the most common choice being $K=1$.

PFCM usually updates the fuzzy membership $u_{ij}$, the possibilistic membership $t_{ij}$, and the center $v_j$ alternatively by

$$\begin{array}{cc}\hfill u_{ij}^{\left(k+1\right)}=& {\displaystyle \frac{{∥x_i-v_j^{\left(k\right)}∥}^{\frac{2}{1-m}}}{{\displaystyle \sum _{l=1}^c}{∥x_i-v_l^{\left(k\right)}∥}^{\frac{2}{1-m}}}},\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill t_{ij}^{\left(k+1\right)}=& {\left[1+{\left({\displaystyle \frac{b}{{\gamma }_j}}{∥x_i-v_j^{\left(k\right)}∥}^2\right)}^{{\displaystyle \frac{1}{q-1}}}\right]}^{-1},\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill v_j^{\left(k+1\right)}=& {\displaystyle \frac{{\displaystyle \sum _{i=1}^n}\left[a{\left(u_{ij}^{\left(k\right)}\right)}^m+b{\left(t_{ij}^{\left(k\right)}\right)}^q\right]x_i}{{\displaystyle \sum _{i=1}^n}\left[a{\left(u_{ij}^{\left(k\right)}\right)}^m+b{\left(t_{ij}^{\left(k\right)}\right)}^q\right]}}.\hfill \end{array}$$

(4)

When the membership weight a in the objective function of PFCM is 0, we realize PCM within the PFCM framework, and then Equations (3) and (4) are the iterative functions, respectively. When the membership weight b in the objective function of PFCM is 0, we realize FCM in the PFCM framework, and then Equations (2) and (4) are the corresponding iterative functions.

2.3. Majorization-Minimization Method

The majorization-minimization (MM) method [31] is an iterative optimization algorithm. It works by constructing a surrogate function that is simpler to minimize.

Consider the following optimization problem:

$$\underset{\theta }{min}g\left(\theta \right),s.t.\theta \in \Theta ,$$

where $\Theta$ is a nonempty closed set in $R^n$ and $g:\Theta \to R$ is a continuous function. The MM method iteratively reduces a series of surrogate functions $f\left(\theta \left|{\theta }_t\right.\right)$ majorizing the objective function $g\left(\theta \right)$ at the current iterate ${\theta }_t$. The principle of majorization involves two key aspects: first, equality is achieved between the surrogate function and the objective function at the current iterate ${\theta }_t$, denoted as $f\left({\theta }_t\left|{\theta }_t\right.\right)=g\left({\theta }_t\right)$; and second, the surrogate function dominates the objective function, meaning $f\left(\theta \left|{\theta }_t\right.\right)\ge g\left(\theta \right)$ for all $\theta$.

The update rule of the MM method is defined as follows [9]:

$${\theta }_{t+1}\leftarrow \arg \underset{\theta }{min}f\left(\theta \left|{\theta }_t\right.\right),$$

which implies the descent property

$$g\left({\theta }_{t+1}\right)\le f\left({\theta }_{t+1}\left|{\theta }_t\right.\right)\le f\left({\theta }_t\left|{\theta }_t\right.\right)=g\left({\theta }_t\right).$$

By successively minimizing these surrogate functions, we aim to converge to a local minimum of the original optimization problem. This process continues until convergence is achieved or another termination criterion is met. It finds wide applications in various fields, including machine learning [9], signal processing [32], and statistics [33].

3. Alternative Derivation Method for Possibilistic Fuzzy C-Means Clustering

In this section, we propose an alternative derivation method for PFCM, which interprets its iterative process using the MM method. Firstly, a simplified optimization model, which only contains the variables U and T, is obtained by eliminating V. Secondly, the MM method is applied to solve the new problem. Finally, we provide proof to demonstrate that this new derivation method is the same as the original PFCM method.

3.1. Formulation

Let $F=U^m$, $G=T^q$, and $Z=aF+bG$, and then according to Equation (4), we have

$$\begin{array}{c}{v}_j={\displaystyle \frac{{\displaystyle \sum _{i=1}^n}\left(a{f}_{ij}+b{g}_{ij}\right)x_i}{{\displaystyle \sum _{i=1}^n}\left(a{f}_{ij}+b{g}_{ij}\right)}}={\displaystyle \frac{X\left(a{f}_j+b{g}_j\right)}{{\left(a{f}_j+b{g}_j\right)}^{T}e}}={\displaystyle \frac{X{z}_j}{z_j^{T}e}}.\hfill \end{array}$$

Substituting $v_j$ into Equation (1), we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \underset{\left(U,T\right)}{min}{J}_{PFCM}\left(U,T\right)=& \sum _{i=1}^n\sum _{j=1}^c{z}_{ij}\left(x_i^T{x}_i-2x_i^T{v}_j+v_j^T{v}_j\right)+\sum _{j=1}^c{\gamma }_j\sum _{i=1}^n{\left(1-t_{ij}\right)}^q\hfill \\ \hfill =& \sum _{i=1}^n\sum _{j=1}^c{z}_{ij}{x}_i^T{x}_i-2\sum _{j=1}^c{z}_j^T{X}^T{v}_j+\sum _{j=1}^c{z}_j^{T}e{v}_j^T{v}_j+\sum _{j=1}^c{\gamma }_j\sum _{i=1}^n{\left(1-t_{ij}\right)}^q\hfill \\ \hfill =& \sum _{i=1}^n\sum _{j=1}^c{z}_{ij}{x}_i^T{x}_i-\sum _{j=1}^c{\displaystyle \frac{z_j^T{X}^{T}X{z}_j}{z_j^{T}e}}+\sum _{j=1}^c{\gamma }_j\sum _{i=1}^n{\left(1-t_{ij}\right)}^q.\hfill \end{array}\end{array}$$

(5)

Because the second term in problem (5) involves a proportional term, it makes problem (5) difficult to solve directly. Therefore, we introduce the MM method to optimize this equivalent optimization problem, and the next primary issue is to find a surrogate function that is easier to optimize. Considering that

$$\varphi \left(Z\right)={\displaystyle \frac{Z^T{X}^{T}XZ}{Z^{T}e}}$$

(6)

is a convex function (as proven in Appendix A), then its opposite function is concave. Based on the property of concave functions, which states that the tangent line of a concave function at any point lies above the graph at that point, the following inequality holds:

$$-\varphi \left(z_j\right)\le -\varphi \left(z_j^{\left(k\right)}\right)-{\left({\omega }_j^{\left(k\right)}\right)}^T\left(z_j-z_j^{\left(k\right)}\right),$$

where

$${\omega }_j^{\left(k\right)}={\displaystyle \frac{2X^{T}X{z}_j^{\left(k\right)}}{{\left(z_j^{\left(k\right)}\right)}^{T}e}}-{\displaystyle \frac{{\left(z_j^{\left(k\right)}\right)}^T{X}^{T}X{z}_j^{\left(k\right)}e}{{\left[{\left(z_j^{\left(k\right)}\right)}^{T}e\right]}^2}}$$

(7)

is the first-order derivative function of $\varphi \left(z_j\right)$ with respect to $z_j$ at the current iterate $\left(u_j^{\left(k\right)},t_j^{\left(k\right)}\right)$. Therefore, $-{\left({\omega }_j^{\left(k\right)}\right)}^T{z}_j$ is further optimized as the surrogate function for $-\varphi \left(z_j\right)$. Substituting this surrogate function into problem (5), we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \underset{\left(U,T\right)}{min}{J}_{PFCM}\left(U,T\right)=& \sum _{i=1}^n\sum _{j=1}^c{z}_{ij}{x}_i^T{x}_i-\sum _{j=1}^c{\left({\omega }_j^{\left(k\right)}\right)}^T{z}_j+\sum _{j=1}^c{\gamma }_j\sum _{i=1}^n{\left(1-t_{ij}\right)}^q\hfill \\ \hfill =& \sum _{i=1}^n\sum _{j=1}^c{z}_{ij}{x}_i^T{x}_i-\sum _{i=1}^n\sum _{j=1}^c{z}_{ij}{\omega }_{ij}^{\left(k\right)}+\sum _{j=1}^c{\gamma }_j\sum _{i=1}^n{\left(1-t_{ij}\right)}^q\hfill \\ \hfill =& \sum _{i=1}^n\sum _{j=1}^c\left(a{u}_{ij}^m+b{t}_{ij}^q\right)\left(x_i^T{x}_i-{\omega }_{ij}^{\left(k\right)}\right)+\sum _{j=1}^c{\gamma }_j\sum _{i=1}^n{\left(1-t_{ij}\right)}^q.\hfill \end{array}\end{array}$$

(8)

Further, the update formulas for the variables U and T are derived using the alternating iteration method.

3.2. Optimization Procedure

We optimize problem (8) with respect to one variable, with the other variables being fixed, which leads to the following two sub-problems.

Firstly, fixing the variable T, the U-update step involves minimizing problem (8) with respect to the variable U, which can be expressed as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \underset{U}{min}& \sum _{i=1}^n\sum _{j=1}^{c}a{u}_{ij}^m\left(x_i^T{x}_i-{\omega }_{ij}^{\left(k\right)}\right),\hfill \\ \hfill s,t,& \sum _{j=1}^c{u}_{ij}=1,0\le u_{ij}\le 1,\left(i=1,\cdots ,n\right).\hfill \end{array}\end{array}$$

(9)

We use the Lagrange multiplier method to derive the iterative function of $u_{ij}$ as follows:

$$u_{ij}^{\left(k+1\right)}={\displaystyle \frac{{\left(x_i^T{x}_i-{\omega }_{ij}^{\left(k\right)}\right)}^{\frac{1}{1-m}}}{{\displaystyle \sum _{l=1}^c}{\left(x_i^T{x}_i-{\omega }_{il}^{\left(k\right)}\right)}^{\frac{1}{1-m}}}}.$$

(10)

Secondly, fixing the variable U, the T-update step involves minimizing problem (8) with respect to the variable T, which can be expressed as follows:

$$\underset{T}{min}\sum _{i=1}^n\sum _{j=1}^{c}b{t}_{ij}^q\left(x_i^T{x}_i-{\omega }_{ij}^{\left(k\right)}\right)+\sum _{j=1}^c{\gamma }_j\sum _{i=1}^n{\left(1-t_{ij}\right)}^q.$$

(11)

Taking the derivative of Equation (11) with respect to $t_{ij}$ and setting it to zero, we have

$$t_{ij}^{\left(k+1\right)}={\displaystyle \frac{1}{1+{\left[\frac{b\left(x_i^T{x}_i-{\omega }_{ij}^{\left(k\right)}\right)}{{\gamma }_j}\right]}^{\frac{1}{q-1}}}}.$$

(12)

Obviously, according to Equation (7), we know that $w_j$ is dependent on U and T. When we obtain U and T by calculating Equations (10) and (12), U and T are repeatedly utilized to update $w_j$ using Equation (7). This means that we can calculate U, T, and $w_j$ iteratively. The detailed process for solving problem (5) is summarized in Algorithm 1.

Algorithm 1 Alternative derivation method for PFCM

1: Input X and c. 2: Initialize U, T, ${\gamma }_j\left(j=1,2,\cdots ,c\right)$, and set $k=0$. 3: repeat 4:     Calculate ${\omega }_j\left(j=1,2,\cdots ,c\right)$ by Equation (7); 5:     Calculate U by Equation (10); 6:     Calculate T by Equation (12); 7:     $k=k+1$; 8: until convergence 9: Output U and T.

Next, we provide proof to demonstrate that this new derivation method is the same as the original PFCM method, as shown in Theorem 1.

Theorem 1.

Algorithm 1 is equivalent to the original PFCM algorithm.

Proof. 

From Equation (7), we can further deduce the value of each item in matrix W as follows:

$${\omega }_{ij}^{\left(k\right)}=2x_i^T{v}_j^{\left(k\right)}-{\left(v_j^{\left(k\right)}\right)}^T{v}_j^{\left(k\right)},$$

(13)

and then we have

$$x_i^T{x}_i-{\omega }_{ij}^{\left(k\right)}={∥x_i-v_j^{\left(k\right)}∥}^2.$$

(14)

Substituting Equation (14) into Equation (10), we observe that Equation (10) is the same as Equation (2). Similarly, substituting Equation (14) into Equation (12), we observe that Equation (12) is the same as Equation (3). Therefore, based on the above analysis, Algorithm 1 is equivalent to the original PFCM algorithm, as their final iterative functions of U and T are the same.    □

To sum up, for the original PFCM problem, we provide a solution using the MM method by proving the convexity of the proportional term in problem (5). This represents a new derivation approach not found in other studies.

4. Majorization-Minimization Method for Possibilistic Fuzzy C-Means Clustering

In this section, another method is proposed to optimize the problem (5), called MMPFCM. Compared to PFCM, MMPFCM obtains better local minima.

4.1. Formulation

Here, we first present a lemma that introduces a simple optimization problem. In this lemma, a new variable s is introduced to eliminate the ratio term in the optimization problem, transforming the original optimization problem into an easily solvable equivalent form. The technique of introducing new variables in the lemma provides a basis for the equivalent transformation of the optimization model in this section.

Lemma 1.

For a constant $\mu >0$, the equation $-{\displaystyle \frac{{\eta }^2}{\mu }}=\underset{s}{min}{s}^2\mu -2s\eta$ holds, and the optimization objective reaches its minimum when $s={\displaystyle \frac{\eta }{\mu }}$.

According to Lemma 1, let ${\mu }_j=z_j^{T}e$ and ${\eta }_j=\sqrt{z_j^T{X}^{T}X{z}_j}$, and we have

$$s_j={\displaystyle \frac{\sqrt{z_j^T{X}^{T}X{z}_j}}{z_j^{T}e}}.$$

(15)

Therefore, in order to eliminate the proportional term in problem (5), we introduce a new variable $s={\left(s_1,s_2,\cdots ,s_c\right)}^T$, and then problem (5) can be transformed into

$$\begin{array}{c}\underset{\left(U,T,s\right)}{min}{J}_{PFCM}\left(U,T,s\right)={\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^c}{z}_{ij}{x}_i^T{x}_i+{\displaystyle \sum _{j=1}^c}\left[s_j^2{z}_j^{T}e-2s_j\sqrt{z_j^T{X}^{T}X{z}_j}\right]+{\displaystyle \sum _{j=1}^c}{\gamma }_j{\displaystyle \sum _{i=1}^n}{\left(1-t_{ij}\right)}^q.\hfill \end{array}$$

(16)

Before solving problem (16), the following theorem provides the equivalence proof between problem (5) and problem (16).

Theorem 2.

Problem (16) is equivalent to problem (5).

Proof. 

In order to prove that those two problems are equivalent, we substitute $s_j$ in Equation (15) into problem (16), and it can be immediately concluded that problem. (16) is equivalent to problem (5).    □

Problem (16) also involves three variables U, T, and s. However, compared to the variable V of $p\times c$ size in PFCM, the variable s in MMPFCM is a vector with c elements, resulting in smaller space complexity. Next, the alternating iteration method is used to solve problem (16).

4.2. Optimization Procedure

We optimize problem (16) with respect to one variable, with the other variables being fixed, which leads to the following three sub-problems.

Firstly, when the variables U and T are fixed, the s-update step is the minimization of problem (16) with respect to the variable s. Because the c components of s are separable, we have the following optimization problem:

$$\underset{s_j}{min}{s}_j^2{z}_j^{T}e-2s_j\sqrt{z_j^T{X}^{T}X{z}_j}.$$

(17)

Taking the derivative of Equation (17) with respect to $s_j$ and setting the derivative value to zero, we obtain $s_j$, as shown in Equation (15).

Secondly, when the variable s is fixed, the optimization problem involving the variables U and T can be denoted as follows:

$$\begin{array}{c}\underset{\left(U,T\right)}{min}{J}_{PFCM}\left(U,T\right)={\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^c}{z}_{ij}{x}_i^T{x}_i+{\displaystyle \sum _{j=1}^c}\left[s_j^2{z}_j^{T}e-2s_j\sqrt{z_j^T{X}^{T}X{z}_j}\right]+{\displaystyle \sum _{j=1}^c}{\gamma }_j{\displaystyle \sum _{i=1}^n}{\left(1-t_{ij}\right)}^q.\hfill \end{array}$$

(18)

Since problem (18) involves an optimization objective with a square root, the derivative for direct optimization is quite complex. Therefore, we continue to rely on the principles of the MM method to search for surrogate functions as a further optimization model. It is obvious that $X^{T}X$ is a positive semidefinite matrix, so $\sqrt{z_j^T{X}^{T}X{z}_j}$ is a convex function about $z_j$. Then, we can immediately conclude that $-\sqrt{z_j^T{X}^{T}X{z}_j}$ is a concave function about $z_j$. Concavity supplies the linear majorization

$$-\sqrt{z_j^T{X}^{T}X{z}_j}\le -\sqrt{{\left(z_j^{\left(k\right)}\right)}^T{X}^{T}X{z}_j^{\left(k\right)}}-{\left({\alpha }_j^{\left(k\right)}\right)}^T\left(z_j-z_j^{\left(k\right)}\right),$$

where

$${\alpha }_j^{\left(k\right)}={\displaystyle \frac{X^{T}X{z}_j^{\left(k\right)}}{\sqrt{{\left(z_j^{\left(k\right)}\right)}^T{X}^{T}X{z}_j^{\left(k\right)}}}}$$

(19)

is the first-order derivative function of $\sqrt{z_j^T{X}^{T}X{z}_j}$ with respect to $z_j$ at the current iterate $\left(u_j^{\left(k\right)},t_j^{\left(k\right)}\right)$. Therefore, $-{\left({\alpha }_j^{\left(k\right)}\right)}^T{z}_j$ is chosen as the surrogate function for $-\sqrt{z_j^T{X}^{T}X{z}_j}$. Substituting this surrogate function into problem (18), we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \underset{\left(U,T\right)}{min}{J}_{PFCM}\left(U,T\right)=& \sum _{i=1}^n\sum _{j=1}^c\left(a{f}_{ij}+b{g}_{ij}\right)x_i^T{x}_i+\sum _{j=1}^c\left[s_j^2{\left(a{f}_j+b{g}_j\right)}^{T}e-2s_j{\left({\alpha }_j^{\left(k\right)}\right)}^T\left(a{f}_j+b{g}_j\right)\right]\hfill \\ \hfill & +\sum _{j=1}^c{\gamma }_j\sum _{i=1}^n{\left(1-t_{ij}\right)}^q.\hfill \end{array}\end{array}$$

(20)

Further, fixing the variables T, the U-update step involves minimizing problem (20) with respect to the variable U, which can be expressed as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & \underset{U}{min}\sum _{i=1}^n\sum _{j=1}^c\left(a{f}_{ij}\right)x_i^T{x}_i+\sum _{j=1}^c\left[s_j^2{\left(a{f}_j\right)}^{T}e-2s_j{\left({\alpha }_j^{\left(k\right)}\right)}^T\left(a{f}_j\right)\right]\hfill \\ \hfill & \iff \hfill \\ \hfill & \underset{U}{min}\sum _{i=1}^n\sum _{j=1}^c\left(a{f}_{ij}\right)x_i^T{x}_i+\sum _{i=1}^n\sum _{j=1}^c\left[s_j^2\left(a{f}_{ij}\right)-2s_j\left(a{f}_{ij}\right){\alpha }_{ij}^{\left(k\right)}\right]\hfill \\ \hfill & \iff \hfill \\ \hfill & \underset{U}{min}\sum _{i=1}^n\sum _{j=1}^c{u}_{ij}^m\left(a{x}_i^T{x}_i+a{s}_j^2-2a{s}_j{\alpha }_{ij}^{\left(k\right)}\right).\hfill \end{array}\end{array}$$

(21)

$u_{ij}$ is solved using the Lagrange multiplier method, and we have

$$u_{ij}^{\left(k+1\right)}=d_{ij}^{{\displaystyle \frac{1}{1-m}}}/\sum _{l=1}^c{d}_{il}^{{\displaystyle \frac{1}{1-m}}}$$

(22)

with $d_{ij}=x_i^T{x}_i+s_j^2-2s_j{\alpha }_{ij}^{\left(k\right)}$.

Then, fixing the variables U, the T-update step involves minimizing problem (20) with respect to the variable T, which can be expressed as follows

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & \underset{T}{min}\sum _{i=1}^n\sum _{j=1}^c\left(b{g}_{ij}\right)x_i^T{x}_i+\sum _{j=1}^c\left[s_j^2{\left(b{g}_j\right)}^{T}e-2s_j{\left({\alpha }_j^{\left(k\right)}\right)}^T\left(b{g}_j\right)\right]+\sum _{j=1}^c{\gamma }_j\sum _{i=1}^n{\left(1-t_{ij}\right)}^q\hfill \\ \hfill & \iff \hfill \\ \hfill & \underset{T}{min}\sum _{i=1}^n\sum _{j=1}^c\left(b{g}_{ij}\right)x_i^T{x}_i+\sum _{i=1}^n\sum _{j=1}^c\left[s_j^2\left(b{g}_{ij}\right)-2s_j\left(b{g}_{ij}\right){\alpha }_{ij}^{\left(k\right)}\right]+\sum _{j=1}^c{\gamma }_j\sum _{i=1}^n{\left(1-t_{ij}\right)}^q\hfill \\ \hfill & \iff \hfill \\ \hfill & \underset{T}{min}\sum _{i=1}^n\sum _{j=1}^c{t}_{ij}^q\left(b{x}_i^T{x}_i+b{s}_j^2-2b{s}_j{\alpha }_{ij}^{\left(k\right)}\right)+\sum _{j=1}^c{\gamma }_j\sum _{i=1}^n{\left(1-t_{ij}\right)}^q.\hfill \end{array}\end{array}$$

(23)

Taking the derivative of Equation (23) with respect to $t_{ij}$ and setting it to zero, we have

$$t_{ij}^{\left(k+1\right)}={\left[1+{\left({\displaystyle \frac{b{d}_{ij}}{{\gamma }_j}}\right)}^{1/\phantom{1(q-1)}(q-1)}\right]}^{-1}$$

(24)

with $d_{ij}=x_i^T{x}_i+s_j^2-2s_j{\alpha }_{ij}^{\left(k\right)}$.

From Equation (19), we find that ${\alpha }_j$ is changed by the change in U and T. So, when we calculate U using Equation (22) and T using Equation (24), ${\alpha }_j$ is updated accordingly, which means that we can calculate ${\alpha }_j$, U, and T iteratively. This process continues until convergence is achieved or another termination criterion is met.

What needs to be pointed out is that this loop is nested within the loop of the proposed method. Therefore, we refer to this loop as the inner loop of the algorithm, with its termination condition set as the maximum number of iterations of the loop being less than or equal to K. It should be noted that this paper selects $K=5$ as the reference value for the comparative experiments, as detailed in Section 6.

In summary, a new optimization method, called MMPFCM, is summarized in Algorithm 2.

Algorithm 2 Majorization-minimization method for possibilistic fuzzy c-means clustering (MMPFCM)

1: Input X and c. 2: Initialize U, T, ${\gamma }_j\left(j=1,2,\cdots ,c\right)$, the number of times for the inner loop K, and set $t=0$. 3: repeat 4:     Calculate $s_j\left(j=1,2,\cdots ,c\right)$ by Equation (15); 5:     for $k=1:K$ 6:         Calculate ${\alpha }_j\left(j=1,2,\cdots ,c\right)$ by Equation (19); 7:         Calculate U by Equation (22); 8:         Calculate T by Equation (24); 9:     end for 10:     $t=t+1$; 11: until convergence 12: Output U and T.

As is well known, PFCM is an extension of FCM and PCM. Therefore, MMPFCM can also be interpreted as a generalization of MMFCM and MMPCM, where MMFCM and MMPCM are two cases when $b=0$ or $a=0$ in MMPFCM, respectively. Relevant experiments regarding these two methods are provided in Appendix B.

4.3. An Interesting Observation

Algorithm 2 consists of two nested loops: an inner loop and an outer loop. Through analysis, we have made an interesting observation, that is, when the inner loop is executed only once, Algorithm 2 degenerates to the original PFCM method. For a detailed analysis, refer to Theorem 3.

Theorem 3.

When $K=1$ in Algorithm 2, Algorithm 2 is equivalent to the original PFCM method in this case.

Proof. 

If the inner loop is executed only once, then Algorithm 2 contains one loop, and its detailed steps are as follows: calculate $s_j$ using Equation (15), calculate ${\alpha }_j$ using Equation (19), calculate $u_{ij}$ using Equation (22), and calculate $t_{ij}$ using Equation (24).

From Equation (19), we can further deduce the value of each item in matrix A as follows:

$${\alpha }_{ij}^{\left(k\right)}={\displaystyle \frac{x_i^{T}X{z}_j^{\left(k\right)}}{\sqrt{{\left(z_j^{\left(k\right)}\right)}^T{X}^{T}X{z}_j^{\left(k\right)}}}}.$$

(25)

Then, substituting Equations (15) and (25) into Equation (22), we have

$$u_{ij}^{\left(k+1\right)}={\displaystyle \frac{{∥x_i-v_j^{\left(k\right)}∥}^{\frac{2}{1-m}}}{{\displaystyle \sum _{l=1}^c}{∥x_i-v_l^{\left(k\right)}∥}^{\frac{2}{1-m}}}},$$

which is the same as Equation (2). Similarly, substituting Equations (15) and (25) into Equation (24), we observe that the obtained $t_{ij}^{\left(k+1\right)}$ is similar to Equation (3). Therefore, based on the above analysis, when the inner loop is executed only once in Algorithm 2, Algorithm 2 degenerates to the original PFCM method. □

In addition, Theorem 1 proves that Algorithm 1 is equivalent to the original PFCM method, so when $K=1$ in Algorithm 2, Algorithm 2 is also equivalent to Algorithm 1.

5. Theoretical Analysis

5.1. Convergence Analysis

If we want to prove the convergence of Algorithm 2, it is essential to initially prove the convergence of the inner loop in Algorithm 2. Therefore, let us first direct our attention to proving the convergence of its inner loop.

Theorem 4.

The inner loop in Algorithm 2 will decrease the objective value of problem (18) in each iteration until it converges.

Proof. 

In Equation (20), let

$$f\left(Z\right)=\sum _{i=1}^n\sum _{j=1}^c{z}_{ij}{x}_i^T{x}_i+\sum _{j=1}^c{s}_j^2{z}_j^{T}e+\sum _{j=1}^c{\gamma }_j\sum _{i=1}^n{\left(1-t_{ij}\right)}^q.$$

Then, Equation (20) becomes $f\left(Z\right)-{\displaystyle \sum _{j=1}^c}2s_j{\left({\alpha }_j^{\left(k\right)}\right)}^T{z}_j$. Let $\overline{U}$ and $\overline{T}$ be the updated U and T in each interation, and then we have

$$f\left(\overline{Z}\right)-\sum _{j=1}^{c}2s_j{\left({\alpha }_j^{\left(k\right)}\right)}^T{\overline{z}}_j\le f\left(Z\right)-\sum _{j=1}^{c}2s_j{\left({\alpha }_j^{\left(k\right)}\right)}^T{z}_j.$$

(26)

Furthermore, according to $\sqrt{z_j^T{X}^{T}X{z}_j}$ is convex about $z_j$, and convexity supplies the linear majorization

$$\sqrt{{\overline{z}}_j^T{X}^{T}X{\overline{z}}_j}-\sqrt{z_j^T{X}^{T}X{z}_j}\ge {\left({\alpha }_j^{\left(k\right)}\right)}^T{\overline{z}}_j-{\left({\alpha }_j^{\left(k\right)}\right)}^T{z}_j,$$

Further, multiplying both sides by ${\displaystyle \sum _j}2s_j$, we have

$$\begin{array}{c}{\displaystyle \sum _j}2s_j{\left({\alpha }_j^{\left(k\right)}\right)}^T{\overline{z}}_j-{\displaystyle \sum _j}2s_j\sqrt{{\overline{z}}_j^T{X}^{T}X{\overline{z}}_j}\le {\displaystyle \sum _j}2s_j{\left({\alpha }_j^{\left(k\right)}\right)}^T{z}_j-{\displaystyle \sum _j}2s_j\sqrt{z_j^T{X}^{T}X{z}_j}.\hfill \end{array}$$

(27)

Adding both sides of Inequality (26) and (27) yields

$$f\left(\overline{Z}\right)-\sum _{j}2s_j\sqrt{{\overline{z}}_j^T{X}^{T}X{\overline{z}}_j}\le f\left(Z\right)-\sum _{j}2s_j\sqrt{z_j^T{X}^{T}X{z}_j}.$$

Therefore, the inner loop in Algorithm 2 decreases the objective value of problem (18) in each iteration until it converges. □

In conclusion, since the inner loop in Algorithm 2 converges, Algorithm 2 also converges.

5.2. Complexity Analysis

For the computational complexity of MMPFCM, since the multiplication operation of matrices generally requires more time than simple addition operations, we focus solely on the multiplication operation.

In Algorithm 2, for Step 4, computing $s_j(j=1,\cdots ,c)$ needs $O\left(npc+pc+nc\right)$. For Step 6, we need $O\left(2npc+pc\right)$ to calculate ${\alpha }_j(j=1,\cdots ,c)$. For Step 7, computing U needs $O\left(np+nc\right)$. For Step 8, we need $O\left(np+nc\right)$ to calculate T. Therefore, the total computational complexity of MMPFCM is $O\left(\left(\left(2npc+2nc+2np+pc\right)t_1+npc+pc+nc\right)t_2\right)$, where $t_1$ and $t_2$ are defined as the number of iterations of the inner loop and the outer loop in Algorithm 2, respectively.

As a consequence, MMPFCM has the same linear complexity with respect to the number of samples, i.e., $O\left(npc{t}_1{t}_2\right)$. Next, we verify the effectiveness of the proposed method through experiments.

6. Experiments

To verify the effectiveness and clustering performance of the proposed algorithm, experimental studies are conducted on twelve real-world datasets, all selected from the UCI Machine Learning Repository (http://archive.ics.uci.edu/ml/index.php, accessed on 26 February 2024). The specifics of these datasets are outlined in

Table 1. Benchmark datasets.

Datasets	Instance	Feature	Class
SCADI	70	205	7
balance	625	4	3
glass	214	9	7
COIL20	1440	1024	20
ORL	400	1024	40
Yale32	165	1024	15
Yale64	165	4096	15
Isolet5	1559	617	26
Urban	168	147	9
Iris	150	4	3
Vehicle	846	18	4
USPS	9298	256	10

.

All the experiments are run on a personal computer with an Intel Core i5-6500 processor and a maximum memory of 16 GB for all processes. The computer operates on Windows 10 with MATLAB R2017b. The convergence criterion is set as $\left|obj\left(it\right)-obj\left(it-1\right)\right|<{10}^{-5}$, where $obj\left(it\right)$ represents the objective function value at the end of the $it$-th iteration. The results are provided in tables and figures to verify the superiority of the proposed method.

6.1. Evaluation Metrics

To evaluate the performance of the proposed algorithm, four external metrics, including the overall F-measure for the entire dataset ($F^{\ast }$), Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and purity, are used to measure the agreement between the ground truth and the clustering results produced by the algorithm [14,34,35]. For all four metrics, higher scores correspond to improved clustering quality.

Metrics that do not require the labels of data are used for the performance evaluation and are called internal metrics. Two internal validity metrics, including the DBI [36] and XB [37], are selected here. It is worth noting that for both metrics, smaller values indicate better clustering performance [5].

The objective function value, time, and number of iterations are the remaining three evaluation metrics used to indicate the efficiency of the algorithms.

6.2. Setting of the Iterations in the Inner Loop

Since the setting of the iterations for the inner loop in Algorithm 2 is the primary issue that needs to be addressed for the execution of this method, in this subsection, we investigate the impact of different K on the clustering performance of the algorithm in order to select the appropriate number of iterations for the subsequent comparative experiments.

To study the effect of K on MMPFCM, we set the values of K as 1, 2, 5, 8, 10, and 30, respectively. MMPFCM is executed 10 times under random initializations, with the clarification that the initializations are the same for different values of K. We record the mean and standard deviation of F*, as well as the purity, across 10 experiments for different values of K. The results are shown in Figure 1.

Although the curves in Figure 1 exhibit minor fluctuations under different values of K, they generally show an upward trend. Additionally, It can be observed that the clustering performance of the algorithm shows a significant improvement when the number of iterations in the inner loop is set within the range of 5 to 10. Therefore, we can choose any number within this range to execute MMPFCM. This new update method increases the flexibility of the original model, allowing for similar fine-tuning to achieve better performance. It should be noted that for convenience, this paper selects $K=5$ as the reference value for all subsequent comparative experiments.

6.3. Comparison between PFCM and MMPFCM

Nie et al. [38] mentioned that a bad local minimum makes the objective value not small enough, which limits the algorithm’s performance. Based on this observation, the first set of experiments is carried out to evaluate the performance of MMPCM, focusing on the objective $J_{PFCM}\left(U,T,V\right)$ in Equation (1). To more intuitively compare the performance of MMPFCM and PFCM, we run both algorithms 10 times under the same random initializations and record their objective function values. Then, we calculate the difference in the objective function values, defined as $J_{MMPFCM}$—$J_{PFCM}$. Figure 2 presents a box plot of these differences across twelve real-world datasets. The green plus sign on each box plot indicates the mean of the differences. The red plus sign indicates the outliers. The red horizontal line represents the median. The length of the box represents the interquartile range (IQR), which is the range from the first quartile (Q1) to the third quartile (Q3) and shows the middle 50% of the data distribution. The whiskers extend from the box to the highest and lowest values, excluding outliers. Typically, the length of the whiskers extends up to 1.5 times the IQR from the edges of the box.

As shown in Figure 2, based on the same optimization objective function and the same random initialization conditions, the green plus signs are all below the horizontal 0 line, indicating that the mean difference in objective function values between MMPFCM and PFCM is less than 0 for these datasets. In other words, the average objective function value of MMPFCM is lower than that of PFCM. The red horizontal lines represent the median and are also below the horizontal 0 line for most datasets. This further demonstrates that the median objective function value of MMPFCM is lower than that of PFCM for most datasets, indicating better performance. Additionally, except for the COIL20 dataset, the boxes for the other datasets are below the horizontal 0 line. Since the difference is defined as $J_{MMPFCM}$—$J_{PFCM}$, the boxes being mostly below 0 indicates that the objective function value of MMPFCM is generally lower than that of PFCM. From this analysis, we can conclude that the proposed method achieves better local optimal solutions under the same initialization conditions.

The second set of experiments involves comparing the clustering performance of these two methods. Given that both the fuzzy membership U and the possibilistic membership T in PFCM can reflect the degree of membership of a point belonging to a particular cluster, we record the DBI and XB calculated by U and T for PFCM and MMPFCM in this set of experiments. The results are listed in

Table 2. DBI and XB calculated by U for PFCM and MMPFCM on real-world datasets. The values are averaged over 10 trials with random initializations. The standard deviations are given below the means, and the best results are shown in bold.

	Datasets	SCADI	Balance	Glass	COIL20	ORL	Yale32	Yale64	Isolet5	Urban	Iris	Vehicle	USPS
		DBI
PFCM	mean	1.430	1.611	4.427	2.284	2.045	2.598	1.860	4.471	1.558	0.639	0.682	3.367
	std	(0.137)	(0.000)	(1.856)	(0.170)	(0.058)	(0.096)	(0.127)	(0.324)	(0.129)	(0.017)	(0.007)	(0.001)
MMPFCM	mean	1.275	1.610	4.305	2.258	2.037	2.561	1.806	4.448	1.507	0.634	0.682	3.366
	std	(0.097)	(0.000)	(1.707)	(0.119)	(0.075)	(0.100)	(0.137)	(0.329)	(0.078)	(0.017)	(0.007)	(0.001)
		XB
PFCM	mean	1.046	0.833	10.193	1.483	1.902	2.319	1.881	3.905	2.926	0.276	0.868	2.055
	std	(0.256)	(0.000)	(2.509)	(0.265)	(0.216)	(0.257)	(0.409)	(1.347)	(1.066)	(0.008)	(0.013)	(0.002)
MMPFCM	mean	0.808	0.833	10.092	1.350	1.811	2.317	1.706	3.871	2.071	0.274	0.867	2.054
	std	(0.168)	(0.000)	(2.202)	(0.125)	(0.205)	(0.281)	(0.429)	(1.281)	(0.110)	(0.008)	(0.013)	(0.002)

and

Table 3. DBI and XB calculated by T for PFCM and MMPFCM on real-world datasets. The values are averaged over 10 trials with random initializations. The standard deviations are given below the means, and the best results are shown in bold.

	Datasets	SCADI	Balance	Glass	COIL20	ORL	Yale32	Yale64	Isolet5	Urban	Iris	Vehicle	USPS
		DBI
PFCM	mean	1.433	1.617	3.415	2.286	2.048	2.596	1.858	4.469	1.609	0.639	0.682	3.366
	std	(0.138)	(0.008)	(1.127)	(0.172)	(0.057)	(0.096)	(0.135)	(0.322)	(0.130)	(0.015)	(0.011)	(0.002)
MMPFCM	mean	1.275	1.615	3.279	2.258	2.041	2.564	1.805	4.447	1.562	0.635	0.681	3.365
	std	(0.097)	(0.008)	(0.888)	(0.118)	(0.077)	(0.099)	(0.135)	(0.329)	(0.088)	(0.016)	(0.011)	(0.003)
		XB
PFCM	mean	0.155	0.059	3.764	1.053	0.497	0.457	0.417	1.321	1.668	0.178	0.797	0.905
	std	(0.033)	(0.000)	(0.327)	(0.172)	(0.052)	(0.050)	(0.091)	(0.451)	(0.371)	(0.005)	(0.010)	(0.001)
MMPFCM	mean	0.125	0.059	3.747	0.973	0.470	0.454	0.374	1.309	1.344	0.177	0.796	0.905
	std	(0.022)	(0.000)	(0.254)	(0.083)	(0.058)	(0.057)	(0.089)	(0.426)	(0.079)	(0.005)	(0.010)	(0.001)

, respectively.

From Table 2 and Table 3, it is evident that regardless of whether the experimental results are obtained through U or T, the DBI and XB of MMPFCM are either less than or equal to those of PFCM, indicating that the proposed method exhibits better clustering performance on these twelve datasets. Moreover, the smaller standard deviation of MMPFCM also demonstrates its stability under the same initialization conditions.

The convergence curves of PFCM and MMPFCM on twelve real-world datasets are shown in Figure 3. The figures illustrate that both algorithms exhibit monotonically decreasing objective values over time. However, MMPFCM obtains better local minima on the SCADI, COIL20, ORL, Yale64, Isolet5, and Urban datasets. Furthermore, the time taken by both methods is at the same linear scale due to their identical linear complexities.

6.4. Comparison between MMPFCM and Other Methods

In order to demonstrate the performance of MMPFCM more comprehensively, MMPFCM is compared with six clustering algorithms, which include the following:

• Fuzzy c-means (FCM) [11];
• Iteratively re-weighted algorithm for fuzzy c-means (IRWFCM) [14];
• An effective optimization method For fuzzy c-means with entropy regularization (IRWERFCM) [16];
• A possibilistic fuzzy c-means clustering algorithm (PFCM) [20];
• Generalized entropy-based possibilistic fuzzy c-means for clustering noisy data and its convergence proof (EPFCMR) [23];
• A feature-weighted suppressed possibilistic fuzzy c-means clustering algorithm and its application to color image segmentation (FW-S-PFCM) [24].

IRWFCM and IRWERFCM are chosen because they utilize a novel method, namely the iteratively re-weighted method, to optimize FCM-type problems, and their advantages lie in their ability to achieve better local minima with fewer iterations. EPFCMR is a generalized entropy-based PFCM, which utilizes functions of distance, not the distance itself, to decrease noise contributions on the cluster centers. FW-S-PFCM introduces a feature-weighted method and a “suppressed competitive learning” strategy into the PFCM model, resulting in improved clustering performance. Additionally, it reduces the number of iterations, sensitivity to membership weights, and initializations of PFCM.

To facilitate a comprehensive comparison between MMPFCM and other algorithms, four external metrics on twelve real-world datasets are presented in

Table 4. Experimental results of different algorithms on real-world datasets. The values are averaged over 10 trials with random initializations. The standard deviations are given after the means, and the best results are shown in bold.

Datasets	Metrics	FCM	IRWFCM	IRWERFCM	PFCM	EPFCMR	FW-S-PFCM	MMPFCM
SCADI	ARI	0.267 (0.064)	0.300 (0.001)	0.327 (0.057)	0.429 (0.064)	0.434 (0.072)	0.261 (0.042)	0.436 (0.080)
	F*	0.535 (0.052)	0.560 (0.003)	0.581 (0.050)	0.630 (0.026)	0.632 (0.028)	0.528 (0.031)	0.637 (0.032)
	NMI	0.492 (0.034)	0.505 (0.000)	0.518 (0.028)	0.524 (0.019)	0.523 (0.019)	0.488 (0.029)	0.531 (0.026)
	Purity	0.734 (0.024)	0.759 (0.005)	0.753 (0.022)	0.631 (0.009)	0.636 (0.018)	0.736 (0.024)	0.637 (0.018)
balance	ARI	0.116 (0.131)	0.114 (0.095)	0.108 (0.077)	0.128 (0.119)	0.130 (0.123)	0.130 (0.011)	0.131 (0.125)
	F*	0.529 (0.119)	0.543 (0.078)	0.539 (0.056)	0.552 (0.089)	0.551 (0.093)	0.577 (0.009)	0.555 (0.088)
	NMI	0.106 (0.119)	0.098 (0.086)	0.102 (0.068)	0.112 (0.100)	0.113 (0.104)	0.112 (0.009)	0.114 (0.105)
	Purity	0.605 (0.102)	0.629 (0.067)	0.635 (0.060)	0.640 (0.075)	0.639 (0.079)	0.656 (0.016)	0.642 (0.077)
glass	ARI	0.235 (0.058)	0.243 (0.056)	0.241 (0.000)	0.221 (0.042)	0.175 (0.005)	0.234 (0.054)	0.246 (0.064)
	F*	0.549 (0.017)	0.552 (0.005)	0.536 (0.000)	0.527 (0.022)	0.457 (0.015)	0.556 (0.014)	0.558 (0.025)
	NMI	0.425 (0.032)	0.426 (0.003)	0.349 (0.000)	0.409 (0.019)	0.322 (0.005)	0.431 (0.023)	0.435 (0.036)
	Purity	0.591 (0.009)	0.592 (0.002)	0.565 (0.000)	0.587 (0.008)	0.538 (0.017)	0.593 (0.008)	0.594 (0.008)
COIL20	ARI	0.593 (0.026)	0.601 (0.018)	0.590 (0.026)	0.573 (0.012)	0.575 (0.007)	0.554 (0.023)	0.593 (0.030)
	F*	0.682 (0.025)	0.695 (0.019)	0.686 (0.018)	0.659 (0.011)	0.664 (0.006)	0.650 (0.022)	0.695 (0.029)
	NMI	0.773 (0.012)	0.782 (0.008)	0.782 (0.015)	0.772 (0.005)	0.778 (0.005)	0.766 (0.010)	0.783 (0.013)
	Purity	0.683 (0.030)	0.691 (0.020)	0.679 (0.022)	0.640 (0.013)	0.640 (0.005)	0.642 (0.018)	0.689 (0.029)
ORL	ARI	0.445 (0.021)	0.459 (0.024)	0.438 (0.028)	0.456 (0.023)	0.450 (0.017)	0.427 (0.015)	0.466 (0.013)
	F*	0.620 (0.016)	0.630 (0.022)	0.609 (0.019)	0.622 (0.022)	0.621 (0.020)	0.603 (0.013)	0.637 (0.010)
	NMI	0.784 (0.006)	0.788 (0.009)	0.776 (0.012)	0.786 (0.010)	0.785 (0.008)	0.777 (0.006)	0.784 (0.006)
	Purity	0.626 (0.020)	0.647 (0.019)	0.610 (0.019)	0.629 (0.020)	0.616 (0.017)	0.607 (0.011)	0.650 (0.012)
Yale32	ARI	0.278 (0.011)	0.276 (0.009)	0.253 (0.029)	0.295 (0.027)	0.289 (0.023)	0.250 (0.024)	0.304 (0.019)
	F*	0.528 (0.014)	0.524 (0.010)	0.500 (0.032)	0.537 (0.032)	0.535 (0.024)	0.496 (0.026)	0.559 (0.025)
	NMI	0.536 (0.006)	0.538 (0.007)	0.524 (0.020)	0.553 (0.020)	0.551 (0.019)	0.521 (0.018)	0.553 (0.013)
	Purity	0.479 (0.023)	0.468 (0.017)	0.453 (0.023)	0.495 (0.032)	0.485 (0.022)	0.453 (0.024)	0.518 (0.023)
Yale64	ARI	0.367 (0.023)	0.375 (0.021)	0.347 (0.043)	0.396 (0.020)	0.402 (0.025)	0.350 (0.016)	0.410 (0.017)
	F*	0.619 (0.028)	0.635 (0.015)	0.606 (0.040)	0.654 (0.021)	0.670 (0.023)	0.602 (0.018)	0.674 (0.019)
	NMI	0.603 (0.019)	0.615 (0.017)	0.594 (0.032)	0.623 (0.014)	0.632 (0.018)	0.599 (0.011)	0.634 (0.011)
	Purity	0.575 (0.021)	0.592 (0.008)	0.550 (0.033)	0.605 (0.021)	0.619 (0.025)	0.556 (0.017)	0.619 (0.021)
Isolet5	ARI	0.434 (0.019)	0.457 (0.023)	0.440 (0.020)	0.449 (0.019)	0.451 (0.019)	0.454 (0.014)	0.460 (0.015)
	F*	0.559 (0.015)	0.578 (0.025)	0.552 (0.016)	0.563 (0.016)	0.570 (0.020)	0.573 (0.015)	0.579 (0.012)
	NMI	0.704 (0.006)	0.711 (0.010)	0.708 (0.013)	0.706 (0.008)	0.708 (0.008)	0.710 (0.013)	0.712 (0.006)
	Purity	0.547 (0.011)	0.565 (0.025)	0.536 (0.023)	0.536 (0.016)	0.549 (0.022)	0.567 (0.019)	0.561 (0.013)
Urban	ARI	0.115 (0.014)	0.121 (0.005)	0.117 (0.002)	0.109 (0.003)	0.120 (0.012)	0.107 (0.030)	0.129 (0.000)
	F*	0.353 (0.011)	0.358 (0.002)	0.364 (0.005)	0.365 (0.006)	0.367 (0.014)	0.346 (0.033)	0.368 (0.000)
	NMI	0.278 (0.017)	0.286 (0.010)	0.271 (0.009)	0.273 (0.003)	0.280 (0.013)	0.266 (0.037)	0.303 (0.000)
	Purity	0.371 (0.013)	0.375 (0.000)	0.373 (0.000)	0.354 (0.003)	0.380 (0.013)	0.362 (0.036)	0.369 (0.000)
Iris	ARI	0.743 (0.000)	0.743 (0.000)	0.667 (0.129)	0.780 (0.007)	0.781 (0.006)	0.716 (0.000)	0.783 (0.005)
	F*	0.899 (0.000)	0.899 (0.000)	0.850 (0.090)	0.917 (0.003)	0.918 (0.003)	0.885 (0.000)	0.919 (0.002)
	NMI	0.758 (0.000)	0.758 (0.000)	0.709 (0.062)	0.763 (0.006)	0.765 (0.004)	0.742 (0.000)	0.767 (0.008)
	Purity	0.900 (0.000)	0.900 (0.000)	0.848 (0.096)	0.918 (0.003)	0.919 (0.003)	0.887 (0.000)	0.919 (0.002)
Vehicle	ARI	0.123 (0.000)	0.123 (0.000)	0.117 (0.022)	0.117 (0.003)	0.129 (0.004)	0.123 (0.000)	0.141 (0.002)
	F*	0.451 (0.000)	0.451 (0.000)	0.462 (0.033)	0.461 (0.003)	0.471 (0.003)	0.455 (0.000)	0.479 (0.002)
	NMI	0.172 (0.000)	0.172 (0.000)	0.159 (0.017)	0.152 (0.003)	0.166 (0.005)	0.187 (0.000)	0.184 (0.003)
	Purity	0.453 (0.000)	0.453 (0.000)	0.439 (0.022)	0.437 (0.004)	0.450 (0.003)	0.453 (0.000)	0.458 (0.002)
USPS	ARI	0.521 (0.014)	0.527 (0.010)	0.536 (0.000)	0.562 (0.005)	0.555 (0.001)	0.526 (0.000)	0.571 (0.000)
	F*	0.682 (0.016)	0.688 (0.012)	0.697 (0.000)	0.720 (0.004)	0.714 (0.002)	0.689 (0.000)	0.725 (0.000)
	NMI	0.605 (0.006)	0.607 (0.004)	0.613 (0.000)	0.611 (0.001)	0.612 (0.001)	0.605 (0.000)	0.610 (0.000)
	Purity	0.718 (0.017)	0.725 (0.012)	0.734 (0.000)	0.724 (0.017)	0.732 (0.020)	0.725 (0.000)	0.718 (0.000)

. The values are averaged over 10 trials with random initializations. The standard deviations are given after the means, and the best results are shown in bold. Note that all experimental results are calculated by the obtained U for different methods. In addition, the corresponding running times of the PFCM-type algorithms are illustrated in Figure 4.

From the experimental results in Table 4 and Figure 4, the following conclusions can be made:

• Comparing the fourth and last columns of each dataset, MMPFCM consistently outperforms PFCM across all four clustering evaluation metrics on ten datasets. In addition, MMPFCM outperforms PFCM in terms of the ARI and FM on the ORL and USPS datasets, and MMPFCM outperforms PFCM in terms of purity on the ORL dataset. These results indicate the superiority of the proposed method under the same initialization conditions.
• PFCM-type clustering algorithms have better clustering results than FCM-type clustering algorithms on the SCADI, balance, Yale32, Yale64, Iris, and USPS datasets. This is because PFCM-type clustering algorithms are better equipped to handle data with noise and outliers.
• The total running time of FW-S-PFCM is the lowest on these twelve datasets, but this is achieved under the condition of tuning more hyperparameters. The time taken by MMPFCM and PFCM is at the same linear level, which confirms that their time complexities have the same linear relationship.

7. Conclusions

This article mainly revisits PFCM using the MM method. By eliminating the variable V, we obtain a simplified model with fewer variables, and then we provide a solution using the MM method by proving the convexity of the proportional term in this model. Through analysis, the new derivation method is shown to be equivalent to the original PFCM. In addition, we introduce a new intermediate variable s to transform the simplified model with a proportional term into an easily solvable equivalent form. Then, we design an iterative sub-problem using the MM method. For convenience, we refer to this method as MMPFCM. The complexity analysis indicates that MMPFCM and PFCM share the same computational complexity. However, MMPFCM uses the intermediate variables s of $c\times 1$ size instead of the variable V of $p\times c$ size to update U and T, resulting in smaller space complexity. Extensive experiments have shown that MMPFCM converges to a better local minimum compared to PFCM. In addition, this new updating approach enhances the flexibility of the original model, allowing for fine-tuning the number of iterations in the inner loop to achieve better performance.

In future work, we will try to apply this strategy to other PFCM-type clustering algorithms for the purpose of obtaining a better local minimum. In addition, since this method has the same computational complexity as the original PFCM, we are also committed to researching an accelerated version of MMPFCM to reduce its running time.