Fuzzy Clustering Approaches Based on Numerical Optimizations of Modified Objective Functions

Abstract

Fuzzy clustering is a form of unsupervised learning that assigns the elements of a dataset into multiple clusters with varying degrees of membership rather than assigning them to a single cluster. The classical Fuzzy C-Means algorithm operates as an iterative procedure that minimizes an objective function defined based on the weighted distance between each point and the cluster centers. The algorithm operates decently in many datasets but struggles with datasets that exhibit irregularities in overlapping shapes, densities, and sizes of clusters. Meanwhile, there is a growing demand for accurate and scalable clustering techniques, especially in high-dimensional data analysis. This research work aims to address these infirmities of the classical fuzzy clustering algorithm by applying several modification approaches on the objective function of this algorithm. These modifications include several regularization terms aiming to make the algorithm more robust in specific types of datasets. The optimization of the modified objective functions is handled based on several numerical methods: gradient descent, root mean square propagation (RMSprop), and adaptive mean estimation (Adam). These methods are implemented in a Python environment, and extensive experimental studies are conducted, following carefully the steps of dataset selection, algorithm implementation, hyper-parameter tuning, picking the evaluation metrics, and analyzing the results. A comparison of the features of these algorithms on various datasets is carefully summarized.

1. Introduction

Unsupervised learning is a sub-domain of machine learning where the algorithms operate on data without prior information being provided in the form of labeled responses or predefined classes. Several important forms of unsupervised learning are developed and successfully applied, including clustering, association rule mining, latent variable models, anomaly detection, etc. Clustering aims at finding unknown internal structures in datasets, grouping the data instances into groups based on their similarities with each other [1]. Besides the classical crisp clustering methods, another significant domain of clustering is soft clustering, where the boundaries between the clusters do not have to be crisply separated; thus, overlapping clusters and partial memberships of data instances into several clusters are naturally occurring. An important form of soft clustering is the Fuzzy C-Means (FCM), a clustering algorithm that extends the classical K-Means algorithm, allowing partial memberships of data instances into multiple clusters with varying membership degrees [2]. The algorithm operates based on the minimization of an objective function expressed as the sum of weighted distances between data instances and cluster centroids. The modus operandi of the fuzzy clustering algorithm is an iterative procedure where the new centroids are re-evaluated based on the current membership degrees of each data instance to the clusters, and the memberships are updated based on the distances of each data instance from the new centroids [3]. This dual updating procedure continues until the changes fall below a predefined threshold, thus achieving convergence. Finally, each data instance is assigned a membership degree, which is a real value in the [0, 1] interval, in such a way allowing for a nuanced representation of the intrinsic structures of the dataset. Applications of fuzzy clustering extend to a myriad of disciplines, including image processing, customer profiling, bioinformatics, text organization, etc. [4].

Despite its overall utility, the classical Fuzzy C-Means algorithm may struggle with datasets that are characterized by complex distributions, such as clusters of varying sizes or densities, overlapping clusters, and the presence of noise and outliers. On such occasions, the application of the standard Fuzzy C-Means algorithm may lead to suboptimal clustering outcomes, where clusters are poorly defined and the membership values of data instances may not be properly assigned into the clusters [5]. To address these issues, ideas of modifying the objective function have been occasionally suggested in the machine learning research community, incorporating additional parameters, employing different distance measures, and adjusting penalization terms [6]. For instance, the usage of regularization parameters can diminish the influence of noise and outliers, thus improving the stability of the clustering procedure [7]. Furthermore, the incorporation of spatial characteristics or density-related information can assist in providing a better definition of the clusters and refine the adjustments for the overlapping clusters [8].

The idea of modifying the objective function of Fuzzy C-Means or K-Means in order to make new versions of the respective algorithms that perform better on specific datasets is known in the machine learning community. There is a variety of modification forms, and also there are several ways how each modification is processed into an iterative procedure [9]. Once the objective function is modified, the iterative procedure for updating the centers of the clusters and the membership values should be properly adapted. There are two main approaches towards its optimization: employing the Lagrange multipliers or using numerical optimization directly on the modified objective function. When Lagrange multipliers are employed, an iterative procedure is generated for updating the centers of the clusters and membership values, but the generation of this procedure may become a challenging differentiation problem for sophisticated modifications of the objective function [10]. Conversely, a numerical optimization algorithm can be applied directly to the objective function without requiring an explicit analytical calculation of the partial derivatives.

In this research work, several modification forms of the objective function will be discussed, providing the overall motivation for each modification. The modified objective functions will be optimized based on the gradient descent numerical method and some of its variants, like root mean square propagation (RMSprop) and adaptive moment estimation (Adam). A diverse set of experimental procedures will be conducted on the modified algorithms, intending to expose the strengths and the vulnerabilities of each algorithm. Working with numerical optimization will provide more flexibility in the spectrum of modifications applicable to the objective functions, as no analytical evaluations will be demanded [11].

2. Classical FCM and Modifications on Its Objective Function

The classical Fuzzy C-Means algorithm (FCM), developed by Dunn and refined by J. C. Bezdek, is the cornerstone of the fuzzy cluster analysis. It is a generalization of the well-known K-Means algorithm, involving the notion of fuzziness degree, consequently allowing for partial memberships of the data instances in the generated clusters. The algorithm is fundamentally an unsupervised procedure, as no prior information (in the form of predefined classes or labeled responses) is provided. FCM works as an iteration scheme, aiming to achieve a nonlinear optimization of an objective function defined as

$$J\left(X, U, V; m, C\right)={\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^C}{\mu }_{ij}^m{d}^2\left(x_i, c_j\right)$$

(1)

Here, X denotes the dataset, $U$ denotes the membership matrix, and $V$ denotes the cluster centers. Hyper-parameter $m$ is the fuzziness degree (or fuzzy exponent) for controlling the level of fuzziness in the generated clusters [12]. Generally, it is given values larger than 1 (but not larger than 2.5). The greater the value of $m$, the more will be the fuzziness and less distinct the generated clusters. In addition, the hyper-parameter $C$ represents the number of clusters, $n$ is the size of the dataset, $x_i$ is the $i$-th instance, $c_j$ is the center of the $j$-th cluster, and ${\mu }_{ij}$ is the membership degree of the $i$-th element in the $j$-th cluster. Two other hyper-parameters of this algorithm are the scale of tolerance $Tol$ and the distance metric (d) [13].

The objective function is optimized using Lagrange multipliers to form a dual updating scheme that re-calculates the cluster centers and the membership degrees of the data instances into the clusters, as in the following pseudo-code [14]:

• Initialize the centers of the clusters, assigning as values random data instances.
• Initialize the partition matrix (assigning 0 to all its entries).
• Set $k = 1$ (number of iterations).
• Evaluate the ${\mathrm{d}}_{\mathrm{i}\mathrm{j}}$ values (i.e., the distances of instances from cluster centers).
• Update membership values as ${\mathrm{\mu }}_{\mathrm{i}\mathrm{j}}={\displaystyle \frac{{\mathrm{d}}_{\mathrm{i}\mathrm{j}}^{-\frac{2}{\mathrm{m}-1}}}{{\displaystyle {\sum }_{\mathrm{k}=1}^{\mathrm{c}}}{\displaystyle {\mathrm{d}}_{\mathrm{i}\mathrm{k}}^{-\frac{2}{\mathrm{m}-1}}}}}$
• Update the centers of the clusters as ${\mathrm{c}}_{\mathrm{i}}={\displaystyle \frac{{\displaystyle {\sum }_{\mathrm{j}=1}^{\mathrm{n}}}{\displaystyle {\mathrm{\mu }}_{\mathrm{i}\mathrm{j}}^{\mathrm{m}}}{\mathrm{X}}_{\mathrm{j}}}{{\displaystyle {\sum }_{\mathrm{j}=1}^{\mathrm{n}}}{\displaystyle {\mathrm{\mu }}_{\mathrm{i}\mathrm{j}}^{\mathrm{m}}}}}$
• $k = k+1$ (increment the number of iterations).
• If $‖{\mathrm{U}}_{\mathrm{k}-1}-{\mathrm{U}}_{\mathrm{k}-2}‖>Tol$ jump to step 4.
• END.

Although the FCM algorithm has a multitude of successful applications, there are conversely many datasets exhibiting complex distributions, such as overlapping clusters and variety in the size and density of the clusters, where the FCM would underperform, generating clusters that are poorly defined, and the membership values of data instances not being properly assigned to the clusters [15]. In this research work, we are going to consider several modifications (described in the following paragraphs) that are either novel in the way the objective function is modified or in the way the iterative procedure is constructed. In each modification, an extra term is involved in the original objective function of the original FCM, aiming to divert the tendencies of the original algorithm to make them more adaptable to certain types of datasets. In all the cases, the coefficient λ, known as the regularization weight, controls the strength of the additional term, thus representing a hyper-parameter that may be tuned.

2.1. An $L_2$ Regularization Term

One of the modifications on the objective function is adding an $L_2$ regularization term to refrain the centroids of the clusters from obtaining large values, thus helping to avoid overfitting. The objective function becomes:

$$J\left(X, U, V; m, C, \lambda \right)={\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^C}{\displaystyle {\mu }_{ij}^m}{d}^2\left(x_i, c_j\right)+\lambda {\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^C}{\displaystyle {\mu }_{ij}^m}{‖c_j‖}^2$$

(2)

Here, ${‖c_j‖}^2$ represents the squared Euclidean norm of the $j$-th cluster center, thus penalizing centers that lie far from the origin. This modification is very useful once the data are previously normalized using z-score or min–max normalizations, which centralize the original data points. Application of the regularization term stimulates simpler models, typically leading to better generalization of data and less susceptibility to noise and outliers [16]. Furthermore, this modification makes the gradient descent procedure more stable, as the regularization adds a smooth constraint in the optimization framework (avoiding local minima). The idea of this modification has been used in hard clustering by W. Sun et al. to address the high-dimensional noise anomaly.

2.2. An Entropy Term

Another modification of the objective function involves an entropy term, which would stimulate more balanced membership degrees of data instances into the clusters:

$$J\left(X, U, V; m, C, \mathrm{\lambda }\right)={\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^C}{\displaystyle {\mu }_{ij}^m}{d}^2\left(x_i, c_j\right)-\mathrm{\lambda }{\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^C}{\mu }_{ij}\log {\mu }_{ij}$$

(3)

This term would penalize extreme membership values (close to 1 or close to 0), so influencing the data to be more distributed into the clusters. This is of particular usefulness in scenarios where data instances are expected to belong to multiple clusters, avoiding the tendency towards one or a small number of clusters in datasets varying in density and spread of the intrinsic clusters [17]. The idea has been used in several works, mostly in the form of a cluster validity measure, but also in the form of objective function modification being optimized via Lagrange multipliers rather than based on numerical methods [18].

2.3. A Sparsity-Inducing Term

Contrary to the previous approach, it is possible to modify the objective function by adding a sparsity-inducing term, which will manifest a tendency towards crisper and more interpretable clusters:

$$J\left(X, U, V; m, C, \lambda \right)={\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^C}{\displaystyle {\mu }_{ij}^m}{d}^2\left(x_i, c_j\right)-\lambda {\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^C}{\mu }_{ij}$$

(4)

This penalization term would cause the data instances to belong to fewer clusters but with higher membership values. This is of considerable usefulness for feature selection in high-dimensional datasets or interpretability in scenarios like image analysis, customer segmentation, etc. [19]. Several variants of this idea have been presented before, primarily aiming to rectify the robustness in the presence of noise and outliers and the clustering of high-dimensional data.

2.4. A Penalty for Large-Sized Clusters

This modification of the objective function aims at keeping the generated clusters in comparable sizes to each other. It is involving an entropy term, which would stimulate more balanced membership degrees of data instances into the clusters:

$$J\left(X, U, V; m, C, \mathrm{\lambda }\right)={\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^C}{\displaystyle {\mu }_{ij}^m}{d}^2\left(x_i, c_j\right)+\mathrm{\lambda }{\displaystyle \sum _{i=1}^n}{\left({\displaystyle \sum _{j=1}^C}{\mu }_{ij}\right)}^2$$

(5)

The squared sum of the membership values acts efficaciously as a penalty for clusters growing large by accumulating a high sum of memberships. The idea of penalizing the large clusters has been employed multiple times, and it is comprised within the more general idea of clustering with constraints [20]. The squared sum of memberships is one of the approaches in this context aiming to ensure that the generated clusters will remain balanced.

2.5. A Spatial Regularization Term

This modification of the objective function aims at keeping the generated clusters smooth in space. It is involving a spatial regularization term, which would stimulate points close to each other to have similar values of the memberships into the clusters.

$$J\left(X, U, V; m, C, \mathrm{\lambda }\right)=\sum _{i=1}^n\sum _{j=1}^C{\mu }_{ij}^m{d}^2\left(x_i, c_j\right)+\mathrm{\lambda }{\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^C}{\displaystyle \sum _{k\in N\left(i\right)}}{\left|{\mu }_{ij}-{\mu }_{kj}\right|}^2$$

(6)

Here, $N\left(i\right)$ represents the neighbourhood set of the data instance $x_i$, i.e., the set of points that are in spatial proximity to $x_i$. This may be determined by an additional radius parameter (the radius of proximity) or naturally based on the adjacency concept in the representative data structure, such as adjacent pixels in an image [21]. The coefficient $\lambda$ controls the strength of the spatial regularization term, thus representing a hyper-parameter that may be tuned. The idea of this modification has been previously used in image analysis, adapting it to the specific scenarios rather than aiming to provide a more general approach [22].

3. Optimization Based on Numerical Techniques

The classical approach based on Lagrange multipliers employs extra variables in order to seamlessly integrate the constraints into the optimization procedure, thus guaranteeing that the constraints are continuously satisfied throughout the procedure. Afterwards, through partial differentiation, the iterative procedure is constructed. The generation of this procedure is well known for the classical forms of the objective function, but it may become a challenging problem for sophisticated modifications of the objective function. In these circumstances, numerical optimization methods may be utilized, providing improved flexibility [23]. In this work, three numerical optimization methods are employed: gradient descent, root mean square propagation (RMSprop) and adaptive moment estimation (Adam). In the following sections, for each of these methods the general theoretical principles are explained; afterwards, the applications of these principles for the optimization of the modified objective functions for fuzzy clustering are elaborated. For each numerical method, the iterative workflow for optimizing the objective function is provided.

3.1. Gradient Descent

Gradient descent is one of the most widely used optimization techniques, especially in the context of machine learning. The fundamental principle of this technique is to iteratively update parameters of interest by tracking the direction of the steepest descent in the objective function that is intended to be minimized. The gradient (i.e., partial derivative) of the objective function with respect to parameters of interest is evaluated, and the parameter values are updated by moving in the opposite direction of the gradient [24].

At each iteration, the algorithm calculates the gradient (the partial derivative) of the objective function with respect to each parameter of interest and updates the parameter values by moving in the opposite direction of the gradient. The learning rate (i.e., the step size) is a parameter that controls the size of each step. For smaller values of the learning rate, the convergence would be slower but would avoid overshooting the minimum. For larger values of the learning rate, the convergence would be reached faster, but there is an augmented risk of instability [25].

The gradient descent technique is helpful in designating the iterative procedure for complex modifications of the objective function. The general form of the modified objective functions considered in this paper was

$$J_{Modified}=J_{FCM}+\lambda R\left(U,c_1,c_2,\dots ,c_C\right)={\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^C}{\displaystyle {\mu }_{ij}^m}{d}^2\left(x_i, c_j\right)+\lambda R\left(U,c_1,c_2,\dots ,c_C\right)$$

(7)

Therefore, based on the gradient descent technique, the memberships of the data instances in the clusters are updated according to the equation:

$${\displaystyle {\mu }_{ij}^{\left(k+1\right)}}={\displaystyle {\mu }_{ij}^k}-\mathrm{\eta }{\displaystyle \frac{\partial \mathrm{ }{\mathrm{J}}_{\mathrm{M}\mathrm{o}\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{e}\mathrm{d}}}{\partial \mathrm{ }{\mu }_{ij}}}={\displaystyle {\mu }_{ij}^k}-\mathrm{\eta }·\mathrm{m}·{\displaystyle {\mu }_{ij}^{m-1}}{d}^2\left(x_i, c_j\right)-\mathrm{\eta }·\lambda ·{\displaystyle \frac{\partial \mathrm{ }\mathrm{R}}{\partial \mathrm{ }{\mu }_{ij}}}$$

(8)

Here, ${\displaystyle {\mu }_{ij}^k}$ represents the membership value at the $k$-th iteration, η represents the learning rate, and $\partial$ represents the partial derivative. Additionally, in order to guarantee that the constraint about the memberships being values in [0, 1] summing up to 1, an extra normalization step is carried out:

$${\displaystyle {\mu }_{ij}^{\left(k+1\right)}}={\displaystyle \frac{{\displaystyle {\mu }_{ij}^{\left(k+1\right)}}}{\sum {\displaystyle {\mu }_{ij}^{\left(k+1\right)}}}}$$

(9)

Furthermore, let us consider the distances being evaluated by the Euclidean distance measure, so

$$d^2\left(x_i, c_j\right)={\left(x_i-c_j\right)}^2$$

Then, the update of centers based on the gradient descent will be

$${\displaystyle c_j^{\left(k+1\right)}}={\displaystyle c_j^{\left(k\right)}}-\mathrm{\eta }{\displaystyle \frac{\partial \mathrm{ }{\mathrm{J}}_{\mathrm{M}\mathrm{o}\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{e}\mathrm{d}}}{\partial \mathrm{ }{c}_j}}={\displaystyle c_i^{\left(k\right)}}-2\mathrm{\eta }·{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}}{\displaystyle {\mu }_{ij}^m}·\left(x_i-c_j\right)-\mathrm{\eta }·\lambda ·{\displaystyle \frac{\partial \mathrm{R}}{\partial \mathrm{ }{c}_{ij}}}$$

(10)

The general modus operandi of the fuzzy clustering algorithm based on numerical optimizations of the modified objective function will be similar to the classical FCM algorithm, the main difference relying on the way the dual updates are handled. The procedure can be summarized by the following pseudo-code:

• Initialize the centers of the clusters, assigning as values random data instances.
• Initialize the partition matrix (assigning 0 to all its entries).
• Set k = 1 (number of iterations).
• Update membership values according to Equation (8).
• Normalize the membership values according to Equation (9).
• Update the centers of the clusters according to Equation (10).
• k = k + 1 (increment the number of iterations).
• If $‖{\mathrm{U}}_{\mathrm{k}}-{\mathrm{U}}_{\mathrm{k}-1}‖>Tol,$ jump to step 4.
• END.

The given pseudo-code is generic, thus being applicable for every modification form of the objective function being employed. According to the specific type of modification, the details rely on the evaluation of the ${\displaystyle \frac{\partial \mathrm{R}}{\partial {\mu }_{ij}}}$ and ${\displaystyle \frac{\partial \mathrm{R}}{\partial c_{ij}}}$ terms.

3.2. Root Mean Square Propagation (RMSprop)

Although gradient descent is one of the most widely used optimization techniques, especially in the context of machine learning, it may suffer from instabilities due to its sensitivity to the learning rate in complex landscapes of the objective functions and its sensitivity to noise. As gradient descent uses a fixed learning rate, this may affect the process when moving through a steep valley, causing oscillations through the sides of the valley or even divergent updates [26]. Overall, a constant learning rate is not appropriate for complex landscapes, as a small value of the learning rate would cause a slow convergence on a flat surface, while a relatively large learning rate would cause oscillations or divergent updates in a steep terrain [27].

The root mean square propagation method modifies the gradient descent method by scaling the learning rate based on the recent values of the gradients. Thus, the learning rate is dynamically adjusted to properly adapt to the current landscape. The weighted quadratic average of the previous gradients is evaluated and applied to scale the learning rate during the iterations, so for a parameter θ, the update would be controlled by the equation:

$${\theta }^{\left(k+1\right)}={\theta }^{\left(k\right)}-{\displaystyle \frac{\eta }{\sqrt{{E\left[g^2\right]}^{\left(k\right)}+ϵ}}}·g^{\left(k\right)}$$

(11)

Here, η represents the learning rate, g^(t) represents the gradient at the $k$-th iteration, E[g²]^(k) represents the weighted quadratic mean of the recent gradients, and $\epsilon$ is a small constant utilized to avoid division by zero. This modification in the update rule makes the root mean square method more stable and particularly efficient in optimizing complex objective functions. Furthermore, it operates robustly in circumstances with noisy or fluctuating gradients, thus requiring significantly less tuning of the learning rate [28].

The root mean square method can be applied in optimizing the objective functions for fuzzy clustering, especially the more complex modification mentioned in the previous sections. So, the gradient of the membership values would be evaluated as

$${\displaystyle g_{{\mu }_{ij}}^{\left(k\right)}}={\displaystyle \frac{\partial \mathrm{ }{\mathrm{J}}_{\mathrm{M}\mathrm{o}\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{e}\mathrm{d}}}{\partial \mathrm{ }{\mu }_{ij}}}=\mathrm{m}·{\displaystyle {\mu }_{ij}^{m-1}}{d}^2\left(x_i, c_j\right)+\lambda ·{\displaystyle \frac{\partial \mathrm{ }\mathrm{R}}{\partial \mathrm{ }{\mu }_{ij}}}$$

(12)

Furthermore, the weighted quadratic average of the gradients would be evaluated (employing a decay factor β, typically having a value close to 0.9) as

$${E\left[{\displaystyle g_{{\mu }_{ij}}^2}\right]}^{\left(k+1\right)}=\beta ·{E\left[{\displaystyle g_{{\mu }_{ij}}^2}\right]}^{\left(k\right)}+\left(1-\beta \right)·{\left({\displaystyle g_{{\mu }_{ij}}^{\left(t\right)}}\right)}^2$$

(13)

In a very similar way, the principle of the moving average characterizing the root mean squares method is applied for the update of the centers. The gradients of the cluster centers are evaluated based on the partial derivatives of the modified objective functions as

$${\displaystyle g_{c_j}^{\left(k\right)}}={\displaystyle \frac{\partial \mathrm{ }{\mathrm{J}}_{\mathrm{M}\mathrm{o}\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{e}\mathrm{d}}}{\partial \mathrm{ }{c}_j}}=2\left({\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}}{\displaystyle {\mu }_{ij}^m}·\left(x_i-c_j\right)\right)+\lambda ·{\displaystyle \frac{\partial \mathrm{R}}{\partial \mathrm{ }{c}_{ij}}}$$

(14)

The modus operandi of the fuzzy clustering algorithm based on optimization of the objective functions via root mean square propagation will remain the same as the classical FCM, with the major differences relying on the way the dual updates are handled. The number of steps is naturally increased as there are several intermediate computations necessary for both the calculations of the membership values and the calculations of the new centers. The procedure can be summarized by the following pseudo-code:

• Initialize the centers of the clusters, assigning as values random data instances.
• Initialize the partition matrix (assigning 0 to all its entries).
• Set $k = 1$ (number of iterations).
• Update the gradients of the memberships according to Equation (12).
• Update the weighted quadratic average according to Equation (13).
• Update memberships, applying Equation (11) for $\theta ={\mu }_{ij}$, so

  $${\displaystyle {\mu }_{ij}^{\left(k+1\right)}}={\displaystyle {\mu }_{ij}^k}-{\displaystyle \frac{\eta }{\sqrt{{E\left[{\displaystyle g_{{\mu }_{ij}}^2}\right]}^{\left(t\right)}+ϵ}}}·{\displaystyle g_{{\mu }_{ij}}^{\left(k\right)}}$$
• Normalize the membership values according to Equation (9).
• Update the gradients of the centers according to Equation (14).
• Update the centers applying Equation (11) for $\theta =c_{ij}$, so

  $${\displaystyle c_j^{\left(k+1\right)}}={\displaystyle c_j^k}-{\displaystyle \frac{\eta }{\sqrt{{E\left[{\displaystyle g_{c_{ij}}^2}\right]}^{\left(t\right)}+ϵ}}}·{\displaystyle g_{c_j}^{\left(k\right)}}$$
• $k = k+1$ (increment the number of iterations)
• If $‖{\mathrm{U}}_{\mathrm{k}}-{\mathrm{U}}_{\mathrm{k}-1}‖>Tol,$ jump to step 4.
• END.

3.3. Adaptive Moment Estimation (Adam)

The adaptive moment estimation (Adam) technique intertwines ideas from root mean square propagation and momentum. It is a popular optimization algorithm with many applications in the domain of machine learning. The algorithm tracks simultaneously two moments: the first moment (i.e., the mean) and the second moment (i.e., the variance) [26]. The rationale behind this approach is that the first moment will assist in maintaining the direction by reducing the oscillations while moving along the landscape, and the second moment will assist in properly adapting the learning rates based on the previous gradient data. The update of the parameters will be handled employing gradients evaluated using both the first and second moments, according to [29]:

$${\theta }^{\left(k+1\right)}={\theta }^{\left(k\right)}-{\displaystyle \frac{\eta ·{\widehat{m}}^{\left(k\right)}}{\sqrt{{\widehat{v}}^{\left(k\right)}+ϵ}}}$$

(15)

Here, η represents the learning rate, ${\widehat{m}}^{\left(k\right)}$ represents the first moment at the k-th iteration after bias correction, ${\widehat{v}}^{\left(k\right)}$ represents the second moment at the k-th iteration after bias correction, and ɛ is a small constant utilized to avoid division by zero. The update principles are similar to the root mean squared method but are now applied twofold [30]:

$$\begin{gathered} m^{\left(k+1\right)}={\beta }_1·m^{\left(k\right)}+\left(1-{\beta }_1\right)·g^{\left(k\right)} \\ {v}^{\left(k+1\right)}={\beta }_2·v^{\left(k\right)}+\left(1-{\beta }_2\right)·{\left(g^{\left(k\right)}\right)}^2 \end{gathered}$$

(16)

Furthermore, both moments are bias-corrected, yielding

$$\begin{gathered} {\widehat{m}}^{\left(k+1\right)}={\displaystyle \frac{m^{\left(k+1\right)}}{1-{{\beta }_1}^{t+1}}} \\ {\widehat{v}}^{\left(k+1\right)}={\displaystyle \frac{v^{\left(k+1\right)}}{1-{{\beta }_2}^{t+1}}} \end{gathered}$$

(17)

When optimizing the modified objective functions based on adaptive moment estimation, the update principles employing two moments will be applied for both updating the membership values and the cluster centers. The primary structure of the algorithm remains the same, consisting of an iterative dual update on membership values and the cluster centers; however, the number of intermediate steps is significantly increased. The complete procedure is described by the following pseudo-code:

• Initialize the centers of the clusters, assigning as values random data instances.
• Initialize the partition matrix (assigning 0 to all its entries).
• Set $k = 1$ (number of iterations).
• Update the gradients of the memberships according to Equation (12).
• Update the first and second moments for each membership value according to Equation (16).
• Apply the bias correction for both moments of the membership values according to Equation (17).
• Update the membership values, applying Equation (15) for $\theta ={\mu }_{ij}$, so

  $${\displaystyle {\mu }_{ij}^{\left(k+1\right)}}={\displaystyle {\mu }_{ij}^{\left(k\right)}}-{\displaystyle \frac{\eta ·{\displaystyle {\widehat{m}}_{{\mu }_{ij}}^{\left(k+1\right)}}}{\sqrt{{\displaystyle {\widehat{v}}_{{\mu }_{ij}}^{\left(k+1\right)}}+ϵ}}}$$
• Normalize the membership values according to Equation (9).
• Update the gradients of the centers according to Equation (14).
• Update the first and second moments of the centers according to Equation (16).
• Apply the bias correction for both moments of the centers according to Equation (17).
• Update the centers, applying Equation (15) for $\theta =c_{ij}$, so

  $${\displaystyle c_j^{\left(k+1\right)}}={\displaystyle c_j^{\left(k\right)}}-{\displaystyle \frac{\eta ·{\displaystyle {\widehat{m}}_{c_j}^{\left(k+1\right)}}}{\sqrt{{\displaystyle {\widehat{v}}_{c_j}^{\left(k+1\right)}}+ϵ}}}$$
• $k = k+1$ (increment the number of iterations).
• If $‖{\mathrm{U}}_{\mathrm{k}}-{\mathrm{U}}_{\mathrm{k}-1}‖>Tol,$ jump to step 4.
• END.

4. Experimental Studies

A series of experimental procedures are carried out to analyze the modified fuzzy clustering algorithms. These algorithms were applied on several slightly distorted versions of benchmark datasets of the UCI Machine Learning Repository (where some artificially noise data are added) in order for their performance to be evaluated. The employed datasets in these studies are Banknotes, Liver, Yeast, Dermatology, Vehicle Silhouettes, and LANDSAT Satellite with an additional quantity of 2–3% artificial noise points, which are randomly positioned at a distance from the cluster centers being 5–15% larger than the distance of the farthest genuine point from the respective cluster center [31]. In the original datasets, these artificially added noise points are labeled according to the cluster the random generation relied on, which is expected to be the closest cluster, too, though not mathematically guaranteed. As these points typically lie near cluster boundaries, they are expected to exhibit partial memberships across clusters.

The Banknotes dataset contains four numerical features derived from wavelet-transformed images of a group of Swiss banknotes, along with the class labels indicating for each banknote whether it is genuine or counterfeit.

The Liver dataset (Indian Liver Patient Dataset) contains two demographic features (age and sex) and eight biochemical test results related to liver health, along with the class label indicating whether the patient is healthy or has a liver disorder.

The Yeast dataset contains eight numerical features representing various physicochemical properties of proteins, such as sequence length and localization signals, as well as the class, which is the sub-cellular localization of proteins across ten possible classes.

The Dermatology dataset contains patient data about some skin diseases manifested with redness and scaling. The dataset comprises 34 clinical attributes, including histological, pathological, and subjective features, used to classify six types of skin diseases.

The Vehicle Silhouettes dataset contains 18 numerical features extracted from the silhouettes of vehicle images. It comprises instances of four classes (Double Decker Bus, Opel Manta 400, Saab 9000, and Chevrolet Van).

The LANDSAT Satellite dataset contains multi-spectral values of pixels from satellite images. It comprises 36 numerical features derived from 3 × 3 pixel neighborhoods in satellite images and the class representing the land cover type (e.g., red soil, grey soil, etc.).

The details of the original datasets, including the number of features, number of instances, and number of clusters, are summarized in

Table 1. Summary of the employed datasets.

Dataset	# Features	# Instances	# Clusters
Banknotes	5	1372	2
Liver	10	583	2
Yeast	8	1484	10
Dermatology	34	366	6
Vehicle Silhouettes	18	846	4
LANDSAT Satellite	36	6435	7

.

Although the categorization into classes for the instances of these datasets is well known, this information will not be provided to the clustering algorithms, so they will operate in an unsupervised way. The process starts with the generation of the clustering results, which are then compared with the known natural classes of the dataset to evaluate the performance of the algorithms. The number of clusters hyper-parameter is set to the respective natural number of clusters of each dataset. The regularization hyper-parameter λ is tuned via grid search among the candidate values [0.1, 0.25, 0.5, 1, 2, 4]. The learning rate is set to 0.01, but it is progressively decreased by 10% in case of divergence. Beyond accuracy assessment, the algorithms will also be examined for their sensitivity to noise, allowing for a more thorough evaluation of their performance. Two criteria are employed for the performance evaluation, which are the fuzzy adjusted Rand index (Fuzzy ARI) and fuzzy normalized mutual information (Fuzzy NMI).

The fuzzy adjusted Rand index is a generalization of the Rand index, adjusting this metric to operate with soft (partial) memberships of the instances in the clusters. The similarity is assessed by comparing the fuzzy partition matrix $U$ with the real class labels $C$, based on mutual agreements of memberships weighted by the membership degrees. While adjusted Rand index measures are based on hard assignments, Fuzzy ARI employs partial memberships, enabling the capture of soft boundaries present in the fuzzy clustering approaches [32]. If the two compared partitions are $U=\left[{\mu }_{ij}\right]$ and $V=\left[v_{ij}\right]$, for any pair of instances $x_k$ and $x_l$, their similarity is evaluated according to partition $U$ and their similarity according to partition $V$, respectively, as

$${\displaystyle s_{kl}^{\left(U\right)}}={\displaystyle \sum _{r=1}^n}{\mu }_{kr}·{\mu }_{lr}$$

$${\displaystyle s_{kl}^{\left(V\right)}}={\displaystyle \sum _{r=1}^n}{v}_{kr}·v_{lr}$$

Based on these mutual similarity values, the adjusted fuzzy Rand index (F-ARI) is evaluated as

$$F-ARI={\displaystyle \frac{{\displaystyle {\sum }_{i<j}}{\displaystyle s_{kl}^{\left(U\right)}}·{\displaystyle s_{kl}^{\left(V\right)}}-\left({\displaystyle {\sum }_{i<j}}{\displaystyle s_{kl}^{\left(U\right)}}\right)\left({\displaystyle {\sum }_{i<j}}{\displaystyle s_{kl}^{\left(V\right)}}\right)/\left(\begin{array}{c}n\\ 2\end{array}\right)}{\frac{1}{2}\left({\displaystyle {\sum }_{i<j}}{\displaystyle s_{kl}^{\left(U\right)}}+{\displaystyle {\sum }_{i<j}}{\displaystyle s_{kl}^{\left(V\right)}}\right)-\left({\displaystyle {\sum }_{i<j}}{\displaystyle s_{kl}^{\left(U\right)}}\right)\left({\displaystyle {\sum }_{i<j}}{s}_{kl}^{\left(V\right)}\right)/\left(\begin{array}{c}n\\ 2\end{array}\right)}}$$

The evaluation score varies in the interval [−1, 1], with value 1 indicating perfect matching, value 0 indicating independence (random labelling) and negative values indicating worse than random scenarios.

Normalized mutual information (NMI) is a metric used to assess the quality of clustering in cases when the intrinsic structures are available (but not used during the clustering procedure). It evaluates the shared information between the generated clusters and the true classes of the dataset. If the two compared partitions are $U=\left[{\mu }_{ij}\right]$ and $V=\left[v_{ij}\right]$, then their fuzzy mutual information is evaluated as

$$F-MI={\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^c}{\displaystyle \sum _{k=1}^c}{\mu }_{ij}·v_{jk}·\log \left({\displaystyle \frac{{\mu }_{ij}·v_{jk}}{p_j^{\left(U\right)}·p_k^{\left(V\right)}}}\right)$$

Here, $p_j^{\left(U\right)}$ and $p_k^{\left(V\right)}$ represent respectively the average membership of cluster $j$ in partition $U$ and the average membership of cluster $k$ in partition $V$, thus:

$$p_j^{\left(U\right)}={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\mu }_{ij}$$

$$p_k^{\left(V\right)}={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{v}_{ik}$$

On the other hand, the Fuzzy Normalized Fuzzy Mutual Information guarantees that the value is between 0 and 1 by dividing the fuzzy mutual information by the sum of the entropies of the two respective partitions:

$$F-NMI=-{\displaystyle \frac{F-MI}{{\displaystyle {\sum }_{i=1}^n}{\displaystyle {\sum }_{j=1}^c}{\mu }_{ij}+{\displaystyle {\sum }_{i=1}^n}{\displaystyle {\sum }_{j=1}^c}{v}_{ij}}}$$

NMI is a broad coverage measure as it comprises both how well each cluster corresponds to a real class in the dataset (homogeneity) and how well the instances of a real class in a dataset correspond to a single cluster (completeness) [33]. The final score is normalized, guaranteeing the result to be in the interval [0, 1], making the clustering results more comparable. The value 1 is an indicator of perfect clustering, while the value 0 indicates independence between the real classes and the clustering results. Fuzzy normalized mutual information is a generalized version of NMI, which considers the membership values (treating the clusters as fuzzy sets) instead of hard assignments into clusters [32]. The evaluation proceeds by calculating the joint entropy for the fuzzy membership values.

Table 2. Fuzzy ARI and Fuzzy NMI scores for the classical FCM algorithm.

Dataset	Classical FCM
	F-ARI	F-NMI
Banknotes	0.73	0.74
Liver	0.68	0.72
Yeast	0.65	0.69
Dermatology	0.70	0.73
Vehicle Silhouettes	0.68	0.74
LANDSAT Satellite	0.69	0.76

, given below, summarizes the performance evaluation of classical Fuzzy C-means based on the fuzzy adjusted Rand index (F-ARI) and the fuzzy normalized mutual information (F-NMI). The FCM algorithm is applied 20 times, and the results shown in the table are the averages of these evaluations.

Table 3. Fuzzy ARI and Fuzzy NMI scores across modifications and optimization methods.

Dataset	Objective Function Modification	GD		RMSprop		Adam
		F-ARI	F-NMI	F-ARI	F-NMI	F-ARI	F-NMI
Banknotes	L2REG	0.73	0.76	0.74	0.77	0.75	0.78
	ENTR-T	0.76	0.79	0.77	0.8	0.78	0.81
	SP-IN	0.75	0.8	0.76	0.81	0.77	0.82
	PEN-LSC	0.78	0.81	0.79	0.82	0.80	0.83
	SPAT-REG	0.77	0.82	0.78	0.83	0.79	0.84
Liver	L2REG	0.68	0.72	0.69	0.73	0.7	0.74
	ENTR-T	0.71	0.75	0.72	0.76	0.73	0.77
	SP-IN	0.72	0.77	0.73	0.78	0.74	0.81
	PEN-LSC	0.74	0.78	0.75	0.79	0.76	0.8
	SPAT-REG	0.73	0.79	0.74	0.8	0.75	0.79
Yeast	L2REG	0.66	0.7	0.67	0.71	0.68	0.72
	ENTR-T	0.69	0.73	0.70	0.74	0.71	0.75
	SP-IN	0.70	0.74	0.71	0.75	0.72	0.76
	PEN-LSC	0.72	0.76	0.73	0.77	0.74	0.78
	SPAT-REG	0.71	0.77	0.72	0.78	0.73	0.79
Dermatology	L2REG	0.70	0.75	0.71	0.76	0.72	0.77
	ENTR-T	0.73	0.78	0.74	0.79	0.75	0.8
	SP-IN	0.74	0.79	0.75	0.8	0.76	0.81
	PEN-LSC	0.75	0.8	0.76	0.81	0.77	0.82
	SPAT-REG	0.75	0.81	0.76	0.82	0.77	0.83
Vehicle Silhouettes	L2REG	0.67	0.73	0.68	0.74	0.69	0.75
	ENTR-T	0.70	0.76	0.71	0.77	0.72	0.78
	SP-IN	0.71	0.76	0.72	0.77	0.75	0.78
	PEN-LSC	0.73	0.77	0.74	0.78	0.73	0.79
	SPAT-REG	0.72	0.78	0.73	0.79	0.74	0.8
LANDSAT Satellite	L2REG	0.71	0.77	0.72	0.78	0.73	0.79
	ENTR-T	0.74	0.8	0.75	0.81	0.76	0.82
	SP-IN	0.76	0.82	0.77	0.83	0.78	0.84
	PEN-LSC	0.76	0.82	0.77	0.83	0.78	0.84
	SPAT-REG	0.76	0.83	0.77	0.84	0.78	0.85

given below summarizes the results of the experimental procedures, where for each dataset are applied the five modified versions of the objective function, respectively L2REG (L₂ Regularization Term explained in Section 2.1), ENTR-T (Entropy term explained in Section 2.2), SP-IN (A Sparsity-Inducing Term explained in Section 2.3), PEN-LSC (A Penalty For Large Sized Clusters explained in Section 2.4), and SPAT-REG (Spatial Regularization Term explained in Section 2.5). For each of the modified versions, firstly, three numerical methods are applied: gradient descent (GD), root mean square propagation (RMSprop), and adaptive moment estimation (Adam). In order to provide a robust evaluation, each method is repeated 20 times, and the values shown in the table are the average scores obtained after these executions.

The Fuzzy ARI and Fuzzy NMI scores are visualized in two different line charts, as shown in Figure 1 and Figure 2 below. The abbreviations for each line given in the chart legend are formed by concatenating the objective function modification abbreviations, such as L2REG (L2 Regularization Term explained in Section 2.1), ENTR-T (Entropy term explained in Section 2.2), SP-IN (A Sparsity-Inducing Term explained in Section 2.3), PEN-LSC (A Penalty For Large Sized Clusters explained in Section 2.4), and SPAT-REG (Spatial Regularization Term explained in Section 2.5) with the abbreviation of the three employed numerical methods for optimization, gradient descent (GD), root mean square propagation (RMSprop), and adaptive moment estimation (Adam).

5. Discussion

The experimental results demonstrate in an obvious manner that modifications to the objective function of the classical Fuzzy C-Means (FCM) algorithm, by applying proper regularization or penalization terms, contribute to the improvement of the quality of the generated clusters. Generally, all five modifications (L2 regularization, entropy term, sparsity-inducing term, penalty for large clusters, and spatial regularization) in all datasets had an overall higher Fuzzy ARI and Fuzzy NMI score compared with the classical FCM, and this is valid for all three optimization techniques (gradient descent, root mean propagation, and adaptive moment estimate). From the summarizing table and the charts can be noted that the spatial regularization (SPAT-REG) and large-cluster penalty (PEN-LSC) modifications typically achieved the highest performance scores, especially when optimized using the Adam method, highlighting their robustness in dealing with noisy or imbalanced data distributions.

The comparison of optimization methods points out that adaptive techniques such as RMSprop and Adam generally perform better compared with the classical gradient descent, offering more stable convergence and less susceptiblity to oscillations for sophisticated objective functions. Especially the adaptive moment estimation (Adam) technique demonstrated very effective results across almost all modifications of the objective function and optimization techniques. Moreover, the numerical methods provide flexibility in designing and optimizing complex objective functions without requiring sophisticated derivatives or the Lagrange multipliers approach. A significant disadvantage of the proposed approaches, compared with the classical methods, is the higher computational complexity, which is considered a natural trade-off for the algorithm simplicity that it offers.

This work also holds possibilities for future directions of research, such as devising hybrid approaches combining multiple regularization techniques to further improve the quality of generated clusters or utilization of adaptive optimization techniques in the hyper-parameter tuning area. Finally, applications of these modified algorithms on new datasets, not well known to the machine learning community, would confirm their validity in real-world problems.

6. Conclusions

This study introduced a comprehensive framework for enhancing the classical Fuzzy C-Means algorithm via several options of modification for the objective function and various numerical optimization strategies applied to them. The incorporation of several modifications to the objective function, such as L2 regularization, entropy term, sparsity-inducing term, penalty for large clusters, and spatial regularization, improved the adaptability to datasets with noise and complex distributions. Each of these modifications was optimized numerically using gradient descent, root mean squared, and adaptive moment estimation, thus avoiding the necessity for analytically derived update rules.

The experimental studies applied on a variety of real-world datasets demonstrated the effectiveness of the proposed modifications. Throughout these experimental studies, an extensive comparison was carried out across various modifications and various numerical optimization techniques. Two significant performance metrics, the fuzzy adjusted rand index and fuzzy normalized information, were applied for the assessment and comparison of the quality of the generated clusters. The findings highlight the practical advantages of combining adjustments on the objective functions with numerical optimization techniques in fuzzy clustering tasks. Among the employed modifications and optimization algorithms, it was noticed that generally, the spatial regularization (SPAT-REG) and large-cluster penalty (PEN-LSC) modifications combined with adaptive moment estimation performed better. Future research may explore hybrid approaches integrating multiple regularization strategies or applying this flexible optimization framework to hyper-parameter tuning problems too.