Anomaly Detection Based on Hybrid Kernelized Fuzzy Density

Abstract

Unsupervised anomaly detection has been extensively studied. However, most existing methods are designed for either numerical or nominal data, which struggle to detect anomalies effectively in real-world mixed-type datasets. Fuzzy information granulation is a key concept in granular computing, which offers a potent framework for managing uncertainty in mixed-type data and provides a viable pathway for unsupervised anomaly detection. Nevertheless, conventional fuzzy information granulation-based detection methods often model only simple, linear fuzzy relations between samples. This limitation prevents them from capturing the complex, nonlinear structures inherent in the data, leading to a degradation in detection performance. To address these shortcomings, we propose a Hybrid Kernelized Fuzzy Density-based anomaly detector (HKFD). HKFD pioneers a hybrid kernelized fuzzy relation by integrating a hybrid distance metric with kernel methods. This new relation allows us to define a hybrid kernelized fuzzy density for each sample within every feature subspace, effectively capturing the local data dispersion. Crucially, we introduce an information-theoretic weighting mechanism. By calculating the fuzzy information entropy of each feature’s distribution, HKFD automatically assigns higher weights to more informative feature subspaces that contribute more to identifying anomalies. The final anomaly factor is then calculated by the weighted fusion of these densities. Comprehensive experiments on 20 datasets demonstrate that HKFD significantly outperforms state-of-the-art methods, achieving superior anomaly detection performance.

1. Introduction

Anomaly detection, also known as outlier detection [1,2], refers to the task of identifying data instances that deviate significantly from the majority of data, often referred to as the norm. These non-conforming instances are defined as anomalies or outliers. This task is of critical importance across a diverse range of high-impact domains, including financial fraud detection, network intrusion security, and medical diagnostics.

Based on the availability of labels, anomaly detection methods can be categorized into supervised, semi-supervised, and unsupervised methods, as illustrated in Figure 1. Supervised methods require a fully labeled dataset containing both normal and anomaly instances, which is often impractical. Anomalies are by nature rare, and their patterns may be unknown in advance. Unsupervised anomaly detection operates under the most challenging and realistic scenario, assuming the dataset is predominantly normal and contains no labels. The core assumption of unsupervised anomaly detection is that anomalies are both few and different from the normal data. Unlike methods requiring prior knowledge, our proposed HKFD is an unsupervised approach that identifies anomalies by exploring intrinsic data structures.

Figure 1. A conceptual taxonomy of anomaly detection approaches categorized by label availability. Our proposed HKFD belongs to the unsupervised category, which directly explores the intrinsic data distribution without requiring labeled samples.

Over the past decades, a multitude of unsupervised anomaly detection methods have been proposed. These methods can be broadly classified into several major categories. Statistical-based methods [3,4] posit a statistical model (e.g., a Gaussian distribution) to represent the normal data. Instances that have a low probability of being generated by this model are declared anomalies. However, these methods rely on strong assumptions about the data distribution, which are often violated in complex and high-dimensional real-world data. Distance-based methods [5,6,7], such as k-Nearest Neighbors (k-NN) [7], are built on the premise that anomalies are located far from their nearest neighbors. Their primary drawback is the curse of dimensionality. In high-dimensional spaces, the contrast in distances between the nearest and farthest neighbors becomes negligible, rendering the distance metric unreliable. Density-based methods [8,9,10], such as Local Outlier Factor (LOF) [8], refine the distance-based concept by comparing the local density of an instance to that of the neighbors. This allows them to identify anomalies in regions of varying density. Nonetheless, their performance is highly sensitive to parameter choices, and their computational complexity, often $O\left(n^2\right)$, makes them unsuitable for large datasets. Clustering-based methods [11,12] operate on the assumption that normal data instances belong to large and dense clusters, while anomalies are either isolated or belong to very small, sparse clusters. The effectiveness of these methods is highly contingent on the success of the underlying clustering algorithm. Consequently, these methods are often optimized for cluster quality rather than for anomaly detection. Deep learning-based methods [13,14] have recently emerged as a powerful paradigm. Methods such as Autoencoders (AE) and Generative Adversarial Networks (GANs) learn a compact, nonlinear representation of the normal data manifold. Instances that cannot be faithfully reconstructed are considered anomalies. While effective, they often require large amounts of clean training data and are prone to overgeneralization.

In parallel to these developments, methods based on Fuzzy Information Granulation (FIG) have emerged as another important category. As a core concept within Granular Computing (GrC), FIG provides a theoretical framework for managing uncertainty. By using fuzzy set theory and the concept of a fuzzy relation, these methods assign degrees of membership rather than binary labels. This makes them inherently suitable for modeling the uncertainty in data and provides a flexible mechanism to uniformly model the similarity between samples in heterogeneous, mixed-type data spaces. Methods based on FIG effectively reflect the anomaly degree of a sample by introducing various uncertainty measures, such as fuzzy information entropy [15] and fuzzy approximation accuracy [1]. However, most existing FIG-based methods only consider simple linear fuzzy relations between samples, failing to effectively model the complex nonlinear relationships within the data. To this end, kernel methods have been introduced into FIG to construct Kernelized FIG (KFIG). In KFIG, the fuzzy relation between samples can be constructed by various kernel methods instead of simple linear distance metrics. The rationale is that traditional linear metrics, such as Euclidean distance, assume simple geometric structures in the input space, which often fail for real-world complex data with nonlinear distributions. By employing kernel functions, samples are implicitly mapped into a high-dimensional feature space. This enhances the model’s ability to capture complex relationships between samples in the data, thereby improving its anomaly detection performance [16,17]. Nevertheless, existing KFIG-based methods still rely on a simple combination of Euclidean distance and a Gaussian kernel, which cannot effectively handle mixed-type data.

Based on the above discussion, in this study, we propose an unsupervised anomaly detection method for mixed-type data, named the Hybrid Kernelized Fuzzy Density-based anomaly detector (HKFD). The framework of HKFD is presented in Figure 2. To simultaneously address the challenges of nonlinearity and data heterogeneity, HKFD first pioneers a hybrid kernelized fuzzy relation. This is achieved by integrating a hybrid distance metric, capable of processing both numerical and categorical attributes, directly into a kernel function. This new relation allows us to define a hybrid kernelized fuzzy density for each sample within every single feature subspace, effectively capturing the local data dispersion in a nonlinear fashion. Furthermore, to distinguish the varying importance of different features, we introduce an information-theoretic weighting mechanism. By calculating the fuzzy information entropy of each feature’s distribution, HKFD automatically assigns higher weights to more informative feature subspaces, which are more critical for identifying anomalies. The final anomaly factor is then calculated as the weighted fusion of these individual densities. The main contributions of this study are summarized as follows:

Figure 2. Framework figure of HKFD.

• We propose HKFD, a novel unsupervised anomaly detection framework designed to effectively handle complex nonlinear relationships in mixed-type datasets.
• We introduce a hybrid kernelized fuzzy relation. This relation integrates a hybrid distance metric with a kernel function, enabling a unified and nonlinear similarity measurement for heterogeneous data.
• We conduct comprehensive experiments on 20 real-world datasets. The results demonstrate that HKFD achieves significant and consistent performance gains over existing state-of-the-art anomaly detection methods.

The remaining sections of this study are organized as follows.

2. Related Works

In this section, we provide a focused review of the literature most relevant to the proposed HKFD, specifically concentrating on anomaly detection methods derived from FIG. The core idea of FIG-based anomaly detection is to quantify an instance’s abnormality by measuring the uncertainty associated with it. A common strategy is to construct fuzzy information granules around data points and then evaluate the properties of these granules. For instance, instances that are isolated or belong to very small, sparse granules are often considered anomalies. Over the years, researchers have proposed various uncertainty measures to quantify this.

For example, Yuan et al. [15] utilized a fuzzy approximate space with a fuzzy similarity relation to construct a fuzzy information entropy-based hybrid attribute anomaly detection method. Yuan et al. [1] introduced fuzzy rough sets to address the anomaly detection problem in mixed-type data. They generalized the anomaly detection model using FRS, constructing a generalized anomaly detection model based on fuzzy rough granules. Yuan et al. [18] first employed a hybrid metric to calculate the fuzzy relation, then built forward and reverse multi-fuzzy granules based on attribute importance and defined an anomaly score based on these ranked sequences. Addressing the limitation that existing techniques are mainly for deterministic data, Liu et al. [19] proposed an anomaly detection method combining fuzzy granules and a Markov random walk. Wang et al. [20] proposed a distance-based fuzzy rough entropy and its correlation measure, designing an anomaly detection method based on this fuzzy rough entropy. Chen et al. [21] proposed a novel information fusion model based on multi-scale fuzzy granules and combined it with fuzzy rough set theory to present an unsupervised anomaly detection algorithm. Subsequently, to enhance model efficiency and the perception of multi-granularity information, many methods combining FIG and granular-ball computing were proposed. For example, Su et al. [22] combined FIG and granular-ball computing, proposing the use of granular-ball fuzzy approximation accuracy to characterize the anomaly degree. Su et al. [23] also proposed detecting anomalies by fusing multi-granular-ball fuzzy information. Furthermore, Su et al. [9] integrated FIG, granular-ball computing, and LOF, proposing the use of local granular-ball density for anomaly detection.

Despite this progress, these methods all rely on simple linear distance metrics to construct the fuzzy relation matrix. This prevents them from effectively modeling the complex nonlinear relationships in the data, leading to a degradation in anomaly detection performance. To this end, Wu et al. [16] combined kernel functions with FIG and applied it to unsupervised anomaly detection. They proposed kernelized fuzzy approximation accuracy to quantify the anomaly degree. Li et al. [17] fused kernel functions, FIG, and granular-ball computing into a unified framework. They proposed multi-granularity kernelized fuzzy approximation accuracy to detect anomalies. However, when constructing the kernelized fuzzy relation, these methods rely solely on simple Euclidean distance, making them ineffective for handling mixed-type data. This specific gap in the literature motivates the development of our proposed HKFD, which is designed to simultaneously model nonlinear relationships and process heterogeneous attributes.

To provide a comprehensive overview of the current research landscape, we systematically summarize the key characteristics of the aforementioned anomaly detection methods in Table 1. The comparison is conducted across several dimensions, including their core mechanisms, supported data types, advantages, disadvantages, and commonly used benchmark datasets.

Table 1. Comparison of existing anomaly detection methods.

Methods	Main Ideas	Data Types	Advantages	Disadvantages	Benchmarks
FIEOD [15]	Fuzzy information entropy	Mixed-type	Handle uncertainty; handle mixed-type data	High complexity; linear fuzzy relation	UCI machine learning repository
FRGOD [1]	Fuzzy approximation accuracy	Mixed-type	Handle uncertainty; handle mixed-type data	High complexity; linear fuzzy relation	UCI machine learning repository
MFGAD [18]	Multi-fuzzy granules	Mixed-type	Handle uncertainty; handle mixed-type data	High complexity; linear fuzzy relation	UCI machine learning repository
FGAS [19]	Markov random walk	Mixed-type	Handle uncertainty; handle mixed-type data	High complexity; linear fuzzy relation	UCI machine learning repository
FREAD [20]	Fuzzy rough entropy	Mixed-type	Handle uncertainty; handle mixed-type data	High complexity; linear fuzzy relation	UCI machine learning repository
MFIOD [21]	Multi-scale fuzzy granules	Mixed-type	Handle uncertainty; handle mixed-type data	High complexity; linear fuzzy relation	UCI machine learning repository
GBFRD [22]	Granular-ball fuzzy approximation accuracy	Numerical	Handle uncertainty; multi-granularity representation; high efficiency	Struggle with mixed-type data; linear fuzzy relation	UCI machine learning repository
MGBOD [23]	Multi-granular-ball fuzzy information	Numerical	Handle uncertainty; multi-granularity representation; high efficiency	Struggle with mixed-type data; linear fuzzy relation	UCI machine learning repository
GBDO [9]	Local granular-ball density	Numerical	Handle uncertainty; multi-granularity representation; high efficiency	Struggle with mixed-type data; linear fuzzy relation	UCI machine learning repository
KFRAD [16]	Kernelized fuzzy approximation accuracy	Numerical	Handle uncertainty; nonlinear fuzzy relation	High complexity; struggle with mixed-type data	UCI machine learning repository
KFGOD [17]	Multi-granularity kernelized fuzzy approximation accuracy	Numerical	Handle uncertainty; multi-granularity representation; high efficiency; nonlinear fuzzy relation	Struggle with mixed-type data	UCI machine learning repository

Based on the comparison in Table 1, the feature handling strategies of these FIG-based methods can be further analyzed through three structured dimensions. First, regarding nonlinearity, while earlier methods like FIEOD and FRGOD rely on linear fuzzy relations that struggle with complex manifolds, recent kernelized approaches (e.g., KFRAD and KFGOD) successfully capture high-order dependencies by mapping data into high-dimensional spaces. Second, in terms of robustness to noise, the fuzzy granulation mechanism inherent in all these methods provides a “soft” boundary, which is naturally more tolerant to data perturbations and outliers than crisp distance-based models. Finally, scalability remains a primary differentiator; granular-ball-based methods (e.g., GBFRD and GBDO) achieve high efficiency by reducing data volume into coarse-grained representations, whereas traditional FIG and kernelized FIG models often entail substantial computational overhead due to the construction of full fuzzy relation matrices. By synthesizing these strategies, our proposed HKFD aims to bridge the gap between nonlinear mixed-type data handling and effective density estimation within the FIG framework.

3. Preliminaries

  We use a dataset $\mathit{S}\in {\mathbb{R}}^{n\times m}$ that contains n samples and m attributes. The i-th sample in $\mathit{S}$ can be denoted as ${\mathit{s}}_i=(s_i^1,s_i^2,\cdots ,s_i^m)\in {\mathbb{R}}^m$, where $s_i^j$ represents the j-th attribute value of sample ${\mathit{s}}_i$.

In contrast to classical sets, which are defined by crisp, binary membership (i.e., an element is either in the set or not), a fuzzy set is characterized by a membership function.

This function maps each element in the universe of discourse to a real value in the interval $[0,1]$, representing its degree of belonging to the set. If f is a mapping from $\mathit{S}$ to $[0,1]$, i.e., $f:\mathit{S}\to [0,1]$, then $f=\{f\left({\mathit{s}}_1\right),f\left({\mathit{s}}_2\right),\cdots ,f\left({\mathit{s}}_n\right)\}$ is a fuzzy set on $\mathit{S}$. The fuzzy set f can also be represented as $f={\displaystyle \frac{f\left({\mathit{s}}_1\right)}{{\mathit{s}}_1}}+{\displaystyle \frac{f\left({\mathit{s}}_2\right)}{{\mathit{s}}_2}}+\cdots +{\displaystyle \frac{f\left({\mathit{s}}_n\right)}{{\mathit{s}}_n}}={\sum }_{i=1}^n{\displaystyle \frac{f\left({\mathit{s}}_i\right)}{{\mathit{s}}_i}}$. It is important to note that this is a standard symbolic notation in fuzzy set theory, where the horizontal bar “/” serves as a delimiter and the plus signs represent a collection, rather than arithmetic division or addition. For any $\mathit{s}\in \mathit{S}$, $f\left(\mathit{s}\right)$ is called the membership function of the fuzzy set f, or the membership degree of the sample $\mathit{s}$ to the fuzzy set f. All fuzzy sets on $\mathit{S}$ can be denoted as $F\left(\mathit{S}\right)$.

The algebra of fuzzy sets is defined by several fundamental operations, analogous to classical set theory. We formally define inclusion, equality, union, intersection, and complement below. Given any two fuzzy sets $f_1, f_2\in F\left(\mathit{S}\right)$, the following operations and relations are defined for all $\mathit{s}\in \mathit{S}$:

• Inclusion relation: $f_1\subseteq f_2$, if $f_1\left(\mathit{s}\right)\le f_2\left(\mathit{s}\right)$;
• Equality relation: $f_1=f_2$, if $f_1\left(\mathit{s}\right)=f_2\left(\mathit{s}\right)$;
• Union operation: $(f_1\cup f_2)\left(\mathit{s}\right)=max\{f_1\left(\mathit{s}\right),f_2\left(\mathit{s}\right)\}=f_1\left(\mathit{s}\right)\vee f_2\left(\mathit{s}\right)$;
• Intersection operation: $(f_1\cap f_2)\left(\mathit{s}\right)=min\{f_1\left(\mathit{s}\right),f_2\left(\mathit{s}\right)\}=f_1\left(\mathit{s}\right)\wedge f_2\left(\mathit{s}\right)$;
• Complement operation: $f_1^c\left(\mathit{s}\right)=1-f_1\left(\mathit{s}\right)$.

The concept of a fuzzy set can be extended to define relationships between pairs of elements. A fuzzy relation R from $\mathit{S}$ to $\mathit{S}$ is a fuzzy set R on $\mathit{S}\times \mathit{S}$, characterized by a mapping, i.e., $R:\mathit{S}\times \mathit{S}\to [0,1]$. For any pair $(\mathit{s},\mathit{t})\in \mathit{S}\times \mathit{S}$, the membership degree $R(\mathit{s},\mathit{t})$ quantifies the degree of relationship between the sample $\mathit{s}$ and the sample $\mathit{t}$. When R is a relation from $\mathit{S}$ to $\mathit{S}$, it is simply called a fuzzy relation on $\mathit{S}$. For a finite dataset, a fuzzy relation R on $\mathit{S}$ is conveniently represented by an $n\times n$ fuzzy matrix, i.e., $\mathit{S}\left(R\right)={\left(r_{ij}\right)}_{n\times n}$. Within this matrix, each row vector $(r_{i1},r_{i2},\cdots ,r_{in})$ can be interpreted as a fuzzy set, describing the relationship of the i-th sample to all other samples. We use $F(\mathit{S}\times \mathit{S})$ to denote the set of all fuzzy relations on $\mathit{S}$. As a fuzzy relation is a specific type of fuzzy set, the standard operations of union, intersection, and complement also apply.

Let C be the set of conditional attributes in $\mathit{S}$. For any attribute subset $B\subseteq C$, we can define a specific fuzzy relation $R_B$ on $\mathit{S}$. This mapping $R_B:\mathit{S}\times \mathit{S}\to [0,1]$ generates a fuzzy relation matrix $\mathit{S}\left(R_B\right)={\left(r_{ij}^B\right)}_{n\times n}$. Each element $r_{ij}^B=R_B({\mathit{s}}_i,{\mathit{s}}_j)$ quantifies a fuzzy relationship between the sample pair $({\mathit{s}}_i,{\mathit{s}}_j)$ with regard to the attributes in B. Of particular importance is the fuzzy similarity relation, which is a fuzzy relation $R_B$ that satisfies two specific properties for any ${\mathit{s}}_i,{\mathit{s}}_j\in \mathit{S}$, i.e., (1) Reflexivity: $R_B({\mathit{s}}_i,{\mathit{s}}_i)=1$; (2) Symmetry: $R_B({\mathit{s}}_i,{\mathit{s}}_j)=R_B({\mathit{s}}_j,{\mathit{s}}_i)$. This fuzzy similarity relation $R_B$ provides a mechanism to granulate the dataset $\mathit{S}$. Each sample induces its own fuzzy information granule based on its relationship to all other samples. This collection of all such granules is defined as the family of fuzzy information granules. For any $B\subseteq C$ and any ${\mathit{s}}_i\in \mathit{S}$. $\mathit{S}$ can be granulated by $R_B$ into a set of fuzzy information granules, thereby forming the family of fuzzy information granules $𝒪\left(R_B\right)$, which is defined as

$$𝒪\left(R_B\right)=\{{\left[{\mathit{s}}_1\right]}_{R_B},{\left[{\mathit{s}}_2\right]}_{R_B},\cdots ,{\left[{\mathit{s}}_n\right]}_{R_B}\},$$

(1)

where ${\left[{\mathit{s}}_i\right]}_{R_B}=r_{i1}^B/{\mathit{s}}_1+r_{i2}^B/{\mathit{s}}_2+\cdots +r_{in}^B/{\mathit{s}}_n=(r_{i1}^B,r_{i2}^B,\cdots ,r_{in}^B)$ is the fuzzy information granule induced by $R_B$ and containing the sample ${\mathit{s}}_i$. Similar to the representation of fuzzy sets, this notation is symbolic in nature, where the bar “/” facilitates the mapping between the relation degree and the pair of samples, rather than denoting arithmetic division.

Within the framework of FIG, fuzzy information entropy serves as a crucial metric for quantifying uncertainty. It is formally defined as follows:

Definition 1.

For any $B\subseteq C$ and any ${\mathit{s}}_i\in \mathit{S}$. The fuzzy information entropy with regard to B is defined as the following:

$$FE\left(B\right)=-{\displaystyle \frac{1}{n}}\sum _{i=1}^n{log}_2{\displaystyle \frac{|{\left[{\mathit{s}}_i\right]}_B|}{n}},$$

(2)

where $|{\left[{\mathit{s}}_i\right]}_B|$ denotes the cardinality of ${\left[{\mathit{s}}_i\right]}_B$, which is calculated by $|{\left[{\mathit{s}}_i\right]}_B|={\sum }_{{\mathit{s}}_j\in \mathit{S}}{R}_B({\mathit{s}}_i,{\mathit{s}}_j)$.

To effectively capture the complex nonlinear relationships between samples, researchers have integrated kernel functions into fuzzy information granulation (FIG) and developed the KFIG framework. A function $K:\mathit{S}\times \mathit{S}\to \mathbb{R}$ is defined as a kernel if it satisfies semi-positivity and symmetry for any $\mathit{s}\in \mathit{S}$ [24]. Furthermore, any kernel K that fulfills the reflexivity condition $K(\mathit{s},\mathit{s})=1$ for all $\mathit{s}\in \mathit{S}$ is also guaranteed to satisfy $T_{\cos }$-transitivity [24]. Several common kernel functions, such as the Rational Quadratic, Spherical, and Gaussian kernels, inherently satisfy all three properties, i.e., reflexivity, symmetry, and $T_{\cos }$-transitivity. As a result, the fuzzy relations derived from these specific kernels are classified as fuzzy T-equivalence relations. The KFIG model is therefore constructed by substituting the standard fuzzy relations of FIG with these kernel functions.

Formally, for any $B\subseteq C$, the kernelized fuzzy relation $K_B$ with regard to B on $\mathit{S}$ is defined as a fuzzy set on $\mathit{S}\times \mathit{S}$. This relation $K_B$ is represented in matrix form as $\mathit{M}\left(K_B\right)={\left(k_{ij}^B\right)}_{n\times n}$, where each element $k_{ij}^B=K_B({\mathit{s}}_i,{\mathit{s}}_j)$ quantifies the kernelized fuzzy relation between samples ${\mathit{s}}_i$ and ${\mathit{s}}_j$ with regard to B. As a special fuzzy relation, $K_B$ can also granulate $\mathit{S}$ to produce multiple kernelized fuzzy information granules, thereby forming the family of kernelized fuzzy information granules $𝒪\left(K_B\right)$, which is defined as the following:

$$𝒪\left(K_B\right)=\{{\left[{\mathit{s}}_1\right]}_{K_B},{\left[{\mathit{s}}_2\right]}_{K_B},\cdots ,{\left[{\mathit{s}}_n\right]}_{K_B}\},$$

(3)

where ${\left[{\mathit{s}}_i\right]}_{K_B}=k_{i1}^B/{\mathit{s}}_1+k_{i2}^B/{\mathit{s}}_2+\cdots +k_{in}^B/{\mathit{s}}_n=(k_{i1}^B,k_{i2}^B,\cdots ,k_{in}^B)$ is the kernelized fuzzy information granule induced by $K_B$ and containing the sample ${\mathit{s}}_i$.

4. Hybrid Kernelized Fuzzy Density-Based Anomaly Detector

In this section, we introduce our novel unsupervised anomaly detection method, HKFD. We first introduce a hybrid kernelized fuzzy relation to effectively handle mixed-type data and model complex nonlinear relationships. Based on this, we define a hybrid kernelized fuzzy density to characterize the local dispersion degree of each sample within each feature subspace. Finally, we develop an information-theoretic weighting scheme and calculate the final anomaly factor by a weighted fusion of these densities.

4.1. Hybrid Kernelized Fuzzy Information Granulation

Considering that existing KFIG-based anomaly detection methods cannot effectively model the fuzzy relations between samples in mixed-type data, we propose a hybrid kernelized fuzzy relation. To effectively handle the nominal attributes in the data, we first introduce the overlap metric [25] to characterize the dissimilarity between samples in the nominal data.

Definition 2.

For any ${\mathit{s}}_i,{\mathit{s}}_j\in \mathit{S}$, the overlap metric $I_B({\mathit{s}}_i,{\mathit{s}}_j)$ between ${\mathit{s}}_i$ and ${\mathit{s}}_j$ on $B\subseteq C$ is calculated by

$$I_B({\mathit{s}}_i,{\mathit{s}}_j)=\left|\{b\in B\mid s_i^b\ne s_j^b\}\right|.$$

(4)

The overlap metric effectively captures the dissimilarity between samples in nominal data through direct attribute comparison. Next, we combine the overlap metric and the classic Euclidean distance to introduce the hybrid distance metric between samples.

Definition 3.

For any ${\mathit{s}}_i,{\mathit{s}}_j\in \mathit{S}$, the hybrid distance metric $H_B({\mathit{s}}_i,{\mathit{s}}_j)$ between ${\mathit{s}}_i$ and ${\mathit{s}}_j$ on $B\subseteq C$ is calculated by

$$H_B({\mathit{s}}_i,{\mathit{s}}_j)=I_{B^{\mathrm{nom}}}({\mathit{s}}_i,{\mathit{s}}_j)+E_{B^{\mathrm{num}}}({\mathit{s}}_i,{\mathit{s}}_j),$$

(5)

where $B^{\mathrm{nom}}$ denotes the nominal attributes in B and $B^{\mathrm{num}}$ denotes the numerical attributes in B; $E_{B^{\mathrm{num}}}({\mathit{s}}_i,{\mathit{s}}_j)$ denotes the Euclidean distance between samples regarding the numerical attributes $B^{\mathrm{num}}$.

Next, we combine the previously proposed hybrid distance metric with the Gaussian kernel function to introduce the hybrid kernelized fuzzy relation, which is capable of effectively handling mixed-type data.

Definition 4.

For any ${\mathit{s}}_i,{\mathit{s}}_j\in \mathit{S}$, the hybrid kernelized fuzzy relation $K_B^{*}({\mathit{s}}_i,{\mathit{s}}_j)$ between ${\mathit{s}}_i$ and ${\mathit{s}}_j$ on $B\subseteq C$ is defined as follows:

$$K_B^{*}({\mathit{s}}_i,{\mathit{s}}_j)=exp\left({\displaystyle \frac{-H_B{({\mathit{s}}_i,{\mathit{s}}_j)}^2}{\delta }}\right),$$

(6)

where δ is the kernel bandwidth.

Based on the proposed hybrid kernelized fuzzy relation, we can achieve an effective granulation of the mixed-type data. Formally, for any $B\subseteq C$, the hybrid kernelized fuzzy relation $K_B^{*}$ with regard to B on $\mathit{S}$ can be represented in matrix form as $\mathit{M}\left(K_B^{*}\right)={\left({\widehat{k}}_{ij}^B\right)}_{n\times n}$, where each element ${\widehat{k}}_{ij}^B=K_B^{*}({\mathit{s}}_i,{\mathit{s}}_j)$ quantifies the hybrid kernelized fuzzy relation between samples ${\mathit{s}}_i$ and ${\mathit{s}}_j$ with regard to B. As a special fuzzy relation, $K_B^{*}$ can also granulate $\mathit{S}$ to produce multiple hybrid kernelized fuzzy information granules, thereby forming the family of hybrid kernelized fuzzy information granules $𝒪\left(K_B^{*}\right)$, which is defined as follows:

$$𝒪\left(K_B^{*}\right)=\{{\left[{\mathit{s}}_1\right]}_{K_B^{*}},{\left[{\mathit{s}}_2\right]}_{K_B^{*}},\cdots ,{\left[{\mathit{s}}_n\right]}_{K_B^{*}}\},$$

(7)

where ${\left[{\mathit{s}}_i\right]}_{K_B^{*}}={\widehat{k}}_{i1}^B/{\mathit{s}}_1+{\widehat{k}}_{i2}^B/{\mathit{s}}_2+\cdots +{\widehat{k}}_{in}^B/{\mathit{s}}_n=({\widehat{k}}_{i1}^B,{\widehat{k}}_{i2}^B,\cdots ,{\widehat{k}}_{in}^B)$ is the hybrid kernelized fuzzy information granule induced by $K_B^{*}$ and containing the sample ${\mathit{s}}_i$. Assuming no ambiguity arises, we simplify ${\left[{\mathit{s}}_i\right]}_{K_B^{*}}$ to ${\left[{\mathit{s}}_i\right]}_B^{*}$ for notational convenience.

Leveraging the hybrid kernelized fuzzy information granules defined above, we extend the classical fuzzy information entropy to propose the hybrid kernelized fuzzy information entropy.

Definition 5.

For any $B\subseteq C$ and any ${\mathit{s}}_i\in \mathit{S}$. The hybrid kernelized fuzzy information entropy with regard to B is defined as follows:

$$HKFE\left(B\right)=-{\displaystyle \frac{1}{n}}\sum _{i=1}^n{log}_2{\displaystyle \frac{|{\left[{\mathit{s}}_i\right]}_B^{*}|}{n}},$$

(8)

where $|{\left[{\mathit{s}}_i\right]}_B^{*}|$ denotes the cardinality of ${\left[{\mathit{s}}_i\right]}_B^{*}$.

Formally, the proposed $HKFE\left(B\right)$ appears identical in form to the classical $FE\left(B\right)$. Their fundamental difference, however, lies in the fuzzy information granules used for their calculation, which are generated by different fuzzy relations. Our $HKFE\left(B\right)$ utilizes the novel hybrid kernelized fuzzy relation to effectively capture the complex nonlinear relationships between samples in mixed-type data. This allows $HKFE\left(B\right)$ to provide a more effective characterization and quantification of the data’s inherent uncertainty, overcoming the limitations of traditional fuzzy entropy measures that rely on linear or single-type distance metrics.

4.2. Detection Method

To effectively characterize the anomaly degree of each sample, we propose the following hybrid kernelized fuzzy density.

Definition 6.

For any $B\subseteq C$ and any ${\mathit{s}}_i\in \mathit{S}$. The hybrid kernelized fuzzy density of ${\mathit{s}}_i$ with regard to B is defined as follows:

$$HKF{D}_B\left({\mathit{s}}_i\right)={\displaystyle \frac{{\sum }_{b\in B}De{n}_b\left({\mathit{s}}_i\right)}{\left|B\right|}},$$

(9)

where $De{n}_b\left({\mathit{s}}_i\right)={\sum }_{{\mathit{s}}_j\in \mathit{S}}{K}_b^{*}({\mathit{s}}_i,{\mathit{s}}_j)/n$.

In the above definition, we define the hybrid kernelized fuzzy density of a sample ${\mathit{s}}_i$ by calculating the average cardinality of its kernelized fuzzy information granules across the different attributes in B. For a sample ${\mathit{s}}_i$, if its $HKF{D}_B\left({\mathit{s}}_i\right)$ is very small, it indicates that its similarity to the majority of other samples is low. Therefore, it has a higher probability of being an anomaly.

In the definition above, all attributes are assigned equal importance. This setting is clearly suboptimal, as in some real-world scenarios, only a small subset of attributes may be relevant for determining whether a sample is an anomaly. Therefore, to effectively measure the importance of different attributes and adaptively assign them appropriate weights, we leverage the previously proposed hybrid kernelized fuzzy information entropy. Finally, we calculate the anomaly factor for each sample as follows:

Definition 7.

For any ${\mathit{s}}_i\in \mathit{S}$. The anomaly factor of ${\mathit{s}}_i$ is defined as follows:

$$AF\left({\mathit{s}}_i\right)=\sum _{c\in C}W\left(c\right)(1-HKF{D}_c\left({\mathit{s}}_i\right)),$$

(10)

where $W\left(c\right)=HKFE\left(c\right)/{\sum }_{c^{\prime }\in C}HKFE\left(c^{\prime }\right)$ denotes the weighting coefficient.

In the anomaly factor defined above, we adaptively assign reasonable weighting coefficients to different attributes based on the proposed hybrid kernelized fuzzy information entropy. The rationale for this weighting scheme is as follows: If the entropy of an attribute is larger, it implies that the distribution of its value domain is more disordered and uncertain. This high degree of uncertainty is often a key manifestation of anomalies, as anomalies are less likely to fit into a clear, low-entropy pattern. Therefore, the hybrid kernelized fuzzy density calculated from such an attribute provides more critical information for separating anomalies from normal samples and should be assigned a greater weight during the fusion process. It is evident that $AF\left({\mathit{s}}_i\right)\in [0,1]$, as it is derived from the normalized kernelized fuzzy memberships. A higher $AF$ value indicates that the sample ${\mathit{s}}_i$ possesses a lower fuzzy density relative to its neighbors, signifying a higher likelihood of being an anomaly, while a lower $AF$ value suggests a normal data point.

4.3. Detection Algorithm

Based on the anomaly detection method proposed above, we formalize it as the HKFD algorithm. As outlined in Algorithm 1, the HKFD algorithm proceeds in several stages. First, we construct the hybrid kernelized fuzzy relation matrix for each attribute $c\in C$, respectively. Next, we calculate the hybrid kernelized fuzzy information entropy $HKFE\left(c\right)$ for each attribute, which is a prerequisite for the subsequent calculation of the attribute weighting coefficients. Then, we calculate the hybrid kernelized fuzzy density $HKF{D}_c\left({\mathit{s}}_i\right)$ for each sample ${\mathit{s}}_i$ under each attribute, as well as the corresponding attribute weighting coefficient $W\left(c\right)$. Finally, we calculate the anomaly factor $AF\left({\mathit{s}}_i\right)$ for each sample by performing a weighted fusion operation of its hybrid kernelized fuzzy densities over all attributes. It should be noted that from the detection task’s point of view, the returned $AF$ values represent the final anomaly scores for all samples. In this unsupervised framework, the ’detection’ is realized through these scores, which characterize the anomaly degree of each sample. These scores enable the ranking of instances and serve as the direct input for evaluating the detection performance via threshold-independent metrics.

The primary computational bottleneck of this algorithm is the construction of the hybrid kernelized fuzzy relation matrix for each attribute. Calculating the $n\times n$ matrix $\mathit{M}\left(K_c^{*}\right)$ for a single attribute c requires comparing all pairs of samples, resulting in a time complexity of $O\left(n^2\right)$. Since this process must be repeated for all m attributes, the total time complexity of the HKFD algorithm is $O\left(m{n}^2\right)$, where m and n represent the total number of attributes and samples in the dataset $\mathit{S}$, respectively.

Algorithm 1: HKFD algorithm

5. Experiments

In this section, we conduct comprehensive experiments to evaluate the performance of our proposed HKFD algorithm. We first introduce the datasets and the baseline methods used in our comparison. Next, we verify the effectiveness of HKFD through multiple experimental metrics. Then, we also conduct statistical analysis to further compare the significance of the results. Finally, we explore the sensitivity of HKFD to its hyperparameter $\delta$.

5.1. Experimental Datasets

We select 20 real-world datasets commonly used for the task of unsupervised anomaly detection. The basic information of these datasets is summarized in Table 2. As shown in the table, the sample size of these datasets ranges from 111 to 4429, the attribute dimensionality ranges from 4 to 279, the number of anomalies ranges from 4 to 221, and the anomaly contamination rate varies from 1.7% to 23.5%.

Table 2. Summary of the benchmark datasets.

No.	Datasets	Abbr.	Samples	Attributes	Outliers	Outlier Ratios
1	Audiology_variant1	Audio	226	69	53	23.5%
2	Lymphography	Lymp	148	8	6	4.1%
3	Monks_0_12_variant1	Monks1	240	6	12	5.0%
4	Monks_0_25_variant1	Monks2	253	6	25	9.9%
5	Monks_0_4_variant1	Monks3	232	6	4	1.7%
6	Mushroom_p_221_variant1	Mush	4429	22	221	5.0%
7	Vote_republican_29_variant1	Vote	296	16	29	9.8%
8	Cardiotocography_2and3_33_variant1	Card	1688	21	33	2.0%
9	Diabetes_tested_positive_26_variant1	Diab	526	8	26	4.9%
10	Glass	Glass	214	9	9	4.2%
11	Iris_Irisvirginica_11_variant1	Iris	111	4	11	9.9%
12	Musk	Musk	3062	166	97	3.2%
13	Pima_TRUE_55_variant1	Pima	555	9	55	9.9%
14	Sonar_M_10_variant1	Sonar	107	60	10	9.3%
15	Wbc_malignant_39_variant1	Wbc	483	9	39	8.1%
16	Wdbc_M_39_variant1	Wdbc	396	31	39	9.8%
17	Arrhythmia_variant1	Arr	452	279	66	14.6%
18	CreditA_plus_42_variant1	Credit	425	15	42	9.9%
19	German_1_14_variant1	German	714	20	14	2.0%
20	Heart270_2_16_variant1	Heart	166	13	16	9.6%

5.2. Experimental Settings

To comprehensively evaluate the effectiveness of HKFD, we select 14 baseline methods covering a wide spectrum of anomaly detection paradigms, including statistical, proximity-based, granular-ball-based, and deep learning-based methods. Specifically, these methods include Rotation-based Outlier Detection (ROD) [26] and Empirical Cumulative distribution functions-based Outlier Detection (ECOD) [27] from the statistical perspective; Local Projection-based Outlier Detection (LPOD) [28], Virtual Outlier Score (VOS) [29], Mean-shift Outlier Detector (MOD+) [30], Directed density ratio Changing Rate-based Outlier Detection (DCROD) [31], and Anomaly Detection based on improved k-nearest Neighbor Rough Sets (ADkNRS) [32] representing the proximity and local density-based paradigm; local Granular-Ball Density-based Outlier detection (GBDO) [9], Granular-Ball Mean-shift Outlier Detector (GBMOD) [33], and Multi-Granular-Ball fuzzy information granules-based Outlier Detection (MGBOD) [23] as state-of-the-art granular-ball-based methods; and Deep Isolation Forest (DIF) [14], Deep Support Vector Data Description (DeepSVDD) [13], outlier detection using structural scores in a high-dimensional space (VarE) [34], and a novel joint learning framework for outlier detection in MIXed-type data with a MIXture of clustered and scattered outliers (MIX) [35] to assess performance against advanced neural networks and specialized mixed-type models. These methods are chosen to ensure a comprehensive and unbiased benchmark across diverse technical frameworks.

To ensure a fair comparison, several baseline methods require hyperparameter tuning to achieve their optimal performance. We establish appropriate search ranges for these methods. For GBDO and GBMOD, their number of neighbors is searched in the range $[2,min\{60,\left|GB\right|\left\}\right]$ with a step size of one, where $\left|GB\right|$ denotes the number of generated granular-balls. For MGBOD, the fuzzy radius is searched from 0 to 1 at intervals of 0.05. For LPOD, VOS, MOD+, DCROD, and ADkNRS, we tune the number of neighbors k within the range $[2,60]$ with a step size of one. ROD and ECOD are parameter-free methods. For VarE, the hyperparameter search space is set to the discrete set $\{{10}^{-3},{10}^{-2},{10}^{-1},1,10,{10}^2,{10}^3\}$. MIX is a stochastic method. We report the best result from ten independent runs as the final experimental outcome. DeepSVDD and DIF are deep learning-based methods, for which we adopt the default network architectures and settings recommended in their original publications. Finally, our proposed method, HKFD, involves a single hyperparameter $\delta$ to adjust the kernel bandwidth. We set the search range from 0.02 to 0.8 with a step of 0.02. We will explore the sensitivity of HKFD to $\delta$ in a subsequent experiment.

We select the Receiver Operating Characteristic (ROC) curve, Area Under the ROC Curve (AUC), and Average Precision (AP) as the evaluation metrics for our experiments. The ROC curve visually represents the trade-off between the true positive rate (recall) and the false positive rate as the detection threshold is varied. AUC summarizes this curve into a single scalar value, representing the probability that the model will rank a randomly chosen positive (anomaly) instance higher than a randomly chosen negative (normal) instance. AP summarizes the precision-recall curve and is particularly informative in highly imbalanced datasets, as it focuses on the performance ranking of the positive class (anomalies). For both AUC and AP, a higher value indicates superior detection performance.

To ensure a fair and rigorous comparison, we adopt an optimization strategy where hyperparameters for all methods are tuned independently for each dataset. Specifically, we conduct a grid search within the predefined ranges mentioned above. Since unsupervised anomaly detection lacks label information for traditional cross-validation, we report the best performance achieved on each dataset to represent the optimal capability of each algorithm.

5.3. Experimental Results

We first present the ROC curves for all methods on the 20 datasets, as shown in Figure 3. From the figure, the ROC curve of our method, HKFD, is closer to the top-left corner of the coordinate axis on datasets such as Monks1, Monks2, Monks3, Mush, Diab, Pima, and German. This indicates that HKFD outperforms other methods on these datasets, demonstrating superior anomaly detection performance. Due to the overlap of numerous ROC curves, the differences between methods can be difficult to further distinguish. Therefore, we further present the experimental results for AUC and AP in tabular form.

Figure 3. Comparison of ROC curves for HKFD and baseline methods on all 20 datasets.

As shown in Table 3, we present the comparison of AUC scores for all 15 methods on the 20 datasets. In this table, HKFD achieves the first-place AUC score on 10 out of the 20 datasets and obtains the highest average AUC score. Our method’s average AUC score is 0.956, which is 2.0% higher than the second-best method. We further display the AP score comparison for the 15 methods on the 20 datasets. As shown in Table 4, HKFD achieves the first-place AP score on 7 out of the 20 datasets and the highest average AP score. Our method’s average AP score is 0.787, which is 10.1% higher than the second-best method. These experimental results consistently demonstrate that HKFD has better anomaly detection performance compared to other methods, thus verifying its effectiveness.

Table 3. Comparison of AUC scores for HKFD against 14 methods across the 20 datasets. The best result on each dataset is highlighted in bold.

Datasets	DeepSVDD	LPOD	MIX	VOS	VarE	MOD+	DCROD	ECOD	ROD	DIF	GBMOD	ADkNRS	MGBOD	GBDO	HKFD (Ours)
Audio	0.694	0.799	0.863	0.651	0.792	0.470	0.645	0.833	0.615	0.549	0.681	0.819	0.651	0.792	0.834
Lymp	0.969	0.673	0.998	0.974	0.507	1.000	0.954	0.996	0.762	0.982	0.998	0.979	0.765	0.989	0.993
Monks1	0.781	0.566	0.985	0.969	0.680	0.732	0.986	0.957	0.762	0.837	0.914	0.967	0.918	0.696	0.996
Monks2	0.752	0.734	0.978	0.916	0.763	0.811	0.992	0.969	0.722	0.724	0.890	0.969	0.896	0.806	1.000
Monks3	0.703	0.657	0.940	0.986	0.727	0.368	0.989	0.987	0.968	0.912	0.917	0.967	0.998	0.816	1.000
Mush	0.917	0.928	0.951	0.533	0.880	0.799	0.596	0.949	0.847	0.847	0.916	0.648	0.852	0.885	0.970
Vote	0.857	0.166	0.995	0.989	0.981	0.273	0.537	0.995	0.995	0.801	0.997	0.500	0.990	0.991	0.995
Card	0.549	0.791	0.616	0.882	0.780	0.788	0.834	0.871	0.893	0.796	0.864	0.863	0.846	0.822	0.879
Diab	0.929	0.966	0.969	0.942	0.965	0.960	0.935	0.979	0.954	0.861	0.945	0.975	0.960	0.926	0.982
Glass	0.541	0.732	0.718	0.708	0.538	0.845	0.863	0.621	0.738	0.775	0.859	0.771	0.712	0.873	0.727
Iris	0.999	1.000	0.986	1.000	1.000	1.000	0.983	0.977	0.866	0.779	1.000	1.000	1.000	1.000	1.000
Musk	0.929	0.075	0.999	0.717	0.346	0.580	0.863	0.956	0.973	0.994	1.000	1.000	1.000	1.000	1.000
Pima	0.849	0.920	0.945	0.935	0.904	0.938	0.928	0.947	0.947	0.839	0.935	0.956	0.947	0.934	0.972
Sonar	0.901	0.997	0.989	0.998	0.955	0.997	0.989	0.966	0.990	0.882	1.000	0.992	0.993	0.998	0.990
Wbc	0.993	0.997	0.997	0.995	0.997	0.987	0.976	0.995	0.985	0.939	0.997	0.996	0.997	0.997	0.997
Wdbc	0.981	1.000	0.996	1.000	0.997	0.942	0.968	0.959	0.995	0.871	0.996	0.997	0.994	0.997	0.999
Arr	0.750	0.799	0.826	0.813	0.739	0.779	0.796	0.807	0.801	0.809	0.810	0.831	0.789	0.776	0.827
Credit	0.959	0.873	0.937	0.934	0.625	0.856	0.841	0.990	0.974	0.910	0.955	0.979	0.980	0.948	0.983
German	0.882	0.949	0.891	0.968	0.919	0.878	0.955	0.966	0.927	0.686	0.937	0.956	0.874	0.889	0.983
Heart	0.930	0.949	0.974	0.964	0.962	0.933	0.970	0.996	0.975	0.817	0.973	0.976	0.986	0.985	0.995
Average	0.843	0.779	0.928	0.894	0.803	0.797	0.880	0.936	0.884	0.831	0.929	0.907	0.907	0.906	0.956

Table 4. Comparisonof AP scores for HKFD against 14 methods across the 20 datasets. The best result on each dataset is highlighted in bold.

Datasets	DeepSVDD	LPOD	MIX	VOS	VarE	MOD+	DCROD	ECOD	ROD	DIF	GBMOD	ADkNRS	MGBOD	GBDO	HKFD (Ours)
Audio	0.476	0.523	0.816	0.401	0.516	0.272	0.385	0.654	0.371	0.311	0.459	0.672	0.493	0.603	0.715
Lymp	0.781	0.520	0.948	0.726	0.095	1.000	0.729	0.915	0.277	0.734	0.958	0.726	0.171	0.811	0.896
Monks1	0.134	0.063	0.730	0.624	0.095	0.158	0.822	0.715	0.129	0.177	0.285	0.483	0.673	0.132	0.946
Monks2	0.326	0.419	0.823	0.545	0.417	0.338	0.928	0.783	0.244	0.205	0.564	0.681	0.748	0.388	1.000
Monks3	0.274	0.054	0.767	0.440	0.153	0.020	0.821	0.813	0.239	0.122	0.514	0.213	0.893	0.075	1.000
Mush	0.454	0.414	0.485	0.193	0.401	0.327	0.139	0.486	0.382	0.366	0.401	0.299	0.388	0.286	0.898
Vote	0.434	0.066	0.948	0.915	0.754	0.073	0.144	0.923	0.953	0.321	0.965	0.098	0.905	0.918	0.919
Card	0.038	0.066	0.050	0.192	0.057	0.071	0.108	0.153	0.122	0.081	0.154	0.144	0.129	0.132	0.135
Diab	0.321	0.661	0.578	0.381	0.699	0.501	0.405	0.638	0.481	0.215	0.437	0.575	0.447	0.378	0.743
Glass	0.071	0.098	0.084	0.124	0.070	0.126	0.185	0.115	0.105	0.126	0.152	0.113	0.087	0.163	0.109
Iris	0.992	1.000	0.857	1.000	1.000	1.000	0.843	0.713	0.429	0.200	1.000	1.000	1.000	1.000	1.000
Musk	0.204	0.017	0.962	0.052	0.022	0.042	0.254	0.492	0.473	0.873	1.000	1.000	1.000	1.000	0.996
Pima	0.347	0.633	0.594	0.466	0.642	0.584	0.517	0.613	0.591	0.304	0.536	0.618	0.560	0.544	0.734
Sonar	0.425	0.977	0.904	0.983	0.774	0.970	0.879	0.661	0.887	0.329	1.000	0.921	0.911	0.977	0.902
Wbc	0.925	0.967	0.967	0.936	0.970	0.817	0.683	0.946	0.825	0.412	0.965	0.959	0.964	0.961	0.968
Wdbc	0.805	1.000	0.918	0.997	0.951	0.520	0.651	0.543	0.894	0.298	0.915	0.940	0.894	0.942	0.989
Arr	0.405	0.525	0.481	0.409	0.288	0.532	0.449	0.465	0.514	0.457	0.468	0.508	0.419	0.487	0.495
Credit	0.696	0.508	0.702	0.783	0.229	0.275	0.407	0.922	0.866	0.444	0.829	0.839	0.846	0.776	0.888
German	0.209	0.493	0.315	0.261	0.362	0.207	0.268	0.358	0.287	0.062	0.214	0.341	0.100	0.292	0.452
Heart	0.675	0.812	0.796	0.799	0.841	0.686	0.813	0.955	0.811	0.268	0.878	0.800	0.920	0.912	0.954
Average	0.450	0.491	0.686	0.561	0.467	0.426	0.522	0.643	0.494	0.315	0.635	0.596	0.627	0.589	0.787

Furthermore, as observed from the results in Table 3 and Table 4, HKFD consistently achieves superior performance on purely nominal datasets, such as Monks1, Monks2, Monks3, Mush, and Vote. This advantage stems from the adoption of the hybrid kernelized fuzzy relation, which provides a more sophisticated mechanism for measuring similarities between categorical attributes compared to existing methods that rely on simple linear distance metrics or are restricted to numerical feature spaces.

Despite its robust performance, HKFD has certain limitations that warrant further investigation. First, the construction of the hybrid fuzzy relation matrix entails a computational complexity of $O\left(m{n}^2\right)$. While acceptable for the benchmark datasets used in this study, this quadratic complexity limits the method’s direct scalability to million-scale datasets. Second, the current HKFD framework is optimized for static batch processing. In scenarios involving high-speed streaming data, the need to re-evaluate fuzzy information granules for new incoming samples could lead to efficiency bottlenecks. Developing an incremental version of HKFD for dynamic environments is a promising direction for future research.

5.4. Statistical Analysis

Referring to existing related research [9,10,22], we employ the Friedman test and the Nemenyi post-hoc test for a rigorous statistical analysis of the experimental results.

The Friedman test is a non-parametric statistical test designed to detect significant differences between multiple competing methods (our 15 methods) across multiple datasets (our 20 datasets). It operates by ranking the algorithms on each dataset based on their performance (e.g., AUC score). The null hypothesis of the Friedman test is that all algorithms perform equivalently, and thus their average ranks are statistically indistinguishable. If the test yields a p-value below a chosen significance level (e.g., $\alpha =0.1$), this null hypothesis is rejected, confirming that statistically significant differences exist among the methods. When the Friedman test confirms a significant overall difference, it is necessary to use a post-hoc test to determine which specific pairs of algorithms are significantly different from each other. For this purpose, we use the Nemenyi post-hoc test. This test calculates a Critical Difference (CD) value. The performance between any two algorithms is considered statistically significantly different if the difference in their average ranks across all datasets exceeds this CD value.

We employ the AUC results for this statistical analysis. As shown in Table 3, our experiments involve 15 methods and 20 datasets. Therefore, in the Friedman test, we can calculate the degrees of freedom for the ${\tau }_F$ distribution to be 14 and 266. We set the significance level $\alpha =0.1$, yielding a test statistic ${\tau }_F=8.3371$, which exceeds the critical value of 1.5303. This indicates that statistically significant differences exist among the 15 methods, necessitating the Nemenyi post-hoc test to further distinguish pairwise differences.

Also, at the $\alpha =0.1$ significance level, we calculate the CD value as ${\mathrm{CD}}_{0.1}=4.4675$. To better compare the methods in the Nemenyi test, we visualize the results in Figure 4. From the figure, our method HKFD only has CD line segment overlaps with ADkNRS, GBMOD, ECOD, MIX, MGBOD, and VOS, for a total of six methods. This shows that there is no statistically significant difference between HKFD and these six methods, but there is a statistically significant difference between HKFD and the remaining eight methods.

Figure 4. Nemenyi test figure on AUC.

5.5. Hyperparameter Sensitivity Analyses

In the calculation of the hybrid kernelized fuzzy relation, HKFD involves one adjustable hyperparameter, $\delta$, which controls the kernel bandwidth. We now analyze the sensitivity of HKFD to $\delta$ by observing the change in its AUC performance as $\delta$ varies.

As shown in Figure 5, HKFD is relatively robust to changes in $\delta$ on most datasets, and its detection performance does not fluctuate excessively as $\delta$ increases. However, on some datasets, such as Glass, Credit, and Musk, the detection performance of HKFD gradually deteriorates as $\delta$ increases. This phenomenon can be explained by the role of the kernel bandwidth. The parameter $\delta$ controls the smoothness of the fuzzy similarity measure. When $\delta$ becomes too large (e.g., approaching 0.8), it leads to an over-smoothing effect. The value of $K^{*}({\mathit{s}}_i,{\mathit{s}}_j)=exp(-H{({\mathit{s}}_i,{\mathit{s}}_j)}^2/\delta )$ will become relatively high even for samples that are moderately far apart. As a result, all samples, including anomalies, are considered more similar to each other. This causes the hybrid kernelized fuzzy density to be high for all samples, effectively flattening the density landscape and thus diminishing the model’s discriminative power.

Figure 5. Hyperparameter sensitivity analysis of HKFD on AUC.

Overall, HKFD is not highly sensitive to the choice of $\delta$ in most cases. Based on the stable performance observed in the lower end of the tested range across many datasets, a recommended $\delta$ value could be selected from the interval $[0.02,0.08]$.

6. Conclusions

In this study, we address the persistent challenges of unsupervised anomaly detection in mixed-type datasets. We proposed a novel framework, the HKFD. The core innovations of HKFD are twofold: First, it introduces a hybrid kernelized fuzzy relation by integrating a hybrid distance metric directly into a kernel function, enabling it to model nonlinear similarities for both numerical and categorical data. Second, it develops an information-theoretic weighting scheme based on hybrid kernelized fuzzy information entropy. This mechanism allows HKFD to evaluate the importance of each feature subspace adaptively, assigning higher weights to more uncertain and informative attributes that are critical for identifying anomalies. Comprehensive experiments were conducted on 20 real-world benchmark datasets against 14 state-of-the-art baseline methods. The results demonstrated the superior effectiveness and robustness of HKFD. Our method achieved the highest average AUC and AP scores. Furthermore, statistical analysis via the Friedman and Nemenyi tests confirmed that HKFD’s performance advantage is statistically significant over the majority of competitors. The sensitivity analysis also showed that HKFD remains effective across a stable range of its kernel bandwidth hyperparameter $\delta$.

Despite its strong performance, HKFD has limitations. Its primary drawback is the computational complexity of $O\left(m{n}^2\right)$, which is inherent in the construction of the kernelized fuzzy relation matrices for all m attributes. This makes it challenging to apply directly to very large-scale datasets. For future work, we will focus on addressing this scalability issue. Potential directions include exploring advanced techniques, such as granular-ball computing or landmark-based approaches, to accelerate the computation. Additionally, we plan to extend the HKFD framework to semi-supervised anomaly detection and investigate the impact of other kernel functions and information-theoretic measures.