Text Classification Using Intuitionistic Fuzzy Set Measures—An Evaluation Study

Abstract

A very important task of Natural Language Processing is text categorization (or text classification), which aims to automatically classify a document into categories. This kind of task includes numerous applications, such as sentiment analysis, language or intent detection, heavily used by social-/brand-monitoring tools, customer service, and the voice of customer, among others. Since the introduction of Fuzzy Set theory, its application has been tested in many fields, from bioinformatics to industrial and commercial use, as well as in cases with vague, incomplete, or imprecise data, highlighting its importance and usefulness in the fields. The most important aspect of the application of Fuzzy Set theory is the measures employed to calculate how similar or dissimilar two samples in a dataset are. In this study, we evaluate the performance of 43 similarity and 19 distance measures in the task of text document classification, using the widely used BBC News and BBC Sports benchmark datasets. Their performance is optimized through hyperparameter optimization techniques and evaluated via a leave-one-out cross-validation technique, presenting their performance using the accuracy, precision, recall, and F1-score metrics.

1. Introduction

Text classification is one of the most important tasks of Natural Language Processing (NLP) and Machine Learning (ML). The role of text classification (or text categorization) is to classify a document into predefined categories automatically. Texts contain much information that can be crucial for businesses; however, the manual extraction of such information can be very difficult and time consuming. For this reason, companies use text classifiers in order to manage the vast plethora of free texts in an effective way.

Text classification has experienced great interest in the literature, with methodologies focusing on various aspects of the classification and analysis process. One of the most important aims in the field is sentiment analysis [1,2], which determines if a text has a positive, negative, or neutral meaning. Other approaches aim to detect the language the text is written in [3,4] or even its intention, especially in customer conversations where they aim to understand the customer’s purposes [5]. All these applications are mainly used in social media monitoring, brand monitoring, customer service, and the voice of the customer. Another common and important application of text classification is topic labeling, which aims to find the topic of a free text. A significant implementation of topic labeling is the organization or recommendation of articles according to their purposes [6].

The most common algorithms for text classification use machine learning and deep learning methods, which present very good results. Naive Bayes (NB), Support Vector Machine (SVM), and K-Nearest Neighbor (KNN) are some of the most-used classifiers in machine learning for text classification. In deep learning, Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) are the most types of networks used to classify documents.

A very fitting mathematical theory for these kinds of problems is Fuzzy Set (FS) theory. FS theory, in contrast to classical set theory which assesses the belongingness of an element to a set in binary terms, allows for gradual assessments. Zadeh [7] proposed the theory of FSs in 1965 which introduced the degree of membership. FSs have become very useful, and their application has been considered in many real-world problems in order to deal with vague information. One of the most successful applications of Fuzzy Sets in the industrial and commercial field was achieved with fuzzy controllers [1]. Furthermore, they are widely used in problems with incomplete and imprecise information such as bioinformatics [8]. Another crucial application of FSs is the automobile industry using Mamdani’s approach. Later, in 1986, Atanassov [9] introduced Intuitionistic Fuzzy Sets (IFSs), which are an improved version of FSs as they are able to solve more complex problems, introducing the non-membership function.

The application of FS theory in pattern recognition, medical diagnosis, image segmentation, or classification in general, has been considered and employed for many years now, and it is enabled by the use of Intuitionistic Fuzzy measures. Those measures aim to calculate the distance or similarity between the fuzzy patterns of a class and that of a new sample to determine to which the class it belongs. For this reason, the literature has focused extensively to develop such mathematical measures, which can be used to calculate the distance between patterns as best as possible, maximizing the distance between samples of different classes and minimizing it on similar samples. Moreover, FSs have the advantage of being able to handle uncertainties, which is a very common drawback of many approaches. Taking into account the important requirements of text classification combined with the usefulness of FS theory, the need to evaluate the existing IFS measures for their application in text document classification is required. For this purpose, in this work, we use a framework for text classification that was proposed by [10], with the aim of evaluating the application of IFS theory for text classification and the different IFS measures that have been proposed in the last 30 years, as well as to highlight the important aspects of the application of FS theory in the specific domain.

This paper contributes to the thorough study of text classification using Fuzzy Sets, with several different distance and similarity measures being tested on two famous text classification datasets. Moreover, the effectiveness of using FSs and IFSs for the specific problem is highlighted through a detailed comparison of the measures, using the framework mentioned above, as well as comparing their application with other, conventional, machine learning models.

The rest of the paper is organized as follows. Section 2 provides the related works on text classification using FS theory. Section 3 presents some information about Fuzzy and Intuitionistic Fuzzy Sets, along with the datasets, tools and evaluation protocol used to conduct the experiments. Section 4 shows the results obtained, and Section 5 discusses the results and compares them to those of conventional machine learning models. Finally, Section 6 concludes the paper.

2. Related Work

The theory of FS was proposed in 1965 by Zadeh [7] in order to deal with a large number of functional problems. In fact, FSs are the sweeping form of classical sets, which were carried out, basically, in control systems, as early as the 1980s. FSs allow to gradually assess if an element belongs to a set via a membership function, which values the belongingness in the real unit interval $[0,1]$. This number reflects the degree of association of the element with a given set, with 0 meaning it does not belong to the set, whereas 1 means that it completely belongs to the set. This degree of association is called the membership value, and the non-association of a number in a set can be described by a value called the non-membership value, represented by 1 minus the membership value of the number.

A very common application of FSs is feature selection for text classification, where the incorporation of the fuzziness and fuzzy logic in feature selection has shown improvements. A. Z. Bushra et al. [11] presented a feature selection methodology aiming to improve the existing issue of selecting overlapping features.

Fuzzy Sets and similarity measures have been exploited for text classification with good results [12]. Fuzzy Sets have been used in a Similarity-Based Concept Mining Model [13] for text categorization into pre-defined category groups. They have also been used for dimensionality reduction of the feature vector in the process of feature clustering in text classification [14]. Y. Jiang et. al [15] proposed a method that classified multi-labeled text using fuzzy similarity and K-Nearest Neighbors (KNNs), which showed very competitive results compared to other methods, avoiding the high computational cost of a KNN using a fuzzy similarity measure to group the patterns into clusters. Moreover, IFSs and similarity measures have been used for text categorization, yielding good results [10]. The proposed framework exploits statistical methods to represent a document as an IFS, “learning” then its IFS patterns. Szmidt and Kacprzyk [16] proposed another text classification algorithm, using IFSs, highlighting their importance for text classification due to their robustness in imbalanced classes.

More recently, the literature has shifted towards combining the theory of fuzzy logic and Fuzzy Sets with machine learning, especially Convolutional Neural Networks (CNNs), as they have shown great success in Natural Language Processing problems. For example, M. Bounabi et al. [17] exploited fuzzy logic and improved the performance of ML models such as Naive Bayes, Support Vector Machines, and Random Forest classifiers. On the other hand, B. Behera [18] combined a Fuzzy-Rough-Set-based robust nearest neighbors algorithm with a CNN-based feature extraction method, for text document classification and feature extraction, showing important improvements compared to other approaches and models. Moreover, S. Puri [19] presented two systems for text classification, called the Concept-Based Mining Model using a Threshold and the Fuzzy Similarity-based Concept Mining Model using Feature Clustering. These models perform the processes of cutting off, feature extraction, and feature selection during the preprocessing steps, showing high feature reduction and classification accuracy rates.

3. Materials and Methods

3.1. Important Definitions

Here, we describe the definitions of Fuzzy Sets and Intuitionistic Fuzzy Sets, as well as the definitions and constraints of a distance and similarity measure when applied in Fuzzy Set theory.

3.1.1. Fuzzy Sets

A Fuzzy Set A in a universe of discourse X is defined, according to FS theory [7], as a set of ordered pairs:

$$A=\{(x,{\mu }_A\left(x\right))/xϵX\}$$

(1)

where the function ${\mu }_A:X\to [0,1]$ defines the degree of membership of the element $xϵX$.

3.1.2. Intuitionistic Fuzzy Sets

Similarly, an Intuitionistic Fuzzy Set A in X is defined as follows [9]:

$$A=\{x,{\mu }_A\left(x\right)),{\nu }_A\left(x\right)/xϵX\}$$

(2)

where the function ${\nu }_A:X\to [0,1]$ defines the degree of non-membership of an element $xϵX$, such that

$$0\le {\mu }_A\left(x\right)+{\nu }_A\left(x\right)\le 1,\forall xϵX$$

(3)

An important extension to IFS theory is the introduction of the hesitation (or hesitancy) degree of an element x belonging to a set A, defined as follows:

$${\pi }_A\left(x\right)=1-{\mu }_A\left(x\right)-{\nu }_A\left(x\right)$$

(4)

It is obvious from Equations (3) and (4) that $0\le {\pi }_A\left(x\right)\le 1,\forall xϵX$. It should also be noted that in the case of FSs, an element’s hesitation degree is ${\pi }_A\left(x\right)=0,\forall xϵX$, which can be described as a particular case of an IFS.

3.1.3. Distance Measure

A distance measure describes how different elements of a set are, and its definition is described as follows.

Definition 1.

A distance d in a nonempty set A is a real function $d:A\times A\to [0,+\infty )$ that satisfies the following conditions, $\forall x,y,zϵA$ [20]:

 C1

$d(x,y)=0\iff x=y$ (coincidence);

 C2

$d(x,y)=d(y,x)$ (symmetry);

 C3

$d(x,z)+d(z,y)\ge d(x,y)$ (triangle inequality).

A very simple function to measure the distance between two FSs A and B is the Euclidean distance [21]:

$$e_1(A,B)=\sqrt{\sum _{i=1}^n{({\mu }_A\left(x_i\right)-{\mu }_B\left(x_i\right))}^2}$$

(5)

The above Equation (5) was extended for application in IFSs to incorporate the non-membership values $\nu (.)$ [22]:

$$e_2(A,B)=\sqrt{\frac{1}{2}{\displaystyle \sum _{i=1}^N}{({\mu }_A\left(x_i\right)-{\mu }_B\left(x_i\right))}^2+{({\nu }_A\left(x_i\right)-{\nu }_B\left(x_i\right))}^2}$$

(6)

as well as the incorporation of the hesitation degrees [23]:

$$e_3(A,B)=\sqrt{\frac{1}{2}{\displaystyle \sum _{i=1}^N}{({\mu }_A\left(x_i\right)-{\mu }_B\left(x_i\right))}^2+{({\nu }_A\left(x_i\right)-{\nu }_B\left(x_i\right))}^2+{({\pi }_A\left(x_i\right)-{\pi }_B\left(x_i\right))}^2}$$

(7)

3.1.4. Similarity Measure

A similarity measure is defined as follows:

Definition 2.

A similarity S in a nonempty set A is a real function $S:A\times A\to [0,1]$ that satisfies the following conditions, $\forall x,y,zϵA$ [24,25,26]:

 C1

$S(x,y)=1$ if and only if $x=y$ (coincidence);

 C2

$S(x,y)=S(y,x)$ (symmetry);

 C3

$ifx\subseteq y\subseteq z$, then $S(x,z)\le S(x,y)$ and $S(x,z)\le S(y,z)$ (triangle inequality).

The first similarity measure was proposed by S. M. Chen [27]:

$$S(A,B)=\frac{1}{N}\sum _{i=1}^N\left(1-\frac{|S_A\left(x_i\right)-S_B\left(x_i\right)|}{2}\right)$$

(8)

where $S_k\left(x_i\right)={\mu }_k\left(x_i\right)-{\nu }_k\left(x_i\right)$, $k=\{A,B\}$.

3.2. Evaluation Protocol

The protocol followed in our study was proposed by P. Intarapaiboon [10]. This protocol performs a pattern learning process, where the IFS patterns of each class of document are learned. Using those learned patterns, any new document can be classified by extracting its IFS patterns. The classification process is performed by finding the class in which the IFS patterns have the lowest distance or highest similarity with the new document.

Figure 1 shows the preprocessing step of the protocol, which includes the document preprocessing and the IFS pattern learning. In this step, the raw documents are cleaned, converted into a bag-of-words vector, and then, transformed into the IFS-based representation. Lastly, the IFS-based representation is used to calculate the IFS patterns of each class of document (called pattern learning).

At the testing step, a test document $d_i$ is converted into the IFS representation, and the class $C_k$ with the closest pattern to the test document is selected as the document’s class. The closest pattern is determined by using a distance or similarity measure.

3.2.1. Bag-of-Words Representation

This type of representation is the most widely used method for representing a text document as numbers in NLP. Based on this process, a text document is represented by the frequency of occurrence of each word. For a bag-of-words to be formulated, it requires a vocabulary of the known words (the words to be included) or the usage of a threshold value to cut off words with low frequency. Thus, a document’s $d_i$ bag-of-words vector V for h words can be represented as follows:

$$V=(n_{i,1},n_{i,2},n_{i,3},\dots ,n_{i,h})$$

(9)

where n is the number of occurrences of each word of the vocabulary.

3.2.2. IFS Representation

To transform a bag-of-words representation of a document $d_i$ into an IFS-based representation, the following steps were followed (as proposed by [10]). Let us define

$$z_{i,j}=\frac{n_{i,j}-mea{n}_j}{st{d}_j}$$

(10)

where $n_{i,j}$ is the number of occurrences of word j in $d_i$ and $mea{n}_j$ and $st{d}_j$ are the mean and standard deviation of occurrences of word j in all documents of each class.

Thus, to calculate the membership and non-membership values of $d_i$, a weighted sigmoid function is used:

$${\mu }_{i,j}=\frac{r_j}{1+e^{-z_{i,j}}}$$

(11)

$${\nu }_{i,j}=\frac{r_j^{*}}{1+e^{z_{i,j}}}$$

(12)

where $r_j$ and $r_j^{*}$ are weights $ϵ[0,1]$. The hesitation degree is then calculated using Equation (4).

3.2.3. IFS Pattern Learning

Lastly, for the pattern learning step, the IFS pattern of class $C_k$ is denoted by $P_k$, is defined by:

$$P_k=\left\{({\tilde{\mu }}_k,{\tilde{\nu }}_k)\right\}$$

(13)

where ${\tilde{\mu }}_k$ and ${\tilde{\nu }}_k$ are the average values of the membership and non-membership values of all the documents belonging to class $C_k$.

3.2.4. Document Classification

A document’s class is determined according to how dissimilar (for distances) or similar (for similarities) its patterns are to the class patterns calculated in the previous step. To calculate a new document’s IFS representation, the $mea{n}_j$ and $st{d}_j$ values calculated during the class pattern learning step are used.

Therefore, to determine a new document’s $d_n$ class $C^{\prime }$, the following equations can be used to classify a test document:

$$C^{\prime }=arg\underset{C_k}{min}\left\{Dist(P_k,IF{S}_{d_n})\right\}$$

(14)

$$C^{\prime }=arg\underset{C_k}{max}\left\{Sim(P_k,IF{S}_{d_n})\right\}$$

(15)

where $Dist$ and $Sim$ are the Distance and Similarity measures, respectively between the IFSs, $P_k$ are the class patterns calculated during the IFS pattern learning step for class $C_k$, and $IF{S}_{d_n}$ is the IFS representation of the new document $d_n$.

3.3. Experimental Setup

3.3.1. Distance and Similarity Measures

In this study, we evaluated the performance of 43 similarity measures and 19 distance measures, all provided by the fsmpy Python library [28].

Table 1. Similarity measures evaluated in the study and provided by the fsmpy library.

Authors Names	Parameters	Measure Name	Reference
S. M. Chen	weights, a, b, c	$S_e$	[30]
D.H. Hong, C. Kim	weights, a, b, c	$M_H$	[31]
L. Dengfeng, C. Chuntian	p, weights	$S_d^p$	[24]
Z. Liang, P. Shi	p, weights	$S_e^p$, $S_s^p$	[32]
H.B. Mitchell	p, weights	$S_{mod}$	[25]
W.L. Hung, M.S. Yang	weights	$S_c$, $S_e$, $S_l$	[26]
H.W. Liu	p, weights, a, b, c	T	[33]
C. Zhang, H. Fu	-	S	[34]
W.L. Hung, M.S. Yang	a	$S_p^c$, $S_p^e$, $S_p^l$	[35]
S. Park, Y.C. Kwun, K.M. Lim	p, weights	$S_{gw}^p$	[36]
W.L. Hung, M.S. Yang	-	$S_{new2}$, $S_{pk1}$, $S_{pk2}$, $S_{pk3}$, $S_{w1}$, $S_{w2}$	[37]
W.L. Hung, M.S. Yang	p	$S_a^c$, $S_a^e$, $S_a^l$	[38]
J. Ye	weights	$C_{IFS}$	[39]
C.M. Hwang, M.S. Yang	-	$S_{IFS}$	[40]
P. Julian, K.C. Hung, S. Lin	p, weights	$S_{new,p}$	[41]
I. Iancu	lamda	${\tilde{S}}_1$, ${\tilde{S}}_5$, ${\tilde{S}}_2$, ${\tilde{S}}_6$, ${\tilde{S}}_2^{\prime }$, ${\tilde{S}}_6^{\prime }$, ${\tilde{S}}_1^c$, ${\tilde{S}}_5^c$	[42]
G. Deng, Y. Jiang, J. Fu	weights, p, u, v	$L_5$, $N_6$, $F_5$, $F_6$, $F_7$	[43]
Y. Song, X. Wang, L. Lei, A. Xue	weights	$S_{WY}$	[44]
S.M. Chen, S. Cheng, T.C. Lan	weights	$S_{CCL}$	[45]
P. Muthukumar, G.S.S. Krishnan	weights	$W_{IFSSs}$	[46]

and

Table 2. Distance measures evaluated in the study and provided by the fsmpy library.

Authors Names	Parameters	Measure Name	Reference
K.T. Atanassov	-	$d_e$, $d_h$, $d_{ne}$, $d_{nh}$	[22]
E. Szmidt, A. Kacprzyk	-	$d_{IFS}^1$, $e_{IFS}^1$, $l_{IFS}^1$, $q_{IFS}^1$	[23]
P. Grzegorzewski	-	$d_{h2}$, $e_h$, $l_h$, $q_h$	[29]
W. Wang, X. Xin	weights, p	$d_1$, $d_2^p$	[20]
Y. Yang, F. Chiclana	-	$d_{eh}$, $l_{eh}$, $e_{eh}$, $q_{eh}$	[47]
I.K. Vlachos, G.D. Sergiadis	-	$D_{IFS}$	[48]

show the names of the authors that proposed the corresponding measure in the first column, the parameters (other than the two Intuitionistic Fuzzy Sets A and B and the type of similarity) that are required by the functions to calculate the corresponding measure, and in the third column, the name of the proposed function that the authors used in the original study to mathematically describe it. As two of the measure $d_h$ proposed by [22,29] have the same name, the latter one was renamed to $d_{h2}$. Furthermore, all the mathematical expressions of the measures are presented in Appendix A.

At this point, it should be mentioned that not all of the measures available in the fsmpy library were compared in this study. A total of 27 measures were not included as their implementations are not trustworthy enough, failing to pass the required tests by the library. Specifically, the rest of the measures proposed by [32] ($S_h^p$), [43] (namely, $L_3$, $L_4$, $L_6$, $N_4$, $N_5$, and $N_7$), [37] ($S_{new1}$) [42] (namely, ${\tilde{S}}_{11}$, ${\tilde{S}}_{19}$, ${\tilde{S}}_{14}$, ${\tilde{S}}_{18}$, ${\tilde{S}}_{13}$, ${\tilde{S}}_{15}$, ${\tilde{S}}_{13}^{\prime }$, ${\tilde{S}}_{15}^{\prime }$, ${\tilde{S}}_{11}^c$, ${\tilde{S}}_{19}^c$, ${\tilde{S}}_{14}^c$, ${\tilde{S}}_{18}^c$, ${\tilde{S}}_2^c$, ${\tilde{S}}_6^c$, ${\tilde{S}}_{13}^c$, ${\tilde{S}}_{15}^c$), [49] ($S_F$) and [27] (T) were not included.

It is worth mentioning that in order to optimize the performance of the measures, a hyperparameter optimization technique was employed using both the fsmpy and Scikit-Optimize libraries. For the optimization process, we employed the Bayesian optimization technique, which, contrary to the Grid Search optimization technique, does not try out all the parameter values, but only a fixed number of parameter settings from specified distributions. The performance of each parameter combination was tested over a 10-fold cross-validation technique, with the best one being selected according to the highest F1-score metric. It should be noted that the weights parameter was set to $weights=\overrightarrow{1/n}$, where n is the number of elements in the set.

3.3.2. Datasets

The datasets that were used in our experiments are presented and described below:

• BBC News: This consists of 2225 articles belonging to 5 topic areas (business, entertainment, politics, sports, and tech) from 2004 and 2005, with a total of 9635 words [50].
• BBC Sports: Similar to the previous one, it contains 737 articles from 5 areas, namely athletics, cricket, football, rugby, and tennis, having a total of 4613 words [50].

Figure 2 shows, for each dataset, the number of samples for each class, along with the mean number of words of the sample texts. More specifically, as shown in Figure 2a, the classes in the BBC News dataset have a balanced distribution of samples, with 386 entertainment-type and 511 sports-type documents. The classes have a balanced word count as well, with the texts belonging to the class business having a mean word count of 339 (the lowest) and tech having 513 (the highest).

For the BBC Sports case though, the distribution of the samples is not as balanced. As shown in Figure 2b, the football class has the highest number of samples (265), with the rest of the classes having 100–150 samples On the other hand, the mean number of words per text is balanced across all classes, with the tennis class having a mean word count of 298 (lowest) and cricket having 391 (highest).

3.3.3. Data Preparation

The data preparation step consisted of the following steps:

• Documents’ conversion to bag-of-words;
• Word-by-word preprocessing:

  (a)

  Remove escape characters;

  (b)

  Convert to lower case;

  (c)

  Lemmatize word.

• Cut-off words with a document frequency lower than a $0.01$ frequency.

It is worth noting that lemmatization is the process of converting variations of a word to its base form, for example: “does”, “doing”, and “did” were converted to “do” and “am” and “are” and “is” to “be”. For this process, the Scikit-learn [51] and Natural Language Toolkit [52] libraries were used. Moreover, the chosen by trial and error cut-off frequency resulted in the BBC News dataset containing a bag-of-words of 2540 words and BBC Sports 2396 words.

3.3.4. Performance Evaluation

The performance of each measure was evaluated using the accuracy, precision, recall, and F1-score metrics, as well as the Degree of Confidence (DoC) proposed by A.G. Hatzimichailidis et al. [53]. To calculate the Degree of Confidence metric, the fsmpy library was used and the Scikit-learn library for the rest of the metrics. These metrics are widely used in classification studies to evaluate the performance of an algorithm or model, with the DoC being used in the literature on FS theory.

More specifically, the DoC measures the confidence at which the measure can recognize a specific sample belonging to a specific pattern. As a result, higher values of the DoC show that the measure has more confidence when predicting a sample’s class. Moreover, to calculate the precision, recall, and F1-score, we used their “macro” average, calculating the metrics for each label and finding their unweighted mean.

4. Results

In our study, we evaluated the performance of 43 similarity measures and 19 distance measures, in the classification of text documents of two different datasets. The experiments were conducted using the Python programming language along with the Scikit-learn and fsmpy libraries. Moreover, we examined the performance of each parameter group by following a grid search technique for $r_j$ and $r_j^{*}$, by selecting five linearly spaced values over the interval $[0.1,1]$. Specifically, the $r_j$ and $r_j^{*}$ values that were tested were the following: $\{0.1,0.325,0.55,0.775,1\}$

After finding the best parameters of each measure, along with their best membership and non-membership weights, we validated their performance on each dataset by employing a leave-one-out cross-validation technique. In this validation technique, a single sample is considered for testing in each fold.

Table 3. Performance of distance measures on BBC News dataset.

Distance Measures	Parameters	Weights (${\mathit{r}}_{{\mathit{j}}_1}$, ${\mathit{r}}_{{\mathit{j}}_1}^{*}$)	Accuracy %	Precision %	Recall %	F1 %	DoC
$d_e$	-	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0903
$d_h$	-	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	3.9711
$d_{ne}$	-	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0018
$d_{nh}$	-	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0016
$d_{IFS}^1$	-	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0903
$e_{IFS}^1$	-	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	3.9711
$l_{IFS}^1$	-	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0018
$q_{IFS}^1$	-	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0016
$d_h$	$p=7.00$	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0016
$e_h$	$p=7.00$	($0.1$, $0.1$)	88.94	90.45	88.98	88.63	0.0031
$l_h$	-	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0903
$q_h$	-	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	3.9711
$d_1$	-	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0018
$d_2^p$	-	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0016
$d_{eh}$	-	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0903
$l_{eh}$	-	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	3.9711
$e_{eh}$	-	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0018
$q_{eh}$	-	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0016
$D_{IFS}$	-	($0.1$, $1.0$)	94.25	94.10	93.98	94.02	5.5846

,

Table 4. Performance of similarity measure on BBC News dataset.

Similarity Measures	Parameters	Weights (${\mathit{r}}_{{\mathit{j}}_1}$, ${\mathit{r}}_{{\mathit{j}}_1}^{*}$)	Accuracy %	Precision %	Recall %	F1 %	DoC
$S_e$	$a=6.00$, $b=-4.00$, $c=7.00$	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0016
$M_H$	$a=6.00$, $b=-4.00$, $c=7.00$	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0016
$S_d^p$	$p=7.00$	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0018
$S_e^p$	$p=7.00$	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0018
$S_s^p$	$p=7.00$	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0018
$S_{mod}$	$p=7.00$	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0018
$S_c$	-	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0016
$S_e$	-	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0025
$S_l$	-	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0031
T	$a=0.34$, $b=0.47$, $c=0.42$, $p=9.00$	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0018
S	-	($0.1$, $1.0$)	50.65	61.21	49.05	39.49	0.0091
$S_p^c$	$a=6.00$	($1.0$, $0.1$)	94.83	94.67	94.62	94.63	0.0019
$S_p^e$	$a=6.00$	($1.0$, $0.1$)	94.83	94.67	94.62	94.63	0.0021
$S_p^l$	$a=6.00$	($1.0$, $0.1$)	94.83	94.67	94.62	94.63	0.0022
$S_{gw}^p$	$p=7.00$	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0025
$S_{new2}$	-	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0023
$S_{pk1}$	-	($1.0$, $0.1$)	50.47	60.86	48.9	39.26	0.0293
$S_{pk2}$	-	($1.0$, $0.1$)	75.06	77.75	74.99	74.56	0.0418
$S_{pk3}$	-	($1.0$, $0.1$)	50.47	60.86	48.9	39.26	0.0157
$S_{w1}$	-	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0017
$S_{w2}$	-	($0.1$, $0.1$)	49.17	39.46	47.68	37.84	0
$S_a^c$	$p=7.00$	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0026
$S_a^e$	$p=7.00$	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0033
$S_a^l$	$a=6.00$, $b=-4.00$, $c=7.00$	($0.1$, $0.1$)	49.66	61.04	48.18	38.51	0.0016
$C_{IFS}$	-	($0.1$, $0.325$)	94.74	94.67	94.57	94.56	0.0033
$S_{IFS}$	-	($0.55$, $0.325$)	93.71	93.44	93.47	93.44	0.0019
$S_{new,p}$	$p=7.00$	($0.325$, $0.1$)	93.17	93.19	92.76	92.86	0.0040
${\tilde{S}}_1$	-	($0.1$, $1.0$)	71.91	80.40	73.01	73.03	0.0097
${\tilde{S}}_5$	-	($1.0$, $0.1$)	59.78	75.68	60.60	61.30	0.0112
${\tilde{S}}_2$	-	($0.55$, $0.1$)	75.55	84.69	74.10	73.14	0.0007
${\tilde{S}}_6$	-	($0.1$, $0.55$)	91.69	92.20	91.84	91.49	0.0006
${\tilde{S}}_2^{\prime }$	-	($1.0$, $1.0$)	55.33	58.77	56.51	55.42	0.0091
${\tilde{S}}_6^{\prime }$	-	($0.775$, $0.325$)	93.48	93.36	93.45	93.29	0.0012
${\tilde{S}}_1^c$	-	($1.0$, $0.325$)	64.9	76.35	65.71	66.39	0.0056
${\tilde{S}}_5^c$	-	($0.1$, $0.1$)	90.25	91.11	89.72	89.72	0
$L_5$	$p=7.00$, $u=4.43$, $v=7.22$	($0.775$, $0.55$)	94.29	94.24	94.09	94.10	0.0289
$N_6$	$p=7.00$, $u=4.43$, $v=7.22$	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0018
$F_5$	$p=7.00$, $u=4.43$, $v=7.22$	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0006
$F_6$	$p=7.00$, $u=4.43$, $v=7.22$	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0
$F_7$	$p=7.00$, $u=4.43$, $v=7.22$	($0.1$, $0.1$)	93.17	93.19	92.76	92.86	0.0028
$S_{WY}$	-	($0.1$, $0.325$)	94.47	94.26	94.23	94.24	0.0005
$S_{CCL}$	-	($0.1$, $1.0$)	49.93	60.98	48.42	38.74	0.0083
$W_{IFSSs}$	-	($1.0$, $0.1$)	65.75	81.88	63.86	59.92	0.0285

,

Table 5. Performance of similarity measure on BBC Sports dataset.

Similarity Measures	Parameters	Weights (${\mathit{r}}_{{\mathit{j}}_1}$, ${\mathit{r}}_{{\mathit{j}}_1}^{*}$)	Accuracy %	Precision %	Recall %	F1 %	DoC
$S_e$	$a=6.00$, $b=-4.00$, $c=7.00$	($0.1$, $0.1$)	25.10	83.07	33.90	25.50	0.0014
$M_H$	$a=6.00$, $b=-4.00$, $c=7.00$	($0.1$, $0.1$)	25.10	83.07	33.90	25.50	0.0014
$S_d^p$	$p=7.00$	($0.1$, $0.1$)	96.34	96.54	96.47	96.48	0.0028
$S_e^p$	$p=7.00$	($0.1$, $0.1$)	96.34	96.54	96.47	96.48	0.0028
$S_s^p$	$p=7.00$	($0.1$, $0.1$)	96.34	96.54	96.47	96.48	0.0028
$S_{mod}$	$p=7.00$	($0.1$, $0.1$)	96.34	96.54	96.47	96.48	0.0028
$S_c$	-	($0.1$, $0.1$)	25.10	83.07	33.90	25.50	0.0014
$S_e$	-	($0.1$, $0.1$)	25.10	83.07	33.90	25.50	0.0023
$S_l$	-	($0.1$, $0.1$)	25.10	83.07	33.90	25.50	0.0028
T	$a=0.34$, $b=0.47$, $c=0.42$, $p=9.00$	($0.1$, $0.1$)	96.34	96.54	96.47	96.48	0.0028
S	-	($0.1$, $1.0$)	28.49	83.19	36.36	29.31	0.0081
$S_p^c$	$a=6.00$	($0.1$, $0.1$)	96.47	96.69	96.78	96.72	0
$S_p^e$	$a=6.00$	($0.1$, $0.1$)	96.47	96.69	96.78	96.72	0
$S_p^l$	$a=6.00$	($0.1$, $0.1$)	96.47	96.69	96.78	96.72	0
$S_{gw}^p$	$p=7.00$	($0.1$, $0.1$)	96.34	96.54	96.47	96.48	0.0045
$S_{new2}$	-	($0.1$, $0.1$)	25.10	83.07	33.90	25.50	0.0021
$S_{pk1}$	-	($0.775$, $0.1$)	26.46	83.12	35.38	27.44	0.0269
$S_{pk2}$	-	($0.775$, $0.325$)	79.65	81.51	77.73	78.79	0.0489
$S_{pk3}$	-	($0.775$, $0.1$)	26.46	83.12	35.38	27.44	0.0144
$S_{w1}$	-	($0.1$, $0.1$)	25.10	83.07	33.90	25.50	0.002
$S_{w2}$	-	($0.1$, $0.1$)	24.42	81.04	33.44	24.17	0
$S_a^c$	$p=7.00$	($0.1$, $0.1$)	25.10	83.07	33.90	25.50	0.0024
$S_a^e$	$p=7.00$	($0.1$, $0.1$)	25.10	83.07	33.90	25.50	0.003
$S_a^l$	$a=6.00$, $b=-4.00$, $c=7.00$	($0.1$, $0.1$)	25.10	83.07	33.90	25.50	0.0014
$C_{IFS}$	-	($0.1$, $0.55$)	96.47	96.99	96.71	96.84	0.0023
$S_{IFS}$	-	($1.0$, $0.775$)	95.93	96.54	96.11	96.31	0.0067
$S_{new,p}$	$p=7.00$	($0.325$, $0.1$)	96.34	96.54	96.47	96.48	0.0064
${\tilde{S}}_1$	-	($0.1$, $1.0$)	69.61	76.86	72.68	71.01	0.0072
${\tilde{S}}_5$	-	($1.0$, $0.1$)	50.47	78.02	54.83	54.77	0.0096
${\tilde{S}}_2$	-	($0.775$, $0.1$)	82.36	86.95	87.83	84.91	0.0012
${\tilde{S}}_6$	-	($0.1$, $0.55$)	94.44	93.89	95.63	94.61	0.0006
${\tilde{S}}_2^{\prime }$	-	($1.0$, $1.0$)	40.30	58.81	47.36	43.41	0.0072
${\tilde{S}}_6^{\prime }$	-	($1.0$, $0.1$)	94.30	93.71	95.61	94.50	0.0018
${\tilde{S}}_1^c$	-	($1.0$, $0.325$)	63.50	73.42	66.40	65.26	0.004
${\tilde{S}}_5^c$	-	($0.1$, $0.1$)	90.09	89.41	91.95	89.84	0
$L_5$	$p=7.00$, $u=4.43$, $v=7.22$	($1.0$, $0.775$)	96.20	96.46	96.34	96.37	0.0371
$N_6$	$p=7.00$, $u=4.43$, $v=7.22$	($0.1$, $0.1$)	96.34	96.54	96.47	96.48	0.0028
$F_5$	$p=7.00$, $u=4.43$, $v=7.22$	($0.1$, $0.1$)	96.34	96.54	96.47	96.48	0.0009
$F_6$	$p=7.00$, $u=4.43$, $v=7.22$	($1.0$, $0.775$)	96.20	96.36	96.40	96.35	0
$F_7$	$p=7.00$, $u=4.43$, $v=7.22$	($0.1$, $0.1$)	96.34	96.54	96.47	96.48	0.0045
$S_{WY}$	-	($0.1$, $0.325$)	95.93	97.33	95.64	96.42	0.0006
$S_{CCL}$	-	($0.1$, $1.0$)	25.78	83.09	34.40	26.22	0.0075
$W_{IFSSs}$	-	($1.0$, $0.1$)	62.82	84.04	65.53	64.82	0.0261

and

Table 6. Performance of distance measures on BBC Sports dataset.

Distance Measures	Parameters	Weights (${\mathit{r}}_{{\mathit{j}}_1}$, ${\mathit{r}}_{{\mathit{j}}_1}^{*}$)	Accuracy %	Precision %	Recall %	F1 %	DoC
$d_e$	-	($0.1$, $0.1$)	93.62	96.32	92.79	94.28	0.1001
$d_h$	-	($0.1$, $0.1$)	25.1	83.07	33.9	25.5	3.423
$d_{ne}$	-	($0.1$, $0.1$)	93.62	96.32	92.79	94.28	0.002
$d_{nh}$	-	($0.1$, $0.1$)	25.1	83.07	33.9	25.5	0.0014
$d_{IFS}^1$	-	($0.1$, $0.1$)	93.62	96.32	92.79	94.28	0.1001
$e_{IFS}^1$	-	($0.1$, $0.1$)	25.1	83.07	33.9	25.5	3.423
$l_{IFS}^1$	-	($0.1$, $0.1$)	93.62	96.32	92.79	94.28	0.002
$q_{IFS}^1$	-	($0.1$, $0.1$)	25.1	83.07	33.9	25.5	0.0014
$d_h$	$p=7.00$	($0.1$, $0.1$)	25.1	83.07	33.9	25.5	0.0014
$e_h$	$p=7.00$	($0.1$, $0.1$)	95.66	95.54	96.15	95.77	0.0034
$l_h$	-	($0.1$, $0.1$)	93.62	96.32	92.79	94.28	0.1001
$q_h$	-	($0.1$, $0.1$)	25.1	83.07	33.9	25.5	3.423
$d_1$	-	($0.1$, $0.1$)	93.62	96.32	92.79	94.28	0.002
$d_2^p$	-	($0.1$, $0.1$)	25.1	83.07	33.9	25.5	0.0014
$d_{eh}$	-	($0.1$, $0.1$)	93.62	96.32	92.79	94.28	0.1001
$l_{eh}$	-	($0.1$, $0.1$)	25.1	83.07	33.9	25.5	3.423
$e_{eh}$	-	($0.1$, $0.1$)	93.62	96.32	92.79	94.28	0.002
$q_{eh}$	-	($0.1$, $0.1$)	25.1	83.07	33.9	25.5	0.0014
$D_{IFS}$	-	($0.1$, $0.55$)	95.39	97.07	95.02	95.94	3.6342

show the performance of the similarity and distance measures on each dataset. Each table shows the name of the measure as mentioned in the corresponding study that introduced it, the best parameters (where applied) obtained from the hyperparameter optimization, the best membership and non-membership weights obtained, the accuracy, precision, recall, and F1-score performance metrics, and the mean DoC of the measure.

Specifically, Table 3 and Table 4 show the performance of the similarity and distance measures on the BBC News dataset and Table 5 and Table 6 their performance on the BBC Sports dataset.

5. Discussion

In general, the application of the evaluation protocol proposed by [10] shows very good classification performance, in most of the similarity or distance measures. First of all, the most important aspect of such an approach for text classification is that it requires no training process. As a result, the performance is not affected by any imbalanced class distribution, with the presented study highlighting this advantage. As shown in Figure 2, the class distribution in the BBC Sports dataset is imbalanced, which would result in an overfitting towards one of the classes if a machine learning model were applied.

Secondly, the results highlight the importance of which IFS measure is used for this type of classification problem. Although most of the measures performed above 90% in accuracy and F1-score, there were a few that performed very poorly.

Thirdly, similarity measures showed very poor prediction confidence, with the distance measure (on the contrary) showing very confident predictions. Despite that, they performed (on average) better than the distance measures, although with a very small difference. Thus, the experimental results showed that the distance should be preferred, as it had high classification performance with very good confidence.

Lastly, by comparing the types of similarities, it can be observed that there was no change in the performance, showing that there was no significant difference between them. This can be highlighted more when taking into account the best parameters obtained from the hyperparameter optimization process, where measures of the same type had the same best parameters, as well as membership and non-membership weights.

Below, we discuss the results obtained for each dataset in more detail.

5.1. BBC News Results

It is obvious that the results obtained from different similarity measures, as well as distance measures were quite similar. Starting with the results obtained from the similarity measures, the ones that performed the best were from W.L. Hung and M.S. Yang [35] with 94.83% accuracy, 94.67% precision, 94.62% recall, and 94.63% F1-score. The next-best-performing measure was by J. Ye [39] with 91.70% and Y. Song, X. Wang, L. Lei, and A. Xue [44] with a 94.56% F1-score. Despite the best performance regarding those metrics, their DoC was not the highest as $S_{pk2}$ had the highest DoC despite having much lower performance. There were, though, quite a few measures that performed below 50%, with the worst performance being at 49.17% accuracy, 39.46% precision, 47.68% recall, 37.84% F1-score, and 0 DoC by $S_{w2}$. Therefore, these types of measures are not well suited for document classification, as they fail to capture those important similarities between the Intuitionistic Fuzzy Sets of the documents and classes. Moreover, the measures’ performance regarding the DoC showed that there were no large differences between the similarities calculated for the chosen classes compared to the other classes. This shows that, despite the very good performance of some measures (or even all of them), they were not that confident for their prediction.

As for the distance measures, the best-performing one was that of I.K. Vlachos and G.D. Sergiadis [48] with 94.25% accuracy, 94.10% precision, 93.98% recall, and 94.02% F1-score, followed by all the Euclidean-based distances, specifically $d_e$, $d_{ne}$, $d_{IFS}^1$, $l_{IFS}^1$, $l_h$, $d_1$, $d_{eh}$, and $e_{eh}$ with 93.17% accuracy, 93.19% precision, 92.76% recall, and 92.86% F1-score. In contrast with the DoC of the similarities, the distances seemed to be more confident about their prediction, with $D_{IFS}$ having the highest confidence of 5.5846, showing large distances between the different classes.

5.2. BBC Sports Results

In the case of the BBC Sports dataset, similar ranking results were obtained, but with better performance. Regarding the similarity measures, the measure proposed by J. Ye [39] performed the best with 96.47% accuracy, 96.99% precision, 96.71% recall, 96.84% F1-score, and 0.0023 DoC, followed by the measures proposed by W.L. Hung and M.S. Yang [35] with 96.47% accuracy, 96.69% precision, 96.78% recall, and 96.72% F1-score. The most confident measure was again $S_{pk2}$, despite performing much worse than $C_{IFS}$.

As for the distance measures, the one proposed by I.K. Vlachos and G.D. Sergiadis [48] performed the best, with 95.39% accuracy, 97.07% precision, 95.02% recall, 95.94% F1-score, and 3.6342 DoC, followed by the same Euclidean-based measures, as in the previous dataset, with 93.62% accuracy, 96.32% precision, 92.69% recall, and 94.28% F1-score. Similarly, the distances showed higher confidence about their predictions, highlighting that the distances between the classes were larger.

5.3. Comparison with Standard Machine Learning Approaches

To better assess the application of the evaluation protocol, as well as the IFS measures included in the study, we compared the best results obtained for each case with standard ML models on the same datasets. For the comparison, we employed the Decision Tree (DT) and K-Nearest Neighbors (KNN) ML models. Both models were trained on the same datasets, by following the same data preparation, performance evaluation, and hyperparameter optimization processes.

Table 7. Performance of the ML models and the best-performing IFS measures on both datasets.

	BBC News				BBC Sports
Model/ Measure	Accuracy (%)	Precision (%)	Recall (%)	F1 (%)	Accuracy (%)	Precision (%)	Recall (%)	F1 (%)
Decision Tree	81.61	81.85	81.04	81.31	88.19	88.14	87.87	87.96
KNN	82.69	86.47	81.63	82.79	88.46	92.96	87.42	89.16
$S_p^c$	94.83	94.67	94.62	94.63	96.47	96.69	96.78	96.72
$C_{IFS}$	94.74	94.67	94.57	94.56	96.47	96.99	96.71	96.84
$D_{IFS}$	94.25	94.10	93.98	94.02	95.39	97.07	95.02	95.94

shows the results obtained from the evaluation of the aforementioned models, along with the results from the best performing similarity and distance measures in both datasets.

The results obtained from the conventional ML models highlight the advantages of the IFS-based classification approaches and IFS measures in general. The first advantage is the better performance of the IFS-based approach on both datasets, showing an improvement of 12% on the BBC News dataset and 10% on the BBC Sports dataset. The best-performing ML model, according to the F1-score metric, on both datasets was the KNN classifier with an 82.79% and 89.16% F1-score, respectively. The Decision Tree model performed at about the same, with 81.61% accuracy, 81.85% precision, 81.04% recall, and 81.31% F1-score, a small difference compared to the KNN.

The second advantage of the IFS-based text classification is that it does not include any training process, by-passing the issue of class imbalance and overfitting on the class with the most samples. This is a very important characteristic, as datasets tend to be very imbalanced, with many methodologies being proposed to tackle this issue. The overfitting of the model (and lack thereof on the $S_p^c$) can be observed in Figure 3, where the confusion matrices are shown during the validation of both. In the first case of the DT (Figure 3a), the model overfitted towards the football class, falsely predicting samples of other classes. On the other hand, the $S_p^c$ (Figure 3b) similarity (or the evaluation protocol in general) was not affected by the class imbalance.

6. Conclusions

In this study, we evaluated the performance of 62 Intuitionistic Fuzzy Set measures on the same base towards classifying texts from two datasets, for the first time in the literature. Their performance was evaluated by following the protocol proposed by P. Intarapaiboon [10], where the documents are represented as Intuitionistic Fuzzy Sets by firstly converting the texts into bag-of-words representations. Then, by using a sigmoid function to calculate the membership and non-membership values of the documents (and the classes), given the membership and non-membership weights, the IFS representation and patterns were obtained. Moreover, we optimized the performance of each measure separately, by finding the combination of parameter values and membership and non-membership weights that yielded the best results in a 10-fold cross-validation technique. Their performance was then validated by employing the leave-one-out cross-validation technique.

The results obtained highlight a number of important findings. Firstly, although the functions are different, in the case of text document classification, their performance was not significantly different, as the results obtained for the different measures were the same in most cases. This can be attributed to the fact that some studies aim to overcome the drawbacks of previous measures, such as the measure proposed by H.B. Mitchell [25], who revised the measure proposed by L. Dengfeng and C. Chuntian [24]. Secondly, Hamming-based distance measures had an F1-score lower than 50%, which shows that the problem of text classification is not as simple as calculating the absolute difference between two sets. Thirdly, the similarities maintained a very low degree of confidence during their predictions, which shows that the resulting class patterns were very similar. Moreover, the DoC requires similarities to be converted to distances via the formula $1-S$. As a result, in combination with the definition of the similarities where $S:A\times A\to [0,1]$, it is obvious that their confidence will be much lower compared to distances, whose definition “allows” them to have much higher values $d:A\times A\to [0,+\infty )$. Lastly, different parameters had different effects on each measure, for example in the case of $S_p^e$, for different values of the parameter a, the performance remained almost the same during the hyperparameter optimization technique, with an std of 0.7%, while $L_5$ had an std of 9% for different values of p.

Despite the wide range of different similarity measures included in the study, there were some measures that were not included. For instance, the similarity measure proposed by Z. Xu [54], which was also included in [10], was not evaluated in the study. Additionally, the types of measures evaluated in our study (distances and similarities) are the first ones that have been proposed and employed for problems such as pattern recognition, medical diagnosis, image segmentation, or classification in general. For this reason, our analysis and evaluation were focused on those. In the future, some other types of measures have to be analyzed in the same manner, such as divergence, correlation, inclusion, and others.