A Multi-Criteria Approach for Arabic Dialect Sentiment Analysis for Online Reviews: Exploiting Optimal Machine Learning Algorithm Selection

Abstract

A sentiment analysis of Arabic texts is an important task in many commercial applications such as Twitter. This study introduces a multi-criteria method to empirically assess and rank classifiers for Arabic sentiment analysis. Prominent machine learning algorithms were deployed to build classification models for Arabic sentiment analysis classifiers. Moreover, an assessment of the top five machine learning classifiers’ performances measures was discussed to rank the performance of the classifier. We integrated the top five ranking methods with evaluation metrics of machine learning classifiers such as accuracy, recall, precision, F-measure, CPU Time, classification error, and area under the curve (AUC). The method was tested using Saudi Arabic product reviews to compare five popular classifiers. Our results suggest that deep learning and support vector machine (SVM) classifiers perform best with accuracy 85.25%, 82.30%; precision 85.30, 83.87%; recall 88.41%, 83.89; F-measure 86.81, 83.87%; classification error 14.75, 17.70; and AUC 0.93, 0.90, respectively. They outperform decision trees, K-nearest neighbours (K-NN), and Naïve Bayes classifiers.

1. Introduction

Machine learning (ML) enables systems to learn, adapt, and enhance knowledge without explicit programming [1]. It was applied across diverse domains such as research and business [2,3]. There are many different ML algorithms and parameters as well as experts’ knowledge about the available classification algorithms and domain in order to select an appropriate solution [4]. The fact that ML-oriented datasets have varying intrinsic characteristics, various experts in the field of data analysis may have incomplete knowledge about the application domains. Moreover, the strengths and capabilities of the candidate classifiers may not be fully understood. It is widely known [5,6] that no ML algorithm performs well for all learning problems. It is, therefore, necessary to evaluate ML algorithms separately for each application scenario [7] because domains have diverse, relevant characteristics [8]. Machine learning algorithms perform well with multiple languages as a significant amount of works focused on the English language, followed by Arabic, Chinese, etc.

In the context of performance measures for classification-based algorithms for Arabic sentiment analysis (SA), the prominent measures employed by most of the researchers in ML comprises precision, recall, F-measure, area under the ROC curve [9], balanced accuracy, and average accuracy. However, an analysis using only one of these may select an algorithm that does not yield the optimum performance [10,11]. Moreover, for morphologically rich languages, such as Arabic or dialect Arabic, additional selection criteria are also relevant, such as training time, execution time, and results inconsistency. A multi-criteria-based evaluation approach is, therefore, necessary to identify the optimal solution [6,12]. The primary objective of this study is to introduce a multi-criteria method to identify an appropriate ML algorithm for sentiment analysis of dialect Arabic and to demonstrate it on a corpus of Saudi tweets. We have compared the performance of five machine learning algorithms classifiers: deep learning, decision tree, Naïve Bayes, K-nearest neighbours, and support vector machine with multi-criteria using dialect Arabic dataset of tweets to select better classifier for dialect Arabic. The result of the experiment shows that deep learning and support vector machine classifiers showed better performance results in terms of accuracy, precision, recall, F-measure, and AUC compared to a decision tree, K-nearest neighbours, and Naïve Bayes classifier.

In consideration of the fact that Arabic is a popular language with over 400 million speakers in North Africa, the Middle East, and the Horn of Africa [13,14], it was revealed that the Arabic language is among the world most influential languages that concur significant parts of the world with incredible speakers [15]. The Arabic language is one of the authority languages utilised at the United Nations and an official language of 27 countries. There are three forms of Arabic: Modern Standard Arabic (MSA), Classical Arabic (CA), and Dialectical Arabic (DA) [16,17,18,19]. MSA is used in formal situations and events, CA is found in religious scripts, and DA is used in everyday life, including for reviews and comments posted on social media. It includes national dialects, such as Egyptian and Moroccan. These have evidently revealed the importance of DA for research-oriented considerations among both novice and veteran researchers.

In more specific terms, dialect Arabic (DA) has been estimated to be the communication medium of almost 295 million individuals living mostly in the Middle East and North Africa [15]. The contemporary advent of a social medium of communication has increased DA exponentially across the globe; however, as a Morphologically Rich Language (MRL) where an unusual amount of data concerning syntactic units and relations communicates at the word level. The huge dialect Arabic (DA) online contents involved sentiments that need to be deciphered for more semantically oriented benefits, which empirically motivates the zeal to conduct this study.

2. Related Work

Recently many research was performed on sentiment analysis to identify and categorise opinions expressed in the text, especially social media, or simply for classification purposes. Some of the Machine Learning algorithms used in sentiment analysis are based on multiple criteria such as accuracy, recall, and F-measure [20,21,22,23,24]. Furthermore, the area under the curve, sensitivity, precision, and F-score are presented in [2,25,26]. Wegrzyn-Wolska, Bougueroua [27] conducted a content analysis of social networks by analysing the intensity and polarity of opinions and developed a system that collects, evaluates, and rates tweets automatically for trend evaluation using a Twitter dataset related to French presidential elections. In [16], supervised sentiment analysis was used for (Arabizi type) Arabic languages that use Latin characters; the result shows a recall up to 86.9%. Salameh, Mohammad [28] proposed a supervised sentiment analysis using word-level and character N-Grams. The result showed that an accuracy of 87% was obtained.

In [29], a Naïve Bayes was used to perform supervised sentiment analysis for Arabic languages. A Multi-algorithm [29] was used, with Naïve Bayes as well as a bag of words. The algorithms were applied to a dataset collected from e-commerce websites, and the result revealed that the algorithm achieved 93.87% accuracy for Naïve Bayes. Alayba, Palade [30] applied the hyper algorithm with Naive Bayes, unigram, and bigram features to the algorithm on the Twitter dataset. The result shows that the accuracy is up to 90%. Similarly, [31] used Naïve Bayes with Precision, Recall, and F-Measure metrics, from which the result sections show that the Naïve Bayes lexicon accuracy reached 90%.

Moreover, [32] applied Naïve Bayes in their experiments with a dataset collected from social networks such as Twitter, Facebook. Hathlian and Hafezs [32] addressed the issue related to the lack of lexicons for analysis and testing in the Arabic sentiment analysis context. A relatively large dataset of modern standard Arabic (MSA) comments and reviews was collected. The result for the Neive Bayes performance accuracy is up to 85%. Moreover, a supervised approach was used by [33]; the study used hyper languages Iraqi, Egyptian and Lebanese dialect Arabic with 10-fold cross-validation. The result showed an accuracy of 87.9%.

Alqarafi, Adeel [34] introduced a semi-supervised approach for constructing an annotated sentiment corpus for the Saudi dialect. The idea is to use a list of lexicons developed using embedding techniques. The annotation of a large number of the corpus is generated from Twitter by manual check in order to remove the incorrect annotated tweets. The result showed that the classifier as Naïve Bayes outperformed other classifiers by up to 91% accuracy. Cambria, Poria [35] provided a review of computational intelligence for affective computing and sentiment analysis. The authors summarise various papers on emotion and sentiment by highlighting their limitations and opportunities.

Abdulkareem and Tiun [33] conducted a sentiment analysis study using k-nearest neighbours in Arabic dialect datasets gathered from Twitter. The study illustrated that by using cross-validation (10-fold), the algorithm achieved up to 87.9% accuracy. Moreover, [36,37] used the support vector classifier and k-nearest neighbours to perform sentiment analysis in Arabic and Saudi dialects’ languages. The results show that the algorithms achieved 95% Accuracy on data collected from Twitter. Khasawneh, Wahsheh [15] Conducted sentiment analysis for Arabic languages by applying a decision tree with Iterative (J48) algorithms. The classifiers were applied to a multi-domain dataset from Facebook and Twitter. The experiment results showed a high result of up to 93%. Abdulkareem and Tiun [33] used decision tree 10-fold cross-validation with other machine learning classifiers to perform a sentiment analysis experiment with hyper languages dialect Arabic collected from Twitter. The result showed an accuracy of 87.9%. A summary of selected algorithms used in Arabic sentiment analysis is shown in Table 1.

Table 1. A summary of the main algorithms for Machine Learning based Arabic SA.

Ref	Algorithms					Descriptions
	SVM	NB	K-NN	DT	DL
[38]	-	90%	-	-	-	Accuracy
[32]	89.5%	85.1%	-	-	-	Accuracy
[39]	73.5	71.6	70	65.1	-	Recall (Average)
[15]	93.3%	91.8%	31.4%	92.4%	-	Accuracy
[40]	74.7%	74.1%	-	-	-	Accuracy
[41]	61.4%	67.9%	-	-	-	Accuracy
[42]	68.2%	61.4%	-	-	-	Accuracy
[43]	73.2	-	-	-	-	Stem + Morph + language independent features
[44]	68.2	61.4	-	-	-	Accuracy, Recall
[45]	71.7	76.8	60	-	-	Accuracy
[46]	85	81.3	52.9	50	-	Accuracy
[29]	94.8	87.3	80.1	-	-	Precision
[47]	73.5	-	72.2	-	-	Accuracy
[42]	87.4	65.9	-	-	-	Accuracy
[48]	66.1	-	-	-	60.4	Accuracy
[49]	77.7	78.2	-	80.9	-	Precision
[50]	84.9	77.5	-	-	-	Accuracy
[51]	93.4	-	-	-	-	Accuracy
[52]	72.6	65.4	-	-	-	Accuracy
[53]	91.5	-	-	-	-	Accuracy
[54]	82.5	85.7	66.7	-	-	Accuracy
[30]	90.9	90.1	-	-	-	Accuracy
[31]	87.8	95.4	-	-	-	Macro-Precision
[16]	86.9	82.1	-	-	-	Recall
[36]	90	-	90.5	-	-	Accuracy
[55]	92	-	-	-	-	Accuracy
[33]	-	93.9	88.8	92.1	-	Recall
[56]	83.2	80.5	82.4	-	-	Macro-F1
[57]	95.1	-	-	-	-	Accuracy
[28]	87	-	-	-	-	Accuracy
[58]	47.4	48.9	57.8	47.6	-	Accuracy
[59]	93	-	-	-	-	Accuracy

SVM = Support Vector Machine, DL = Deep Learning, NB = Naïve Bayes, k-NN = K-nearest Neighbours, and DT = Decision Tree.

3. Materials and Methods

This section outlines a five-step method to select machine learning algorithms for Arabic sentiment analysis: (1) Data collection; (2) Pre-processing; (3) Feature selection; (4) Data analysis; (5) Evaluation (Figure 1). The novel component of this method is the multiple criteria final evaluation stage. Figure 1 used to presents an overview of the five-step methods.

Figure 1. Overview of the five-stage method to evaluate SA algorithm for Arabic.

3.1. Data Collection

A set of 11,647 Saudi dialect Arabic tweets [60,61] was used with sentiment labels from online tools. Moreover, a large number of students were invited to register in the system available online and classify the texts made up of short review sentences to positive and negative, which include 5255 positive and 6392 negative tweets. The corpus consists of only Saudi dialect Arabic reviews about various domains such as restaurants, shopping, fashion, education, entertainment, hotels, motors, and tourism. In this study, only the positive and negative reviews were considered, while neutral, sarcastic, modern standard Arabic (MSA), and unknown tweets were excluded.

3.2. Pre-Processing

The pre-processing stages include filtering, tokenisation, and Arabic stemming.

3.2.1. Tokenisation: Or 1.1 (Tokenisation)

Tokenisation involves the breakdown of a naturally occurring input string of orthographic symbols into elementary symbols named “tokens” for subsequent processing. Since there is no single correct tokenisation, a tokenisation system is a purely analytical tool [62].

Tokenisation systems segment the text of a document into a sequence of tokens. There are numerous options for specifying the splitting points. The default setup involves the use of all non-letter characters. The outcome generates tokens that consist of one single word, which is the most suitable option for sentiment analysis [61].

3.2.2. Filtering Step Words

The next pre-processing stage excludes Arabic stopwords with the help of “arabicStemR”, a package for stemming Arabic in the R programming language. Frequent words with less than three letters are removed. In general, stop words usually only convey syntactic meanings. These include the words “or” (أو, ( فثً $\stackrel{ً}{◌}$)), “in” (“ح ٍث), “were” (“على), “on” (“لمن), “whose“ (كثث), “all” ( غ ٍثث), and “beyond” (“مثثن)”. The omission of stop words makes the documents shorter in length, which leads to quicker processing and increases the effectiveness of term indexing [63].

3.2.3. Stemming

A light stemmer algorithm was used to remove the most frequently occurring prefixes and suffixes. This algorithm alters the form of the words in certain cases [64]. The initial “و)” and definite articles “لل ”,” وال“،”ال“،”كال“، ”فال“، ”بال” and suffixes “ھا ”,“ ان“،”ات“،”ون“،”ين“،”يه “،”ية“،”ه“،”ة“،”ي” are removed by the light stemmer and light10 algorithms. These algorithms allowed recurrent suffixes and prefixes to be removed from strings.

3.3. Feature Selection

The tweets were labelled as positive or negative. In addition, the result was transferred to two Arabic language experts at the University of Khartoum to validate the labels.

Manual Annotations

An Arabic language expert assessed the accuracy of a random sample of the RapidMiner polarity labels [65]. More than 5000 mixed datasets were annotated manually. Moreover, the must set the polarity of the sentence based on three classes, these being neutral, negative, or positive. While this process, the inter-agreement between them was measured using the Kappa coefficient.

$$k=\frac{P_{agree}-P_{chance}}{1-P_{chance}}$$

(1)

$P_{agree}$ = Proportion of trials in which judges agree.

$P_{chance}$ = Proportion of trials in which agreement would be expected due to chance.

The tweets and their analogous annotations are saved in a database. Each tweet needs to own at least three labels to have completed labelling. These human manual labels are recognised as the ideal golden truth, and the accuracy of the classifiers is calculated concerning these labels. Moreover, the following formula was used to estimate the necessary sample size for the check:

$$n=\frac{2{\left(Z_a+Z_{1-\beta }\right)}^{2\sigma 2,}}{{\Delta }^2}$$

(2)

where $Z$ is a constant (set by convention according to the accepted $\alpha$ error and whether it is a one-sided or two-sided effect) and $n$ is the required sample size for $Z_a$ [66].

3.4. Data Analysis

3.4.1. Machine Learning Algorithms

The five most common classifiers were selected for comparative analysis: decision tree, support vector machine, K-nearest neighbours, deep learning, and Naïve Bayes. These are popular and have been reasonably accurate in previous similar tasks [67,68,69]. In addition, more than 21 existing ML classifiers for Arabic language performance analysis and recommendation methodologies were rigorously examined. The best performing classifiers with higher frequencies were selected to represent the top five ML algorithms. In the end, we evaluate the performance of the top five ML classifiers selected.

Support Vector Machine

Support vector machines determine the maximum margin hyperplane that offers the highest separation between the classes, while the samples that are closest to this hyperplane are called the support vectors [70].

The optimal hyperplane function is expressed as

$$h\left(x\right)=wTx+b,$$

(3)

For any new point z, the class of support vector machines classifier is predicted as

$$ˆy=sign\left(h\left(z\right)\right)=sing\left(wTz+b\right)$$

(4)

where the sign “·” function returns −1 if its argument is negative, and +1 if its argument is positive [63].

Support vector machines are intrinsically two-class classifiers; a multiclass problem is formed through the construction of multiclass support vector machines, where a two-class classifier is developed over a feature vector $\mathsf{\Phi }\left(\overrightarrow{x,}{y}^{\prime }\right)$ obtained from the pair that constitutes the class of the datum and the input characteristics.

The classifier selects the class at test time using the following expression

$$y=argma{x}_{y^{\prime } }{\overrightarrow{w}}^T\mathsf{\Phi }\left(\overrightarrow{x,}{y}^{\prime }\right)$$

(5)

In the course of training, the margin refers to the gap or difference between the value for the correct class and the other closest class, and therefore the quadratic program formulation will require the following:

Naïve Bayes

For the Naïve Bayes classifier, the probabilities of classes are ascertained with the help of a feature vector table. The class that has the maximum posterior probability is then assigned the classification. In general, the text classification is carried out through the implementation of two models of the Naïve Bayes approach: Bernoulli’s multivariate and multinomial. The Naïve Bayes is basically a stochastic model applied to generate documents, which adheres to the Bayes’ rule expressed as follows [71,72]:

$$\frac{P\left(c|x\right)=P\left(c\right).P\left(x\right)}{P\left(x\right)}$$

(6)

Above:

-

P(c|x) is the posterior probability of class (c, target) given predictor (x, attributes);

-

P(c) is the prior probability of class;

-

P(x|c) is the likelihood which is the probability of predictor given class;

-

P(x) is the prior probability of predictor.

To find the probability of the tweets class given the tweet, we adopt Equation (2), as used in [73].

$$P\left(\frac{C}{t}\right)=P\left(C\right)P{{\displaystyle \prod }}_{i=1}^{n}P\left(f_i\left|C\right.\right)$$

(7)

Here, C represents the classes negative, positive, and neutral; t represents the tweet; and f represents the feature. The NLTK implementation of the Naïve Bayes algorithm was used [74].

K-Nearest Neighbours

In the k-nearest neighbours algorithm, an unknown example is compared with the k training examples that are its closest neighbours. In the process of applying the k-nearest neighbours algorithm to a new example, the first step is to identify the k closest training examples. The term “closeness” is defined as a distance in the n-dimensional space, which is embodied using the $n$ attributes in the training example set. Other metrics can also be used to represent the distance between the unknown example and the training examples—for instance, the Euclidean distance. The normalisation of data before training and application of the k-nearest neighbours algorithm is recommended because the distances usually depend on absolute values. The accurate configuration is defined by the parameters of the operator and the metric used. In the second step, the unknown example is classified by the k-nearest neighbours algorithm based on the majority vote of the identified neighbours. The predicted value, in the case of a regression, is defined as the average of the values of the found or identified neighbours.

The k nearest neighbours are assigned a weight of 1/k and all others a weight of 0 by the k-nearest neighbours classifier. This generalisation can be further applied to the weighted closest or nearest neighbours classifiers. In other words, where the $i$th nearest neighbours are assigned a weight $\left(w_{ni}\right)$ with ${{\displaystyle \sum }}_{i=0}^n{w}_{ni}=1$, an analogous result is applied to the strong and robust consistency of weighted nearest neighbours classifiers [74].

$C_n^{wnn}$ signifies the weighted nearest classifier with weights ${\left\{w_{ni}\right\}}_{i=1}^n$. The excess risk has the following asymptotic expansion subject to the regularity conditions applicable for class distributions [75].

$$R_R{C}_n^{wnn}-R_R\left(C^{Bayes}\right)=B_1{s}_n^2+B_2{t}_n^2)\left\{1+o\left(1\right)\right\},$$

(8)

For constants $B_1$ and $B_2$ where

$$s_n^2={{\displaystyle \sum }}_{i=1}^n{w}_{ni}^{2}and t_n=n^{-2/d}{{\displaystyle \sum }}_{i=1}^n{w}_{ni}\left\{i^{1+2/d}-{\left(i-1\right)}^{1+2/d}\right\}$$

(9)

The following expressions represent the optimal weighting scheme ${\left\{w_{ni}^{*}\right\}}_{i=1}^n$ that balances the two terms in the display above set $w_{ni}^{*}=\frac{1}{k^{*}}\left[1+\frac{d}{2}-\frac{d}{2k^{*}{}^{2/d}}\left\{i^{1+2/d}-{\left(i-1\right)}^{1+2/d}\right\}\right]$ for $i=1,2,\dots , k^{*}$ and $w_{ni}^{*}=0$ for $i=k^{*}+1,\dots ,n.$ The dominant term in the asymptotic expansion of the excess risk, after the consideration of the optimal weights, is $\mathcal{O}\left(n^{-\frac{4}{d+4}}\right)$.

Decision Tree

In a decision tree, a node indicates the test of an attribute value, and a branch signifies the result of the test. An ensemble of classifiers is created in a decision tree through the formation of various decision trees at the training stage that uses arbitrary feature selection and bagging methodology. The decision tree creates two kinds of nodes: the interior node linked with a feature and the leaf node labelled as a class [49].

Mathematical Formulation

With training vectors $x_i\in R^n, i=1,\dots ,I$ and a label vector, $y\in R^l$, a decision tree recurrently partitions and separates the space so that the samples with the same labels are classified together.

In the following expressions, $Q$ represents the data at node m. Every candidate split $\theta =\left(j, t_m\right)$ comprises a threshold $t_m$ and a feature, and the data are partitioned into $Q_{left}\left(\theta \right) \mathrm{and} Q_{right}\left(\theta \right)$ subsets.

$$Q_{left}\left(\theta \right)=\left.\left(x,y\right)\right|x_j<=t_m$$

(10)

$Q_{right}\left(\theta \right)=Q\backslash Q_{left}\left(\theta \right)$ the impurity at $m$ is calculated with the help of an impurity function $H\left(\mathrm{n}\right)$, the selection of which is based on the task being solved (regression or classification).

$$G\left(Q,\theta \right)=\frac{n_{left}}{N_m} H\left(Q_{left}\left(\theta \right)\right)+\frac{n_{right}}{N_m}H\left(Q_{right}\left(\theta \right)\right)$$

(11)

The parameters that minimise the impurity is selected

$${\theta }^{*}=argmi{n}_{\theta }G\left(Q,\theta \right)$$

(12)

Recourse for subsets $Q_{left}\left({\theta }^{*}\right)$ and $Q_{right}\left({\theta }^{*}\right)$ until the maximum permissible depth is attained, $N_m<mi{n}_{samples} or N_m=1$.

Deep Learning

Deep artificial neural networks are a group of algorithms reaching new levels of accuracy for several important issues, such as image recognition, sound recognition, recommender systems [76,77,78]. A deep neural network is a type of neural network. It consists of three parts: the input layer (X), the multi-layer hidden layer (Hn), and the output layer (Y). Figure 2 shows the multi-layer structure of a deep learning algorithm, which can express complex non-linear relationships. It uses neural networks to simulate human decision-making. Deep learning can be computationally expensive and needs large datasets for training. This is because there are many parameters to learn. For example, a deep learning algorithm might be trained to “learn” what a cat looks like. It would take an enormous dataset of pictures for it to understand the minor details that distinguish a cat from a wolf or a fox. Deep learning works by:

Figure 2. The Multi-layer Structure of a Deep Learning Algorithm.

• Using a cascade of multiple layers of non-linear processing units for feature extraction and transformation. Every successive layer uses the output from the previous layer as input;
• Learning multiple levels of representation that correspond to different levels of abstraction within a hierarchy of concepts [76].

3.4.2. Advantages and Disadvantages of the Prominent ML Algorithms for Arabic SA

The top five ML algorithms’ key advantages and disadvantages were discussed in Table 2, which is a cover classifier that is used for a different type of Arabic dataset collected from various domains such as book reviews, hotels, and restaurants.

Table 2. Key Advantages and Disadvantages of five Machine Learning Algorithms.

Algorithm	Advantage	Disadvantage
Support Vector Machines	Non-linear decision boundaries can be modelled by support vector machines. There are different kernels to make the selection. They are also equitably robust against overfitting, specifically in high-dimensional space [49].	Support vector machines are memory intensive. They are difficult to scale up to massive datasets and are complex to tune because the weights have given to the selection of the right kernel. In the industry at present, random forests are generally preferred over support vector machines [79].
Decision Tree	Decision trees are comparatively robust to outliers and are capable of learning non-linear relationships. When it comes to practice, ensembles perform exceptionally well, winning several classical (that is, non-deep-learning) machine learning competitions.	Individual trees that are unconstrained are inclined towards overfitting as they can continue branching until they store the training data in memory. Ensembles can improve or alleviate this [80].
Naïve Bayes	Despite the fact that the assumption of conditional independence seldom holds, Naïve Bayes models perform well. The scale and are easy to implement.	Naïve Bayes models, as a result of their absolute simplicity, are frequently outclassed by models that are suitably trained and tuned with the help of the earlier listed algorithms.
Deep Learning	When it comes to the classification of text, audio, or image data, deep learning performs extremely well [81].	Deep neural networks are similar to regression and require huge amounts of data for training, and therefore they are not regarded as a general-purpose algorithm.
Nearest Neighbours	Nearest neighbours algorithms are “instance-based,” which implies that every training observation is considered. They then predict new observations through the search for the most similar or parallel training observations and combining their values.	As these algorithms are highly memory-intensive, their performance is weak for high-dimensional data. A meaningful distance function is required by them to evaluate and calculate similarity. However, in practice, tree ensembles or training regularised regression nearly always optimises the use of time.

3.5. Evaluation Metrics

To evaluate ML algorithms, we performed 10-fold cross-validation with 90% as a test set. We assessed their performance using Accuracy, Precision, Recall, and F-measure, which were proposed for performance evaluation in multi-label learning [82], as defined below.

The Accuracy provided by Equation (13) measures the most intuitive performance measure, and it is merely a ratio of correctly predicted observation to the total observations [83].

$$\mathrm{Accuracy}=\frac{\sum true positive+\sum True negative}{\sum Total population}$$

(13)

On the other hand, Precision provided by Equation (14) measures the number of True Positives divided by the number of True Positives and False Positives [83].

$$\mathrm{Precision}=\frac{true positive}{true positive+false positive}$$

(14)

In this context, Recall measures expressed by Equation (15) are also referred to as the true positive sensitivity or rate, and precision is also known as the positive predictive value (PPV). Accuracy and true negative rate are the other relevant measures that are used in classification. Specificity is another term for the true negative rate [83].

$$Recall=\frac{true positive}{true positive+false negative}$$

(15)

F-measure, as expressed in Equation (16), is a measure that represents the combined precision, while recall indicates the harmonic mean of precision and recall, and the balanced F-score or the traditional F-measure [83].

$$F-measure=\frac{2.Precision. Recall}{\left(Precision+Recall\right)}$$

(16)

In addition to this, another two evaluation measures were included, which is significant for the assessment of the performance. The first measure is the area under the curve (AUC) [84].

$$F_{abs}=\left(\frac{AU{C}_{non-IV}}{AU{C}_{IV}}\right)\times \left(\frac{Dos{e}_{IV}}{Dos{e}_{non-IV}}\right)$$

(17)

Given that an algorithm’s runtime may vary based on different inputs of the same size, the send one is a runtime (times complexity). The worst-case time complexity refers to the maximum amount of time needed for inputs of a certain volume. On the other hand, the average-case complexity, which is relatively less common and generally stated explicitly, denotes the average of the time taken for inputs of a given size. The time complexity is usually represented as a function of the size of the input in both cases [85].

4. Results and Discussion

The results are discussed separately for each evaluation criterion. Moreover, to ensure the performance of the classifiers, we combined various domains to test the accuracy of five machine learning algorithms using Arabic dialect features.

4.1. Accuracy

Figure 3 illustrates the performance of Naïve Bayes, decision tree, SVM, K-NN, and deep learning classifiers. It indicates that deep learning shows better accuracy (85.25%) compared to a decision tree, Naïve Bayes, k-NN, and SVMs. However, SVM also has shown slightly better accuracy, which is 82.30% better than decision tree, k-NN, and Naïve Bayes, 79.31%, 61.41%, and 57.56%, respectively.

Figure 3. Accuracy of Top Five Machine Learning Algorithms.

4.2. Precision

Figure 4 shows the various result of precision for deep learning, decision tree, Naïve Bayes, K-NN, and SVM classifiers. It appears that the decision tree is core (96.67%) compared to deep learning, Naïve Bayes, k-NN, and SVMs. However, deep learning and SVM also showed a slightly better result, which is 85.30%, 83.87%, respectively, better than Naïve Bayes, and K-NN.

Figure 4. Precision Result for Five Machine Learning Algorithms.

4.3. Recall

Figure 5 illustrates the various result of deep learning, decision tree, Naïve Bayes, K-NN, and SVM classifiers. It indicates that deep learning shows better accuracy (85.25%) compared to a decision tree, Naïve Bayes, k-NN, and SVMs. However, SVM also has shown slightly better accuracy, which is 82.30% better than decision tree, k-NN, and Naïve Bayes, 79.31%, 61.41%, and 57.56%, respectively.

Figure 5. Recall Result for Five Machine Learning Algorithms.

4.4. F-Measure

Figure 6 illustrates various accuracies of deep learning, decision tree, Naïve bays, K-NN, and SVM classifiers. It appears that deep learning shows the better result of F-measure is core (86.81%) compared to a decision tree, Naïve Bayes, K-NN, and SVMs. However, SVM also showed a slightly better result, which is 83.87% better than decision tree (77.36%), K-NN (73.32%), and Naïve Bayes (46.681%).

Figure 6. F-Measure Result for Five Machine Learning Algorithms.

4.5. Correct and Wrong Detection

Table 3 illustrates the correct detection of deep learning, decision tree, Naïve Bays, K-NN, and SVM classifiers. It indicates that deep learning shows the highest accurate detections (9929) and lowest wrong detections (1718) compared to a decision tree, Naïve Bayes, K-NN, and SVMs. The SVM also has shown slightly higher accurate detection (9585) and low wrong detections (2062). However, the result is better than the decision tree’s correct detections (9237) and low incorrect detections (2410), K-NN’s accurate detections (7153) and low wrong detections (4494), and Naïve Bayes’ accurate detections (6704) and low wrong detections (4943).

Table 3. Correct and Wrong Detection.

	DL	DT	NB	K-NN	SVM
Correct Detection	9929	9237	6704	7153	9585
Wrong Detection	1718	2410	4943	4494	2062
Total	11647	11647	11647	11647	11647

4.6. Classification Error

Figure 7 illustrates various Classification errors of deep learning, decision tree, Naïve Bayes, K-NN, and SVM algorithms. It indicates that deep learning shows the lowest classification error, which is 14.75%, compared to a decision tree, Naïve Bayes, K-NN, and SVM. However, SVM also has shown a slightly lower classification error, which is 17.70%. However, the result is better than the decision tree, with a classification error of 20.69%; the K-NN classification error of 38.59%; and Naïve Bayes, with a higher classification error of 42.44%.

Figure 7. Classification Error Result for Five Machine Learning Algorithms.

4.7. CPU Time

Figure 8 illustrates various time complexity of deep learning, decision tree, Naïve bays, K-NN, and SVM classifiers. It appears that SVM is faster with core 897 ms compared to K-NN, Naïve Bayes, decision tree, and deep learning. However, k-nearest neighbours showed a slightly better result, which is (1690 ms) better than decision tree (3870 ms), deep learning (8464 ms), and Naïve Bayes (1800 ms).

Figure 8. Runtime Result for Five Machine Learning Algorithms.

4.8. Area under the Curve (AUC)

Figure 9 presents a graph of the various area under the curve results for the five classifiers, namely, deep learning, decision tree, Naïve Bayes, K-NN, and SVM classifiers. It appears that deep learning shows a better result of AUC as core 0.928 compared to a decision tree, Naïve Bayes, K-NN, and SVMs. However, SVM also showed a slightly better result, which is 0.900 better than the decision tree (0.878), Naïve Bayes (0.610), and K-NN (0.500).

Figure 9. Area Under the Curve (AUC) Result for Five Machine Learning Algorithms.

4.9. Overall Result

In this section, we discuss the overall performance result for the top five machine learning classifiers, such as deep learning, decision tree, Naïve Bayes, K-nearest neighbours, and SVM. Generally, Table 4 and Figure 10 consists of the dialect Arabic dataset, in which the performance of the classifiers involved are dependent on the multi-criteria evaluating strategy as well as the dataset. For example, decision tree precision reached 96.67%, and K-Nearest neighbours recall reached 96.67%. However, the other metrics are low for decision trees and k-nearest neighbours. Moreover, the classification error for k-nearest neighbours reached 38.59%, and the decision tree reached 20.69% for classification error. Moreover, Naïve Bayes obtained the lowest result for the recall, which is 33.59%, and the higher result is 75.49% for precision, and also the classification error and the runtime is slightly high, which is near to 2000 MS compared to other classifier metrics. The accuracy for the decision tree, K-nearest neighbours, and Naïve Bayes are 79.31%, 61.41%, and 57.56%, respectively.

Table 4. Summary of the Overall Performance of Machine Learning Classifiers.

Model	Acc	P	R	Fm	CE	AUC	RT
Deep Learning	85.25%	85.30%	88.41%	86.81%	14.75%	0.928	8464
Decision Tree	79.31%	96.67%	64.52%	77.36%	20.69%	0.878	3870
Naïve Bayes	57.56%	75.49%	33.59%	46.48%	42.44%	0.610	1800
K-Nearest Neighbours	61.41%	59.14%	96.14%	73.23%	38.59%	0.500	1690
Support Victor Machine	82.30%	83.87%	83.89%	83.87%	17.70%	0.900	897

Acc = Accuracy, P = Precision, R = Recall, Fm = F-measure, CE = Classification Error, AUC = Area Under the Curve, RT = Run Time.

Figure 10. Overall Result for Five Machine Learning Algorithms.

On the other hand, deep learning reached a higher result for recall (88.41%) compared with other metrics: F-measure (86.81%), precision (85.30%), and accuracy (85.25%). Classification error shows better results compared to another classifier, which is 14.75%. However, the runtime is shown as 3464 MS, which needs improvement. Meanwhile, the support vector machine classifier is demonstrating a better runtime with 897 MS. Furthermore, other metrics are found within the range of 90 in the area under the curve and show a slightly higher classification error of 17.70% compared to a decision tree, K-nearest neighbours, and Naïve Bayes classifier.

5. Conclusion and Future Work

In this study, dialect Arabic data from Twitter was used. Moreover, the original dataset was used with AYLIEN tools to label as positive and negative. The dataset was pre-processed by performing data selection, data cleaning, and construction before being used for modelling. After testing the top five classifiers deep learning, decision tree, K-nearest neighbours, Naïve Bayes, and support vector machine, the result of the experiment shows that deep learning and support vector machine classifier showed better performance results in terms of accuracy, precision, recall, F-measure, and AUC compared to a decision tree, K-nearest neighbours, and Naïve Bayes classifiers. However, this does not imply that deep learning and support vector machine classifiers would always be the best algorithms, as some algorithms appeared to be promising using different datasets; this is because the dataset differs considering the location and the population of the data that was collected. In this study, we further compared the performance of five machine learning algorithms classifier: deep learning, decision tree, Naïve Bayes, K-nearest neighbours, and support vector machine with multi-criteria using dialect Arabic dataset of tweets, to select a better classifier for dialect Arabic. This study can be used to aid novice and prominent researchers in comprehending the theories and concepts of ML algorithms, performance measures, and the sentiment analysis of dialect Arabic texts for a sustainable solution to its morphologically oriented challenges. As for future works, the study highlighted three feasible future research directions that can lead to the production of useful studies related to the Arabic dialect sentiment domain. They include: (i) Using different dialect Arabic training and testing datasets; (ii) using more than the top five machine learning classification algorithms on the same data sources; and (iii) mixing the Arabic dialect dataset with modern standard Arabic dataset to generalise the outcome.