Chicken Swarm-Based Feature Subset Selection with Optimal Machine Learning Enabled Data Mining Approach

Abstract

Data mining (DM) involves the process of identifying patterns, correlation, and anomalies existing in massive datasets. The applicability of DM includes several areas such as education, healthcare, business, and finance. Educational Data Mining (EDM) is an interdisciplinary domain which focuses on the applicability of DM, machine learning (ML), and statistical approaches for pattern recognition in massive quantities of educational data. This type of data suffers from the curse of dimensionality problems. Thus, feature selection (FS) approaches become essential. This study designs a Feature Subset Selection with an optimal machine learning model for Educational Data Mining (FSSML-EDM). The proposed method involves three major processes. At the initial stage, the presented FSSML-EDM model uses the Chicken Swarm Optimization-based Feature Selection (CSO-FS) technique for electing feature subsets. Next, an extreme learning machine (ELM) classifier is employed for the classification of educational data. Finally, the Artificial Hummingbird (AHB) algorithm is utilized for adjusting the parameters involved in the ELM model. The performance study revealed that FSSML-EDM model achieves better results compared with other models under several dimensions.

1. Introduction

Data mining (DM) is the procedure of understanding data through cleaning raw data, discovering patterns, producing models, and testing the models. It comprises of several fields such as statistics, machine learning (ML), and database systems. Education DM (EDM) is an emergent field with an arising strategy to investigate the various types of data which are obtained from an education background [1]. It is an interdisciplinary area which inspects data mining (DM), man-fabricated consciousness, and measurable demonstrating with the data produced using an academic organization [2]. EDM uses a calculation method for taking care of elucidating academic data considering a definitive point of examining academic enquiries. To make a nation stand out among different nations across the globe, education frameworks should encounter an essential advancement by re-planning their design. The concealed data and examples from various data sources can be extricated by adjusting the strategies for DM. For summing up the outcomes of students with their qualifications, they investigate the abuse of DM in the academic fields. Crude data can be altogether moved through DM models. The data achieved from an education organization go through examination of various DM strategies [3]. The strategy identifies the conditions wherein students can strive to have a positive impact [4].

Student performance prediction (SPP) has different definitions according to troublesome perspectives; however, the measured assessment assumes a significant part in current education establishments. SPP seems effective in aiding all partners in the education interaction. For students, SPP can assist them with picking reasonable courses or activities and make their arrangements for academic durations [5]. For educators, SPP can assist with changing learning material and presenting programs compatible with the students’ capacity, and help identify struggling students. For education chiefs, SPP can assist with checking the education program and enhancing the course framework. Generally, partners in the education advancement have intentions to further develop the education outcome. Moreover, the data-driven SPP study provides a goal of reference for the education framework. Weka, a compelling DM technique was utilized to produce the outcome [6].

The increment of educational information from distinct sources has resulted in desperation for the EDM research [7]. This can help to further objectives and characterize specific goals of education and highlight subset determinations by disposing of the component that is repetitive/is not important. The set of components chosen should follow the Occam Razor rule to provide the best outcome in light of the goal [8]. The data size to be dealt with has expanded in the past five years; hence, the choice is turning into a necessity before any sort of arrangement happens. It is not quite the same as the element extraction strategy because the determination method does not change the first portrayal of the data [9]. The least complex method includes choice, where how much quality in an examination is diminished by choosing just the main view of the circumstances such as a more elevated level of exercises [10].

Since the high dimensionality raises computational costs, it is essential to define a way to reduce the number of considered features. Feature selection (FS) allows reducing a high dimensionality problem and selecting a suitable number of features. This study designed a Feature Subset Selection with optimal machine learning for the Educational Data Mining (FSSML-EDM) model. The proposed FSSML-EDM model involves the Chicken Swarm Optimization-based Feature Selection (CSO-FS) technique for electing feature subsets. Next, the extreme learning machine (ELM) classifier was employed for the classification of educational data. Finally, the Artificial Hummingbird (AHB) algorithm was utilized for adjusting the parameters involved in the ELM model. The performance study revealed the effectual outcomes of the FSSML-EDM model over the compared models under several dimensions. Our contributions are summarized as follows:

• We propose a model comprising data preprocessing, CSO-FS, ELM classification, and AHB parameter;
• We designed a new CSO-FS technique to reduce the curse of the dimensionality problem and enhanced the classification performance;
• We employed the ELM classification model with the AHB-based parameter optimization technique for the EDM process;
• We validated the performance of the FSSML-EDM model using the benchmark dataset from the UCI repository.

2. Literature Review

Injadat et al. [11] explored and analyzed two distinct datasets at two distinct phases of course delivery (20% and 50%) utilizing several graphical, statistical, and quantitative approaches. The feature analysis offers understanding as to the nature of distinct features regarded and utilizes in the selection of ML techniques and their parameters. Moreover, this work presents a systematic model dependent upon the Gini index and p-value for selecting an appropriate ensemble learner in a group of six potential ML techniques. Ashraf et al. [12] progressed an accurate prediction pedagogical method, considering the pronounced nature and novelty of presented approach in Educational Data Mining. The base classifications containing RT, j48, kNN, and naïve Bayes (NB) were estimated on a 10-fold cross-validation model. In addition, the filter procedure as over-sampling (SMOTE) and under-sampling (Spread subsampling) were exploited to examine some important alterations in outcomes amongst meta and base classifications.

Dabhade et al. [13] forecasted student academic performance in a technical institution in India. The data pre-processed and factor analysis were executed on the attained dataset for removing the anomaly from the data, decreasing the dimensionality of the data, and attaining the most correlated feature. Nahar et al. [14] generated two datasets concentrating on two distinct angles. In the primary dataset classification and forecast, the type of students (bad, medium, and good) on a particular course was dependent upon its prerequisite course efficiency. This can be executed during the artificial intelligence (AI) course. The secondary dataset also classified and forecasted the last grade (A, B, C) of an arbitrary subject; our data can be established in such a way that the data are only concentrated on the efficiency of the midterm exam.

Despite all the studies performed on FS process, to the best of our knowledge, only few works have carried out an FS-based classification model for EDM. Earlier works have used ML models for EDM without contributing much significance to the FS process. At the same time, the parameters involved in the ML models (i.e., ELM) considerably affect the overall classification performance. Since the trial-and-error method for parameter tuning is a tedious and erroneous process, metaheuristic algorithms can be applied. Therefore, metaheuristic optimization algorithms can be designed to optimally tune the parameters related to the ML models to improve the overall classification performance.

3. The Proposed Model

In this study, a new FSSML-EDM technique was developed for mining educational data. The proposed FSSML-EDM model involves data preprocessing at the initial stage to transform the input data into a compatible format. Then, the preprocessed data are passed into the CSO-FS technique for electing feature subsets. Next, the ELM classifier can be employed for the effective identification and classification of educational data. Finally, the AHB algorithm is utilized for effectively adjusting the parameters involved in the ELM model. The outcome of the ELM model is the classification output. Figure 1 depicts the block diagram of FSSML-EDM technique.

3.1. Process Involved in CSO-FS Technique

At the initial stage, the presented FSSML-EDM model incorporates the design of the CSO-FS technique for electing feature subsets. The CSO algorithm is chosen over other optimization algorithms due to its simplicity and high parallelism. The CSO simulates the chicken movement and the performance of the chicken swarm; the CSO is explained as follows: CSO has several groups, and all the groups have a dominant rooster, some hens, and chicks [15]. The rooster, hen, and chick from the group are found dependent upon their fitness value. The rooster (group head) is the chicken which is an optimum fitness value. However, the chick is the chicken which has the worse fitness value. The majority of chickens are hens and it can be selected arbitrarily to stay in that group. The dominance connection and mother–child connection from the group remain unaltered and upgrade during (G) time steps. The movement of chickens are expressed under the equation which utilizes to the rooster place upgrade provided by Equation (1):

$$X_{ij}^{r+1}=X_{i,j}^t\times \left(1+randn\left(0, {\sigma }^2\right)\right)$$

(1)

whereas:

$${\sigma }^2=\{\begin{array}{ll}1\hfill & \mathrm{if} f_i\le f_k\hfill \\ \exp \left(\frac{f_k-f_i}{\left|f_i+\epsilon \right|}\right)\hfill & \mathrm{Otherwise}\hfill \end{array}$$

In which $k\in \left[1, N_r\right],$ $k\ne i,$ and N_r refers the amount of chosen roosters. $X_{i,j}$ signifies the place of rooster number $i$ in $j\mathrm{th}$ dimensional under $t$ and $t+1$ iteration, $randn \left(O, {\sigma }^2\right)$ is utilized for generating Gaussian arbitrary numbers with mean $0$ and variance ${\sigma }^2;$ $\epsilon$ refers to the constant with minimum value; and $f_i$ is the fitness value to the equivalent rooster $i.$ The equation which utilizes the hen place upgrade is provided by Equations (2)–(4):

$$X_{i,j}^{t+1}=X_{i,j}^t+S_{1}randn\left(X_{r1,j}^t-X_{i,j}^t\right)+S_{2}randn\left(X_{r2,j}^t-X_{i,j}^t\right)$$

(2)

In which:

$$S_1=\exp \left(\frac{f_i-f_{r1}}{\left|f_i\right|+\epsilon }\right)$$

(3)

and:

$$S_1= \exp \left(f_{r2}-f_i\right)$$

(4)

where $r1,$ $r\in \left[1, \dots , N\right], r1\ne rr$ refers to the index of the rooster, but $r2$ implies the chicken in the swarm which is a rooster or hen and a uniform arbitrary number is created by $randn$. Finally, the equation that utilizes the chick place upgrade is provided by Equation (5):

$$X_{i,j}^{t+1}=X_{i,j}^t+FL\left(X_{m,j}^t-X_{i,j}^t\right), FL\in \left[0,2\right]$$

(5)

where $X_{m,j}^t$ signifies the place of $i\mathrm{th}$ chick mother.

3.2. ELM Based Classification

At this stage, the ELM classifier can be employed for the effective identification and classification of educational data. The ELM model has $n$ input layers, $l$ hidden layers, and $m$ output layers. Initially, considering the training instance $\left\{X, Y\right\}=\left\{x_i, y_i\right\}\left(i=1,2, \dots , Q\right)$, and it is comprised of the input feature $X=\left[x_{i1}{x}_{i2}\dots x_{iQ}\right]$ and matrix $Y=\left[y_{j1}{y}_{j2}\dots y_{jQ}\right]$ with training instance, where the matrix $X$ and $Y$ are expressed by [16]:

$$X=\left[\begin{array}{cccc}{x}_{11}\hfill & x_{12}& \cdots & x_{1Q}\\ x_{21}\hfill & x_{22}& \cdots & x_{2Q}\\ ⋮\hfill & ⋮& \ddots & ⋮\\ x_{n1}\hfill & x_{n2}& \cdots & x_{nQ}\end{array}\right],$$

(6)

$$Y=\left[\begin{array}{cccc}{y}_{11}\hfill & y_{12}& \cdots & y_{mQ}\\ y_{21}\hfill & y_{22}& \cdots & y_{mQ}\\ ⋮\hfill & ⋮& \ddots & ⋮\\ y_{m1}\hfill & y_{m2}& \cdots & y_{mQ}\end{array}\right],$$

where $n$ and $m$ parameters denote the dimension of input and output matrix. Next, ELM set the weights amongst input and hidden layers randomly:

$$w=\left[\begin{array}{cccc}{w}_{11}\hfill & w_{12}& \cdots & w_{1n}\\ w_{21}\hfill & w_{22}& \cdots & w_{2n}\\ ⋮\hfill & ⋮& \ddots & ⋮\\ w_{l1}\hfill & w_{l2}& \cdots & w_{ln}\end{array}\right],$$

(7)

where $w_{ij}$ denotes the weight from $i$th hidden and $j$th input layers. Figure 2 showcases the framework of ELM. Then, ELM considers the weight from output and hidden layers that can be shown below:

$$\beta =\left[\begin{array}{lll}{\beta }_{11}\hfill & \begin{array}{cc}{\beta }_{12}& \cdots \end{array}\hfill & {\beta }_{1m}\hfill \\ {\beta }_{21}\hfill & \begin{array}{cc}{\beta }_{22}& \cdots \end{array}\hfill & {\beta }_{2m}\hfill \\ ⋮\hfill & \begin{array}{cc}⋮& \ddots \end{array}\hfill & ⋮\hfill \\ {\beta }_{l1}\hfill & \begin{array}{cc}{\beta }_{l2}& \cdots \end{array}\hfill & {\beta }_{lm}\hfill \end{array}\right],$$

(8)

where ${\beta }_{jk}$ indicates the weight from $jth$ hidden and ith output layers. Next, ELM set a bias of hidden layers randomly:

$$B={[b_1{b}_2 \cdots b_n]}^T.$$

(9)

After that, ELM chooses the network activation function. According to, output matrix $T$ is characterized by:

$$T={[t_1, t_2, \dots , t_Q]}_{m\times Q}.$$

(10)

The column vector of output matrix $T$ is shown in the following:

$$t_j=\left[\begin{array}{l}{t}_{1j}\hfill \\ t_{2j}\hfill \\ ⋮\hfill \\ t_{mj}\hfill \end{array}\right]=\left[\begin{array}{l}{\displaystyle {\displaystyle \sum }_{i=1}^l}{\beta }_{i1}g\left(w_i{x}_j+b_i\right)\hfill \\ {\displaystyle {\displaystyle \sum }_{i=1}^l}{\beta }_{i2}g\left(w_i{x}_j+b_i\right)\hfill \\ ⋮\hfill \\ {\displaystyle {\displaystyle \sum }_{i=1}^l}{\beta }_{im}g\left(w_i{x}_j+b_i\right)\hfill \end{array}\right]\left(j=1,2,3, \dots , Q\right).$$

(11)

Moreover, considering Equations (10) and (11), it is modelled in the following:

$$H\beta =T^{\prime },$$

(12)

where $T^{\prime }$ indicates transpose of $T$ and $H$ represents simulation outcomes of hidden neurons. To achieve better solution with less error, the least square model is employed for determining the weight matrix measure of $\beta$ [17,18].

$$\beta =H^{+}{T}^{\prime }.$$

(13)

To enhance the normalization ability of the system and provide stable outcomes, the regularization parameter of $\beta$ is used. The amount of hidden layers is minimal in contrast with the amount of training samples, $\beta$ is shown in the following [19]:

$$\beta ={\left(\frac{I}{\lambda }+H^{T}H\right)}^{-1}{H}^T{T}^{\prime }.$$

(14)

When the amount of hidden layers is maximal than the amount of training samples, $\beta$ is denoted as follows [20]:

$$\beta =H^T{\left(\frac{I}{\lambda }+H{H}^T\right)}^{-1}{T}^{\prime }.$$

(15)

3.3. AHB Based Parameter Optimization

Finally, the AHB algorithm is utilized for effectively adjusting the parameters involved in the ELM model with the goal of attaining maximum classification performance [21]. The AHB algorithm is an optimization approach stimulated from the foraging and flight of hummingbirds. The three major models are provided as follows: in a guided foraging model, three flight behaviors are utilized in foraging (axial, diagonal, and omnidirectional flight). It can be defined as follows:

$$v_i\left(t+1\right)=x_{i,ta}\left(t\right)+h·b·\left(x_i\left(t\right)-X_{i,ta}\left(t\right)\right)h\sim N\left(0,1\right)$$

(16)

where $x_{i,ta}\left(t\right)$ characterizes the location of the targeted food source, $h$ signifies the guiding factor, and $x\left(t\right)$ represents the location of $i\mathrm{th}$ food source at time t. The location updating of the ith food source is provided by:

$$x_{Ai}\left(t\right)=\{\begin{array}{ll}{x}_i\left(t\right)\hfill & f\left(x_i\left(t\right)\right)\le f\left(v_i\left(t+1\right)\right)\hfill \\ v_i\left(t+1\right)\hfill & f(x_{i\left(t\right))}>f\left(v_i\left(t+1\right)\right)\hfill \end{array}$$

(17)

where $f\left(x_i\left(t\right)\right)$ and $f\left(v_i\left(t+1\right)\right)$ denote the value of function fitness for $x\left(t\right)$ and $v_i\left(t+1\right)$. the local search of hummingbirds in the territorial foraging strategy is provided in the following:

$$v_i\left(t+1\right)=x_{i\left(t\right)}+g·b·\left({\chi }_{i\left(t\right)}\right)g\sim N\left(0,1\right)$$

(18)

where $g$ represents the territorial factor. The arithmetical formula for the migration foraging of hummingbirds is provided by:

$$x_{wor}\left(t+1\right)=lb+r·\left(ub-lb\right)$$

(19)

where $x_{wor}$ indicates the source of food with worst population rate of nectar refilling, $r$ represents a random factor, and $ub$ and $lb$ denote the upper and lower limits, respectively.

4. Experimental Validation

The proposed FSSML-EDM model was simulated using a benchmark dataset from UCI repository, which comprises of 649 samples with 32 features and 2 class labels as illustrated in

Table 1. Dataset details.

Details	Values
Number of samples	649
Feature count	32
Class count	2
Source	https://archive.ics.uci.edu/ml/datasets/student+performance (accessed on 30 January 2022)

. The parameter settings are provided as follows: learning rate, 0.01; dropout, 0.5; batch size, 5; and number of epochs, 50. For experimental validation, the dataset is split into 70% training (TR) data and 30% testing (TS) data.

Figure 3 highlights a set of confusion matrices produced by the FSSML-EDM model on the test data. The figure indicates that the FSSML-EDM model resulted in effectual outcomes. On entire dataset, the FSSML-EDM model identified 545 samples into pass and 86 samples into fail. In addition, on 70% of the training dataset, the FSSML-EDM model identified 390 samples into pass and 53 samples into fail. Moreover, on 30% of testing dataset, the FSSML-EDM model identified 155 samples into pass and 33 samples into fail.

Table 2. Result analysis of FSSML-EDM technique with distinct measures and datasets.

Class Labels	Accuracy	Precision	Recall	F-Score	MCC	Kappa Score
Entire Dataset
Pass	97.23	97.50	99.27	98.38	89.08	-
Fail	97.23	95.56	86.00	90.53	89.08	-
Average	97.23	96.53	92.64	94.45	89.08	88.91
Training Set (70%)
Pass	97.58	97.50	99.74	98.61	89.57	-
Fail	97.58	98.15	84.13	90.60	89.57	-
Average	97.58	97.82	91.94	94.60	89.57	89.22
Testing Set (30%)
Pass	96.41	97.48	98.1	97.79	88.22	-
Fail	96.41	91.67	89.19	90.41	88.22	-
Average	96.41	94.58	93.65	94.1	88.22	88.2

offers a comprehensive EDM outcome of the FSSML-EDM model on test dataset. The experimental values indicated that the FSSML-EDM model accomplished maximum outcomes on all datasets. Figure 4 provides brief classification results of the FSSML-EDM model on entire dataset. It can be inferred from the figure that the FSSML-EDM model classified pass instances for $acc{u}_y, pre{c}_n$, $rec{a}_l$, $F_{score}, MCC$, and kappa of 97.23%, 97.50%, 99.27%, 98.38%, and 89.08% respectively. Moreover, the figure shows that the FSSML-EDM model classified fail instances for $acc{u}_y, pre{c}_n$, $rec{a}_l$, $F_{score}, MCC$, and kappa of 97.23%, 95.56%, 86%, 90.53%, and 89.08%, respectively.

Figure 5 provides detailed classification results of the FSSML-EDM model on 70% of the training dataset. The figure reveals that the FSSML-EDM model classified pass instances for $acc{u}_y, pre{c}_n$, $rec{a}_l$, $F_{score}, MCC$, and kappa of 97.58%, 97.50%, 99.74%, 98.61%, and 89.57%, respectively. In addition, the figure shows that the FSSML-EDM model classified fail instances for $acc{u}_y, pre{c}_n$, $rec{a}_l$, $F_{score}, MCC$, and kappa of 97.58%, 98.15%, 84.13%, 90.60%, and 89.57% respectively.

Figure 6 offers brief classification results of the FSSML-EDM approach on 30% of testing dataset. The figure exposes that the FSSML-EDM algorithm classified pass instances for $acc{u}_y, pre{c}_n$, $rec{a}_l$, $F_{score}, MCC$, and kappa of 96.41%, 97.48%, 98.1%, 97.798%, and 88.22%, respectively. Moreover, the figure shows that the FSSML-EDM approach classified fail instances for $acc{u}_y, pre{c}_n$, $rec{a}_l$, $F_{score}, MCC$, and kappa of 96.41%, 91.67%, 89.19%, 90.41%, and 88.22%, respectively.

Figure 7 illustrates the training and validation accuracy inspection of the FSSML-EDM technique on the applied dataset. The figure of the FSSML-EDM approach offers maximum training/validation accuracy on the classification process.

Next, Figure 8 reveals the training and validation loss inspection of the FSSML-EDM approach on the applied dataset. The figure shows that the FSSML-EDM algorithm offers reduced training/accuracy loss on the classification process of the test data.

A brief precision-recall examination of the FSSML-EDM model on the test dataset is portrayed in Figure 9. By observing the figure, it is noticed that the DLBTDC-MRI model accomplished maximum precision-recall performance under all classes.

A detailed ROC investigation of the FSSML-EDM approach on the distinct datasets is portrayed in Figure 10. The results indicate that the FSSML-EDM technique exhibited its ability in categorizing two different classes such as pass and fail on the test datasets.

Table 3. Comparative analysis of FSSML-EDM approach with existing methods [21].

Methods	Precision	Recall	Accuracy	F-Score	MCC	Kappa
FSSML-EDM	94.58	93.65	96.41	94.10	88.22	88.20
IEAFSSNFC	93.81	92.39	90.33	93.01	73.78	73.37
NFC Algorithm	81.76	91.41	81.66	86.99	56.21	50.10
NN Algorithm	67.58	85.76	64.30	76.03	57.52	67.82
SVM Algorithm	68.83	86.77	66.36	76.63	66.27	66.17
DT Algorithm	78.22	93.40	77.81	83.02	52.87	59.80
RF Algorithm	74.95	91.90	75.40	83.39	62.68	58.34

reveals an extensive comparative study of the FSSML-EDM model with existing models such as improved evolutionary algorithm-based feature subsets election with neuro-fuzzy classification (IEAFSSNFC) [22], neuro-fuzzy classification (NFC) [21], neural network (NN), support vector machines (SVM), decision tree (DT), and random forest (RF). Figure 11 inspects the comparative $pre{c}_n$, $rec{a}_l$, and $acc{u}_y$ investigation of the FSSML-EDM model with recent methods. The figure reveals that the NN and SVM models showed poor performance with lower values of $pre{c}_n$, $rec{a}_l$, and $acc{u}_y$. The NFC, DT, and RF models have showed slightly improved values of $pre{c}_n$, $rec{a}_l$, and $acc{u}_y$. Moreover, the IEAFSSNFC model resulted in reasonable $pre{c}_n$, $rec{a}_l$, and $acc{u}_y$ of 93.81%, 92.39%, and 90.33%, respectively. Furthermore, the FSSML-EDM model accomplished effectual outcomes with maximum $pre{c}_n$, $rec{a}_l$, and $acc{u}_y$ of 94.58%, 93.65%, and 96.41%, respectively.

Figure 12 inspects the comparative $F_{score}$, $MCC$, and $kappa$ analysis of the FSSML-EDM method with existing algorithms. The figure reveals that the NN and SVM methods showed poor performance with lower values of $F_{score}$, $MCC$, and $kappa$. Similarly, the NFC, DT, and RF approaches showed slightly improved values of $F_{score}$, $MCC$, and $kappa$.

Moreover, the IEAFSSNFC model resulted a in reasonable $F_{score}$, $MCC$, and $kappa$ of 93.01%, 73.78%, and 73.37%, respectively. Additionally, the FSSML-EDM methodology accomplished effectual outcomes with maximum $F_{score}$, $MCC$, and $kappa$ of 94.10%, 82.22%, and 88.20%, respectively. Therefore, the FSSML-EDM model has the capability of assessing student performance in real time.

5. Conclusions

In this study, a new FSSML-EDM technique was developed for mining educational data. The proposed FSSML-EDM model involves three major processes. At the initial stage, the presented FSSML-EDM model incorporates the design of CSO-FS technique for electing feature subsets. Next, ELM classifier can be employed for the effective identification and classification of educational data. Finally, the AHB algorithm is utilized for effectively adjusting the parameters involved in the ELM model. The performance study revealed the effectual outcomes of the FSSML-EDM model over the compared models under several dimensions. Therefore, the FSSML-EDM model can be used as an effectual tool for EDM. In the future, feature reduction and outlier removal models can be employed to improve performance. In addition, the proposed model is presently tested on small-scale dataset, which needs to be explored. As a part of the future scope, the performance of the proposed model will be evaluated on a large-scale real-time dataset.