Language Inference Using Elman Networks with Evolutionary Training

Abstract

In this paper, a novel Elman-type recurrent neural network (RNN) is presented for the binary classification of arbitrary symbol sequences, and a novel training method, including both evolutionary and local search methods, is evaluated using sequence databases from a wide range of scientific areas. An efficient, publicly available, software tool is implemented in C++, accelerating significantly (more than 40 times) the RNN weights estimation process using both simd and multi-thread technology. The experimental results, in all databases, with the hybrid training method show improvements in a range of 2% to 25% compared with the standard genetic algorithm.

1. Introduction

Recurrent neural networks (RNN) are well-established mathematical models, and they have been used widely in many scientific and industrial areas such as text processing [1,2], speech recognition [3], sentiment classification [4], handwriting recognition [5], medical signal processing [6], etc. Elman [7] is a multi-layer neural network with recurrent units in the first layer. In its simplest form, the output of the Elman network is derived from the recurrent neuron outputs of the first layer. In the neuron internal states, the information from previous time non-linear transformations is stored and processed, also known as the memory of the cell. These models have been implemented in a wide variety of complex multidimensional classification problems [8,9]. Due to their complex structure, error propagation [10] and hill-climbing techniques [11,12] inherit many deficiencies. A well-known alternative, which avoids trapping the model in local minima, is the use of evolutionary methods such as the standard genetic algorithm (SGA), or some variants of the SGA [13], to estimate the internal weights or/and the structure of the Elman network [14,15], also referred to in the literature as the GA-Elman algorithm. Furthermore, the decoding of the model’s output differs significantly from application to application. In natural language classification problems, it is common to use the final output of the network for determining the class of a data instance [16], while in more complex models, the output of the recurrent nodes [17] is part of a greater network. The use of evolutionary algorithms with Elman networks has been successful in several applications [18,19].

In this article, a novel approach to decoding the outputs of the Elman network for binary classification and a hybrid GA–Elman algorithm, which fully automatically estimates the synaptic weights, are presented. The input data of the network are treated as positive or negative symbolic sequences, while the output of the Elman network is decoded to a scalar value representing the probability of a data instance being recognized as a positive symbolic sequence by the classifier. The hybrid GA-Elman training algorithm estimates the weights of the model using both genetic operators and invoking gradient -ased local search [20,21] every few iterations. Even though this approach limits this classifier to binary classification problems, this method is still applicable to a wide range of problems, with several advantages discussed in Section 5, and extended easily to multi-class problems. The datasets used, either artificially created or obtained from popular databases, are transformed to fit the encoding scheme of our method. All the experiments were conducted tenfold by randomly splitting the instances in test and evaluation data.

The rest of this article has the following structure: in Section 2, the proposed neural classifier and the corresponding training method are described in detail. The implementation details and the specific optimization techniques and tools are presented in Section 3. In Section 4, evaluation experiments from various scientific areas such as biology, chemistry, mathematics, etc., show the capabilities of the proposed algorithm. The accuracy of the proposed classifier and the computational advantages of the proposed classifier are summarized. In the last section, some conclusions from the use of the proposed method and implementation advances are given.

2. Method Description

A simple one-layer recurrent neural network (Elman [7]) transforms non-linearly each input symbol to an output symbol through a real-values vector mapping method using time-invariant data and memory-information-derived previous time non-linear transformations of the input sequence (also known as the memory of the cell).

In order to use an Elman-type recurrent neural network for symbol sequences classification, known also as artificial language modeling, several modifications to its original version are required.

The complete network and the binary classification process is separated in three distinguished parts: the data encoding-decoding process, the RNN activation function and the estimation of the output sequence mean log-probability. The binary classification criterion of the mean log-probability of the recurrent neural network outputs is a novel approach, as in the vast majority of implementations [8,9] the output vector at the $T_{th}$ (final) step is interpreted as the output of the classifier.

2.1. Data Encoding–Decoding Process

The proposed method can be used to define binary classifiers of symbol sequences built on Elman recurrent neural networks using sets of positive and negative samples. Each individual symbol is mapped to a unique vector that consists of all zeros except the corresponding symbol index, which is set to one. Therefore, if the alphabet of the input sequences defined by the set $S=\{q_1,q_2,\dots ,q_M\}$, then the binary representation of the input symbols in the M dimensional space, where M is the number of distinct symbols, can be easily derived by $q_i\to \mathbf{x}={[0,\dots ,0,1,0,\dots 0]}^T$, where the value 1 is met in the $i_{th}$ position of the vector $\mathbf{x}$. Given a sequence of T symbols, the corresponding vectors in the RNN input is equal to $\mathbf{x}\left(t\right)$ for $t=1\dots T$.

2.2. Elman Network

Given the input vector of the cell $\mathbf{x}\left(\mathbf{t}\right)$ at time t and the previous hidden state of the memory cell $\mathbf{h}(\mathbf{t}-\mathbf{1})$, represented by a N-dimensional vector, at time $t-1$, the new state of the cell $\mathbf{h}\left(\mathbf{t}\right)$ is estimated by:

$$\mathbf{h}\left(t\right)=\mathbf{f}({\mathbf{W}}_{\mathbf{hh}}\ast \mathbf{h}(\mathbf{t}-\mathbf{1})+{\mathbf{W}}_{\mathbf{xh}}\ast \mathbf{x}\left(\mathbf{t}\right)+{\mathbf{b}}_{\mathbf{h}})$$

(1)

where $f(.)$ is a non-linear activation function, ${\mathbf{W}}_{\mathbf{hh}}$ are the hidden state synaptic weights, ${\mathbf{W}}_{\mathbf{xh}}$ are the linear transformation weights of the input and ${\mathbf{b}}_{\mathbf{h}}$ is the input bias factor. At time $t=0,\mathbf{h}\left(0\right)=\mathbf{0}$.

The N-dimensional state $\mathbf{h}\left(\mathbf{t}\right)$ of the recurrent network in time t is then converted to an M-dimensional vector $\mathbf{p}\left(\mathbf{t}\right)$:

$$\mathbf{p}\left(t\right)=\sigma ({\mathbf{W}}_{\mathbf{hp}}\ast \mathbf{h}\left(t\right)+{\mathbf{b}}_p),$$

(2)

where $\sigma (.)$ represent the softmax function, ${\mathbf{W}}_{\mathbf{hp}}$ is the output synaptic weights and ${\mathbf{b}}_{\mathbf{p}}$ is the output bias factor.

The softmax function is used to derive the symbols probabilities at the RNN output $\mathbf{p}\left(t\right)$, and is formally defined as:

$$p_i={\mathbf{p}}_i=\sigma \left({\mathbf{x}}_{\mathbf{i}}\right)=\frac{e^{{\mathbf{x}}_{\mathbf{i}}}}{\sum e^{{\mathbf{x}}_{\mathbf{i}}}},i=1,M$$

(3)

It is well-known that the softmax [22] function is an efficient transformation function of a real-valued vector into a vector of probabilities due to the monotonicity of the exponential function into the set of positive real numbers and the normalization factor of their sum.

2.3. Estimation of Mean Log-Probability

The mean log-probability of the input sequence at the output of the recurrent cell is given by:

$$LogProb\left(\mathbf{x}\right)=\frac{1}{T}\sum _{t=0}^{T-1}lo{g}_2\left(q\left(t\right)\right)$$

(4)

In Equation (4), $q\left(t\right)$ represents the element of the output $\mathbf{p}\left(t\right)$ at index $q\left(t\right)=argmax\left(\mathbf{p}\right(t\left)\right)$.

The input sequence is classified as a positive example if the mean log-probability is greater than a threshold. The optimal threshold is estimated during the training process.

2.4. Training Method

Typically, the Elman networks require an extremely long training time when the network weights are estimated from positive ${\mathrm{\Omega }}^{+}$ and negative ${\mathrm{\Omega }}^{-}$ examples using the traditional error-backpropagation through time class of algorithms. Moreover, in the gradient vanishing problem, the convergence to local-minima of the error function is inevitable in these approaches. Therefore, a genetic algorithm is adopted due to the stochastic nature of this evolutionary search approach, the ability to perform the computations in parallel and the excellent behavior in the presence of large number of local optima of the optimized function.

The weights of the Elman network are derived by a modified genetic algorithm with local search for enhanced convergence. Local search is a procedure in which, at every $T_l$ iterations of the genetic algorithm, $N_t$ chromosomes are randomly selected and optimized through a non-evolutionary method. In the experiments conducted in this paper, the ADAM method was selected as local search optimizer.

The weights of the Elman network can be encoded in different depths of accuracy, i.e. double, single precision floating or fixed point numbers. In typical recurrent structures, the accuracy of an objective function is more sensitive with respect to the recurrent part of the structure, in our case the $W_{hh}$ weights.

The proposed training method is presented in the pseudo-code of Algorithm 1. The fitness of each chromosome is estimated from the log-probabilities of the RNN output for all positive and negative available examples. The overall fitness criterion decreases the log-probability for the negative examples, also increasing the log-probability for the positive examples:

$$F({\mathrm{\Omega }}^{+}\cup {\mathrm{\Omega }}^{-})=\sum _{\mathbf{x}\in {\mathrm{\Omega }}^{+}}LogProb\left(\mathbf{x}\right)-\sum _{\mathbf{x}\in {\mathrm{\Omega }}^{-}}LogProb\left(\mathbf{x}\right)$$

(5)

The goal of the training method is to maximize the fitness metric of Equation (5).

Algorithm 1 Training Elman-Classifiers

(a)  Set $i=0$ (iteration counter) and MaxIters(maximum number of iterations);  (b)  Set the $p,p_m,p_s$ the population size, mutation rate and selection rate of the algorithm;  (c)  Set $T_l,N_l$ the frequency and number of chromosomes for local optimization;  (d)  Randomly initialize each chromosome in ${\left[0,1\right]}^n$.  (a)  For each individual in population calculate its fitness  (b)  $i=i+1$  (a)  Select the best $p_s$ individuals of the population;  (b)  Apply one-point crossover to the selected individuals to derive the new population;  (c)  Randomly mutate chromosomes in respect to $p_m$.  If i mod $T_l==0$ : apply local optimization to $N_l$ random chromosomes  If $i>MaxIters$ :  (a)  Calculate the optimal threshold where the minimum classification error occurs;  (b)  Return the best chromosome and the optimal threshold.  else : goto 2.

The optimal threshold is estimated by scanning the log-probability space and detecting the value where the classification error of the trained Elman RNN classifier is minimum for the training set of both positive and negative examples ${\mathrm{\Omega }}^{+},{\mathrm{\Omega }}^{-}$.

Randomly selected chromosomes are optimized for a small number of iterations through the selected local optimization method with respect to the fitness metric.

For the local optimization method, the ADAM [23] optimizer was selected as a lightweight but fast converging algorithm. The improvements in the convergence of the model in all experiments are presented in

Table 1. Experimental results on selected datasets.

DATASET	GA-Elman	GA-Elman + ADAM
Regex	78.42%	87.75%
Primes	62.80%	68.50%
Flu	69.00%	75.25%
LogP	64.13%	71.63%
Sentiment	53.83%	54.01%
Mnist	85.63%	87.38%
Dnacvs	61.78%	66.50%
Prop	66.36%	80.53%

.

3. Implementation

The software developed for the algorithm evaluation, the experiments and the corresponding data are freely distributed in itsoulos.teiep.gr/GeneticRNN.tar.gz (accessed on 29 August 2022).

The proposed classifier and the training method have several benefits in regards to their complexity and their performance on any modern computing system. Modern CPUs mainly offer two types of parallelism: SIMD instructions and concurrent execution of parallel sections. These capabilities are covered by the OpenMP standard that is implemented in the most popular C/C++ compilers [24,25]. The use of these capabilities are quantified in

Table 2. Comparison between implementations.

DATASET	Non-Optimzed	2 Threads + SIMD	8 Threads + SIMD
Regex	205 min.	42 min. (×4.88)	19.4 min. (×10.56)
Primes	162 min.	31 min. (×5.22)	14.4 min. (×11.25)
Prop	111 min.	21.2 min. (×5.23)	10.5 min. (×10.57)

.

3.1. Hardware Acceleration

The recurrent neural network was implemented explicitly using SIMD instructions through the OpenMP API [26] as it yielded better results than the automatic vectorization capabilities in the case of the GCC compiler that was used. The steps 2 and 4 of the training algorithm, described in Section 2.4, were implemented by using the multi-process capabilities of the OpenMP standard. It is important to mention that, while multi-threading programming can also be used for the Elman classifier, it is best to minimize the number of threads created, as the accumulated overhead may reduce the acceleration rate significantly. Moreover, the fitness estimation procedure is completed in almost constant time in our method, so the static scheduling capabilities [27] of the OpenMP API further minimize overheads by spawning threads for chunks of estimations rather than estimating the fitness of each single chromosome in a single thread.

3.2. Software Techniques

In order to obtain the gradient of the objective function (Equation (5)) with respect to the corresponding data or a batch of data for the ADAM optimizer, the differentiation was implemented with the automatic differentiation (AD) method of the Adept library [28]. This approach enables exact estimation of the gradient for any continuous and smooth function, as in the case of our objective function or any alternative non-linear function adopted by the user. Furthermore, this is performed by template programming in C++, which enables the compiler to generate code at compile time, which enables even more optimization. The results of the previously mentioned techniques are quantified in Table 2, whereas the partially optimized code (-O2 default flag in GCC) and optimized code with vector processing and multi-threading for 2 and 8 thread run times are presented.

4. Experiments

4.1. Datasets

A series of datasets from different fields were used as test cases for the evaluation of the proposed classifier and the corresponding training method. In

Table 3. Datasets summary.

DATASET	Positive	Negative	Symbols	Av. Length	STDEV Length
Regex	300	300	6	17	1.5
Primes	500	500	2	12	1.2
Flu	200	200	4	147	13.6
LogP	3000	3000	32	12	8.6
Sentiment	1480	1302	28	86	23.5
Mnist	1000	1000	2	784	0.0
Dnacvs	1000	1000	4	55	0.0
Prop	90	90	10	22	6.3

, the number of positive/negative sequences, the number of distinct symbols in the dataset, the average length of each instance and the standard deviation are summarized.

Regex Data. Two artificial regular expressions $R{E}_1,R{E}_2$, where $R{E}_1\cap R{E}_2=Ø$ and

• $R{E}_1=\left(a\right|b\left|c\right|d\left|e\right|f)\{6,7\}\left(a\right|b\left|c\right)\{2,3\}\left(a\right|b\left|c\right|d\left|e\right|f)\{6,7\}\left(a\right|b\left|c\right)\{1,3\}$
• $R{E}_2=\left(a\right|b\left|c\right|d\left|e\right|f)\{6,7\}\left(d\right|e\left|f\right)\{2,3\}\left(a\right|b\left|c\right|d\left|e\right|f)\{6,7\}\left(d\right|e\left|f\right)\{1,3\}$

where $\left(a\right|b\left|c\right|d\left|e\right|f\left)\right\{6,7\}$ is a sequence of six to seven characters from a–f [29] and were used to measure both classification accuracy and encoding efficiency in the well control environment of simple artificial languages with a small number of terminal symbols.

A total number of 300 positive and 300 negative instances were randomly created from the above regular expressions as positive and negative data. The intersection of the positive and negative data sequences is an empty set.

Biological datasets. Two additional biological datasets with only four terminal symbols (A,G,C and T) were sampled from the well-known genome sequences NCBI and [30], called Flu and Dnacvs, as shown in the experimental result Table 1.

The Flu database consists of 200 sequences of DNA from the flu virus strains and prion strains randomly selected from the NCBI dataset [31], aiming to successfully distinguish if an arbitrary length genome sequence represents a virus (positive sequences) or a prion (negative sequences).

The Dnacvs dataset consists of genomes that represent either coding (positive sequences) or non-coding genomic DNA sequences (negative sequences) [30]. A total number of 1000 positive sequences and 1000 negative sequences were randomly sampled, and the goal of the classifier was to distinguish, given an arbitrary DNA sequence, if this sequence can be translated to a protein or not.

Chemical dataset. The LogP dataset was used to predict if a chemical substance has positive [32] or negative LogP [33] value using only the molecule in SMILES [34] representation. If the LogP of a molecule instance is positive, the instance is considered a positive example, else the molecule is considered a negative example. The database consists of 3000 positive and 3000 negative instances, while there are 32 distinct terminal symbols in the dataset.

Image dataset. In this dataset, two hand-written numbers (0, 8) from the MNIST dataset were selected to represent the two classes of the classifier: positive and negative examples. These numbers were selected because they are similar enough to pose a difficult problem for the proposed method. Each instance is a sequence of 784 ($28\times 28$ pixels) binary numbers representing the hand-written numbers. A total of 1000 positive and 1000 negative examples were used.

The Sentiment dataset. The dataset [35] was used to build an Elman classifier capable of distinguishing if a sentence describes a movie in a positive or negative mood. The database consists of 1480 positive sequences and 1302 negative sequences. There are 28 terminal symbols in this database representing the English letters and some punctuation symbols.

Mathematical datasets. The Prop dataset represents sequences of decimal numbers for positive data and random sequences of decimal numbers as negative data. The positive sequences of digits were generated by taking a positive integer number b and then by successively multiplying this number by $1,\dots ,10$ and concatenating the products. The numbers b were in the range $[10,100]$ for the positive sequences.

The Primes dataset consists of 500 random primes starting from 101 and 500 random not-prime integers starting from 100, both in binary form. While the prime identification problem has been up to now unsolvable by a finite rules grammar, it is well-known that there are several noisy patterns in the binary representation of primes. The experiments carried out with the Primes dataset explore the capabilities of the RNN training method to detect primes from their binary representation.

4.2. Experimental Results

In Table 1, the experimental results derived by the proposed training method using the evaluation datasets are presented. Each row contains the dataset name, the RNN classification accuracy when the training process is completed with the aid of the local search optimization procedures, or using only the genetic learning part of the algorithm. Each experiment was conducted ten times with a different seed for the random generator each time, and the average rates are presented, minimizing the influence of the initial random selection and the random processes involved in the genetic learning algorithm. The values presented in

Table 4. Algorithm parameters .

Parameter	Value
Memory size ($size\left(h\right)$)	10
Chromosomes	500
Generations	300
Elitism	1 chromosome
Selection ($p_s$)	0.9
Mutation ($p_m$)	0.05
Local Chromosomes ($N_t$)	4
Local Frequency ($T_l$)	20

were used for each experiment. The accuracy gain from the use of local optimization is 2–25% depending on the dataset. The size of the memory of the neuron N in all experiments was set to 10, which means that the number of independent variables in the proposed classifier is extremely small. For $N=10$, the total number of synaptic weights, depending on the number of independent symbols in each experiment, varies from 150 to 250 double precision numbers.

For the MNIST dastaset especially, there have been several methods tested [36,37] using Elman networks. The method proposed in [36], for example, proposes a more complicated inference method and barely outperforms our method for similar tasks in number classification. Our method outperforms the method proposed in [37], but the scope of our dataset is limited (similarly to [36]) in comparison to the test conducted in [37].

5. Conclusions

A novel Recurrent Neural Network classifier trained by combined evolutionary and gradient-based techniques is described in this article and implemented. The information combining with the log-prob decoding from the iterative outputs of the classical Elman network shows promising potential in a great variety of binary classification problems. Furthermore, the computational overhead from the log-prob decoding is minimal, as this method does not introduce complex structures. It is also shown that the proposed classifier and the corresponding training method can be applied in different scientific areas including chemical, image, sentiment, mathematical and biological problems. Moreover, the use of local search methods in the training process, combined with an evolutionary learning method, outperforms in all cases the accuracy of the typical GA-Elman algorithm. It is notable that despite the small computational complexity of the model, the classification rates are robust, even in the case of severe scientific problems.

The limitation of this method is scalability in complex models. This limitation is present in the vast majority of evolutionary methods as each candidate solution must be evaluated on the whole dataset. So, complex models, where the evaluation process is computationally expensive, will need extensive training time.

Future Plans

This method can be extended in order to fit several types of recurrent neural networks. We would like to implement this method in LSTMs, GRUs and the corresponding variants of these architectures. This method can be used without changing anything in the majority of neural-network architectures.