1. Introduction
In the area of artificial intelligence (AI), two important perspectives stand out. The first is the applied rule that represents the given problem. The applied rule is vital in decision making in order to explain the nature of the problem. The second perspective is the automation process based on the rule which leads to neuro symbolic integration. These two perspectives rely heavily on the practicality of the symbolic rule that governs the AI system. The use of a satisfiability (SAT) perspective in software and hardware system theories is currently one of the most effective methods in bridging the two perspectives. SAT offers the promise, and often even the reality, that the model checks efforts with feasible industrial application. There were several practical applications of SAT that can be mentioned in this section. Ref. [
1] utilized Boolean SAT by integrating satisfiability modulo theories (SMT) in tackling the scheduling problem. The proposed SMT method was reported to outperform other existing methods. Ref. [
2] discovered vesicle traffic network by model checking that incorporates Boolean SAT. The proposed SAT model established a connection between vesicle transport graph connectedness and underlying rules of SNARE protein. In another development, [
3] developed several SAT formulations to deal with the resource-constrained project scheduling problem (RCPSP). The proposed method is reported to solve various benchmark instances and outperform the existing work in terms of computation time and optimality. SAT formulation is a dynamic language that can be used in representing problem in hand. Ref. [
4] proposed a special SAT in modelling the circuit. The proposed method reconstructed the accurate circuit configuration up to 90%. The application of SAT in very-large-scale integration (VLSI) inspires the authors to extend the application of SAT into pattern reconstruction [
5] where they used the variable in SAT as a building block of the desired pattern. The practicality of SAT motivates researchers to implement SAT in navigating the structure in an artificial neural network (ANN).
Logic programming in ANN has been initially proposed by [
6]. In his work, logic programming can be embedded into the Hopfield neural network (HNN) by minimizing the logical inconsistencies. This is also a pioneer to the Wan Abdullah method which obtains the synaptic weight by comparing cost function with Lyapunov energy function. Ref. [
7] further developed the idea of the logic programming in HNN by implementing Horn satisfiability (HornSAT) as a logical structure of HNN. The proposed network achieved more than 80% global minima ratio but high computation time due to the complexity of the learning phase. Since then, logic programming in ANN was extended to another type of ANN. Ref. [
8] initially proposed logic programming in radial basis function neural network (RBFNN) by calculating the centre and width of the hidden neurons that corresponds to the logical rule. In the proposed method, the dimensionality of the logical rule from input to output can be reduced by implementing Gaussian activation function. The further development of logic programming in RBFNN were proposed in [
9] where the centre and the width of the RBFNN are systematically calculated. In another development, [
10] proposed a systematic logical rule by implementing a 2-satisfiability logical rule (2SAT) in HNN. The proposed hybrid network is incorporated with effective learning methods, such as genetic algorithm [
11] and artificial bee colony [
12]. The proposed network managed to achieve more than 95% of global minima ratio and can sustain a high number of neurons. In another development, [
13] proposed the higher order non-systematic logical rule, namely random
k satisfiability (RAN
kSAT) that consists of random first-, second-, and third-order logical rule. The proposed works run a critical comparison among a combination of RAN
kSAT and demonstrate the capability of non-systematic logical rule to achieve optimal final state. The practicality of the SAT in HNN was explored in pattern satisfiability [
5] and circuit satisfiability [
4] where the user can capture the visual interpretation of logic programming in HNN. However, up to this point, the choice of SAT structure in HNN has received very little research attention, despite its practical importance.
Current data mining were reported to achieve good accuracy but the interpretability of the output is poorly understood due to emphasize of the black box model. In other words, the output makes sense for the AI but not for the user. One of the most useful applications of logic programming in HNN is logic mining. Logic mining is a relatively new perspective in extracting the behaviour of the dataset via logical rule. This method is a pioneer work of [
14]. In this work, the proposed RA extracted individual logical rule that represents the performance of the students. The logical rule extracted from the datasets is based on the number of induced Horn logics produced by HNN. Thus, there is very limited effort to identify the “best” induced logical rule that represent the datasets. To complement the limitation of the previous RA, several studies include specific SAT logical rules to be embedded into HNN. Ref. [
15] introduced 3-satisfiability (3SAT) as a logical rule in HNN, thus creating the first systematic logic mining technique, i.e., the
k satisfiability reverse analysis method (
kSATRA). The proposed hybrid logic mining is used to extract logical rule in several fields of studies, such as social media analysis [
15] and cardiovascular disease [
16]. In another development, different types of logical rule (2SAT) have been implemented by [
17]. They proposed 2SATRA by incorporating the 2SAT logical rule in extracting a diabetes dataset [
17] and student’s performance dataset [
18]. Ref. [
19] utilized 2SATRA by extracting logical rule for football datasets in several established football league in the world. Pursuing that, the ability of 2SATRA is further tested when the proposed method is implemented in e-games. The 2SATRA has been proposed to extract the logical rule that explains the simulation game of the League of Legend (LOL) [
20]. The proposed method achieved an acceptable range of logical accuracy. The application of logic mining was extended to several prominent areas, such as extracting the price information from commodities [
21]. Another interesting development for
kSATRA is by incorporating energy in induced logic. Ref. [
22] proposed an energy-based 2-satisfiability-based reverse analysis method (E2SATRA) for e-recruitment. The proposed method reduced the suboptimal induced logic and increased the classification accuracy of the network. Despite the increase in application in data mining, the existing logic mining endured a significant drawback. The induced logic produced by the proposed method suffers from a limited amount of search space. This is due to the positioning of the neurons in
kSAT formulation which affects the classification ability of 2SATRA. In this case, the optimal choice of the neuron pair in the
kSAT clause in logic mining is crucial to avoid possible overfitting.
There were various studies that implemented regression analysis in ANN. Standalone regression analysis was prone to data overfitting [
23], easily affected by outlier [
24], and mostly limited to a linear relationship [
25]. Due to the above weaknesses, regression analysis was implemented to complement the intelligent system. In most studies, regression analysis will be utilized in the pre-processing layer before it can be processed by the ANN. Ref. [
26] proposed a combination of regression analysis with a RBFNN. The proposed method formed a prediction model for national economic data. Ref. [
27] proposed an ANN that combines with regression analysis via a mean impact value. The proposed hybrid network identifies and extracts input variables that deal with irregularity and vitality of Beijing International Airport’s passenger flow dataset. In [
28], ANN is used to predict the water turbidity level by using optical tomography. The proposed ANN utilized the regression analysis value as an objective function of the network. Ref. [
29] fully utilized logistic regression to identify significant microseismic parameters. The significant parameters will be trained by a simple neural network which results in the highly accurate seismic model. By nature, ANN is purely unsupervised learning and logistic regression analysis displays a major improvement to the overall performance. Although there were many studies conducted to confirm the benefit logistic regression analysis in classification and prediction paradigm, regression analysis has never been implemented in classifying the SAT logical rule. Regression analysis has the ability to restructure the logical rule based on the strength of relationship for each
k variables in the
kSAT clause. In that regard, the ANN will learn the correct logical structure and the probability to achieve highly accurate induced logical rule will increase dramatically. In that regard, relatively few studies have examined the effectiveness of regression in analysing data features that correspond to the
kSAT. The choice of variable pair in the 2SAT clause can be made optimally by implementing regression analysis without interrupting the value of the cost function.
Unfortunately, there is no recent effort to discover the optimal choice that leads to the true outcome of the
kSAT. The closest work that addresses this issue is shown by [
30]. This work [
30] utilized neuron permutation to obtain the most accurate induced logical rule by considering
neuron arrangement in
kSAT. Hence, the aim of this paper is to effectively explore the various possible logical structures in 2SATRA. The proposed logic mining model identifies the optimal neuron pair for 2SAT clause forming a new logical formula. Pearson chi-square association analysis will be conducted to examine the connectedness of the neuron with respect to the outcome. By doing so, the new 2SAT formula learned by HNN as an input logic and the new induced logical rule can be obtained. Thus, the contributions of this paper are:
- (a)
To formulate a novel supervised learning that capitalize correlation filter among variables in the logical rule with respect to the logical outcome;
- (b)
To implement the obtained supervised logical rule into HNN by minimizing the cost function which minimizes the final energy;
- (c)
To develop a novel logic mining based on the hybrid HNN integrated with the 2-satisfiability logical rule;
- (d)
To construct the extensive analysis for the proposed logic mining in doing various datasets. The proposed logic mining will be compared to the existing state of the art logic mining.
An effective 2SATRA model incorporating a new supervised model will be compared with the existing 2SATRA model for several established datasets. In
Section 2, we describe satisfiability programming in HNN in detail. In
Section 3, we describe some simulation of HNN by using simulated result. Discussion follows in
Section 4. The concluding remarks in
Section 5 complete the paper.
4. Satisfiability in Discrete Hopfield Neural Network
HNN [
33] consists of interconnected neurons without a hidden layer. Each neuron in HNN is defined in bipolar state
that represents true and false, respectively. An interesting feature about HNN is the ability to restructure the neuron state until the network reached its minimum state. Hence, the proposed HNN achieved the optimal final state if the collection of neurons in the network reached the lowest value of the minimum energy. The general definition of HNN with the
i-th activation is given as follows
where
and
represent a threshold and synaptic weight of the network, respectively. Without compromising the generality of HNN, some study used
as the threshold value. Note that
is the number of 2SAT variables.
is also defined as the connection between neuron
and
. The idea of implementing
in HNN (HNN-2SAT) is due to the need of some symbolic rule that can govern the output of the network. The cost function
of the proposed
in HNN is given as follows:
where
is the number of
clause. The definition of the clause
is given as follows [
9]
where
is the negation of literal in
. It is also worth mentioning that
if the
is because the neuron state
associated to
is fully satisfied. Each variable inside a particular
will be connected by
. Structurally, the synaptic weight of
is always symmetrical for both the second- and third-order logical rule:
with no self-connection between neurons:
Note that Equations (5)–(8) only account for a non-redundant logical rule because the logical redundancies will result in the diminishing effect of the synaptic weight. The goal of the learning in HNN is to minimize the logical inconsistency that leads to
or
. Although synaptic weight of the HNN can be properly trained by using conventional method, such as Hebbian learning [
33], ref. [
14] demonstrated that the Wan Abdullah method can obtain the optimal synaptic weight with minimal neuron oscillation compared to Hebbian learning. For example, if the embedded logical clause is
, the synaptic weights will read
. During retrieval phase of HNN-2SAT, the neuron state will be updated asynchronously based on the following equation.
where
is a final neuron state with pre-defined threshold
. In terms of output squashing, the Sigmoid function can be used to provide non-linearity effects during neuron classification. Potentially, the final state of the neuron must contain information that lead to
, and the quality of the obtained state can be computed by using Lyapunov energy function:
According to [
33], the symmetry of the synaptic weight is sufficient condition for the existence of the Lyapunov function. Hence, the value of
in Equation (10) decreases monotonically with network. The absolute minimum energy
can pre-determined by substituting interpretation that leads to
. In this case, if the obtained neuron state can satisfy
, the final neuron state achieved global minimum energy. Note that the current conventions of
can be converted to binary by implementing different a Lyapunov function coined by [
6].
5. Proposed Method
2SATRA is a logic mining method that can extract a logical rule from the dataset. The philosophy of the 2SATRA is to find the most optimal logical rule of Equation (1), which corresponds to the dynamic system of Equation (9). In the conventional 2SATRA proposed by [
20], the choice of variable in 2SATRA will be determined randomly which leads to poor quality of the induced logic. The choices of the neurons are arranged randomly before the learning of HNN can take place. In this section, chi-square analysis will be used during the pre-processing stage. The aim of the association method is to assign the two best neurons/clauses that correspond to the outcome
. These neurons will take part during the learning phase of HNN-2SAT which leads to better induced logic. In other words, the additional optimization layer is added to reduce the pre-training effort for 2SATRA to find the best logical rule.
Let
the number of neurons represent the attribute of the datasets
where each neuron is converted into bipolar interpretation
. Necessarily, 2SATRA is required to select
neurons that will be learned by HNN-2SAT. In this case, the number of possible neuron permutation after considering the learning logic
structure is
. By considering the relationship between
and neuron
, we can optimally select the pair of
for each clause
. The
selection for each
is given as follows:
where
is the
P value between
and the neuron
.
signifies the minimized value of
recorded between
and
, and the value of
is pre-defined by the network. Note that
does not significy a self-connection between the same neurons. By considering the best- and worst-case scenario, the neuron will be chosen at random if
. If the examined neurons do not achieve the pre-determined association, HNN-2SAT will reset the search space, which fulfils the threshold association value. Hence, by using Equation (11), the proposed 2SATRA is able to learn the early feature of the dataset. After obtaining the right set of neurons for
, the dataset will be converted into bipolar representation:
Note that we only consider the second-order clause or
for each clause in
. Hence, the collection of
that leads to positive outcome of the learning data or
will be segregated. By calculating the collection of
that leads to
, the optimum logic
is given as follows:
where
is the number of
that leads to
. Hence, the logical feature of the
can be learned by obtaining the synaptic weight of the HNN. In this case, the cost function in Equation (11) which corresponds to
will be compared to Equation (5). By using Equation (9), we obtain the final neuron state
.
Since the proposed HNN-2SAT only allows an optimal final neuron state, the quality of the
will be verified by using
. In this case,
that leads to local minima will not be considered. Hence, the classification of the induced
is as follows:
where
can be obtained from Equation (10). It is worth mentioning that if the two neurons do not have the strong association, the neurons will not be considered. Thus, if the association value for all neurons does not achieve the threshold variable
, the proposed network will be reduced to conventional
kSATRA proposed by [
21,
31].
Figure 1 shows the implementation of the proposed supervised logic mining or (S2SATRA). Algorithm 1 shows Pseudo code of the Proposed S2SATRA.
Algorithm 1. Pseudo code of the Proposed S2SATRA. |
| Input: Set all attributes with respect to . |
| Output: The best induced logic . |
1 | Begin |
2 | Initialize algorithm parameters; |
3 | Define the Attribute for with respect to ; |
4 | Find the correlation value between with ; |
5 | fordo |
6 | | if | Equation (11) is satisfied then |
7 | | | Assign as , and continue; |
8 | | while | do |
9 | | | Using the found attributes, find using Equation (13); |
10 | | | Check the clause satisfaction for ; |
11 | | | Compute using Equation (10); |
12 | | | Compute the synaptic weight associated with using the WA method; |
13 | | | Initialize the neuron state; |
14 | | | for | |
15 | | | | Compute using Equation (9); |
16 | | | | Convert to the logical form using Equation (14); |
17 | | | | Evaluate the by using Equation (10); |
18 | | | | If | Condition (15) is satisfied then |
19 | | | | | Convert to induced logic ; |
20 | | | | | Compare the outcome of the with and continue; |
21 | | | | ; |
22 | | | end for | |
23 | | | ; |
24 | | end while | |
25 | end for |
26 | End |
8. Discussion
The optimal logic mining model requires pre-processing structures for neurons before the can be learned by HNN. Currently, the logic mining model specifically optimizes the logic extraction from the dataset without considering the optimal . The mechanism that optimizes the optimal neuron relationship before the learning can occur is remain unclear. In this sense, identifying a specific pair of neurons for will facilitate the dataset generalization and reduce computational burden.
As mentioned in the theory section, S2SATRA is not merely a modification of a conventional logic mining model, but rather it is a generalization that absorbs all the conventional models. Thus, S2SATRA not only inherits many properties from a conventional logic mining model but it adds supervised property that reduces the search space of the optimal
. The question that we should ponder is: what is the optimal
for the logic mining model? Therefore, it is important to discuss the properties of S2SATRA against the conventional logic mining model in extracting optimal logical rule from the dataset. According to the previous logic mining model, such as [
20,
21,
31], the quality of attributes is not well assessed since the attributes were randomly assigned. For instance, [
21] achieved high accuracy for specific combination of attributes but the quality of different combination of the attributes will result in low accuracy due to a high local minima solution. A similar neuron structure can be observed in E2SATRA, as proposed by [
24], because the choice of neurons is similar during the learning phase. Practically speaking, this learning mechanism [
20,
21,
22,
31] is natural in real life because the neuron assignment is based on trial and error. However, the 2SATRA model needs to sacrifice the accuracy if there is no optimum neuron assignment. By adding permutation property, as carried out in Kasihmuddin et al. [
30], P2SATRA is able to increase the search space of the model in the expense of higher computational complexity. To put things into perspective, 10 neurons require learning 18,900 of
learning for each neuron combination before the model can arrive to the optimal result. Unlike our proposed model, S2SATRA can narrow down the search space by checking the degree of association among the neurons before permutation property can take place. Supervised features of S2SATRA recognized the pattern produced by the neurons and align it with the
clause. Thus, the mutual interaction between association and permutation will optimally select the best neuron combination.
As reported in
Table 7 and
Table 8, the number of associations for analysis required for
n attributes to create optimal
was reduced by
. In other words, the probability of P2SATRA to extract optimal
is lower compared to the S2SATRA. As the
supplied to the network has changed, the retrieval property of the S2SATRA model will improve. The best logic mining model demonstrates a high value of
and
with a minimized value of
and
. P2SATRA is observed to outperform the conventional logic mining in terms of performance metrics because P2SATRA can utilize the permutation attributes. In this case, the higher the number of permutations, the higher probability for the P2SATRA to achieve correct
and
. Despite a robust permutation feature, P2SATRA failed to disregard the non-significant attributes which leads to
. Despite achieving high accuracy, the retrieved final neuron state is not interpretable. E2SATRA is observed to outperform 2SATRA in terms of accuracy because the induced logic in E2SATRA is the only amount in the final state that reached global minimum energy. The dynamic of the induced logic in E2SATRA follows the convergence of the final state proposed in [
22] where the final state will converge to the nearest minima. Although all the final state in E2SATRA is guaranteed to achieve global minimum energy, the choice of attribute that is embedded to the logic mining model is not well structured. Similar to 2SATRA and P2SATRA, the interpretation of the final attribute will be difficult to design. In another development, 2SATRA is observed to outperform the RA proposed by [
14] in terms of all performance metric. Although the structure of RA is not similar to 2SATRA in creating the
, the induced logic
has a general property of
. In this case,
is observed to create a rigid induced logic (at most 1 positive literal) and can reduce the possible solution space of the RA. In this case, we only consider the dataset that satisfies the
that will lead to
.
In contrast, S2SATRA employs a flexible
logic which accounts for both positive and negative literal. This structure is the main advantage over the traditional RA proposed by [
14]. S2ATRA is observed to outperform the rest of the logic mining model due the optimal choice of attributes. In terms of feature, S2SATRA can capitalize the energy feature of E2SATRA and the permutation feature of P2SATRA. Hence, the induced logic obtained will always achieve global minimum energy and only relevant attribute
will be chosen to be learned in HNN. Another way to explain the effectiveness of logic mining is the ability to consistently find the correct logical rule to be learned by HNN. Initially, all logic mining models begin with HNN which has too many ineffective synaptic weights due to suboptimal features. In this case, S2SATRA can reduce the inconsistent logical rule that leads to suboptimal synaptic weight.
S2SATRA is reported to outperform almost all the existing logic mining models in terms of all performance metrics. S2SATRA has the capability to differentiate between and , which leads to high Acc and F-score values. Since S2SATRA is able to obtain more , the Pr and Sen will increase compared to the other existing methods. In terms of Pr and Sen, S2SATRA is reported to succesfully predict during the retrieval phase. In other words, the existing 2SATRA model is less sensitive to the positive outcome which leads to a lower value of Pr and Se. It is worth mentioning that the overfitting nature of the retrieval phase will lead to which can only produce more positive neuron states. This phenomenon was obvious in the existing method where the HNN tends to converge to only a few final states. This result has a good agreement with the McNemar’s test where the performance of S2SATRA is significantly different from the existing method. The optimal arrangement of the signifies the importance of the association among the attributes towards the retrieval capability of the S2SATRA. Without proper arrangement, the obtained tends to overfit which leads to a high classification error. S2SATRA can only utilize correlation analysis during the pre-processing stage because correlation analysis provides preliminary connection between the attribute and .
It is worth noting that although there are many developments of the supervised learning method, such as a decision tree, a support vector machine, etc., none of these methods can provide the best approximation to the logical rule. Most of the mentioned methods are numerically compatible as an individual classification task, but not as a classification via a logical rule. For instance, a decision tree is effective in classifying the outcome of the dataset but S2SATRA is more effective in generalizing the datasets in the form of induced logics. The obtained induced logic can be utilized for a similar classification task. In term of parameter settings, S2SATRA is not dependent on any free parameter. The only parameter that can improve S2SATRA is the number of
Trial. Increasing the number of trials will increase the number of the final state that corresponds to the
. The main problem with this modification is that increasing the number of trials will lead to an unnecessary high computation time. Hence, in this experiment, the number of
Trial still follows the conventional settings in [
38]. It is worth noting that S2SATRA achieved the lowest accuracy for F1. This is due to imbalanced data, which leads to non-optimal induced logic. Correlation analysis cannot discriminate the correct relationship between variables and
. Generally, S2SATRA improved the pre-processing phase of the logic mining which leads to an improved learning phase due to the correct combination of
. The correct combination of
will lead to optimal
which can generalize the dataset.
Finally, we would like to discuss the limitations of the study. The limitation of the S2SATRA is the computation time due to the complexity of the learning phase. Since all logic mining models utilized the same learning model to maximize the fitness of the solution, computation time is not considered as a significant factor. As the number of attribute or clausal noise increases, the learning error will exponentially increase. Hence, metaheuristics and accelerating algorithms, such as in [
41], are required to effectively minimize the cost function in Equation (5) within a shorter computational time. This phenomenon can be shown when the number of neurons
in the logic mining model is trapped in a trial-and-error state. In terms of satisfiability, all the proposed 2SATRA models do not consider non-satisfiable logical structure or
, such as maximum satisfiability [
42] and minimum satisfiability [
43]. This is due to the nature of 2SATRA that only consider data point that leads to positive outcome or
. In terms of network, HNN is chosen compared to other ANN structures, such as feedforward because feedback to the input is compatible to the cost function
. Another problem that might arise for feedforward ANN, such as within the radial basis function neural network (RBFNN), is the training choice. For instance, the work of [
9,
44] can produce a single induced logic due to the RBFNN structure. This will reduce the accuracy of the S2SATRA model. A convolution neural network (CNN) is not favoured because propositional logic only deals with bipolar representation and multiple layers only increase the computational cost for the S2SATRA. In another perspective, weighted satisfiability that randomly assign the negation of the neuron will reduce the generality of the induced logic. In this case, 2SATRA model must add one additional layer during the retrieval phase to obtain which logical weight yields the best accuracy. Unlike several learning environments in HNN-2SAT [
45], learning iteration will not be restricted and will be terminated when
. A restricted value of the learning iteration will lead to more induced logic trapped in local minimum energy. As a worst-case scenario, a logic mining model, such as E2SATRA, will not produce any induced logic in restricted learning environment. Hence, all the 2SATRA models exhibit the same learning rule via the Wan Abdullah method [
6]. In addition, all the logic mining models, except for RA and conventional logic mining, follow the condition of
. In this case, only induced logic that can achieve global minimum energy will be considered during the retrieval phase. This is supported by [
33] where the final state of neuron that represents the induced logic will always converge to the nearest minimum. By employing the Wan Abdullah method and HTAF [
4], the number of solutions that corresponds to the local minimum solution will reduce dramatically. The neuron combination is limited to only
because the higher the value of
, the higher the learning error and HNN tends to be trapped in a trial-and-error state.
The experimental results presented above indicate that the S2SATRA improved the classification performance more than other existing logic mining model and created more solution variation. Another interesting phenomenon we discovered is that supervised learning features in S2SATRA reduce the permutation effort in finding the optimal
. As a result, HNN can retrieve the logical rule to do with acquiring higher accuracy. Additionally, we observed that when a number of clausal noise was added, S2SATRA shows a better result compared to the existing model. It is expected that our work can give inspiration to other logic mining models, such as [
20,
21], to extract the logical rule effectively. The robust architecture of S2SATRA provides a good platform for the application of real-life bioinformatics. For instance, the proposed S2SATRA can extract the best logical rule that classifies single-nucleotide polymorphisms (SNPs) inside known genes associated with Alzheimer’s disease. This can lead to large-scale S2SATRA design, which has the ability to classify and predict.