A Novel Boolean Kernels Family for Categorical Data ^†

Abstract

Kernel based classifiers, such as SVM, are considered state-of-the-art algorithms and are widely used on many classification tasks. However, this kind of methods are hardly interpretable and for this reason they are often considered as black-box models. In this paper, we propose a new family of Boolean kernels for categorical data where features correspond to propositional formulas applied to the input variables. The idea is to create human-readable features to ease the extraction of interpretation rules directly from the embedding space. Experiments on artificial and benchmark datasets show the effectiveness of the proposed family of kernels with respect to established ones, such as RBF, in terms of classification accuracy.

1. Introduction

Large-margin kernel machines (e.g., SVM) are recognized state-of-the-art algorithms in machine learning applications. They are broadly applied to several domains, such as text categorization, spam filtering, RNA function prediction, and so on. However, since these methods typically work on an implicitly defined feature space by resorting to the well-known kernel trick, the interpretability of the resulting model is difficult.

This last aspect is often crucial in specific application areas, such as the medical ones, in which the simple predictive answer is not enough. Being this a requirement for the acceptance of these black-box models by end users, in the last decade, several methods have been introduced for rule extraction from SVMs (see [1] for a recent survey). The majority of the proposed approaches try to extract if–then rules over the input variables and this task is generally hard when common kernels, e.g., the polynomial and RBF ones, are used.

On the other hand, Decision Trees (DT), thanks to their easy logical interpretation, are very appreciated, especially by non-expert users. The shortcoming of DTs is that, in general, they are not as accurate as more complex methods. In the case of binary valued input data, an alternative approach to make SVM more interpretable consists in defining features that are easy to interpret, for example, features that are propositional (i.e., logical) formulas applied to the input vectors. In particular, Boolean kernels are kernel functions in which the binary input vectors are mapped into an embedding space formed by propositional formulas of the input variables, and, in such space, the dot product is performed.

More formally, a Boolean kernel function $\kappa :{\{0,1\}}^n\times {\{0,1\}}^n\to \mathbb{N}$ is defined as $\kappa (\mathit{x},\mathit{z})=\langle \varphi \left(\mathit{x}\right),\varphi \left(\mathit{z}\right)\rangle$ where $\varphi :{\{0,1\}}^n\to {\{0,1\}}^N$ is the embedding Boolean function with $\mathit{x},\mathit{z}\in {\{0,1\}}^n$. When the input space is binary, the linear kernel can be seen as a limit case of a Boolean kernel in which the features simply represent the Boolean literals themselves.

In this paper, we propose a new family of Boolean kernels able to produce feature spaces composed by arbitrarily complex propositional formulas. In particular, we first introduce the basic Boolean kernels [2], namely the conjunctive kernel and the disjunctive kernel, for both the monotone and non-monotone case. On top of these kernels, we then propose more complex kernels such as the Disjunctive Normal Form kernel and the Conjunctive Normal Form kernel. For all the proposed kernels, an efficient method to compute them is provided. We assess the quality of the proposed kernels in terms of classification accuracy on several categorical datasets, and compare their performance against state-of-the-art kernels, such as the RBF kernel, and other Boolean kernels proposed in the literature.

The reminder of this paper is structured as follows: in Section 2, we give an overview of the existing work related to Boolean kernels. In Section 3, we present the proposed Boolean kernels family and, in Section 4, we discuss their computational complexity. Finally, Section 5 shows all the performed experiments on several benchmark categorical datasets.

2. Related Work

Sadohara [3] was the first to introduce the idea of Boolean kernel. In that work, the concept of Boolean kernel is actually related to a single kernel called DNF kernel. Specifically, he proposed a SVM for learning Boolean functions: since every Boolean (i.e., logical) function can be expressed in terms of Disjunctive Normal Form (DNF) formulas, the proposed kernel creates a feature space containing all possible conjunctions of negated or non-negated Boolean variables.

For instance, the feature space for a two variables, e.g., $x_1,x_2$, DNF contains the following $3^2-1$ features: $x_1,x_2,\neg x_1,\neg x_2,x_1\wedge x_2,x_1\wedge \neg x_2,\neg x_1\wedge x_2,\neg x_1\wedge \neg x_2$. The resulting decision function of a kernel machine which employs the DNF kernel can be represented as a weighted linear sum of conjunctions (Representer Theorem [4,5]), which in turn can be seen as a kind of “soft” DNF.

Formally, the DNF kernel between $\mathit{x},\mathit{z}\in {\mathbb{R}}^n$ is defined as

$${\kappa }_{\mathrm{dnf}}(\mathit{x},\mathit{z})=-1+\prod _{i=1}^n(2{\mathit{x}}_i{\mathit{z}}_i-{\mathit{x}}_i-{\mathit{z}}_i+2),$$

while its monotone (i.e., without negations) form is the following

$${\kappa }_{\mathrm{mdnf}}(\mathit{x},\mathit{z})=-1+\prod _{i=1}^n({\mathit{x}}_i{\mathit{z}}_i+1).$$

By restricting the domain of the vectors in ${\{0,1\}}^n$, the computation of the kernels is simplified as follows

$${\kappa }_{\mathrm{dnf}}(\mathit{x},\mathit{z})=-1+2^{\langle \mathit{x},\mathit{z}\rangle +\langle \overline{\mathit{x}},\overline{\mathit{z}}\rangle },{\kappa }_{\mathrm{mdnf}}(\mathit{x},\mathit{z})=-1+2^{\langle \mathit{x},\mathit{z}\rangle },$$

where $\overline{\mathit{x}}={\mathbf{1}}_n-\mathit{x}$ is the vector with all the binary entries swapped. Note that the sum $\langle \mathit{x},\mathit{z}\rangle +\langle \overline{\mathit{x}},\overline{\mathit{z}}\rangle$ simply counts the number of common “bits” between $\mathit{x}$ and $\mathit{z}$. These last two kernels were independently discovered in [6,7].

A drawback of this type of kernels is the exponential growth of the size of the feature space with respect to the number of involved variables, i.e., $3^n-1$ for n variables. To give the possibility of controlling the size of the feature space, Sadohara et al. [8] proposed a variation of the DNF kernel in which only conjunctions with up to d variables (i.e., d-ary conjunctions) are considered. Over binary vectors, this kernel, dubbed d-DNF kernel, is defined as

$${\kappa }_{\mathrm{dnf}}^d(\mathit{x},\mathit{z})=\sum _{i=1}^d\left(\begin{array}{c}\langle \mathit{x},\mathit{z}\rangle +\langle \overline{\mathit{x}},\overline{\mathit{z}}\rangle \\ i\end{array}\right),$$

and trivially, if $d=n$, ${\kappa }_{\mathrm{dnf}}^d(\mathit{x},\mathit{z})={\kappa }_{\mathrm{dnf}}(\mathit{x},\mathit{z})$. A nice property of the d-DNF kernel is that it yields a nested sequence of hypothesis spaces, i.e., ${\mathcal{H}}_1\subseteq {\mathcal{H}}_2\subseteq \dots \subseteq {\mathcal{H}}_n$. Thus, choosing a degree d (also known as “arity”) for the kernel implicitly means controlling the capacity of the hypothesis space, which is a very important aspect in learning. The same “trick” can be applied to the monotone d-DNF kernel [9]:

$${\kappa }_{\mathrm{mdnf}}^d(\mathit{x},\mathit{z})=\sum _{i=1}^d\left(\begin{array}{c}\langle \mathit{x},\mathit{z}\rangle \\ i\end{array}\right).$$

Instead of limiting the number of involved variables, Zhang et al. [10] proposed a parametric version of the DNF and mDNF kernel. Specifically, given $\mathit{x},\mathit{z}\in {\{0,1\}}^n$ and $\sigma >0$, then

$$\begin{array}{cc}\hfill {\kappa }_{\mathrm{dnf}}^{\left(\sigma \right)}(\mathit{x},\mathit{z})& =-1+\prod _{i=1}^n(\sigma {\mathit{x}}_i{\mathit{z}}_i+\sigma (1-{\mathit{x}}_i)+\sigma (1-{\mathit{z}}_i)+1),\hfill \\ \hfill {\kappa }_{\mathrm{mdnf}}^{\left(\sigma \right)}(\mathit{x},\mathit{z})& =-1+\prod _{i=1}^n(\sigma {\mathit{x}}_i{\mathit{z}}_i+1).\hfill \end{array}$$

The parameter $\sigma$ induces an inductive bias towards simpler or more complex DNF formulas. Specifically, for $\sigma$ in the range $[0,1]$ a bias towards shorter DNF is given, while for $\sigma >1$ the bias is more towards longer DNF. When $\sigma =1$, then ${\kappa }_{\mathrm{dnf}}^{\left(\sigma \right)}(\mathit{x},\mathit{z})={\kappa }_{\mathrm{dnf}}(\mathit{x},\mathit{z})$, and the same for the monotone DNF kernel. In the following, we refer to these kernel as $\sigma$-DNF and $\sigma$-mDNF kernel, respectively. Zhang et al. also proved that, for binary input vectors, the polynomial kernel, i.e., ${\kappa }_{\mathrm{POLY}}^p(\mathit{x},\mathit{z})={(\sigma \langle \mathit{x},\mathit{z}\rangle +c)}^p$, $c\in {\mathbb{R}}_{+}$, is a Boolean kernel, even though they did not provide any formal definition of Boolean kernel. Nonetheless, an important observation is that the embedding space of a polynomial kernel is composed by all the monomials (that are conjunctions) up to the degree p. Thus, the only difference between the polynomial one and the d-DNF kernel are the weights associated to the features. It is also noteworthy that, in the binary case, the embedding of the polynomial kernel contains sets of equivalent features, e.g., for $p=4$ and $\mathit{x}\in {\{0,1\}}^2$, the value of the features (in the feature space) ${\mathit{x}}_1^3{\mathit{x}}_2,{\mathit{x}}_1^2{\mathit{x}}_2^2,{\mathit{x}}_1{\mathit{x}}_2^3$ are equivalent to the feature ${\mathit{x}}_1{\mathit{x}}_2$.

A kernel related to the polynomial is the all-subset kernel [11,12], defined as

$${\kappa }_{\subseteq }(\mathit{x},\mathit{z})=\prod _{i=1}^n({\mathit{x}}_i{\mathit{z}}_i+1),$$

which considers a space with a feature for each subset of the input variables, including the empty subset. It is different from the polynomial because it does not limit the number of considered monomials, and it gives the same weight to all the features. It is easy to see that the all-subset kernel and the monotone DNF kernel are actually the same kernel up to the constant $-1$, i.e., ${\kappa }_{\subseteq }(\mathit{x},\mathit{z})={\kappa }_{\mathrm{mdnf}}(\mathit{x},\mathit{z})+1$.

Both the polynomial and the all-subsets kernel have limited control of which features they use and how they are weighted. The polynomial kernel uses only monomials of degree up to p with a weighting scheme depending on a parameter (c). The all-subsets, instead, makes use of the monomials corresponding to all possible subsets of the n input variables.

A restricted version of the all-subset kernel is the ANOVA kernel [11] in which the embedding space is formed by monomials with a fixed degree d without repetition. For example, given $\mathit{x}\in {\{0,1\}}^3$ the feature space of the all-subset kernel would be made by the features ${\mathit{x}}_1,{\mathit{x}}_2,{\mathit{x}}_3,{\mathit{x}}_1{\mathit{x}}_2,{\mathit{x}}_1{\mathit{x}}_3,{\mathit{x}}_2{\mathit{x}}_3,{\mathit{x}}_1{\mathit{x}}_2{\mathit{x}}_3$ and ∅, while for the ANOVA kernel of degree 2 it would be composed by ${\mathit{x}}_1{\mathit{x}}_2,{\mathit{x}}_1{\mathit{x}}_3$ and ${\mathit{x}}_2{\mathit{x}}_3$. Formally, the ANOVA kernel is defined as follows

$${\kappa }_{\mathrm{A}}^d(\mathit{x},\mathit{z})=\sum _{1\le i_1<i_2<\dots <i_d\le n}\prod _{j=1}^d{\mathit{x}}_{i_j}{\mathit{z}}_{i_j},$$

where $i_1,i_2,\dots ,i_d$ are all the possible sets of indices of cardinality d, taken from the set $\{1,\dots ,n\}$.

In [13,14], Boolean kernels are used for studying the learnability of logical formulas, specifically DNF formulas, using maximum margin algorithms, such as SVM. In particular, in [14], the authors showed the learning limitations of some Boolean kernels inside the PAC (Probably Approximately Correct) framework. Moreover, in [13], Kowalczyk at al. proposed a generalization of the Sadohara mDNF kernel. A special case of this kernel is represented by the $\sigma$-mDNF kernel.

From the application point of view, Boolean kernels have been successfully applied on many learning tasks, such as face recognition [15,16], spam filtering [17], load forecasting [18], and generic binary classification tasks [8,10].

3. A New Family of Boolean Kernels

In this section, we propose a new family of Boolean kernels which owns the characteristic of creating feature spaces that are very easy to understand, since they are based on logic. Specifically, features are logical formulas (of a fixed form) over the input Boolean variables.

Firstly, we present the most basic Boolean kernel and then, for both the monotone and the non-monotone cases, we propose kernels which mimic the conjunctive operator (and) and the disjunctive operator (or). Then, we provide an efficient way to combine these “base” Boolean kernels to obtain more complex ones, such as kernels with feature spaces composed by normal form formulas.

Throughout the paper, unless specified otherwise, we refer to vectors $\mathit{x},\mathit{z}\in {\{0,1\}}^n$ as binary (Boolean) vectors of dimension $n\in {\mathbb{N}}_{+}$. We also use $\mathcal{X}\equiv \left\{i\right|x_i=1\}$ and $\mathcal{Z}\equiv \left\{i\right|z_i=1\}$ as the sets interpretation of those vectors, while the set $\mathcal{U}\equiv \{1,\dots ,n\}$ indicates the universal set. Given a set $\mathcal{A}$, we refer to its i-th element with ${\mathcal{A}}_i$ for some enumeration of the elements of $\mathcal{A}$. With the notation ${\left[\mathcal{A}\right]}^k\equiv \{\mathcal{S}\subseteq \mathcal{A}\mid |\mathcal{S}|=k\}$, we express the set of all the subsets of $\mathcal{A}$ of cardinality k. It is worth noticing that, for any binary vector $\mathit{x}$, ${\parallel \mathit{x}\parallel }_2^2={\parallel \mathit{x}\parallel }_1$ holds, which is the number of ones contained in it. For the sake of brevity, we refer to this quantity with $\left|\mathit{x}\right|$. Moreover, ${\mathbf{1}}_n$ denotes the n-dimensional vector with all entries equal to 1, and with the notation $\langle ·,·\rangle$ we refer to the dot product. The symbol ⊙ denotes the entry-wise multiplication between matrices. Finally, given a function $f:\mathcal{X}\to {\mathcal{Y}}^n$, $\mathcal{Y}\subseteq \mathbb{R}$, then $\left|f\right|$ denotes the dimension of its codomain, i.e., n.

For each of the proposed kernels, the embedding function is provided in the general form $\varphi :\mathit{x}\mapsto {\varphi }^{\left(b\right)}{\left(\mathit{x}\right)}_{b\in \mathbb{B}}$, where $\mathbb{B}$ is a set of Boolean functions (formulas) over variables of $\mathit{x}$ such that ${\varphi }^{\left(b\right)}\left(\mathit{x}\right)=b\left(\mathit{x}\right)$ is a truth value associated with the application of the formula b to $\mathit{x}$. For example, let $b\left(\mathit{x}\right)=x_1\wedge x_2$ and $\mathbf{v}=[0,1,0,1]$, then $b\left(\mathbf{v}\right)=0$, that is false. For the sake of simplicity, for each new kernel, only the set $\mathbb{B}$ from which the Boolean formulas are taken is defined.

3.1. Monotone Boolean Kernels

A Boolean function (or formula) $f:{\{0,1\}}^n\to \{0,1\}$ is called monotone if replacing a 0 (i.e., false) with a 1 (i.e., true) in the input can only increase f’s value, i.e., the truth value can only change from false to true. In other words, a formula f is monotone if it does not have any not operator.

3.1.1. Monotone Literal Kernel

In logic, a literal is an atomic formula or its negation. Here, we are in the monotone setting, so we refer to a literal only in its positive form. In this case, the embedding is formed by Boolean literals taken from the set $\mathbb{B}\equiv \{f_i\mid f_i\left(\mathit{x}\right)=x_i\}$. Hence, the monotone Literal (mL) kernel, ${\kappa }_{\mathrm{mL}}(\mathit{x},\mathit{z})$, will count how many true (i.e., positive) input literals the vectors have in common. Actually, ${\kappa }_{\mathrm{mL}}$ is exactly the linear kernel ${\kappa }_{\mathrm{LIN}}(\mathit{x},\mathit{z})=\langle \mathit{x},\mathit{z}\rangle$, which simply performs the dot product between the two input binary vectors.

3.1.2. Monotone Conjunctive Kernel

In Boolean algebra, given two Boolean variables $x,z\in \{0,1\}$, the conjunction (i.e., and) between x and z, denoted by $x\wedge z$, is satisfied if and only if $x=z=1$, that is if and only if both variables are true. Given two vectors $\mathit{x},\mathit{z}\in {\{0,1\}}^n$, the monotone Conjunctive (mC) kernel [19] ${\kappa }_{\mathrm{mC}}^c(\mathit{x},\mathit{z})$ counts how many monotone conjunctions of the input variables, of a fixed arity c, are satisfied in both $\mathit{x}$ and $\mathit{z}$. In particular, the embedding is defined by Boolean formulas taken from $\mathbb{B}\equiv \{f_i\mid f_i\left(\mathit{x}\right)={\bigwedge }_{j\in {\left[\mathcal{U}\right]}_i^c}{x}_j\}$, which represent all the possible conjunctions of c literals (i.e., variables) taken from $\mathit{x}$. The dimension of the resulting feature space is $\left(\genfrac{}{}{0pt}{}{n}{c}\right)$, that is the number of all the combinations of c different variables taken from the input n-dimensional space. To count all the possible conjunctions of c variables satisfied in both $\mathit{x}$ and $\mathit{z}$, we need to calculate the number of combinations of c monomials that can be formed by using all the positive variables in both the vectors, that is the value of the kernel ${\kappa }_{\mathrm{mL}}(\mathit{x},\mathit{z})$. Hence, we obtain:

$${\kappa }_{\mathrm{mC}}^c(\mathit{x},\mathit{z})=\left(\begin{array}{c}{\kappa }_{\mathrm{mL}}(\mathit{x},\mathit{z})\\ c\end{array}\right)=\left(\begin{array}{c}\langle \mathit{x},\mathit{z}\rangle \\ c\end{array}\right).$$

It is easy to see that for binary vectors ${\kappa }_{\mathrm{mC}}^c$ is actually the ANOVA kernel of degree c [11].

As shown in [19], we can express the Sadohara mDNF Kernel [3] as a linear combination of mC-kernels of arity $1\le d\le n$ as in the following:

$${\kappa }_{\mathrm{mdnf}}(\mathit{x},\mathit{z})=2^{\langle \mathit{x},\mathit{z}\rangle }-1=\sum _{d=0}^n\left(\begin{array}{c}\langle \mathit{x},\mathit{z}\rangle \\ d\end{array}\right)-1=\sum _{d=1}^n{\kappa }_{\mathrm{mC}}^d(\mathit{x},\mathit{z}).$$

A similar construction also holds for the d-mDNF kernel.

3.1.3. Monotone Disjunctive Kernel

The disjunction of two Boolean variables $x,z\in \{0,1\}$, denoted by $x\vee z$, is not satisfied if and only if $x=z=0$, that is if and only if both variables are false, or, in other words, it is satisfied anytime at least one of the variables is true. The monotone Disjunctive (mD) kernel [19], ${\kappa }_{\mathrm{mD}}^d(\mathit{x},\mathit{z})$ counts how many monotone disjunctions, of a fixed arity d, are satisfied in both $\mathit{x}$ and $\mathit{z}$. The embedding of this kernel is defined by Boolean formulas taken from $\mathbb{B}\equiv \{f_i\mid f_i\left(\mathit{x}\right)={\bigvee }_{j\in {\left[\mathcal{U}\right]}_i^c}{x}_j\}$, with a feature space of dimension $\left(\genfrac{}{}{0pt}{}{n}{d}\right)$. To explain how to count the common positive disjunctions in both $\mathit{x}$ and $\mathit{z}$, we can rely on the analogy between binary vectors and sets. An active disjunction of d literals for $\mathcal{X}$ can be defined as a set of d elements taken from the universe $\mathcal{U}$, let us call it ${\mathcal{U}}_d$, such that $\exists a\in {\mathcal{U}}_d|a\in \mathcal{X}$. Anytime $\exists a,b\in {\mathcal{U}}_d\mid a\in \mathcal{X}\wedge b\in \mathcal{Z}$ (potentially $a=b$), then ${\mathcal{U}}_d$ is an active subset for $\mathcal{X}$ and $\mathcal{Z}$. Using this interpretation, we can say that the value of the kernel is the number of active subsets ${\mathcal{U}}_d$ in common between $\mathcal{X}$ and $\mathcal{Z}$. We can count the number of these subsets in a negative fashion. Starting from the number of all possible subsets, which is $(\genfrac{}{}{0pt}{}{\left|\mathcal{U}\right|}{d})$, we remove the inactive subsets for $\mathcal{X}$ and for $\mathcal{Z}$. An inactive subset for $\mathcal{X}$ is a set such that it does not contain any element of $\mathcal{X}$, and the number of this kind of sets is $(\genfrac{}{}{0pt}{}{|\mathcal{U}\setminus \mathcal{X}|}{d})$. Analogously, we can do the same for $\mathcal{Z}$. Now, we have removed twice the subsets formed by elements taken from $\overline{\mathcal{X}\cup \mathcal{Z}}\equiv \mathcal{U}\setminus (\mathcal{X}\cup \mathcal{Z})$ and hence we need to add its contribution once, that is $(\genfrac{}{}{0pt}{}{|\mathcal{U}\setminus (\mathcal{X}\cup \mathcal{Z}\left)\right|}{d})$. We can now define ${\kappa }_{\mathrm{mD}}^d$ as

$$\begin{array}{cc}\hfill {\kappa }_{\mathrm{mD}}^d(\mathit{x},\mathit{z})& =\left(\genfrac{}{}{0pt}{}{\left|\mathcal{U}\right|}{d}\right)-\left(\genfrac{}{}{0pt}{}{|\mathcal{U}\setminus \mathcal{X}|}{d}\right)-\left(\genfrac{}{}{0pt}{}{|\mathcal{U}\setminus \mathcal{Z}|}{d}\right)+\left(\genfrac{}{}{0pt}{}{|\mathcal{U}\setminus (\mathcal{X}\cup \mathcal{Z}\left)\right|}{d}\right)\hfill \\ & =\left(\genfrac{}{}{0pt}{}{n}{d}\right)-\left(\genfrac{}{}{0pt}{}{n-\langle \mathit{x},\mathit{x}\rangle }{d}\right)-\left(\genfrac{}{}{0pt}{}{n-\langle \mathit{z},\mathit{z}\rangle }{d}\right)+\left(\genfrac{}{}{0pt}{}{n-\langle \mathit{x},\mathit{x}\rangle -\langle \mathit{z},\mathit{z}\rangle +\langle \mathit{x},\mathit{z}\rangle }{d}\right).\hfill \end{array}$$

(1)

3.2. Non-Monotone Boolean Kernels

Converse to the monotone case, non-monotone Boolean formulas can contain negated variables, e.g., $\neg x_i$, thus the mL-kernel is not expressive enough to be the simplest non-monotone Boolean kernel because it does not consider negated variables.

3.2.1. Non-Monotone Literal Kernel

To include the contribution of the negated variables, we need to add the number of false variables in common between $\mathit{x}$ and $\mathit{z}$ to the mL-kernel. This can be calculated by the negation kernel, defined as

$$\begin{array}{cc}\hfill {\kappa }_{\mathrm{NEG}}(\mathit{x},\mathit{z})& =\langle ({\mathbf{1}}_n-\mathit{x}),({\mathbf{1}}_n-\mathit{z})\rangle \hfill \\ & =n-\langle \mathit{x},\mathit{x}\rangle -\langle \mathit{z},\mathit{z}\rangle +\langle \mathit{x},\mathit{z}\rangle ,\hfill \end{array}$$

in which the embedding is defined by Boolean functions taken from $\mathbb{B}\equiv \{f_i\mid f_i\left(\mathit{x}\right)=\neg x_i\}$. Hence, the non-monotone Literal (L) kernel, ${\kappa }_L(\mathit{x},\mathit{z})$, counts how many true and false variables $\mathit{x}$ and $\mathit{z}$ have in common, and it is defined as a sum of kernels [11] as in the following:

$$\begin{array}{cc}\hfill {\kappa }_L(\mathit{x},\mathit{z})& ={\kappa }_{\mathrm{mL}}(\mathit{x},\mathit{z})+{\kappa }_{\mathrm{NEG}}(\mathit{x},\mathit{z})\hfill \\ & =n-\langle \mathit{x},\mathit{x}\rangle -\langle \mathit{z},\mathit{z}\rangle +2\langle \mathit{x},\mathit{z}\rangle .\hfill \end{array}$$

3.2.2. Non-Monotone Conjunctive Kernel

The non-monotone Conjunctive (C) kernel counts how many non-monotone conjunctions of a certain arity c are satisfied in both $\mathit{x}$ and $\mathit{z}$. The embedding is defined by boolean functions in the set $\mathbb{B}$ of all the non-monotone conjunctions of c literals. Formally, given the set ${\mathcal{U}}_c^{\ast }\equiv \{\mathcal{S}\subseteq \{1,\dots ,2n\}\mid |\mathcal{S}|=c,\nexists i,j\in \mathcal{S}\mathrm{s}.\mathrm{t}.i=2j\}$ and the function $g(\mathit{x},i)={\mathit{x}}_i$ if $i\le n$ or $g(\mathit{x},i)=\neg {\mathit{x}}_i$ otherwise, then $\mathbb{B}\equiv \{f_i\mid f_i\left(\mathit{x}\right)={\bigwedge }_{j\in {\left[{\mathcal{U}}_c^{\ast }\right]}_i}g(\mathit{x},j)\}$. Since we are considering conjunctions of variables, this kernel will count how many combinations of (possibly negated) common variables there are between $\mathit{x}$ and $\mathit{z}$. Thus, relying on these considerations and on the definition of ${\kappa }_L$, we can finally define the ${\kappa }_{\mathrm{C}}^c(\mathit{x},\mathit{z})$ as follows:

$${\kappa }_{\mathrm{C}}^c(\mathit{x},\mathit{z})=\left(\genfrac{}{}{0pt}{}{{\kappa }_L(\mathit{x},\mathit{z})}{c}\right)=\left(\genfrac{}{}{0pt}{}{n-\langle \mathit{x},\mathit{x}\rangle -\langle \mathit{z},\mathit{z}\rangle +2\langle \mathit{x},\mathit{z}\rangle }{c}\right).$$

Similar to the monotone case, we can express the Sadohara DNF Kernel [3] as a linear combination of C-kernels of arity $1\le d\le n$ as in the following:

$$\begin{array}{cc}\hfill {\kappa }_{\mathrm{dnf}}(\mathit{x},\mathit{z})& =2^{\langle \mathit{x},\mathit{z}\rangle +\langle \overline{\mathit{x}},\overline{\mathit{z}}\rangle }-1=\sum _{i=0}^n\left(\genfrac{}{}{0pt}{}{\langle \mathit{x},\mathit{z}\rangle +\langle \overline{\mathit{x}},\overline{\mathit{z}}\rangle }{i}\right)-1=\sum _{i=0}^n\left(\genfrac{}{}{0pt}{}{{\kappa }_{\mathrm{mL}}(\mathit{x},\mathit{z})+{\kappa }_{\mathrm{NEG}}(\mathit{x},\mathit{z})}{i}\right)-1\hfill \\ & =\sum _{i=0}^n\left(\genfrac{}{}{0pt}{}{{\kappa }_L(\mathit{x},\mathit{z})}{d}\right)-1=\sum _{i=1}^n{\kappa }_{\mathrm{C}}^d(\mathit{x},\mathit{z}).\hfill \end{array}$$

An analogous construction also holds for the d-DNF kernel.

3.2.3. Non-Monotone Disjunctive Kernel

The non-monotone Disjunctive (D) kernel counts how many non-monotone disjunctions, of a certain arity d, are satisfied in both $\mathit{x}$ and $\mathit{z}$. The embedding is defined by Boolean formulas in the set $\mathbb{B}$ of all the non-monotone disjunctions of d literals. Formally, $\mathbb{B}\equiv \{f_i\mid f_i\left(\mathit{x}\right)={\bigvee }_{j\in {\left[{\mathcal{U}}_d^{\ast }\right]}_i}g(\mathit{x},j)\}$, with ${\mathcal{U}}_d^{\ast }$ and g defined as in the previous section. As for the monotone case, we derive the kernel function in a negative fashion. The number of every possible combination of arity d of variables that can be also in their negated form is $2^d\left(\genfrac{}{}{0pt}{}{n}{d}\right)$. For both $\mathit{x}$ and $\mathit{z}$, we have to discard the combinations that are false, which are exactly $\left(\genfrac{}{}{0pt}{}{n}{d}\right)$ because for each set of d different variables, there exists only one assignment of the negations such that the disjunction is false. For example, given the variables $x_1=1,x_2=0$ and $x_3=1$, only the disjunction $\neg x_1\vee x_2\vee \neg x_3$ is false and all the others $2^3-1$ negation assignments are true. Then, we have to re-add the false combinations that have been discarded twice, that is the combinations made with variables that are false in both the vectors. This can be seen as the opposite of what C-kernel computes, but, since we generate all the possible combinations with all the possible negation assignments, the counting is actually the same as the C-kernel. Finally, we can define the ${\kappa }_{\mathrm{D}}^d(\mathit{x},\mathit{z})$ as

$${\kappa }_{\mathrm{D}}^d(\mathit{x},\mathit{z})=(2^d-2)\left(\genfrac{}{}{0pt}{}{n}{d}\right)+{\kappa }_{\mathrm{C}}^d(\mathit{x},\mathit{z})$$

3.3. Boolean Kernels Combination

Given the Boolean kernels defined in the previous sections, we have now all the basic elements to build new Boolean kernels that represent a specific Boolean concept.

Table 1. Summary of the just presented Boolean kernels: $\mathit{x},\mathit{z}\in {\{0,1\}}^n$ and $\left|\varphi \right|$ stands for the dimension of the feature space.

$\mathit{\kappa }$	$\mathit{\kappa }(\mathit{x},\mathit{z})$	$\mathit{\kappa }(\mathit{x},\mathit{x})$	$\left|\mathit{\varphi }\right|$
${\kappa }_{\mathrm{mL}}$	$\langle \mathit{x},\mathit{z}\rangle$	$\left|\mathit{x}\right|$	n
${\kappa }_{\mathrm{mC}}^c$	$\left(\genfrac{}{}{0pt}{}{{\kappa }_{\mathrm{mL}}(\mathit{x},\mathit{z})}{c}\right)$	$\left(\genfrac{}{}{0pt}{}{\left|\mathit{x}\right|}{c}\right)$	$\left(\genfrac{}{}{0pt}{}{n}{c}\right)$
${\kappa }_{\mathrm{mD}}^d$	$\left(\genfrac{}{}{0pt}{}{n}{d}\right)-\left(\genfrac{}{}{0pt}{}{n-\left|\mathit{x}\right|}{d}\right)-\left(\genfrac{}{}{0pt}{}{n-\left|\mathit{z}\right|}{d}\right)+\left(\genfrac{}{}{0pt}{}{n-\left|\mathit{x}\right|-\left|\mathit{z}\right|+\langle \mathit{x},\mathit{z}\rangle }{d}\right)$	$\left(\genfrac{}{}{0pt}{}{n}{d}\right)-\left(\genfrac{}{}{0pt}{}{n-\left|\mathit{x}\right|}{d}\right)$	$\left(\genfrac{}{}{0pt}{}{n}{d}\right)$
${\kappa }_{\mathrm{NEG}}$	$n-\left|\mathit{x}\right|-\left|\mathit{z}\right|+\langle \mathit{x},\mathit{z}\rangle$	$n-\left|\mathit{x}\right|$	n
${\kappa }_L$	${\kappa }_{\mathrm{mL}}+{\kappa }_{\mathrm{NEG}}$	n	$2n$
${\kappa }_{\mathrm{C}}^c$	$\left(\genfrac{}{}{0pt}{}{{\kappa }_L(\mathit{x},\mathit{z})}{c}\right)$	$\left(\genfrac{}{}{0pt}{}{n}{c}\right)$	$2^c\left(\genfrac{}{}{0pt}{}{n}{c}\right)$
${\kappa }_{\mathrm{D}}^d$	$(2^d-2)\left(\genfrac{}{}{0pt}{}{n}{d}\right)+{\kappa }_{\mathrm{C}}^d(\mathit{x},\mathit{z})$	$(2^d-1)\left(\genfrac{}{}{0pt}{}{n}{d}\right)$	$2^d\left(\genfrac{}{}{0pt}{}{n}{d}\right)$

shows a summary of all the presented Boolean kernels. It is easy to see that all the kernels are function of dot products of the input vectors, and this allows us to create new kernels by replacing those dot products with other Boolean kernels. The logical interpretation of the new kernel depends on how the base kernels are combined. In the following, we present some new Boolean kernels generated by using the above mentioned method.

3.3.1. DNF Kernels

A disjunctive normal form (DNF) is a normalization of a logical formula which is a disjunction of conjunctive clauses. In the monotone case, both conjunctive and disjunctive clauses are monotone, i.e., the literals are only in their positive form. Since DNFs are disjunctions of conjunctive clauses, we can combine the embedding maps of the mC-kernel and the mD-kernel in this way ${\varphi }_{\mathrm{mDNF}}^{d,c}:\mathit{x}\mapsto {\varphi }_{\mathrm{mD}}^d\left({\varphi }_{\mathrm{mC}}^c\left(\mathit{x}\right)\right),$ obtaining the desired feature space for the monotone DNF, which leads to the definition of the mDNF-kernel as in the following:

$${\kappa }_{\mathrm{mDNF}}^{d,c}(\mathit{x},\mathit{z})=\left(\genfrac{}{}{0pt}{}{\left(\genfrac{}{}{0pt}{}{n}{c}\right)}{d}\right)-\left(\genfrac{}{}{0pt}{}{\left(\genfrac{}{}{0pt}{}{n}{c}\right)-{\kappa }_{\mathrm{mC}}^c(\mathit{x},\mathit{x})}{d}\right)-\left(\genfrac{}{}{0pt}{}{\left(\genfrac{}{}{0pt}{}{n}{c}\right)-{\kappa }_{\mathrm{mC}}^c(\mathit{z},\mathit{z})}{d}\right)+\left(\genfrac{}{}{0pt}{}{\left(\genfrac{}{}{0pt}{}{n}{c}\right)-{\kappa }_{\mathrm{mC}}^c(\mathit{x},\mathit{x})-{\kappa }_{\mathrm{mC}}^c(\mathit{z},\mathit{z})+{\kappa }_{\mathrm{mC}}^c(\mathit{x},\mathit{z})}{d}\right).$$

Note that we need to know the dimension of the feature space of the mC-kernel. The features of this kernel are actually monotone DNF formulas with exactly d disjunctions of conjunctive clauses of arity c. In the non-monotone case, we proceed in a similar way but, since the conjunctive clauses are non-monotone, we have to use the C-kernel instead of the mC-kernel. So, the feature space can be created by composing the embedding map of the mD-kernel with the one of the C-kernel, that is, ${\varphi }_{\mathrm{DNF}}^{d,c}:\mathit{x}\mapsto {\varphi }_{\mathrm{mD}}^d\left({\varphi }_{\mathrm{C}}^c\left(\mathit{x}\right)\right).$ Consequently, the DNF-kernel is defined as follows:

$${\kappa }_{\mathrm{DNF}}^{d,c}(\mathit{x},\mathit{z})=\left(\genfrac{}{}{0pt}{}{2^c\left(\genfrac{}{}{0pt}{}{n}{c}\right)}{d}\right)-\left(\genfrac{}{}{0pt}{}{2^c\left(\genfrac{}{}{0pt}{}{n}{c}\right)-{\kappa }_{\mathrm{C}}^c(\mathit{x},\mathit{x})}{d}\right)-\left(\genfrac{}{}{0pt}{}{2^c\left(\genfrac{}{}{0pt}{}{n}{c}\right)-{\kappa }_{\mathrm{C}}^c(\mathit{z},\mathit{z})}{d}\right)+\left(\genfrac{}{}{0pt}{}{2^c\left(\genfrac{}{}{0pt}{}{n}{c}\right)-{\kappa }_{\mathrm{C}}^c(\mathit{x},\mathit{x})-{\kappa }_{\mathrm{C}}^c(\mathit{z},\mathit{z})+{\kappa }_{\mathrm{C}}^c(\mathit{x},\mathit{z})}{d}\right).$$

3.3.2. CNF Kernels

A logic formula is in conjunctive normal form (CNF) if it is composed of conjunctions of disjunctive clauses. Clearly, in its monotone form it does not contain any negated literal. By using a similar approach as for the mDNF-kernel, the feature space is defined by the function ${\varphi }_{\mathrm{mCNF}}^{d,c}:\mathit{x}\mapsto {\varphi }_{\mathrm{mC}}^d\left({\varphi }_{\mathrm{mD}}^c\left(\mathit{x}\right)\right),$ and hence in the kernel function we replace the dot products inside the mC-kernel with the mD-kernel as in the following:

$${\kappa }_{\mathrm{mCNF}}^{d,c}(\mathit{x},\mathit{z})=\left(\genfrac{}{}{0pt}{}{{\kappa }_{\mathrm{mD}}^d(\mathit{x},\mathit{z})}{c}\right).$$

The resulting feature space is composed of monotone CNF formulas with exactly c conjunctions of disjunctive clauses of arity d. By swapping the mD-kernel with the D-kernel, we can easily obtain the non-monotone CNF kernel:

$${\kappa }_{\mathrm{CNF}}(\mathit{x},\mathit{z})=\left(\genfrac{}{}{0pt}{}{{\kappa }_{\mathrm{D}}^d(\mathit{x},\mathit{z})}{c}\right),$$

which has associated the embedding function ${\varphi }_{\mathrm{CNF}}^{d,c}:\mathit{x}\mapsto {\varphi }_{\mathrm{mC}}^d\left({\varphi }_{\mathrm{D}}^c\left(\mathit{x}\right)\right).$

For the sake of brevity, in the rest of the paper, we indicate the mDNF-kernel having d disjunctions and c conjunctions with either the notation mDNF(d,c)-kernel or simply mDNF(d,c).

4. Computational Complexity

The computational complexity of the Boolean kernels described in the previous sections is bounded by the complexity of the calculation of the binomial coefficient, that is $\mathcal{O}\left(k\right)$ with k the arity of the combinations. Hence, computing an entire kernel matrix over n-dimensional examples taken from a dataset with l examples would lead to a complexity of $\mathcal{O}((k+n)·l^2)$. Even though it is not possible to reduce such complexity, we can take advantage of the recursive nature of the binomial coefficient to compute the kernels in a recursive fashion. By doing so, it is possible to compute higher degree kernel matrices by leveraging on kernel matrices of lower degrees.

Let the matrix ${\mathit{K}}^0$ be the base (Boolean) kernel matrix over the dataset $\mathcal{D}$, such that ${\mathit{K}}_{i,j}^0=\langle \varphi \left({\mathit{x}}_i\right),\varphi \left({\mathit{x}}_j\right)\rangle$, for ${\mathit{x}}_i,{\mathit{x}}_j\in \mathcal{D}$ and some $\varphi$ with codomain ${\{0,1\}}^n$. Then, we can recursively define the Boolean kernels for both monotone and the non-monotone case as described in the following.

4.1. mC-Kernel

By definition, the mC-kernel matrix of arity 1, that is ${\mathit{K}}_{\mathrm{mC}}^1$, is equivalent to the base kernel matrix ${\mathit{K}}^0$. By using ${\mathit{K}}^0$ as base case, we can recursively define the mC-kernel matrix as

$${\mathit{K}}_{\mathrm{mC}}^{c+1}={\mathit{K}}_{\mathrm{mC}}^c\odot \left(\frac{1}{c+1}\left({\mathit{K}}_{\mathrm{mC}}^1-c{\mathbf{1}}_l{\mathbf{1}}_l^{⊺}\right)\right).$$

4.2. mD-Kernel

Let us define the matrices

$$\mathbf{S}=\mathrm{diag}\left({\mathit{K}}^0\right)·{\mathbf{1}}_l^{⊺},{\mathbf{N}}_x=n{\mathbf{1}}_l{\mathbf{1}}_l^{⊺}-\mathbf{S},\mathrm{and}{\mathbf{N}}_{xz}={\mathbf{N}}_x-{\mathbf{S}}^{⊺}+{\mathit{K}}^0.$$

Then, we define, recursively in its parts, the mD-kernel matrix ${\mathit{K}}_{\mathrm{mD}}^d$ as

$${\mathit{K}}_{\mathrm{mD}}^d={\mathbf{N}}^{\left(d\right)}-{\mathbf{N}}_x^{\left(d\right)}-{{\mathbf{N}}_x^{\left(d\right)}}^{⊺}+{\mathbf{N}}_{xz}^{\left(d\right)},$$

where

$$\begin{array}{cc}\hfill {\mathbf{N}}^{(d+1)}& ={\mathbf{N}}^{\left(d\right)}\odot \left(\frac{n-d}{d+1}{\mathbf{1}}_l{\mathbf{1}}_l^{⊺}\right),\hfill \\ \hfill {\mathbf{N}}_x^{(d+1)}& ={\mathbf{N}}_x^{\left(d\right)}\odot \left(\frac{1}{d+1}\left({\mathbf{N}}_x-d{\mathbf{1}}_l{\mathbf{1}}_l^{⊺}\right)\right),\mathrm{and}\hfill \\ \hfill {\mathbf{N}}_{xz}^{(d+1)}& ={\mathbf{N}}_{xz}^{\left(d\right)}\odot \left(\frac{1}{d+1}\left({\mathbf{N}}_{xz}-d{\mathbf{1}}_l{\mathbf{1}}_l^{⊺}\right)\right),\hfill \end{array}$$

with the corresponding base cases ${\mathbf{N}}^{\left(1\right)}=n{\mathbf{1}}_l{\mathbf{1}}_l^{⊺}$, ${\mathbf{N}}_x^{\left(1\right)}={\mathbf{N}}_x$ and ${\mathbf{N}}_{xz}^{\left(1\right)}={\mathbf{N}}_{xz}$.

4.3. C-Kernel

By relying on the previous definition of $\mathbf{S}$ and the base case kernel matrix

$${\mathit{K}}_{\mathrm{C}}^1=n{\mathbf{1}}_l{\mathbf{1}}_l^{⊺}-\mathbf{S}-{\mathbf{S}}^{⊺}+2{\mathit{K}}^0,$$

the C-kernel matrix can be recursively defined by

$${\mathit{K}}_{\mathrm{C}}^{c+1}={\mathit{K}}_{\mathrm{C}}^c\odot \left(\frac{1}{c+1}\left({\mathit{K}}_{\mathrm{C}}^1-c{\mathbf{1}}_l{\mathbf{1}}_l^{⊺}\right)\right).$$

4.4. D-Kernel

Using the previous definitions of the matrices ${\mathbf{N}}^{\left(d\right)}$ and ${\mathit{K}}_{\mathrm{C}}^c$, we can define the D-kernel matrix, recursively in its parts, as

$${\mathit{K}}_{\mathrm{D}}^{d+1}=\left((2^d-2){\mathbf{1}}_l{\mathbf{1}}_l^{⊺}\right)\odot {\mathbf{N}}^{\left(d\right)}-{\mathit{K}}_{\mathrm{C}}^d.$$

Both the standard and the recursive definition have been implemented in a freely available Python module pyros available at the following URL: https://github.com/makgyver/pyros.

5. Evaluation

5.1. Evaluation Protocol

In all experiments, the kernels have been normalized using the well-known formula

$$\tilde{\kappa }(\mathit{x},\mathit{z})=\frac{\kappa (\mathit{x},\mathit{z})}{\sqrt{\kappa (\mathit{x},\mathit{x})\kappa (\mathit{z},\mathit{z})}}.$$

We assessed the proposed Boolean kernels by using SVM as kernel machine and we compared them with the linear kernel, the RBF kernel, the (monotone) DNF kernel proposed by Sadohara et al. [3], the d-DNF kernel [9], the $\sigma$-mDNF kernel [10] and the Tanimoto kernel.

Both validation and test were evaluated in terms of the Area Under the ROC Curve (AUC). The used validation method is a five-fold nested cross validation. For each dataset, the test was repeated 10 times and the average performances were recorded. Specifically, we validated the misclassification cost parameter C for the SVM in the set $\{2^{-4},2^{-3},\dots ,2^4\}$; for each of the proposed kernels we validated both the conjunctive arity c (when applicable) and the disjunctive arity d (when applicable) in the set $\{1,\dots ,5\}$, for the RBF kernel we validated the hyper-paramater $\gamma \in \{{10}^{-4},\dots ,{10}^4\}$, while for the d-mDNF we fixed $d=5$. Finally, for the $\sigma$-mDNF kernel we validated $\sigma$ in the set $\{0.2,0.5,1,2\}$. All the experiments were implemented in Python using the machine learning module Scikit-learn [20]. The source code is freely available at https://github.com/makgyver/pyros.

The benchmark datasets used for the experiments are reported in

Table 2. Datasets information: name, number of instances (# Examples), number of features (# Features), class distribution and number of active variables for example.

Dataset Name	# Examples	pos/neg (%)	# Features	$\mathit{m}={\parallel \mathit{x}\parallel }_1$
zoo	101	40/60	36	16
promoters	106	50/50	228	57
lymphography	148	45/55	44	15
house-votes	232	46/54	32	16
soybean	266	54/46	97	35
spect	267	79/21	44	22
breast	277	71/29	41	9
primary-tumor	339	41/59	34	15
monks-1	432	50/50	17	6
crx	653	55/45	40	9
tic-tac-toe	958	65/35	27	9
flare	1066	49/51	41	11
car	1728	30/70	21	6
dna_bin	2000	47/53	180	47
splice	3175	48/52	240	60
kr-vs-kp	3196	52/48	73	36

. These datasets are freely available from the UCI repository [21] and the KEEL repository [22]. We selected datasets with binary or categorical features and for each of them the following preprocessing steps were performed:

• Instances with missing attributes were removed.
• categorical features, including the binary ones, were mapped into binary features by means of the one-hot encoding [23]. This preprocessing keeps for every example in the dataset the same number of ones , or in other words every input vector has the same euclidean norm.
• Non-binary tasks were artificially transformed into binary ones, by arranging the classes into two groups while trying to keep the number of instances balanced.

5.2. Experimental Results

The first set of experiments assessed the quality of the proposed kernels. The average AUCs over 10 runs as well as the standard deviations are reported in

Table 3. AUC performances on benchmark datasets. For each dataset the best performing kernel is highlighted with both the boldface font and with a dot (·).

Dataset	mC	mD	C	D	mDNF	mCNF	DNF	CNF
breast	$\underset{\pm 1.27}{71.19}$	$\underset{\pm 1.51}{\mathbf{71.68}·}$	$\underset{\pm 1.87}{71.52}$	$\underset{\pm 1.80}{71.70}$	$\underset{\pm 1.73}{71.21}$	$\underset{\pm 2.07}{71.31}$	$\underset{\pm 1.82}{71.54}$	$\underset{\pm 1.90}{71.50}$
car_bin	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.08}{99.97}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.12}{99.62}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$
crx	$\underset{\pm 0.39}{91.77}$	$\underset{\pm 0.37}{92.04}$	$\underset{\pm 0.22}{\mathbf{92.11}·}$	$\underset{\pm 0.28}{91.99}$	$\underset{\pm 0.34}{91.81}$	$\underset{\pm 0.38}{91.86}$	$\underset{\pm 0.29}{92.08}$	$\underset{\pm 0.30}{92.03}$
dna_bin	$\underset{\pm 0.05}{99.03}$	$\underset{\pm 0.06}{98.98}$	$\underset{\pm 0.05}{\mathbf{99.07}·}$	$\underset{\pm 0.05}{98.96}$	$\underset{\pm 0.05}{\mathbf{99.07}·}$	$\underset{\pm 0.04}{99.01}$	$\underset{\pm 0.05}{\mathbf{99.07}·}$	$\underset{\pm 0.05}{99.06}$
flare_bin	$\underset{\pm 0.13}{94.80}$	$\underset{\pm 0.13}{94.91}$	$\underset{\pm 0.13}{94.91}$	$\underset{\pm 0.10}{94.91}$	$\underset{\pm 0.11}{94.88}$	$\underset{\pm 0.11}{94.89}$	$\underset{\pm 0.10}{\mathbf{94.92}·}$	$\underset{\pm 0.13}{94.90}$
house-votes-84	$\underset{\pm 0.26}{99.23}$	$\underset{\pm 0.28}{99.19}$	$\underset{\pm 0.25}{\mathbf{99.27}·}$	$\underset{\pm 0.31}{99.18}$	$\underset{\pm 0.28}{99.19}$	$\underset{\pm 0.29}{99.18}$	$\underset{\pm 0.30}{99.18}$	$\underset{\pm 0.32}{99.16}$
kr-vs-kp	$\underset{\pm 0.02}{99.95}$	$\underset{\pm 0.01}{99.74}$	$\underset{\pm 0.02}{99.94}$	$\underset{\pm 0.01}{99.74}$	$\underset{\pm 0.01}{\mathbf{99.96}·}$	$\underset{\pm 0.01}{\mathbf{99.96}·}$	$\underset{\pm 0.01}{\mathbf{99.96}·}$	$\underset{\pm 0.01}{\mathbf{99.96}·}$
lymphography_bin	$\underset{\pm 1.45}{92.45}$	$\underset{\pm 1.09}{92.81}$	$\underset{\pm 1.10}{92.77}$	$\underset{\pm 1.23}{92.73}$	$\underset{\pm 1.33}{92.40}$	$\underset{\pm 1.25}{92.67}$	$\underset{\pm 1.13}{92.86}$	$\underset{\pm 1.16}{\mathbf{93.03}·}$
monks-1	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 1.40}{91.52}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 1.86}{87.75}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$
primary-tumor	$\underset{\pm 0.61}{72.88}$	$\underset{\pm 0.56}{72.92}$	$\underset{\pm 0.79}{72.80}$	$\underset{\pm 0.58}{72.72}$	$\underset{\pm 0.54}{\mathbf{72.97}·}$	$\underset{\pm 0.51}{72.95}$	$\underset{\pm 0.82}{72.70}$	$\underset{\pm 0.61}{72.86}$
promoters	$\underset{\pm 1.06}{97.51}$	$\underset{\pm 0.92}{97.39}$	$\underset{\pm 0.87}{97.39}$	$\underset{\pm 0.87}{97.28}$	$\underset{\pm 1.09}{\mathbf{97.57}·}$	$\underset{\pm 1.13}{97.49}$	$\underset{\pm 0.89}{97.43}$	$\underset{\pm 0.88}{97.45}$
soybean	$\underset{\pm 0.08}{99.66}$	$\underset{\pm 0.09}{99.64}$	$\underset{\pm 0.09}{\mathbf{99.69}·}$	$\underset{\pm 0.07}{99.66}$	$\underset{\pm 0.10}{99.66}$	$\underset{\pm 0.09}{99.65}$	$\underset{\pm 0.07}{99.68}$	$\underset{\pm 0.06}{\mathbf{99.69}·}$
spect	$\underset{\pm 2.07}{83.60}$	$\underset{\pm 2.08}{83.58}$	$\underset{\pm 1.98}{83.79}$	$\underset{\pm 2.06}{83.75}$	$\underset{\pm 2.04}{83.61}$	$\underset{\pm 2.04}{83.63}$	$\underset{\pm 2.00}{\mathbf{83.93}·}$	$\underset{\pm 2.08}{83.81}$
splice	$\underset{\pm 0.04}{\mathbf{99.35}·}$	$\underset{\pm 0.04}{99.20}$	$\underset{\pm 0.02}{99.28}$	$\underset{\pm 0.04}{99.19}$	$\underset{\pm 0.03}{\mathbf{99.35}·}$	$\underset{\pm 0.04}{\mathbf{99.35}·}$	$\underset{\pm 0.02}{99.29}$	$\underset{\pm 0.02}{99.29}$
tic-tac-toe	$\underset{\pm 0.61}{98.03}$	$\underset{\pm 0.43}{97.86}$	$\underset{\pm 0.37}{\mathbf{98.25}·}$	$\underset{\pm 0.37}{\mathbf{98.25}·}$	$\underset{\pm 0.60}{98.03}$	$\underset{\pm 0.61}{98.03}$	$\underset{\pm 0.37}{\mathbf{98.25}·}$	$\underset{\pm 0.37}{\mathbf{98.25}·}$
zoo	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$
Rank	4.38	5.13	3.06	5.44	4.00	3.88	2.88	3.13

. The last row of each table summarizes the average rank achieved by all kernels over the benchmark datasets.

It is evident from the tables that normal form (NF) kernels perform on average better than the other Boolean kernels, with the only exception of the C-kernel on the AUC metric. This is reasonable since normal form kernels are a generalization of the (m)C-kernel and the (m)D-kernel. However, after the validation procedure, there is no guarantee that a NF kernel is always better than their correspondent base kernels, as underlined by the very good performance of the C-kernel. Another interesting observation is that both the D-kernel and the mD-kernel are almost always the worst performing ones, and this can be explained by the fact that they are less expressive than the competing kernels.

Since NF kernels have shown very good performances, we built, for each normal form, the average kernel over all the degrees, that is:

$${\kappa }_{\mathrm{avg}.\mathrm{NF}}=\frac{1}{C\times D}\sum _{d=1}^D\sum _{c=1}^C{\kappa }_{NF}^{d,c}(\mathit{x},\mathit{z}).$$

In this way, given a normal form, the feature space of the resulting kernel contains all the normal form formulas for each of the degrees $(c,d)$ with $1\le c\le C$ and $1\le d\le D$. Since we have fixed the maximum degrees C and D or all the kernels, and the ${\kappa }_{\mathrm{avg}.\mathrm{NF}}$ has no other hyper-parameters, it did not require any further validation.

The average AUCs over the 10 runs as well as the standard deviations are reported in

Table 4. AUC performances on benchmark datasets. For each dataset the best performing kernel is highlighted with both the boldface font and with a dot (·).

Dataset	Linear	RBF	DNF [3]	d-mDNF	Tanimoto	$\mathit{\sigma }$-mDNF	avg.mDNF	avg.mCNF	avg.DNF	avg.CNF
breast	$\underset{\pm 0.91}{70.69}$	$\underset{\pm 1.63}{71.21}$	$\underset{\pm 1.55}{71.56}$	$\underset{\pm 1.50}{71.82}$	$\underset{\pm 1.99}{\mathbf{72.32}·}$	$\underset{\pm 1.77}{71.24}$	$\underset{\pm 1.69}{70.09}$	$\underset{\pm 1.85}{72.16}$	$\underset{\pm 1.97}{72.18}$	$\underset{\pm 1.73}{72.03}$
car_bin	$\underset{\pm 0.13}{98.98}$	$\underset{\pm 0.07}{99.96}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.04}{99.82}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$
crx	$\underset{\pm 0.41}{91.08}$	$\underset{\pm 0.36}{92.03}$	$\underset{\pm 0.21}{91.96}$	$\underset{\pm 0.23}{91.82}$	$\underset{\pm 0.26}{\mathbf{92.14}·}$	$\underset{\pm 0.36}{91.99}$	$\underset{\pm 0.21}{91.78}$	$\underset{\pm 0.30}{91.96}$	$\underset{\pm 0.27}{92.02}$	$\underset{\pm 0.25}{92.09}$
dna_bin	$\underset{\pm 0.10}{98.21}$	$\underset{\pm 0.06}{98.84}$	$\underset{\pm 0.19}{97.25}$	$\underset{\pm 0.05}{98.50}$	$\underset{\pm 0.05}{98.92}$	$\underset{\pm 0.06}{98.52}$	$\underset{\pm 0.05}{98.51}$	$\underset{\pm 0.05}{99.04}$	$\underset{\pm 0.05}{99.07}$	$\underset{\pm 0.05}{\mathbf{99.07}·}$
flare	$\underset{\pm 0.14}{94.85}$	$\underset{\pm 0.14}{\mathbf{94.87}·}$	$\underset{\pm 0.18}{93.72}$	$\underset{\pm 0.12}{94.53}$	$\underset{\pm 0.17}{94.84}$	$\underset{\pm 0.10}{94.82}$	$\underset{\pm 0.17}{93.89}$	$\underset{\pm 0.12}{94.76}$	$\underset{\pm 0.18}{94.76}$	$\underset{\pm 0.16}{94.86}$
house-v.	$\underset{\pm 0.18}{\mathbf{99.38}·}$	$\underset{\pm 0.17}{99.36}$	$\underset{\pm 0.24}{98.67}$	$\underset{\pm 0.26}{98.96}$	$\underset{\pm 0.26}{99.26}$	$\underset{\pm 0.21}{99.16}$	$\underset{\pm 0.23}{98.85}$	$\underset{\pm 0.20}{98.87}$	$\underset{\pm 0.25}{98.94}$	$\underset{\pm 0.25}{98.94}$
kr-vs-kp	$\underset{\pm 0.01}{99.14}$	$\underset{\pm 0.01}{\mathbf{99.98}·}$	$\underset{\pm 0.01}{99.94}$	$\underset{\pm 0.01}{99.97}$	$\underset{\pm 0.01}{99.94}$	$\underset{\pm 0.01}{\mathbf{99.98}·}$	$\underset{\pm 0.01}{99.97}$	$\underset{\pm 0.01}{99.94}$	$\underset{\pm 0.01}{99.97}$	$\underset{\pm 0.01}{99.94}$
lympho.	$\underset{\pm 1.20}{92.37}$	$\underset{\pm 1.28}{92.62}$	$\underset{\pm 0.82}{92.20}$	$\underset{\pm 1.01}{93.08}$	$\underset{\pm 1.10}{93.03}$	$\underset{\pm 1.17}{93.10}$	$\underset{\pm 0.89}{92.86}$	$\underset{\pm 1.02}{93.07}$	$\underset{\pm 1.01}{93.13}$	$\underset{\pm 1.08}{\mathbf{93.34}·}$
monks-1	$\underset{\pm 3.12}{46.32}$	$\underset{\pm 0.47}{99.10}$	$\underset{\pm 1.52}{89.87}$	$\underset{\pm 1.35}{91.98}$	$\underset{\pm 0.23}{99.70}$	$\underset{\pm 0.32}{99.71}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$
primary-t.	$\underset{\pm 0.81}{73.31}$	$\underset{\pm 0.90}{72.75}$	$\underset{\pm 0.89}{73.01}$	$\underset{\pm 0.39}{\mathbf{73.41}·}$	$\underset{\pm 0.68}{73.00}$	$\underset{\pm 0.54}{72.99}$	$\underset{\pm 0.55}{73.37}$	$\underset{\pm 0.54}{73.06}$	$\underset{\pm 0.45}{73.12}$	$\underset{\pm 0.45}{73.34}$
promoters	$\underset{\pm 0.98}{97.23}$	$\underset{\pm 0.98}{97.37}$	$\underset{\pm 0.88}{95.07}$	$\underset{\pm 0.82}{\mathbf{97.65}·}$	$\underset{\pm 0.94}{97.31}$	$\underset{\pm 0.92}{97.29}$	$\underset{\pm 0.75}{97.61}$	$\underset{\pm 0.91}{97.46}$	$\underset{\pm 0.92}{97.37}$	$\underset{\pm 0.83}{97.41}$
soybean	$\underset{\pm 0.12}{99.47}$	$\underset{\pm 0.10}{99.68}$	$\underset{\pm 0.08}{99.56}$	$\underset{\pm 0.07}{99.72}$	$\underset{\pm 0.06}{99.65}$	$\underset{\pm 0.10}{99.74}$	$\underset{\pm 0.09}{\mathbf{99.73}·}$	$\underset{\pm 0.07}{99.72}$	$\underset{\pm 0.09}{99.71}$	$\underset{\pm 0.09}{99.71}$
spect	$\underset{\pm 1.82}{\mathbf{83.81}·}$	$\underset{\pm 1.82}{83.70}$	$\underset{\pm 1.90}{78.57}$	$\underset{\pm 1.94}{82.34}$	$\underset{\pm 1.92}{84.14}$	$\underset{\pm 1.77}{83.43}$	$\underset{\pm 1.97}{82.27}$	$\underset{\pm 2.04}{82.25}$	$\underset{\pm 1.99}{82.53}$	$\underset{\pm 1.95}{82.42}$
splice	$\underset{\pm 0.03}{98.48}$	$\underset{\pm 0.04}{99.12}$	$\underset{\pm 0.13}{98.27}$	$\underset{\pm 0.04}{\mathbf{99.31}·}$	$\underset{\pm 0.04}{99.11}$	$\underset{\pm 0.05}{99.01}$	$\underset{\pm 0.04}{99.30}$	$\underset{\pm 0.03}{99.25}$	$\underset{\pm 0.03}{99.29}$	$\underset{\pm 0.03}{99.28}$
t-t-t	$\underset{\pm 0.43}{97.86}$	$\underset{\pm 0.18}{98.53}$	$\underset{\pm 0.03}{99.94}$	$\underset{\pm 0.02}{\mathbf{99.97}·}$	$\underset{\pm 0.05}{99.89}$	$\underset{\pm 0.05}{99.88}$	$\underset{\pm 0.03}{99.96}$	$\underset{\pm 0.03}{99.95}$	$\underset{\pm 0.03}{99.96}$	$\underset{\pm 0.02}{99.95}$
zoo	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$	$\underset{\pm 0.00}{\mathbf{100.00}·}$
Rank	7.25	5.25	7.81	4.13	5.00	5.13	5.31	4.56	3.50	3.44

.

In general, we can see that the normal form kernels seem to achieve performances almost always comparable or higher than the competing kernels. Good performance is also achieved by the d-mDNF kernel which we have shown is the summation of d mC-kernels.

An interesting observation can be done regarding the monks-1 dataset which can be explained by a mDNF rule. The obtaining results on monks-1 show that most of the proposed Boolean kernels achieve an AUC of 100% while all other kernels are not able to achieve this perfect score. This is because these Boolean kernels contain the target formula, or a set of related formulas, in the feature space.

All the reported results are further confirmed by other experiments with other metrics, such as precision, recall and F1, however they are not reported here for space reasons.

6. Conclusions

We present a family of Boolean kernels designed to have feature spaces composed by logical formulas that can be exploited to interpret the solution of a large-margin kernel machine, such as an SVM. For all kernels, we provide the logic interpretation and an efficient way to compute them. Experimental results on many categorical datasets show that the presented kernels achieve a performance comparable to the performance of state-of-the-art kernels (such as the RBF) and other Boolean kernels. In particular, we observed that, in general, those kernels corresponding to normal form achieve very good performances. In the future, we aim to develop methods able to learn how to combine different Boolean kernels, for example by means of Multiple Kernel Learning algorithms. Moreover, we also aim to find efficient and effective ways to interpret the solution of SVMs based on Boolean kernels.