Privacy-Preserving Feature Selection with Fully Homomorphic Encryption

Abstract

For the feature selection problem, we propose an efficient privacy-preserving algorithm. Let D, F, and C be data, feature, and class sets, respectively, where the feature value $x\left(F_i\right)$ and the class label $x\left(C\right)$ are given for each $x\in D$ and $F_i\in F$. For a triple $(D,F,C)$, the feature selection problem is to find a consistent and minimal subset $F^{\prime }\subseteq F$, where ‘consistent’ means that, for any $x,y\in D$, $x\left(C\right)=y\left(C\right)$ if $x\left(F_i\right)=y\left(F_i\right)$ for $F_i\in F^{\prime }$, and ‘minimal’ means that any proper subset of $F^{\prime }$ is no longer consistent. On distributed datasets, we consider feature selection as a privacy-preserving problem: assume that semi-honest parties $\mathsf{A}$ and $\mathsf{B}$ have their own personal $D_{\mathsf{A}}$ and $D_{\mathsf{B}}$. The goal is to solve the feature selection problem for $D_{\mathsf{A}}\cup D_{\mathsf{B}}$ without sacrificing their privacy. In this paper, we propose a secure and efficient algorithm based on fully homomorphic encryption, and we implement our algorithm to show its effectiveness for various practical data. The proposed algorithm is the first one that can directly simulate the CWC (Combination of Weakest Components) algorithm on ciphertext, which is one of the best performers for the feature selection problem on the plaintext.

1. Introduction

1.1. Motivation

This study proposes a secure feature selection protocol that works effectively as a preprocessor for traditional machine learning (ML). Let us consider a scenario where different data owners are interested in private ML model training (e.g., logistic regression [1], SVM [2,3], and decision tree [4,5]) on their combined data. There is a large advantage to securely training these ML models on distributed data due to competitive advantage or privacy regulations. Feature selection is the problem of finding a subset of relevant features for model training. Using well-chosen features can lead to more accurate models, as well as speedup during model training [6].

Consider a data set D associated with a feature set F and a class variable C, where all feature values $x\left(F_i\right)$ $(F_i\in F)$ and the corresponding class label $x\left(C\right)$ are defined for each datum $x\in D$. In

Table 1. An example dataset shown in [7].

D	${\mathit{F}}_1$	${\mathit{F}}_2$	${\mathit{F}}_3$	${\mathit{F}}_4$	${\mathit{F}}_5$	C
$x_1$	1	0	1	1	1	0
$x_2$	1	1	0	0	0	0
$x_3$	0	0	0	1	1	0
$x_4$	1	0	1	0	0	0
$x_5$	1	1	1	1	0	1
$x_6$	0	1	0	1	0	1
$x_7$	0	1	0	0	1	1
$x_8$	0	0	0	0	1	1
$I(F_i;C)$	0.189	0.189	0.049	0.000	0.000

, for example, we show a concrete example. Given a triple $(D,F,C)$, the feature selection problem is to find a minimal $F^{\prime }\subseteq F$ that is relevant to the class C. The relevance of $F^{\prime }$ is evaluated, for example, by $I(F^{\prime };C)$, which measures the mutual information between $F^{\prime }$ and C. On the other hand, $F^{\prime }$ is minimal, if any proper subset of $F^{\prime }$ is no longer consistent.

To the best of our knowledge, the most common method for identifying favorable features is to choose features that show higher relevance in some statistical measures. Individual feature relevance can be estimated using statistical measures such as mutual information and Bayesian risk. For example, at the bottom row of Table 1, the mutual information score $I(F_1;C)$ of each feature $F_i$ to class labels is described. We can see that $F_1$ is more important than $F_5$, because $I(F_1;C)>I(F_5;C)$. $F_1$ and $F_2$ of Table 1 will be chosen to explain C based on the mutual information score. However, a closer examination of D reveals that $F_1$ and $F_2$ cannot uniquely determine C. In fact, we find $x_2$ and $x_5$ with $x_2\left(F_1\right)=x_5\left(F_1\right)$ and $x_2\left(F_2\right)=x_5\left(F_2\right)$ but $x_2\left(C\right)\ne x_5\left(C\right)$. On the other hand, we can see that $F_4$ and $F_5$ uniquely determine C using the formula $C=F_4\oplus F_5$ while $I(F_4;C)=I(F_5;C)=0$. As a result, the traditional method based on individual feature relevance scores misses the correct answer.

Thus, we concentrate on the concept of consistency: $F^{\prime }\subseteq F$ is considered to be consistent if, for any $x,y\in D$, $x\left(F_i\right)=y\left(F_i\right)$ for all $F_i\in F^{\prime }$ implies $x\left(C\right)=y\left(C\right)$. In machine learning research, consistency-based feature selection has received a lot of attention [8,9,10,11,12]. CWC (Combination of Weakest Components) [8] is the simplest of such consistency-based feature selection algorithms, and even though CWC uses the most rigorous measure, it shows one of the best performances in terms of accuracy as well as computational speed compared to other methods [7]. Throughout the proposed secure protocol, none of the parties learns the values of the data as all computations are done over ciphertexts. Next, the parties train an ML model over the pre-processed data using existing privacy-preserving training protocols (e.g., logistic regression training [13] and decision tree [14]). Finally, they can disclose the trained model for common use.

To design a secure protocol for feature selection, we focus on the framework of homomorphic encryption. Given a public key encryption scheme E, let $E\left[m\right]$ denote a ciphertext of integer m; if $E[m+n]$ can be computed from $E\left[m\right]$ and $E\left[n\right]$ without decrypting them, then E is said to be additive homomorphic, and if $E\left[mn\right]$ can also be computed, then E is said to be fully homomorphic. Furthermore, modern public key encryption must be probabilistic: when the same message m is encrypted multiple times, the encryption algorithm produces different ciphertexts of $E\left[m\right]$.

Various homomorphic encryption schemes have been proposed to satisfy these homomorphic properties over the last two decades. The first additive homomorphic encryption was proposed by Paillier [15]. Somewhat homomorphic encryption that allows a sufficient number of additions and a limited number of multiplications has also been proposed [16,17,18], and we can use these cryptosystems to compute more difficult problems, such as the inner product of two vectors. Gentry [19] proposed the first fully homomorphic encryption (FHE) with an unlimited number of additions and multiplications, and since then, useful libraries for fully homomorphic encryption have been developed, particularly for bitwise operations and floating-point operations. TFHE [20,21] is known as the fastest fully homomorphic encryption that is optimized for bitwise operations.

For the private feature selection problem, we use TFHE to design and implement our algorithm. In this case, we assume two semi-honest parties $\mathsf{A}$ and $\mathsf{B}$: each party complies with the protocol but tries to infer as much as possible about the secret from the information obtained. The parties have their own private data $D_{\mathsf{A}}$ and $D_{\mathsf{B}}$ and they jointly compute advantageous features for $D_{\mathsf{A}}\cup D_{\mathsf{B}}$ while maintaining their privacy. The goal is to jointly compute the CWC algorithm result on $D=D_{\mathsf{A}}\cup D_{\mathsf{B}}$ without revealing any other information.

In this paper, we describe the simplest case where there are two data owners, and they perform the cooperative secure computation. More generally, there are many data owners, and they encrypt with their own public keys. Since homomorphic operations cannot be applied to two data encrypted with different public keys, a simple approach would be for the server to attempt to re-encrypt them with some common public key. However, there is no guarantee that the server or the new public key can be trusted. To solve this problem, the framework of multi-key homomorphic encryption was proposed. This allows FHE operations on data encrypted with different keys, i.e., we can extend the two-party computation model to a more general case because TFHE has the required property. Using this property, its application to the framework of oblivious neural network inference [22] has been proposed.

This should be a realistic requirement, if one wants to draw some conclusions from data that are privately distributed over more than one party. Multi-party computation (MPC) can provide effective technical solutions to realize this requirement in many cases. In MPC, certain computations that essentially rely on the distributed data are performed through cooperation among the parties. In particular, fully homomorphic encryption (FHE) is one of the critical tools of MPC. One of the most significant advantages of FHE-based MPC is thought to be that FHE realizes outsourced computation in a simple and straightforward manner: parties encrypt their private data with their public keys and send the encrypted data to a single trusted party with sufficient computational power to perform the required computation; although the computational results of the trusted party may be incorrect, if some malicious parties send incorrect data, honest parties are at least convinced that their private data have not been stolen as far as the cryptosystem used is secure. In contrast, when a party shares his/her secret with other parties to perform MPC, even if it uses a secure secret sharing scheme, collusion of a sufficient number of compromised parties may reveal the party’s secret. In general, it is difficult to prove the security of MPC protocols for the situation where we cannot deny the existence of active malicious parties, and hence, the security is very often proven assuming that all the parties are at worst semi-honest. In reality, however, even this relaxed assumption is unable to hold. Thus, the property that a party can protect its private data only relying on its own efforts should be counted as an important advantage of FHE-based MPC.

On the other hand, the current implementations of FHE are thought to be significantly inefficient, and consequently, their ranges of application are actually limited. This is currently true, but may not be true in the future: the Goldwasser–Mmicali (GM) cryptosystem [23] is considered as the first scheme with provable security. Unfortunately, because the GM cryptosystem encrypts data in a bitwise manner, it has turned out not to have sufficient efficiency in time and memory to be used in the real world. In 2001, however, RSA-OAEP was finally proven to have both provable security and realistic efficiency [24,25], and is widely used through SSL/TLS. Thus, studying FHE-based MPC does not merely have theoretical meaning, but also will yield significant contributions in terms of application to the real world in the future.

In this paper, we propose an MPC protocol which relies on FHE-based outsourced computation as well as mutual cooperation among parties. The target of our protocol is to perform the computation of CWC, a feature selection algorithm known to be accurate and efficient, preserving the privacy of the participating parties. If we fully perform CWC by FHE-based outsourced computation, we have to pay unnecessarily large costs in time in the phase of sorting the features of CWC. Therefore, in our proposed scheme, we add ingenuity so that two parties cooperate with each other to sort the features efficiently.

Converting CWC into its privacy-preserving version based on different primitives of MPC—for example, based on secret sharing techniques—is not only interesting but also useful both in theory and in practice. We will pursue this direction as well in our future work.

1.2. Our Contribution and Related Work

Table 2. Time and space complexities of the baseline and improved algorithms for secure CWC, where k is the number of features and $m,n$ are the numbers of positive and negative data, respectively. We assume that the time of the respective operation (e.g., encryption/addition/multiplication/comparison) in FHE is $O\left(1\right)$.

Algorithm	Time	Space
CWC on plaintext [8]	$O(kmn+klogk)$	$O\left(kmn\right)$
secure CWC (baseline)	$O(kmnlogk+k{log}^{2}k)$	$O\left(kmn\right)$
improved	$O(kmn+k{log}^{2}k+klogklogmn)$	$O\left(kmn\right)$

summarizes the complexities of the proposed algorithms in comparison to the original CWC on plaintext. The baseline is a naive algorithm that can simulate the original CWC [8] over ciphertext using TFHE operations. The bottleneck of private feature selection exists in the sorting task over ciphertext, as we mention in the related work below. Our main contribution is the improved algorithm, shown as ‘improved’, which significantly reduces the time complexity caused by the sorting task. We also implement the improved algorithm and demonstrate its efficiency through experiments in comparison to the baseline.

There are mainly two private computation models, secret sharing-based MPC and public key-based MPC, and secret sharing-based MPC currently has an advantage. On the other hand, we focus on the convenience of FHE. Public key-based MPC can establish a simple mechanism to obtain results while keeping the learning model possessed by the server and the personal information of many data owners confidential from each other, relying only on cryptographic strength. The secret sharing-based MPC is faster but requires at least two trusted parties that do not collude with each other, which creates a different problem to cryptographic strength.

Other drawbacks of public key-based MPC are its security against the chosen plaintext attack (CPA) and computational cost. TFHE is, however, computationally secure against the chosen ciphertext attack (CCA), which assumes a stronger adversary than CPA so that an attacker cannot obtain meaningful information from plaintext or ciphertext within polynomial time.

In this section, we discuss related work on private feature selection as well as the benefits of our method. Rao [26] et al. proposed a homomorphic encryption-based private feature selection algorithm. Their protocol allows the additive homomorphic property only, which invariably leaks statistical information about the data. Anaraki and Samet [27] proposed a different method based on the rough set theory, but their method suffers from the same limitations as Rao et al., and neither method has been implemented. Banerjee et al. [28], and Sheikhalishahi and Martinellil [29] have proposed MPC-based algorithms that guarantee security by decomposing the plaintext into shares, as a different approach to the private feature selection, while achieving cooperative computation. Li et al. [30] improved the MPC protocol on the aforementioned flaw and demonstrated its effectiveness through experiments.

These methods avoid partial decoding under the assumption that the mean of feature values provides a good criterion for feature selection. This assumption, however, is heavily dependent on data. The most important task in general feature selection is feature value-based sorting, and CWC and its variants [7,8,11] demonstrated the effectiveness of sorting with the consistency measure and its superiority over other methods. On ciphertext, this study realizes the sorting-based feature selection algorithm (e.g., CWC).

We focus on the learning decision tree by MPC [31] as another study that employs sorting for private ML, where the sorting is limited to the comparison of N values of fixed length in $O\left(N{log}^{2}N\right)$ time by a sorting network. In the case of CWC, however, the algorithm must sort N data points, each of which has a variable length of up to M, so a naive method requires $O(MNlogN+N{log}^{2}N)$ time. Our algorithm reduces this complexity to $O(MN+N{log}^{2}N+NlogNlogM)$, which is significantly smaller than the naive algorithm depending on M and N. Through experiments, we confirm this for various data, including real datasets for ML.

Although sorting itself is not ML, a fast-sorting algorithm is an important preprocess for ML model training. In the previous result [7], it was shown that sorting-based feature selection can classify with higher accuracy than other heuristic methods. Furthermore, preprocessing by sorting has proven to be an important task in decision tree model training [31]. On the other hand, it is also well known that sorting can speed up ML model training. For example, in SVM, which is widely used in text classification and pattern recognition, the problem of finding the convex hull of n points in Euclidean space can be reduced from $O\left(n^2\right)$ to $O\left(nlogn\right)$ time by preprocessing it with an appropriate sorting algorithm.

2. Preliminaries

2.1. Consistency Measure

First of all, we review the notion of the consistency measure employed in our problem. A consistency measure $\mu :2^F\to [0,\infty )$ for a feature set F is a function to represent how far the data deviate from a consistent state and is required to satisfy determinisity ($\mu \left(F\right)=0$ if and only if F is consistent) and monotonicity ($F\subseteq G$ implies $\mu \left(F\right)\ge \mu \left(G\right)$). The following consistency measures satisfy this requirement.

• ${\mu }_{\mathrm{bin}}\left(F\right)=0$, F is consistent; 1, otherwise (binary consistency [11])
• ${\mu }_{\mathrm{icr}}\left(F\right)={\sum }_x\left(\mathrm{Pr}(F=x)-{max}_c\mathrm{Pr}(F=x,C=c)\right)$ (ICR [32])
• ${\mu }_{\mathrm{rs}}\left(F\right)=1-{\sum }_c\frac{F{D}_{C=c}}{\left|D\right|},FD=\{D_{F=x}\mid D_{F=x}\subseteq D\}$ (rough set [33])
• ${\mu }_{\mathrm{ie}}\left(F\right)={\sum }_x{\sum }_{c\ne c^{\prime }}\frac{|D_{F=x,C=c}|·|D_{F=x,C=c^{\prime }}|}{{\left|D\right|}^2}$ (inconsistent pair [34])

The fully homomorphic encryption used in this study is specialized for binary operations. Therefore, among these consistency measures, we employ ${\mu }_{\mathrm{bin}}$.

2.2. CWC Algorithm over Plaintext

We generally assume that the dataset D associated with F and C contains no errors, i.e., if $x\left(F_i\right)=y\left(F_i\right)$ for all i, $x\left(C\right)=y\left(C\right)$. When D contains such errors, they are removed beforehand and D contains not more than one $x\in D$ with the same feature values.

In Algorithm 1, we describe the original algorithm for finding a minimal consistent feature for two-class data. Given D with $F_i$ and $C=\{0,1\}$, a datum $x\in D$ of $x\left(C\right)=1$ is referred to as a positive datum and $y\in D$ of $y\left(C\right)=0$ is referred to as a negative datum. Let n represent the number of positive data and $m=\left|D\right|-n$. We consider two-dimensional bit array ${\mathbf{B}}_i[1..n][1..m]$ such that, for any $1\le p\le n$ and $1\le q\le m$, ${\mathbf{B}}_i\left[p\right]\left[q\right]=0$ if $x_p\left(F_i\right)=y_q\left(F_i\right)$ and ${\mathbf{B}}_i\left[p\right]\left[q\right]=1$ otherwise, where $x_p$ is the p-th positive datum $(1\le p\le n)$ and $y_q$ is the q-th negative datum $(1\le q\le m)$. ${\mathbf{B}}_i\left[p\right]\left[q\right]=0$ means that $F_i$ is not consistent with the pair $(x_p,y_q)$ because $x_p\left(F_i\right)=y_q\left(F_i\right)$ despite $x_p\left(C\right)\ne y_q\left(C\right)$. Recall that $F_i$ is said to be consistent only if $x\left(F_i\right)=y\left(F_i\right)$ implies $x\left(C\right)=y\left(C\right)$ for any $x,y\in D$. As a result, $\left|\right|{\mathbf{B}}_i\left|\right|$ is defined to be the number of 1s in ${\mathbf{B}}_i$.

For a subset $F^{\prime }\subseteq F$, $F^{\prime }$ is said to be consistent, if for any $p\in [1,n]$ and $q\in [1,m]$, there exists i such that $F_i\in F^{\prime }$ and ${\mathbf{B}}_i\left[p\right]\left[q\right]=1$ hold. CWC uses this to remove irrelevant features from F in order to build a minimal consistent feature set. We note that finding the smallest consistent feature set is clearly NP-hard. There is a simple reduction from the minimum set cover to this problem as follows: given $S_1,\dots ,S_k\subseteq S$$\left(\right|S|=n)$ with the intention that $S_i$ is regarded as ${\mathbf{B}}_i$ in CWC, covering any element of S corresponds to the condition that for any $j\in \{1,\dots ,n\}$, there exists at least one i such that ${\mathbf{B}}_i\left[j\right]=1$.

Since the point of ${\mathbf{B}}_i$ is that it contains information, for every pair of data across different classes, whether $F_i$ is consistent with the pair or not, it can be easily extended to multi-class data that have more than two classes. Although we focus on two-class data for the sake of simplicity, for multi-class data, the $mn$-factors in the complexities are replaced with the number of pairs of data across different classes, which is upper bounded by $\left|D\right|\left(\right|D|-1)/2$. Moreover, in extending to multi-class data, it is convenient to consider ${\mathbf{B}}_i$ as an appropriately serialized one-dimensional bit string because there is no way to represent it as a dense two-dimensional bit array. Hence, in what follows, we treat ${\mathbf{B}}_i$ as a bit string.

Algorithm 1 The algorithm CWC for plaintext

1: Input: A dataset D associated with features $F=\{F_1,\dots ,F_k\}$ and class $C=\{0,1\}$. 2: Output: A minimal consistent subset $S\subseteq F$. 3: Sort $F_1,\dots ,F_k$ in the incremental order of $\left|\right|{\mathbf{B}}_i\left|\right|$. 4: Let $\pi$ be the sorted indices of $\{1,\dots ,k\}$. 5: for  $i=1,\dots ,k$ 6:     if $F\setminus \left\{F_{\pi \left[i\right]}\right\}$ is consistent then 7:         update $F\leftarrow F\setminus \left\{F_{\pi \left[i\right]}\right\}$ 8:     end if 9: end for

Table 3. An example dataset D with $F=\{F_1,F_2,F_3,F_4\}$ and $C=\{0,1\}$. The data consist of two positive data $\{x_1,x_2\}$ and five negative data $\{y_1,y_2,y_3,y_4,y_5\}$.

${\mathit{x}}_{\mathit{i}}\in \mathit{D}$	${\mathit{F}}_1$	${\mathit{F}}_2$	${\mathit{F}}_3$	${\mathit{F}}_4$	C
$x_1$	0	1	1	0	1
$x_2$	0	0	1	1	1
$y_i\in D$	$F_1$	$F_2$	$F_3$	$F_4$	C
$y_1$	1	0	1	0	0
$y_2$	1	1	0	0	0
$y_3$	0	1	0	1	0
$y_4$	1	0	1	0	0
$y_5$	1	1	0	0	0

shows an example of D, and 

Table 4. The bit string ${\mathbf{B}}_i$ for the example dataset D of Table 3. Each column $(x_p,y_q)$ is 0 if $x_p\left(F_i\right)=y_q\left(F_i\right)$. For example, ${\mathbf{B}}_1=(1,1,0,1,1,1,1,0,1,1)$ because $x_p\left(F_1\right)=y_q\left(F_1\right)$ only for the two pairs $(x_1,y_3)$ and $(x_2,y_3)$.

${\mathit{B}}_{\mathit{i}}$	$({\mathit{x}}_1,{\mathit{y}}_1)$	$({\mathit{x}}_1,{\mathit{y}}_2)$	$({\mathit{x}}_1,{\mathit{y}}_3)$	$({\mathit{x}}_1,{\mathit{y}}_4)$	$({\mathit{x}}_1,{\mathit{y}}_5)$	$({\mathit{x}}_2,{\mathit{y}}_1)$	$({\mathit{x}}_2,{\mathit{y}}_2)$	$({\mathit{x}}_2,{\mathit{y}}_3)$	$({\mathit{x}}_2,{\mathit{y}}_4)$	$({\mathit{x}}_2,{\mathit{y}}_5)$
${\mathbf{B}}_1$	1	1	0	1	1	1	1	0	1	1
${\mathbf{B}}_2$	1	0	0	1	0	0	1	1	0	1
${\mathbf{B}}_3$	0	1	1	0	1	0	1	1	0	1
${\mathbf{B}}_4$	0	0	1	0	0	1	1	0	1	1

shows the corresponding ${\mathbf{B}}_i$. Consider the behavior of CWC in this case. All ${\mathbf{B}}_i$$(1\le i\le 4)$ are computed as preprocessing. Then, the features are sorted by the order $\left|\right|{\mathbf{B}}_2\left|\right|=5\le \left|\right|{\mathbf{B}}_4\left|\right|=5\le \left|\right|{\mathbf{B}}_3\left|\right|=6\le \left|\right|{\mathbf{B}}_1\left|\right|=8$ and $\pi =(2,4,3,1)$. By the consistency order $\pi$, CWC checks whether $F_{\pi \left[i\right]}$ can be removed from the current F. Using the consistency measure, CWC removes $F_2$ and $F_4$ and the resulting $\{F_1,F_3\}$ is the output. In fact, we can predict the class of x by the logical operation $\overline{x\left(F_1\right)}\wedge x\left(F_3\right)$.

2.3. Security Model

2.3.1. Indistinguishable Random Variables

Let $\mathbb{N}$ denote the set of natural numbers. A function $ϵ:\mathbb{N}\to [0,1]$ is called negligible, if $\forall c>0,\exists k,\forall n\ge k,ϵ\left(n\right)<1/n^c$. Let $X=\{X_k\mid k\in \mathbb{N}\}$ and $Y=\{Y_k\mid k\in \mathbb{N}\}$ be sequences of random variables such that $X_k$ and $Y_k$ are defined over the same sample space. We say that X and Y are indistinguishable, denoted by $X{\equiv }_{c}Y$, if, and only if, $Pr[X_n=Y_n]$ is a negligible function.

2.3.2. Security of Multi-Party Computation (MPC)

Although the discussion of this section can be extended to MPC schemes which involve more than two parties, merely for simplicity, we focus on the case where only two parties are involved.

A two-party protocol is a pair $\mathsf{\Pi }=({\mathcal{P}}_1,{\mathcal{P}}_2)$ of PPT Turing machines with input and random tapes. Let $x_i$ be an input of ${\mathcal{P}}_i$ and $y_i$ be an output of ${\mathcal{P}}_i$, respectively.

We assume a semi-honest adversary $\mathcal{A}$ and consider a protocol $(\mathcal{A},{\mathcal{P}}_2)$, replacing ${\mathcal{P}}_1$ in $\mathsf{\Pi }$ by $\mathcal{A}$, where $\mathcal{A}$ takes $x_1$ as input and apparently follows the protocol. Let ${\mathtt{REAL}}_{\mathsf{\Pi },\mathcal{A}}(x_1,x_2)$ denote the random variable representing the output $(y_1,y_2)$ of $(\mathcal{A},{\mathcal{P}}_2)$, and we define the class ${\mathtt{REAL}}_{\mathsf{\Pi },\mathcal{A}}={\left\{{\mathtt{REAL}}_{\mathsf{\Pi },\mathcal{A}}(x_1,x_2)\right\}}_{x_1,x_2}={\{y_1,y_2\}}_{x_1,x_2}$.

On the other hand, let $\mathcal{F}$ denote the functionality that the protocol $\mathsf{\Pi }$ is trying to realize, i.e., $\mathcal{F}$ is a PPT that simulates the honest $({\mathcal{P}}_1,{\mathcal{P}}_2)$ so that $\mathcal{F}(x_1,x_2)\equiv ({\mathcal{P}}_1\left(x_1\right),{\mathcal{P}}_2\left(x_2\right))$. Here, we assume a completely reliable third party, denoted by $\mathcal{F}$. In this ideal world, for this $\mathcal{F}$ and any adversary $\mathcal{B}$ acting as ${\mathcal{P}}_1$ with input $x_1^{\prime }$, possibly $x_1^{\prime }\ne x_1$, we define the random variable ${\mathtt{IDEAL}}_{\mathcal{F},\mathcal{B}}(x_1,x_2)=\left(\mathcal{B}(x_1,{\mathcal{F}}_1(x_1^{\prime },x_2),{\mathcal{F}}_2(x_1^{\prime },x_2))\right)$, where ${\mathcal{F}}_i(·,·)$ denotes the i-th component of the output of $\mathcal{F}(·,·)$ for $i=1,2$. Similarly, we denote the class ${\mathtt{IDEAL}}_{\mathcal{F},\mathcal{B}}={\left\{{\mathtt{IDEAL}}_{\mathcal{F},\mathcal{B}}(x_1,x_2)\right\}}_{x_1,x_2}$.

Using such random variables, we define the security of protocol $\mathsf{\Pi }$ as follows.

Definition 1.

It is said that a protocol Π securely realizes a functionality $\mathcal{F}$ if, for any attacker $\mathcal{A}$ against Π, there exists an adversary $\mathcal{B}$, and ${\mathtt{REAL}}_{\mathsf{\Pi },\mathcal{A}}{\equiv }_c{\mathtt{IDEAL}}_{\mathcal{F},\mathcal{B}}$ holds.

The definitions stated above can be intuitively explained as follows. Exactly conforming to the protocol, a semi-honest adversary $\mathcal{A}$ plays the role of ${\mathcal{P}}_1$ to steal any secrets. The information sources which $\mathcal{A}$ can take advantage of are the following three:

1.

the input tape to ${\mathcal{P}}_1$;

2.

the conversation with ${\mathcal{P}}_2$;

3.

the execution of the protocol.

While the information that $\mathcal{A}$ can obtain from the first and third sources is exactly $x_1$ and $y_1$, respectively, we call the information from the second source a view.

To denote it, we use the symbol ${\mathsf{View}}_{{\mathcal{P}}_1}$.

Since the protocol inevitably requires that $\mathcal{A}$ obtains the information of $x_1$ and $y_1$, the security of the protocol questions what $\mathcal{A}$ can obtain, in addition to what can be computationally inferred from $x_1$ and $y_1$. If there exists such information, its source must be ${\mathsf{View}}_{{\mathcal{P}}_1}$.

The security criterion of simulatability requires that ${\mathsf{View}}_{P_1}$ can be simulated on the input of $x_1$ and $y_1$. In more formal terms, there exists a PPT Turing machine Sim that outputs a view on the input of $x_1$ and $y_1$ such that the output view cannot be distinguished from ${\mathsf{View}}_{{\mathcal{P}}_1}$ by any PPT Turing machine. When ${\mathsf{View}}_{{\mathcal{P}}_1}$ is simulatable, we see that Sim can generate by itself what Sim can obtain from ${\mathsf{View}}_{P_1}$. Therefore, Sim cannot cannot obtain any information in addition to what Sim can compute from $x_1$ and $y_1$.

2.3.3. IND-CPA

Indistinguishability against chosen plaintext attack (IND-CPA) is an important criterion for the secrecy of a public key cryptosystem. We let $\mathsf{\Pi }=(\mathtt{Gen},\mathtt{Enc},\mathtt{Dec})$ denote a public key cryptosystem consisting of key generation, encryption, and decryption algorithms. To describe IND-CPA, we introduce the IND-CPA game played between an adversary $\mathcal{A}$ and an oracle $\mathcal{O}$: $\mathcal{A}$ is a PPT Turing machine, and k is the security parameter.

1.

$\mathcal{O}$ generates a public key pair $(sk,pk)\leftarrow \mathtt{Gen}\left(1^k\right)$.

2.

$\mathcal{A}$ generates two messages $(m_0,m_1)$ of the same length arbitrarily and throws a query $(m_0,m_1)$ to $\mathcal{O}$.

3.

On receipt of $(m_0,m_1)$, $\mathcal{O}$ selects $b\in \{0,1\}$ uniformly at random, computes $c=\mathtt{Enc}(pk,m_b)$, and replies to $\mathcal{A}$ with c.

4.

$\mathcal{A}$ guesses on b by examining c and outputs the guess bit $b^{\prime }$.

We view b and $b^{\prime }$ as random variables whose underlying probability space is defined to represent the choices of the public key pair, b and $b^{\prime }$. The advantage of the adversary $\mathcal{A}$ is defined as follows to represent the advantage of $\mathcal{A}$ over tossing a fair coin to guess $\mathcal{O}$’s secret b:

$${\mathsf{Adv}}_{\mathcal{A}}=2·Pr[b^{\prime }=b]-1.$$

when we let

$$Pr[b^{\prime }=0|b=0]=\frac{1}{2}+{\alpha }_0\mathrm{and}Pr[b^{\prime }=1|b=1]=\frac{1}{2}+{\alpha }_1,$$

We have

$${\mathsf{Adv}}_{\mathcal{A}}={\alpha }_0+{\alpha }_1$$

This definition of the advantage is consistent with the common definition found in many textbooks:

$${\mathsf{Adv}}_{\mathcal{A}}=Pr[b^{\prime }=0|b=0]-Pr[b^{\prime }=1|b=0]$$

Definition 2.

A public key cryptosystem Π is secure in the sense of IND-CPA, or simply IND-CPA secure, if ${\mathsf{Adv}}_{\mathcal{A}}$ as a function in k is a negligible function.

2.4. TFHE: A Faster Fully Homomorphic Encryption

The proposed private feature selection is based on FHE. We review the TFHE [21], one of the fastest libraries for bitwise addition (this means XOR ‘⊕’) and bitwise multiplication (AND ‘·’) over ciphertext. On TFHE, any integer is encrypted bitwise: For ℓ-bit integer $m=(m_1,\dots ,m_{\ell })$, we denote its bitwise encryption by $E\left[m\right]\equiv (E\left[m_1\right],\dots ,E\left[m_{\ell }\right])$, for short. These bitwise operations are denoted by $f_{\oplus }(E\left[x\right],E\left[y\right])\equiv E[x\oplus y]$ and $f_{·}(E\left[x\right],E\left[y\right])\equiv E[x·y]$ for $x,y\in \{0,1\}$ and the ciphertexts $E\left[x\right]$ and $E\left[y\right]$. The same symbol is used to represent an encrypted array. For example, when x and y are integers of length ℓ and ${\ell }^{\prime }$, respectively, $E(x,y)$ denotes

$$E[x,y]\equiv (E\left[x\right],E\left[y\right])\equiv ((E\left[x_1\right],\dots ,E\left[x_{\ell }\right]),(E\left[y_1\right],\dots ,E\left[y_{{\ell }^{\prime }}\right])).$$

TFHE allows all arithmetic and logical operations via the elementary operations $E[x\oplus y]$ and $E[x·y]$. In this section, we will consider how to build the adder and comparison operations. Let $x,y$ represent ℓ-bit integers and $x_i,y_i$ represent the i-th bit of $x,y$, respectively. Let $c_i$ represent the i-th carry-in bit and $s_i$ is the i-th bit of the sum $x+y$. Then, we can obtain $E[x+y]$ by the bitwise operations of ciphertexts using $s_i=x_i\oplus y_i\oplus c_i$ and $c_{i+1}=(x_i\oplus c_i)·(y_i\oplus c_i)\oplus c_i$. We can construct other operations such as subtraction, multiplication, and division based on the adder. For example, $E[x-y]$ is obtained by $E[x+(-y\left)\right]$, where $(-y)$ is the bit complement of y obtained by $y_i\oplus 1$ for all i-th bits. On the other hand, we examine the comparison. We want to obtain $E[x<?y]$ without decrypting x and y where $x<?y=1$ if $x<y$ and $x<?y=0$ otherwise. We can obtain the logical bit for $x<?y$ as the most significant bit of $x+(-y)$ over ciphertexts here. Similarly, for the equality test, we can compute the encrypted bit $E[x=?y]$.

Adopting these operations of TFHE, we design a secure multi-party CWC. In this paper, we omit the details of TFHE (see, e.g., [20,21]).

We should note that the secrecy of TFHE definitely impacts the security of our scheme. In fact, in our two-party feature selection scheme, the party B sends his/her inputs in an encrypted form to the party A, and A performs the computation of feature selection on the encrypted inputs. If the encrypted inputs could be easily infiltrated, any ingenious devices to secure the scheme would be meaningless.

Therefore, in designing our scheme, it was a matter of course to require our FHE cryptosystem to be IND-CPA secure. In fact, TFHE is known to be IND-CPA secure. Regarding this, we should note the following:

• By definition, encryption with an ID-CPA cryptosystem is probabilistic. That is, the result $E\left[x\right]$ of encryption unpredictably differs every time the encryption is performed. For this reason, by $E\left[x\right|t]$, we denote a ciphertext generated at time t. In particular, the notation of $E\left[x\right|\ast ]$ means that the ciphertext has been generated at a time different from any other encryption events.
• When we consider the IND-CPA security of an FHE cryptosystem, we should note that the way in which the oracle $\mathcal{O}$ generates c with $D\left(c\right)=m_b$ is not unique. For example, the oracle may compute c from two ciphertexts of additive shares of $m_b$, say $E\left[r\right]$ and $E[m_b\oplus r]$, by $c=f_{\oplus }(E\left[r\right],E[m_b\oplus r])$. The IND-CPA security of an FHE cryptosystem should require that $\mathcal{A}$ cannot guess b with effective advantage, no matter how c has been generated. This, however, holds, if the result of performing $E\left[x\right],f_{\oplus }(E\left[x\right],E\left[y\right])$ and $f_{·}(E\left[x\right],E\left[y\right])$ distributes uniformly, and TFHE is known to satisfy this condition.

3. Algorithms

3.1. Baseline Algorithm

We present the baseline algorithm, a privacy-preserving variant of CWC. In this subsection, we consider a two-party protocol, in which a party $\mathsf{B}$ has its private data and outsources CWC computation to another party $\mathsf{A}$, where this protocol can be extended to more than two data owners using the multi-key homomorphic encryption [22]. During the computation, party $\mathsf{A}$ should not gain other information than the number n of positive data, the number m of negative data, and the number k of features. It should be noted that party $\mathsf{B}$ can hide the actual number of data by inserting dummy data and telling $\mathsf{A}$ the inflated numbers n and m. Dummy data can be distinguished by adding an extra bit that indicates that the datum is a dummy if the bit is 1. The values of features and dummy bits of data in each class are encrypted by $\mathsf{B}$’s public key and sent to $\mathsf{A}$.

The baseline algorithm consists of three tasks: computing encrypted bit string $E\left[{\mathbf{B}}_i\right]$, sorting $E\left[{\mathbf{B}}_i\right]$s, and executing feature selection on $E\left[{\mathbf{B}}_i\right]$s. In the baseline algorithm, all inputs are encrypted and they are not decrypted until the computation is completed. Thus, for simplicity, we omit the notation E in the following presentation.

3.1.1. Computing ${\mathbf{B}}_i$

We can compute ${\mathbf{B}}_i[m(p-1)+q]$ by $(x_p\left(F_i\right)\oplus y_q\left(F_i\right))\vee x_p\left(d\right)\vee y_q\left(d\right)$, where $x_p\left(d\right)$ and $y_q\left(d\right)$ represent the dummy bits for data $x_p$ and $y_q$, respectively. $(x_p\left(F_i\right)\oplus y_q\left(F_i\right))$ becomes 0 if $F_i$ is inconsistent for the pair of $x_p$ and $y_q$. Since we want to ignore the influence of dummy data, the part ‘$\vee x_p\left(d\right)\vee y_q\left(d\right)$’ is added to make the whole value 1 (meaning that it is consistent) when one of $x_p$ and $y_q$ is a dummy. It takes $O\left(kmn\right)$ time and space in total.

3.1.2. Sorting $\mathbf{B}$s

We can compute $\parallel {\mathbf{B}}_i\parallel$ in encrypted form by summing up the values in ${\mathbf{B}}_i$ in $O(mnlog(mn\left)\right)$ time (noting that each operation on integers of $log\left(mn\right)$ bits takes $O(log(mn\left)\right)$ time). Instead, we can set an upper bound $b_{\mathit{max}}$ of the bits used to store the consistency measure to reduce the time complexity to $O\left(mn{b}_{\mathit{max}}\right)$.

Then, sorting $\mathbf{B}$s in the incremental order of consistency measures can be accomplished using any sorting network in which comparison and swap are performed in encrypted form, without leaking information about feature ordering. It should be noted that in this approach, the algorithm must spend $\mathsf{\Theta }(mn+logk)$ time to swap (or pretend to swap) two-bit strings and original feature indices of $logk$ bits regardless of whether the two features are actually swapped or not. Because this is the most complex part of our baseline algorithm, we will demonstrate how to improve it. Using an AKS sorting network [35] of size $O(klogk)$, the total time for sorting ${\mathbf{B}}_i$s is $O(mn{b}_{\mathit{max}}+(mn+b_{\mathit{max}}+logk)klogk)$.

In our experiments, we employ a more practical sorting network of Batcher’s odd-even mergesort [36] of size $O\left(k{log}^{2}k\right)$. A simple oblivious radix sort [37] in the $O(klogk)$ algorithm under the assumption that the bit length of each integer is constant was recently proposed.

3.1.3. Selecting Features

Let $(F_{\pi \left(1\right)},\dots ,F_{\pi \left(k\right)})$ be the sorted list of features. We first compute a sequence of bit strings $(Z_2,\dots ,Z_k)$ of length $mn$ each such that $Z_i\left[h\right]={\bigvee }_{j=i+1}^k{\mathbf{B}}_{\pi \left(j\right)}\left[h\right]$ for any $2\le i\le k$ and $1\le h\le mn$; namely, $Z_i$ is the bit array storing the cumulative or each position h for ${\mathbf{B}}_{\pi (i+1)},{\mathbf{B}}_{\pi (i+2)},\dots ,{\mathbf{B}}_{\pi \left(k\right)}$. Note that $Z_i\left[h\right]=0$ indicates that the set $\{F_{\pi (i+1)},F_{\pi (i+2)},\dots ,F_{\pi \left(k\right)}\}$ of features is inconsistent with regard to a pair $(x_p,y_q)$ satisfying $h=m(p-1)+q$, and $\{F_{\pi (i+1)},F_{\pi (i+2)},\dots ,F_{\pi \left(k\right)}\}$ is inconsistent if and only if the bit string $Z_i$ contains 0. See

Table 5. Sorted $\mathbf{B}$s for the example dataset D of Table 3 and the corresponding $Z_i$s.

i	$\mathit{\pi }\left(\mathit{i}\right)$	${\mathit{B}}_{\mathit{\pi }\left(\mathit{i}\right)}$											${\mathit{Z}}_{\mathit{i}}$
1	2	${\mathbf{B}}_2=$	1	0	0	1	0	0	1	1	0	1	$Z_1=$	1	1	1	1	1	1	1	1	1	1
2	4	${\mathbf{B}}_4=$	0	0	1	0	0	1	1	0	1	1	$Z_2=$	1	1	1	1	1	1	1	1	1	1
3	3	${\mathbf{B}}_3=$	0	1	1	0	1	0	1	1	0	1	$Z_3=$	1	1	0	1	1	1	1	0	1	1
4	1	${\mathbf{B}}_1=$	1	1	0	1	1	1	1	0	1	1	$Z_4=$	0	0	0	0	0	0	0	0	0	0

for Zs in our running example. The computation requires $O\left(kmn\right)$ time and space.

We simulate Algorithm 1 on encrypted $\mathbf{B}$s and Zs for feature selection. Furthermore, we use two 0-initialized bit arrays, R of length k and S of length $mn$. $R\left[i\right]$ is meant to store 1 if the i-th feature (in sorted order) is selected. S is used to keep track of the cumulative or for the bit strings of the currently selected features. Namely, $S\left[h\right]$ is set to ${\bigvee }_{\alpha =1}^{\ell }{\mathbf{B}}_{\pi \left(j_{\alpha }\right)}\left[h\right]$ if ℓ features $\{F_{\pi \left(j_1\right)},\dots ,F_{\pi \left(j_{\ell }\right)}\}$ have been selected at this time.

Assume that we are in the i-th iteration of the loop of Algorithm 1. Note that, at this time, F contains features $\{F_{\pi \left(i\right)},F_{\pi (i+1)},\dots ,F_{\pi \left(k\right)}\}$ and currently selected features, and $F\setminus \left\{F_{\pi \left(i\right)}\right\}$ is consistent if ${\bigwedge }_{h=1}^{mn}(Z_i\left[h\right]\vee S\left[h\right])$ is 1. Because we keep $F_{\pi \left(i\right)}$ in F if $F\setminus \left\{F_{\pi \left(i\right)}\right\}$ is inconsistent, the algorithm sets $R\left[i\right]=\neg {\bigwedge }_{h=1}^{mn}(Z_i\left[h\right]\vee S\left[h\right])$. After computing $R\left[i\right]$, we can correctly update S by $S\left[h\right]\leftarrow S\left[h\right]\vee (R\left[i\right]\wedge {\mathbf{B}}_{\pi \left(i\right)}\left[h\right])$ for every $1\le h\le mn$ in $O\left(mn\right)$ time. Therefore, the total computational time is $O\left(kmn\right)$.

3.1.4. Summing Up Analysis

The sorting step takes $O(mn{b}_{\mathit{max}}+(mn+b_{\mathit{max}}+logk)klogk)$ time. Because CWC works with any consistent measure, we do not need to use $\parallel {\mathbf{B}}_i\parallel$ in full accuracy, so we assume that $b_{\mathit{max}}$ is set to be constant. Under the assumption, we obtain the following theorem.

Theorem 1.

For the two-party feature selection problem, we can securely simulate CWC in $O(kmnlogk+k{log}^{2}k)$ time and $O\left(kmn\right)$ space without revealing the private data of the parties under the assumption that TFHE is secure.

Proof. 

According to the discussion above, computing ${\mathbf{B}}_i$ for all features takes $O\left(kmn\right)$ time and space, sorting features takes $O(mn{b}_{\mathit{max}}+(mn+b_{\mathit{max}}+logk)klogk)=O(kmnlogk+k{log}^{2}k)$ time, and selecting features takes $O\left(kmn\right)$ time.

Finally, party $\mathsf{A}$ computes in $O(klogk)$ time an integer array P with $P\left[h\right]=R\left[h\right]·\pi \left(h\right)$, which stores the original indices of selected features. In the outsourcing scenario, party $\mathsf{A}$ simply sends P to party $\mathsf{B}$ as the result of CWC. In the joint computing scenario, party $\mathsf{A}$ randomly shuffles P to conceal $\pi$ to $\mathsf{B}$. As a result, we can securely simulate CWC in $O(kmnlogk+k{log}^{2}k)$ time and $O\left(kmn\right)$ space.    □

3.2. Improvement of Secure CWC

Sorting is a major bottleneck for private CWC. The reason for this is that pointers cannot be moved across ciphertexts. For example, consider the case of a secure integer. Let the variables x and y contain integers a and b, respectively. In this case, by performing the secure operation $a<?b$, the result is obtained as $a<?b=c\in \{0,1\}$. Using this logical bit c, we can swap the values of x and y in $O\left(1\right)$ time, satisfying $x<y$ by the secure operation $x\leftarrow c·a+\overline{c}·b$ and $y\leftarrow \overline{c}·a+c·b$.

In the case of CWC, however, each integer i of feature $F_i$ is associated with the bit string ${\mathbf{B}}_i$. Since any x cannot be decrypted, we cannot swap the pointers appropriately. Therefore, the baseline algorithm swaps ${\mathbf{B}}_i$ explicitly. As a result, the computation time for sorting increases to $O\left(mnk{log}^{2}k\right)$. Our main contribution is to improve this complexity to $O(mnk+k{log}^{2}k)$ by reducing the cost for such explicit sorting.

Based on the FHE, we propose the improved secure CWC (Algorithm 2), which reduces the time complexity to $O(mnk+k{log}^{2}k)$. An example run of Algorithm 2 is illustrated in Figure 1. As shown in this example, the party $\mathsf{A}$ can securely sort k randomized features in $O\left(k{log}^{2}k\right)$ time using a suitable sorting network, and then, according to the result of sorting, $\mathsf{A}$ swaps each associated bit string of length $nm$ in $O\left(kmn\right)$ time. Following this preprocessing, the parties securely obtain minimal consistent features by decrypting the output of CWC. Finally, we have the following result.

Algorithm 2 Improved secure CWC between parties $\mathsf{A}$ and $\mathsf{B}$

1: Preprocessing: Party $\mathsf{A}$ has $E_{\mathsf{B}}\left[\mathcal{F}\right]=E_{\mathsf{B}}[{\mathcal{F}}_1,\dots ,{\mathcal{F}}_k]$ for ${\mathcal{F}}_i=(F_i,\parallel {\mathbf{B}}_i\parallel ,{\mathbf{B}}_i)$ encrypted with party $\mathsf{B}$’s public key, where each datum x is encrypted at time 0 as $E_{\mathsf{B}}\left[x\right|0]$. 2: Party $\mathsf{A}$: Generates $r_i$ for $i=1,\dots ,n$ uniformly at random. Sends $(E_{\mathsf{B}}[{\mathbf{B}}_i+r_i|1],E_{\mathsf{A}}\left[r_i\right|1])$ for $i=1,\dots ,n$. 3: Party $\mathsf{A}$: Calculates $E_{\mathsf{B}}\left[i\right|2]$ for $i=1,\dots ,n$. Securely sorts $(E_{\mathsf{B}}\left[F_i\right|0],E_{\mathsf{B}}[\parallel {\mathbf{B}}_i\parallel \left|0\right],E_{\mathsf{B}}\left[i\right|2])$ for $i=1,\dots ,n$ in increasing order of $\parallel {\mathbf{B}}_i\parallel$. As a result, obtains $(E_{\mathsf{B}}\left[F_{i_j}\right|3],E_{\mathsf{B}}[\parallel {\mathbf{B}}_{i_j}\parallel \left|3\right],E_{\mathsf{B}}\left[i_j\right|3])$ for $j=1,\dots ,n$. Generates a permutation $\pi \in {\mathfrak{S}}_n$ uniformly at random and memorizes it. Sends $(E_{\mathsf{B}}\left[i_{\pi \left(1\right)}\right|3],\dots ,E_{\mathsf{B}}\left[i_{\pi \left(n\right)}\right|3])$. 4: Party $\mathsf{B}$: Decrypts $(i_{\pi \left(1\right)},\dots ,i_{\pi \left(n\right)})$. Generates $r_i^{\prime }$ for $i=1,\dots ,n$ uniformly at random. Sends $(E_{\mathsf{B}}[{\mathbf{B}}_{i_{\pi \left(j\right)}}+r_{i_{\pi \left(j\right)}}+r_{i_{\pi \left(j\right)}}^{\prime }|4],E_{\mathsf{A}}[r_{i_{\pi \left(j\right)}}+r_{i_{\pi \left(j\right)}}^{\prime }|4])$ for $j=1,\dots ,n$. 5: Party $\mathsf{A}$: Decrypts $r_{i_{\pi \left(j\right)}}+r_{i_{\pi \left(j\right)}}^{\prime }$ for $j=1,\dots ,n$. Obtains $E_{\mathsf{B}}\left[{\mathbf{B}}_{i_{\pi \left(j\right)}}\right|5]$$j=1,\dots ,n$. Obtains $E_{\mathsf{B}}\left[{\mathbf{B}}_{i_j}\right|5]$$j=1,\dots ,n$ through permutation by ${\pi }^{-1}$. 6: Party $\mathsf{A}$: Simulates CWC for resulting $E_{\mathsf{B}}\left[\mathcal{F}\right]$.

Theorem 2.

Algorithm 2 can simulate CWC in $O(kmn+k{log}^{2}k+klogklogmn)$ time and $O\left(kmn\right)$ space under the assumption that FHE executes each bit operation in $O\left(1\right)$ time.

Proof. 

Compared to the baseline, additional space is required for $\pi$ and $r_i$ and $r_i^{\prime }$. Thus, the space complexity remains $O\left(kmn\right)$. For the time complexity, the main task is to sort k-triple $(F_i,\parallel {\mathbf{B}}_i\parallel ,{\mathbf{B}}_i)$ in the increasing order of $\parallel {\mathbf{B}}_i\parallel$. The improved algorithm sorts only the pairs $x_i=(F_i,\parallel {\mathbf{B}}_i\parallel )$ of integers, where the size of $x_i$ is $O(logk+logmn)$ bits. For each $x_i,x_j$, we can check if $\parallel {\mathbf{B}}_i\parallel \le \parallel {\mathbf{B}}_j\parallel$ in $O(logmn)$ time and we can swap them in $O(logk+logmn)$ time using homomorphic operations in FHE. It follows that the time for sorting all $x_i$$(i=1,\dots ,k)$ is $O(klogk(logk+logmn\left)\right)$ time. After sorting the pairs, the algorithm moves all ${\mathbf{B}}_i$ to the correct positions according to the rank of $x_i$$(i=1,\dots ,k)$. This cost is $O\left(kmn\right)$. Therefore, time complexity is $O(kmn+k{log}^{2}k+klogklogmn)$.    □

Theorem 3.

Algorithm 2 is secure under the assumption that the employed FHE is IND-CPA secure.

Proof. 

We show the security by constructing simulators for parties $\mathsf{A}$ and $\mathsf{B}$, respectively.

$\mathsf{B}$’s view (what $\mathsf{B}$ can obtain from $\mathsf{A}$) is the following:

• $(E_{\mathsf{B}}[{\mathbf{B}}_i+r_i|1],E_{\mathsf{A}}\left[r_i\right|1])$ for $i=1,\dots ,n$;
• $E_{\mathsf{B}}\left[i_{\pi \left(1\right)}\right|3],\dots ,E_{\mathsf{B}}\left[i_{\pi \left(n\right)}\right|3]$.

Their probability distributions are uniform and independent of each other. Hence, the simulator for $\mathsf{B}$ can replace them with

• $(E_{\mathsf{B}}[{\mathbf{B}}_i+r_i^{\prime \prime }|6],E_{\mathsf{A}}\left[r_i^{\prime \prime }\right|6])$ for $i=1,\dots ,n$ and $r_i^{\prime \prime }$, which are selected uniformly at random;
• $E_{\mathsf{B}}\left[{\pi }^{\prime }\left(1\right)\right|6],\dots ,E_{\mathsf{B}}\left[{\pi }^{\prime }\left(n\right)\right|6]$ for ${\pi }^{\prime }\in {\mathfrak{S}}_n$, which is selected uniformly at random.

Note that, even if an adversary knows ${\mathbf{B}}_i$, it is computationally impossible to distinguish between $E_{\mathsf{B}}[{\mathbf{B}}_i+r_i^{\prime \prime }|6]$ and $E_{\mathsf{B}}\left[r_i^{\prime \prime }\right|6]$ by the IND-CPA security of the cryptosystem $E_{\mathsf{B}}$.

Next, we construct a simulator Sim for the party $\mathsf{A}$. Although what $\mathsf{A}$ can obtain from $\mathsf{B}$ is

$$\left\{\left.\left(E_{\mathsf{B}}[{\mathbf{B}}_{i_{\pi \left(j\right)}}+r_{i_{\pi \left(j\right)}}+r_{i_{\pi \left(j\right)}}^{\prime }|4],E_{\mathsf{A}}[r_{i_{\pi \left(j\right)}}+r_{i_{\pi \left(j\right)}}^{\prime }|4]\right)\right|j=1,\dots ,n\right\}$$

this is equivalent to $\{E_{\mathsf{B}}\left[{\mathbf{B}}_{i_j}\right|5]\mid j=1,\dots ,n\}$ after decryption and permutation.

On the other hand, the sequence $(i_1,\dots ,i_n)$ is not explicitly given to $\mathsf{A}$, and $\mathsf{A}$ recognizes it through the alignment between

• $(E_{\mathsf{B}}[\parallel {\mathbf{B}}_{i_1}\parallel \left|3\right],\dots ,E_{\mathsf{B}}[\parallel {\mathbf{B}}_{i_n}\parallel \left|3\right])$ and
• $(E_{\mathsf{B}}\left[{\mathbf{B}}_{i_1}\right|5],\dots ,E_{\mathsf{B}}\left[{\mathbf{B}}_{i_n}\right|5])$.

Therefore, we define $\mathsf{A}$’s view to be

$${\mathtt{View}}_{\mathsf{A}}=\left\{\left.\left(E_{\mathsf{B}}[\parallel {\mathbf{B}}_{i_j}\parallel \left|3\right],E_{\mathsf{B}}\left[{\mathbf{B}}_{i_j}\right)\left|5\right]\right)\right|j=1,\dots ,n\right\}$$

with $\parallel {\mathbf{B}}_{i_1}\parallel \le \cdots \le \parallel {\mathbf{B}}_{i_n}\parallel$.

On the other hand, we define the view that $\mathtt{Sim}$ should generate as follows. While $\mathsf{A}$ can generate $\{E_{\mathsf{B}}[\parallel {\mathbf{B}}_{i_j}\parallel \left|3\right]\mid j=1,\dots ,n\}$ with $\parallel {\mathbf{B}}_{i_1}\parallel \le \cdots \le \parallel {\mathbf{B}}_{i_2}\parallel$, $\mathsf{A}$ needs $\mathsf{B}$’s cooperation to generate $\{E_{\mathsf{B}}\left[{\mathbf{B}}_{i_j}\right|5]\mid j=1,\dots ,n\}$. Without $\mathsf{B}$’s cooperation, $\mathtt{Sim}$ selects ${\pi }^{\prime }\in {\mathfrak{S}}_n$ uniformly at random, and generates its own view to be

$${\mathtt{View}}_{\mathtt{Sim}}=\left\{\left.\left(E_{\mathsf{B}}[\parallel {\mathbf{B}}_{i_j}\parallel \left|3\right],E_{\mathsf{B}}\left[{\mathbf{B}}_{{\pi }^{\prime \prime }\left(j\right)}\right|\ast ]\right)\right|j=1,\dots ,n\right\}.$$

Sim can compute $E_{\mathsf{B}}\left[{\mathbf{B}}_{{\pi }^{\prime \prime }\left(j\right)}\right|\ast ]$ from $E_{\mathsf{B}}\left[0\right|\ast ]$ and $E_{\mathsf{B}}\left[{\mathbf{B}}_{{\pi }^{\prime \prime }\left(j\right)}\right|0]$ taking advantage of the homomorphic property of the encryption system $E_{\mathsf{B}}$.

Furthermore, we define a distinguisher $\mathcal{D}$ as a PPT Turing machine that tries to distinguish between ${\mathtt{View}}_{\mathsf{A}}$ and ${\mathtt{View}}_{\mathtt{Sim}}$ on the input of $\left\{\right(E_{\mathsf{B}}[\parallel {\mathbf{B}}_i\parallel \left|0\right],E_{\mathsf{B}}\left[{\mathbf{B}}_i\right|0])\mid i=1,\dots ,n\}.$

When we let $Pr[Y=\mathsf{A}\mid X=\mathsf{A}]=1/2+{\alpha }_1$ and $Pr[Y=\mathtt{Sim}\mid X=\mathtt{Sim}]=1/2+{\alpha }_2$, the advantage of $\mathcal{D}$ is defined as ${\alpha }_1+{\alpha }_2$.

We show that, if $\mathcal{D}$’s advantage $\alpha$ is not negligible, we can construct a PPT attacker $\mathtt{Attck}$ that can break the IND-CPA security of the encryption system $E_{\mathsf{B}}$ with a non-negligible advantage. Our attacker $\mathtt{Attck}$ plays the IND-CPA game, exploiting an oracle ${\mathcal{O}}_{\mathrm{IND}}$ as follows:

1.

$\mathtt{Attck}$ generates ${\mathbf{B}}_1,{\mathbf{B}}_2$ with $\parallel {\mathbf{B}}_1\parallel \le \parallel {\mathbf{B}}_2\parallel$;

2.

$\mathtt{Attck}$ lets $x_1=\parallel {\mathbf{B}}_1\parallel$ and $x_2=\parallel {\mathbf{B}}_2\parallel$ and throws a query $(x_1,x_2)$ to ${\mathcal{O}}_{\mathrm{IND}}$;

3.

${\mathcal{O}}_{\mathrm{IND}}$ selects $i\in \{1,2\}$ uniformly at random and sends $c=E_{\mathsf{B}}\left[x_i\right|-1]$ to $\mathtt{Attck}$;

4.

$\mathtt{Attck}$ initializes $\mathcal{D}$ by inputting $(E_{\mathsf{B}}[\parallel {\mathbf{B}}_1\parallel \left|0\right],E_{\mathsf{B}}[\parallel {\mathbf{B}}_1\parallel \left|0\right]),(E_{\mathsf{B}}[\parallel {\mathbf{B}}_2\parallel \left|0\right],E_{\mathsf{B}}[\parallel {\mathbf{B}}_2\parallel \left|0\right])$;

5.

First query.$\mathtt{Attck}$ throws to $\mathcal{D}$ the query: $(E_{\mathsf{B}}[\parallel {\mathbf{B}}_2\parallel \left|2\right],c),(E_{\mathsf{B}}[\parallel {\mathbf{B}}_2\parallel \left|2\right],E_{\mathsf{B}}\left[{\mathbf{B}}_2\right|2])$;

6.

If $\mathcal{D}$ replies with $\mathsf{A}$, $\mathtt{Attck}$ outputs 1 and terminates.

7.

Second query.$\mathtt{Attck}$ generates $c^{\prime }$ by adding $E_{\mathsf{B}}\left[0\right|3]$ to c. Note that ${\mathcal{D}}_{\mathsf{B}}\left(c^{\prime }\right)={\mathcal{D}}_{\mathsf{B}}\left(c\right)$ holds. $\mathtt{Attck}$ throws to $\mathcal{D}$ the query: $(E_{\mathsf{B}}[\parallel {\mathbf{B}}_1\parallel \left|3\right],E_{\mathsf{B}}\left[{\mathbf{B}}_2\right|3]),(E_{\mathsf{B}}[\parallel {\mathbf{B}}_2\parallel \left|3\right],c^{\prime })$.

8.

If $\mathcal{D}$ replies with $\mathtt{Sim}$, $\mathtt{Attck}$ outputs 1 and terminates.

9.

$\mathtt{Attck}$ outputs 2.

We evaluate $\mathtt{Attck}$’s advantage as follows. We assume ${\mathcal{D}}_{\mathsf{B}}\left(c\right)=x_1$. The probability of this case is $1/2$. The probability that $\mathcal{D}$ replies with $\mathsf{A}$ to the first query or $\mathcal{D}$ replies with $\mathtt{Sim}$ to the second query is

$$\frac{1}{2}+{\alpha }_1+\frac{1}{2}+{\alpha }_2-(\frac{1}{2}+{\alpha }_1)(\frac{1}{2}+{\alpha }_2)=\frac{3}{4}+\frac{\alpha }{2}-{\alpha }_1{\alpha }_2\ge \frac{3}{4}+\frac{\alpha }{4},$$

since the first and second queries are mutually independent.

When assuming $D_B\left(c\right)=x_2$, we see that $Pr\left[\mathcal{D}\mathrm{outputs}\mathtt{Sim}\mathrm{at}\mathrm{the}\mathrm{first}\mathrm{query}\right]-1/2$ is negligible. Otherwise, $\mathcal{D}$ can be used as an attacker to break the IND-CPA security of $E_B$. Therefore, $Pr\left[\mathrm{Attck}\mathrm{outputs}2\right]-1/4$ is negligible. Consequently, we have

$$Pr\left[\mathtt{Attck}’\mathrm{s}\mathrm{guess}\mathrm{is}\mathrm{right}\right]\ge \frac{1}{2}\left(\frac{3}{4}+\frac{\alpha }{4}\right)+\frac{1}{2}·\frac{1}{4}=\frac{1}{2}+\frac{\alpha }{8}.$$

Since we assume that $\alpha$ is not negligible, neither is $\alpha /4$.    □

4. Experiments

We implemented the baseline and improved algorithms for secure CWC in C++ using the TFHE library (https://tfhe.github.io/tfhe (accessed on 28 January 2021)).

The experiments were carried out on a machine equipped with an Intel Core i7-6567U (3.30 GHz) processor and 16GB of RAM. In the following, m (resp. n) is the number of positive (resp. negative) data and k is the number of features.

Table 6. Running time (s) of baseline algorithm (naive secure CWC). Task 1: computing ${\mathbf{B}}_i$s. Task 2: sorting ${\mathbf{B}}_i$s. Task 3: feature selection.

k	$\mathbf{mn}$	Task 1	Task 2	Task 3
10	100	60.3	835.8	111.9
	500	300.5	4252.4	558.1
	1000	601.4	8867.0	1114.2
50	100	301.8	6292.6	589.3
	500	1502.9	30,364.6	2941.0
	1000	3007.0	62,124.6	5919.7
100	100	603.7	16,148.5	1179.0
	500	3005.9	76,315.2	5952.5
	1000	6014.1	154,143.5	11,867.0

summarizes the running time of the baseline algorithm (naive implementation of Algorithm 1 using TFHE) for random data generated for $k\in \{10,50,100\}$ and $mn\in \{100,500,1000\}$. The complexity analysis shows that the running time increases in proportion to $mn$. This experimental result confirms this in real data. The table clearly shows that the sorting process is the bottleneck.

Table 7. Running time (s) of baseline and improved algorithms. Here, ‘baseline’ is same as Task 2 in Table 6 (i.e., the bottleneck); ‘improved:’ is the running time of corresponding task in the improved algorithm, where ‘sorting’ and ‘other tasks’ are the details.

k	$\mathbf{mn}$	Baseline	Improved:	Sorting	Other Tasks
10	100	835.8	203.7	69.4	134.2
	500	4252.4	286.2	89.5	196.6
	1000	8867.0	302.5	98.9	203.6
100	100	16,148.5	3311.2	1865.5	1445.7
	500	76,315.2	4601.9	2647.7	1954.1
	1000	154,143.5	4671.4	2660.8	2010.5

compares the running time of preprocessing in the baseline and improved algorithms. According to the results, the proposed algorithm significantly improves the bottleneck in naive CWC for secure computing. We should note that the baseline and improved algorithms both compute exactly the same solution as the CWC on plaintexts. We also show the details of the improved algorithm: ‘sorting’ means the time for sorting of the triples $(F_i,|\left|{\mathbf{B}}_i\right||,i)$ of integers; ‘other task’ means the time for remaining tasks, including generating/adding/subtracting random noise $r_i$, moving ${\mathbf{B}}_i$, decrypting integers, etc.

Table 8. Running time (s) of improved algorithm for real data in UCI Machine Learning Repository.

Dataset	k	$\mathit{mn}$	Time	Sorting	Other Tasks
Letter	16	196	252.2	80.6	171.4
Breast Cancer	10	2464	312.6	103.1	209.5
Covertype	54	979	1653.5	836.9	816.6

displays the running time of the improved algorithm for real data available from the UCI Machine Learning Repository (https://archive.ics.uci.edu/ml/index.php (accessed on 26 January 2022)), because, since these datasets contain more than three feature/class values, we treated them as a binary classification between one feature/class and the other.

We demonstrated that the proposed algorithm works well for real-world multi-level feature selection problems. We only evaluated the running time in this experiment, but the relevance of the extracted features is guaranteed because the secure CWC algorithm produces the same solution as the original [8].

5. Conclusions

On the basis of fully homomorphic encryption, we proposed a faster private feature selection algorithm that allows us to securely compute functional features from distributed private datasets. Our algorithm can simulate the original CWC algorithm, which chooses favorable features by sorting. In addition to the improvement in computational complexity, the proposed algorithm solves the private feature selection problem in practical time for a variety of real data. One of the remaining challenges is to improve sorting at a lower cost because CWC does not always require exact sorting. Then, ambiguous sorting possibly reduces the computation time, maintaining solution quality. At this time, the proposed algorithm is not applicable to real numbers for feature value. This is because TFHE [21] is not suitable for floating-point operations. Extending the TFHE library to enable secure feature selection for real-valued data is a future challenge.

A well-known feature selection method is to filter features by computing Gini impurity scores [14]. In this method, the optimal threshold for filtering is determined by the order of sorting of each feature. However, since sorting is time-consuming even for secret sharing-based MPC [38], a simpler method of determining the threshold by calculation has been proposed [30], and its effectiveness has been confirmed by experiments. On the other hand, this study focuses on consistency measure-based feature selection. As we mentioned previously, the consistency measure-based method has been confirmed to have advantages over other methods. This study proposes the first secure protocol that enables consistency measure-based feature selection in practical time.

We next compare our proposal with other methods in the framework of private feature selection. A secret sharing-based MPC using the distributed secure sum is proposed [29]. In this method, it is known that statistical information about the data is leaked during the computation. The authors in [30] propose an honest-majority three-party protocol that improves on the drawback of [29]. This method is fast but requires at least two trusted parties. In addition, [30] considered both semi-honest and malicious adversaries, but in this study, the parties are assumed to be semi-honest. A feature selection using homomorphic encryption was proposed in [26]. This method uses only the additive homomorphic property in a two-party model, which limits its computational power. Therefore, statistical information about the data is leaked because partial decryption is required during communication between the parties. Moreover, this protocol has not been implemented. The recent approach by [39] is not based on cryptography and does not provide a formal privacy guarantee, and it leaks information through the disclosure of intermediate representations. Although our method is inferior to secret sharing-based MPC in terms of practical computing time, it can handle feature selection from many data owners, even in situations where only they themselves can be trusted, and does not leak information during computation.

In addition, we discuss future issues and prospects related to secret computation with FHE, which were not discussed in detail in this study. First, this study assumes that the data are consistent, which cannot be applied to real-world data. If the data are inconsistent, e.g., when the data are merged, there are two entries with the same feature values they but are classified into different classes, the party can ignore these entries without decoding. Since the outsourced party can perform a comparison of two integers without decoding, they can use the encrypted logical bit to change the class label of these irrelevant entries to a special value, effectively ignoring them.

Next, we consider how to speed up the computation of real numbers using FHE. CKKS [40] and TFHE are the current state-of-the-art methods for computing real numbers on FHE. CKKS speeds up arithmetic operations on real numbers by converting reals to integers through scaling, performing arithmetic operations on the integers, and then converting the results back to reals. However, CKKS has the drawback that it cannot compute nonlinear functions (e.g., ReLU) or perform comparison operations, making it difficult to apply to ML. On the other hand, TFHE can evaluate comparison operations and NAND circuits, making all computations theoretically possible. Unfortunately, the runtime on TFHE has an overhead of approximately 10,000 times that on the plaintext, and thus the GPGPU-based architecture is currently being studied for speedup. Currently, the fastest implementation is around 20 times faster than algorithms on CPUs [41]. Thus, the application and speedup of TFHE to ML is a promising research area for the future.

In conclusion, it should be noted that with the development of FHE, practical algorithms for more challenging problems such as large-scale genome analysis (GWAS) [42,43] and deep learning [44,45,46] are emerging.