IoT Application-Layer Protocol Vulnerability Detection using Reverse Engineering

Abstract

Fuzzing is regarded as the most promising method for protocol vulnerabilities discovering in network security of Internet of Things (IoT). However, one fatal drawback of existing fuzzing methods is that a huge number of test files are required to maintain a high test coverage. In this paper, a novel method based on protocol reverse engineering is proposed to reduce the amount of test files for fuzzing. The proposed method uses techniques in the field of protocol reverse engineering to identify message formats of IoT application-layer protocol and create test files by generating messages with error fields according to message formats. The protocol message treated as a sequence of bytes is assumed to obey a statistic process with change-points indicating the boundaries of message fields. Then, a multi-change-point detection procedure is introduced to identify change-points of byte sequences according to their statistic properties and divide them into segments according to their change-points. The message segments are further processed via a position-based occurrence probability test analysis to identify keyword fields, data fields and uncertain fields. Finally, a message generation procedure with mutation operation on message fields is applied to construct test files for fuzzing test. The results show that the proposed method can effectively find out the message fields and significantly reduce the amount of test files for fuzzing test.

1. Introduction

Fuzzing is a widely used security technique for discovering vulnerability in network protocol by sending a series of test files with random or fault data to software system implementing specific protocol and observing software exceptions to detect vulnerabilities within the protocol.

Currently, there exist mainly two kinds of fuzzing techniques, i.e., mutation-based and generation-based fuzzing [1]. The former generates test files by injecting random or fault data into sample messages (message is the basic data unit exchanged between processes of application-layer protocol ), while the latter constructs fault-injected messages as test files based on specific protocol specification. The mutation-based fuzzing emits a fatal problem that too many fault-injected messages are required to maintain a high test coverage, such as FileFuzz (http://www.securiteam.com/tools/5PP051FGUE.html) and SPIKEfile (https://www.ee.oulu.fi/research/ouspg/SPIKEfile). However, the amount of fault-injected files is ${256}^L$, where L is the power of sample message’s length, and it would take tremendously long time to handle so great amount of fault-injected test files especially when L is large. Actually, a protocol’s software system parses inputs by considering their formats and treats any files which does not obey the rule of its format as invalid input, in which case a software system will throw an error and quit before it reaches the fault segment(s). Therefore, many of fault-injected test files are not necessary for successful fuzzing test. The generation-based fuzzing generates test files by considering the format of input messages, such as PROTOS (https://www.ee.oulu.fi/roles/ouspg/Protos). One advantage of such fuzzing tools is that it reduces the number of test files greatly and introduces nearly no sacrifice on test coverage [2]. However, one has to figure out the message formats and configure generation-based fuzzer accordingly. Currently, the message formats are mainly collected or analyzed in a manual way, which is a time-consuming and error-prone process. To address these issues, protocol reverse engineering [3] is introduced to obtain protocol specification automatically. The protocol specification including message format a set of rules that describe or model a network protocol. Then a field-based fault-injected message generation procedure conducted by the message format is applied to create fuzzing test files.

Protocol message is treated as a byte sequence which could be divided into a sequence of fields. A keyword field usually holds a command, operator or state code of protocol, while a data field is variable subsequence whose content is always changeable, such as the value of some parameters of communication. Generally, message format is recovered by identifying all fields in byte sequences. However, it is hard to locate the boundary of fields and a great challenge to identify fields in message, since a priori information about them is usually not available. The byte sequence of protocol message is supposed to obey an underlying stochastic process in which different fields have their own distribution of symbols and change-points are the boundaries of fields. Apparently, each change-point implies an end point of one field and a start point of another field. With these assumption, our goal of field boundary detection is essentially the problem of multi-change-point detection. This problem can be addressed using change-point detection [4] widely used in time series analysis. When change-points are localized successfully, messages are divided into field sequences. However, the type of fields are still uncertain. Thus, a further inference procedure, named position-based occurrence probability test analysis, is proposed to determine field type( keyword fields and data fields). Firstly, fields with approximate zero-probability distribution are classified as data fields. Then, the rest ones are further processed in a position-based statistic test. Specifically, a reference position would be selected for every field, and each field are tested by binomial test to make sure whether their positions are equal to the reference position with probability 1 given a significance level $\alpha$. The fields passing these tests are chosen as keyword fields, while the rest ones are considered as uncertain fields.

2. Related Work

Recently, the security and privacy issues for Internet of Things have attracted a lot of research interests [5,6,7,8,9,10]. In particular, the analysis of applications and protocols in real-time network traffic monitoring is a fundamental and critical building block in network management and security systems for IoT infrastructures [11,12,13,14,15]. In this part, we review the recent works in application-layer network protocol vulnerability analysis and detection.

Fuzzing helps protocol vulnerabilities detection to gain higher benefit-to-cost ratio with no or less increasing in computing complexity. It aims to reveal bugs in protocols which would be exploited by adversary to launch attack or activate their malicious code. Currently, research on network protocol fuzzing test is a heat topic in network security. AutoFuzz [16] identified the variable parts of sample messages and fuzzes protocol implementation by sending messages with invalid symbols or messages. AspFuzz [17] leveraged the accessible protocol specifications on RFCs (Request for Comments) to generate fault-injected messages for test files. Then, AspFuzz sent both anomalous and reordered messages to discover vulnerabilities. SecFuzz [18] focused on fuzzing security protocol implementation, but it did not consider the specification of target protocol as well. Zhao et al. [19] used regression finite state machine to infer a state transition diagram of protocol so as to reveal potential vulnerabilities in wireless protocols.

In recent years, a range of works about protocol reverse engineering [20] have been published. Early in 2005, Marshall A. Beddoe held the protocol informatics project [21] and applied bio-informatics algorithms to identify the fields in packets based on alignment algorithms. Cui et al. went further than Beddoe and presented Discoverer [22] to recover protocol message format using both sequence aligning and recursively clustering algorithm. However, Discoverer need some a priori information about the delimiters used by protocol, such as space and comma, which is used to help tokenization, i.e., breaking message into token sequence. Recently, Tao et al. [23] combines hierarchical clustering algorithm, multi-sequence alignment and Bayesian decision model to determine the field boundary of binary protocol in bit granularity. Chen et al. [24] introduce deep learning algorithm to analyze mobile applications. Xiao et al. [25] propose a method based on heuristic rule to reverse analysis of the incomplete flow. In our approach, we make no assumption about the delimiters. We treat the byte sequence of message as a stochastic process and detect field boundaries according to their statistical properties.

As a paralleled method to understand the unknown protocols, binary analysis-based techniques, such as Polyglot [26], Tupni [27], AutoFormat [28], Prospex [29] Dispatcher [30] and so on, also draw much research attention. They are practical in some special scenarios where binary codes are available and executable in a specific sandbox-like environment. Moreover, binary analysis method would fail if programs make use of some confusion techniques like obfuscation to keep themselves away from being reverse-engineered.

As in many other security application domains [31,32,33,34,35,36], data mining and machine learning techniques have been widely adopted in the domain of IoT security and IoT traffic analysis. One of the key challenges is the data privacy problem, especially in collaborative and cloud-based learning scenarios. Several recent studies have proposed novel data privacy preserving approaches for addressing the problem [37,38,39,40,41,42].

3. Problem Formulation

Suppose that the alphabet used by protocol messages is defined as $\mathsf{\Sigma }=\left\{0\mathtt{x}00,0\mathtt{x}01,0\mathtt{x}02,\dots ,0\mathtt{x}FF\right\}$. A string $\omega$ is defined as a finite set of ordered letters in $\mathsf{\Sigma }$. That is $\omega =a_1{a}_2\dots a_n$ ( $a_1,a_2,\dots ,a_n\in \mathsf{\Sigma }$). All strings over alphabet $\mathsf{\Sigma }$ forms a super set ${\mathsf{\Sigma }}^{\ast }$. As a basic data unit used by IoT protocol, protocol message m is essentially strings made up of a sequence of message fields. Thus, we mark message field as $\varrho \in {\mathsf{\Sigma }}^{\ast }$.

In this paper, a protocol message is assumed to be a byte sequence undergoing hidden statistical process, denoted as $\mathsf{\Theta }$, whose statistical feature would shift on and on when the byte sequence goes from one message field to another. As $\mathsf{\Theta }$ passes from one field (${\varrho }_i$) to another (${\varrho }_j$), the statistical characteristic would change significantly. Thus, a change-point would occur just in the boundary of two different message fields. Inspired by this observation, the problem of message field identification can be transformed to be a change-point detection issue in the statistical process undergone by protocol message.

Given a string ${\omega }_o=\overline{x_1\dots x_n}$, a q-length prefix of the last letter (i.e., $x_n$) in ${\omega }_o$ is marked as $\mathfrak{T}({\omega }_o,q)$, while the set of such prefixes whose lengths are no longer than Q in ${\omega }_o$ is marked as $\mathcal{T}(x_n,Q)$. For instance, $\mathfrak{T}(\overline{x_1\dots x_4},2)=\overline{x_2{x}_3}$, $\mathfrak{T}(\overline{x_1\dots x_4},3)=\overline{x_1{x}_2{x}_3}$, and $\mathcal{T}({\omega }_o,Q)=\{\mathfrak{T}({\omega }_o,q):1\le q\le min(Q,n-1),Q\in \mathbb{R}\}$.

The prefix conditional probability of $\overline{x_1\dots x_n}$ is defined as

$$\begin{array}{cc}\hfill p_n& =P\left(x_n|\mathcal{T}(\overline{x_1\dots x_n},n-1)\right).\hfill \end{array}$$

(1)

Let $m=\overline{x_1{x}_2{x}_3\dots }$ to be a Q-order Markov process. Then, the likelihood of $x_n$ given $\overline{x_1,\dots ,x_{n-1}}$ is

$$\begin{array}{c}\hfill P(x_n|\overline{x_1,\dots ,x_{n-1}})=P(x_n|\overline{x_{n-Q}{x}_{n-Q+1}\dots x_{n-1}}),\end{array}$$

(2)

where $Q\in \mathbb{R}$ and $n>Q$.

Suppose that the byte sequence of protocol message obeys Q-order Markov process, then Equation (1) would be rewritten as follows.

$$\begin{array}{cc}\hfill p_n& =P\left(x_n|\mathcal{T}(\overline{x_1\dots x_n},Q)\right)\hfill \\ & ={\displaystyle \sum _{q=1}^{min\{Q,n-1\}}}{\omega }_{q}P(x_n|\mathfrak{T}(\overline{x_1\dots x_n},q)),\hfill \end{array}$$

(3)

where ${\omega }_q$ is the weight of $P(x_n|\mathfrak{T}(\overline{x_1\dots x_n},q))$. Essentially, ${\omega }_q$ can be regarded as the importance of $\mathfrak{T}(\overline{x_1\dots x_n},q)$ for predicting the context of $x_n$.

The larger q is, the more important it is for $\mathfrak{T}(\overline{x_1\dots x_n},q)$ in predicting the context of $x_n$. For instance, it is much more important for $P(“e”|“xampl”)$ than $P(“e”|“pl”)$ to foresee that the context of “e” is “example” instead of “multiple”. As a result, the weight of ${\omega }_q$ in this paper is defined as

$$\begin{array}{cc}\hfill {\omega }_q& =\frac{q^2}{{\displaystyle \sum _{q^{\prime }=1}^{min\{Q,n-1\}}}{(q^{\prime })}^2},n>1.\hfill \end{array}$$

(4)

Additionally, $P(x_n|\mathfrak{T}(\overline{x_1\dots x_n},q))$ is calculated by

$$\begin{array}{c}\hfill P(x_n|\mathfrak{T}(\overline{x_1\dots x_n},q))=\frac{\nu (\overline{x_{n-q}\dots x_{n-1}{x}_n})}{\nu (\overline{x_{n-q}\dots x_{n-1}})},\end{array}$$

(5)

where $\nu (\omega )$ is the frequency of $\omega$ in training dataset $\mathcal{D}$.

As shown in Figure 1, the prefix conditional probability of $\overline{x_1\dots x_n}$ would be very high when $x_n$ and $\mathfrak{T}(x_n,q)$ locate in the same field, otherwise it would be low.

3.1. Minmax Formulation for Field Detection

There exist mainly two formulations of change-point detecting problem: Bayesian formulation and minmax formulation. The Bayesian formulation [43] assumes that the change-point $\gamma$ obeys a prior distribution which is known in prior, while the minmax formulation [44] supposes that the change-point as well as its statistical distribution are unknown to us.

In this paper, the statistical distribution of change-points in protocol message is unknown. As a result, the change-point detection problem should be represented in minmax formulation. Page [45] proposed a cumulative sum (CUSUM) algorithm to implement an optimal solution to minmax formulated problems. Accordingly, a CUSUM-LIKE algorithm is proposed to search fro multiple change-points in this paper. Since the statistic feature of message fields is unknown in prior, the likelihood ratio from post-change probability to pre-change probability, denoted as $L(X_n)$ cannot be calculated directly by $L(X_n)=f_{{\varrho }_1}(X_n)/f_{{\varrho }_0}(X_n)$. Thus, $L(X_n)$ is replaced with a new metric in this paper as

$$\begin{array}{c}\hfill C_n=-log(\frac{p_n}{p_{n-1}}),n>1.\end{array}$$

(6)

Suppose $\gamma$ is a change-point in a message and $x_n$ is the n-th letter in the message. We assume that the post-change distribution of $x_n$ is $f_{{\varrho }_1}(X_n)$, while the pre-change distribution of $x_n$ is $f_{{\varrho }_0}(X_n)$, then prefix conditional probability of $x_n$, i.e., $p_n$, would be much less than $p_{n-1}$, which results in a high and positive value of $C_n$. When $n<\gamma$, if $x_n$ and $x_{n-1}$ locate in the same field, that is they obey the same distribution, so that $|1-p_n/p_{n-1}|\le ϵ$, where $ϵ$ is a small and positive value, given as a threshold. if $x_n$ and $x_{n-1}$ locate in different fields, that means $n-1$ is also a change-point which should be detected before $\gamma$. On the other hand, the value of $C_n$ is likely to be bigger than the given threshold $ϵ$ when $n>\gamma$. As a result, a detection indicator metric which could be regarded recursively for multi-change-point detection should be defined as:

$$\begin{array}{c}\hfill V_n=max\{0,V_{n-1}+C_n\},n\ge 1,V_0=0.\end{array}$$

(7)

The stopping condition can be set as

$$\begin{array}{c}\hfill {\tau }^{\ast }=min\{n\ge 1:V_n\ge \upsilon \},\end{array}$$

(8)

where $\upsilon$ is a threshold of detection indicator.

3.2. Multi-Change-Point Detection

Since the problem of message field identification in this paper is actually a multi-change-point detection problem, the detection procedure has to be extended to a multi-round procedure presented in Section 3.1 and called MultiCUSUM.

A variable ${\chi }_n$ indicating the underlying state of $x_n$ is defined as

$$\begin{array}{c}\hfill {\chi }_n=\left\{\begin{array}{c}1,n\in \{n^{\prime }\ge 1:V_{n^{\prime }}\ge \upsilon \}\hfill \\ 0,\mathtt{otherwise}.\hfill \end{array}\right.\end{array}$$

(9)

Accordingly, the detection statistic is

$$\begin{array}{c}\hfill V_n=\left\{\begin{array}{cc}max\{0,u_0+C_n\},\hfill & {\chi }_{n-1}=1,\hfill \\ max\{0,V_{n-1}+C_n\},\hfill & {\chi }_{n-1}=0,\hfill \end{array}\right.\end{array}$$

(10)

where $u_0$ is the initial condition in a new round of detection procedure started once the previous change-point has been found.

The stopping time in the k-th iteration, denoted as ${\displaystyle {\tau }_k^{\ast }}$, is defined as

$$\begin{array}{c}\hfill {\displaystyle {\tau }_k^{\ast }}=min\{n>{\displaystyle {\tau }_{k-1}^{\ast }}:V_n\ge {\upsilon }_k\},k\ge 1,{\displaystyle {\tau }_0^{\ast }}=0,\end{array}$$

(11)

where

$$\begin{array}{c}\hfill {\upsilon }_k=\left\{\begin{array}{cc}1-\zeta ,\hfill & {\displaystyle {\tau }_{k-1}^{\ast }}<n<{\displaystyle {\tau }_{k-1}^{\ast }}+3\hfill \\ min\{\rho \ast {\mu }_k,1-\zeta \},\hfill & n\ge {\displaystyle {\tau }_{k-1}^{\ast }}+3\hfill \end{array}\right.\end{array}$$

(12)

with ${\mu }_k$ as the mean of $\{C_{{\displaystyle {\tau }_{k-1}^{\ast }}+1},\dots ,C_{n-1}\}$ and $\rho$ as the coefficient of ${\mu }_k$.

3.3. Message Segmenting Algorithm

A message segmenting algorithm, as shown in Algorithm 1, is proposed to segment protocol message m into a set of message fields. In Algorithm 1, the message m consists of a set of All messages associated with a specific protocol in $\mathcal{D}$ are concatenated one by one to form a new message m according to their appearance time. Then, a Q-depth suffix trie $\mathbf{T}$ is built to store sub-strings of m with max length of $Q+1$ (line 1). The prefix conditional probability $p_n$ is calculated according to Equation (3) (line 2) to enable the multi-change-point detection procedures (MultiCUSUM()). The identified change-points are put into ${\mathcal{P}}_1$ (line 3).

Algorithm 1 Message Segmenting Algorithm

Input: Message $m=\overline{x_1\dots x_N}$ Output: Segment set $\mathsf{\Omega }$ 1: $\mathbf{T}\leftarrow$ QSufTrie(m); # Creating Q-depth suffix trie 2: $\mathbf{P}\leftarrow$ condProb(m,$\mathbf{T}$); # Compute the conditional probabilities: $\mathbf{P}=\{p_n:n=1,\dots ,N\}$ 3: ${\mathcal{P}}_1\leftarrow$ MultiCUSUM(m,$\mathbf{P}$); # Change-point detection, ${\mathcal{P}}_1$ is the change-point set 4: $m^R\leftarrow$$\overline{x_N{x}_{N-1}...x_1}$; # Reverse the message stream 5: ${\mathbf{T}}^R\leftarrow$ QSufTrie($m^R$); 6: ${\mathbf{P}}^R\leftarrow$ condProb($m^R$,${\mathbf{T}}^R$); 7: ${\mathcal{P}}_2\leftarrow$ MultiCUSUM($m^R$,${\mathbf{P}}^R$); 8: $\mathcal{P}\leftarrow {\mathcal{P}}_1\cup {\mathcal{P}}_2$ 9: $\mathsf{\Omega }\leftarrow$ MsgSeg(m,$\mathcal{P}$);

Actually, not all change-points are not so sensitive to the prefix conditional probability of $P(x_n|\overline{x_1\dots x_{n-1}})$ to be detected by the aforementioned procedure, instead they are more sensitive to the postfix conditional probability of $P(x_n|\overline{x_{n+1}\dots x_N})$ which is essentially the prefix conditional probability $x-n$ in a special string that is the reverse-order of original message. Therefore, we reverse the letter order of m (i.e., $m^R=x_N{x}_{N-1}\dots x_1$) and perform the same detection procedure again on $m^R$ to search for such type of change-points (line 4∼7) and put the results in ${\mathcal{P}}_2$.

Finally, the two sets of change-points are merged by $\mathcal{P}={\mathcal{P}}_1\cup {\mathcal{P}}_2$ and the message m is segmented into segments based on the change-points in $\mathcal{P}$(line 8∼9).

4. Inferring Message Fields

4.1. Occurrence Probability Analysis

To relief the burden of position-based statistic test analysis, a pre-processing called occurrence probability analysis is applied to filter out the obvious part of data fields whose occurrence probability is very low. Given a dataset $\mathcal{D}$ and its size of M, and the occurrence probability of a string $\omega \in \mathsf{\Omega }$ in $\mathcal{D}$, denoted as $p_D(\omega )$, is defined as the ratio between the amount of messages containing $\omega$, denoted as ${\nu }_m(\omega )$, and the size of dataset.

The data field is variational and their occurrence probabilities of each value in a data field are always very small, which nearly approaches zero. Therefore, the data field can be found by searching for those string segments whose occurrence probabilities are statistically zero. In this paper, the occurrence probabilities of message segments is assumed to obey binomial distribution and the binomial test in the statistics field is considered to test whether the occurrence probability of each message segment is zero.

Let the hypothesis be

$$\begin{array}{cc}\hfill H_0:& p_D(\omega )\to 0,\omega \in \mathsf{\Omega },\hfill \end{array}$$

(13)

where $\alpha$ is a significance level.

The strings in $\mathcal{F}$ could be chosen as data fields according to

$$\begin{array}{c}\hfill {\mathcal{F}}_d=\left\{\omega :\frac{{\nu }_m(\omega )}{M}\le \alpha ,\omega \in \mathsf{\Omega }\right\}.\end{array}$$

(14)

4.2. Position-Based Statistic Test Analysis

Apparently, a keyword field would frequently appear in many messages with similar function and its positions are also relatively stable. That means both frequency and position are important features for us to infer keyword fields from segment set $\mathcal{F}$. As a result, a position-based statistic test is introduced to select keyword fields from $\{\omega :\omega \in (\mathcal{F}-{\mathcal{F}}_d)\}$ by testing the position of segment is fixed or quasi-fixed in messages.

Specifically, four kinds of positions of $\omega$ are considered in our scheme. That is

• $P^{\omega ,1}$: the distance between the message head and the position of $\omega$ in the message.
• $P^{\omega ,2}$: the distance between the message tail and the position of $\omega$ in the message.
• $P^{\omega ,3}$: the distance between the head of a line which containing $\omega$ and the position of $\omega$ in that line.
• $P^{\omega ,4}$: the distance between the tail of a line which containing $\omega$ and the position of $\omega$ in that line.

Let $P^{\omega ,r}=({\displaystyle p_1^{\omega ,r}},\dots ,{\displaystyle p_n^{\omega ,r}}),r\in \{1,2,3,4\}$ and define the support rate of ${\displaystyle p_i^{\omega ,r}}$, marked as $N({\displaystyle p_i^{\omega ,r}})$, as the number of ${\displaystyle p_i^{\omega ,r}}$ in $\mathcal{D}$. Based on binomial test (see Section 4.1), the keyword fields are chosen by

$$\begin{array}{c}\hfill {\mathcal{F}}_k=\left\{\omega :\frac{\underset{i,r}{max}\left\{N({\displaystyle p_i^{\omega ,r}})\right\}}{{\sum }_{r,i}N({\displaystyle p_i^{\omega ,r}})}>1-\alpha ,\omega \in \mathcal{F}-{\mathcal{F}}_d\right\},\end{array}$$

(15)

given $\alpha$ as the significance level.

Equation (15) infers keywords whose positions are fixed by searching for segments satisfying ${max}_{i,r}\{N({\displaystyle p_i^{\omega ,r}})\}$. It has good performance on those $\omega$ which have one dominated position. For instance, “GET” in HTTP messages has one dominated position, i.e., in the head of a request message. However, some other keywords have more than one dominated position, and there are multiple peaks in $N({\displaystyle p_i^{\omega ,r}})$.

Aiming to address multi-peak issue, an algorithm (called MDL-PTA) based on the minimal description length (MDL) [46] criteria is introduced to enable the position-based statistic test analysis, as shown in Algorithm 2.

k reference positions, $B_k=\{b_1,\dots ,b_k\}$, whose support rates are the first k top values in $\{N({\displaystyle p_j^{\omega ,r}}):{\displaystyle p_j^{\omega ,r}}\in P^{\omega ,r}\}$ (line 5) are selected for each $\omega$, and $P^{\omega ,r}$ is divided into k clusters, $C_k=\{c(b_1),\dots ,c(b_k)\}$, according to the distance between ${\displaystyle p_j^{\omega ,r}}$ and reference position $b_m$, $m=1,2,\dots ,k$ (line 6).

The entropy of $C_k$ is calculated through following equation:

$$\begin{array}{c}\hfill E_k=-{\displaystyle \sum _{i=1}^k}\left(\frac{|c(b_i)|}{{\displaystyle {\sum }_{j=1}^k}|c(b_j)|}log\frac{|c(b_i)|}{{\displaystyle {\sum }_{j=1}^k}|c(b_j)|}\right).\end{array}$$

(16)

The model complexity of $C_k$ is $(logk)/2$ and the sum of description length of $C_k$ is calculated in line 7, that is

$$\begin{array}{c}\hfill L_k=E_k+\frac{logk}{2}.\end{array}$$

(17)

The k-th model in the model set $\mathsf{\Psi }$ is represented as $\{B_k,C_k,L_k\}$. The optimal model with minimal description length would be selected from $\mathsf{\Psi }$ (line 11). Apparently, the computation complexity would be very high if all models in $\mathsf{\Psi }$ are considered. Meanwhile, a keyword should not have lots of reference positions. As a result, only the top K models in $\mathsf{\Psi }$ are considered in Algorithm 2 (line 4∼10).

Algorithm 2 MDL-PTA Algorithm

Input:K, $\mathcal{D}$ and $\omega \in (\mathcal{F}-{\mathcal{F}}_d)$ Output: true if $\omega$ is a keyword field, or false otherwise. 1: $\mathsf{\Psi }\leftarrow \{\}$ 2: $P^{\omega ,r}\leftarrow$ GetPos($\mathcal{D}$,$\omega$) # Get positions of $\omega$ 3: $\mathcal{N}\leftarrow \{N({\displaystyle p_j^{\omega ,r}}):{\displaystyle p_j^{\omega ,r}}\in P^{\omega ,r}\}$ 4: for i = 1 K do 5:  $B_i\leftarrow$ TopK($\mathcal{N}$,i) # Get reference positions 6:  $C_i\leftarrow$ Cluster($P^{\omega ,r}$,$B_i$) # Cluster $P^{\omega ,r}$ 7:  $L_i\leftarrow$ CalDLen($C_i$) # Compute the description length of $C_i$ 8:  ${\psi }_i\leftarrow \{B_i,C_i,L_i\}$ 9:  $\mathsf{\Psi }\leftarrow \mathsf{\Psi }\cup \{{\psi }_i\}$ 10: end for 11: ${\psi }^{\ast }\leftarrow$ minDLen($\mathsf{\Psi }$) # Get the model with minimal description length 12: for all$b_i\in B^{\ast }$do 13:  res←TestPos($b_i$,$c^{\ast }(b_i)$) # Check whether $b_i$ satisfies Eq. (19) 14:  if res==true then 15:   return  true 16:  end if 17: end for 18: return  false

The optimal model chosen by the Algorithm 2 is ${\psi }^{\ast }=\{C^{\ast },B^{\ast },L^{\ast }\}$. For each reference position $b_i\in B^{\ast }$ in $C^{\ast }=\{c^{\ast }(b_1),\dots ,c^{\ast }(b_{k^{\ast }})\}$, the following hypothesis is tested via binomial test:

$$\begin{array}{cc}\hfill H_0:& P(x=b_i:x\in c^{\ast }(b_i))\to 1,\hfill \end{array}$$

(18)

where $b_i\in B^{\ast }$ and $i=1,\dots ,k^{\ast }$.

The accept condition is

$$\begin{array}{c}\hfill \left\{b\in B^{\ast }:\frac{N(b)}{{\sum }_{x\in c^{\ast }(b)}N(x)}>1-\alpha \right\},\end{array}$$

(19)

The segment set passing hypothesis test is regarded as keyword set ${\mathcal{F}}_k$, and the rest ones are uncertain fields.

The inferred message fields would be further refined and some semantic information of message fields would be determined. Specifically, continuous segments of data (or uncertain) fields would be merged into a single segment which is data (or uncertain) field. Regular expressions representing some specific semantic information, such as IP address, File names, URLs, Timestamp and so on, are applied to match the message fields so that some semantic of message fields would be inferred.

5. Evaluation

In this section, experiments are performed to evaluate the effectiveness of the proposed method. The experiments comprise of two parts: message segmentation evaluation and fuzzing test. The proposed message segmentation approach is implemented on a system called QCD-PInfer whose system architecture is shown in Figure 2.

There are totally six typical protocols (HTTP, FTP, SMTP, POP, DNS and QQ) which are widely used in the application-layer are selected to test the effectiveness and efficiency of message segmentation. The recall and precision of keyword inference are shown in

Table 1. The recall of keyword inference.

System	HTTP	FTP	SMTP	POP	SSDP	BitTorrent
QCD-PInfer	87.0	92.9	85.7	84.0	74.1	100
Discoverer	78.3	60.7	64.3	40.0	33.3	100
PI	4.4	3.6	7.1	4.0	18.5	50.0

and

Table 2. The precision of keyword inference.

System	HTTP	FTP	SMTP	POP	SSDP	BitTorrent
QCD-PInfer	66.7	97.0	35.0	95.8	66.0	66.7
Discoverer	7.2	23.3	19.2	22.8	33.9	5.3
PI	100	100	20.0	16.7	35.6	33.3

. Please note that, the ground truth of keywords are those keywords which are occurred in the test set. Both DNS and QQ are not taken into account for evaluating the quality of keyword set, since the two are binary protocols and there is no concept of keyword defined in binary protocol.

By comparison, QCD-PInfer has a higher recall rate than Discoverer and PI. In particular, PI’s recall rate is much low: the recall rates for HTTP, FTP, SMTP and POP are less than 10%.

Discoverer is prone to infer too many segments as keywords, so that its precision is much lower than that of the proposed system. Although PI’s recall rate is very low, its precision for HTTP and FTP is extremely high. However, PI’s precision for other protocols are still very low. It is worth mentioning that PI infers too few keywords, always less than 5 for all protocols being considered.

The F-scores of the experiment results are shown in Figure 3. The proposed system has the highest F-score for all the six protocols, which means our method performs well in keyword inference.

In fuzzing test, QCD-PInfer is extended with fuzzing function to implement an automatic fuzzing tool (APREFuzz). APREFuzz can identify vulnerability in a system being tested which is designed to introduce information-centric network into IoT devices to enable their caching capability. The protocol used by target system under testing comprises of 5 type of messages responsible for sending interesting, distributing data, pushing data, responding with target data and responding with no answer, respectively.

Firstly, message fields are identified using QCD-PInfer system, and message format are reconstructed.

Secondly, test files are generated by inserting fault data into one field according to the message format. Please note that, for a real fuzz test, fault data may inserted into more than one field. However, as a proof-of-concept system, APREFuzz considers the scenarios with only one field being fault-injected currently. Actually, it is not difficult to extend the system to consider fault-injected in multiple fields. When inserting the fault data, keyword fields are only replacing with inferred keywords according to message formats, data fields would be replaced by random data, while uncertain fields would be replaced with either inferred keywords or random data. In our experiments, the uncertain fields are treated the same as data fields.

Finally, the target system are treated as a black box and supposed to be unknown to us. APREFuzz sends test files to target system file-by-file and monitors the reactive of target system via analyzing the response.

In our experiments, APREFuzz extracted 7 keyword fields and infers 7 data fields in the sample message. One data field is found that it contains only figures. The amount of inferred keywords is 12. We take 11 abnormal strings into account for inserting fault data into the data fields except the one containing only figures. For the special field that containing only figures, 21 boundary figures are used to be injected. As a result, the amount of fault-injected files generated by APREFuzz is 248 (=$(12+11)\times 7+21\times 1+11\times 6$). On the other hand, FileFuzz generates $393,216$ (=$1.5\times 1024\times 2^8$) fault-injected files by replacing each byte with values from $0\mathtt{x}00$ to $0\mathtt{x}FF$. When test files are sent to target system, APREFuzz monitored one exception that the system fails to respond, while FileFuzz monitor none. The exceptions maybe indicates a vulnerability which would be leveraged to launch a DoS attack, or some attacks that would ruin the system’s availability. Actually, other tools are needed to analyze the exception deeply and figure out its type and impact. However, that work has surpassed the discussion scope concerned in this paper so that it will not be presented here.

6. Conclusions

The proposed method applies protocol reverse engineering approach to improve IoT protocol fuzzing performance by creating valid and effective test files based on protocol message format and reducing greatly the size of test files. It considers the statistical attributes of message fields to locate their boundaries by searching for change-points in the messages and reconstruct the message format. A CUSUM-LIKE algorithm is presented to address the problem of multi-change-point detection. Additional procedures including occurrence probability test and position test are further employed to classify the message segments into keyword fields, data fields and uncertain fields. The results show that the extracted message formats are useful for generating test files for network protocol fuzzing.

In the future, the proposed APREFuzz with enough improvement based on current version would be a practical and powerful tool to generate test files automatically for fuzzing test carried on IoT protocols or devices to reveal their hidden vulnerabilities. It also would contribute to strengthening the IoT security in effective and efficient way, and even to be a security tool for improving protocol fuzzing in many other types of network.