Multiset Lempel–Ziv Jaccard Distance

The performance of pattern classification is affected significantly by feature selection. However, for security applications, selecting proper features is difficult, as malicious software continuously changes its characteristics. Thus, compression-based pattern recognition has attracted much attention because it does not require explicit feature selection to design proper distance measures. LZJD (Lempel–Ziv Jaccard Distance), in particular, has been useful for malware classification, as it computes compression distances without actually compressing objects and is suitable for handling large files like malware. LZJD extracts a compression dictionary for every object in advance and estimates a similarity between two objects by comparing their compression dictionaries. However, LZJD ignores the similarity between words in a compression dictionary. As a result, even if the dictionary has many similar words, they are simply processed as different words. To exploit the similarity between words, we propose to remove the last characters of words in the dictionary and to unify similar words that share the same prefix. This unification of words turns the compression dictionary into a multiset of words. Hence, our compression distance is named MLZJD (Multiset LZJD). In addition, the unification of words in MLZJD decreases the number of word kinds in compression dictionaries and contributes to speeding up the distance computation. We experimentally show that MLZJD halves the execution time as compared with LZJD, while hardly damaging the classification accuracy. Even on condition that the compression distances are approximated with Min-Hash, MLZJD achieves a much shorter running time than LZJD, while retaining almost the same classification accuracy as LZJD.