# Multi-PQTable for Approximate Nearest-Neighbor Search

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

- We propose a product quantization table (PQTable) algorithm on the basis of the PQ algorithm, according to the ability of the Hash Table to quickly find the required content. This algorithm can implement a non-exhaustive approximate nearest-neighbor search algorithm, aiming at quickly and accurately retrieving the vector candidate sets in a large-scale dataset.
- We also propose a multi-PQTable query strategy for ANN search. Besides, we generate several nearest-neighbor vectors for each sub-compressed vector of the query vector to reduce the failure rate and improve the recall in image retrieval.

## 2. Product Quantization

_{i}is a codeword or centroid.

- (1)
- Uniformly split the vector X into M distinct sub-vectors u
_{j}(x), 1 ≤ j ≤ M. The dimension of the sub-vector is D* and D* = D/M, where D is a multiple of M. Therefore, the vector X can be seen as a series of sub-vectors, and $X=\left[{u}_{1}\left(x\right),{u}_{2}\left(x\right),\dots {u}_{M}\left(x\right)\right]$. - (2)
- Each sub-vector is quantized and compressed by the K-means algorithm, and the corresponding codebook set C
_{j}is obtained. - (3)
- The Codebook C of the vector X is the Cartesian product generated from all the set C
_{j}, and C = C_{1}× C_{2}× ⋯ × C_{M}.

**SDC**: the vector x and the vector y can be represented by their corresponding centroids q(x) and q(y). The distance D(x, y) between vector x and vector y can be approximated to the distance D(q(x), q(y)) between their centroids q(x) and q(y). That is to say, $D\left(x,y\right)\approx D\left(q\left(x\right),q\left(y\right)\right)$, as shown in Equation (5).

_{j}(x), q

_{j}(y))

^{2}can be quickly obtained from the lookup table according to the index value of the j-th sub-quantizer. The lookup table contains the square of the distance between all the sub-quantizer centroids.

**ADC**: this method only needs to represent the vector y with its centroid q(y). The distance D(x, y) between the vector x and the vector y can be approximated to the distance D(x, q(y)). That is, $D\left(x,y\right)\approx D\left(x,q\left(y\right)\right)$, as shown in Equation (6).

## 3. Multi-PQTable for ANN Search

#### 3.1. Problem Description

_{q}and the image dataset I = {I

_{1}, I

_{2}, …, I

_{N}}, N represents the size of the dataset.

_{1}, Y

_{2}, …, Y

_{k}}, which are closest to the query image I

_{q}from the dataset I. The image retrieval task can be transformed into vector retrieval, as shown in Figure 2. The specific steps of image retrieval are as follows:

- (1)
- Extract the features for the query image I
_{q}and for the image dataset I by feature extraction tools such as SIFT, GIST, CNN, and so on. Correspondingly obtain the image feature vector Q = [q_{1}, q_{2}, …, q_{n}] and the image feature dataset X = {X_{1}, X_{2}, …, X_{N}}, where Q is the feature vector of the query image I_{q}, X_{i}= [x_{i}_{1}, x_{i}_{2}, …, x_{iD}], and D is the dimension of the feature vector. - (2)
- Obtain the top-k vector candidate subsets S
_{c}= {S_{1}, S_{2}, …, S_{k}} through calculating and sorting according to the query vector Q. - (3)
- Correspondingly obtain the top-k image candidate sub-dataset Y = {Y
_{1}, Y_{2}, …, Y_{k}} via the linking relationship between the vectors and the images.

#### 3.2. PQTable Algorithm

_{1}, X

_{2}, …, X

_{N}}, X

_{i}∈ R

^{D}, X

_{i}$=\left[{X}_{i}^{1},{X}_{i}^{2},\dots ,{X}_{i}^{M}\right]$. PQ(X

_{i}) represents the product quantization process of vector X

_{i}, as shown in Equation (7).

_{i}∈ R

^{D}, ${X}_{i}^{j}$ ∈ R

^{D}

^{/M}, 1 ≤ i ≤ N, 1 ≤ j ≤ M.

^{M}.

_{1}, q

_{2}, …, q

_{m}}, and we can obtain a compression vector PQ(Q), where the quantizers come from the quantization process of set X. Then, we can use the compression vector PQ(Q) to search in the PQTable. Finally, we can quickly get the candidate set according to the mapping relationship between the original vector and the compressed vector.

#### 3.3. Multi-PQTable Query Strategy

^{M}. The computational cost of retrieval vectors in PQTable is O(LM), and the computational cost of the PQ algorithm is O(KD + NM) ≈ O(NM). When N = 1 × 10

^{9}, K = 256 = 2

^{8}, M = 8, L = K

^{M}= (2

^{8})

^{M}= 256

^{M}= 256

^{8}= 1.84 × 10

^{19}, there is obviously L >> N. Thus, their computing time is sorted as O(LM) >> O(NM). In other words, the PQTable increases the computational time of vector retrieval. When L >> N, the identifiers in the PQTable are mostly empty and sparse, which easily leads to retrieval failure.

^{M}

^{*}. When searching via the query vector Q, the compression vector PQ(Q) is also divided into T sub-compression vectors, namely, PQ(Q) = [subPQ(Q)

^{1}, subPQ(Q)

^{2}, …, subPQ(Q)

^{T}]. Then, we query the subPQ(Q)

^{t}in the subPQTable

^{t}and get the corresponding candidate set ${S}_{C}^{t}$, where 1 ≤ t ≤ T. Finally, we union the T candidate sets and obtain the final candidate set S

_{C}, as shown in Equation (8).

^{M*}= (2

^{8})

^{M}

^{/2}= (2

^{4})

^{M}= 16

^{M}= 16

^{8}= 4.2 × 10

^{9}. When T = 4, M* = M/T = M/4, there is L* = K

^{M*}= (2

^{8})

^{M}

^{/4}= (2

^{2})

^{M}= 4

^{M}= 4

^{8}= 65,536. Apparently, L* << L, and O(L*M) << O(LM). According to the above analysis, it is found that the multi-PQTable query strategy greatly reduces the computational time and effectively improves the speed of vector retrieval. When T = 2, the process of the PQTable query strategy is as shown in Figure 5.

^{t}of the query vector Q. The purpose is to reduce the failure rate and improve the recall rate. The specific steps for generating several neighbor vectors are as follows:

- (1)
- Creating a generator table. We define K* − 1 as the largest parameter and create a two-dimensional generator table with a size of U × V, where K is a positive integer, and K* << K, U = K*
^{M*}, V = M*, M* = M/T. We sequentially fill the table with integers from 0 to K* − 1 and get a generator table (GenTable). - (2)
- Generating several nearest-neighbor vectors. Each subPQ(Q)
^{t}is added and subtracted to every row data in the GenTable. Then, we can get the corresponding nearest-neighbor vector set (NNVS). - (3)
- Filtering the elements in nearest-neighbor vectors. We validate each vector in the NNVS and filter out the vectors whose elements are less than zero. Finally, we get the final NNVS.

^{t}= [54,122,35,63], U = 4

^{4}= 64, V = 4; the process of generating NNVS is shown in Figure 6. The generating nearest-neighbor vector set (genNNVS) algorithm is shown in Algorithm 1.

**Algorithm 1.**The input of the algorithm includes the generator table GenTable[U][V] and the sub-compression vector subPQ[V]. The output is the nearest-neighbor vector set NNVS. Lines 3–10 shows that each subPQ(Q)

^{t}is added and subtracted to every row data in the GenTable. Combining lines 7–9 with lines 12–14, we obtain that the algorithm filters out the vectors whose elements are less than zero.

Algorithm 1: Generating Nearest-Neighbor Vector Set (genNNVS). | |

Input: | |

GenTable[U][V], subPQ[V] | |

Output: | |

NNVS | |

1: | for i <= U do |

2: | Flag = true; |

3: | for j <= V do |

4: | V_{0}[j] = subPQ [j] + GenTable[i][ j]; |

5: | V_{1}[j] = subPQ [j] − GenTable[i][j]; |

6: | if V_{1}[j] < 0 then |

7: | flag = false; |

8: | end if |

9: | end for |

10: | NNVS add V_{0}; |

11: | if flag == true then |

12: | NNVS add V_{1}; |

13: | end if |

14: | end for |

_{1}, X

_{2}, …, X

_{N}}, the process of quickly obtaining the candidate set S

_{C}through the multi-PQTable is shown in Figure 7. Besides, the approximate nearest-neighbor search algorithm based on the multi-PQTable is shown in Algorithm 2.

**Algorithm 2.**The input of the algorithm includes the candidate set size k, the query vector Q, the generator table GenTable[U][V], and T subPQTables, where subPQTables = {subPQTable

^{1}, subPQTable

^{2}, …, subPQTable

^{T}}. The output is the candidate set S

_{Ck}, whose size is k. Line 1 indicates that the query vector q is quantized and compressed by the PQ algorithm. Line 2 indicates that the compressed vector PQ(Q) is split into T sub-compressed vectors, i.e., subPQs = {subPQ

^{1}[V], subPQ

^{2}[V], …, subPQ

^{T}[V]}. Line 5 represents the generating NNVS for each sub-compressed vector subPQ

^{t}[V]. Lines 6–10 describe that the algorithm searches all vector in the subPQTable

^{t}and gets the corresponding candidate set ${S}_{C}^{t}$, where 1 ≤ t ≤ T. Line 13 indicates that the algorithm obtains the final candidate set S

_{Ck}through calculating similarity and sorting for their original vectors. The algorithm can also set the similarity threshold (e.g., α = 0.5) in advance and only add the vector whose similarity is greater than the threshold to the candidate set S

_{Ck}.

Algorithm 2: The Approximate Nearest-Neighbor (ANN) Search Algorithm based on the Multi-PQTable | |

Input: | |

k, Q, GenTable[U][V], subPQTables = {subPQTable^{1}, subPQTable^{2}, …, subPQTable^{T}} | |

Output: | |

S_{Ck} | |

1: | PQ(Q); |

2: | subPQs = {subPQ^{1}[V], subPQ^{2}[V], …, subPQ^{T}[V]}; |

3: | while t <= T do |

4: | t = t + 1; |

5: | NNVS^{t} = genNNVS(GenTable[U][V], subPQ^{t}[V]); |

6: | foreach vector ∈ NNVS do |

7: | IDs←search vector in subPQTables^{t}; |

8: | sc←obtain the vector set through IDs; |

9: | S_{C}^{t} = S_{C}^{t}∪sc; |

10: | end foreach |

11: | S_{C} = S_{C}∪S_{C}^{t}; |

12: | end while |

13: | S_{Ck} = S_{C}; |

14: | size←get size of S_{C}; |

15: | if size > k then |

16: | S_{Ck}←calculate similarity and sort; |

17: | end if |

## 4. Experiments and Analysis

#### 4.1. Experimental Settings

^{4}levels. Therefore, we randomly select an image from the MNIST dataset and obtain a one-dimensional pixel matrix. We add some random factors into the one-dimensional pixel matrix and repeat them three times to synthesize a three-dimensional pixel matrix. We input the three-dimensional pixel matrix into the TSPP-Net model and get the corresponding feature vectors. After repeating the process for 1 × 10

^{9}times, we can make the scale of image feature vectors reach 100 million levels. Since the image in MNIST is quite different from the image in Caltech 101, the above operation has no effect on the result of image retrieval. At the same time, we quantize and compress vectors through PQ, OPQ, and LOPQ and then construct the corresponding multi-PQTables.

#### 4.2. Experimental Results

## 5. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Lowe, D.G. Distinctive Image Features from Scale-Invariant Keypoints. Int. J. Comput. Vis.
**2004**, 60, 91–110. [Google Scholar] [CrossRef] - Figat, J.; Kornuta, T.; Kasprzak, W. Performance Evaluation of Binary Descriptors of Local Features. In Proceedings of the International Conference on Computer Vision and Graphics, Warsaw, Poland, 15–17 September 2014; pp. 187–194. [Google Scholar]
- Bay, H.; Tuytelaars, T.; Gool, L.V. SURF: Speeded Up Robust Features. In European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 2006; pp. 404–417. [Google Scholar]
- Boulkenafet, Z.; Komulainen, J.; Hadid, A. Face Antispoofing Using Speeded-Up Robust Features and Fisher Vector Encoding. IEEE Signal Process. Lett.
**2017**, 24, 141–145. [Google Scholar] - Oliva, A.; Torralba, A. Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope. Int. J. Comput. Vis.
**2001**, 42, 145–175. [Google Scholar] [CrossRef] - Oliva, A.; Torralba, A. Building the gist of a scene: The role of global image features in recognition. Prog. Brain Res.
**2006**, 155, 23–36. [Google Scholar] [PubMed] - Sánchez, J.; Redolfi, J. Exponential family Fisher vector for image classification. Pattern Recognit. Lett.
**2015**, 59, 26–32. [Google Scholar] [CrossRef][Green Version] - Jégou, H.; Douze, M.; Schmid, C.; Pérez, P. Aggregating local descriptors into a compact image representation. In Proceedings of the IEEE Conference on Computer Vision & Pattern Recognition, San Francisco, CA, USA, 13–18 June 2010. [Google Scholar]
- Amato, G.; Bolettieri, P.; Falchi, F.; Gennaro, C. Large Scale Image Retrieval Using Vector of Locally Aggregated Descriptors. In International Conference on Similarity Search & Applications; Springer: Berlin/Heidelberg, Germany, 2013. [Google Scholar]
- Sun, P.X.; Lin, H.T.; Luo, T. Learning discriminative CNN features and similarity metrics for image retrieval. In Proceedings of the IEEE International Conference on Signal Processing, Communications and Computing, Hong Kong, China, 5–8 August 2016. [Google Scholar]
- Fu, R.; Li, B.; Gao, Y.; Wang, P. Content-based image retrieval based on CNN and SVM. In Proceedings of the IEEE International Conference on Computer and Communications, Chengdu, China, 14–17 October 2016; pp. 638–642. [Google Scholar]
- Melekhov, I.; Kannala, J.; Rahtu, E. Siamese network features for image matching. In Proceedings of the International Conference on Pattern Recognition, Cancun, Mexico, 4–8 December 2016; pp. 378–383. [Google Scholar]
- Appalaraju, S.; Chaoji, V. Image Similarity Using Deep CNN and Curriculum Learning. arXiv
**2017**, arXiv:1709.08761. [Google Scholar] - Li, Y.; Miao, Z.; Wang, J.; Zhang, Y. Deep binary constraint hashing for fast image retrieval. Electron. Lett.
**2018**, 54, 25–27. [Google Scholar] [CrossRef] - Hoffer, E.; Ailon, N. Deep Metric Learning Using Triplet Network. In Similarity-Based Pattern Recognition; Springer: Berlin/Heidelberg, Germany, 2015; pp. 84–92. [Google Scholar][Green Version]
- Schroff, F.; Kalenichenko, D.; Philbin, J. FaceNet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 815–823. [Google Scholar]
- Liu, Y.; Chao, H. Scene Classification via Triplet Networks. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.
**2018**, 11, 220–237. [Google Scholar] [CrossRef] - Kumar, Y.S.; Pavithra, N. KD-Tree approach in sketch based image retrieval. In Proceedings of the International Conference on Mining Intelligence and Knowledge Exploration, Hyderabad, India, 9–11 December 2015; pp. 247–258. [Google Scholar]
- Kao, B.; Lee, S.D.; Lee, F.K.; Cheung, D.W.; Ho, W.S. Clustering Uncertain Data Using Voronoi Diagrams and R-Tree Index. IEEE Trans. Knowl. Data Eng.
**2010**, 22, 1219–1233. [Google Scholar][Green Version] - Viet, H.H.; Anh, D.T. M-tree as an index structure for time series data. In Proceedings of the International Conference on Computing, Management and Telecommunications, Ho Chi Minh City, Vietnam, 21–24 January 2013; pp. 146–151. [Google Scholar]
- Wieschollek, P.; Wang, O.; Sorkine-Hornung, A.; Lensch, H. Efficient Large-scale Approximate Nearest Neighbor Search on the GPU. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2017; pp. 2027–2035. [Google Scholar]
- Amsaleg, L. Locality sensitive hashing: A comparison of hash function types and querying mechanisms. Pattern Recognit. Lett.
**2010**, 31, 1348–1358. [Google Scholar][Green Version] - Abdulhayoglu, M.A.; Thijs, B. Use of locality sensitive hashing (LSH) algorithm to match Web of Science and Scopus. Scientometrics
**2018**, 116, 1229–1245. [Google Scholar] [CrossRef] - Jégou, H.; Douze, M.; Schmid, C. Product Quantization for Nearest Neighbor Search. IEEE Trans. Pattern Anal. Mach. Intell.
**2011**, 33, 117–128. [Google Scholar] [CrossRef] - Ge, T.; He, K.; Ke, Q.; Sun, J. Optimized Product Quantization for Approximate Nearest Neighbor Search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 2946–2953. [Google Scholar]
- Ge, T.; He, K.; Ke, Q.; Sun, J. Optimized Product Quantization. IEEE Trans. Pattern Anal. Mach. Intell.
**2014**, 36, 744–755. [Google Scholar] [CrossRef] - Kalantidis, Y.; Avrithis, Y. Locally Optimized Product Quantization for Approximate Nearest Neighbor Search. Available online: http://openaccess.thecvf.com/content_cvpr_2014/papers/Kalantidis_Locally_Optimized_Product_2014_CVPR_paper.pdf (accessed on 15 May 2018).
- Martinez, J.; Hoos, H.H.; Little, J.J. Stacked Quantizers for Compositional Vector Compression. arXiv
**2014**, arXiv:1411.2173. [Google Scholar] - Wang, J.; Li, Z.; Du, Y.; Qu, W. Stacked Product Quantization for Nearest Neighbor Search on Large Datasets. In Proceedings of the IEEE Trustcom, Tianjin, China, 23–26 August 2016; pp. 787–795. [Google Scholar]
- Babenko, A.; Lempitsky, V. Additive Quantization for Extreme Vector Compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 931–938. [Google Scholar]
- Yuan, X.; Liu, Q.; Long, J.; Hu, L.; Wang, Y. Deep Image Similarity Measurement based on the Improved Triplet Network with Spatial Pyramid Pooling. Information
**2019**, 10, 129. [Google Scholar] [CrossRef] - Hu, F.; Zhu, Z.; Mejia, J.; Tang, H.; Zhang, J. Real-time indoor assistive localization with mobile omnidirectional vision and cloud GPU acceleration. AIMS Electron. Electr. Eng.
**2017**, 1, 74–99. [Google Scholar] [CrossRef] - Bing, Z.; Xin-xin, Y.A. A content-based parallel image retrieval system. In Proceedings of the International Conference on Computer Design and Applications (ICCDA), Qinhuangdao, China, 25–27 June 2010; pp. 332–336. [Google Scholar]

**Figure 10.**The mean average precision (mAP) of the multi-PQTable query strategy varies with the variable K* of the generator table.

No. | Parameter Settings | Average Retrieval Time (×10^{3}/ms) | ||
---|---|---|---|---|

PQ | OPQ | LOPQ | ||

1 | Original Algorithm | 14.29 | 14.23 | 14.16 |

2 | T = 2, K* = 0 | 45.93 | 45.86 | 45.77 |

3 | T = 2, K* = 2 | 63.56 | 62.95 | 62.84 |

4 | T = 2, K* = 4 | 217.69 | 198.31 | 197.92 |

5 | T = 4, K* = 0 | 0.004 | 0.002 | 0.003 |

6 | T = 4, K* = 2 | 0.012 | 0.008 | 0.009 |

7 | T = 4, K* = 4 | 0.21 | 0.16 | 0.17 |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Yuan, X.; Liu, Q.; Long, J.; Hu, L.; Wang, S. Multi-PQTable for Approximate Nearest-Neighbor Search. *Information* **2019**, *10*, 190.
https://doi.org/10.3390/info10060190

**AMA Style**

Yuan X, Liu Q, Long J, Hu L, Wang S. Multi-PQTable for Approximate Nearest-Neighbor Search. *Information*. 2019; 10(6):190.
https://doi.org/10.3390/info10060190

**Chicago/Turabian Style**

Yuan, Xinpan, Qunfeng Liu, Jun Long, Lei Hu, and Songlin Wang. 2019. "Multi-PQTable for Approximate Nearest-Neighbor Search" *Information* 10, no. 6: 190.
https://doi.org/10.3390/info10060190