Deep Learning Triplet Ordinal Relation Preserving Binary Code for Remote Sensing Image Retrieval Task

Abstract

As satellite observation technology rapidly develops, the number of remote sensing (RS) images dramatically increases, and this leads RS image retrieval tasks to be more challenging in terms of speed and accuracy. Recently, an increasing number of researchers have turned their attention to this issue, as well as hashing algorithms, which map real-valued data onto a low-dimensional Hamming space and have been widely utilized to respond quickly to large-scale RS image search tasks. However, most existing hashing algorithms only emphasize preserving point-wise or pair-wise similarity, which may lead to an inferior approximate nearest neighbor (ANN) search result. To fix this problem, we propose a novel triplet ordinal cross entropy hashing (TOCEH). In TOCEH, to enhance the ability of preserving the ranking orders in different spaces, we establish a tensor graph representing the Euclidean triplet ordinal relationship among RS images and minimize the cross entropy between the probability distribution of the established Euclidean similarity graph and that of the Hamming triplet ordinal relation with the given binary code. During the training process, to avoid the non-deterministic polynomial (NP) hard problem, we utilize a continuous function instead of the discrete encoding process. Furthermore, we design a quantization objective function based on the principle of preserving triplet ordinal relation to minimize the loss caused by the continuous relaxation procedure. The comparative RS image retrieval experiments are conducted on three publicly available datasets, including UC Merced Land Use Dataset (UCMD), SAT-4 and SAT-6. The experimental results show that the proposed TOCEH algorithm outperforms many existing hashing algorithms in RS image retrieval tasks.

1. Introduction

With the rapid development of satellite observation technology, both the amount and the quality of remote sensing (RS) images have improved dramatically. An era of remote sensing image big data has arrived. An increasing number of researchers are focusing on the task of large-scale RS image retrieval, due to its broad applications, such as disaster prevention, soil erosion monitoring, disaster rescue scenario and short-term weather forecasting [1,2,3,4,5]. The content-based image retrieval (CBIR) [6,7] method extracts feature information representing RS image content and finds similar RS images by comparing the distance values among their feature information. However, the feature information in CBIR is always represented as high dimensional float point data and it is difficult to directly compute the similarity relationship based on the original high dimensional feature information. Fortunately, hashing methods [1,2,3,4,5,8,9] can map high dimensional float point data into compact binary codes and return the approximate nearest neighbors according to Hamming distance; this measure effectively improves the retrieval speed. In summary, the content-based image retrieval method assisted by hashing algorithms enables the efficient and effective retrieval of target remote sensing images from a large-scale dataset.

In recent years, many hashing algorithms [10,11,12,13,14] have been proposed to achieve the approximate nearest neighbor (ANN) search task, due to its advantage of computation and storage. According to the learning framework, the existing hashing algorithms can be roughly divided into two types: the shallow model [12,13,14] and the deep model [10,11,15,16]. Conventional shallow hashing algorithms, such as locality sensitive hashing (LSH) [14], spectral hashing (SH) [17], iterative quantization hashing (ITQ) [13] and k-means hashing (KMH) [12], have been applied to various approximate nearest neighbor search tasks, including image retrieval. Locality sensitive hashing [14] is a kind of data-independent method, which learns hashing functions without a training process. LSH [14] randomly generates linear hashing functions and encodes data into binary codes according to their projection signs. Spectral hashing (SH) [17] utilizes a spectral graph to represent the similarity relationship among data points. The binary codes in SH are generated by partitioning a spectral graph. Iterative quantization hashing [13] considers the vertexes of a hyper cubic as encoding centers. ITQ [13] rotates the principal component analysis (PCA) projected data and maps the rotated data to the nearest encoding center. The encoding centers in ITQ are fixed and they are not adaptive to the data distribution [12]. To fix this problem, k-means hashing [12] learns the encoding centers by simultaneously minimizing the quantization error and the similarity loss. KMH [12] encodes the data as the same binary code as the nearest center. For the image search task, the shallow model first learns the high dimensional features, such as scale-invariant feature transform (SIFT) [18] or a holistic representation of the spatial envelope (GIST) [19], then retrieves similar images by mapping these features into the compact Hamming space. In contrast, the deep learning model enables end-to-end representation learning and hash coding [10,11,20,21,22]. In particular, the deep learning to hash, such as deep Cauchy hashing (DCH) [11] and twin-bottleneck hashing (TBH) [10], proves crucial to jointly learn, thereby similarly preserving the representations and control quantization error of converting continuous representations to binary codes. Deep Cauchy hashing [11] defines a pair-wise similarity preserving restriction based on Cauchy distribution and it heavily penalizes the similar image pairs with large Hamming distance. Twin-bottleneck hashing [10] proposes a code-driven graph to represent the similarity relationship among data points and aims to minimize the loss between the original data and decoded data. These deep learning to hash methods have shown state-of-the-art results for many datasets.

Recently, many hashing algorithms have been applied to the large-scale RS image search task [1,2,3,4,5]. Partial randomness hashing [23] maps RS images into a low dimensional Hamming space by both the random and well-trained projection functions. Demir et al. [24] proposed two kernel-based methods to learn hashing functions in the kernel space. Liu et al. [25] fully utilized the supervised deep learning framework and hashing learning to generate the binary codes of RS images. Li et al. [25] carried out a comprehensive study of DHNN systems and aimed to introduce the deep neural network into the large-scale RS image search task. Fan et al. [26] proposed a distribution consistency loss (DCL) to capture the intra-class distribution and inter-class ranking. Both deep Cauchy hashing [11] and the distribution consistency loss functions [26] employ pairwise similarity [15] to describe the relationship among data. However, the similarity relationship among RS images is more complex. In this paper, we propose the triplet ordinal cross entropy hashing (TOCEH) to deal with the large-scale RS image search task. The flowchart of the proposed TOCEH is shown in Figure 1.

As shown in Figure 1, the TOCEH algorithm consists of two parts: the triplet ordinal tensor graph generation part and the hash code learning part. In part 1, we first utilize the AlexNet [27] pre-trained on the ImageNet dataset [28] to extract the 4096-dimension image feature information of the target domain RS images. Then, we separately compute the similarity and dissimilarity graph among the high dimensional features. Finally, we establish the triplet ordinal tensor graph representing the ordinal relation among any triplet RS images. Part 2 utilizes two fully connected layers to generate binary codes. During the training process, we define two excellent objection functions, including the triplet ordinal cross entropy loss and the triplet ordinal quantization loss to guarantee the performance of the obtained binary codes and utilize the back-propagation mechanism to optimize the variables of the deep neural network. The main contributions of the proposed TOCEH are summarized as follows:

1.

The learning procedure of TOCEH takes into account the triplet ordinal relations, rather than the pairwise or point-wise similarity relations, which can enhance the performance of preserving the ranking orders of approximate nearest neighbor retrieval results from the high dimensional feature space to the Hamming space.

2.

TOCEH establishes a triplet ordinal graph to explicitly indicate the ordinal relationship among any triplet RS images and preserves the ranking orders by minimizing the inconsistency between the probability distribution of the given triplet ordinal relation and that of the ones derived from binary codes.

3.

We conduct comparative experiments on three RS image datasets: UCMD, SAT-4 and SAT-6. Extensive experimental results demonstrate that TOCEH generates highly concentrated and compact hash codes, and it outperforms some existing state-of-the-art hashing methods in large-scale RS image retrieval tasks.

The rest of this paper is organized as follows. Section 2 introduces the proposed TOCEH algorithm. Section 2.1 shows the important notation. The hash learning problem is stated in Section 2.2. The tensor graph representing the triplet ordinal relation among RS images is introduced in Section 2.3. We provide the formulation of triplet ordinal cross entropy loss and triplet ordinal quantization loss in Section 2.4 and Section 2.5, respectively. The extensive experimental evaluations are presented in Section 3. Finally, we set out a conclusion in Section 4.

2. Triplet Ordinal Cross Entropy Hashing

2.1. Notation

In this paper, we use the letters B and X to separately represent the data matrix in the Hamming and Euclidean spaces. The columns in the data matrix are denoted as the letters with subscript. The important notations are summarized in

Table 1. The important notations used in this paper.

Notation	Description
B	Compact binary code matrix
Bi, Bj, Bk	The i-th, j-th, k-th column in B
H(∙)	Hashing function
X	Data matrix in the Euclidean space
xi, xj, xk	The i-th, j-th, k-th column in X
G	Triplet ordinal graph in the Euclidean space
Ĝ	Triplet ordinal relation in the Hamming space
gijk	The entry (i, j, k) in G
S	Similarity graph
DS	Dissimilarity graph
N	The number of training samples
L	The number of k-means centers
P(∙)	Probability distribution function
dh(∙,∙)	Hamming distance function
M	Binary code length
1	The binary matrix with all values of 1

.

2.2. Hashing Learning Problem

The purpose of the hashing algorithm [3,10,11] is to learn the hashing function H(∙), mapping the high dimensional float point data x into the compact Hamming space as defined in Equation (1). B(x) represents the compact binary code of x.

$$B(x)=(sign(H(x)-0.5)+1)/2$$

(1)

With the assistance of the obtained hashing function H(∙), we can encode RS image content as compact binary code and efficiently achieve RS image search task according to their Hamming distances [1,2,3,4,5,23,24,25]. Furthermore, to guarantee the quality of the RS image search result, we expect the triplet ordinal relation among RS images in the Hamming space to be consistent with that in the original space [29,30]. To illustrate this requirement, a simple example is provided below. Here, x_i, x_j and x_k separately represent RS image content information. In the original space, the image pair (x_i, x_j) is more similar than the image pair (x_j, x_k). After mapping them into the Hamming space, the Hamming distance of the data pair (x_i, x_j) should be smaller than that of the data pair (x_j, x_k). This constraint is defined as in Equation (2).

$$\begin{array}{ll}& \left|\right|H(x_i)-H(x_j)|{|}_1\le ||H(x_k)-H(x_j)|{|}_1\\ s.t.\hspace{1em}& \left|\right|x_i-x_j|{|}_2^2\le ||x_k-x_j|{|}_2^2\end{array}$$

(2)

The constraint in Equation (2) guarantees that the ranking order of the retrieval result in the Hamming space is consistent with that in the Euclidean space. Thus, the hashing algorithm, satisfying the triplet ordinal relation preserving constraint, can achieve RS image ANN search tasks [31,32,33,34,35].

2.3. Triplet Ordinal Tensor Graph

To learn the triplet ordinal relation preserving hashing functions, the first problem is how to efficiently compute the probability distribution of the triplet ordinal relation among the training set in the original space.

Generally, we select the triplet data (x_i, x_j, x_k) from the training set to compute their ordinal relation, where the data pair (x_i, x_j) has a small Euclidean distance value and (x_j, x_k) is considered as the dissimilar data pair. However, this mechanism needs to randomly select triplet samples and compare the distance values among all data points. It has a high time complexity and costly memory. Furthermore, it is difficult to define the similar and dissimilar data pairs for the problem without supervised information.

In this paper, to solve the above problem, we employ a tensor ordinal graph G to represent the ordinal relation among the triplet images (x_i, x_j, x_k). We establish the tensor ordinal graph G by tensor production and each entry in G is calculated as G(ij, jk) = S(i, j)∙DS(j, k). S(i, j) is the similarity graph as defined in Equation (3). A larger value of S(i, j) means the data pair (x_i, x_j) is more similar. DS(i, j) is the dissimilarity graph and its value is calculated as DS(i, j) = 1/S(i, j).

$$S(i,j)=\{\begin{array}{cc}0,& i=j\\ e^{-\left|\right|x_i-x_j|{|}_2^2/2{\sigma }^2},& otherwise\end{array}$$

(3)

We further process G to obey the binary distribution as in Equation (4). g_ijk is the entry of G(i, j, k).

$$\{\begin{array}{cc}{g}_{ijk}=1,& G(i,j,k)>1\\ g_{ijk}=0,& G(i,j,k)\le 1\end{array}$$

(4)

Given N training samples, the size of the similarity graph and dissimilarity graph is N × N. The tensor product of the two graphs is shown in Figure 2, and its size is N ² × N ². However, the proposed TOCEH only concerns the relative similarity relationship among the data pairs (x_i, x_j) and (x_j, x_k). The corresponding elements are marked blue. There are N rectangles and each rectangle contains N × N elements. We pick up these elements and restore them into a matrix with the size of N × N × N.

Finally, the ordinal relation among any triplet items can be represented by the triplet ordinal graph G, as defined in Equation (5).

$$\{\begin{array}{cc}S(i,j)>S(k,j),& g_{ijk}=1\\ S(i,j)\le S(k,j),& g_{ijk}=0\end{array}$$

(5)

To illustrate the cases defined in Equation (5), a simple explanation is provided below. For the triplet item (x_i, x_j, x_k), the value of the (ij, kj)-th entry is G(ij, kj) = S(i, j)∙DS(k, j) = S(i, j)/S(k, j). If the triplet ordinal relation is S(i, j) > S(k, j), we have G(ij, kj) > 1 and g_ijk = 1; otherwise, we have G(ij, kj) ≤ 1 and g_ijk = 0. Thus, the value in G can correctly indicate the true ordinal relation among any triplet items.

As described above, we can establish a tensor ordinal graph G with size N³ to represent the triplet ordinal relation among N images. In practice, during the training procedure, we use L (L ≪ N) k-means centers to establish the tensor ordinal graph, which can reduce the training time complexity.

2.4. Triplet Ordinal Cross Entropy Loss

In this section, we define Ĝ as RS images’ triplet ordinal relation in the Hamming space. As discussed in Section 2.2, an ideal hashing algorithm should minimize the inconsistency between Ĝ and G. In this paper, the above requirement is achieved by minimizing the cross entropy value, as defined in Equation (6).

$$\min CEH(G,\stackrel{⌢}{G})=\min -P(G)\log P(\stackrel{⌢}{G})$$

(6)

P(G) defined in Equation (7) computes the probability distribution of RS images’ triplet ordinal relation in the Euclidean space.

$$\{\begin{array}{cc}{w}_{ijk}=\frac{T_1}{T}& g_{ijk}=1\\ w_{ijk}=\frac{T_0}{T}& g_{ijk}=0\end{array}$$

(7)

The definitions of T₁, T₀ and T are shown in Equation (8). T₁ is the number of samples with a value of 1 in the matrix G and T₀ is the number of samples with a value of 0 in the matrix G. T is the total number of the elements in the matrix G.

$$\begin{array}{l}{T}_1={\displaystyle \sum _{i,j,k=1}^N{g}_{i,j,k}}\\ T_0={\displaystyle \sum _{i,j,k=1}^N(1-g_{i,j,k})}\\ T={\displaystyle \sum _{i,j,k=1}^N|2\cdot g_{i,j,k}-1|}\end{array}$$

(8)

P(Ĝ) is a conditional probability of the triplet ordinal relation with given binary codes. As the samples are independent from each other, we calculate P(Ĝ) by Equation (9).

$$P(\stackrel{⌢}{G})={{\displaystyle \Pi }}_{i,j,k=1}^{N}P(g_{ijk}|B_i,B_j,B_k)$$

(9)

P(g_ijk|B_i, B_j, B_k) is the probability of the triplet images satisfying the ordinal relation g_ijk, and the samples’ are assigned the binary codes (B_i, B_j, B_k). The definition is shown in Equation (10).

$$P(g_{ijk}|B_i,B_j,B_k)=\{\begin{array}{cc}\varphi (d_h(B_k,B_j)-d_h(B_i,B_j)),& g_{ijk}=1\\ 1-\varphi (d_h(B_k,B_j)-d_h(B_i,B_j)),& g_{ijk}=0\end{array}$$

(10)

We further rewrite the definition of P(g_ijk|B_i, B_j, B_k) as in Equation (11).

$$P(g_{ijk}|B_i,B_j,B_k)=\varphi {(d_h(B_k,B_j)-d_h(B_i,B_j))}^{g_{ijk}}{(1-\varphi (d_h(B_k,B_j)-d_h(B_i,B_j)))}^{1-g_{ijk}}$$

(11)

d_h(∙,∙) returns the Hamming distance and ϕ(∙) computes the probability value. If g_ijk = 1, the probability value should be close to 1 as d_h(B_k, B_j)-d_h(B_i, B_j) gets larger and the probability value should be close to 0 as d_h(B_k, B_j)-d_h(B_i, B_j) gets smaller. The characteristic of the function (∙) is shown in Figure 3.

In this paper, the sigmoid function is considered as the function (∙) as in Equation (12).

$$\varphi (d_h(B_k,B_j)-d_h(B_i,B_j))=\frac{1}{1+e^{-\alpha (d_h(B_k,B_j)-d_h(B_i,B_j))}}$$

(12)

By merging Equations (7), (9), (11) and (12) into Equation (6), we reach the final triplet ordinal relation preserving objective function, as shown in Equation (13).

$$\begin{array}{ll}L& =-w_{ijk}\log {{\displaystyle \Pi }}_{i,j,k=1}^{N}P(g_{ijk}|B_i,B_j,B_k)\\ & ={\displaystyle \sum _{i,j,k=1}^N-w_{ijk}\log P(s_{ijk}|B_i,B_j,B_k)}\\ & ={\displaystyle \sum _{i,j,k=1}^N-w_{ijk}\log {(\frac{1}{1+e^{-\alpha (d_h(B_k,B_j)-d_h(B_i,B_j))}})}^{g_{ijk}}{(1-\frac{1}{1+e^{-\alpha (d_h(B_k,B_j)-d_h(B_i,B_j))}})}^{1-g_{ijk}}})\\ & ={\displaystyle \sum _{i,j,k=1}^N{w}_{ijk}(g_{ijk}\log (1+e^{-\alpha (d_h(B_k,B_j)-d_h(B_i,B_j))})+(1-g_{ijk})\log (1+\frac{1}{e^{-\alpha (d_h(B_k,B_j)-d_h(B_i,B_j))}}))}\\ & ={\displaystyle \sum _{i,j,k=1}^N{w}_{ijk}(g_{ijk}\log (e^{-\alpha (d_h(B_k,B_j)-d_h(B_i,B_j))})+\log (1+\frac{1}{e^{-\alpha (d_h(B_k,B_j)-d_h(B_i,B_j))}}))}\end{array}$$

(13)

2.5. Triplet Ordinal Quantization Loss

Generally, the sign function is adopted to map the real-valued data output by the last layer of deep neural network into binary codes. However, it generates discrete values and makes the objective function non-deterministic polynomial (NP) hard for optimization [20,36]. To fix this problem, the continuous tanh(⋅) function is utilized instead of the sign(⋅) function in this paper. Furthermore, to minimize the quantization loss caused by the continuous relaxation procedure, we expect the output of the tanh(⋅) function to be close to ±1. Here, we utilize the triplet ordinal cross entropy to formulate the quantization loss. We define the binary code obtained by the tanh(⋅) function as Bⁱ_tah. B_ref is the reference binary code. The ideal encoding result is 1. Thus, we formulate the quantization loss Q as in Equation (14).

$$\begin{array}{c}Q={\displaystyle \sum _{i=1}^N-\log P(1|(|\left|B_{tah}^i\right||,1,|\left|B_{ref}\right||))}\\ ={\displaystyle \sum _{i=1}^N-\log \varphi (-d_h(\left|\right|B_{tah}^i\left|\right|,1)+\delta )}\\ ={\displaystyle \sum _{i=1}^N\log (1+e^{-\alpha (-d_h(\left|\right|B_{tah}^i\left|\right|,1)+\delta )})}\end{array}$$

(14)

In Equation (14), the triplet ordinal relation among (||Bⁱ_tah ||, 1 and ||B_ref||) is defined as 1 and it indicates that the data pair (||Bⁱ_tah ||, 1) is more similar than the data pair (1, ||B_ref||). Therefore, to minimize the quantization loss, the Hamming distance of the data pair (||B_tah||, 1) should be smaller than the Hamming distance δ = d_h(||B_ref||, 1). During the training procedure, we tune the value of δ to balance the optimization complexity and the approximation performance. A small δ value let the encoding results be close to the output of sign function and the training process will become hard. In contrast, a large δ value creates low optimization complexity, but it leads to poor approximation results.

After applying the continuous relaxation mechanism, we compute the Hamming distance of one data pair by Equation (15). ⨂ computes the sum of bitwise production value. f₈(⋅) represents the output of the deep neural network’s last layer.

$$d_h(B_i,B_j)=\frac{1}{2}(M-\tanh (f_8(x_i))\otimes \tanh (f_8(x_j)))$$

(15)

Finally, we utilize the back propagation mechanism to optimize the variables of the deep neural network by simultaneously minimizing the triplet ordinal relation cross entropy loss in Equation (13) and the quantization loss in Equation (14).

3. Experimental Setting and Results

In this section, we introduce the comparative experimental setting and evaluate the approximate nearest neighbor search performance of the proposed TOCEH and some state-of-the-art hashing methods.

3.1. Datasets

The comparative experiments are conducted on three large-scale RS image datasets, including UC Merced land use dataset (UCMD) [37], SAT-4 dataset [38] and SAT-6 dataset [38]. The details of these three RS image datasets are introduced below.

1.

UCMD [37] stores aerial image scenes with a human label. There are 21 land cover categories, and each category includes 100 images with the normalized size of 256 × 256 pixels. The spatial resolution of each pixel is 0.3 m. We randomly choose 420 images as query samples and the remaining 1680 images are utilized as training samples.

2.

The total number of images in SAT-4 [38] is 500k and it includes four broad land cover classes: barren land, grass land, trees and other. The size of images is normalized to 28 × 28 pixels and the spatial resolution of each pixel is 1 m. We randomly select 400k images to train the network and the other 100k images to test the ANN search performance.

3.

The SAT-6 [38] dataset contains 405k images covering barren land, buildings, grassland, roads, trees and water bodies. These images are normalized to 28 × 28 pixels size and the spatial resolution of each pixel is 1 m. We randomly select 81k images as query set and the other 324k images as training set.

Some sample images of the above three datasets are shown in Figure 4, Figure 5 and Figure 6, and the statistics are summarized in

Table 2. Statistics and several parameter settings of three datasets.

	UCMD	SAT4	SAT6
Class Number	21	4	6
Image Size	256 × 256	28 × 28	28 × 28
Dataset Size	2100	500,000	405,000
Training Set	1470	400,000	360,000
Query Set	630	100,000	45,000
Ground Truth	100	1000	1000

.

3.2. Experimental Settings and Evaluation Matrix

To verify the ANN search performance of the proposed TOCEH method, many state-of-the-art hashing methods, including locality sensitive hashing (LSH) [14], spectral hashing (SH) [17], iterative quantization hashing method (ITQ) [13], k-means hashing (KMH) [12], partial randomness hashing (PRH) [23], deep variational binaries (DVB) [39], deep hashing (DH) [40], DeepBit [41], deep Cauchy hashing (DCH) [11] and twin-bottle neck hashing (TBH) [10] are utilized as the baseline methods. LSH [14], SH [17], ITQ [13] and KMH [12] belong to the shallow methods. During the ANN search experiments, we extract the content information from RS images by AlexNet and the features are represented as 4096-dimension float point data. Then, these shallow hashing methods map the 4096-dimension features into the compact Hamming space and achieve the ANN search task according to the Hamming distance. DCH [11], TBH [10], DVB [39], DH [40], DeepBit [41] and the proposed TOCEH are deep learning hashing methods. They directly generate the RS image’s binary feature using an end-to-end mechanism.

The training process and comparative experiments are conducted on a high-performance computer with GPU Tesla T4 16 GB, CPU Intel Xeon 6242R 3.10 GHz and 64 GB RAM.

To evaluate the ANN search performance, two widely used standards, mean average precision (mAP) and recall curves, are employed in this paper.

The recall curve represents the fraction of the positive samples that are successfully retrieved. The definition of recall is shown in Equation (16). #(⋅) returns the number of samples.

$$recall=\frac{\#(retrieved positive samples)}{\#(all positive samples)}$$

(16)

Mean average precision value expresses the return rate of positive samples as defined in Equation (17). |total| is the total number of retrieved samples. K_i returns the number of positive samples of the i-th query sample. rank(j) is the ranking number of the j-th positive sample in the retrieved results.

$$mAP=\frac{1}{\left|total\right|}{\displaystyle \sum _{i=1}^{\left|total\right|}\frac{1}{K_i}{\displaystyle \sum _{j=1}^{K_i}\frac{j}{rank(j)}}}$$

(17)

3.3. Experimental Results

3.3.1. Qualitative Analysis

In this section, we show the qualitative image search results on the UCMD dataset [37]. The proposed TOCEH and the other seven state-of-the-art methods separately map the image content information into 64-, 128- and 256-bit binary code. The images with minimal Hamming distance to the query sample are returned as retrieval results and the false images are marked with red rectangles, as shown in Figure 7, Figure 8 and Figure 9.

From the RS image retrieval results, we intuitively know that TOCEH owns the best retrieval results. When encoding RS image content as a 64-bit binary code in Figure 6, TOCEH and TBH [10] return two false positive images. Correspondingly, the number of false images retrieved by the other six methods is larger than two. Furthermore, the false RS images’ ranking position in TOCEH is higher than that in TBH [10], which gives TOCEH a larger mAP value. In Figure 7, the length of the binary code is 128. One RS image is incorrectly returned by TOCEH, TBH [10], DCH [11] and PRH [23], and the false image has a relatively higher ranking position in TOCEH. As the number of binary bits increases to 256, only TOCEH and TBH [10] retrieve no false image, as shown in Figure 8.

3.3.2. Quantitative Analysis

In this section, we adopt recall curves and mAP to quantitatively analyze the ANN search performance of the proposed TOCEH and the other seven state-of-the-art hashing methods. These hashing methods separately generate 64-, 128-, and 256-bit binary code to represent the image content. The mAP values are in

Table 3. Comparison of mAP with different binary code lengths on UCMD.

	TOCEH	TBH	DVB	DCH	DeepBit	PRH	DH	KMH	ITQ	SH	LSH
64-bit	0.3914	0.3415	0.3261	0.2917	0.2657	0.2462	0.2296	0.2135	0.1986	0.1724	0.1637
128-bit	0.5479	0.4638	0.4259	0.3963	0.3781	0.3527	0.3467	0.2816	0.2462	0.2015	0.1842
256-bit	0.5837	0.4975	0.4757	0.4319	0.4197	0.3746	0.3528	0.3168	0.2673	0.2351	0.2148

,

Table 4. Comparison of mAP with different binary code lengths on SAT-4.

	TOCEH	TBH	DVB	DCH	DeepBit	PRH	PRH	KMH	ITQ	SH	LSH
64-bit	0.7011	0.5768	0.5271	0.4862	0.4522	0.4361	0.4139	0.3946	0.3657	0.3482	0.3407
128-bit	0.7236	0.6124	0.5537	0.4986	0.4794	0.4528	0.4385	0.4173	0.3856	0.3724	0.3615
256-bit	0.7528	0.6345	0.6149	0.5128	0.5068	0.4857	0.4653	0.4361	0.4285	0.4152	0.3986

and

Table 5. Comparison of mAP with different binary code lengths on SAT-6.

	TOCEH	TBH	DVB	DCH	DeepBit	PRH	DH	KMH	ITQ	SH	LSH
64-bit	0.7124	0.5826	0.5446	0.4936	0.4725	0.4586	0.4352	0.4125	0.3764	0.3695	0.3628
128-bit	0.7351	0.6268	0.5841	0.5174	0.4921	0.4795	0.4596	0.4281	0.3927	0.3864	0.3752
256-bit	0.7842	0.6527	0.6261	0.5394	0.5175	0.4972	0.4628	0.4516	0.4359	0.4238	0.4175

. The recall curves are shown in Figure 10, Figure 11 and Figure 12.

From the quantitative results, we know TOCEH achieves the best ANN search performance. LSH [14], the data-independent hashing algorithm, randomly generates hashing projection functions without a training process. As a result, the ANN search performance of LSH cannot drastically improve as the number of binary bits increases [9]. In contrast, the proposed TOCEH and the other nine comparative hashing methods utilize a machine learning mechanism to obtain the hashing functions, which are adaptive to the training data distribution. Thus, these machine-learning-based hashing algorithms achieve a better ANN search performance than LSH. SH [17] establishes a spectral graph to measure the similarity relation among samples, and divides the samples into different cluster groups by spectral graph partition. Then, SH [17] assigns the same code to the samples in the same group. For a large-scale RS image dataset, the time complexity of establishing a spectral graph would be high. Both ITQ [13] and KMH [12] first learn encoding centers, then assign the samples as the same binary code as their nearest center. ITQ [13] considers the fixed vertexes of a hyper cubic as centers, but they are not well adapted to the training data distribution. KMH [12] learns the encoding centers with minimal quantization loss and similarity loss by a k-means iterative mechanism. This measure effectively helps KMH improve the ANN search performance. To balance the training complexity and ANN search performance, PRH [23] employs the partial randomness and partial learning strategy to generate hashing functions. LSH [14], SH [17], ITQ [13], KMH [12] and PRH [23] belong to the shallow hashing algorithms, and their performances relate to the quality of the intermediate high dimensional features. To eliminate this effect, TOCEH, TBH [10], DVB [39], DH [40], DeepBit [41] and DCH [11] adopt a deep learning framework to learn the end-to-end binary feature, which can further boost the ANN search performance. The classical DH [40] proposes three constraints at the top layer of the deep network: the quantization loss, balance bits and independent bits. However, the pair-wise similarity preserving or the triplet ordinal relation preserving is not considered in DH. This may lead a poor performance of DH. The same problem also exists in DeepBit [41]. However, DeepBit augments the training data with different rotations and further updates the parameters of the network. This measure helps DeepBit to obtain a better ANN search performance than DH. For most deep hashing, it is hard to unveil the intrinsic structure of the whole sample space by simply regularizing the output codes within each single training batch. In contrast, the conditional auto-encoding variational Bayesian networks are introduced in DVB to exploit the feature space structure of the training data using the latent variables. DCH [11] pre-trains a similarity graph and expects that the probability distribution in the Hamming space should be consistent with that in the Euclidean space. TBH [10] abandons the process of the pre-computing similarity graph and embeds it in the deep neural network. TBH aims to preserve the similarity between the original data and the data decoded from the binary feature. Both TBH [10] and DCH [11] aim to preserve the pair-wise similarity, and it is difficult to capture the hyper structure among RS images. TOCEH establishes a tensor graph representing the triplet ordinal relation among RS images in both Hamming space and Euclidean space. During the training process, TOCEH expects that the triplet ordinal relation graphs have the same distribution in different spaces. Thus, it can enhance the ability of preserving the Euclidean ranking orders in the Hamming space. As discussed above, TOCEH can achieve the best RS image retrieval results.

3.3.3. Ablation Experiments

To guarantee the ANN search performance of the obtained binary codes, the TOCEH algorithm proposes two key components: the triplet ordinal cross entropy loss and the triplet ordinal quantization loss. Here, we conduct the comparative experiments to analyze these two components. TOCEL only utilizes the triplet ordinal cross entropy loss as the objective function for deep learning binary code. The deep hashing TOQL only employs the triplet ordinal quantization loss as the objective function. TOCEH, TOCEL and TOQL separately map the data into 64- and 128-bit binary code. The ANN search results are shown in Figure 13, Figure 14 and Figure 15.

From the comparative results, we know that both the triplet ordinal cross entropy loss and the triplet ordinal quantization loss play important roles in improving the performance of TOCEH. The triplet ordinal cross entropy loss minimizes the inconsistency between the probability distributions of the triplet ordinal relations in different spaces. For example, the data pair (x_i, x_j) is more similar than data pair (x_j, x_k) in the Euclidean space. Then, to minimize the triplet ordinal cross entropy loss, it should be a larger probability to assign x_i and x_j as similar binary codes. Without the triplet ordinal cross entropy loss, TOQL randomly generates the samples’ binary codes. LSH algorithm also randomly generates the hashing functions. Thus, the ANN search performance of TOQL is almost the same as that of LSH. To fix the NP hard problem of the objective function, we apply the continuous relaxation mechanism to the binary encoding procedure. Furthermore, we define the triplet ordinal quantization loss to minimize the loss between the binary codes and the corresponding continuous variable. Without the triplet ordinal quantization loss, the difference between the optimized variables and the binary encoding results would become larger in TOCEL. Thus, TOCEL has a relatively inferior ANN search performance. As discussed above, both the triplet ordinal cross entropy loss and the triplet ordinal quantization loss are necessary for the TOCEH algorithm.

4. Conclusions

In this paper, to boost the RS image search performance in the Hamming space, we propose a novel deep hashing method called triplet ordinal cross entropy hashing (TOCEH) to learn an end-to-end binary feature of an RS image. Generally, most of the existing hashing methods place emphasis on preserving point-wise or pair-wise similarity. In contrast, TOCEH establishes a tensor graph to capture the triplet ordinal relation among RS images and defines the triplet ordinal relation preserving problem as the formulation of minimizing the cross entropy value. Then, TOCEH achieves the aim of preserving triplet ordinal relation by minimizing the inconsistency between the probability distributions of the triplet ordinal relations in different spaces. During the training process, to avoid the NP hard problem, we apply continuous relaxation to the binary encoding process. Furthermore, we define a quantization function based on the triplet ordinal relation preserving restriction, which can reduce the loss caused by the continuous procedure. Finally, the extensive comparative experiments conducted on three large-scale RS image datasets, including UCMD, SAT-4 and SAT-6, show that the proposed TOCEH outperforms many state-of-the-art hashing methods in RS image search tasks.