A Sketch-Based Cross-Modal Retrieval Model for Building Localization Without Satellite Signals

Abstract

In existing non-satellite navigation systems, visual localization is widely adopted for its high precision. However, in scenarios with highly similar building structures, traditional visual localization methods that rely on direct coordinate prediction often suffer from decreased accuracy or even failure. Moreover, as scene complexity increases, their robustness tends to decline. To address these challenges, this paper proposes a Sketch Line Information Consistency Generation (SLIC) model for indirect building localization. Instead of regressing geographic coordinates, the model retrieves candidate building images that correspond to hand-drawn sketches, and these retrieved results serve as proxies for localization in satellite-denied environments. Within the model, the Line-Attention Block and Relation Block are designed to extract fine-grained line features and structural correlations, thereby improving retrieval accuracy. Experiments on multiple architectural datasets demonstrate that the proposed approach achieves high precision and robustness, with mAP@2 values ranging from 0.87 to 1.00, providing a practical alternative to conventional coordinate-based localization methods.

1. Introduction

Location plays an important role in various fields, from transportation and outdoor exploration to agriculture and the military, covering a wide range of applications [1,2,3,4,5,6,7,8]. It helps people achieve accurate navigation in different environments, improving efficiency, safety and productivity. Moreover, as a crucial component of location technology, semantic location takes into account not only the user’s intended destination but also factors in the surrounding context and semantic information, offering a more comprehensive and user-centric location experience [9,10,11,12,13,14,15,16]. Compared to traditional location systems, semantic location is characterized by its intelligence, offering a significantly enhanced and personalized location experience with greater accuracy.

Among the methods of semantic location, sketch information is a crucial component of the semantics. Sketches are simplified, rapid, unfinished drawings designed to capture the main features of ideas, concepts or observations, emphasizing personal expression. Consequently, sketches inherently contain a wealth of high-level semantic information. Especially with the wide use of touch screen devices such as mobile phones and tablets, it is effortless to record the information in the form of sketches, making sketches a quite convenient practice. Therefore, combining sketches with location, using simple strokes to depict the target location and matching with the real location, the work of positioning and navigation can be effectively accomplished. Furthermore, even in situations where satellite signals are unavailable or unreliable for positioning purposes, by building a sketch-location database for a specific area, the location and navigation task can be completed at a lower cost and higher stability [17].

Meanwhile, sketch-based image retrieval (SBIR) [18,19,20,21,22,23]—matching natural photos with free-hand sketches—has drawn a lot of interest. Various advancements in SBIR architectures have evolved based on several foundational concepts, including multiple independent networks [24], multi-layer networks [25], and semi-heterogeneous networks [26]. However, these networks are oriented to a wide range of objects, and most of them are sketch retrieval networks about animals or objects in life. As a result, when the image retrieval of sketches is oriented to architectural objects, its accuracy often cannot meet the requirements. In order to solve the above problems, this paper proposes a Sketch Line Information Consistency Generation model, which uses the attention mechanism to extract fine texture features such as lines in sketches and buildings.

Therefore, we propose a novel semantic localization method based on line information. This method accomplishes localization tasks by matching hand-drawn sketches with actual building groups, as shown in Figure 1. The fundamental principle of this localization method revolves around achieving alignment between the image and the sketch. The primary challenges lie in the cross-domain matching problem and the issue of high similarity features. To address these challenges, we employ a two-branch structured generative adversarial network in the overall network architecture. This network aims to discover two types of mapping functions simultaneously, mapping the original image and sketch into a shared semantic space with matching distributions, thus enhancing the matching process. Furthermore, the second challenge involves the precision of locating highly similar buildings. Although the two mapping functions ensure a shared space, the distance between feature vectors from different categories may not be well preserved, leading to confusion in locating similar buildings. To tackle this issue, we introduce the L-A and R-A modules within the neural network’s training process. These modules utilize the distinctive line information in buildings as the foundation for feature vector convergence, maximizing the differentiation between different categories. In addition, we leverage Gradient-Weighted Class Activation Mapping (Grad-CAM) [27] to visualize and analyze class-discriminative regions, which provides guidelines for abstracting architectural sketches into effective line representations. This integration helps explain why line features are emphasized in our design and further justifies the role of sketches in cross-modal localization. Lastly, we consider the selection of sketch datasets for training the network. The dataset is curated from individuals without a drawing background, simulating architectural outline information obtained through recall, thus enhancing the model’s universality.

In summary, the main contributions of this paper can be concluded as follows.

• A semantic location method based on sketch retrieval is introduced as an innovative approach for achieving precise positioning and navigation, without relying on satellite technology. The method involves matching hand-drawn architectural sketches created by individuals without any prior art skills to real-world buildings. The result is a remarkably robust and effective location-based navigation system, which is validated through retrieval accuracy experiments on multiple architectural datasets in Section 3.4.
• To derive a set of universal sketching principles, sketch-original datasets encompassing various architectural styles were constructed, and corresponding sketching experiments were carried out. The Grad-CAM method is used to visualize the position of key lines and further simplify the way sketches are drawn. Ultimately, three fundamental principles for retaining essential lines were formulated based on distinctive line characteristics, and their impact on retrieval accuracy is demonstrated in the sketch simplification experiments of Section 3.4.
• In the context of aligning sketches with their corresponding original drawings, a novel approach involves the use of a two-branch generative network that relies on line information. Two types of modules, L-A Block and R-A Block, are proposed in dealing with the matching problem of buildings with similar height. The former extracts the line features and content information of the building, while the latter deeply associates the information with the help of the characteristics of the association network. The effectiveness of these modules is confirmed by the ablation studies in Section 3.5.

The rest of this article is organized as follows. Section 1 introduces related works on both measurement and feature aspects in semantic navigation and sketch-based image retrieval. Section 2 briefly presents the proposed system framework. Section 3 provides the details of the sketch line information consistency generative model and presents the experimental results. Finally, the conclusion is summarized at the end of Section 4.

2. Materials and Methods

In this work, we introduce the SLIC model for sketch-based building localization. The main framework is built upon a generative network based on CycleGAN, aiming to establish a shared semantic space for retrieval and matching. Simultaneously, an auxiliary framework combines attention mechanisms to align the original sketches with the common semantic space, effectively separating highly similar buildings. The complete architecture of our model is illustrated in Figure 2.

The SLIC model is a generative model that effectively aligns the line information of original buildings and sketch buildings while ensuring consistency in spatial features during the alignment process. It begins by establishing a common semantic space for mapping architectural sketches and original images. The auxiliary modules in SLIC, namely the L-A module and the R-A module, play crucial roles: The L-A module is utilized for extracting line features specific to each type of architectural standard. These features are instrumental in executing alignment operations during mapping, thus accomplishing the preliminary task of building positioning. Meanwhile, the R-A module delves deeper into the correlation of line features, particularly when dealing with highly similar buildings. Its purpose is to achieve high-precision positioning by refining the alignment process, ensuring accurate mappings even in cases of architectural similarity.

We define $\mathcal{D}=\{\mathbf{X},\mathbf{Y}\}$ as the dataset comprising sketches and images belonging to known categories. The dataset includes sketch images $\mathbf{X}={\left\{{\mathbf{x}}_i\right\}}_{i=1}^N$ and natural images $\mathbf{Y}={\left\{{\mathbf{y}}_i\right\}}_{i=1}^N$, where N represents the category numbers in the sketch-image datasets. It is worth noting that sketches and images share the same labeling convention, simplifying the association of corresponding pairs. The auxiliary information $\mathbf{L}$ is defined as the output from the line attention network. The information effectively directs the attention of the generation network towards texture details, enhancing matching accuracy, particularly in scenarios where real image texture information is faint. The primary objective of our model is to learn two functions, denoted as $G_{\mathrm{s}}(·)$ and $G_{\mathrm{i}}(·)$, for mapping sketches and natural images into a shared semantic space. When provided with a sketch and an image from distinct domains, these proposed functions, $G_{\mathrm{s}}:{\mathbb{R}}^d⟶{\mathbb{R}}^M$ and $G_{\mathrm{i}}:{\mathbb{R}}^d⟶{\mathbb{R}}^M$, respectively, map them to the same semantic space where retrieval and matching tasks can be efficiently performed. Following the mapping process described above, we obtain features ($I_{\mathrm{ie}}$ & $I_{\mathrm{se}}$) from both the sketch and original images. These features are then fed into the R-A module for similarity matching, which ultimately yields the final positioning results.

2.1. Cycle-Consistent Generative Model

When dealing with the training of depth equations for sketches and images, we adopt a cycle-consistent generative frame. Each branch in the frame is aligned with a common discriminator. The cycle consistency not only ensures that the sketch and image are mapped to the semantic space but also ensures that the spatial location is consistent in the inverse mapping back to the original space. Therefore, it is only necessary to add a loss function to the category information to achieve the generation of discriminative features.

The unique semantic information of architecture lies in its rich texture, whether sketch or original drawing, contour or line information should be the main focus. Our main goal is to enable the two kinds of mapping equations to learn this kind of texture information separately and transfer to the common semantic space. However, different from Zhu et al. [28], who adopt two types of images in different domains to achieve cyclic consistency, we use line feature information on the common semantic space as an intermediary to achieve cyclic consistency of sketches and pictures, respectively. By training $G_{\mathrm{s}}(·)$, the architectural sketch $\mathbf{X}$ is mapped to the $\mathbf{L}$ so that ${\widehat{\mathbf{l}}}_i=G_{\mathrm{s}}\left({\mathbf{x}}_i\right)$, and ${\mathbf{l}}_i\in \mathbf{L}$ is the line information used to complete the adversarial training. Thereby, the trained $G_{\mathrm{s}}(·)$ is able to transform the modality $\mathbf{X}$ to a modality $\widehat{\mathbf{L}}$ that satisfies the distribution in $\mathbf{L}$.

For adversarial loss, there exist three classes of adversarial loss regarding sketch, image, and semantics. We have designed image and sketch generators, $G_{\mathrm{s}}:\mathbf{X}\to \mathbf{L}$, $G_{\mathrm{i}}:\mathbf{Y}\to \mathbf{L}$, $F_{\mathrm{s}}:\mathbf{L}\to \mathbf{X}$, and $F_{\mathrm{i}}:\mathbf{L}\to \mathbf{Y}$, and three corresponding adversarial discriminators: $D_{\mathrm{s}}(·)$, $D_{\mathrm{i}}(·)$, and $D_{\mathrm{se}}(·)$, where $D_{\mathrm{se}}$ is used to distinguish between the semantic information of the original line $\left\{l\right\}$ and the semantic information of the sketch $\left\{G_{\mathrm{s}}\left(x\right)\right\}$ and the semantic information of the image $\left\{G_{\mathrm{i}}\left(y\right)\right\}$. $D_{\mathrm{s}}$ is used to distinguish the original sketch information $\left\{x\right\}$ from the sketch information transformed from the semantic space $\left\{F_{\mathrm{s}}\left(l\right)\right\}$. Similarly, $D_{\mathrm{i}}$ is used to distinguish $\left\{y\right\}$ from $\left\{F_{\mathrm{i}}\left(l\right)\right\}$. For $G_{\mathrm{s}}$ and $G_{\mathrm{i}}$, the objective function is as follows:

$$\begin{array}{c}\hfill L_{\mathrm{ad}}(G_{\mathrm{s}},G_{\mathrm{i}},D_{\mathrm{se}},\mathbf{x},\mathbf{y},\mathbf{l})=2\times \mathbb{E}\lfloor \log D_{\mathrm{se}}\left(\mathbf{l}\right)\rfloor \\ \hfill +\mathbb{E}\left[\log (1-D_{\mathrm{se}}\left(G_{\mathrm{s}}\left(\mathbf{x}\right)\right))\right]+\mathbb{E}\left[\log (1-D_{\mathrm{se}}\left(G_{\mathrm{i}}\left(\mathbf{y}\right)\right))\right]\end{array}$$

(1)

Among them, $G_{\mathrm{s}}$ and $G_{\mathrm{i}}$ intend to minimize the objective function, while $D_{\mathrm{se}}$ tries to maximize the objective function. Similarly, for $F_{\mathrm{s}}$ and $D_{\mathrm{s}}$, the objective function is as follows:

$$\begin{array}{c}\hfill L_{\mathrm{ad}}(F_{\mathrm{s}},D_{\mathrm{s}},\mathbf{x},\mathbf{l})=\mathbb{E}\lfloor \log D_{\mathrm{s}}\left(\mathbf{x}\right)\rfloor \\ \hfill +\mathbb{E}\left[\log (1-D_{\mathrm{s}}\left(F_{\mathrm{s}}\left(\mathbf{l}\right)\right))\right]\end{array}$$

(2)

$F_{\mathrm{s}}$ minimizes the objective function and the discriminator $D_{\mathrm{s}}$ intends to maximize it. The same goes for $F_{\mathrm{i}}$ and $D_{\mathrm{i}}$.

$$\begin{array}{c}\hfill L_{\mathrm{ad}}(F_{\mathrm{i}},D_{\mathrm{i}},\mathbf{y},\mathbf{l})=\mathbb{E}\lfloor \log D_{\mathrm{i}}\left(\mathbf{y}\right)\rfloor \\ \hfill +\mathbb{E}\left[\log (1-D_{\mathrm{i}}\left(F_{\mathrm{i}}\left(\mathbf{l}\right)\right))\right]\end{array}$$

(3)

For the cycle consistency loss, while adversarial learning can help reduce the gap between the sketch domain and image domain, it does not guarantee the alignment of information in terms of position when they are transformed into the common semantic space. To address this concern, we employ the cycle consistency approach. This approach maps both the sketch and the image to the corresponding common semantic space and then transforms them back to their original feature spaces, ensuring spatial alignment.

Specifically, when training the two transformations: $G_{\mathrm{s}}:\mathbf{X}\to \mathbf{L}$ and $F_{\mathrm{s}}:\mathbf{L}\to \mathbf{X}$, they are designed to be inverse operations of each other. The objective function is as follows:

$$\begin{array}{c}\hfill L_{\mathrm{cy}}(G_{\mathrm{s}},F_{\mathrm{s}})=\mathbb{E}\left[{∥F_{\mathrm{s}}\left(G_{\mathrm{s}}\left(\mathbf{x}\right)\right)-\mathbf{y}∥}_1\right]\\ \hfill +\mathbb{E}\left[{∥G_{\mathrm{s}}\left(F_{\mathrm{s}}\left(\mathbf{l}\right)\right)-\mathbf{l}∥}_1\right]\end{array}$$

(4)

Similarly, we apply cycle-consistency loss to these two types of equations $G_{\mathrm{i}}:\mathbf{Y}\to \mathbf{L}$ and $F_{\mathrm{i}}:\mathbf{L}\to \mathbf{Y}$:

$$\begin{array}{c}\hfill L_{\mathrm{cy}}(G_{\mathrm{i}},F_{\mathrm{i}})=\mathbb{E}\left[{∥F_{\mathrm{i}}\left(G_{\mathrm{i}}\left(\mathbf{y}\right)\right)-\mathbf{y}∥}_1\right]\\ \hfill +\mathbb{E}\left[{∥G_{\mathrm{i}}\left(F_{\mathrm{i}}\left(\mathbf{l}\right)\right)-\mathbf{l}∥}_1\right]\end{array}$$

(5)

For classification loss, the adversarial loss and cycle consistency loss only ensure that the corresponding sketches and images are closer to each other in the new domain and retain certain position information. However, it does not guarantee that the trained generator has category-level discrimination. Therefore, we add the classification loss to the features generated by the generator:

$$\begin{array}{c}\hfill L_{\mathrm{cl}}\left(G_{\mathrm{s}}\right)=-\mathbb{E}\left[\log P\left(ca\right|G_{\mathrm{s}}\left(\mathbf{x}\right);\theta )\right]\end{array}$$

(6)

where ca stands for the class label. Similarly, the classification loss is added to the generator $G_{\mathrm{i}}$ as well.

$$\begin{array}{c}\hfill L_{\mathrm{cl}}\left(G_{\mathrm{i}}\right)=-\mathbb{E}\left[\log P\left(ca\right|G_{\mathrm{i}}\left(\mathbf{y}\right);\theta )\right]\end{array}$$

(7)

Through the loss setting of the above circular architecture, two types of features $I_{\mathrm{ie}}$ and $I_{\mathrm{se}}$ can be obtained for matching. Further feature extraction is carried out by the subsequent R-A network to achieve accurate matching of buildings with high similarity.

2.2. L-A Block and R-A Block

The proposed cyclic consistency framework can effectively align two heterogeneous modalities while preserving spatial information. However, its performance can be affected when dealing with poor separation of architectural buildings and complex backgrounds (e.g., nighttime scenes) or high similarity between buildings (e.g., similar color or outline profiles). Therefore, we need to design a network module that can enhance the distinctive line features of buildings. The module consists of two stages: the first stage (L-A Block) is to extract standard features with sketch-like style and image content, which are used to determine the alignment criteria for the common space; the second stage (R-A Block) is to deeply explore the correlations of features in the common space, which is used to distinguish similar types of buildings.

2.2.1. Line-Attention Block

The L-A Block draws inspiration from the ideas presented in SANET [29], characterized by its ability to take both a content image and a style image as input, generating a stylized image that combines semantic structures from the former with characteristics from the latter. In our approach, we utilize the pre-trained VGG-16 network to extract style features from both the sketch and the original image. As shown in Figure 3, the output from different VGG layers serves as the basis for extracting angle and length features, and an attention mechanism is used to efficiently extract line information, even in complex environments. The network is instrumental in improving the model’s ability to handle challenging scenarios where building lines are less distinct from their surroundings.

The fourth and fifth layers of the VGG-16 network are selected as the main feature extraction networks for angle change and length information and denoted as $VG{G}_4$ and $VG{G}_5$, respectively. Meanwhile, the sketch is defined as the style image, and the original image is defined as the content image, which is the input into the two feature extraction networks, and we can obtain the information from different layers, and we denote them as follows. 

$$\begin{array}{c}\hfill F_x^4=VG{G}_4\left(\mathbf{x}\right),F_y^4=VG{G}_4\left(\mathbf{y}\right)\\ \hfill F_x^5=VG{G}_5\left(\mathbf{x}\right),F_y^5=VG{G}_5\left(\mathbf{y}\right)\end{array}$$

(8)

After extracting feature information from both types of images, the primary objective is to calculate the weight matrix within the attention mechanism. With a focus on the straightforward line style of the sketch and an emphasis on content extraction from the original image, the following operations are conducted across different layers.

For the extraction of length information, a relatively shallow fourth layer is employed. The attention weight matrix is derived through matrix multiplication of the sketch and the original image features. Subsequently, the weight matrix is multiplied with the original image features to obtain features that emphasize length information ($L_{len}$).

$$\begin{array}{c}\hfill L_{len}={\displaystyle \frac{1}{C\left(F^4\right)}}\sum _{\forall j}exp\left(f{\left(\overline{F_x^4}\right)}^{T}g{\left(\overline{F_y^4}\right)}_j\right)h\left({\left(F_y^4\right)}_j\right)\end{array}$$

(9)

where function $f(·)$ means $W\times F$, and W is the learned weight matrix, which is implemented as $1\times 1$ convolutions. Meanwhile, $\overline{F}$ denotes a mean-variance channel normalized version of F and $C\left(F^4\right)={\sum }_{\forall j}exp\left(f{\left(\overline{F_x^4}\right)}^{T}g{\left(\overline{F_y^4}\right)}_j\right)$.

Likewise, the angle information ($L_{ang}$) can be obtained as follows.

$$\begin{array}{c}\hfill L_{ang}={\displaystyle \frac{1}{C\left(F^5\right)}}\sum _{\forall j}exp\left(f{\left(\overline{F_x^5}\right)}^{T}g{\left(\overline{F_y^5}\right)}_j\right)h\left({\left(F_y^5\right)}_j\right)\end{array}$$

(10)

We combine the above two output features as

$$\begin{array}{c}\hfill L_{emb}=con{v}_{3\times 3}(L_{len}+upsampling\left(L_{ang}\right))\end{array}$$

(11)

Finally, the stylized line information feature L is synthesized by feeding $L_{emb}$ into the decoder as follows:

$$\begin{array}{c}\hfill L=Decoder\left(L_{emb}\right)\end{array}$$

(12)

After obtaining both types of line information, the integration into the sketch–image matching model serves as a benchmark for optimizing the matching performance.

2.2.2. Relation Block

The task of building localization based on sketches can be preliminarily accomplished using the modules described above. However, when dealing with buildings which are highly similar in appearance, such as those with rectangular shapes and comparable color schemes, there is a risk of confusion and misidentification. To overcome this challenge, a deep information relation module is proposed, designed to extract more refined and distinctive architectural relation from the data.

Inspired by the relational network approach proposed in [30], we aim to incorporate a relational network module after extracting the standard feature vector to better distinguish between similar architectural features in depth. In this work, we assume $C(·,·)$ to be concatenation of two standard feature in depth. The relational network architecture $R_{\psi }$ consists of two fully connected (FC) layers, each followed by a ReLU activation function and a dropout layer. This configuration produces a matching score in the range of (0, 1), which represents the degree of similarity between the input features.

$$\begin{array}{c}\hfill R(I_{\mathrm{ie}},I_{\mathrm{se}})=\mathrm{sigmoid}\left(R_{\psi }\left(C(I_{\mathrm{ie}},I_{\mathrm{se}})\right)\right)\end{array}$$

(13)

This framework differs from the correlation of global image features used in the original literature [30]. The previous approach consumes a higher amount of computational resources, whereas the proposed method utilizes deep information mining and reasoning on the extracted standard features to achieve more efficient retrieval in cases where computation is limited.

2.3. Overall Cost Function

To effectively align sketch and image modalities while ensuring spatial consistency, category discrimination, and robust cross-modal retrieval, we design a composite cost function that integrates adversarial alignment, cycle consistency, category classification, and relation-based retrieval losses. These components jointly minimize the discrepancy between the model predictions and the ground-truth targets at multiple levels of representation. Specifically, to reduce the distribution gap between sketches, images, and the shared semantic line-space $\mathbf{L}$, we employ adversarial training with generators $G_s:\mathbf{X}\to \mathbf{L}$ and $G_i:\mathbf{Y}\to \mathbf{L}$, inverse mappings $F_s$ and $F_i$, and discriminators $D_{se}$, $D_s$, and $D_i$ for semantic, sketch, and image domains, respectively. The adversarial objective is defined as

$$L_{\mathrm{ad}}=L_{\mathrm{ad}}^{se}+L_{\mathrm{ad}}^{sk}+L_{\mathrm{ad}}^{im},$$

(14)

where

$$\begin{array}{cc}\hfill L_{\mathrm{ad}}^{se}=& {\mathbb{E}}_{\ell \sim p\left(\mathbf{L}\right)}\left[\log D_{se}(\ell )\right]+{\mathbb{E}}_{x\sim p\left(\mathbf{X}\right)}\left[\log (1-D_{se}\left(G_s\left(x\right)\right))\right]\hfill \\ \hfill & +{\mathbb{E}}_{y\sim p\left(\mathbf{Y}\right)}\left[\log (1-D_{se}\left(G_i\left(y\right)\right))\right],\hfill \end{array}$$

(15)

$$L_{\mathrm{ad}}^{sk}={\mathbb{E}}_x\left[\log D_s\left(x\right)\right]+{\mathbb{E}}_{\ell }\left[\log (1-D_s\left(F_s(\ell )\right))\right],$$

(16)

$$L_{\mathrm{ad}}^{im}={\mathbb{E}}_y\left[\log D_i\left(y\right)\right]+{\mathbb{E}}_{\ell }\left[\log (1-D_i\left(F_i(\ell )\right))\right].$$

(17)

This adversarial loss encourages the generated distributions from $G_s$ and $G_i$ to become indistinguishable from the real distributions in their respective domains. To further ensure that the mapping preserves spatial and structural information, we incorporate an $L_1$ cycle consistency loss that reconstructs sketches and images after forward and backward mappings:

$$\begin{array}{cc}\hfill L_{\mathrm{cy}}=& {\mathbb{E}}_x\parallel F_s\left(G_s\left(x\right)\right){-x\parallel }_1+{\mathbb{E}}_{\ell }{\parallel G_s\left(F_s(\ell )\right)-\ell \parallel }_1\hfill \\ \hfill & +{\mathbb{E}}_y\parallel F_i\left(G_i\left(y\right)\right){-y\parallel }_1+{\mathbb{E}}_{\ell }{\parallel G_i\left(F_i(\ell )\right)-\ell \parallel }_1.\hfill \end{array}$$

(18)

This ensures semantic alignment without introducing spatial distortion by enforcing invertibility between mappings. Additionally, to retain category-level discriminative information in the semantic space, we apply a standard cross-entropy classification loss to both sketch and image features:

$$L_{\mathrm{cl}}=-{\mathbb{E}}_{(x,c)}\log P\left(c\right|G_s\left(x\right))-{\mathbb{E}}_{(y,c)}\log P\left(c\right|G_i\left(y\right)),$$

(19)

where $P\left(c\right|·)$ is the predicted probability of class c. For sketch–image retrieval, we use a relation head $R(·,·)$ to output similarity score $s\in (0,1)$ between sketch features $I_{\mathrm{se}}$ and image features $I_{\mathrm{ie}}$ and train it with binary cross-entropy against the ground-truth pair label $z\in \{0,1\}$:

$$L_{\mathrm{rel}}=-\mathbb{E}\left[z\log s+(1-z)\log (1-s)\right].$$

(20)

Finally, the overall cost function is expressed as

$$L_{\mathrm{total}}={\lambda }_{ad}{L}_{\mathrm{ad}}+{\lambda }_{cy}{L}_{\mathrm{cy}}+{\lambda }_{cl}{L}_{\mathrm{cl}}+{\lambda }_{rel}{L}_{\mathrm{rel}},$$

(21)

where ${\lambda }_{ad}$, ${\lambda }_{cy}$, ${\lambda }_{cl}$, and ${\lambda }_{rel}$ are hyperparameters that balance the contributions of each loss component. During training, the total loss is minimized with respect to the generators, classifiers, and relation network, while the adversarial components are maximized with respect to the discriminators following a standard GAN min–max optimization scheme. This formulation explicitly connects the differences between ground-truth and predicted outputs at distributional, reconstruction, categorical, and pairwise retrieval levels within a unified optimization objective.

2.4. Algorithm Description

To enhance the reproducibility of our method, we summarize the key steps of the proposed Sketch Line Information Consistency Generation (SLIC) model using pseudocode. The training procedure integrates the adversarial loss, cycle-consistency loss, and classification loss defined in Equations (1)–(7), while the inference stage describes the sketch-to-image retrieval process. The overall optimization objective follows Equation (8).

The training algorithm (Algorithm 1) ensures that sketches and images are aligned in a shared semantic space by jointly optimizing adversarial, cycle consistency, and classification losses. The retrieval algorithm (Algorithm 2) performs inference by matching a query sketch to candidate building images, thereby enabling indirect localization in satellite-denied environments.

Algorithm 1 Training procedure of the SLIC model

Require:  Sketch dataset $\mathbf{X}$, Image dataset $\mathbf{Y}$, Line-attention features $\mathbf{L}$ Ensure:  Trained model parameters 1: Initialize parameters of generators $G_s,G_i,F_s,F_i$ and discriminators $D_s,D_i,D_{se}$ 2: for each training iteration do 3:   Sample mini-batch $\mathbf{x}\in \mathbf{X}$, $\mathbf{y}\in \mathbf{Y}$, $\mathbf{l}\in \mathbf{L}$ 4:   Adversarial loss (Equations (1)–(3)):  Compute $L_{ad}(G_s,G_i,D_{se},\mathbf{x},\mathbf{y},\mathbf{l})$, $L_{ad}(F_s,D_s,\mathbf{x},\mathbf{l})$, $L_{ad}(F_i,D_i,\mathbf{y},\mathbf{l})$ 5:   Cycle-consistency loss (Equations (4) and (5)): Compute $L_{cy}(G_s,F_s)$ and $L_{cy}(G_i,F_i)$ 6:   Classification loss (Equations (6) and (7)): Compute $L_{cl}\left(G_s\right)$ and $L_{cl}\left(G_i\right)$ 7:   Total loss (Equation (8)): $L_{total}={\lambda }_{ad}\left(L_{ad}\right)+{\lambda }_{cy}\left(L_{cy}\right)+{\lambda }_{cl}\left(L_{cl}\right)$ 8:   Update $G_s,G_i,F_s,F_i$ by minimizing $L_{total}$ 9:   Update $D_s,D_i,D_{se}$ by maximizing adversarial objectives 10: end for 11: return Trained model parameters

Algorithm 2 Sketch-to-Image Retrieval with SLIC

Require:  Query sketch $S_q$, Image database $\mathcal{I}$, Trained model parameters $\theta$ Ensure:  Retrieved building image $I^{*}$ 1: Extract sketch features $S_e=\mathrm{VGG}-16\left(S_q\right)$ 2: for each image $I\in \mathcal{I}$ do 3:   Extract image features $I_e=\mathrm{VGG}-16\left(I\right)$ 4:   Compute similarity score $sim(S_e,I_e)$ using R-A Block 5: end for 6: Rank all images by similarity scores 7: return Top-1 (or Top-K) retrieved image $I^{*}$

3. Results

3.1. Simulation Setup

The effect of network retrieval was evaluated through a series of buildings in different environments and ablation experiments. To establish a standardized set of sketching rules, key stroke experiments were executed. The experiment involved selectively removing unnecessary strokes from the sketch dataset by setting accuracy thresholds for matching. To confirm the universality of the stroke rule and the network, we explored a range of architectural building styles and conducted matching experiments. This included images of buildings from different regions and at various time points. To assess the impact of line attention networks on the overall network retrieval performance, ablation experiments were conducted, with consideration given to the matching effect of buildings at different times. All the above experiments were carried out using the Python programming language (version 3.8, Python Software Foundation, Wilmington, DE, USA) and PyTorch (version 1.9, Meta Platforms, Inc., Menlo Park, CA, USA), and were run on an Intel Core i7-12700KF CPU (Intel Corporation, Santa Clara, CA, USA) with 32 GB of RAM (Kingston Technology, Fountain Valley, CA, USA).

3.2. Datasets

In research of localization, there is still limited exploration of semantic localization based on architectural sketches, which has resulted in a lack of publicly available datasets. To more effectively validate our model and establish reliable sketching guidelines, we constructed both a sketch dataset and a building image dataset, as illustrated in Figure 4. The building image dataset consists of 576 original building photographs, covering four categories of building groups with distinct architectural line styles. To ensure diversity and robustness, the images were captured under both daytime and nighttime conditions. As shown in Figure 5, we collected 2466 sketches corresponding to the 576 building photographs, with each image described from multiple viewing angles. All sketches were drawn by 21 participants without prior drawing experience, ensuring diverse drawing styles that closely resemble real human navigation scenarios. This comprehensive dataset design not only allows for a rigorous evaluation of our model’s robustness across varying environments but also provides a valuable resource for future research in sketch-based localization.

The dataset focuses on building groups with relatively similar outlines and line structures to emphasize fine-grained discrimination, and this choice inevitably limits domain adaptability. Moreover, the current dataset was collected from adult participants without prior drawing experience, which reflects the target user group of typical navigation tasks but does not capture other possible factors such as hurried drawing conditions, contributions from children, or sketches containing uncontrolled noise. In future work, we plan to expand the dataset by including a broader range of architectural styles, as well as more diverse participant conditions, so as to further improve both generalization and realism.

3.2.1. Sketch Dataset

In order to better simulate the scene of human semantic navigation, the selection of sketch data set has a high standard. The main crowd of sketch drawing is without painting foundation, which ensures that the model can learn the characteristics of different style lines as much as possible, in contrast to the network trained with neat sketches formed by painting foundation. In addition, in the initial sketch drawing, due to the lack of a guiding method for drawing architectural sketches, the external outline and internal detail features of the building are preserved to some extent. Subsequently, the number of strokes of the building sketch will be controlled to simplify the rendering process and extract only the key features required for building classification as much as possible. Finally, these relatively complex sketches will be cut through subsequent experiments so as to achieve the purpose of simplifying the sketch as much as possible.

3.2.2. Buildings Dataset

For the building images, four distinct styles architecture are selected to ensure adaptability to different building types. These include SJTU, PARIS, OXBUILD, and structures around OXFORD. For example, OXBUILD’s buildings exhibit intricate backgrounds and a wide array of architectural lines, offering a complex and diverse visual landscape. PARIS, on the other hand, is dominated by larger structures, resulting in simpler backgrounds with predominantly straight architectural lines. During the process of image acquisition, it was essential to replicate the real-world scenario of architectural navigation. The process required traversing the spaces between buildings to capture images from various angles. Consequently, the images were obtained along the path, each showcasing buildings from different perspectives. Furthermore, when capturing color images of the buildings, meticulous care was taken to exclude foreground objects like trees and vehicles. The behavior was to prevent foreground elements from occupying a significant portion of the image, which could otherwise diminish the prominence of the building in the picture. Such a setup ensures that the building stands out distinctly and does not hinder the subsequent matching experiments between the color image and the skeletal representation.

3.2.3. Night Dataset

In addition to the daytime data, a class of nighttime test data is also collected for selected buildings. Since nighttime navigation is in demand to some extent, verifying the retrieval effect of nighttime data is also to verify whether the nighttime navigation success probability can be guaranteed. Considering that the line attention network structure is an important part of extracting the edge information of color pictures, the effectiveness of this module can also be verified from the side through the night data set and ablation experiments.

In the final stage of preprocessing for both color images and stick figures within each class, the objective is to prepare the data for subsequent tasks, such as model training. The images undergo normalization to standardize their pixel values, and all images are uniformly resized to 256 × 256 pixels. This standardization ensures consistent input dimensions for the neural network. Furthermore, to establish a balanced training dataset, the paper employs a random division approach, allocating 70% of the data to the training set. This random division strategy is employed to mitigate potential biases and the undue influence of peculiarities in the training results.

3.3. Experiments on the Principles of Core Sketching

To establish general rules for sketching strokes, a stroke deletion experiment was conducted to identify key strokes. The primary objective was to extrapolate shared features from a single building class to most building classes. SJTU was chosen as the experimental data set, and sketches with different stroke numbers were drawn for SJTU’s unified architecture. In order to preliminarily identify the key focus areas of the proposed framework, we leverage the Grad-CAM algorithm. Grad-CAM utilizes gradient backpropagation to assign weights to the various convolutional layers, and then aggregates these weighted contributions to obtain the importance of different spatial locations within the input image. According to Figure 6, we can observe the line angle characterization in the sketch drawing: the core internal structure of the building and the outline of the building may become the focus of attention. Based on this phenomenon, further sketch drawing line optimization experiments can be conducted.

To assess the impact of stroke deletion, a reduction principle was applied: reduce the sketch’s complexity while maintaining consistent sketch retrieval results and accurate single-point positioning. The results of our experiment are as follows: the average stroke count for stick figure drawings was kept under eight strokes, and three fundamental drawing principles were derived.

Three representative types of redundant strokes are illustrated in Figure 7. In each subfigure, the first column shows the original building image, the second column presents the corresponding sketch containing redundant strokes, and the third column displays the simplified sketch after applying the drawing principles. As shown in Figure 7a, both the inner and outer contour lines that run in parallel are simplified into a single outline. The contrast is particularly visible on the right, where redundant contour features at the top and interior of the structure have been removed. In Figure 7b, only one instance of repeated internal features is preserved. Specifically, among multiple ‘M’-shaped structures, a single representative is retained to avoid redundancy. Finally, Figure 7c demonstrates the removal of internal dividing lines. Overlapping architectural features are simplified to retain only the outer outline, and extension lines on the right side are eliminated to achieve a cleaner representation.

Through the guidance of Grad-CAM heatmaps and systematic stroke simplification experiments, we established a quantitative sketching principle that identifies and preserves key structural strokes essential for retrieval accuracy. As shown in

Table 1. Accuracy degradation after removing different types of key strokes.

Removed Stroke Type	mAP@all	mAP@3
Full sketch (baseline)	0.90	0.91
Parallel inner/outer outlines	0.87	0.88
Multiple identical features	0.85	0.86
Internal dividing lines	0.84	0.84

, we progressively removed different stroke types and evaluated the resulting performance. When non-critical strokes were removed, the retrieval accuracy remained above the robustness threshold of 0.90, indicating that these strokes could be safely simplified without harming localization. In contrast, removing key strokes—such as parallel inner/outer contours, repeated architectural features, or internal dividing lines—caused accuracy to drop below 0.90 (e.g., mAP@all decreased to 0.84–0.87), confirming their importance for preserving discriminative structures. This quantitative analysis validates that the model effectively leverages critical line information for matching and remains robust to variations in drawing style, as long as key strokes are preserved. Furthermore, we confirmed the generalizability of these sketching principles by evaluating them on three additional architectural categories beyond the SJTU dataset.

3.4. Experiments on Universality of Drawing Principles and Network Structures

To verify the universality of the model and the sketching principles mentioned above, four types of architectural scenes were selected with distinct styles for testing the model’s adaptability. As shown in Figure 8, these scenes include SJTU, PARIS, OXBUILD, and OXFORD, each representing different architectural characteristics. For instance, PARIS images are characterized by simplicity, featuring mostly straight lines or simple curves with minimal internal texture details. In contrast, OXFORD buildings exhibit more complex line information, with their outlines primarily composed of curves and intricate internal textures, including repetitive patterns. Matching and positioning experiments were conducted on these architectural scenes with varying background complexity and line features.

We evaluated the model’s performance using single-point matching accuracy and mAP (mean Average Precision) as the assessment criteria to confirm the model’s universality. Furthermore, three drawing principles were employed as a basis to create architectural sketches with fewer strokes for these scenes. In order to ensure that the model exhibits both navigation and positioning accuracy, as well as fault tolerance, two images with the smallest Euclidean distance between the generated sketch and the original image from the architectural database are selected. These selected images are then regarded as the final matching objects. Moreover, three different sketches for a building are drawn. Successful matching is determined by the identification of a correct corresponding building between the two sketches with the smallest Euclidean distance, and the matched building’s name is labeled accordingly.

In

Table 2. Comparison of sketch retrieval performance across different architectural styles, where S, P, F, and B represent SJTU, PARIS, OXFORD, and OXBUILD scenes, respectively. A check mark (✓) indicates a correct retrieval, and a cross mark (✗) indicates a failed retrieval.

Place	Scenario									mAP@2
SJTU	$S_{\mathit{field}}$	$S_{\mathit{jiang}}$	$S_{\mathit{gym}}$	$S_{\mathit{dining}}$	$S_{\mathit{east}}$	$S_{\mathit{library}}$	$S_{\mathit{statue}}$	$S_{\mathit{gate}}$	$S_{\mathit{middle}}$	0.90
	✓	✓	✓	✓	✓	✗	✓	✓	✓
PARIS	$P_{\mathit{square}}$	$P_{\mathit{tower}}$	$P_{\mathit{triangle}}$	$P_{\mathit{windmill}}$	$P_{\mathit{clock}}$	$P_{\mathit{palace}}$	$P_{\mathit{eiffel}}$	$P_{\mathit{dome}}$	$P_{\mathit{gate}}$	1.00
	✓	✓	✓	✓	✓	✓	✓	✓	✓
OXFORD	$F_{\mathit{tower}}$	$F_{\mathit{dome}}$	$F_{\mathit{square}}$	$F_{\mathit{bridge}}$	$F_{\mathit{steep}}$	$F_{\mathit{pillar}}$	$F_{\mathit{palace}}$	$F_{\mathit{mindwill}}$	$F_{\mathit{glass}}$	0.88
	✓	✗	✓	✓	✓	✓	✓	✓	✓
OXBUILD	$B_{\mathit{palace}}$	$B_{\mathit{column}}$	$B_{\mathit{highrise}}$	$B_{\mathit{tower}}$	$B_{\mathit{bridge}}$	$B_{\mathit{dome}}$	$B_{\mathit{church}}$	$B_{\mathit{build}}$	$B_{\mathit{railway}}$	0.87
	✓	✓	✓	✗	✓	✓	✓	✓	✓

, the actual matching results for nine buildings from four different categories are presented. The experimental findings indicate that, in each building category, there is at most one mismatch, indicating the versatility of the network structure across various architectural styles. Furthermore, the best performance is observed in the PARIS dataset, which can be attributed to several factors. On the one hand, most of the buildings in the PARIS dataset are large, and they have fewer redundant background elements, thereby reducing interference. On the other hand, the overall building outlines in this dataset are composed of simple lines, making them easier to sketch. In contrast, the OXBUILD dataset contains several challenging cases, where different buildings exhibit similar tower-and-arch structures. As illustrated in Figure 9, simplified sketches drawn by participants may correspond to multiple buildings with similar outlines, leading to ambiguous retrieval results. These “bad cases” highlight a key limitation of purely outline-based representations and explain the few mismatches observed in the experiments.

Additionally, in order to assess the overall accuracy of the model and ensure the effectiveness of positioning, the corresponding mAP@2 values were calculated. For three types of buildings with relatively complex backgrounds or containing intricate interiors, the sketching principles established in previous experiments were applied, and the sketches were simplified. The mAP@2 values for these three building categories are 0.90, 0.88, and 0.87, respectively. The above experiments demonstrate that the model maintains high positioning accuracy across various architectural styles. Additionally, the applicability of the sketching rules has been validated, indicating their robustness across different environments. In terms of efficiency, the end-to-end retrieval for a single query takes approximately 16 ms on an NVIDIA RTX 4080 GPU, which meets real-time requirements.

3.5. Comparative Experiment and L-A Block Ablation Experiment

To further validate the network’s localization accuracy capability, three different types of retrieval networks were selected for comparison. The first category comprises traditional feature extraction operators, which extract features across different domains. The second category consists of neural network feature extraction operators, but the matching still occurs across different domains. The third category involves domain-transformed neural network algorithms that fuse features from different domains.

From

Table 3. SJTU-style comparison results. The bold values indicate the best performance in each column.

Method	mAP@1	mAP@2	mAP@3
DAISY [31]	0.61	0.59	0.57
HOG [32]	0.74	0.72	0.68
RESNET [33]	0.82	0.85	0.79
VGG [34]	0.85	0.83	0.80
SEM-PCYC [35]	0.85	0.83	0.81
ZSE [36]	0.87	0.84	0.82
Ours	0.90	0.90	0.91

, we can observe that for the building localization task, the neural network operators outperformed the traditional operators in terms of feature extraction, maintaining a certain advantage in retrieval time, and the VGG module achieved the highest precision, making it a suitable image feature extraction operator. Meanwhile, the SOTA models in the sketch retrieval task also achieved high precision on the building localization task, indicating that domain fusion can enhance the matching precision in tasks that utilize sketches for localization. However, it was noted that the failure cases typically involved similar types of buildings. The line-attention matching model proposed in this paper specifically addresses this problem, achieving the highest precision in building localization and maintaining stable performance as the retrieval quantity increases, thereby demonstrating strong robustness, as analyzed in the following section.

For evaluating the effect of the line attention network on the overall network, a comparison was made between the original network and a simplified network that excluded the attention network. Moreover, in adverse shooting conditions, like low light, the building’s outline tends to blur, posing challenges for accurate matching. We aspire to conduct a more in-depth examination of the line attention network’s performance under such circumstances. Therefore, a dataset of buildings during the night was introduced, as shown in Figure 10, which is characterized by the complex background weakening the texture information in the buildings.

In comparing the experimental results of day and night data under the simplified network, it can be observed that when nighttime images are used as the test set, both retrieval accuracy and single-point localization performance degrade significantly. As shown in

Table 4. Ablation experiments across times. Drop is the relative decrease compared to the attention-enabled network.

Time	Network	mAP@1	mAP@2	mAP@3	Drop@1%	Drop@2%	Drop@3%
day	attention	0.90	0.90	0.91	–	–	–
	w/o attention	0.85	0.82	0.88	5.6%	8.9%	3.3%
night	attention	0.74	0.74	0.75	–	–	–
	w/o attention	0.62	0.64	0.65	16.2%	13.5%	13.3%

, MAP@3 decreases from 0.88 to 0.65, MAP@2 decreases from 0.82 to 0.64, and the single-point localization accuracy drops from 0.85 to 0.62. This indicates that the difficulty of feature extraction is markedly increased when dealing with nighttime images. Furthermore, the ablation results reveal that removing the attention block leads to relative performance drops of 5.6%, 8.9%, and 3.3% for mAP@1, mAP@2, and mAP@3 under daytime conditions, and more substantial drops of 16.2%, 13.5%, and 13.3% under nighttime conditions. These findings highlight that the attention mechanism plays a critical role in enhancing robustness, particularly in low-light scenarios.

However, when observing the performance of the network with attention mechanism, its overall performance is greatly improved compared with the simplified network, especially when facing scenes with complex backgrounds or unprominent architectural textures. It improves the accuracy of MAP@3 by about 3% over the simplified network’s MAP@3 during the day, and it improves the accuracy by about 10% during the night. In addition, we also visualize the vector distance of the final match, where a smaller vector distance between the sketch and the image indicates a better match. To make the contrast more obvious, we apply post-processing in the 0–1 range, where yellow tends to 0 and purple tends to 1. As shown in Figure 11, the yellow squares are concentrated and located on the diagonal in the network with attention module, which also means that the corresponding matching is better completed. In networks without attention modules, the yellow squares are slightly scattered, resulting in poor results. Therefore, the network with attention mechanism has a strong ability to extract line information features, which ensures that the network still has good performance when the real picture highlights line information with difficulty.

4. Conclusions

When seeking effective localization methods under satellite failure conditions, traditional visual localization approaches face limitations in both accuracy and robustness, especially in complex environments. To address these challenges, we introduce the Sketch Line Information Consistency Generation model, which incorporates sketch information to reduce environmental influences and enhance robustness. Notably, sketches are created by students with no drawing experience to ensure they possess natural features recognizable by people’s subconscious. The model utilizes a line-based generative adversarial network to align sketches with original images and improves localization accuracy through attention mechanisms and correlation modules. The versatility and effectiveness of our model are validated through a series of experiments, including tests on overall building structures, stroke counting, and line attention modules. Quantitative evaluations further confirm its advantages, the proposed method achieves a top-1 retrieval accuracy (mAP@1) of 0.90, and maintains high precision (mAP@3 of 0.91) across different architectural styles. In addition, the end-to-end retrieval latency remains within 16 ms on an NVIDIA RTX 4080 GPU, satisfying real-time localization requirements. The experiments demonstrate that the model can be effectively applied to semantic localization tasks across multiple styles and environments, and the established sketching principles ensure strong robustness during localization.