Enhancing Keyword Spotting via NLP-Based Re-Ranking: Leveraging Semantic Relevance Feedback in the Handwritten Domain

Abstract

Handwritten Keyword Spotting (KWS) remains a challenging task, particularly in segmentation-free scenarios where word images must be retrieved and ranked based on their similarity to a query without relying on prior page-level segmentation. Traditional KWS methods primarily focus on visual similarity, often overlooking the underlying semantic relationships between words. In this work, we propose a novel NLP-driven re-ranking approach that refines the initial ranked lists produced by state-of-the-art KWS models. By leveraging semantic embeddings from pre-trained BERT-like Large Language Models (LLMs, e.g., RoBERTa, MPNet, and MiniLM), we introduce a relevance feedback mechanism that improves both verbatim and semantic keyword spotting. Our framework operates in two stages: (1) projecting retrieved word image transcriptions into a semantic space via LLMs and (2) re-ranking the retrieval list using a weighted combination of semantic and exact relevance scores based on pairwise similarities with the query. We evaluate our approach on the widely used George Washington (GW) and IAM collections using two cutting-edge segmentation-free KWS models, which are further integrated into our proposed pipeline. Our results show consistent gains in Mean Average Precision (mAP), with improvements of up to $2.3\%$ (from $94.3\%$ to $96.6\%$) on GW and $3\%$ (from $79.15\%$ to $82.12\%$) on IAM. Even when mAP gains are smaller, qualitative improvements emerge: semantically relevant but inexact matches are retrieved more frequently without compromising exact match recall. We further examine the effect of fine-tuning transformer-based OCR (TrOCR) models on historical GW data to align textual and visual features more effectively. Overall, our findings suggest that semantic feedback can enhance retrieval effectiveness in KWS pipelines, paving the way for lightweight hybrid vision-language approaches in handwritten document analysis.

1. Introduction

In recent years, the demand for efficient information retrieval from large-scale historical document collections has spurred significant interest in recognition-free document indexing. Traditional Optical Character Recognition (OCR) systems often fall short when faced with degraded, cursive, or stylistically inconsistent handwriting found in archival material. To overcome these limitations, Keyword Spotting (KWS) [1] has emerged over the past two decades as a powerful alternative. It enables the retrieval of instances of a given word from document images without requiring full transcription. KWS systems bypass full recognition either by comparing image representations of words with a reference query word image representation (Query-by-Example, QbE) or by textual embeddings, typically expressed as latent representations of visual features among the query string and its potential instances (Query-by-String, QbS).

Among various categorizations of the KWS paradigm, we can distinguish two primary directions. Segmentation-based approaches assume document page segmentation at the line or word image level, prior to feature extraction and representation. Instead, segmentation-free approaches operate directly on full document pages. While segmentation-based techniques [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23] benefit from a more controlled setting, they rely heavily on accurate preprocessing and annotations, limiting their scalability. On the contrary, segmentation-free methods [24,25,26,27,28,29,30,31,32,33] offer a more robust and scalable alternative, especially in zero-shot or low-supervision regimes.

Despite their advancements, current KWS systems have approached a saturation point in leveraging the visual domain alone. Deep neural architectures have significantly improved word image representation. However, they often struggle to capture semantically related words that are visually dissimilar within the document space. This motivates the integration of Natural Language Processing (NLP) techniques as a complementary semantic layer on top of the visual representations.

1.1. Motivation

The central motivation of this work lies in the observation that many top-ranked instances retrieved by state-of-the-art KWS models are either visually misleading or semantically weak matches. As such, there is an increasing need to amplify KWS performance by embedding language-aware mechanisms that can exploit semantic proximity between the query and the retrieved results. Recent advancements in language models, such as Word2Vec [34], FastText [35], and BERT [36], allow for rich contextual representations that encode semantic similarity beyond surface-level matching.

In this work, we introduce an unsupervised semantic relevance feedback mechanism that operates in the latent space of word meanings. By projecting the transcription of each retrieved instance into a high-dimensional semantic space, we evaluate its relevance with respect to the query and re-rank the list accordingly. This re-ranking is driven by a hybrid similarity score that incorporates both the original visual similarity (verbatim) and the semantic proximity in the language domain using their weighted sum (Equation (6)). Our aim is to improve the Mean Average Precision (mAP) of string-based queries without requiring manual annotation or supervised fine-tuning.

1.2. Contribution

The key contributions of this work can be summarized as follows.

• We present a novel framework for semantic-aware re-ranking in handwritten document image KWS by integrating cutting-edge NLP techniques into the retrieval pipeline. To the best of our knowledge, this is the first attempt to incorporate transformer-based Large Language Models (LLMs) for uncovering semantic relationships with top-ranked query instances, enabling post-retrieval re-ranking that enhances overall KWS performance.
• We explore two distinct decoding strategies for transcribing word image instances from the initial ranked list: (i) TrOCR [37], a transformer-based vision-to-text model that maps visual input directly to character sequences, and (ii) the character counting and Connectionist Temporal Classification (CTC)-based re-scoring mechanism introduced in [33], a compact Convolutional Neural Network (CNN)-based segmentation-free approach that scores query matches based on dense character probability maps and sequence alignment. These transcriptions are subsequently embedded using state-of-the-art language models (e.g., RoBERTa [38]), projecting each word into a semantically meaningful vector space for downstream re-ranking.
• We propose a new ranking scheme where each word instance is assigned a composite score: a weighted sum of its verbatim (visual-based) similarity and its semantic similarity to the query. This allows semantically relevant words, possibly missed in the initial visual ranking, to be spotted higher in the final ordering.
• We perform extensive ablation studies on the George Washington (GW) and IAM datasets using two cutting-edge reference KWS systems that perform directly at page level, namely, KWS-Simplified [33] and WordRetrievalNet [29], evaluating improvements in mAP across different embedding strategies and similarity metrics.
• Finally, our results illustrate that NLP-based re-ranking not only improves standard KWS performance but also opens the path toward semantically aware information retrieval systems. These systems bridge the gap between visually dissimilar yet semantically related word instances while maintaining a plug-and-play design that is both dataset-agnostic and independent of baseline model retraining. This generalization capability is further supported by the low variability in re-ranking effectiveness across cross-validation folds. Throughout our experiments, we observe consistently improved mAP and low standard deviation. Taken together, our approach pushes the frontier of recognition-free document understanding by integrating semantic reasoning into post-retrieval analysis.

Our work is organized as follows: Section 2 reviews the related works with regard to document-level keyword spotting, focusing on segmentation-free approaches where processing is performed directly on the page image without requiring explicit word or line segmentation. In addition, we cover related attempts within the relevance feedback domain and re-ranking schemes that have inspired our proposed methodology. Section 3 details the baseline KWS reference systems, their integrated architecture within the NLP-based techniques used to extract semantic similarities from the initial exact keyword matches that can improve the overall performance. We also introduce a novel re-ranking scheme that balances verbatim and semantic KWS in a unified framework. Section 4 presents extensive ablation studies validating the impact of semantic relevance on retrieval performance, followed by a deeper discussion and interpretation of our findings, which confirm our initial intuition in this respect. Finally, Section 5 summarizes our contributions and outlines potential directions for future research.

2. Related Works

2.1. Segmentation-Free Keyword Spotting: Key Approaches and Trends

One of the major issues of the preprocessing stage is that possible segmentation errors are regularly conveyed in the spotting phase. In particular, accurate word segmentations are difficult to obtain in handwritten and degraded documents. For this reason, several segmentation-free word spotting techniques have emerged.

Early segmentation-free KWS methods addressed the problem of avoiding explicit word or line segmentation by analyzing entire document pages directly. This direction was initially dominated by hand-crafted feature extraction and region-of-interest proposals. Leydier et al. [39] and Zhang and Tan [40] used local keypoints and gradient-based descriptors or Heat Kernel Signatures (HKSs), matched through elastic or manifold-based similarity metrics. However, these techniques incurred high computational costs and did not scale well.

A more scalable direction emerged with patch-based sliding-window frameworks [24,41,42,43], where descriptors like Scale-Invariant Feature Transform (SIFT), Histogram of Gradients (HoG), or pixel densities are extracted over image regions. In this line, Rusiñol et al. [24] enhanced retrieval effectiveness via a Latent Semantic Indexing (LSI) projection, while Rothacker et al. [42] used a Bag-of-Features Hidden Markov Model (BoF-HMM) formulation for robust query modeling.

Graph- and component-based techniques [26,44,45] modeled spatial or structural properties of documents using connected components (CCs) or grapheme graphs, typically matched using graph edit distances or geometric constraints. Kovalchuk et al. [26], winner of the segmentation-free track in the ICFHR 2014 KWS competition, proposed a fast and simple CC-based matching scheme. Other approaches such as [44,45] explored contextual or geometric graph structures, often leveraging binarized document representations. While segmentation-free in design, these methods were sensitive to noise and relied on handcrafted heuristics for constructing word hypotheses.

These early systems, while pioneering, were progressively replaced by deep learning-based methods, which enabled the end-to-end learning of discriminative features and more efficient document-wide inference. Early deep approaches often relied on region proposal mechanisms to locate candidate word regions, but more recent work has shifted toward fully dense or proposal-free retrieval architectures.

Ghosh and Valveny [46] combined region-proposal CNNs with attribute-based deep embeddings (PHOCNet representations [3]) to aggregate features across word-like regions. Wilkinson et al. [28] introduced Ctrl-F-Net, an end-to-end architecture that employs a ResNet34 backbone, Region Proposal Networks (RPNs), and word string embeddings such as the Pyramidal Histogram of Characters (PHOC) [2] and Discrete Cosine Transform of Words (DCToW) [47], to enable robust QbS retrieval. Rothacker et al. [48] further enhanced region detection under uncertainty by incorporating extremal region proposals and class activation maps into the word spotting pipeline.

At the multi-task and multi-scale learning frontier, Zhao et al. [29] integrated a Feature Pyramid Network (FPN) into a CNN architecture, jointly training for pixel classification, bounding box regression, and visual-to-textual embedding learning (via DCToW). These contributions have advanced the segmentation-free paradigm toward dense, discriminative, and scalable retrieval pipelines. Complementary efforts by Wilkinson and Nettelblad [30] introduced a weakly supervised bootstrapping strategy using HMM-based alignment, while Retsinas et al. [33] proposed an efficient proposal-free approach based on character counting and CTC-based re-scoring.

While most prior works in this area focus on visual representations and similarity, recent trends point toward bridging the gap between visual and semantic domains using embeddings that encode language-aware properties [49]. However, such semantic alignment has remained under-explored in segmentation-free KWS settings. This motivates our work, which aims to enhance KWS effectiveness by introducing relevance-aware re-ranking based on language models and semantic embeddings of word image transcriptions.

In what follows, we review relevant literature in retrieval enhancement, particularly focusing on relevance feedback and re-ranking schemes that aim to improve the final ranked list beyond raw visual similarity. This includes both supervised and unsupervised paradigms and sets the foundation for our proposed embedding-based re-ranking method that leverages NLP-driven semantic proximity in the re-ranking process.

2.2. Retrieval Enhancement via Relevance Feedback and Re-Ranking

A prominent challenge in KWS systems is the presence of false positives within the top-ranked retrieval results. To address this, several techniques for refining the ranked list have been proposed, commonly grouped under the umbrella of relevance feedback. These can be broadly classified into supervised and unsupervised methods.

2.2.1. Supervised Relevance Feedback

In supervised relevance feedback (SRF), the user labels a small subset of the retrieval results as relevant or irrelevant, enabling the system to reformulate the query representation or re-rank the results accordingly. Rocchio-style vector updates [50] have been adapted for KWS by Rusiñol et al. [51], combining query refinement with score re-weighting. More recently, Wolf et al. [52] introduced a CNN-based confidence estimation pipeline, employing dropout-based uncertainty, surrogate meta-classifiers, and sigmoid-activated trust scores to identify and suppress false positives in segmentation-free word spotting.

Despite the rapid evolution of relevance feedback techniques in vision-language retrieval, such as CLIP-based interactive search [53], approaches of this type have not yet been adopted by the document analysis and recognition community. In fact, to the best of our knowledge, no new supervised feedback strategies have been introduced in segmentation-free KWS settings since 2020. This highlights a critical research gap and further motivates our embedding-based re-ranking proposal.

2.2.2. Unsupervised Feedback and Re-Ranking

The obvious benefits of supervised relevance feedback lie in facilitating user judgments to guide ranking refinement. However, this process can be costly and subjective, particularly in degraded or historical documents. This limitation has motivated the adoption of unsupervised alternatives, such as pseudo-relevance feedback (PRF). Almazán et al. [43] introduces a two-stage ranking, combining fast approximate retrieval with Fisher vector-based re-ranking and iterative query expansion. Similar approaches have been employed by Ghosh and Valveny [54] and Shekhar and Jawahar [55], who incorporate spatial pyramid refinements. Vats and Fornés [56] propose a local query expansion strategy based on confidence thresholds and keypoint matching, repeated across document pages to enhance robustness.

While these methods have advanced re-ranking pipelines in segmentation-free KWS, they typically operate on low-level visual similarity. Hence, they fail to capture deeper semantic relations, which are particularly important when dealing with ambiguous visual forms or out-of-vocabulary (OOV) terms. This motivates a shift toward semantic reasoning via language-aware models.

Recent advancements in general information retrieval (IR), including query expansion via neural embeddings, transformer-based re-ranking, and interactive CLIP-style search, have demonstrated substantial gains in retrieval quality. However, such methods often target textual corpora, document-level retrieval, or long natural language queries. Therefore, they are not directly applicable to the constrained setting of segmentation-free word image retrieval. Notable examples include re-ranking in Question Answering (QA) pipelines [57], semantic query expansion with conditional Variational Autoencoders (VAEs) [58], and surveys of deep Query Expansion (QE) techniques [59], all of which focus on textual or structured data rather than single-word visual spotting tasks.

For this reason, and to maintain conceptual coherence, we restrict our focus to visual-domain methods that have been explicitly applied within the keyword spotting literature. We next propose an alternative, embedding-based re-ranking scheme that introduces language-informed semantic priors into the unsupervised refinement process.

2.3. Semantic Knowledge Transfer

In a traditional KWS system, document regions are represented using descriptors such as PHOC or DCToW and matched against a query based solely on visual appearance and character-level similarity. However, words are more than sequences of characters, since they also carry semantic meaning. Even humans might struggle in interpreting handwritten text, and as such, often rely on contextual understanding of the broader context to disambiguate visually similar words. Figure 1 showcases an example of this peculiarity where a human transcriber has to interpret letters forming a valid word (i.e., ten), while that word should fit logically within the surrounding sentence and passage. This intuition motivates the integration of semantic reasoning into word spotting systems.

Semantic KWS was first introduced by Wilkinson et al. [47] as a method to enrich document image search with language-level knowledge. Their approach retrieved exact matches and expanded them using semantically related terms (e.g., synonyms, inflected forms, or categorical relationships). This form of semantic expansion (expressed as inexact or partial matching) has proved particularly useful for dealing with hyphenation, word splits across line breaks, or approximate matches.

Semantic retrieval methods can be broadly divided into ontology-based and context-based techniques [60]. Ontology-based approaches [61,62] use resources such as WordNet [63] to identify categorical or lexical similarities.

On the contrary, context-based approaches are grounded in the distributional hypothesis [64], which posits that words appearing in similar contexts tend to share similar meanings. Recent advancements in NLP have produced a wide range of text embedding techniques, including Word2Vec, GloVe, FastText, and transformer-based models like BERT, which generate dense vector representations that encode semantic similarity beyond surface-level string matching. These embeddings have opened new opportunities for mapping visual content, such as handwritten words, into semantically meaningful spaces.

To bring this semantic capability into keyword spotting, two general strategies have emerged: (1) learning visual-to-semantic mappings directly in an end-to-end, recognition-free manner, and (2) transcribing word images first, followed by text-based embedding to obtain semantic representations.

End-to-end approaches are particularly common in segmentation-based settings, where they circumvent explicit recognition; a process known to yield irrecoverable errors when mapped directly to embedding spaces. This paradigm was introduced by Wilkinson et al. [47], who trained a two-stage CNN with cosine embedding loss to project word images into a pre-trained semantic space. Subsequent work by Krishnan et al. [65] extended this approach using the HWNet architecture, recently evolved into a joint embedding framework (HWNet v3) for recognition and retrieval [19] to jointly learn syntactic (PHOC or DCToW) and semantic (e.g., FastText) representations. Tüselmann et al. [49] further evaluated the impact of different embeddings—including FastText, GloVe, and BERT—using the same architecture and explored combinations thereof for document-level semantic understanding.

These learned embeddings have enabled downstream tasks such as Named Entity Recognition (NER), Visual Question Answering (VQA), and Named Entity Linking (NEL) directly on image documents. However, direct embedding methods remain limited in practice: they require large amounts of annotated training data, and no publicly available pre-trained models currently exist for generic semantic KWS. For example, Wilkinson et al. [47] suggest training on all 40 volumes of The Writings of George Washington from the Original Manuscript Sources, 1745–1799 [66] to adequately capture corpus-level semantics. Other works [19,49] rely on large-scale synthetic datasets of handwritten word images rendered from frequent English words.

Despite these efforts, recent analysis [60] reveals that visual–semantic embeddings often retain mostly syntactic characteristics. This suggests a gap between visual and semantic domains and highlights the underutilized potential of modern pre-trained language models in keyword spotting. To address this, we propose an alternative path: leveraging pre-trained transformer-based language models (RoBERTa, MPNet [67], and MiniLM [68]) to inject semantic reasoning into the KWS pipeline through post-hoc re-ranking. By embedding transcriptions of retrieved word images and comparing them to the query in a semantic space, we enable relevance-aware refinement that complements visual similarity. This lightweight, flexible mechanism enhances both exact and semantic KWS, bridging the gap between vision and language without requiring end-to-end retraining.

3. Materials and Methods

In this section, we introduce a post-processing, semantically-aware relevance feedback mechanism aimed at enhancing the retrieval performance of KWS systems. We begin by describing two state-of-the-art KWS systems in Section 3.1 and Section 3.2, each embodying a distinct approach to traditional segmentation-free KWS, with their ranked lists serving as the basis for our re-ranking. To conclude, we present our proposed framework in Section 3.3.

An overview of the proposed architecture is illustrated in Figure 2. Given a query, the KWS system retrieves the top-k most visually similar image regions from the document collection. Each retrieved word image is decoded, and an LLM projects both its transcription and the query in a shared subspace to compute their semantic similarity. The final ranking is determined by combining both similarity measures.

Additionally, the re-ranking capability of the proposed pipeline extends beyond conventional KWS objectives. As shown in Figure 2, visually similar but irrelevant terms (e.g., “letter”, “understand”, “better”) are replaced with semantically relevant military terms (e.g.,“sergeant”, “regiment”). Although this could yield more meaningful retrieval results for users, such qualitative improvements are not reflected in verbatim KWS metrics.

3.1. WordRetrievalNet

The WordRetrievalNet is a state-of-the-art segmentation-free QbS KWS system introduced in [29]. It operates in two stages:

• Offline stage: A deep neural network (DNN) is trained to generate a database of candidate bounding boxes along with their representations in a latent space.
• Online stage: A query is matched against the database, returning a ranked list of bounding boxes with the highest cosine similarity to the query representation.

Its end-to-end design avoids the complex pre- and post-processing steps required by traditional KWS methods. Additionally, the FPN-based architecture [69,70] extracts multi-scale features directly from document images, which are processed by three prediction heads:

• A classification head that identifies pixels belonging to a positive word area;
• A regression head that predicts the offsets between each pixel of a positive word area and the corresponding bounding box containing it;
• An embedding head that maps word areas into the latent space (DCToW, PHOC, etc.).

The network is trained in a supervised manner, minimizing the loss function:

$${\mathcal{L}}_{all}={\mathcal{L}}_{cls}+{\mathcal{L}}_{bbox}+{\mathcal{L}}_{embed}.$$

(1)

For the classification task, a loss based on the Dice similarity coefficient [70,71] is used:

$${\mathcal{L}}_{cls}=1-{\displaystyle \frac{{\displaystyle 2\sum _{i,j}{\widehat{y}}_{cls}^{i,j}·y_{cls}^{i,j}}}{{\displaystyle \sum _{i,j}{\left({\widehat{y}}_{cls}^{i,j}\right)}^2+\sum _{i,j}{\left(y_{cls}^{i,j}\right)}^2}}},$$

(2)

where ${\widehat{y}}_{cls}^{i,j}$, $y_{cls}^{i,j}$ denote the values of pixel $(i,j)$ in the word classification prediction ${\widehat{y}}_{cls}$ and the ground truth $y_{cls}$, respectively. This loss function counteracts the bias introduced by the imbalance between word pixels and background pixels.

For the bounding box regression, a modified Intersection over Union (IoU) loss, called distance–IoU loss ($DIoU$) [72], is employed, as expressed by Equation (3):

$${\displaystyle {\mathcal{L}}_{bbox}={\displaystyle \frac{1}{\left|C\right|}}\sum _{i\in C}DIoU\left({\widehat{y}}_{bbox},y_{bbox}\right),}$$

(3)

where C represents the collection of positive elements in the word pixel classification, ${\widehat{y}}_{bbox}$ is the predicted bounding box, and $y_{bbox}$ is the ground truth box.

For the word embedding, the cosine loss is used ${\mathcal{L}}_{embed}=1-cos\left({\widehat{y}}_{embed},y_{embed}\right)$, which penalizes the dissimilarity between the representations ${\widehat{y}}_{embed}$ and $y_{embed}$.

During inference, the set of positive word pixels and their offsets are combined to construct bounding boxes for the candidate word areas. A Non-Maximum Suppression (NMS) filter is applied to reduce the density of predictions. The embedding of each bounding box is computed as the mean embedding of the pixels it contains. Figure 3 summarizes the above-mentioned key model components.

3.2. Segmentation-Free KWS Simplified

Retsinas et al. [33] introduce a segmentation-free QbS KWS system (KWS-Simplified) that formulates KWS as a character counting problem. Their goal is to identify image regions containing the same character histogram as the query. Unlike WordRetrievalNet, the lightweight system architecture eliminates synthetic training data requirements. However, to refine the initial predictions and compute their similarity with the query, several post-processing steps are applied. These steps include (1) Pyramidal Counting, where a descriptor similar to PHOC is constructed and compared against the query descriptor, and (2) CTC-based re-scoring via force alignment. This alignment approach is used to improve the bounding box estimation and refine the ranking of candidates, in line with recent developments in CTC alignment and scoring [73]. NMS is further applied to reduce overlapping predictions.

The system consists of a ResNet [74] backbone with two prediction heads: (1) a decoder head that estimates the probability distribution of each character occurrence across a document image and (2) a scaler head that predicts the character scale at each image location. Figure 4 depicts this architecture.

For a given character c, let $F(i,j,c)$ denote the feature probability distribution output by the decoder head at image coordinates $(i,j)$ and let $S(i,j)$ represent the predicted scale factor at the same coordinates. The number of occurrences of character c within a bounding box spanning from $(s1,s2)$ to $(e1,e2)$ is given by Equation (4):

$$y_c=\sum _{i=s_1}^{e_1}\sum _{j=s_2}^{e_2}F(i,j,c)·S(i,j)$$

(4)

The counting loss is defined as ${\mathcal{L}}_{count}={\parallel y_c-t_c\parallel }_2$, where $t_c$ is the target count histogram.

During the training of the network, the loss function employed combines the counting loss with the CTC loss [75]. Following the approach of Retsinas et al. [33], a weighting factor of 10 is applied to the counting loss to balance its contribution with the CTC objective:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{CTC}}+10·{\mathcal{L}}_{\mathrm{count}}.$$

(5)

Herein, the feature map produced by the decoder head undergoes column-wise max-pooling before being fed into the CTC loss. Figure 5 overviews these post-processing scoring steps.

3.3. Proposed Framework

We conclude this section by presenting our proposed framework: an unsupervised relevance feedback pipeline enabled by the incorporation of semantic information during the re-ranking process. Its goal is to enhance retrieval performance through the suppression of false positive results that appear high in the initial ranking, while simultaneously promoting instances whose original rank underestimates their actual relevance.

The proposed framework operates through three stages. Initially, a verbatim retrieval step takes place; a KWS system generates a ranked list of candidate image regions, ordered by their visual resemblance to a given query. Then, a decoder transcribes each image region (retrieved query instance) from the list. Subsequently, a semantically aware LLM [38] embeds these transcriptions into a semantic space, where spatial proximity reflects semantic relatedness. The final ranking occurs, wherein candidates are reordered based on their combined verbatim and semantic similarity.

We examine multiple variations of the framework:

• Two distinct decoder architectures;
• Three alternative state-of-the-art semantic LLMs;
• Two late fusion strategies.

The first decoder is adapted from the KWS-Simplified network. Notably, its character probability prediction head can operate as an independent module that generates transcriptions when a softmax function is applied to its output. For the second decoder, we utilize TrOCR, a state-of-the-art text recognition model. It integrates a Vision Transformer (ViT) encoder [76] with a transformer-based text generator (typically initialized using either RoBERTa or MiniLM). The combined model has been pre-trained on large-scale synthetic textual data and can be fine-tuned on both machine-printed and handwritten document collections.

For the generation of semantic embeddings, we use three state-of-the-art BERT-like LLMs derived from RoBERTa, MPNet, and MiniLM, respectively. Each of these models has been specifically adapted for the task of semantic search through supervised contrastive learning [38]. This paradigm trains models to differentiate between semantically related sentence pairs and randomly sampled negatives. In order to handle OOV terms, while maintaining fixed-dimensional embeddings, the WordPiece [36] subword tokenization technique is used. The final embedding is obtained by mean pooling all token embeddings. Semantic similarity is quantified as the cosine similarity between the generated embedding vectors. Mean pooling is chosen for its increased robustness against transcription errors in individual subtokens.

Finally, we evaluate two strategies that fuse visual and semantic relevance: weighted combination of similarities and semantic pruning. In the weighted combination strategy, we compute a weighted average of the verbatim and semantic similarity scores using Equation (6):

$${\mathrm{sim}}_{\mathrm{combined}}=a·{\mathrm{sim}}_{\mathrm{semantic}}+(1-a)·{\mathrm{sim}}_{\mathrm{verbatim}},$$

(6)

where $a\in [0,1]$ is a hyperparameter controlling the relative importance of semantic similarity in the final score. The resulting values are used to re-rank the candidate list, incorporating both semantic and verbatim relevance.

On the other hand, semantic pruning refers to the process of filtering candidate items by discarding those with a semantic similarity below a predefined threshold. The remaining candidates are then re-ranked based on their verbatim similarity.

4. Experimental Evaluation

4.1. Datasets

We conducted our experiments on two standard benchmarks for evaluating and comparing KWS systems [1]: the George Washington (GW) dataset [77] and as well as the IAM Handwriting Database (IAM) [78].

The GW database comprises 20 handwritten letters from George Washington’s Papers at the Library of Congress [79]. These 18th-century documents, written in historical English by Washington and his aides, contain 4860 annotated words with corresponding bounding boxes. Due to the minimal variation in the writing style, it can be characterized as a single-writer dataset.

Unlike segmentation-based KWS approaches that employ the standardized partition of Almazán et al. [2] for the GW collection, no official partition exists for the evaluation of segmentation-free methods. Instead, we follow the established experimental practices used in prior works [27,28,29,33]. Given its limited size, it is customary to adopt a four-fold cross-validation scheme, where each fold consists of five pages. Thus, four experimental iterations are conducted. During each iteration, one fold is reserved as the test set, while the remaining three serve as the training set. Additionally, one page from the training set is set aside for validation. Test queries are extracted from the unique transcriptions of the test pages by removing punctuation and lowercasing, whereas stopwords are retained as queries.

Table 1. The partition of GW used in our experiments.

Fold No.	Page IDs Across Each Fold
1	274, 309, 276, 272, 303
2	306, 273, 301, 300, 278
3	270, 302, 277, 275, 308
4	307, 304, 279, 271, 305

presents the exact partition of the dataset used in our experiments (the partition was obtained by shuffling the page indices 0–19 using NumPy with seed 0).

The IAM database contains 1539 pages of modern cursive handwritten English text produced by 657 writers. The pages are segmented and annotated, comprising a total of 115,320 words. The variability introduced by the multi-writer setting is a principal factor contributing to the difficulty of the dataset. We use the official partition of the database, as is common practice in the literature [28,33]; however, unlike GW, no cross-validation is performed. The test queries comprise all unique, lowercased transcriptions from the test set, excluding words that contain non-alphanumeric characters, punctuation marks, erroneous annotations, or words appearing in the official stopword list [2,3].

4.2. Evaluation Protocol

We follow the evaluation procedure outlined in [27], which extends the standard Almazán protocol [2] for assessing the performance of segmentation-free QbS KWS methods [28,29,33].

To evaluate the ranking of retrieved document regions, we compute the Mean Average Precision (mAP, a commonly used metric in KWS evaluation. As the name suggests, mAP is calculated as the mean of the Average Precision (AP) across a set of given test queries. The AP for a given query is defined in Equation (7):

$$AP={\displaystyle \frac{1}{R}}\sum _{k=1}^{n}P@k·rel\left(k\right),$$

(7)

where $P@k$ is the precision of the top k retrieved results (i.e., the fraction of the top k retrieved instances that are relevant), $rel\left(k\right)$ is an indicator function that returns 1 if the retrieved instance at index k is relevant and 0 otherwise, and R denotes the total number of relevant instances for the query. Finally, in the segmentation-free KWS setting, a retrieved region is considered relevant if it sufficiently overlaps with an annotated region in the ground truth. Overlap is measured using the Intersection over Union (IoU) criterion, which is satisfied when it exceeds a predefined threshold. Commonly used thresholds, which we also employ in our study, are $25\%$ (mAP@25) and $50\%$ (mAP@50). If multiple retrieved regions overlap with a single ground truth region, only one is considered relevant—typically, the one with the largest overlap.

4.3. Implementation Details

Both WordRetrievalNet and KWS-Simplified offer open-source implementations, as well as pre-trained models. However, while KWS-Simplified includes training on the IAM dataset, neither system has been pre-trained on GW.

In order to enable a direct comparison between these architectures and to establish a rigorous baseline against which we evaluate the relative gains of our framework, we trained both systems on GW using the typical four-fold cross-validation scheme described in Section 4.1 and the partition shown in Table 1. For each cross-validation iteration, we trained a WordRetrievalNet model for 120 epochs and a KWS-Simplified model for 200 epochs, evaluating mAP@25 performance on the validation set every 10 epochs and retaining the best-performing model as the final baseline. The training configuration of each system (e.g., optimizer selection, hyperparameter values) followed the settings reported by Zhao et al. [29] and Retsinas et al. [33] as those yielding the highest mAP performance. The reproduced and reported mAP scores are recorded in

Table 2. Comparison of reproduced and reported mAP scores and their standard deviations for the two baseline models, WordRetrievalNet and KWS-Simplified, on the GW dataset.

Model	mAP@25	mAP@50
WordRetrievalNet (Reproduced)	94.31 ± 1.8	88.29 ± 4.0
WordRetrievalNet (Reported)	96.46	94.06
KWS-Simplified (Reproduced)	89.74 ± 0.7	72.29 ± 3.0
KWS-Simplified (Reported)	91.6	66.4

.

Similarly, WordRetrievalNet was trained on the official IAM split for 80 epochs, achieving an mAP@25 of $79.15\%$ and an mAP@50 of $72.85\%$. It is worth noting that Zhao et al. [29] do not report results for this dataset.

While our reproduced results deviate from prior works, our goal is to establish a baseline reference system, and therefore, such observed divergences are to be expected and can be attributed to several factors.

First and foremost, the original implementations used different partitions of GW. In the case of WordRetrievalNet, an additional source of variability arises from the use of randomly generated synthetic training data.

Second, we note minor implementation-specific differences in the training data preparation pipelines. Since both systems are segmentation-free, they employ a detection phase in which the model identifies image regions that are likely to contain words. These candidate regions are then refined to match the query and produce the final retrieval results. To train such a detector effectively, it is not sufficient to use only the tight bounding boxes of the ground truth word images. Instead, it is beneficial to enlarge these regions so it can learn to segment words more reliably. Overestimating the bounding box is generally preferable to underestimating it, as missing a word entirely would hinder recall.

In the training of KWS-Simplified on GW, the authors generate training images by extending the ground truth bounding boxes by a constant factor. In our implementation, we enlarged the word image areas by a factor of 2.5. We observed this larger context window helped the model learn more robust representations corresponding to the segmented word regions, which in turn led to a substantial improvement in mAP@50.

Furthermore, the implementation of WordRetrievalNet uses training data extracted as image crops that cover regions larger than the ground truth word bounding boxes, often including multiple word instances. We note here that recovering the exact parameterization that yielded the optional result in the original work was not possible. Therefore, we modified the patch cropping algorithm used for extracting positive examples during the training of WordRetrievalNet, aiming to improve the computational efficiency of the training loop. The original algorithm repeatedly sampled image patches until no word within a patch crossed its boundary or until a limit of 1000 iterations was reached. In practice, this limit was frequently exhausted, creating an artificial bottleneck that significantly slowed down training and rendered the computation CPU-bound. Our modified approach addresses this issue by retaining only the words for which at least 70% of the bounding box falls within the sampled patch. This trade-off between computational efficiency and training quality was preferable, especially within the mAP@50 metric, where the impact is more noticeable.

Finally, some variability is inevitably introduced by the stochastic nature of neural network training, such as the random weight initialization and the random sampling during optimization.

As discussed earlier in Section 3.3, we evaluate two distinct decoder architectures for transcribing retrieved image regions. Each KWS-Simplified-based decoder inherently shares weights with the corresponding KWS-Simplified backbone trained on the same partition, thereby requiring no additional training before deployment. For the TrOCR-based alternative, we initialized the model in each cross-validation iteration of GW using weights from a handwritten text recognition (HTR) model [80] pre-trained on the IAM database. Next, the model was fine-tuned for 20 epochs using the set of word images from the training set of the current iteration, while the queries from one page were used for validation. We employed the AdamW optimization algorithm [81] with an initial learning rate of $5\times {10}^{-5}$, which increased linearly during a short warm-up period and then decayed linearly for the remaining epochs. The fine-tuning reduced the average Character Error Rate (CER) on the validation sets of GW from 26.76% to 11.05% (i.e., a standard metric in OCR and HTR, which measures the number of character-level errors in a predicted transcription compared to the ground truth). Although the strong $3.42\%$ CER performance reported by TrOCR on IAM [37] suggests that there is room for improvement on GW, accurate retrieval does not necessarily require explicit transcription. We report CER solely to support reproducibility. Moreover, given this performance, we used the decoder in our experiments on IAM without further training.

To embed the decoded transcriptions into a semantic space, we employed three pre-trained models from the SentenceTransformers library [82]: (1) stsb-roberta-base [83] (the RoBERTa architecture fine-tuned on the Semantic Textual Similarity benchmark [84]); (2) all-mpnet-base-v2 [85] (the MPNet architecture fine-tuned on a corpus of diverse datasets comprising over one billion sentence pairs); and (3) all-MiniLM-L12-v2 [86] (the MiniLM architecture fine-tuned on the same diverse corpus).

Finally, both WordRetrievalNet and the three semantic embedding models utilize cosine distance to compute the verbatim and semantic similarities, respectively. In comparison, the KWS-Simplified method relies on a similarity measure derived from CTC scores. To ensure that these values are on a comparable scale when combined, we normalize the CTC-based scores to the range $[-1,1]$ using the minimum and maximum values observed in the training set.

4.4. Ablation Experiments on Fusion Strategy and Decoder Choice

This section presents ablation studies conducted to evaluate the impact of different configurations on the proposed pipeline. These experiments aim to isolate the effects of the re-ranking strategy and assess the generalization ability of our method across different initial retrieval systems.

Table 3. Mean Average Precision of the WordRetrievalNet backbone with the weighted combination strategy across semantic importance thresholds and embeddings.

Semantic Weight	Semantic LLM	George Washington Dataset				IAM Handwriting Database
		KWS-Simplified Decoder		TrOCR Decoder		KWS-Simplified Decoder		TrOCR Decoder
		mAP@25	mAP@50	mAP@25	mAP@50	mAP@25	mAP@50	mAP@25	mAP@50
0.0 *	all -MiniLM -L12 -v2	94.31 ± 1.8	88.29 ± 4.0	94.31 ± 1.8	88.29 ± 4.0	79.15	72.85	79.15	72.85
0.1		95.80 ± 1.5	89.51 ± 3.8	94.51 ± 1.8	88.43 ± 4.0	80.60	73.98	82.04	75.40
0.2		96.10 ± 1.5	89.75 ± 3.8	93.96 ± 2.0	87.84 ± 4.0	79.29	72.62	80.59	73.77
0.3		96.30 ± 1.3	89.79 ± 3.7	93.20 ± 2.1	87.07 ± 3.8	75.87	69.30	77.68	71.05
0.4		96.04 ± 1.5	89.64 ± 3.6	91.71 ± 2.5	85.73 ± 3.9	71.36	65.13	74.04	67.66
0.5		95.34 ± 1.4	89.01 ± 3.5	89.05 ± 2.9	83.19 ± 3.8	65.95	60.19	70.68	64.57
0.6		94.39 ± 1.5	88.20 ± 3.5	84.82 ± 4.2	79.41 ± 4.2	60.26	54.99	67.17	61.33
0.7		93.25 ± 1.6	87.37 ± 3.5	79.04 ± 4.8	74.03 ± 3.7	55.95	50.99	64.47	58.88
0.8		91.92 ± 1.9	86.26 ± 3.5	73.12 ± 6.1	68.42 ± 4.5	53.12	48.38	62.88	57.42
0.9		90.93 ± 2.0	85.42 ± 3.6	69.68 ± 6.6	65.20 ± 4.8	51.46	46.88	62.22	56.83
1.0		89.83 ± 2.2	84.28 ± 3.7	64.58 ± 7.5	60.07 ± 6.0	47.96	43.71	54.34	49.48
0.0 *	all -mpnet -base -v2	94.31 ± 1.8	88.29 ± 4.0	94.31 ± 1.8	88.29 ± 4.0	79.15	72.85	79.15	72.85
0.1		95.90 ± 1.4	89.67 ± 3.8	94.68 ± 1.6	88.62 ± 3.9	80.84	74.18	82.12	75.43
0.2		96.27 ± 1.3	89.86 ± 3.8	94.09 ± 1.7	87.98 ± 3.8	80.01	73.21	80.77	74.02
0.3		96.50 ± 1.4	89.97 ± 4.0	93.07 ± 2.1	87.02 ± 3.7	77.55	70.82	77.87	71.29
0.4		96.56 ± 1.5	90.08 ± 3.9	91.78 ± 2.4	85.85 ± 3.7	73.86	67.22	74.73	68.33
0.5		96.00 ± 1.3	89.62 ± 3.6	89.42 ± 2.9	83.85 ± 3.8	69.24	62.95	71.23	65.12
0.6		95.04 ± 1.2	88.82 ± 3.5	85.13 ± 3.6	79.84 ± 3.8	63.70	57.97	67.84	61.99
0.7		93.83 ± 1.6	87.86 ± 3.4	79.10 ± 5.3	74.10 ± 4.1	58.18	52.97	65.43	59.78
0.8		92.57 ± 1.7	86.71 ± 3.4	74.06 ± 6.2	69.30 ± 4.6	53.92	49.10	63.80	58.25
0.9		91.53 ± 1.8	85.84 ± 3.6	70.50 ± 6.4	65.90 ± 4.6	51.43	46.89	62.78	57.36
1.0		90.21 ± 2.1	84.45 ± 3.7	64.91 ± 7.5	60.31 ± 6.0	49.26	44.84	54.99	50.11
0.0 *	stsb -roberta -base	94.31 ± 1.8	88.29 ± 4.0	94.31 ± 1.8	88.29 ± 4.0	79.15	72.85	79.15	72.85
0.1		95.83 ± 1.6	89.60 ± 3.9	94.58 ± 2.0	88.55 ± 4.2	81.16	74.49	81.88	75.18
0.2		96.19 ± 1.3	89.82 ± 3.7	94.01 ± 2.1	87.91 ± 4.2	80.39	73.55	80.97	74.22
0.3		96.59 ± 1.3	90.17 ± 3.7	92.87 ± 2.9	86.92 ± 4.5	77.40	70.68	77.51	70.90
0.4		96.44 ± 1.6	89.99 ± 3.7	90.75 ± 3.2	84.96 ± 4.4	73.54	66.94	74.29	67.87
0.5		96.32 ± 1.6	89.90 ± 3.7	87.91 ± 3.5	82.32 ± 4.3	69.45	63.25	71.28	65.11
0.6		96.04 ± 1.7	89.57 ± 3.8	84.15 ± 4.3	78.83 ± 4.9	64.83	59.05	68.70	62.85
0.7		95.27 ± 1.7	88.91 ± 4.0	80.37 ± 4.4	75.25 ± 4.7	61.28	55.73	66.71	61.03
0.8		94.37 ± 1.8	88.08 ± 3.9	77.09 ± 4.4	72.16 ± 4.3	58.23	53.05	65.16	59.66
0.9		93.54 ± 2.0	87.46 ± 4.0	74.05 ± 4.8	69.22 ± 4.1	56.22	51.22	64.13	58.74
1.0		92.32 ± 2.2	86.19 ± 3.9	67.98 ± 5.4	63.34 ± 4.3	53.99	49.11	56.03	51.24

* This is essentially the baseline model. It does not use a decoder.

and

Table 4. Mean Average Precision of the KWS-Simplified backbone with the weighted combination strategy across semantic importance thresholds and embeddings.

Semantic Weight	Semantic LLM	George Washington Dataset				IAM Handwriting Database
		KWS-Simplified Decoder		TrOCR Decoder		KWS-Simplified Decoder		TrOCR Decoder
		mAP@25	mAP@50	mAP@25	mAP@50	mAP@25	mAP@50	mAP@25	mAP@50
0.0 *	all -MiniLM -L12 -v2	89.74 ± 0.7	72.29 ± 3.0	89.74 ± 0.7	72.29 ± 3.0	86.40	63.73	86.40	63.73
0.1		90.62 ± 0.5	72.77 ± 3.0	90.27 ± 0.8	72.53 ± 3.1	86.71	63.86	87.49	64.22
0.2		90.68 ± 0.4	72.82 ± 3.0	90.38 ± 0.7	72.57 ± 3.1	86.79	63.94	87.84	64.47
0.3		90.70 ± 0.5	72.84 ± 3.0	90.35 ± 0.7	72.55 ± 3.2	86.57	63.72	88.01	64.58
0.4		90.72 ± 0.5	72.84 ± 3.0	90.30 ± 0.7	72.51 ± 3.2	86.11	63.36	88.06	64.60
0.5		90.79 ± 0.4	72.90 ± 3.0	90.32 ± 0.6	72.55 ± 3.2	85.65	63.01	87.74	64.43
0.6		90.85 ± 0.5	72.93 ± 3.0	90.32 ± 0.6	72.54 ± 3.2	84.66	62.38	87.28	64.17
0.7		90.91 ± 0.5	72.97 ± 3.0	90.26 ± 0.7	72.51 ± 3.2	83.32	61.47	86.30	63.56
0.8		90.83 ± 0.4	72.90 ± 2.9	90.12 ± 0.6	72.42 ± 3.1	81.31	60.20	84.66	62.49
0.9		90.82 ± 0.4	72.89 ± 2.9	89.93 ± 0.6	72.29 ± 3.0	77.61	57.85	81.51	60.47
1.0		90.43 ± 0.7	72.73 ± 2.7	87.16 ± 0.6	70.39 ± 2.1	71.97	54.39	76.74	57.24
0.0 *	all -mpnet -base -v2	89.74 ± 0.7	72.29 ± 3.0	89.74 ± 0.7	72.29 ± 3.0	86.40	63.73	86.40	63.73
0.1		90.63 ± 0.5	72.78 ± 3.0	90.22 ± 0.7	72.49 ± 3.1	86.69	63.88	87.53	64.25
0.2		90.69 ± 0.5	72.83 ± 3.0	90.31 ± 0.7	72.51 ± 3.1	86.70	63.87	87.89	64.51
0.3		90.72 ± 0.5	72.84 ± 3.0	90.29 ± 0.6	72.49 ± 3.1	86.56	63.75	88.16	64.80
0.4		90.81 ± 0.5	72.91 ± 3.1	90.32 ± 0.7	72.54 ± 3.1	86.32	63.59	88.25	64.88
0.5		90.86 ± 0.5	72.96 ± 3.1	90.29 ± 0.7	72.53 ± 3.2	85.91	63.33	87.97	64.69
0.6		90.87 ± 0.4	72.94 ± 3.0	90.26 ± 0.6	72.49 ± 3.1	85.03	62.71	87.52	64.37
0.7		90.94 ± 0.5	72.97 ± 3.1	90.10 ± 0.7	72.39 ± 3.1	84.15	62.10	86.60	63.77
0.8		90.91 ± 0.5	72.96 ± 3.0	90.06 ± 0.6	72.35 ± 3.0	82.56	61.09	85.07	62.78
0.9		90.85 ± 0.4	72.92 ± 3.0	89.74 ± 0.4	72.09 ± 2.9	79.47	59.19	81.97	60.86
1.0		90.47 ± 0.5	72.78 ± 2.8	87.06 ± 0.7	70.23 ± 2.1	74.04	55.89	77.30	57.78
0.0 *	stsb -roberta -base	89.74 ± 0.7	72.29 ± 3.0	89.74 ± 0.7	72.29 ± 3.0	86.40	63.73	86.40	63.73
0.1		90.62 ± 0.5	72.78 ± 3.0	90.21 ± 0.6	72.46 ± 3.0	86.58	63.82	87.35	64.13
0.2		90.63 ± 0.5	72.79 ± 3.0	90.27 ± 0.6	72.47 ± 3.0	86.78	63.85	87.78	64.52
0.3		90.66 ± 0.5	72.80 ± 3.0	90.27 ± 0.6	72.47 ± 3.1	86.72	63.74	88.04	64.72
0.4		90.76 ± 0.5	72.88 ± 3.0	90.32 ± 0.6	72.52 ± 3.1	86.45	63.59	88.12	64.84
0.5		90.89 ± 0.5	72.97 ± 3.1	90.33 ± 0.5	72.54 ± 3.1	85.87	63.29	87.62	64.60
0.6		90.90 ± 0.4	72.97 ± 3.1	90.30 ± 0.4	72.50 ± 3.0	85.09	62.79	86.78	64.16
0.7		90.92 ± 0.4	72.97 ± 3.1	90.13 ± 0.4	72.41 ± 2.9	83.82	61.99	85.44	63.32
0.8		90.86 ± 0.4	72.92 ± 3.0	89.96 ± 0.5	72.28 ± 2.9	81.96	60.79	83.06	61.79
0.9		90.74 ± 0.4	72.86 ± 3.0	89.75 ± 0.5	72.12 ± 2.8	79.45	59.12	80.17	59.88
1.0		90.33 ± 0.5	72.71 ± 2.8	87.08 ± 0.9	70.24 ± 2.0	76.10	57.18	76.49	57.40

* This is essentially the baseline model. It does not use a decoder.

present the numerical results obtained after re-ranking the initial WordRetrievalNet and KWS-Simplified ranking lists using the proposed weighted combination strategy. These results confirm our intuition that the combination of verbatim and semantic information actually enhances the performance. On the contrary,

Table 5. mAP comparison of semantic pruning pipelines on the GW dataset.

Threshold	Semantic LLM	WordRetrievalNet				KWS-Simplified
		KWS-Simplified Decoder		TrOCR Decoder		KWS-Simplified Decoder		TrOCR Decoder
		mAP@25	mAP@50	mAP@25	mAP@50	mAP@25	mAP@50	mAP@25	mAP@50
0.1	all -MiniLM -L12 -v2	94.25 ± 1.7	88.24 ± 3.9	94.16 ± 1.7	88.20 ± 4.1	89.63 ± 0.8	72.18 ± 3.1	89.50 ± 0.9	72.05 ± 3.0
0.3		92.02 ± 2.3	86.32 ± 3.7	77.32 ± 6.3	71.77 ± 4.3	87.52 ± 1.2	70.26 ± 3.7	71.46 ± 5.1	56.03 ± 3.9
0.5		89.87 ± 2.3	84.52 ± 3.7	67.59 ± 6.7	63.15 ± 4.8	86.53 ± 1.6	69.69 ± 4.1	63.25 ± 6.1	49.08 ± 4.3
0.7		88.58 ± 2.6	83.68 ± 3.8	63.25 ± 6.3	59.34 ± 4.7	85.62 ± 2.2	69.01 ± 4.7	59.35 ± 5.9	45.98 ± 4.3
0.9		87.81 ± 2.7	83.06 ± 3.8	60.10 ± 6.6	56.45 ± 5.3	85.24 ± 2.3	68.71 ± 4.7	56.53 ± 5.9	43.64 ± 4.4
0.1	all -mpnet -base -v2	94.29 ± 1.8	88.28 ± 4.0	93.96 ± 2.1	87.97 ± 4.2	89.72 ± 0.7	72.27 ± 3.0	89.34 ± 0.9	71.90 ± 2.9
0.3		91.74 ± 2.0	86.21 ± 3.4	76.40 ± 5.8	71.46 ± 4.0	87.91 ± 1.4	70.75 ± 3.8	71.68 ± 4.9	56.63 ± 3.4
0.5		89.51 ± 2.2	84.27 ± 3.7	66.96 ± 7.0	62.76 ± 5.4	86.46 ± 2.0	69.72 ± 4.5	62.64 ± 5.7	48.51 ± 4.4
0.7		88.50 ± 2.3	83.51 ± 3.8	63.11 ± 6.2	59.20 ± 4.7	85.72 ± 1.9	69.04 ± 4.5	59.04 ± 5.8	45.68 ± 4.0
0.9		87.50 ± 2.9	82.76 ± 4.0	59.70 ± 6.8	56.05 ± 5.4	85.24 ± 2.5	68.71 ± 4.8	56.15 ± 6.0	43.26 ± 4.4
0.1	stsb -roberta -base	94.33 ± 1.8	88.32 ± 4.0	93.49 ± 1.9	87.45 ± 4.2	89.74 ± 0.7	72.29 ± 3.0	88.96 ± 1.1	71.52 ± 3.2
0.3		94.32 ± 2.1	88.33 ± 4.2	85.81 ± 3.4	80.42 ± 4.3	89.05 ± 1.2	71.73 ± 3.4	81.33 ± 2.7	64.75 ± 3.0
0.5		92.95 ± 2.0	87.08 ± 3.7	73.83 ± 4.6	69.09 ± 4.0	88.00 ± 1.6	70.90 ± 4.1	69.24 ± 3.8	54.77 ± 3.5
0.7		90.72 ± 2.5	85.36 ± 3.7	65.99 ± 5.3	62.04 ± 4.0	86.28 ± 2.4	69.62 ± 4.7	62.33 ± 4.6	48.99 ± 3.3
0.9		87.82 ± 2.9	83.02 ± 3.9	60.61 ± 6.2	56.93 ± 4.9	85.35 ± 2.4	68.83 ± 4.8	57.18 ± 5.4	44.19 ± 3.9

shows the results for the alternative strategy based on semantic pruning. Despite its aim to alleviate the retrieval by identifying semantically irrelevant instances, the strategy is overly simplistic and effectively hampers system performance. The performance on GW is reported as the average mAP@25 and mAP@50 across the experimental iterations, along with the corresponding standard deviations (SDs) across the four trials. The contribution of each pipeline component is discussed in detail below.

4.4.1. Impact of the Baseline KWS Model

The effect of our semantic re-ranking pipeline varies depending on the baseline KWS model. On the one hand, WordRetrievalNet exhibits stronger improvements compared to the KWS-Simplified variant. As shown in Table 3 and Figure 6, the gains on the GW dataset reach $\sim 2.3\%$ in mAP@25 (from 94.31% to 96.59%) and $\sim 0.9\%$ in mAP@50 (from 88.29% to 90.17%). Even more noticeable are the improvements on the IAM dataset, where mAP@25 rises by $\sim 3\%$ (from 79.15% to 82.12%) and mAP@50 by $\sim 2.6\%$ (from 72.85% to 75.43%).

On the other hand, KWS-Simplified achieves smaller gains, as reported in Table 4 and Figure 7. On GW, the re-ranking yields an increase of $\sim 1.2\%$ in mAP@25 (from 89.74% to 90.94%) and $\sim 0.7\%$ in mAP@50 (from 72.29% to 72.97%). On IAM, the respective gains are $\sim 1.9\%$ (mAP@25: 86.40% to 88.25%) and $\sim 1.2\%$ (mAP@50: 63.73% to 64.88%).

These differences can be attributed to the design of each backbone. WordRetrievalNet produces a larger and richer set of candidate detections for each query, since it performs query-independent spotting by precomputing visual features (e.g., DCToW) across the document. This allows the re-ranking module to refine the initial results more effectively, as it has access to more potential matches, including those ranked low in the initial retrieval.

Conversely, KWS-Simplified retrieves a very limited set of candidates, typically just a handful per query. This restricts the impact of re-ranking, as relevant instances not retrieved initially cannot be recovered in later stages. Therefore, KWS-Simplified may benefit from integrating query expansion techniques aimed at enlarging the initial candidate pool, making semantic re-ranking more effective.

4.4.2. Impact of the Decoder

The decoder module directly influences the effectiveness of the re-ranking process, as reflected in the performance variations across datasets and architectures. On GW, the KWS-Simplified decoder yields higher mAP gains than TrOCR, improving mAP@25 by $\sim 2.3\%$ (from 94.31% to 96.59%) and mAP@50 by $\sim 0.9\%$ (from 88.29% to 90.17%). Unlike the first case, TrOCR shows smaller gains: $\sim 0.6\%$ for mAP@25 (from 89.74% to 90.38%) and $\sim 0.3\%$ for mAP@50 (from 72.29% to 72.57%).

This pattern reverses on IAM, where the TrOCR decoder, when paired with WordRetrievalNet, achieves stronger improvements: $\sim 3\%$ (mAP@25: from $79.15\%$ to $82.16\%$) and $\sim 2.6\%$ (mAP@50: from $72.85\%$ to $75.43\%$). The KWS-Simplified decoder on the same dataset shows slightly lower gains: $\sim 2\%$ for mAP@25 and $\sim 1.6\%$ for mAP@50. These trends are summarized in Table 3.

Such results are consistent with prior work in KWS: the final performance is shaped not only by the quality of the decoder but also by the accuracy of the initial bounding boxes [87]. Even a strong decoder like TrOCR may struggle on GW due to limited candidate diversity, whereas it benefits more on IAM, where segmentation is more fine-grained and the linguistic space is richer. This highlights the interplay between decoder expressiveness and the underlying retrieval quality.

Ultimately, these findings emphasize the importance of holistic system design. Decoder selection should be based not only on CER or transcription quality but also on how well it complements the retrieval front end. While our current strategy employs a simple late-fusion scheme, it still provides meaningful gains with no additional supervision and minimal computational overhead.

4.4.3. Semantic Embedding Models

We expected that the more diversely trained semantic embedding models (all-MiniLM-L12-v2, all-mpnet-base-v2) would outperform stsb-roberta-base. However, across all model and decoder combinations, we observed at most a $0.5\%$ difference in both mAP@25 and mAP@50, with performance typically being indistinguishable across GW and IAM benchmarks. This can be attributed to factors such as transcription errors, the reliance on word-level context alone, and dataset limitations (i.e., the small size of GW), all of which prevent the more expressive models from demonstrating their full semantic representational power.

4.4.4. Fusion Strategy: Weighted Combination

The weighted combination of verbatim and semantic relevance scores leads to consistent performance gains, as showcased in Table 3 and Table 4, along with Figure 6 and Figure 7. Nonetheless, the extent of the improvement and the importance of the semantic relevance level depend on the performance of the underlying keyword spotter and the decoder architecture.

On the GW dataset, the already high performance of WordRetrievalNet shifts the optimal semantic relevance weight toward lower values, favoring verbatim relevance, for both decoders. This effect is less pronounced for the KWS-Simplified decoder, where optimal performance is achieved when the semantic weight lies within [0.2, 0.5]. Instead, for the TrOCR-based decoder, as well as the WordRetrievalNet on IAM, the best performance occurs when the semantic importance is minimal, typically within [0.1, 0.2]. Outside these ranges, and particularly when relying solely on semantic similarity, performance deteriorates. This decline indicates that recognition errors introduced during transcription propagate into the semantic space as well. Additionally, it highlights the importance of leveraging both verbatim and semantic signals for effective re-ranking. In each of these cases, the performance drops sharply as the level of semantic importance increases. For instance, in the TrOCR-decoded pipeline, mAP@25 decreases from $94.31\%$ to approximately $65\%$, and mAP@50 drops from $88.29\%$ to about $62\%$ on average over all choices of semantic embeddings.

By comparison, when re-ranking the obtained results from the KWS-Simplified backbone, as shown in Table 4, the initial ranking is less accurate, and in turn, the best results are skewed toward the semantic end. In this case, the re-ranking consistently outperforms the baseline for the KWS-Simplified decoder but remains on par with its TrOCR counterpart. Notably, there appears to be a synergy between the KWS-Simplified decoder and the baseline model. Since the decoder is an integral part of the baseline, the initially predicted bounding boxes are optimally aligned for the decoder to use. Consequently, the incorporation of semantic relevance offers clear advantages for this model.

4.4.5. Fusion Strategy: Semantic Pruning

Thus far, we have presented results based on the weighted combination strategy, which consistently delivers the best performance among the tested fusion methods. On the contrary, semantic pruning, a simpler alternative, performs noticeably worse across all configurations, as shown in Table 5 and Figure 8.

The key limitation of semantic pruning lies in its rigid filtering rule, which accepts only candidates exceeding a predefined similarity threshold. While this may boost semantic purity, it often leads to valid instances being discarded due to minor recognition or decoding errors. As a result, retrieval performance deteriorates, especially under stricter thresholds.

Despite its suboptimal results in verbatim keyword spotting, this strategy serves an illustrative role: it highlights the value of combining semantic and lexical signals rather than treating them in isolation. In future applications, semantic pruning might be better suited for tasks where conceptual alignment or query expansion is more critical than exact string matches.

4.4.6. Qualitative Analysis

Previous sections evaluated the quantitative performance of the proposed framework through mAP metrics. We now examine its qualitative benefits: the system successfully augments retrieval with semantic matches while preserving exact lexical matching capabilities. Visual examples demonstrate these enhancements, which transcend standard evaluation indices of verbatim KWS.

Figure 9 shows three top ten ranked lists for the query “forgot”: the first ranked by conventional KWS (verbatim) similarity, the second ranked by semantic similarity, and the last ranked by the combined similarity of our proposed system. For each instance, the top-left corner displays the global rankings using different colors: purple for verbatim, yellow for semantic, and blue for combined similarity. The bottom-right corner shows the similarity scores with the query, following the same color scheme. The ground-truth bounding box is highlighted in green.

The relevant instance corresponding to the query “forgot” is initially ranked tenth, following several instances of “fort”, due to a suboptimal bounding box prediction. Both the semantic ranking and the combined re-ranking successfully identify its true relevance, boosting its score while reducing the scores of visually similar false positives corresponding to “fort”. However, visually similar results still dominate the combined ranking due to a number of factors, including the limited size and variability of GW, as well as significant class imbalance, meaning a considerable number of words either have only a few or entirely lack semantically relevant instances in the dataset. Although the re-ranking process successfully moves the correct instance to the first position, its bounding box remains unchanged. As a result, under stricter overlap thresholds, the correction is effectively ignored, explaining the higher performance improvement in mAP@25 over mAP@50.

Another qualitatively interesting result of the proposed pipeline, potentially useful in recognition-free word-image retrieval, is illustrated in Figure 10. In a similar fashion to the previous example, we present a ranked list for the query “soldiers”. While the vocabulary of the GW dataset generally lacks semantic depth due to its limited size, this effect is less prevalent in certain areas, such as military terminology. In such a scenario, the qualitative benefit is clear: when a user searches for military-related terms (e.g., “soldiers”), it is more likely for the system to retrieve other relevant military terms than it is for it to retrieve unrelated results. Additionally, note that there is only one instance of the word “soldiers” in the dataset. The second instance in the initial verbatim retrieval task is an erroneous duplication of the first, possibly due to suboptimal NMS. The final re-ranking correctly discards this duplicate, a qualitative improvement not effectively measured by mAP.

4.5. Discussion

In a nutshell, our semantic re-ranking framework operates as a modular, plug-and-play extension to existing KWS systems, requiring neither retraining nor dataset-specific adaptation. In contrast with approaches that rely on fine-tuned embeddings or corpus-dependent training, our method is able to generalize effectively across datasets. This generalization capability is further supported by consistent performance gains observed across both datasets. To ensure robustness of the GW results despite the limited size of the dataset, we reported mAP scores averaged over four experimental trials. The consistently low standard deviation observed for all experiments showcases the reliability and stability of the proposed semantic augmentation pipeline. This robustness is further evidenced by the overall effectiveness of the method considering different semantic embedding models, which highlights its adaptability to varying NLP backends.

The broader candidate pool and richer semantic space of IAM allow the semantic re-ranking to manifest more clearly, facilitating its ability to exploit higher-level meaning when more linguistic diversity is available. These results validate the generalization capacity of our method to adapt to heterogeneous data scenarios. Notably, WordRetrievalNet features improvements of $3\%$ in mAP@25 (from $79.15\%$ to $82.12\%$) and $2.6\%$ in mAP@50 (from $72.85$% to $75.43\%$), while KWS-Simplified achieves $+1.85\%$ (from $86.40\%$ to $88.25\%$) and $+1.15\%$ (from $63.73\%$ to $64.88\%$), respectively.

In our numerical results over GW, we observe consistent improvements for both baseline models when semantic relevance is incorporated. This behavior holds across both mAP@25 and mAP@50 metrics, regardless of the decoder architecture. WordRetrievalNet achieves the most substantial gain, with mAP@25 increasing by $2.3\%$ (from $94.31\%$ to $96.59\%$), while KWS-Simplified improves by $1.2\%$ (from $89.74\%$ to $90.94\%$).

Although the relative improvements in mAP@50 are more modest (typically less than $1\%$), this does not fully capture the value of our method. The high baseline performance suggests that remaining false negatives are inherently difficult cases, often involving retrieved instances that fall below the IoU threshold due to imperfect bounding box predictions by the underlying word spotter. As a result, even when re-ranking elevates a semantically relevant instance, it may not be counted as correct under the strict overlap criterion. This evaluation limitation underscores the need for post-retrieval refinements that can adjust the predicted bounding boxes. Future extensions such as query expansion [43], late fusion [88], or joint re-ranking and refinement modules could alleviate this issue.

A natural direction for future exploration involves moving beyond late fusion toward fully trainable semantic re-ranking modules. Integrating fusion architectures inspired by recent vision-language models, such as CLIP-Rerankers [89] or BLIP [90], could enable richer cross-modal interactions between query and candidate embeddings. While such models introduce additional training complexity, they offer the potential to jointly optimize retrieval, decoding, and semantic alignment in a single unified pipeline.

5. Conclusions

This work introduced a lightweight, modular re-ranking pipeline for segmentation-free keyword spotting (KWS) that incorporates semantic relevance signals derived from large pre-trained language models. Motivated by the observation that many top-ranked results from visual-only KWS systems are semantically misaligned with the user’s intent, we proposed a simple yet effective mechanism that blends visual and semantic similarity scores to re-order retrieved word instances without retraining any component of the baseline system.

Experimental results on both the George Washington and IAM datasets validate our initial hypothesis: semantic feedback can systematically enhance precision while maintaining recall. Considering two representative segmentation-free baselines (WordRetrievalNet and KWS-Simplified) and different decoding strategies, consistent improvements in Mean Average Precision (mAP) were observed, most notably, up to a ∼2.3% absolute gain in mAP@25 for WordRetrievalNet on GW, as well as a ∼3% improvement on IAM when the WordRetrievalNet is paired with TrOCR. This confirms that integrating contextual word embeddings into the retrieval loop improves retrieval quality even in recognition-free setups.

The robustness of our framework is further supported by its low variance among all cross-validation folds and its insensitivity to the choice of semantic embedding model. The optimal fusion weights suggest that a balanced contribution between verbatim and semantic signals yields the best performance. Furthermore, qualitative examples demonstrate that semantically relevant but visually dissimilar terms can be effectively elevated in the ranking, demonstrating the ability of the method to perform beyond mere pattern matching.

Looking ahead, several promising research directions emerge:

• Exploring transformer-based late fusion strategies that support end-to-end trainable re-ranking, inspired by cross-modal architectures such as CLIP-Rerankers and BLIP;
• Incorporating dynamic query expansion mechanisms via LLMs to improve semantic coverage and reduce reliance on exact phrasing;
• Integrating joint optimization objectives with vision–language models to enhance alignment between visual content and textual intent.

We also acknowledge the emergence of hybrid architectures in document layout analysis that integrate object detectors with Vision Transformers. A notable example is the Vision Grid Transformer (VGT) proposed by [91], which adopts a two-stream framework combining visual and grid-based textual features, achieving state-of-the-art results across multiple benchmarks. Despite their impressive performance, such models typically rely on large-scale annotated datasets—a requirement that is often infeasible to meet in historical manuscript collections. This limitation reinforces the relevance of our segmentation-free KWS approach, which is designed to operate effectively even under limited supervision.

Ultimately, our findings encourage a shift in perspective wherein deep language models need not be confined to transcription but can actively contribute to the semantic understanding and retrieval of handwritten documents. We hope this work paves the way toward more intelligent, generalizable, and semantically aware document analysis systems.