Multi-Level Feature-Matching System for Counterfeit Seal Image Recognition ^†

Abstract

As document forgery schemes become increasingly sophisticated, organizations face mounting challenges in authenticating seals found on official documents. In this study, we collaborated with law enforcement agencies in Taiwan to develop an AI-driven system that supports the rapid identification of forged seals. Instead of relying on manual inspection, the system leverages deep neural networks to analyze overall and fine visual features of seal images. By integrating advanced image enhancement, similarity measurement, and feature comparison modules, the system efficiently filters and ranks potential matches from a dedicated police database. Evaluation on a dataset containing several hundred forged seal images demonstrates that the system achieves greater than 90% accuracy for detecting counterfeit seals. The solution not only reduces the time and effort required for verification but also provides investigators with immediate access to relevant case histories, thereby strengthening the overall fraud prevention workflow.

1. Introduction

Rapid digital advancements and sophisticated fraud tactics have posed serious threats to public security through scams involving forged seals. Taiwan’s Criminal Investigation Bureau reports impersonation fraud accounted for over NT$837 million in losses from 2022 to 2023, with an extra NT$135 million in the first half of 2024 [1]. Criminals impersonate officials using forged seals and realistic documents to demand payments, deceiving victims unaware of verification methods. Manual verification is delayed by complex procedures.

We propose an AI seal image retrieval system capable of real-time analysis and identification to address this issue. This system leverages image matching and retrieval techniques. Although image retrieval is widely applied in location recognition and search engines, its application in seal identification remains unexplored.

Recent studies have focused on deep learning to improve keypoint detection and matching. SuperPoint [2] introduced a self-supervised framework for keypoint detection and description, while the Detect-to-Describe Network (D2-Net) [3] used end-to-end Convolutional Neural Networks (CNNs) to enhance matching precision. Building on this, SuperGlue [4] integrates Graph Neural Networks (GNNs) with attention mechanisms to model contextual relationships between keypoints, enabling robust matching. It allows accurate matching even under tilted angles, partial occlusions, or surface distortions, common challenges in real-world seal images, making SuperGlue especially effective for forged seal identification.

Additionally, early retrieval methods focused on regional features to improve precision, as relying solely on global features often led to incorrect matches, particularly when the query image only partially overlapped with the target. For example, Wu et al. [5] proposed a regional feature-based framework that leverages CNNs and Local Feature Self-Attention [6], which extracts visual tokens using spatial attention and refines them through self- and cross-attention mechanisms. Building upon this idea, recent research has shifted toward two-stage retrieval frameworks that integrate both global and regional features, a strategy that has since become the mainstream approach [7]. In this configuration, global features are employed in the first stage for coarse filtering, while regional features are used in the second stage for re-ranking. Yang et al. [8] further improved this design with a dual-branch architecture, the Deep Orthogonal Local and Global Feature Framework, to enhance regional features using self-attention and extract components that are orthogonal to global features, thereby generating more representative features for effective retrieval.

Based on the results, we adopted a three-step retrieval approach. The first step involves image preprocessing to enhance seal quality and ensure consistent input, including noise removal, alignment correction, and clean seal extraction. In the second step, global features are used for large-scale filtering to narrow down candidate matches. Finally, the regional features are applied for fine comparison. This approach significantly reduces human processing time while maintaining high accuracy. The results show that the system automatically detected 97% of counterfeit official seals and 92% of forged company seals. This fast and accurate system reduces the workload on law enforcement, supports fraud prevention, and enhances public financial security through the use of AI-driven applications.

2. Three-Step Forged Seal Image Recognition

We propose a three-step forged seal image recognition method (Figure 1), designed to analyze and determine whether a given seal image matches any known counterfeit cases and to assess the likelihood of forgery. The process begins with preprocessing seal images as Step 1, which involves removing irrelevant parts of the input seal image and extracting a single seal to optimize feature-matching performance. Step 2 involves low-level feature extraction using the Visual Geometry Group 16 (VGG16) pre-trained model to identify highly similar candidate seals from the seal database based on global features. Step 3 applies the SuperGlue model to compare the detailed regional features, capturing fine and texture patterns of the seal image more precisely. The final step balances identification accuracy and efficiency, progressively narrowing down potential counterfeit seals. The following sections provide detailed descriptions of each step.

2.1. Step 1: Seal Image Preprocessing

To optimize feature matching, the system removes irrelevant components, such as black document text, which often overlap red seals. The input image is converted into the hue, saturation, and intensity (HIS) color space to separate color information. Red and black masks are generated, and overlapping pixels are identified to isolate the red seal. Non-overlapping black areas are removed to suppress text interference. The “I” channel is used for binarization [9], with Otsu’s algorithm [10] determining the optimal threshold. YOLOv8 [11] detects the seal region, which is cropped and resized to 224 × 224 to ensure input consistency (Figure 2).

Seal images can be misaligned due to shooting angles or distorted by thin paper. To ensure reliable on-site verification, the system must correct such issues automatically. Therefore, Geometric Transformation Matrices [12] and the Gradient Descent algorithm [13] are used to optimize image alignment. The peak signal-to-noise ratio (PSNR) [14] serves as the objective function to minimize the distortion’s impact on feature-matching accuracy. We define the optimization parameter vector $p$ = [$\theta$, $C_x$, $C_y$, $t_x$, $t_y$, $H_h$, $H_v$], which includes the rotation angle ($\theta$), scaling factors ($C_x$, $C_y$), translation distances ($t_x$, $t_y$), and shear degrees ($H_h$, $H_v$). Using the transformation matrix $A\left(p\right)$ to rectify the input image I, we can obtain:

$$A\left(p\right)=R·S·T·H,$$

(1)

The transformation matrices are defined as follows.

• Rotation matrix $R$ is defined as:

$$\left[\begin{array}{ccc}\cos \theta & -\sin \theta & 0\\ \sin \theta & \cos \theta & 0\\ 0& 0& 1\end{array}\right].$$

(2)

• Scaling matrix $S$ is defined as:

$$\left[\begin{array}{ccc}{C}_x& 0& 0\\ 0& C_y& 0\\ 0& 0& 1\end{array}\right].$$

(3)

• Translation matrix $T$ is defined as:

$$\left[\begin{array}{ccc}1& 0& t_x\\ 0& 1& t_y\\ 0& 0& 1\end{array}\right].$$

(4)

• Shear matrix $H$ is defined as:

$$\left[\begin{array}{ccc}1& H_v& 0\\ H_h& 1& 0\\ 0& 0& 1\end{array}\right].$$

(5)

Since $PSNR$ is computed as:

$$PSNR=10 \times {log}_{10}\left({\displaystyle \frac{{MAX}^2}{MSE}}\right),$$

(6)

where MAX represents the maximum pixel value of an image (e.g., the maximum value of an 8-bit pixel is 255). To maximize PSNR, the loss function L is defined as:

$$L= {log}_{10}\left(MSE\right),$$

(7)

where

$$MSE={\displaystyle \frac{1}{mn}}{\sum }_{i=0}^{m-1}{\sum }_{j=0}^{n-1}{\left[R\left(i,j\right)-{A(I;p)}_{i,j}\right]}^2,$$

(8)

where $R\left(i,j\right)$ and $A\left(I;p\right)$ represent the pixel values of the reference image and the rectified image, respectively, and m and n denote the height and width of the image. To minimize the loss function $L\left(p\right)$, the gradient descent method is used to update the parameter vector $p$ as:

$$p^{t+1}=p^t-\eta {\displaystyle \frac{\mathsf{\partial }L\left(p\right)}{\mathsf{\partial }p}},$$

(9)

where η is the learning rate, which controls the step size of parameter updates.

Geometric transformations, such as rotation, scaling, translation, and shearing, aim to comprehensively correct the image and minimize the negative impact of the shooting environment. This process significantly enhances the accuracy and reliability of feature matching.

2.2. Step 2: VGG16-Based Global Feature Matching

After preprocessing and correction, the system extracts low-level features using the pre-trained VGG16 model [15]. Since seal characteristics primarily consist of lines and text, features are extracted from the first three layers (conv1_2, conv2_2, conv3_2) to capture essential variations as shown in Figure 3 and Figure 4. The figures highlight key differences between seals.

The feature extraction process can be mathematically represented as:

$$F = \phi l\left(I\right), l \in \left\{conv{1}_1, conv{2}_2, conv{3}_3\right\},$$

(10)

where $F$ represents the extracted features, $\phi l$ denotes the feature extraction function, $I$ is the input image, and $l$ indicates the convolutional layer used. To quantify the distance between two images in the feature space, Euclidean Distance is adopted as the metric, which is defined as:

$$D\left(F^{\left(1\right)}, F^{\left(2\right)}\right)= \sqrt{{\sum }_{i=1}^n{\left(F_i^{\left(1\right)}-F_i^{\left(2\right)}\right)}^2},$$

(11)

where $F^{\left(1\right)}$ and $F^{\left(2\right)}$ are feature vectors from the same convolutional layer of two images, with $F_i^{\left(1\right)}$ and $F_i^{\left(2\right)}$ representing values at the i-th dimension. n is the feature dimension. Furthermore, to further evaluate the similarity between the two images, Cosine Similarity is used as the metric, defined as:

$$S= {\displaystyle \frac{F^{\left(1\right)}·F^{\left(2\right)}}{\Vert F^{\left(1\right)}\Vert ·\Vert F^{\left(2\right)}\Vert }},$$

(12)

where $F^{\left(1\right)}·F^{\left(2\right)}$ represents the dot product. The terms $\Vert F^{\left(1\right)}\Vert$ and $\Vert F^{\left(2\right)}\Vert$ denote their respective L2 norms (Euclidean Norm) as:

$$\Vert F\Vert = \sqrt{{\sum }_{i=1}^n{F}_i^2},$$

(13)

where $F_i$ represents the i-th component of the feature vector $F$, and n is the dimension of the feature vector. The average similarity between all images in the dataset and the input image is computed to obtain a comprehensive similarity evaluation as:

$$\overline{S_l}= {\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{S}_{il},$$

(14)

where $S_{il}$ represents the similarity between the input image and the i-th image in the dataset at the l-th convolutional layer. N is the number of images in the dataset, and $\overline{S_l}$ is the average similarity of the dataset at the given convolutional layer.

Although the VGG16 model’s low-level features are useful for broad comparisons, they struggle to capture fine differences in complex seal images (Figure 5). To address this limitation, Support Vector Machine (SVM) [16] is used to further classify and filter results. The extracted feature value $F$ is input into the SVM for analysis to identify highly similar candidate seals. This approach efficiently narrows the search range and filters out irrelevant images early, improving overall computational efficiency.

2.3. Step 3: SuperGlue-Based Local Feature Matching

To overcome the limitations of global low-level features, we introduce a second-step local feature matching based on the SuperGlue feature point matching algorithm [6]. This method utilizes deep geometric reasoning and GNNs to perform accurate keypoint matching between seal images, as shown in Figure 6.

The process begins by extracting keypoints and descriptors using the SuperPoint model [4]. Given two input images A and B, the extracted keypoint sets are defined as:

$$A=\left\{A_1, A_2,\dots ,A_m\right\}, B=\left\{B_1, B_2,\dots ,B_n\right\},$$

(15)

where A₁ and B₁ represent the keypoints of images A and B, respectively, extracted using SuperPoint [4]. Each keypoint consists of a coordinate position (p_i) and a descriptor (d_i).

$$A_i=\left(p_i^A,d_i^A\right){, B}_i=\left(p_i^B,d_i^B\right).$$

(16)

To improve matching accuracy, the SuperGlue method employs a multilayer perceptron as a keypoint encoder, embedding positional information into the initial keypoints (q_i) for GNN processing. Its multi-GNN module captures both intra-image and cross-image relationships. Using self-attention and cross-attention, SuperGlue integrates local and global contexts, updating keypoint representations through a shared attention mechanism as:

$$h_i^{\prime }=\sum _j{\alpha }_{ij}{v}_j, v_j=W_v{h}_j,$$

(17)

where $W_v$ is the learned weight matrix, $h_j$ represents the information on neighboring keypoints (previous state), and $h_i^{\prime }$ is the updated feature representation (new state). In the coefficient, i refers to the currently updated keypoint, and j represents the keypoints associated with i. The attention weight ${\alpha }_{ij}$ is computed as the Softmax similarity between the key ($k_j$) and the query ($q_j$) as:

$${\alpha }_{ij}={Softmax}_j\left(q_j^T{k}_j\right).$$

(18)

The newly computed keypoint information allows better utilization of contextual information, enhancing the robustness of matching. The optimal matching layer is combined with Optimal Transport to compute the matching probability. The matching matrix S is computed as:

$$S_{ij}= <f_i^A, f_j^B>,$$

(19)

where $f_i^A$ and $f_j^B$ represent keypoint features in SuperGlue. They are designed to adapt to scene variations, such as perspective and lighting changes, through a learned transformation function as:

$$f_i^k=W_f{h}_i+b, k\in \left\{A,B\right\},$$

(20)

where $W_f$ is the learned weight matrix, $b$ is a bias term, and $f_i^A$ represents the updated keypoint information. Finally, the Sinkhorn algorithm [17] is applied for regularization as:

$$S\leftarrow Sinkhorn\left(S\right).$$

(21)

Finally, SuperGlue utilizes the cross-entropy loss to train the model, thereby enhancing the precision of the matching results. During training, the supervision signals originate from the ground truth matching matrix, which is defined as:

$$\mathcal{M}=\left\{\left(i, j\right)\right\}\subset A\times B,$$

(22)

where A and B represent the keypoint sets of the respective images, and $A\times B$ indicates all possible keypoint pairings. The loss is computed over this set to enable the supervised model to learn accurate matching relationships. The loss function is defined as:

$$Loss=-\sum _{\left(i, j\right)\in \mathcal{M}}\mathit{log}{P}_{i, j}-\sum _{i\in \mathcal{L}}\mathit{log}{P}_{i, n+1}-\sum _{j\in \mathcal{T}}\mathit{log}{P}_{m+1, j},$$

(23)

where $P_{i,j}$ denotes the predicted matching probability, where i and j correspond to different keypoints, and $\mathcal{L}$ and $\mathcal{T}$ represent the sets of all keypoints in images A and B, respectively. The first term in the loss function, $\sum _{\left(i, j\right)\in \mathcal{M}}\log P_{i, j}$, calculates the loss for correctly matched keypoints based on probability, enhancing the prediction confidence of Ground Truth matches. To account for unmatched keypoints, SuperGlue adds an extra row and column ($n+1$, $m+1$), respectively. The unmatched terms $\sum _{i\in \mathcal{L}}\log P_{i, n+1}$ and $\sum _{j\in \mathcal{T}}\log P_{m+1, j}$ penalizes false matches, enhancing robustness.

Compared with the low-level global features extracted from the VGG16 model, SuperGlue focuses on learning geometric relationships between local keypoints, enabling precise alignment under perspective changes or distortions. It enhances the accuracy of forged seal identification, particularly when distinguishing between highly similar samples.

3. Results

3.1. Forged Seal Dataset from Police Department

In this study, we cooperated with the Kaohsiung City Police Department to obtain a dataset of forged seal images (

Table 1. Dataset of forged seal cases from police database.

Dataset	Number of Seal Group	Total Seal Image	Comparison Pairs
			Number of the Same Seal Pair	Number of Different Seal Pair
Official seal	12	176	2554	12,846
Company seal	72	360	707	42,364
Total	84	536	3261	55,210

). The police department provides a dataset of 536 forged seal images, including 12 forged official seals, totaling 176 images.

According to the database design established by the police department, each forged seal may appear on different forged documents and be regarded as a different image of the same seal group (Figure 7). Therefore, each forged seal is a group, and the various document images using this forged seal are different images of the same group. In addition to the official seals, we also obtained 72 forged company seals, corresponding to a total of 360 images, as shown in Table 1. In the experiments, we applied different methods to compare any two seal images within the same seal group (i.e., using the same forged seal but appearing in different forged documents), referred to as the Same Seal Pair, totaling 3261 pairs. On the contrary, the comparison between different forged seals (different groups), called the Different Seal Pair, totals 55,210 pairs.

3.2. Comparison

F1-score is adopted to evaluate the performance of our methods by computing the key metrics, including true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). It captures the system’s ability to identify both authentic and forged seals, providing a fair and robust performance assessment defined as:

$$F1-Score={\displaystyle \frac{2}{{\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}^{-1}+{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}^{-1}}};$$

(24)

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{TP}{TP+FP}};$$

(25)

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{TP}{TP+FN}}.$$

(26)

Data preprocessing significantly improved performance for official seals, raising the F1-score from 24 to 90% (

Table 2. Results of using VGG16-based global feature matching only (Step 2) with and without Preprocessing (Step 1).

Dataset	Type of Comparison Pairs	F1-Score
		Without Preprocessing (Step 2 Only)	With Preprocessing (Step 1 + 2)
Official seal	Same seal	24%	90%
	Different seal	100%	98%
Company seal	Same seal	0%	0%
	Different seal	100%	100%

). However, even with preprocessing, company seals remained challenging due to complex patterns, revealing the limitations of VGG16 global features. However, since only global features need to be compared, the VGG16-based method still has the advantage of lower execution time. In our case, comparing all seal pairs only takes a few seconds.

To address the limitations of low-level features, a SuperGlue-based method is employed for local feature matching. The SuperGlue-based approach outperforms the VGG16-based method for both the official and company seals (

Table 3. Results of using SuperGlue-based Local Feature Matching only with Preprocessing (Step 1 + Step 3).

Dataset	Type of Comparison Pairs	Threshold Setting	F1-Score
Official seal	Same seal	150	97%
		200	98%
	Different seal	150	100%
		200	100%
Company seal	Same seal	150	92%
		200	88%
	Different seal	150	100%
		200	100%

). This improvement is attributed to SuperGlue’s ability to perform more detailed and accurate feature comparisons, thanks to its higher computational complexity. A crucial parameter in this process is the feature point threshold, which defines the minimum number of matching keypoints required to determine whether two seal images are identical. A key parameter called the feature point threshold defines the minimum number of matching keypoints required to determine whether two images are considered the same. A lower threshold allows more matches but may increase false positives, while a higher threshold results in stricter comparisons. By setting the threshold = 150, the F1-Score reached 97% for official and 92% for company seals, confirming the system’s effectiveness in identifying fine similarities.

By combining the advantages of the VGG16-based method and the SuperGlue-based method, we established a comparison system that is both highly efficient and highly accurate. When a picture of a suspected forged seal was input into the system, we performed data preprocessing, and then randomly selected a picture from each seal group to compare with the input picture using the VGG16-based method for global feature comparison. For example, the input picture was compared with the 84 pictures randomly selected from each seal group in Table 1, and the seal group that was closest to it was identified and selected. Then, the SuperGlue-based method was used to compare with all or W pictures in the selected Seal Group, and the closest H pictures were listed for police review. Users can change the values of W and H through system settings. Currently, W is set to the number of all pictures in the selected seal group, and H = 10. This system design balances speed and accuracy, demonstrating a significant performance improvement, as shown in

Table 4. Comparisons of different methods for the same seal pairs.

Method	F1-Score		Execution Time 1
	Official Seal	Company Seal
Step 1 + Step 2	90%	0%	Fast (in seconds)
Step 1 + Step 3	97%	92%	Slow (in minutes)
Step 1 + Step 2 + Step 3	97%	92%	Fast (in seconds)

¹ Execution on Intel i7-13700 CPU with one Nvidia 3080Ti GPU.

.

4. System Development

To support police operations, we developed an AI-based seal image recognition system (Figure 8). This system has a LINE interface that mimics handheld devices used by law enforcement. Officers can upload photos of suspected forged documents to the cloud for analysis. The cloud AI compares these images with the police database, identifies similar forged seals, computes similarity scores, and retrieves related archived cases. This process enables officers to quickly assess the authenticity of seals and generate a report for future investigations.

5. Conclusions

We developed an AI-based system for forged seal recognition and retrieval, combining preprocessing, global feature filtering, and local feature matching. The system achieved high accuracy, specifically 97% for official seals and 92% for company seals, demonstrating its strong effectiveness. By enabling real-time verification from a single photo, the system enables law enforcement to prevent fraud, reduce manual workload, and enhance investigation efficiency. It offers a practical tool for identifying counterfeit seals during on-site inspection and supports future deployment in actual police workflows.