Research on Multi-Path Feature Fusion Manchu Recognition Based on Swin Transformer

Abstract

Recognizing Manchu words can be challenging due to their complex character variations, subtle differences between similar characters, and homographic polysemy. Most studies rely on character segmentation techniques for character recognition or use convolutional neural networks (CNNs) to encode word images for word recognition. However, these methods can lead to segmentation errors or a loss of semantic information, which reduces the accuracy of word recognition. To address the limitations in the long-range dependency modeling of CNNs and enhance semantic coherence, we propose a hybrid architecture to fuse the spatial features of original images and spectral features. Specifically, we first leverage the Short-Time Fourier Transform (STFT) to preprocess the raw input images and thereby obtain their multi-view spectral features. Then, we leverage a primary CNN block and a pair of symmetric CNN blocks to construct a symmetric spectral enhancement module, which is used to encode the raw input features and the multi-view spectral features. Subsequently, we design a feature fusion module via Swin Transformer to fuse multi-view spectral embedding and thereby concat it with the raw input embedding. Finally, we leverage a Transformer decoder to obtain the target output. We conducted extensive experiments on Manchu words benchmark datasets to evaluate the effectiveness of our proposed framework. The experimental results demonstrated that our framework performs robustly in word recognition tasks and exhibits excellent generalization capabilities. Additionally, our model outperformed other baseline methods in multiple writing-style font-recognition tasks.

1. Introduction

Manchu, once an official language of China, has left an abundance of historical archives in both central and local institutions. The First Historical Archives of China alone preserves more than 200,000 Manchu documents. However, due to the critically endangered status of Manchu (UNESCO, 2009), few individuals remain proficient in the language. Consequently, the development and utilization of these archives lag significantly behind other historical records [1]. Therefore, the development of Manchu recognition systems is crucial for the exploitation of Manchu historical documents and the preservation of the Manchu cultural heritage.

The Manchu writing system shares similarities with Mongolian in its vertical writing style and ligature-based structure. Unlike the Latin and Cyrillic alphabets, where characters maintain consistent forms regardless of their position within a word, Manchu characters change based on their position and context. Some characters have different appearances depending on whether they appear at a word’s beginning, middle, or end, as shown in Figure 1. This position-dependent variation complicates the Manchu writing system. Additionally, the presence of polysemy and mandatory ligature further increases the difficulty of detecting and recognizing Manchu characters.

Manchu character recognition methods [2,3] can work effectively with smaller datasets. Most Manchu words feature a central trunk flanked by left and right strokes. Segmentation-based methods, such as the projection method and the important contour method [4], first identify the vertical axis of the trunk. They then perform horizontal segmentation. These approaches rely on carefully designed feature extraction, image processing, and pattern recognition. However, segmentation-based methods often struggle with accuracy and robustness. Challenges include stroke adhesion, character diversity, and polysemy. Manchu characters are much more complicated than English characters. There are a total of 236 nominal characters, presentation characters, and mandatory ligature [5], so it is difficult to include all the Manchu character usage rules in a data set consisting of only a few hundred words.

The fundamental premise of the Manchu word recognition method is to regard the Manchu word as a cohesive entity. These methods employ neural networks for feature extraction, significantly reducing the reliance on manual design. Convolutional neural networks (CNNs) are widely used for image classification, and some studies have approached the Manchu recognition problem as an image classification task. ResNet and CNNs with Spatial Pyramid Pooling have been employed for Manchu recognition, yielding higher recognition accuracy than segmentation-based methods [6,7]. Recognizing Manchu through image classification requires a large-scale dataset, and the training and validation sets must comprehensively cover the usage rules of Manchu characters to ensure robust generalization. Given that Manchu is a pinyin script, it is nearly impossible for the dataset to encompass all Manchu words. Additionally, while CNNs are effective in extracting local image features, such as strokes, shapes, and details, through convolutional processing, their localized convolutional kernels limit their ability to capture long-range contexts and global dependencies.

In contrast, the self-attention mechanism used in Transformers and the recursive architecture of Long Short-Term Memory Networks (LSTMs) excel at modeling long-range dependencies, demonstrating strong performance in managing long sequences and capturing semantic relationships. Researchers have successfully applied LSTMs and Transformer modules to text recognition tasks. For instance, the Transformer-based model TrOCR (Transformer-based Optical Character Recognition) has shown robust performance on the IC3 benchmark dataset, achieving state-of-the-art capabilities in text recognition [8]. However, the features extracted by the Transformer encoder can be influenced by high-frequency signals, which may reduce recognition accuracy for noisy images. Furthermore, the encoder may struggle to produce distinctive encodings for highly similar Manchu characters, limiting its ability to identify subtle differences.

We propose a multi-branch fusion framework based on the Swin Transformer to address the limitations of existing methods. The STFT is employed to convert Manchu word images into spectral features, facilitating the extraction of diverse information patterns. The overall architecture employs a three-branch parallel structure: two symmetric CNN blocks process horizontal and vertical spectral features, respectively, while a third CNN branch is dedicated to processing the original image. Subsequently, the outputs from the two spectral features branches are fused by a Swin Transformer, and the resulting features are then concatenated with those from the main CNN branch. This combined feature representation is finally fed into a Transformer decoder to achieve global feature integration and learn high-quality embeddings.

Our principal contributions can be summarized as follows.

• We propose a novel multi-path architecture based on the CNN and Transformer framework for Manchu recognition. The primary branch processes the raw image to extract spatial features, while two additional branches perform directional spectral analysis through horizontal and vertical STFT decomposition.
• We propose a cross-fusion module that integrates features from the three branches. This module effectively combines information across different domains.
• We created a dataset of single-font and multi-font Manchu words for use by other researchers (https://github.com/zy691/Manchu-Recognition (accessed on 24 August 2025)).

2. Related Work

2.1. Segmentation-Based Recognition Methods

Most Manchu characters resemble Mongolian characters, and their recognition methods are highly similar. Early research on Manchu recognition primarily relied on segmentation-based algorithms. The process involves splitting Manchu words into individual characters, recognizing those characters, and then reconstructing the full words from the sequences.

Traditional Manchu recognition techniques used handcrafted feature extraction, classical image processing, and pattern recognition. These methods included stroke classification by statistical patterns or segmentation-based recognition using image processing. Examples include the improved projection method [3,9], the multi-feature integration approach [10] (combining projection, chain code, endpoints, and intersections), and deep learning-based methods [11,12].

The segmentation-based recognition method relies heavily on precise segmentation, as any errors in this process can directly result in recognition failure. Furthermore, these methods identify characters by isolating them from Manchu words, which causes a loss of semantic information associated with the characters within the context of the words. Although these methods have achieved some success, their accuracy and robustness are more susceptible to the complexities of Manchu characters, including issues like stroke stacking, font variation, and semantic nuances.

2.2. Segmentation-Free Recognition Methods

To overcome the limitations of segmentation-based recognition methods, researchers are increasingly turning to neural networks to recognize entire words directly. The main idea behind these approaches is to treat each Manchu word as a single, inseparable entity, which redefines the text recognition challenge as a large-scale image classification problem [4]. For thousands of Manchu-word pure images, traditional CNN models, such as VGG, ResNet, and GoogleNet, can achieve classification results with an accuracy rate of over 99% [7]. However, their effectiveness significantly decreases when dealing with degraded images, multi-font images, or images that were not included in their training data. Due to the unique challenges associated with Manchu recognition, researchers have proposed several improved models to enhance performance. Li et al. implemented a spatial pyramid pooling layer in place of the last max-pooling layer in a CNN to recognize the arbitrary size of Manchu words and improve the recognition accuracy to 97.68% [6]. Zhang et al. employed CNN to recognize an image covered by a sliding window to identify Manchu words, with the recognition accuracy reaching 98.84% on the test set of 671 Manchu words [13]. Zheng et al. implemented a nine-layer CNN to recognize multi-font Manchu words, achieving accuracy rates of 88% to 95% across different fonts [14].

The Manchu language is written in a pinyin script [15], which makes it challenging for traditional word recognition methods based on classification to accommodate all Manchu words. Consequently, recent studies have shifted their focus toward developing word recognition methods that utilize sequence models and attention mechanisms [16]. Ren et al. proposed an EGA model that incorporates ECA-Net, a BiGRU encoder with an attention mechanism, and a BiLSTM decoder. By utilizing data augmentation techniques during training, EGA achieved a handwriting recognition accuracy of 89.33% for Mongolian [17]. Cui et al. proposed a TAMN model, which utilizes a spatial transformation network to correct deformation, employs gated recursive convolutional layers combined with triplet attention to extract features, and utilizes MoGrafter LSTM with attention decoding for sequence prediction, achieving a recognition accuracy of 90.30% for Mongolian deformed images [18]. Zhang et al. proposed a segmentation-free method that treats Mongolian word recognition as a sequence-to-sequence mapping problem, utilizing an attention-based LSTM model to directly convert images into text letters, with an average recognition accuracy of 89.3% across four different printed fonts in Mongolian [19]. Sun et al. introduced a method that processes word images using a CNN feature extractor and feeds the features to a Transformer for prediction, achieving an average recognition accuracy of 94.57% for four different printed Mongolian fonts [20]. Recognizing handwritten, deformed, or multi-font words presents challenges. In these cases, sequence-based LSTMs or Transformers provide superior contextual understanding and flexibility.

3. Methodology

Figure 2 shows the overall architecture of our proposed framework. The preprocessing module begins by applying a STFT to the original Manchu image to extract its spectral features along the horizontal and vertical axes. These features are then fed into a symmetric spectral encoding enhancement module, which processes them through two separate spectral feature encoders. Concurrently, a main encoder processes the original Manchu image as the third branch. The features output from the two spectral feature encoders are subsequently fused using a Swin Transformer block. Finally, this fused result is concatenated with the features from the main encoder to generate a decoding vector, which is then processed by a Transformer decoder to output the final character position encoding.

3.1. Preprocessing Module

A Manchu word image, denoted as I, is a matrix of size $H\times W$, $I\in {\mathbb{R}}^{H\times W}$. For the non-standard raw image of a Manchu word, it is first resized to the same width and then filled to a uniform height.

$$I=\mathrm{Resize}(I_{\mathrm{raw}},H\times W)$$

(1)

Resize: scales (expands/reduces) and fills the input image to the desired dimensions.

The cursive nature of the Manchu script, characterized by heavily interconnected strokes, poses a significant challenge for character segmentation. To address this, we propose the use of the Short-Time Fourier Transform (STFT). STFT is adept at extracting localized spectral features from continuous signals, which can effectively represent the subtle variations and transitions within the Manchu strokes, thereby providing a robust basis for their differentiation and recognition.

To perform the Short-Time Fourier Transform on Manchu word images, the images are initially converted into one-dimensional sequences, either in rows or columns, as shown in Figure 3. Let $w\left(n\right)=w(-n)$ be a real-symmetric preprocessing window function. $X(f,\tau )$ represents the Short-Time Fourier Transform coefficient of a one-dimensional sequence $x\left(n\right)$ at the time frame $\tau$, with multiple analyzed frequencies, where the total number of frequencies is L.

$$X(f,\tau )=\sum _{n=0}^{L-1}x\left(n\right)·x(n-\tau )·e^{-j2\pi fn/L}$$

(2)

where $\tau ={\tau }_0,{\tau }_1,{\tau }_2,\dots ,{\tau }_{M-1}$ are the starting positions of each sliding window, M is the maximum number of sliding windows, and f is a multiple of $1/L$.

The Short-Time Fourier Transform of a word image, denoted as $I_T$, is defined as the modulus of the complex number.

$$I_T=\left|X(f,\tau )\right|$$

(3)

$I_T$ is a matrix of size $H_T\times W_T$, $I_T\in {\mathbb{R}}^{L\times M}$. The two spectral features are denote as $I_{HT}$ and $I_{VT}$, respectively.

3.2. Symmetric Spectral Encoding Enhancement Module

The performance of image classification can be enhanced by employing various subnetworks to extract spatial and spectral features, which are then fused together [21,22,23,24]. Inspired by this approach, this paper utilizes multiple subnetworks to encode image and spectral features separately. Given that some Manchu characters are symmetrical either vertically or horizontally, the spectral features in these directions alone cannot effectively distinguish between the symmetrical characters. Therefore, this paper improves feature discrimination capability by integrating spectral features from two directions using a symmetric spectral encoding enhancement module.

Figure 4 shows a framework that uses CNNs as encoders. The inputs for the three encoders are I, $I_{HT}$, and $I_{VT}$, respectively. The outputs from the main encoder and the symmetric spectral encoding enhancement module, which are O, $O_{VT}$, and $O_{HT}$, respectively, are all sequences. Specifically, $O\in {\mathbb{R}}^{1\times N\times C}$, $O_{VT}\in {\mathbb{R}}^{1\times N\times C_{VT}}$, $O_{HT}\in {\mathbb{R}}^{1\times N\times C_{HT}}$, where N is the sequence length; C, $C_{VT}$, and $C_{HT}$ are the channel numbers of each respective sequence.

Each encoder contains seven consecutive CNN modules $(i=1,\dots ,7)$ with specialized operations: For the spatial branch $(j=1)$:

$${\varphi }_i^1=\left\{\begin{array}{cc}\mathrm{Conv}{3\times 3}_{\mathrm{stride}(1,1)}\left(I\right)\hfill & \mathrm{if}i=1\hfill \\ \mathrm{ReLU}\left(\mathrm{Conv}3\times 3\left(\mathrm{MaxPool}{2\times 1}_{\mathrm{stride}(2,1)}\left({\varphi }_{i-1}^1\right)\right)\right)\hfill & \mathrm{if}i=2\hfill \\ \mathrm{Conv}3\times 3\left(\mathrm{BatchNorm}2\mathrm{d}\left(\mathrm{ReLU}\left({\varphi }_{i-1}^1\right)\right)\right)\hfill & \mathrm{if}i>2\mathrm{and}i\mathrm{is}\mathrm{odd}\hfill \\ \mathrm{BatchNorm}2\mathrm{d}\left(\mathrm{Conv}3\times 3\left({\varphi }_{i-1}^1\right)\right)\hfill & \mathrm{if}i>2\mathrm{and}i\mathrm{is}\mathrm{even}\hfill \end{array}\right.$$

(4)

For the spectral branches $(j=2,3)$:

$${\varphi }_i^j=\left\{\begin{array}{cc}\mathrm{Conv}{3\times 3}_{\mathrm{stride}(2,2)}(I_{HT}/I_{VT})\hfill & \mathrm{if}i=1\hfill \\ \mathrm{ReLU}\left(\mathrm{Conv}3\times 3\left(\mathrm{MaxPool}{2\times 1}_{\mathrm{stride}(2,1)}\left({\varphi }_{i-1}^j\right)\right)\right)\hfill & \mathrm{if}i=2\hfill \\ \mathrm{Conv}3\times 3\left(\mathrm{BatchNorm}2\mathrm{d}\left(\mathrm{ReLU}\left({\varphi }_{i-1}^j\right)\right)\right)\hfill & \mathrm{if}i>2\mathrm{and}i\mathrm{is}\mathrm{odd}\hfill \\ \mathrm{BatchNorm}2\mathrm{d}\left(\mathrm{Conv}3\times 3\left({\varphi }_{i-1}^j\right)\right)\hfill & \mathrm{if}i>2\mathrm{and}i\mathrm{is}\mathrm{even}\hfill \end{array}\right.$$

(5)

The architecture displays three key distinctions between branches: (1) The spatial branch $(j=1)$ maintains full resolution, while the spectral branches $(j=2,3)$ utilize stride(2,2) downsampling in their initial layer. (2) All branches alternate between BatchNorm-after-Conv and Conv-after-BatchNorm patterns for odd/even modules. (3) The second module universally incorporates MaxPool2x1 with stride(2,1) for dimension reduction while preserving the horizontal structure critical for Manchu’s vertical writing system. This design simultaneously captures spatial textures and multi-view spectral features via specialized pathway configurations.

3.3. Feature Fusion Module

The Swin Transformer [25] has demonstrated remarkable success in various computer vision tasks [26,27], particularly in handling high-resolution images and multi-scale features [28,29]. Inspired by its effectiveness, this study employs the Swin Transformer for feature fusion. This choice is motivated by the unique characteristics of the Manchu script, where the shape of a single letter is heavily influenced by its adjacent letters, introducing significant long-range dependencies. Conventional adaptive feature fusion methods, with their limited receptive fields, often struggle to capture such complex contextual relationships and are susceptible to interference when integrating spatial and spectral features. The self-attention mechanism within the Swin Transformer, however, is specifically designed to model these long-range dependencies, making it an ideal solution to overcome the aforementioned limitations.

The core component of the feature fusion module is a Swin Transformer block. The Swin Transformer block consists of two key modules: the Window Multi-head Self-Attention Mechanism (W-MSA) and the Shifted Window Multi-head Self-Attention Mechanism (SW-MSA). The input to this block is a fused feature, denoted as $F_I$, which is generated by fusing $O_{VT}$ and $O_{HT}$ from the symmetric spectral encoding module.

Denote the result of the patch embedding of $F_I$ as $X_{\mathrm{strip}}$, $X_{\mathrm{strip}}\in {\mathbb{R}}^{N\times (C_{VT}+C_{HT})}$. $X_{\mathrm{strip}}$ is divided into non-overlapping $M\times W$ windows, with each window $X_{\mathrm{win}}\in {\mathbb{R}}^{M\times D}$ processed by windowed multi-head self-attention:

$$X_{\mathrm{win}}^{\prime }=\mathrm{LayerNorm}\left(X_{\mathrm{win}}\right)+\mathrm{W}-\mathrm{MSA}\left(\mathrm{LayerNorm}\left(X_{\mathrm{win}}\right)\right)$$

(6)

where the window attention mechanism is computed as follows:

$$W-\mathrm{MSA}\left(X_{\mathrm{win}}\right)=\mathrm{MultiHead}(X_{\mathrm{win}}^Q,X_{\mathrm{win}}^K,X_{\mathrm{win}}^V)$$

(7)

The multi-head attention operation combines h attention heads through

$$\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}({\mathrm{head}}_1,\dots ,{\mathrm{head}}_h)W^O$$

(8)

With each attention head calculated as follows:

$${\mathrm{head}}_i=\mathrm{Softmax}\left({\displaystyle \frac{Q{W}_i^Q{\left(K{W}_i^K\right)}^T}{\sqrt{d_k}}}\right)\left(V{W}_i^V\right)$$

(9)

where $W_i^Q$, $W_i^K$$\in {\mathbb{R}}^{D\times d_k}$, and $W_i^V$$\in {\mathbb{R}}^{D\times d_v}$ project queries, keys, values for head i, $d_k={\displaystyle \frac{D}{h}}$ maintains constant compute across heads, and $W^O$$\in {\mathbb{R}}^{h{d}_v\times D}$ combines head outputs.

Alternating layers apply a half-window shift ($⌊M/2⌋$ pixels vertically) to capture cross-window dependencies:

$$X_{\mathrm{shifted}}=\mathrm{Roll}(X_{\mathrm{win}},⌊M/2⌋,\mathrm{axis}=0)$$

(10)

Processed by shifted window attention:

$$\mathrm{SW}-\mathrm{MSA}\left(X_{\mathrm{shifted}}\right)=\mathrm{MultiHead}(X_{\mathrm{shifted}}{W}^Q,X_{\mathrm{shifted}}{W}^K,X_{\mathrm{shifted}}{W}^V)$$

(11)

Each Swin Transformer block sequentially applies

$$X^{\prime }=\mathrm{LayerNorm}\left(X\right)+\mathrm{W}-\mathrm{MSA}\left(\mathrm{LayerNorm}\left(X\right)\right)$$

(12)

$$X^{\prime \prime }=\mathrm{LayerNorm}\left(X^{\prime }\right)+\mathrm{MLP}\left(\mathrm{LayerNorm}\left(X^{\prime }\right)\right)$$

(13)

The output of the Swin Transformer block $F_O$ is a sequence, $F_O\in {\mathbb{R}}^{N\times {\displaystyle \frac{C_{VT}+C_{HT}}{2}}}$. This output ($F_O$) is then concatenated with the main encoder’s output to form the final production, $D_I$, $D_I\in {\mathbb{R}}^{N\times ({\displaystyle \frac{C_{VT}+C_{HT}}{2}}+C)}$, which serves as the input for the decoder.

3.4. Transformer Decoder Module

The Transformer decoder has powerful sequence modeling capabilities [30,31]. The fused features from the Swin Transformer module are processed through a 6-layer Transformer decoder architecture. We made a critical modification to the standard decoder by removing the positional encoding entirely. This design choice is motivated by the unique characteristics of the Manchu script. The script lacks positional invariance—identical characters exhibit different morphological forms depending on their position within a word (initial, medial, or final positions). Conventional positional encoding may increase recognition error rates for Manchu text. The decoder processes the Transformer’s fused features through

$$\mathrm{Decoder}\left(X\right)={\mathrm{LayerNorm}}_6\left({\mathrm{FFN}}_6\left({\mathrm{MHA}}_6(\dots {\mathrm{LayerNorm}}_1\left({\mathrm{MHA}}_1\left(X\right)\right)\dots )\right)\right)$$

(14)

where each of these layers contains multi-head self-attention (MHA), a feed-forward network (FFN), and layer normalization (LayerNorm).

4. Experiment

4.1. Manchu Datasets

This study used two datasets for model evaluation.

Dataset 1: A single-font dataset containing 130,917 words in the Abkai font [32].

Dataset 2: A multi-font dataset comprising 1570 words from 11 different fonts. We used the KaikKi Manchu Dictionary, which includes 2395 Manchu words and phrases [7], and removed the phrases to generate the second dataset from the remaining words using a Python2.0 program. The twelve fonts used in both datasets are displayed in Figure 5.

Our dataset is composed of Dataset 1 and Dataset 2, which together contain a total of 12 fonts and 148,187 Manchu images.

We randomly selected 90,000 Manchu words from the single-font dataset and divided them into a training set of 50,000 words, including all words with a character length of less than 4, and a test set of 40,000 words. Both sets contain the same collection of Manchu characters as described in [5]. Manchu characters are written in three different formats: initial, medial, and final, depending on their position in the word. The distribution of nominal characters and mandatory ligatures at the beginning, middle, and end of the word is illustrated in Figure 6.

Despite the differences between the test and training sets, Figure 6 illustrates that the character distribution is similar. This suggests that both datasets adhere to the same character usage rules. Furthermore, the 130,917-word dataset reflects character usage rules that align with the training and test sets.

From the multi-font dataset comprising 17,270 images, we randomly selected approximately 14,000 images (around 80%) as the training set, with the remainder used as the test set.

4.2. Data Enhancements

To improve model resilience to document deformations and quality variations, we implemented two augmentation techniques:

Random Rotation enhances robustness to skewed text by applying a randomly selected angle $\theta$ constrained to $(-5^{\circ },5^{\circ })$ to each pixel $(x,y)$ in Manchu images, thereby ensuring characters remain intact and simulating real-world applications.

$$\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right]=\left[\begin{array}{cc}cos\theta & -sin\theta \\ sin\theta & cos\theta \end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]$$

(15)

where $(x,y)$ are original coordinates and $(x^{\prime },y^{\prime })$ are transformed coordinates. Bilinear interpolation computes the rotated pixel value $I(x^{\prime },y^{\prime })$ using four neighboring points:

$$I(x^{\prime },y^{\prime })=\sum _{i=1}^2\sum _{j=1}^{2}I(x_i,y_i)w_i{w}_j$$

(16)

where $(x_0,y_0)$, $(x_0,y_1)$, $(x_1,y_0)$, and $(x_1,y_1)$ are the values of the four neighbors of the target point $(x^{\prime },y^{\prime })$. We calculate the value of the target point.

The weights $w_i$ and $w_j$ are calculated based on the horizontal and vertical distances between $(x^{\prime },y^{\prime })$ and its neighbors. This two-stage interpolation first processes horizontal then vertical axes:

$$I_0\left(x^{\prime }\right)=I(x_0,y_0)·w_1+I(x_1,y_0)·w_2$$

(17)

$$I_1\left(x^{\prime }\right)=I(x_0,y_1)·w_1+I(x_1,y_1)·w_2$$

(18)

$$I(x^{\prime },y^{\prime })=I_0\left(x^{\prime }\right)·w_3+I_1\left(x^{\prime }\right)·w_4$$

(19)

Gaussian Blur addresses quality degradation from lighting and document wear. The Gaussian kernel $G(x,y)$ with standard deviation $\sigma$ is

$$G(x,y)={\displaystyle \frac{1}{2\pi {\sigma }^2}}{e}^{-{\displaystyle \frac{x^2+y^2}{2{\sigma }^2}}}$$

(20)

Convolution with a $(2k+1)\times (2k+1)$ kernel implements the blur:

$$I_{\mathrm{blurred}}(x,y)=\sum _{i=-k}^k\sum _{j=-k}^{k}G(i,j)·I(x+i,y+j)$$

(21)

where $G(i,j)$ are kernel weights and $I(x+i,y+j)$ are neighboring pixel values. This preserves stroke boundaries while reducing noise, which is particularly beneficial for historical documents.

4.3. Implementation Details

In our experiment, a Manchu word is represented by $X=[x_1,x_2,\dots ,x_i,\dots ,x_n]$, where $x_i$ denotes the ith character. When X is input to the Transformer decoder, we encode it as

$$X_{\mathrm{input}}=[SOS,O\left(x_1\right),O\left(x_2\right),\dots ,O\left(x_i\right),\dots ,O\left(x_n\right),EOS,0_{n+2},\dots ,0_{N-1}]$$

(22)

where $SOS$ denotes the start token vector, $O\left(x_i\right)$ denotes the 32-float one-hot vector of $x_i$, $EOS$ denotes the end token vector, $0_{n+2}$ denotes the $n+2$th zero vector, and N denotes the maximum word length.

The output from the Transformer decoder is encoded as follows:

$$X_{\mathrm{output}}=[O\left(x_1\right),O\left(x_2\right),\dots ,O\left(x_i\right),\dots ,O\left(x_n\right),EOS,0_{n+2},\dots ,0_N]$$

(23)

The list of characters used in the experiment is shown in

Table 1. Character index.

Character	Index	Character	Index	Character	Index	Character	Index
PAD	0	SPACE	1	SOS	2	EOS	3
A	4	I	5	E	6	U	7
G	8	N	9	H	10	B	11
K	12	L	13	R	14	O	15
M	16	T	17	Ū	18	S	19
D	20	C	21	J	22	Y	23
F	24	Š	25	W	26	P	27
Z	28	’	29	Ž	30	-	31

.

The Manchu words in the experimental dataset varied in length from 1 to 22 characters, with an average length of 10.19 characters. Additionally, 95% of the words fell between 5 and 15 characters. Given that the longest word in the dataset is 22 characters, we set the maximum length, N, to 24. The predicted output $\widehat{Y}$ is a 24 × 32 matrix that is computed by the Softmax function.

$${\widehat{Y}}_p=\mathrm{Softmax}\left({\widehat{Y}}^{\left(p\right)}\right)={\left[{\displaystyle \frac{e^{{\widehat{y}}_i^{\left(p\right)}}}{{\sum }_{j=1}^{32}{e}^{{\widehat{y}}_j^{\left(p\right)}}}}\right]}_{i=1}^{32}\mathrm{for}p\in \{1,\dots ,24\}$$

(24)

where ${\widehat{Y}}_p$ is a vector of the predicted probability for the pth character, and ${\widehat{y}}_i^{\left(p\right)}$ is the element in row p, column i of $\widehat{Y}$. The denominator ${\sum }_{j=1}^{32}{e}^{{\widehat{y}}_j^{\left(p\right)}}$ is the sum of the exponentiations of all elements in the pth row of $\widehat{Y}$, ensuring that the total sum of the probability is 1.

To align with our model’s architectural design, which inherently integrates a Softmax layer into the Transformer decoder’s output, and to prevent redundant computation, we adopt the Negative Log-Likelihood Loss (NLLLoss) as the optimization objective. The loss function is formulated as follows:

$$\mathrm{Loss}=-{\displaystyle \frac{1}{24}}\sum _{p=1}^{24}log\left({\widehat{Y}}_{p,Y_{\mathrm{target},p}}\right)$$

(25)

where ${\widehat{Y}}_{p,Y_{\mathrm{target},p}}\in {\mathbb{R}}^{24\times 32}$ is the predicted probability of the pth character on the target category.

In our experiment, we adjusted the size of the Manchu word images to 64 × 384 and set the window size to 256, the hop size to 64 in STFT, C to 256, $C_{HT}$ to 128, and $C_{VT}$ to 128. We implemented our model using PyTorch2.0 and trained the network with the Adam optimizer. The training parameters are as follows: the batch size is 8, the learning rate is $3\times {10}^{-5}$, the learning rate decay factor is 0.9, and the step size is 5. We trained the model for 60 epochs when using CNNs as the encoder and for 100 epochs when using Swin Transformers as the encoder.

4.4. Evaluation Index

The primary evaluation metric is recognition accuracy, which measures the proportion of correctly identified Manchu words relative to the total number of tested words. TP refers to the count of Manchu words that were accurately recognized. FP represents the count of Manchu words that were incorrectly identified as matches. The recognition accuracy is calculated using the following formula:

$$\mathrm{Accuracy}={\displaystyle \frac{TP}{TP+FP}}$$

(26)

The computational efficiency of the proposed model was assessed using two primary metrics: the average inference time per Manchu word ($T_{\mathrm{W}}$) and the average inference time per Manchu character ($T_{\mathrm{C}}$). These metrics are formally defined by the following equations:

$$\left\{\begin{array}{c}\hfill T_{\mathrm{W}}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{T}_i\\ \hfill T_{\mathrm{C}}={\displaystyle \frac{{\sum }_{i=1}^N{T}_i}{{\sum }_{i=1}^N{L}_i}}\end{array}\right.$$

(27)

where N is the total number of words, $T_i$ is the processing time for the i-th word, and $L_i$ is the total number of characters of the i-th word.

4.5. Ablation Study

We performed various ablation studies to show the effectiveness of the encoder and decoder in the model. Each alternative network was trained using the same parameters, and the results are shown in Figure 7.

Encoder: We compared the performance of the 3-path Swin Transformer with that of the 3-path CNNs used as encoders. Both methods first converted the three data types into one-dimensional sequences of 192 and then utilized the Swin Transformer block for data fusion. There is only a slight difference in recognition accuracy between the two approaches on both the test and training datasets. When the data is serialized into one-dimensional formats, the Swin Transformer and CNNs can produce similar results.

Path number: We compared the performance of the 3-path Swin Transformer with the standard Swin Transformer, the 2-path Swin Transformer, the 1-path CNNs, and the 3-path CNNs as encoders. The 3-path encoders achieved a higher recognition accuracy on the test dataset. This advantage was particularly pronounced when the test dataset consisted of degraded images. This improvement can be attributed to the richer feature data extracted from the original images, combined with the STFT data processed in both directions.

Fusion: We compared convolutional blocks and Swin Transformer blocks for data fusion in three-path CNNs. The Swin Transformer blocks exhibited notably higher recognition accuracy, particularly with degraded images, due to their effectiveness in modeling long-range dependencies.

Sequences generation methods: We tested two methods for generating sequences in Swin Transformer. The one-dimensional PatchMerging operation is more robust to degraded images than the two-dimensional PatchMerging operation.

Decoder: We compared the performance of the two decoders. The Transformer decoder outperformed the MLP in terms of recognition accuracy and overall convergence.

4.6. Hyperparameter Analysis

We tested the impact of key hyperparameters:

STFT Window Size:

Table 2. Hyperparameter study results of the proposed model on Dataset 2 (* denotes degraded image).

Experiment	Parameters	Dataset	Accuracy
STFT	n_fft = 2 ×W, hop_length = W	test	99.75%
			99.69% *
		train	99.99%
	n_fft = 4 × W, hop_length = W	test	99.75%
			99.69% *
		train	99.98%
	n_fft = 2 × H, hop_length = H	test	99.54%
			99.33% *
		train	99.97%
Decoder Feature Length	512	test	99.81%
			99.71% *
		train	99.98%
	384	test	99.81%
			99.71 % *
		train	99.97%

indicates that n_fft = 2 × W and hop_length = W can achieve a higher recognition accuracy, which is less computationally expensive than n_fft = 4 × W, and the model uses n_fft = 2 × W and hop_length = W to compute STFT.

Decoder Feature Length: We tested two different decoder feature lengths, and there was only a slight difference in the accuracy of the Transformer decoder for Manchu word recognition at the decoder feature length of 384 and 512.

4.7. Results Analysis

The experimental results are compared with existing Manchu word recognition methods, as shown in

Table 3. Comparison of the proposed model with other methods (* denotes degraded image).

Word Count	Font Count	Test Dataset	Method	Accuracy
—	—	500	Fusion Decision [9]	87.4%
11,252	1	9993	Attention-based CRNN [33]	97.44%
671	1	671,000	Deep CNN [13]	98.87% *
671	1	671,000	Ours	99.55%
14,850	—	12,919	SSC3 [34]	97.16%
1423	4	5237	CNN+Transformer [20]	94.57%
200,000	4	20,000	Seq2seq with Attention [14]	96%
1570	11	3269	Ours	99.75%
1570	11	3269	Ours	99.69% *
130,917	1	40,000	Ours	99.81%
130,917	1	40,000	Ours	99.71% *
130,917	1	130,917	Ours	99.85%

. We trained the model using both single-font and multi-font training datasets. The recognition accuracy on the single-font test dataset was 99.81%, while the accuracy for the test dataset composed of degraded images was 99.71%. When the test dataset was expanded to include 130,917-word samples, the recognition accuracy for degraded images slightly improved to 99.72%. On the multi-font test dataset, the recognition accuracy was 99.75%, with the recognition accuracy for degraded images being 99.69%. These experimental results demonstrate that the model has strong generalization capabilities. However, our model’s inference time is relatively long, with $T_W$ being 0.1753 s and $T_C$ being 0.0306 s.

The word recognition accuracy of the proposed model is 2.37 percentage points higher than that of the attention-based CRNN method, which has an accuracy of 97.44%, and 0.95 percentage points higher than that of Deep CNN, with an accuracy of 98.87% (using only 671 words). For multi-font recognition, the word recognition accuracy of the proposed model is 4.0 percentage points higher than that of the CNN Transformers model. This demonstrates the robustness of our model in recognizing Manchu words, particularly its stable recognition ability when faced with degraded images and varying writing styles.

The superior performance of our model is attributed to the effective integration of Swin Transformer’s contextual modeling and STFT-based frequency analysis within our multi-branch design. This approach has proven particularly effective in addressing challenges such as positional allomerism and ligature formation, which often hinder traditional recognition systems. Overall, our advancements establish this method as a state-of-the-art solution for archival document processing and real-world Manchu text recognition applications.

5. Discussion

This study shows that the Transformer decoder is suitable for Manchu recognition in extracting semantic features. After training with tens of thousands of words, the architecture combining the encoder and the Transformer decoder yields impressive recognition results. There is only a minor recognition accuracy difference when using CNNs, Swin Transformer, or multi-path Swin Transformer as encoders for clean Manchu word images. For degraded images, the multi-path Swin Transformer or CNN encoder achieves better results. Previous studies have used small datasets or clean images for training, while this study utilized a large-scale collection of degraded images. The experimental results demonstrate that the proposed model achieves outstanding word recognition accuracy for degraded images and multi-font Manchu words.

Table 4. Cross domain accuracy of the proposed model.

Font	BiaoHei	GuFeng	LiuYe	ShuKai	WenJian	WenQin
Pure image	95.30%	92.26%	64.28%	60.94%	99.94%	81.67%
Degraded image	98.79%	91.94%	61.51%	57.13%	99.75%	80.47%
Font	XingShu	YaBai	YingBi	ZhengBai	ZhengHei	Abkai
Pure image	91.69%	99.87%	69.94%	98.22%	99.94%	81.61%
Degraded image	88.08%	99.94%	72.23%	93.97%	99.30%	75.77%

presents the cross-domain recognition results for various fonts. In our multi-font dataset, we selected 10 fonts of Manchu word images for training to recognize Manchu words that were not included in the training. As shown in the table, the model achieved a higher cross-domain recognition accuracy with Manchu text fonts that have a typographic style. In contrast, the accuracy for handwriting-style fonts remained relatively low. This indicates that the model, which was trained using a general training method, has limited capabilities for causal inference.

A comparison of recognition errors between CNN and three-path CNN encoders on the single-font test dataset is shown in Figure 8. This paper categorizes the types of errors into five distinct categories:

1. Error Type 1: Duplicate Structural Errors. These errors occur when duplicate characters are incorrectly merged or omitted.

2. Error Type 2: Single-Character Recognition Errors. These refer to mistakes in accurately recognizing individual characters.

3. Error Type 3: Semantic Errors. These errors occur when the recognized word is reasonable in some contexts but incorrect in the specific context; for example, “NA” may be recognized as “A”.

4. Error Type 4: Translation Invariance Errors. These errors arise from the translation invariance characteristic of CNNs.

5. Error Type 5: Other Errors. This category includes errors not covered by the previous types.

The results indicate that the three-path CNN encoder effectively reduces errors associated with translational invariance, addressing the inherent challenges posed by this issue in CNNs. Additionally, the model significantly minimizes recognition errors caused by repetitive structures, demonstrating an enhanced ability to handle the unique repetitive pattern structures in Manchu. However, semantic errors continue to present a major challenge. This suggests that while improvements have been made in structural processing, contextual constraints still lead to semantic errors.

The preservation of historical Manchu documents is challenged by time-induced damage, such as image degradation, creasing, and curling. These physical defects, compounded by the cursive nature of Manchu script, significantly hinder character recognition. While our current model demonstrates a baseline capability, its accuracy on these “extreme” cases needs substantial improvement. Future work will therefore focus on enhancing the model’s generalization to achieve higher recognition rates for damaged documents.

6. Conclusions

This paper presents a novel multi-path hyperarchitecture for recognizing Manchu script, which effectively combines spatial and spectral features. The proposed model employs three parallel paths: one path analyzes spatial features from the original image, while the other two paths process spectral features through dimensionally-specific STFT analysis. This innovative design allows for complementary feature extraction, where the spatial branch captures local structural patterns and the spectral branches model global textural characteristics. By integrating CNN and the Transformer architecture, the model gains significant advantages in modeling long-range dependencies inherent in Manchu’s vertically connected script. This approach effectively addresses challenges related to positional allography and ligature formations. Comprehensive evaluation across multiple dataset scales demonstrates the model’s robust performance, achieving state-of-the-art recognition accuracy while maintaining strong generalization capabilities. The parallel processing framework enhances feature representation for Manchu recognition, establishing a versatile approach that can be adapted to other complex writing systems with similar structural characteristics.