LLM for Japanese Text OCR: Automating Kuzushiji Recognition Using Decoder-Only OCR Architecture ^†

Abstract

This study presents an application of DTrOCR approach to Japanese text optical recognition, particularly addressing the possibility of processing sophisticated writing styles such as kuzushiji or handwritten calligraphy. This approach utilizes the recently developed method which applies decoder-only transformer to the process of optical character recognition (OCR). Specifically, the current research examines the adaption of this method to the task of Japanese character recognition under the constraints of limited training and computational resources. We experimented with training the model both with and without fine-tuning using the corpora with the synthesized printed texts as well as the texts from kuzushiji—traditional Japanese cursive writing. Our findings suggest that a fine-tuned DTrOCR-based approach is promising for kuzushiji automatic recognition and can retain its accuracy on other types of Japanese texts using cursive writing.

1. Background

OCR (optical character recognition) is one of the constituent domains in computer vision. OCR method application areas include digitizing archived documents and books, automating document workflows in accounting systems, translating texts in photographs and images, as well as integration in language learning applications.

Neural networks and machine learning algorithms have significantly transformed the landscape of text recognition, especially for languages utilizing a large number of complex symbols, varying in form and writing styles, for instance, Japanese and Chinese.

Most of the known traditional OCR methods are based on autoencoder architecture [1,2,3], some leveraging a pretrained language model as a decoder [4]. LLM-based decoders ensure more accurate predictions, which leverage not only the visual features of particular characters but also the context derived from the previously predicted words and characters. Fujitake suggested the DTrOCR (Decoder-only Transformer for Optical Character Recognition) method [5], where the model employs a pretrained LLM as a decoder. The method has fewer parameters compared to many established traditional methods, while producing better accuracy [5].

This paper explores the potential of using DTrOCR for Japanese text recognition, specifically addressing two major use cases:

• Automatic printed text recognition based on a synthesized Japanese text corpus;
• Recognition of kuzushiji texts.

2. Introduction to Kuzushiji

Kuzushiji (literally meaning “deformed characters”) is a highly cursive Japanese writing style used in many Japanese historical documents for over a thousand years prior to 1900 (see for example, [6]). Figure 1 shows an example of an Edo-period illustrated book printed in ukiyo-e, the style of Japanese woodblock prints and paintings that emerged and developed mostly during the Edo period (1603–1868) but still has a significant impact on modern and contemporary arts and cultural studies [7,8], where kuzushiji writing can often be seen. The illustrated explanations about the technology of making such books can be found in [9], pp. 26–27.

Different characters can be distorted in kuzushiji by varying degrees. Figure 2 shows the cover and fragment from the typical dictionary illustrating such distortions. Interestingly, more frequently used characters can be even more deformed compared to others [10]. This imposes significant challenges in reading and understanding such documents. Kuzushiji is considered so challenging to learn that students enrolled in programs on Japanese history and philology often take specialized advanced courses to learn how to read kuzushiji [11,12].

3. Related Work

3.1. OCR for Japanese Texts

Numerous recent research works on Japanese OCR pertain to very specific tasks [14,15,16] varying from the recognition of documents belonging to specific time periods [15] and old cursive writings [14] to constructing text recognition systems for rather unique use cases, such as the recognition of mathematical formulae alongside text [16]. Some of them use transformer-based decoders [15] and even utilize multimodal LLMs [17].

However, only a few existing approaches aim to create a text recognition model that would perform well on a variety of text types. Such models exist for languages like English and Chinese [4,5], but to the best of our knowledge, they have not been applied to Japanese text recognition.

The majority of high-quality data currently available for LLM training is in English and Chinese [18], while the data for common text types such as scene and handwritten text in Japanese is much more limited. That is why Japanese is often considered as one of the examples of so-called “impossible” languages [19]. Specifically for the purposes of OCR, the above-mentioned limitation can be overcome through the use of a methodology that would be highly specific to the task [15] or through the use of synthesized data [17] along with oversampling techniques [14].

3.2. Modern Multimodal OCR Methods

Most modern multimodal OCR approaches mainly feature an encoder–decoder structure. Before the introduction of transformers, most methods used a combination of CNNs and RNNs. The encoder aims to extract visual representations of text images, while the decoder maps the representations to text. In recent years, transformers have been leveraged for OCR tasks [1,4,20,21,22]. For example, TrOCR [4] uses a combination of a vision transformer model as an encoder and a transformer language model as a decoder. MaskOCR project delivers another example [22], where a pair of transformers are used as components of an encoder–decoder architecture. This project also employs masked modeling to train both the encoder and decoder. Both of the mentioned methods achieved state-of-the-art results at the time of their publication, showcasing the potential of transformer architecture for OCR.

3.3. DTrOCR Original Method

DTrOCR [5] is one of the most recent OCR methods. The key peculiarity of this method is that it does not utilize an encoder, while a pretrained language model serves as a decoder; the latter is additionally trained for the OCR task. According to the reported results, while keeping the training costs at the same level, such a retrained model significantly improves performance compared to training the decoder from scratch.

Particularly, according to the results reported in [5], the DTrOCR method outperforms existing state-of-the-art methods on various benchmarks for a selection of text recognition tasks, including scene, handwritten, and printed text recognition. It performs nearly equally well in recognizing both English and Chinese texts, suggesting the generality of the method.

4. Method

4.1. Original Architecture

As mentioned previously, the pretrained language model is trained only to process a language and possesses the language knowledge; however, that knowledge is not connected to the image features the decoder receives. To teach the model to catch the connections between known tokens and specific image features, additional training is performed.

Before being passed to the language model, the input image undergoes patch embedding. The patch embedding module converts an image into a sequence of small, fixed-size 1D vectors (patches) by dividing, flattening, and linearly projecting them. This enables transformers to process images as sequences of tokens and preserves the local structure while reducing computational complexity. Those sequences are then passed into the decoder with positional embeddings. After receiving a special token [SEP], the decoder predicts the first-word token. Then, it begins to predict subsequent tokens auto-regressively until [EOS]—another special token—is passed to it. The diagram in Figure 3 illustrates the overall workflow.

4.2. Model Pipeline

For the pretrained language model that serves as a decoder, similarly to [5], we use the Japanese-GPT2-small model from Rinna [23], the latter being a 12-layer, 768-hidden-size transformer-based language model [24] trained on the Japanese CC-100 dataset [25] and Japanese Wikipedia. This model is chosen due to its similarity in parameters to the GPT2 version used in DTrOCR project [5]. For patch embedding, we utilize ViT (Vision Transformer) [26] patch embedding. Both models are available on Hugging Face [27]—a leading present-day open-source AI platform and community.

4.3. Pretraining with Synthesized Datasets

For pretraining, 4,000,000 images were generated: half contain examples of Japanese texts in a modern layout organized in horizontal lines (yokogaki, in Japanese), while the remaining 2,000,000 samples contain text in a traditional Japanese writing style in vertical columns (tategaki). Text Renderer [28] was used for synthesizing this corpus. Figure 4 shows a few examples of generated images of both types. We used Japanese CC-100 [25] as a basis for synthesizing the data corpus because it was used to train the original GPT2-small language model [24]. Specifically, to ensure the model’s resistance to noise, we used 41 different Japanese fonts at font sizes ranging from 10 to 30, while the number of characters ranged between 1 and 8 in different colors.

Additionally, the original implementation of Text Renderer crops the generated images to the size of the text so that the text is centered and fills the entire image. We decided to disable that cropping with the expectation of improving the robustness of the final model and adjusting it to the different text placements in actual images without additional image processing.

We shuffled and split both the tategaki and yokogaki samples into training and testing sets in a proportion of 90% training data and 10% testing data. Finally, we applied data augmentation to all the synthesized dataset elements. For each image, we generated three additional augmented versions. We applied the RandAugment [29] three-layer policy and the magnitude of three, corresponding to low-intensity transformations. We also added random inversion with a probability of 0.2.

Thus, the overall training was performed using 15,200,000 synthesized images split into 475,000 batches, where each batch contained 32 images.

4.4. Fine-Tuning on the Kuzushiji-Kanji Dataset

Our primary interest is to test the method’s ability to be applied to difficult text types that pose serious challenges for reading, comprehension, and automatic recognition. Kuzushiji is a perfect fit. Improving methods for performing automatic recognition of kuzushiji writing is an important current scientific and technological task that has an impact on the development of OCR instruments, cultural studies, and education.

The dataset used in this study is represented by the Kuzushiji-Kanji (KKanji) dataset by ROIS-DS [30], an imbalanced dataset with a total of 3832 Kanji characters (64 × 64 grayscale, 140,424 images), ranging from 1766 examples to only a single example per class.

One of our research questions was to examine to what extent model performance is affected by additional fine-tuning and training after the initial training. For that purpose, we decided to test it with kuzushiji data. The fine-tuning was conducted following the original research methodology [5]. Before that, we had split the data into training and testing sets with the same ratio as the synthesized data. Similar to the training, all data was passed for fine-tuning in random order.

5. Results

The training and fine-tuning were performed on an RTX 3060. Training on synthesized data with augmentation took approximately 50 h, while fine-tuning on kuzushiji with augmentation took 2 h. We used accuracy as the main metric to assess the model’s performance. Additionally, we measured balanced accuracy on KKanji to better account for its imbalanced nature.

The tested scenarios for model training were as follows:

• Model A: Training only on the synthesized corpus;
• Model B: Model A + augmentation;
• Model C: Model B + fine-tuning on kuzushiji;
• Model D: Model C with extra augmentation;
• Model E: Training only on augmented kuzushiji;
• Model F: Model B + extra training on augmented kuzushiji.

Table 1. Influence of augmentation and fine-tuning on accuracy.

Model	Synthesized	Kuzushiji	Balanced Accuracy
A (synthesized)	78.195%	49.208%	49.231%
B (synthesized+augmented)	87.947%	50.703%	49.289%
C (model B + fine-tune)	86.978%	78.318%	63.547%
D (model C + augmentation)	83.527%	88.923%	73.392%
E (only training on kuzushiji)	12.277%	91.794%	79.408%
F (model B + training on kuzushiji)	28.133%	96.439%	89.832%

shows the efficiency of augmentation, which demonstrated a significant impact on the model performance. On synthesized images, the model gained over 10% accuracy in comparison to models A and B. A similar effect was observed when augmentation was applied to the training set of kuzushiji characters used for fine-tuning. Furthermore, fine-tuning has a noticeable negative impact on the accuracy of the original synthesized corpus: the more data used, the more loss of accuracy was observed.

Models E and F were trained on kuzushiji, both with the use of augmentation. Model E was trained only on kuzushiji, while model F is based on model B and additionally trained on kuzushiji. As one might expect, model E shows high accuracy on kuzushiji as it was trained specifically for it. Model F shows even better results. However, although the accuracy of model F is high on kuzushiji tests, it drops significantly on the synthesized corpus. Therefore, in a scenario with an already trained text recognition model, fine-tuning it for a specific task would be a more efficient choice than training it from scratch.

We also included the balanced accuracy values to Table 1, since it is calculated as the average of model sensitivity (true-positive rate) and specificity (true-negative rate) and therefore offers a more reliable performance measure compared to standard accuracy. For most models (except model A) the balanced accuracy values are lower than standard accuracy. This is to be expected, as there are classes weighted the same as others but having very few samples (even just one in multiple cases—Figure 5 shows a few examples). Such classes contribute to producing a negative impact on the overall accuracy. The balanced accuracy decrease is more significant for the models fine-tuned on kuzushiji, since a lower learning rate makes it difficult for the model to correctly learn the features of the less populated classes.

Table 2. Influence of sample quantity on balanced accuracy.

Threshold	Accuracy	Balanced Accuracy
5	96.896%	92.314%
10	97.209%	93.524%
20	97.552%	94.388%
50	98.037%	94.865%

illustrates the sensitivity of the model F (without augmentation) to filtering out the classes with a number of training samples fewer than the given threshold. As one can see, both accuracy and balanced accuracy continue to increase as the classes with fewer training samples are filtered out.

6. Case Study: Recognition of Handwritten Calligraphy-Style Writing

Based on the promising results of our experiments with kuzushiji writing, we ran a preliminary experiment with an attempt to use the model trained on kuzushiji to recognize the handwritten calligraphy style text, which is actually a fragment from an anonymized handwritten private letter. Figure 6 draws on the fragments of the document used for our case study experiment.

Such texts often pose difficulties even for readers with a good Japanese language background. We asked a Japanese expert to translate and mark up the input text in order to highlight the characters, which required some more time and context exploration even for an expert to decipher.

Since the handwritten letters contain Japanese alphabetical characters not present in the KKanji dataset, for this case study, model F (see Table 1) was enhanced with additional training on the K-49 dataset by ROIS-DS [30], which is an imbalanced dataset consisting of 49 characters including 48 Hiragana symbols and one Hiragana iteration mark.

Figure 7 depicts the marked-up translation along with the recognition output produced by the OCR system.

We must acknowledge that the recognition accuracy of this fragment of the handwritten text is just around 68%, which is lower compared to kuzushiji recognition accuracy, and the model struggles with the characters having too few samples in the Kuzushiji dataset. Also, we need to note that for this case study, in our prediction correctness assessment, we ignored the diacritic marks such as dakuten (Dakuten—a small diacritic mark placed at the top-right of Japanese hiragana and katakana characters to modify their pronunciation so that the voiced consonants are produced) and handakuten (Handakuten—a small diacritic mark placed at the top-right of Japanese hiragana and katakana characters to modify their pronunciation so that the semi-voiced consonants are produced) since the original dataset does not differentiate between them.

Though the results achieved in this case study show that the kuzushiji-trained model applied to the handwritten contemporary calligraphy is not as straightforwardly efficient as it is for predicting the kuzushiji characters themselves, the task suggests an interesting use case for the future work.

7. Discussion

7.1. Observations of the Major Outcomes

The performance of the model on kuzushiji, especially after training, suggests that it has great potential to be used for handwritten-text processing. Model fine-tuning must be enough to reap most of the benefits of training without losing its ability to perform on the other types of texts.

These observations are generally consistent with the findings in DTrOCR [5]. However, while the results reported in [5] suggest that the fine-tuning leads to improvement of recognition quality on all types of data, in our experiments, the accuracy decreased on the synthesized images compared to kuzushiji recognition quality, mainly due to the model being fine-tuned just on kuzushiji data.

Compared to the results reported in [5], our experiments with applying the model to Japanese texts do not reach the same level of performance: lower accuracy values can be explained by a significantly smaller amount of training and fine-tuning data, as well as their insufficient variability. To improve the accuracy of kuzushiji recognition, more samples are required for augmentation along with using oversampling techniques, the latter having been successfully applied to the same dataset as reported in [14].

7.2. Further Steps

After confirming the method’s potential for universal Japanese text recognition, the next step would require a more accurate replication of the original research, which would involve a significant increase in the training data, adding synthesized scene and handwriting texts, as well as increasing the amount and variability of data samples.

The text generation software used in the original research does not support Japanese handwritten text; however, there are other models and generation methods available. Additionally, while the real data available for Japanese texts is not as varied and abundant as that for English and Chinese, there are still various available datasets that could be utilized for fine-tuning and training the model, for example:

• ICDAR 2019 multilingual dataset (scene text) [31];
• Manga109 (Japanese comics) [32,33];
• Kuzushiji Recognition (full pages of kuzushiji text) [34].

7.3. Prospective Application Areas

Various domains of digital humanities and education suggest plenty of promising cross-disciplinary applications of the method.

The results achieved during the experiments on kuzushiji recognition and case study on calligraphy-like handwritten letters unveil the possibilities to use similar models for transcribing and searching historical documents, reviving the antique woodcut printed books for contemporary readers, unscrambling the inscriptions in ukiyo-e pictures like those that can be seen in Figure 8, accompanying museum artifacts on exhibition display, and even unraveling the engravings on old memorial stones and monuments. The latter case could be an insightful opportunity to enhance traveling and history learning experiences to make them genuinely meaningful and truly memorable [35].

Language learning applications (specifically, in the domain of iCALL) can benefit from applying the method to the common types of texts, such as printed and scene texts, but also to developing technology-enhanced educational resources on learning kuzushiji characters or other palaeographic [36] writing styles used in ancient Japanese artistic and classical literature.