Deteriorated Characters Restoration for Early Japanese Books Using Enhanced CycleGAN

Round 1

Reviewer 1 Report

The paper logically and succinctly describes an improved method of the CycleGAN technique applied to old writings already digitized. The proposed technique can contribute to an increase in the readability of old texts, whose supports have suffered damage over time.

Author Response

Dear reviewer

   First of all, we would like to thank the reviewers for their effort with thoughtful suggestions and comments on our manuscript. We have carefully studied the valuable comments and believe that all of the suggestions and comments have been taken into account in our revised manuscript. The change is marked in red color in the manuscript and attached to the file.

--------------------------------

Overall Comments

The paper logically and succinctly describes an improved method of the CycleGAN technique applied to old writings already digitized. The proposed technique can contribute to an increase in the readability of old texts, whose supports have suffered damage over time.

Response:

Thank you very much for your kind comments. We will continue to put our heart into the research and give more constitutions to cultural heritage preservation.

Best wishes.

Author Response File: Author Response.pdf

Reviewer 2 Report

the manuscript suggests a GAN-based approach for virtual restoration. In general, the proposed approach is innovative and interesting. I suggest a minor review to improve some aspects.

introduction: I suggest to the authors to insert a quick state of the art on the increase of machine learning and deep learning (or chemometric) techniques in the cultural heritage sector in order to highlight the innovativeness of the proposed approach.

Materials: The conditions of acquisition of the images are not described correctly. I suggest the authors detail how the images were acquired, illuminating working distances. Using GAN the authors probably had to normalize the acquisition conditions before proceeding.

Computational times: is it possible to quantify processing times? You could enter the characteristics of the data processing system and indicate the processing time.

Your case study is very interesting and well executed. One of the future developments could also be on works with greater colorimetric variability. The authors can insert some considerations in the conclusions on the complexity of managing a highly variable data.

Author Response

Dear reviewer

   First of all, we would like to thank the reviewers for their effort with thoughtful suggestions and comments on our manuscript. We have carefully studied the valuable comments and believe that all of the suggestions and comments have been taken into account in our revised manuscript. The change is marked in red color in the manuscript and attached to the file.

--------------------------------

Overall Comments

The manuscript suggests a GAN-based approach for virtual restoration. In general, the proposed approach is innovative and interesting. I suggest a minor review to improve some aspects.

Your case study is very interesting and well executed. One of the future developments could also be on works with greater colorimetric variability. The authors can insert some considerations in the conclusions on the complexity of managing a highly variable data.

Response:

Thank you very much for your valuable suggestions. We have added details as follows:

(Page 1, Lines 27-35 in the revised manuscript):

Currently, lots of researchers apply deep learning and machine learning methods for cultural heritage protection, organization, etc. Literature organization is one of the hot topics in this area, such as the re-organization of OBI [6, 7], Kuzushi-ji [1,2,8], and Rubbing [9]. In detail, Yue et al. aim to achieve a good accuracy recognition for Oracle Bone Inscription, which is ancient characters described on the tortoises’ shells and animals’ bones. Zhang et al. combine simple deep learning models and lexical analysis to recognize rubbing characters described on the bones from 3000 to 100 years ago. Lyu et al. try to use deep learning and image processing method for detecting and recognizing Kuzushi-ji. It is also the target literature of this paper.

(Page 3, Lines 44-54):

GAN-based ancient characters restoration is focused on supervised image-to-image translation for inpainting damaged characters [11,12]. Su et al. inpainted masked characters in the book of the Qing dynasty and Yi, handwritten ancient Chinese characters, with unmasked images and applied them to the practical ancient text. Wenjun et al. inpainted large-area damaged characters in the ancient Yongle Encyclopedia with ground truth and examples of them. To our knowledge, this is the first time the practical restoration of Kuzushi-ji and the early Japanese books using deep learning. When applying supervised image-to-image translation for Kuzushi-ji, the various damaged and distorted letter styles make it hard to prepare train data. Moreover, most damaged characters are weak damages, which makes GANs training difficult.

Response:

Thank you very much for your careful suggestion. The acquisition conditions of the images are very important for this research. Since the conditions are changed depending on the literature’s size, shape, etc. Therefore, a digitization manual is made for literature and published in Japan, which is used for image acquisition. Although it is provided only in Japanese now, I think it is able to be translated into English in the near future.

Furthermore, we add an explanation on Page 8, Lines 205-209.

The acquisition conditions of the images are very important for this research. Since the conditions are changed depending on the literature’s size, shape, etc. Therefore, a digitization manual is made for literature and published in Japan, which is used for image acquisition[31]. The images of literature are taken based on this manual.    

National Institution of Japanese Literature, Manual for the digitization of Japanese historical literature,

https://www.nijl.ac.jp/pages/cijproject/images/digitization-manual/digitization-manual_NIJL-202205.pdf

Response:

Thank you very much for your careful suggestion. We add the experimental results in Table 7, and the hardware environment in the article of Page 11, Lines 281-287.

Table 7. The computation time of full-page restoration.

Table 7 reports the processing times of full-page restoration by binarization and enhanced CycleGAN. For the CycleGAN-based method, we measure the times on the CPU and the GPU, which AMD Ryzen 9 3590X 16-Core Processor CPU and Nvidia GeForce GTX 1080ti GPU. Deep learning methods need longer processing time than traditional image processing. However, the restoration is automatic, far faster than human work. Furthermore, proposal achieves a better improvement than image processing method.

Response:

Thank you very much for your insightful comment. As you pointed out, robustness for the colorimetric variability is our future work. In this paper, we aim to restore black-and-white or specific book text, so CycleGAN-based methods work well. For example, in Figure 10 of page 15, the methods subjectively restore the faded color of the background and match all of the background colors. On the other hand, our proposal may unintentionally unify the background with just one color when training numerous books. Therefore, we added our proposal limitation and future work in the conclusion as follows:

(Page 13, Lines 304-309):

In this paper, we achieve slight-to-moderate damage restoration on a specific early Japanese book. However, our method has a limitation for complete damage because the damaged character is few, and it is based on CycleGAN. Furthermore, the early Japanese books have multiple types of deterioration and color configuration, and paper texture. In terms of future work, we aim to realize a high degree of freedom restoration for ancient documents.

Author Response File: Author Response.pdf

Reviewer 3 Report

Review of the Manuscript “Deteriorated Characters Restoration for Early Japanese Books using Enhanced CycleGAN”

The indisputable advantage of this research is an attempt to combine humanities and computer sciences, or, to be more precise, to study ancient manuscripts with the help of IT technologies. The relevance and even urgency of such works is beyond doubt, because the researched materials (which are Early Japanese books) deteriorate over time and may be irretrievably lost to us. Accordingly, there is a need to digitize them.

The article is well illustrated. As can be seen from the images, the manuscripts that are represented as damaged are not so destroyed that they cannot be restored at all.

I am interested in the question, do the authors consider different degrees of damage (for example, no damage, slight damage, moderate damage, severe damage, complete damage, etc.)? And what were the most destroyed texts they had to work with?

Also, I would like to point out a couple of inaccuracies in the table below.

Minor editing of English language required. 

Author Response

Dear reviewer

   First of all, we would like to thank the reviewers for their effort with thoughtful suggestions and comments on our manuscript. We have carefully studied the valuable comments and believe that all of the suggestions and comments have been taken into account in our revised manuscript. The change is marked in red color in the manuscript and attached to the file.

--------------------------------

Overall Comments

The indisputable advantage of this research is an attempt to combine humanities and computer sciences, or, to be more precise, to study ancient manuscripts with the help of IT technologies. The relevance and even urgency of such works is beyond doubt, because the researched materials (which are Early Japanese books) deteriorate over time and may be irretrievably lost to us. Accordingly, there is a need to digitize them. The article is well illustrated. As can be seen from the images, the manuscripts that are represented as damaged are not so destroyed that they cannot be restored at all.

Response:

Thank you for your insightful comments. The standard CycleGAN cannot restore slight damage, although this damage accounts for almost deterioration in the real early Japanese books. Our proposal can restore not only remarkable damage such as moderate damage but also slight damage, but it is not effective in achieving optimal performance for the complete damage. It is because the damage character is few in the books and CycleGAN-based method has a limitation for extreme changes. In this study, we pay attention to slight-to-moderate damage but aim to restore few completed lost part in the future.   

Additionally, we add the future work in the article of Page 13, Lines 304-309:

In this paper, we achieve slight-to-moderate damage restoration on a specific early Japanese book. However, our method has a limitation for complete damage because the damaged character is few, and it is based on CycleGAN. Furthermore, the early Japanese books have multiple types of deterioration and color configuration, and paper texture. In terms of future work, we aim to realize a high degree of freedom restoration for ancient documents.

Response:

Thank you very much for your knid comments. I revised these points and some grammar mistakes, colored by red(Page 11, Lines 278-279; Page 1, Lines 38-39).

Author Response File: Author Response.pdf