Learning System for Japanese Kanji Calligraphy with Computerized Supervision

Abstract

The most popular way of learning oriental calligraphy has been by practicing the calligraphy under the supervision of a human teacher, but finding a good instructor can be difficult. There are a number of studies in the literature that have evaluated calligraphic characters in holistic ways, but such systems do not support detailed supervision of scripting errors. This study proposes a Kanji calligraphy learning system with computerized supervision and analyzes the learning efficiency of the system, where the supervision includes symmetries between strokes. The proposed system compares a written calligraphic character of a user to the model of a human expert, and indicates error spots with explanations. An experiment with 22 participants proved that this system was more efficient at reducing the number of scripting errors in comparison to the traditional manner of a human expert. The main contribution of this paper was to identify and reveal the efficacy of computerized supervision in comparison to a human supervisor. The proposed system decreased the writing-error-rates of learners from 32.7% to 3.4%, whereas the traditional practice reduced the error rates from 31.0% to 6.8%. This result shows that computerized supervision is more efficient than human supervision for learning calligraphy.

1. Introduction

Oriental calligraphy is a visual art that represents languages and emotions using signs and symbols, which was developed and has been evolving for centuries in countries that utilize Chinese characters in their scripting system. The calligraphy of Japanese Kanji, which is the adopted logographic of Chinese characters in Japan, was thus greatly influenced by Chinese calligraphy [1]. The Kanji calligraphy is well-known to people in many countries and has received much artistic attention throughout history. Symmetries are easily found in Chinese characters: some characters have completely symmetrical shapes, and most of the characters have strokes that are scripted symmetrically. Three types of symmetry exist in Chinese characters: angle, distance, and length ratios between strokes [2]. These symmetry-related factors have been utilized to evaluate calligraphic characters [3] and understood as a personal feature of the writer [2].

Most of the previous studies on oriental calligraphy have focused on generating calligraphy-style handwritten characters. Lyu et al. proposed a system that compresses synthetic images with specific styles from standard Chinese font images and created a large-scale dataset for Chinese calligraphy synthesis [4]. Xu et al. proposed a shape and hierarchical parameters analysis-based calligraphy artwork generation system from existing calligraphic artwork samples [5]. Wong and Ip presented a technique for image coherence regarding the local distribution of gray values of the image in a particular resolution or scale and showed its application to the generation of Chinese calligraphic images [6].

Some studies have proposed systems of visualizing the process of scripting/drawing in real time using digital devices such as tablet styluses or traditional brushes fitted with digital sensors [7,8,9,10]. These systems can express a brush-style typeface with a good quality, although the type of drawing obtained with a pen tablet is unnatural because the system is not robust enough for interactive drawing. Wada and Shin generated a 3D animation of a virtual-brush’s movements and calligraphic scripting from digital data recorded with a pen tablet [11]. The simulation of ink diffusion during virtual writing is another related topic of consideration, with various affecting factors (ink, paper, and brush conditions) [12]. By simulating the various effects during scripting, the calligraphic artwork generated by the virtual brush system appears more realistic.

The education on Kanji calligraphy has long been conducted by human supervisors using traditional tools—brush, ink, and paper [12,13]. However, preparing the learning tools can be difficult according to use situations, and finding a good supervisor can be considered even more difficult [14,15,16]. Moreover, it is a burden for the teachers to supervise the calligraphy because many students are often assigned to a single teacher and supervising multiple learners simultaneously while providing sufficient support for all of them is a time-consuming job.

There have been a few reported studies on automatic learning systems for calligraphy. Some of these studies have focused on providing a visual example of calligraphic writing, while others have attempted to evaluate the writing. However, considerable gaps exist between having human supervision and utilizing the computerized system. The computer systems have been incapable of providing detailed supervisions on the writings; they only evaluate the overall scores.

Narita and Matsumaru presented a calligraphy learning system with a leap motion sensor and a projector [13,17]. A user writes a character on paper, on which an example of the written character is projected. The system virtually generates calligraphic writing from the leap motion sensors and compares the generated writing to the projected example. The evaluations of the scripted calligraphy were conducted by pixelwise comparisons of the example with the reference and by generating correlation coefficients for writing speed and velocity. The evaluation of four letters with three participants showed that the participants’ scripting became more similar to that of the examples after practice.

Muranaka et al. proposed a calligraphy learning system with video tutorials [18]. A user drew a character with brush on paper, where the video tutorials were displayed on a one-way mirror that was installed diagonally above the paper. The user was able to see both the tutorial and his/her own writing. Participants were able to improve the quality of their calligraphy scripting, but an interactive learning process was not proposed.

Han et al., proposed an automatic grading system with a fuzzy algorithm, which evaluated scripted characters on papers with brush and ink. This system evaluated handwritten characters in a holistic approach, similar to Narita and Matsumaru’s method. Feature values were calculated for balance, character size, and tilting degree of the horizontal/vertical strokes, and compared these to standard values for evaluation. A holistic evaluation, however, does not provide special information on scripting errors, and the users do not know which parts of the characters have errors.

This paper introduces a calligraphy learning system that specifically indicates error spots. It was difficult for users to understand error spots in previous systems because the written characters were evaluated in holistic ways. We introduce the present calligraphy learning system for Japanese Kanji in later sections and show experimental results to prove its efficiency.

2. Calligraphy Learning System Overview

Screenshots of the developed calligraphy learning system are shown in Figure 1. At the system start, a user selects a letter to learn and checks the stroke order to draw. Figure 1a shows a virtual paper and a brush to script the letter. The virtual pen moves as the user moves the digital pen on/above the surface of the tablet, and lines and dots are displayed on the virtual paper when the pen touches the tablet. The proposed system reflects pen movement in real time and can therefore assist the user with scripting characters by watching the screen, not the pen and tablet [11].

After scripting the character, the user checks the evaluation results and advice, which are generated automatically. The evaluation provides results and advice by indicating the stroke numbers, error spots, and types of errors. Then, an animation of the writing process is played on the virtual paper. This animation is automatically created from the reference calligraphic data by a teacher from a Japanese elementary school. Through this procedure, the user learns to write the selected Japanese Kanji letter; this procedure of practice was repeated five times in later experiments.

3. Selection of Letters for Learning Calligraphy

The sheer number of Japanese Kanji characters is one of the reasons that learning Japanese calligraphy is difficult. The number of letters in Kanji for daily life is close to 3000 (2964 letters for JIS level-1), and it is difficult to practice calligraphy for all the characters individually. Therefore, it is important to choose letters that include different styles of calligraphic strokes. The letter “Yong” (永), which has eight different and representative styles of strokes in a single letter, is traditionally used as the beginning letter to learn Japanese calligraphy [19]. Learning calligraphy by writing this single letter is a system that abstracts up to eight complex point drawings for each of the characters (there were more than 20,000 Chinese letters during the Nara/Heian period of Japan; therefore it is difficult to learn all possible letters of the calligraphy) [20]. Practicing the letter (永) is important, but would not be enough to master all the different styles of calligraphic strokes (2964 JIS level-1 Kanji is composed of 32,398 strokes) because it would not cover all the strokes in Kanji. It takes a great deal of time to learn all of these strokes [21,22]. To select the letters to evaluate the proposed calligraphy learning system, this study suggests a minimum number of Kanji list with a computerized clustering algorithm.

An online handwritten character database [23] is used to cluster the strokes. This database was recorded with a pen tablet (WACOM, Japan), which includes traces of the XY-coordinates of the pen together with the information on pressures and elevation angles at each of the data points. The characters in the database were normalized to the size of 256 × 256, and all the strokes of the characters were separated. The strokes were adjusted so that their starting points were $\left(0,0\right)$.

3.1. Stroke Clustering

Strokes were clustered by the K-means clustering algorithm [24], where the distance between two strokes ($S$ and $S^{\prime }$) was calculated as follows:

$$D\left(S,S^{\prime }\right)=\frac{1}{n}{\displaystyle \sum }_{j=1}^n\sqrt{{\left(X_j-{X^{\prime }}_j\right)}^2+{\left(Y_j-{Y^{\prime }}_j\right)}^2}$$

(1)

where $X_j$ and $Y_j$ are the $x$ and $y$ coordinates of the $j$-th data point of a stroke. The value of K was determined by a simple clustering algorithm [25], which classifies strokes within a threshold as being in the same group. The distance calculation for the simple clustering was conducted with the following equation:

$$d=\frac{s+m+e}{3}$$

(2)

$$s=\frac{\left|s{(x)}_{ref}-s{(x)}_{inp}\right|+\left|s{(y)}_{ref}-s{(y)}_{inp}\right|}{2}$$

(3)

$$m=\frac{\left|m{(x)}_{ref}-m{(x)}_{inp}\right|+\left|m{(y)}_{ref}-m{(y)}_{inp}\right|}{2}$$

(4)

$$e=\frac{\left|e{(x)}_{ref}-e{(x)}_{inp}\right|+\left|e{(y)}_{ref}-e{(y)}_{inp}\right|}{2}$$

(5)

where $s$, $m$, and $e$ are the characteristic points of each stroke (i.e., starting point, middle point that is obtained by calculating the mid-point of the ordered series of strokes, and endpoint).

Table 1. Number of clusters after stroke clustering with different threshold values.

Threshold *	Number of Clusters
0	32,389
5	1156
6	817
7	601
8	455
9	364
10	294
11	236
12	197
13	168
14	142

* The unit of the threshold is the distance between two strokes calculated as Equation (2).

shows the clustering results by the simple clustering algorithm with different threshold values. The number of clusters was reduced as the threshold value increased, which means that fewer clusters represented all the strokes in Japanese Kanjis. The strokes in the same cluster were averaged to generate the representative strokes for the cluster. We chose the threshold value of eight for this study, as it was previously reported that reducing the stroke number to less than 400 could cause a decrease in the recognition accuracy to under 95% [26].

The threshold was set to minimize the number of clusters and stroke distortion.

Therefore, we reviewed the representative strokes in the 455 cluster and found many strokes with similar shapes but different sizes or writing directions (see Figure 2). These differences could be caused by the relative positions of the stroke in a character or by the complexity of a character (see Figure 3 for an example). Since these strokes were difficult to classify by automatic algorithms (reducing the number of clusters to less than 400 caused recognition problems, as described above), we classified them manually. In the end, the strokes of Japanese calligraphy were compressed to 44 from 32,398 (see Figure 4).

3.2. Suggestion of Kanji Characters for Learning Calligraphy

This study suggests the use of 19 letters for Kanji calligraphy learning, which includes the 44 types of shortlisted strokes shown in Section 3.1, as shown in 0. The selection of the letters was conducted manually because it was difficult to build an automatic procedure. The letters were selected to minimize the number of characters, for the selected characters to have relatively small numbers of strokes, and to minimize repetition of the same stroke types in a letter. The number of strokes in the selected letters was distributed from 1 to 8, such that users could experience various styles in the limited number of examples chosen. The total number of strokes in the selected letters was 83, which is less than twice the total number of stroke types.

4. Evaluation of Calligraphy Writing

4.1. Experimental Overview

The proposed system was evaluated with 19 characters comprising the 44 types of strokes shown in

Table 2. List of letters including all the stroke types possible in 2964 Kanji letters.

Character	Number of Strokes	Number of Strokes that Occurred Once in a Letter	Number of Repeated Strokes in a Letter	Number of Unique Strokes *
邸	8	7	1	3
呉	7	6	1	2
汎	6	4	2	1
迅	6	5	1	2
団	6	5	1	2
写	5	4	1	3
凸	5	4	1	1
永	5	5	0	1
心	4	4	0	3
円	4	3	1	1
孔	4	4	0	3
幻	4	4	0	1
日	4	3	1	3
方	4	4	0	3
女	3	3	0	1
及	3	3	0	2
七	2	2	0	1
乃	2	2	0	1
乙	1	1	0	2
Total	83	73	10	36

* The term “unique strokes” denotes that the stroke type appears in one character only.

and Figure 4. Twenty-two Japanese people (aged 18 to 55 years, 20 males and 2 females) participated in scripting Kanji calligraphy, and the learning efficiency was evaluated in comparison to the traditional method with brush and paper. None of the participants were experts in Japanese calligraphy. The scripted characters were evaluated at the stroke and character levels.

For the readability of the explanation, we first provide an example of the evaluation process of a Kanji character after the features for the evaluation at the stroke and character level are listed. We chose the letter “邸” for the example because it is the most complex letter among the selected letters. Thereafter, the overall experimental results for the 19 characters produced by the 22 writers are provided.

4.2. Calligraphy Evaluation

The proposed system evaluates calligraphy in nine main categories. The evaluation categories were pen pressure, fades, hooks, horizontal/vertical strokes, distance between strokes, relative positions and lengths of strokes, and angles between strokes or segments. Six out of ten categories (except brush pressure, fade, hook, and others) are related to the symmetry, which concerns angles, distances, and the length ratio between strokes. All the evaluation categories and their details are listed in

Table 3. Features for calligraphy evaluation, categorized into 10 groups.

Evaluation Category	Description	Features for Evaluation
Brush pressure	Thickness of stroke at the stroke starting/end points	- Pen pressure at stroke starting/end and corner points
Fade	Gradual change of stroke width at stroke end	- Starting/end position
		- Direction
		- Smoothness
Hook	Sudden change of stroke direction in a stroke	- Starting/end position
Horizontal/ vertical stroke	Horizontal or vertical strokes without curves	- Tilting angle
		- Heading/sharpness at the end
Distance	Distance between strokes	- Distance between strokes
Position	Position of stroke relative to other strokes	- Relative position of strokes’ starting and end points
Length	Length of stroke	- Length of strokes/segments
Angle	Angle between two strokes or segments	- Angles at between segments in a stroke
		- Angles between strokes
Balance	Balance of strokes and character shapes	- Ratio of stroke lengths between segments/strokes
		- Ranks of stroke lengths
		- Relative tilting angles of strokes
		- Ratio of height to width of a character
		- Horizontal/vertical position of central vertical/horizontal stroke
Other	Errors that are not categorized above	- Curvature of corners
		- Writing speed
		- Pen movements at corner (should move slightly upward)

. A written character was evaluated at each of the feature points compared to the checklist based on a dictionary [27]. The feature points of the strokes were the starting, end, and corner points of strokes—a part of the strokes divided by a corner point was noted as a segment.

4.3. An Example of Calligraphy Evaluation

This section shows an example of the calligraphy evaluation with the letter “邸,” which is the most complex letter among the selected letters. A written character is evaluated at twenty checkpoints—starting and end points of each stroke and corner points—as shown in Figure 5, labeled “a” to “t”. The strokes were evaluated by comparing the elevation angles, pen pressure, tilting angles, x and y coordinates, and the distance between two points.

Table 4. Checklist for calligraphic evaluation of the letter “邸”.

Stroke Number	Evaluation Checklist	Place Tags * and Conditions
First stroke	Brush pressure	a, b
	Angle	a to b
Second stroke	Horizontal/Vertical stroke	c = d **
	Angle	d to e
	Distance between two points	d, e
	Brush pressure	c, d, e
Third stroke	Angle	f to g
	x-coordinate	a > g
	Brush pressure	f, g
Fourth stroke	x-coordinate	(b − a)/2 + a > h
	x-coordinate	h > i
	y-coordinate	d = i
	Angle	i to j
	Distance between two points	i, j
	Brush pressure	h, i, j
Fifth stroke	x-coordinate	d = k
	Angle	k to l
	x-coordinate	i < l
	Brush pressure	k, l
Sixth stroke	y-coordinate	h = m
	Angle	m to n
	Brush pressure	m, n, o
	Angle	n to o
	x-coordinate	s > o
Seventh stroke	x-, y-coordinate	o = p
	x-, y-coordinate	d, n = q
	Angle	q to r
	Brush pressure	p, q, r
Eighth stroke	Brush pressure	s, t
	y-coordinate	k > t
	Distance between two points	k, t

* The tags are as illustrated in Figure 5. ** Two coordinates were considered equal if the distance between the coordinates was smaller than a threshold.

describes the details of all the checkpoints. We developed these checkpoints by referencing a dictionary [27].

We consider the pressure of the pen in this study, and the pen tablet is used to evaluate the strength of the pen pressure on a scale of 0 to 1023. There are 20 detail points in the strokes where the pen pressure needs to be either strengthened or weakened. Points a, c, d, g, h, i, l, n, q, and s require a higher pressure from the pen between 800 and 1023, while points b, e, f, j, k, m, o, p, r, and t require a weaker writing pressure of less than 799. The system records the pen pressure at each point, and if it falls within the specified values, the user is informed that this is correct. Table 4 summarizes the above descriptions.

Figure 6 depicts an example of calligraphy scripting with the proposed system showing the various problems associated with pen pressure. The system evaluated the writing and listed the problems for each stroke based on the XY-coordinates, pen pressure, and elevation angle. The system notifies the user that “the writing is good” if all the strokes are written correctly.

Figure 6b shows an example of calligraphic scripting displaying a number of mistakes, and the problems found by the proposed system from the example are as listed in

Table 5. Writing issues discovered by the proposed system from the example of Figure 6.

Problem No.	Stroke No.	Issue
1	2	The angle of hook is not correct.
2	6	The 6th stroke should start at the same vertical level as the 4th stroke.
3	6	The horizontal stroke should not move upward or downward.
4	7	The corner point of the 7th stroke should be at the same horizontal level as the one of the 6th and at the same vertical level to the 2nd stroke.

. This result also shows the limitation of the proposed system. The sharpness at the stroke-ends was not scripted well in the example, but these errors were not detected in Table 5. Details of these evaluation errors should be discussed in further studies.

4.4. Learning Performance of Calligraphy

Experiments were conducted to compare the educational effectiveness of learning with this method compared to tutoring in the traditional process of scripting on paper. We evaluated the time required to learn and write Kanji characters and found some striking differences in the learning of the same Kanji character. The twenty-two participants were divided into two equal and random groups, where the participants in one group practiced with the proposed system and the participants in the other group practiced with the traditional system—brush, papers, textbook, and dictionary. There was no statistical difference between the two groups. Each group had eleven participants, including ten males and one female. The mean age of the participants in the group with the traditional system was 27.9 and the mean age of the participants in the group with the proposed system was 24.6, but the age difference between the two groups was insignificant (p = 0.748, U = 65.5, Mann-Whitney U test, two-tailed).

The participants of the traditional system repeatedly scripted letters on papers with a brush, while reading the textbook and dictionary to check the stroke order and writing styles. The participants in both groups practiced five times for each letter.

Figure 7 shows the writing error rates of the participants. The error rates were measured by the number of accurately written strokes, as follows:

$$\mathit{\text{writing\_error\_rates}}=1-\frac{\mathit{\text{number\_of\_accuractly\_written\_strokes}}}{\mathit{\text{total\_number\_of\_strokes}}}$$

(6)

The error rates were calculated after an automatic evaluation of the test group, whereas the evaluation was conducted by a teacher from a Japanese elementary school for the control group, because the participants in the control group wrote on paper. The mean error rate of the test group at the last trial was 3.4%, whereas the one for the control group was 6.8%. There were no significant differences between the two groups from the first to third trials, but the writing accuracy of the test group was significantly higher than that of the control group (Mann–Whitney U test; p < 0.01 and U = 103.5 for the 4th; and p < 0.05 and U = 98.0 for the last trial). For the statistical test, an independent sample Kruskal-Wallis test was conducted, and the difference in terms of the error rates was found to be significant (p < 0.001, N = 110). Thereafter, a Mann-Whitney U test was conducted, comparing the error rates of the two groups. This results show that the proposed system was more effective for learning Japanese Kanji calligraphy in comparison to the traditional brush and pen-based system.

It is notable that the mean error rates of the test group decreased more rapidly, and the error rate of the 4th trial was slightly higher than the error rate of the 5th trail of the control group. This implies that a user with the traditional system may be required to practice the scripting of a set of characters to achieve similar skills.

Figure 8 shows two pairs of writing samples of the proposed system for the first and the last trials. Many types of errors (character balance, overlap of strokes, tilted strokes, and positions of stroke coordinates) were observed during the first trial; however, the calligraphy in the final trial shows a more balanced structure and most errors related to strokes are corrected. Figure 9 confirms that a decrease in the errors occurs for all Japanese Kanji letters. The mean error rates of the last trial were less than half of those for the first trial for most letters (15 out of 19 letters). The letter “七” showed the largest decrease, for which the final error rate was 13.5% of the first trial.

Figure 10 shows the total number of errors that occurred during the first and last trials for each type of error. The errors were considerably reduced through practice. Items such as “fade” and “pen pressure” were relatively easy to understand, and the reduced rates for these errors were high (the most frequent error in the first trial was pen pressure (563); however, the number of errors became less than one-third (153) through practice). However, the learning efficiencies were relatively low for “angle” and “length.” The error of “angle” occurred 238 times in the last trail, which was 70.2% of that at the beginning (167 errors); the error of “length” occurred 187 times in the last trial, which was 68.0% of the one at the beginning. It is an interesting finding that the learning efficiencies were different, depending on the error types. We presume that the low efficiencies for “angle” and “length” could be because of their uniqueness (the stroke length varies for different stroke type) because there is little chance to learn the stroke lengths and angles between strokes. The reason for this difference and the ways to increase the efficiencies for different types of errors should be addressed in future research.

5. Conclusions

Traditional ways of learning Japanese Kanji calligraphy include writing on paper with a brush and ink, under the supervision of a teacher. This method has been verified throughout history; however, preparing the equipment and finding a good supervisor are not always easy, depending on the circumstances. This paper introduced a novel system for calligraphy learning, which provides automatic advice by indicating the errors in Kanji calligraphy. The main contributions of this paper are as follows: (1) a calligraphy learning system that indicates errors in the consideration of symmetry was introduced, and (2) it was proved that calligraphy learning with computerized supervision can be more efficient in reducing the number of errors than the traditional method with a human supervisor.

Although this study showed that the proposed system outperformed the human supervision regarding calligraphy teaching, this does not ensure that the proposed system is always better. Excellent teachers may supervise the learners better, and the learners can ask calligraphy-related questions to improve their knowledge. Moreover, personal relationships between the teacher and learners could not be mimicked by the proposed system. It would not be possible for the proposed system to substitute human teachers entirely. These limitations need to be addressed in future studies. From another perspective, direct comparisons of learning efficacy among the computer-assisted calligraphy learning systems are necessary. Verifying whether training with the proposed system is effective for other characters is also a significant related topic. A direct comparison of the comments from the proposed system and human teachers would verify our results with a different perspective. Further, variant brush effects (e.g., dirt, blurring of characters) should be included for a better representation of virtual brush artwork.