Brush Stroke-Based Writing Trajectory Control Model for Robotic Chinese Calligraphy

Abstract

Engineering innovations play a critical role in achieving the United Nations’ Sustainable Development Goals, especially in human–robotic interaction and precise engineering. For the robot, writing Chinese calligraphy with hairy brush pen is a form of precision operation. Existing writing trajectory control models mainly focus on writing trajectory models, and the fine-grained trajectory control model based on brush strokes is not studied. The problem of how to establish writing trajectory control based on brush stroke model needs to be solved. On the basis of the proposed composite-curve-dilation brush stroke model (CCD-BSM), this study investigates the control methods of intelligent calligraphy robots and proposed fine-grained writing trajectory control models that conform to the rules of brush calligraphy to reflect the local writing characteristics. By decomposing and refining each writing process, control models in the process of brush movement are analyzed and modeled. According to the writing rules, fine-grained writing trajectory control models of strokes are established based on the CCD-BSM. The parametric representations of the control models are built for the three stages of initiation, execution, and completion of strokes writing. Experimental results demonstrate that the proposed fine-grained control models can exhibit excellent performances in basic strokes and Chinese characters with better writing capabilities. Compared with existing models, the writing results demonstrate the advantages of our proposed model in terms of high average similarity with two quantitative indicators Cosine similarity (CSIM) and Structural similarity index measure (SSIM), which are 99.54% and 97.57%, respectively.

1. Introduction

Engineering innovations in today’s world, especially technologies in human–robotic interaction and precise engineering, are pivotal in achieving the United Nations Sustainable Development Goals. Writing Chinese calligraphy with a hairy brush pen by an intelligent robot is a form of precision operation. Writing trajectory control model, as a crucial component in intelligent robotic calligraphy, plays an important role in the rendering effect of robotic calligraphy works [1,2,3,4]. Unlike soft robotic system [5] and existing intelligent writing robots that use hard pens [6,7], the use of a hairy brush pen requires the consideration of the brush stroke model, that is, the brush pen with a certain pressing height and inclined angle on the paper to form a special graphic [8].

To understand the significance of the results, we explain the meaning of specialized terms based on the writing process of calligraphic:

Bifeng: The tip of the hairy brush pen.

Yunbi: The writing process in which a stroke is written.

Qibi: The beginning of writing a stroke.

Shoubi: The ending of writing a stroke.

Xingbi: The hairy brush pen moves horizontally with a certain pressing height and inclined angle.

zhongfengxingbi: The brush tip remains in the center of the stroke at all times.

cefengxingbi: The brush tip deviates from the centerline of the stroke.

Most of the existing control models used traditional B-splines for trajectory interpolation. Wen Y et al. [9] proposed a robotic trajectory tracking control method, using cubic spline interpolation to generate a closed geometric three-dimensional path. The three-dimensional kinematic model of the end-effector was established to control the input velocity and tangent vector of the trajectory path. By transforming the writing problem into an optimization problem, Zhenyu X et al. [10] studied a method to minimize the input energy of the writing trajectory by modeling characters. With normalized uniform B-splines as basic functions, skeleton images of characters were generated from trajectories. The single-pixel characters were then transformed into corresponding written characters based on the pix2pix image transformation framework using deep learning. However, employing B-spline interpolation for writing control cannot comprehensively reflect the local writing characteristics of strokes or Chinese characters, resulting in significant discrepancies between the generated models and actual characters.

Recently, the machine learning-based approach [11], deep reinforcement learning and generative adversarial networks [12,13,14,15,16,17,18,19] have been used to predict flow status and to train robots for writing tasks widely. Generative adversarial networks—actor–critic (GAN-AC) [14] integrated deep reinforcement learning with generative adversarial network. During learning and training, the actor–critic (AC) algorithm was used to control the robot to write repeatedly and generate images multiple times. However, extensive repeated writing operations will inevitably lead to a rapid decline and exhaustion for the service life of the robot. Moreover, the common problem of these models is the lack of a brush stroke model, as they merely trained the robot to write strokes repeatedly without conforming to the three-stage principle of calligraphy writing named “qibi”, “xingbi” and “shoubi” (initiation, execution and completion of strokes).

To solve the above problems, a previously proposed brush stroke model named CCD-BSM [20] is indispensable. CCD-BSM is a closed composite curve dilation brush stroke model, which was designed with an oblique section of a cone and two tangent parabolas and can be fixed through analytical geometric calculation. Then, a basic stroke graphic can be generated based on the designed composite curve [20]. In the proposed model, only four measurable and controllable parameters were defined, including the length of a hairy brush bundle, maximum radius of a brush bundle, depth to which the pen was pressed, and inclined angle between the pen and vertical direction when touching the paper. All parameters of the CCD-BSM are determined according to the specification and posture of the hairy brush pen with rigorous mathematical proof, without manual setting by experience or parameter estimation on a number of samples [20]. Compared with existing dynamic brush models, CCD-BSM leads to good generalization and robustness for different pens as the specifications of brush pens are taken into consideration. The writing trajectory is composed of several trajectory points by our CCD-BSM. It is defined by the specification of the hairy brush pen, pressed depth (H), inclined angle (α) and rotation angle (θ). We have provided the mathematical relationship between H, α, θ and the brush stroke model CCD-BSM of the trajectory [20].

Based on the existing CCD-BSM, this paper studies the writing trajectory control model of calligraphy robots and proposed a fine-grained modeling method to reflect the local writing characteristics. By decomposing and refining each writing process of brush, the commonly used control models for “xingbi”, “zhuanbitiaofeng” and “zhebi” are analyzed and built. Based on these different “yunbi” control methods, the trajectory control model based on the CCD-BSM is established; furthermore, the trajectory control model for single strokes can be developed. According to the writing rules of brush, the corresponding description and representation of the three steps of “qibi”, “xingbi” and “shoubi” are given. The experimental results demonstrate that the proposed fine-grained trajectory control model exhibited excellent performance in both basic strokes and Chinese characters. The trajectory control process and the robotic writing effects are illustrated in Figure 1. By modeling different control methods of “yunbi”, the fine-grained control models based on the brush stroke model are proposed, along with three-stage control models conforming to calligraphy rules. Moreover, writing tests and performance evaluations were carried out to verify the performance of our proposed control model. Compared to most existing models, it holds significant advantages with good generalization and robustness.

The main contributions of this study are summarized as follows:

• By decomposing and refining each writing process, the commonly used control methods and models for “xingbi”, “zhuanbitiaofeng” and “zhebi” are analyzed and built.
• Based on the existing CCD-BSM, the writing trajectory control models and methods of calligraphy robots are studied and established. According to the writing rules of brush, the corresponding description and representation of the three steps of “qibi”, “yunbi” and “shoubi” are given.
• The fine-grained writing trajectory control models that adhere to the rules of brush calligraphy are proposed. The experimental results show that compared with other models the proposed model has better performance in both basic strokes and Chinese characters with good generalization and robustness.

2. Related Works

• Based on Bézier curve and B-Spline curve

The traditional cubic B-spline algorithm [21,22,23] has been widely used to plan Cartesian space paths, optimize the trajectory and control robot writing. Mueller S et al. [24] established a 6-DOF robotic writing system based on KUKA. Based on the B-Spline curve, basic strokes were trained. B-splines of reference strokes and actual contours written by robots were obtained. The visual feedback mechanism was utilized in the learning process to calculate the error between the reference characters and the drawing characters; then, the trajectory was updated to generate a training function. But the writing system can only reproduce simple strokes in the database. Lin H I et al. [25] proposed a trajectory model to provide three-dimensional coordinates for robot control. Berio D et al. [26] used a spline-based method to edit polygon and synthesize a continuous virtual trajectory.

2.

Based on learning control and optimal control

Wu R et al. [27] proposed a robot writing framework based on a robotic hand–eye coordination method. Berio et al. [28,29] proposed methods for generating interactive curves and paths. The stochastic formulas of optimal control theory were used to calculate variations in dynamic systems. Huebel N et al. [30] calculated the error between reference characters and drawn characters based on the projected positions to form the paper coordinate position of the robot’s writing trajectory. They used a P-type iterative learning controller (ILC) by a learning node to obtain the Z-axis coordinate position and eventually generate the three-dimensional coordinates $(x,y,z)$ as the robot’s trajectory. Inspired by the pseudo-spectral optimal control, Wang S et al. [31] parameterized the execution trajectory of each stroke as Chebyshev polynomials and applied the pseudo-spectral optimal control (PSOC) method to optimize the open-loop control trajectory of the robotic end-effector.

Mueller S et al. [24] proposed a vision feedback-based iterative optimization writing method, but its iterative optimization process is computationally complex and time-consuming. They compared the writing results with reference strokes to evaluate and update the writing trajectory. Based on error data points and previous splines, the positions of control point for the next iteration were determined to update the contours of strokes.

Ma Z et al. [32] established a closed-loop calligraphy system to reduce errors and perform aesthetic optimization. By predicting the position of the n + 1th stroke based on the positions of the first n strokes, they controlled the writing strategy of the robotic calligraphy. A constrained optimization problem was defined to optimize the aesthetic evaluation and control the writing trajectory of each stroke by the robot.

3.

Based on deep learning

The Long short-term memory network—generative adversarial network (LSTM-GAN) [12] enabled the robot to learn and generate sequences of Chinese character strokes, namely, writing trajectories, and optimize these trajectories based on the writing results. In the absence of a dataset of the robot’s motion trajectories, the combined LSTM-GAN network structure can convert pixel-level stroke images into vector trajectory sequences for robot control. Zhou P et al. [17] extracted parameters of character images with the target style, then converted them into control parameters to guide the writing process and generate writing trajectory precisely.

In summary, existing robotic writing trajectory control methods for robotic Chinese calligraphy are mainly based on a Bézier curve and B-Spline curve, learning control and optimal control and deep learning. However, these models and methods lack the fine-grained modeling method, generalization and the brush stroke model to reflect the local writing characteristics and artistic presentation. As a basic unit of strokes and characters, the brush stroke model should be considered in the trajectory control model. Thus, the fine-grained modeling method that combines the writing rules of the hairy brush pen and the brush stroke model with good generalization and robustness needs to be addressed urgently.

3. Methods and Models

The construction of control methods and models are required based on the CCD-BSM brush stroke model during the “yunbi” process. Then, on this basis, the writing trajectory can be obtained for robotic brush calligraphy by combining different methods of “yunbi”.

3.1. Control Method of “Yunbi”

During the writing process, the influence factors of “yunbi” control methods mainly include the drop height, tip lag of brush pen and the deviation of the direction angle of brush tip. Among them, the drop height depends on lifting and pressing operations in the vertical direction. The tip lag is mainly determined by the bending of the brush hairs. The deviation angle of the brush tip is jointly determined by the rotation of the brush handle around its own axis, combined with the movement of the handle on the paper. The writing trajectory control models based on the brush stroke model are composed of different control methods. First, the control models for “xingbi”, “zhuanbitiaofeng” and “zhebi” are studied, and then the control models of strokes and Chinese characters are further investigated.

3.1.1. Control Model for “Xingbi”

During the “xingbi” process, the pressing height of the brush pen is set to H and the inclined angle is $\alpha$. The brush tip moved at a constant speed along the trajectory on the paper, and the writing path can be approximated as a straight line from point T₁ to point T₂. At this point, the brush handle lagged behind the brush tip, and the trajectory on the paper is from ${T_1}^{\prime }$ to ${T_2}^{\prime }$. The direction of the brush tip is opposite to the direction of “yunbi”, moving along $T_2{T}_1$, as show in Figure 2.

Let the position of the brush tip $P$ be a point on the straight line $T_1{T}_2$, and the corresponding position of the brush handle be $P^{\prime }$. At this point, the stroke width is $w_p$. The pressing height and inclined angle of the brush pen are H and $\alpha$, respectively. The control model shown in Equations (1)–(3) can be denoted as follows.

$$T_1{T_1}^{\prime }=T_2{T_2}^{\prime }=d$$

(1)

$$\overrightarrow{P{P}^{\prime }}=d\overrightarrow{T_1{T}_2}/‖\overrightarrow{T_1{T}_2}‖$$

(2)

$$w(H,\alpha )=w_p$$

(3)

3.1.2. Control Model for “Zhuanbitiaofeng”

The “zhuanbitiaofeng” in brush calligraphy can be viewed as an arc motion on the paper, equivalent to the brush pen being pressed onto a disk with a certain pressing height H and inclined angle $\alpha$. The brush remains stationary, while the disk rotates counterclockwise continuously. After a time interval $T$, it will reach a stable state. The brush handle rotates around its own axis, and the brush rotates around the central axis of the disk with the circular motion. Its rotational speed is equal to the speed of the brush handle’s self-rotation. Due to the brush tip lagging behind the brush handle during the rotation, the brush bundle generates concentric arcs as trajectories on the horizontal surface. The writing trajectory as shown in Figure 3 is generated by the brush pen during the process of “zhuanbitiaofeng”. Taking the center of the concentric arcs as the origin of the reference coordinate system, the radius of the inner and outer arcs are set as r and R, respectively.

During the “zhuanbitiaofeng”, when the brush tip moved to position A, the brush handle is located at position $A^{\prime }$, and the distance between them is d. To obtain an arc with radius $r$, the brush handle must lie on the extension line of the brush tip trajectory and maintain a constant offset of d from the brush tip. According to geometric calculations, when the brush tip moves to the trajectory point T with parameters H and $\alpha$, the brush handle is always in the tangent direction of the concentric arc; the relationship is shown in Equation (4).

$$OT\perp T{T}^{\prime }$$

(4)

where $OT=r$, $T{T}^{\prime }=d$.

At this time, the brush handle takes $A^{\prime }$ as the starting point of the writing trajectory and moves in a circular arc with radius $O{T}^{\prime }$. Compared to the brush tip, its initial direction angle leads ahead of the brush tip by $\alpha$. The solutions for $O{T}^{\prime }$ and $\alpha$ are given by Equations (5) and (6), respectively.

$$O{T}^{\prime }=\sqrt{r^2+d^2}$$

(5)

$$\alpha ={\tan }^{-1}(d/r)$$

(6)

From the above analysis, the width of the concentric arc is set as $w_o$ and the rotation angle of the brush handle is ${\theta }_r$. For any point T on the concentric arc AB, when the brush tip moves from point A to point T along the writing trajectory, the brush handle rotates from point $A^{\prime }$ to point $T^{\prime }$. The control model can be yielded as shown in Equations (7)–(10).

$$w(H,\alpha )=w_o$$

(7)

$$O{T}^{\prime }=\sqrt{r^2+d^2}$$

(8)

$$\angle AO{T}^{\prime }=\angle AOT+\theta$$

(9)

$${\theta }_r=\angle AOT$$

(10)

where $\theta =\alpha$.

3.1.3. Control Model for “Zhebi”

Unlike “xingbi” and “zhuanbi”, “zhebi” involves the brush tip being pressed onto the paper with a certain pressing height (H) and inclined angle (α). As the parameters were controllable, we used Aubo-I5 script programming to set different combinations of the parameters; then, the end-effector of robotic arm can be adjusted to the parameters with certain H and α. After following a straight trajectory to a specific point, the writing direction is changed, and the brush pen then proceeds to the next point along either a straight or curved trajectory.

Assume the writing trajectory of “yunbi” consists of a curve segment formed by four trajectory points $T_1{T}_2{T}_3{T}_4$, where $T_1{T}_2$ and $T_3{T}_4$ are straight segments. The radius of the arc $T_2{T}_3$ is $r$, and the central angle is $\angle T_{2}O{T}_3$. $T_1{T}_2$ and $T_3{T}_4$ are tangent to the arc $T_2{T}_3$ at points $T_2$ and $T_3$, respectively, as shown in Figure 4a. Through the writing method of “zhebi”, it can be seen that when the brush tip moves from point $T_1$ to point $T_2$, it does not follow the original arc trajectory at point $T_2$, as in Figure 4a, but instead it moves toward the upper right along $T_2{T}_4$, resulting in a polyline writing trajectory $T_1{T}_2{T}_4$, as shown in Figure 4b. At this point, the arc segment $T_2{T}_3$ becomes a straight line $T_2{T_3}^{\prime }$ with radius $r=0$. The writing path of the brush handle is a curved segment ${T_1}^{\prime }{T_2}^{\prime }{T_3}^{\prime }{T_4}^{\prime }$, composed of two straight lines ${T_1}^{\prime }{T_2}^{\prime }$, ${T_3}^{\prime }{T_4}^{\prime }$, and an arc segment ${T_2}^{\prime }{T_3}^{\prime }$. ${T_2}^{\prime }{T_3}^{\prime }$ takes $T_2{T_2}^{\prime }$ as the radius and $T_2$ as the center. The relationship is shown in Equation (11).

$$T_2{T_2}^{\prime }=T_2{T_3}^{\prime }=T_4{T_4}^{\prime }=d$$

(11)

In summary, when the “bifeng” reaches point $T_2$ along $T_1{T}_2$, the brush handle turns to the upper right and writes along the arc segment ${T_2}^{\prime }{T_3}^{\prime }$. Assuming that the pressing height and inclined angle of the brush pen remain unchanged during this process, it continues writing along ${T_3}^{\prime }{T_4}^{\prime }$ after reaching point ${T_3}^{\prime }$. The control model for the “zhebi” method of brush tip at point $T_2$ is shown in Equations (12) and (13).

$$T_2{T_3}^{\prime }=d\overrightarrow{T_2{T}_4}/‖\overrightarrow{T_2{T}_4}‖$$

(12)

$${\theta }_r=〈\overrightarrow{T_1{T}_2},\overrightarrow{T_2{T}_4}〉$$

(13)

where ${\theta }_r$ represents the rotational angle of the brush handle around the axis of rotation.

3.2. Control Method of the Brush Stroke Model

The writing models of different control methods at given points have been deduced above. However, to generate the control model of strokes, the trajectory control model based on the brush stroke model is needed. Assuming the trajectory points are $P_{i-1}{P}_i{P}_{i+1}$ in sequence, the vector of the brush stroke model CCD-BSM is shown in Equation (14), and its control model is illustrated in Figure 5.

$$\sigma =(\overrightarrow{P_i{P}_{i+1}},w_i,\Delta w_i,{\theta }_i,\Delta {\theta }_i)$$

(14)

$$w_i=AB$$

(15)

$$\Delta w_i=CD-AB$$

(16)

$${\theta }_i=〈\overrightarrow{P_{i-1}{P}_i},\overrightarrow{P_i{P}_{i+1}}〉$$

(17)

$$\Delta {\theta }_i=〈\overrightarrow{P_{i-1}{P}_i},\overrightarrow{P_{i+1}{P}_{i+1}^{\prime }}〉$$

(18)

where $P_i$ and $P_{i+1}$ represent the midpoints of $AB$ and $CD$, respectively. $w_i$ and $\Delta w_i$ indicate the width of the stroke and the change in width, respectively. $\overrightarrow{P_{i-1}{P}_i}\perp \overrightarrow{AB}$, $\overrightarrow{P_{i+1}{P}_{i+1}^{\prime }}\perp \overrightarrow{CD}$. The input angle and the rotation angle of the brush tip are represented by ${\theta }_i$ and $\Delta \theta$, respectively.

As shown in Figure 5, the graphic formed on the paper along the writing trajectory $P_i{P}_{i+1}$ is the ABCD portion of the brush stroke model (the region enclosed by red straight and curved lines), where $P_i$ and $P_{i+1}$ represent the midpoints of AB and CD, respectively, and ${P^{\prime }}_i$ and ${P^{″}}_{i+1}$ lie on the extensions of $P_{i-1}{P}_i$ and $P_i{P}_{i+1}$. The writing process of this brush stroke model is considered as the brush tip performing a transition to the lower right at point $P_i$, making $P_i{P}_{i+1}$ the new writing direction. After writing along $P_i{P}_{i+1}$ to $P_{i+1}$, the brush tip transitions again to the lower right, and writes along the direction of $P_{i+1}$${P^{\prime }}_{i+1}$.

When the brush tip moves to $P_i$ along the trajectory $P_{i-1}{P}_i$, the brush handle is located at point ${P^{\prime }}_i$. As the brush moves along ${P^{\prime }}_i{P^{″}}_i$, the brush tip rotates by ${\theta }^{\prime }r$ according to Equation (19). The brush handle then proceeds from ${P^{″}}_i$ to ${P^{″}}_{i+1}$ along ${P^{″}}_i{P^{″}}_{i+1}$ with a constant descending height and tilt angle, satisfying the relationship described by Equation (20). As the width of the stroke transitions from AB to CD due to the “zhebi”, the brush handle continues its movement along path ${P^{″}}_{i+1}$${P^{\prime }}_{i+1}$ until reaching the point ${P^{\prime }}_{i+1}$, with the rotation angle of the brush handle ${\theta }^{″}r$ shown in Equation (21).

$${{\theta }^{\prime }}_r=〈\overrightarrow{P_i{P^{\prime }}_i},\overrightarrow{P_i{P}_{i+1}}〉$$

(19)

$$\overrightarrow{{P^{″}}_i{P^{″}}_{i+1}}=\overrightarrow{P_i{P}_{i+1}}$$

(20)

$${{\theta }^{″}}_r=〈\overrightarrow{P_{i+1}{P^{″}}_{i+1}},\overrightarrow{P_{i+1}{P^{\prime }}_{i+1}}〉$$

(21)

where ${{\theta }^{\prime }}_r$ and ${{\theta }^{″}}_r$ represent the rotation angles of the brush tip and brush handle, respectively.

The complete writing process is the superposition of the aforementioned three segments. In the extremely small case of $P_i{P}_{i+1}$, the three segments can be treated as one segment. Consequently, the control of the brush handle can be simplified as moving the brush handle directly from ${P^{\prime }}_i$ to ${P^{\prime }}_{i+1}$ along $P{\prime }_i$${P^{\prime }}_{i+1}$, as illustrated in Figure 5. The control relationship can be derived as shown in Equation (22).

$$\overrightarrow{{P^{\prime }}_i{P^{\prime }}_{i+1}}=\overrightarrow{P_i{P}_{i+1}}+\overrightarrow{P_{i+1}{P^{\prime }}_{i+1}}-d\overrightarrow{P_{i-1}{P}_i}/‖\overrightarrow{P_{i-1}{P}_i}‖$$

(22)

$${\theta }_r=〈\overrightarrow{P_{i-1}{P}_i},\overrightarrow{P_{i+1}{P^{\prime }}_{i+1}}〉$$

(23)

$$\Delta H=w^{-1}\left(CD\right)-w^{-1}\left(AB\right)$$

(24)

where ${\theta }_r$ represents the rotation angle of the brush handle, and $\Delta H$ denotes the variation in the descending height of the brush.

The control vector based on the brush stroke model is defined as shown in Equation (25). Thus, the brush movement corresponding to Figure 5 can be derived as expressed in Equation (26).

$$M=\left(\overrightarrow{v},\Delta H,\Delta \theta \right)$$

(25)

where $\overrightarrow{v}$ represents the vector of the brush handle on the paper plane based on CCD-BSM, and $\Delta H$ represents the variation in the descending height of the brush. $\Delta \theta$ denotes the rotation angle of the brush handle.

$$M_i=\left(\overrightarrow{{P^{\prime }}_i{P^{\prime }}_{i+1}},w^{-1}\left(CD\right)-w^{-1}\left(AB\right),〈\overrightarrow{P_{i-1}{P}_i},\overrightarrow{P_{i+1}{P^{\prime }}_{i+1}}〉\right)$$

(26)

3.3. Writing Trajectory Control Models of Strokes

Based on the brush stroke model, the stroke width of the writing trajectory is determined by both the parameters of the CCD-BSM and the writing trajectory control model during the actual writing process.

3.3.1. Control Model During the Phase of Stroke Execution

The writing trajectory control of the brush stroke model during this phase is shown in Figure 6, with the vector representation described by Equation (27).

$${\sigma }_i=\left(\overrightarrow{P_i{P}_{i+1}},w_i,w_{i+1}-w_i,〈\overrightarrow{P_{i-1}{P}_i},\overrightarrow{P_i{P}_{i+1}}〉,〈\overrightarrow{P_{i-1}{P}_i},\overrightarrow{P_i{P}_{i+1}}〉\right)$$

(27)

By substituting Equations (29)–(31), the writing trajectory control parameters can be obtained. Thus, the control sequence for the phase of “xingbi” can be derived as shown in Equation (28).

$$M_i=\left(\overrightarrow{{P^{\prime }}_i{P^{\prime }}_{i+1}},\Delta H_i,\Delta {\theta }_i\right)$$

(28)

$$\overrightarrow{{P^{\prime }}_i{P^{\prime }}_{i+1}}=\overrightarrow{P_i{P^{\prime }}_{i+1}}-d‖\overrightarrow{P_{i-1}{P}_i}‖/‖\overrightarrow{P_{i-1}{P}_i}‖$$

(29)

$$\Delta {\theta }_i={\theta }_r=〈\overrightarrow{P_{i-1}{P}_i},\overrightarrow{P_{i+1}{P^{\prime }}_{i+1}}〉$$

(30)

$$\Delta H_i=w^{-1}\left(CD\right)-w^{-1}\left(AB\right)$$

(31)

where $\Delta H_i$ denotes the variation in the descending height of the brush. $\Delta {\theta }_i$ represents the rotation angle of the brush handle.

3.3.2. Control Model During the Phase of Stroke Initiation

The stroke initiation phase is typically composed of multiple brush stroke models, corresponding to multiple writing units. Based on the different writing methods, the motion of the “nifengrubi” of the brush handle is formed by the combinations of ${\sigma }_{FA}$, ${\sigma }_A$, ${\sigma }_{AH}$ and ${\sigma }_{HI}$, while the motion of the “shunfengrubi” of the brush handle is composed of the combinations of ${\sigma }_A$, ${\sigma }_{AH}$ and ${\sigma }_{HI}$, as shown in Figure 7. The points $A^{\prime }$ and $H^{\prime }$ are taken on the extensions of lines FA and AH, respectively, such that ${AA}^{\prime }={HH}^{\prime }=d$; $IJ\perp CD$. Additionally, the point $I^{\prime }$ is taken on line IJ, such that ${II}^{\prime }=d$.

• “Nifengrubi”

“Nifengrubi” can be considered as the superposition of two linear brush movements, FA and AH. In the first segment, the brush handle moves from the initial position point F along the straight line FA to point $A^{\prime }$, while the descending height of the brush gradually increases. The corresponding vectors are described in Equations (32) and (33), respectively.

$${\sigma }_{FA}=\left(\overrightarrow{FA},0,w_{\min },0,0\right)$$

(32)

$$M_{{FA}^{\prime }}=\left(\overrightarrow{{FA}^{\prime }},\Delta h_{FA},0\right)$$

(33)

Due to the characteristics of the “nifengrubi”, in the second segment, the brush handle moves from point $A^{\prime }$ back to A, and the descending height of the brush decreases from $w_{\min }$ to 0. The corresponding vectors for this segment are described in Equations (34) and (35), respectively.

$${\sigma }_A=\left(\overrightarrow{{\delta }_A},w_{\min },-w_{\min },0,0\right)$$

(34)

$$M_{A^{\prime }A}=\left(\overrightarrow{A^{\prime }A},-\Delta h_{FA},0\right)$$

(35)

$$\overrightarrow{{AA}^{\prime }}=d\overrightarrow{FA}/‖\overrightarrow{FA}‖$$

(36)

$$\overrightarrow{{FA}^{\prime }}=\overrightarrow{FA}+\overrightarrow{{AA}^{\prime }}$$

(37)

where $\overrightarrow{{\delta }_A}$ represents the direction vector of $IH$, and $‖\overrightarrow{{\delta }_A}‖\to 0$.

2.

“Shunfengrubi” forward-tip entry

In the case of “Shunfengrubi”, the brush tip starts directly from point A, omitting the first two stages of the reverse tip entry.

3.3.3. Control Model During the Phase of Stroke Termination

The stage of the stroke termination is also typically composed of multiple brush stroke models, corresponding to several brush stroke units. Based on the analysis of the brush stroke movement during the stage of stroke termination, the movement of the brush handle is composed of the combination of ${\sigma }_{IH}$, ${\sigma }_{HA}$ and ${\sigma }_{AF}$. Figure 8 illustrates the brush movements during the stage of “shoubi” termination. Take points $H^{\prime }$, $A^{\prime }$ and $F^{\prime }$ on the IH extension line, HA extension line and AF extension line, respectively, so that ${HH}^{\prime }={AA}^{\prime }={FF}^{\prime }=d$; ${II}^{\prime }$ connects to the brush stroke movements of the stroke execution phase, and ${II}^{\prime }=d$.

Similar to the stroke initiation phase, the control of the brush handle during the stroke termination phase mainly includes two different methods: “changfengshoubi” and “loufengshoubi”—hidden-tip termination and exposed-tip termination.

• “Changfengshoubi”—Hidden-Tip Termination

In hidden-tip termination, the “bifeng” (the brush tip) moves from point A to point F, while the brush handle moves from point $A^{\prime }$ to point $F^{\prime }$. Simultaneously, the brush is gradually lifted, and at point $F^{\prime }$, the tip of the brush pen is just above the paper surface. The corresponding brush stroke model vector ${\sigma }_{AF}$ and brush movement $M_{{AF}^{\prime }}$ are described in Equations (38) and (39), respectively.

$${\sigma }_{AF}=\left(\overrightarrow{AF},w_{\min },-w_{\min },0,0\right)$$

(38)

$$M_{{AF}^{\prime }}=\left(\overrightarrow{{AF}^{\prime }},-h_{\min },0\right)$$

(39)

$$\overrightarrow{A^{\prime }{F}^{\prime }}=\overrightarrow{{AF}^{\prime }}-\overrightarrow{{AA}^{\prime }}$$

(40)

$$\overrightarrow{{AF}^{\prime }}=\overrightarrow{AF}+\overrightarrow{{FF}^{\prime }}$$

(41)

$$\overrightarrow{{FF}^{\prime }}=d\overrightarrow{AF}/‖\overrightarrow{AF}‖$$

(42)

where $w_{\min }$ can be derived from the descending height $h_{\min }$ of the brush pen (based on CCD-BSM stroke model).

2.

“Loufengshoubi”—Exposed-Tip Termination

In the “loufengshoubi”, the brush tip directly leaves the paper surface from point A, skipping the latter two stages of hidden-tip termination.

4. Experimental Results and Discussion

4.1. Experimental Hardware System

In this system, we used Aubo-I5 script programming. Based on a high-precision robotic writing system, a collaborative 6-DOF robot “Aubo-I5” with repeatability of ±0.05 mm was adopted. The writing results were collected using the robotic writing system and a normal camera. The configuration of our experimental hardware system is shown in Figure 9. The AUBO-I5 robot is a rotationally articulated robot that satisfies the composite Pieper criterion, and the parameters corresponding to each joint angle can be obtained by using a hybrid closed solution that satisfies the Pieper criterion. Based on the robotic manipulator maps, the robot’s movement can be controlled to complete the writing operation.

In the experiment, two pens with different specifications were used and affixed to the end-effector of the robot during a writing test. The length and the radius of the brush bundle of pen 1 were 48 mm and 6 mm, respectively, while the length and the radius of the brush bundle of pen 2 were 32 mm and 4.5 mm, respectively.

4.2. Robot Writing Test

4.2.1. Writing Test of Basic Strokes

The writing results for the stroke “horizontal” are shown in Figure 10. From left to right, these are the reference stroke, the strokes written when the direction of the brush tip is opposite to the writing direction of the horizontal stroke, and the writing result when the direction of the brush tip is the same as the writing direction of the horizontal stroke.

The writing results of the stroke “vertical” are shown in Figure 11. To achieve the writing effect of reference strokes, the process of the robot writing the stroke “vertical” should follow the rule of “intending to go down, first go up”.

Figure 12 and Figure 13 show the writing results of strokes “short left-falling” and “long left-falling” respectively.

4.2.2. Writing Test of Chinese Characters

Taking the Yan-style calligraphy characters “bu”, as well as the regular script character “qu”, as examples, Figure 14 shows the actual writing result of the character “bu”. From left to right, these are reference calligraphy images, the writing results without the “yunbi” methods and the writing results after incorporating the “yunbi” methods. As can be seen from Figure 15, the writing result improved after adding the “yunbi” methods.

Figure 15 shows the writing result of the character “qu”. Figure 15a is the reference calligraphy image, while Figure 15b,c show the actual writing images by the robotic arm based on the same parameter settings.

4.2.3. Writing Test with Different Brush Pen

To verify that the writing robot has a good writing ability with different brushes, we conducted a writing test by replacing a different type of brush (No. 2 brush) and evaluated the performance. Figure 16 and Figure 17 show the writing result of the character “bu” and “qu” with the No. 2 brush, respectively. In terms of the overall structure, the Chinese characters written with the No. 2 brush are similar to the reference Chinese characters after applying the three steps of “yunbi”, which indicated that the model has good generalization and robustness for different pens.

4.3. Performance Evaluation

4.3.1. Writing Speed Evaluation

When the robot performs writing tasks, the range of writing speed variation can reflect the stability of the writing process. A smaller speed variation interval indicates a more stable writing process. To further demonstrate the stability of the writing process, we evaluated speed variations during the writing of five basic strokes “horizontal”, “vertical”, “short left-falling”, “long left-falling” and “right-falling”.

The planning writing positions and postures were applied to conduct 100 experimental tests on five strokes, and the results are shown in

Table 1. Writing speed changes of basic strokes (mm/s).

Strokes	Maximum Speed	Minimum Speed	Range of Variation
“Horizontal”	22.87	21.32	1.55
“Short left-falling”	20.15	19.54	0.61
“Vertical”	20.28	19.81	0.47
“Long left-falling”	21.66	20.23	1.43
“Right-falling”	22.72	20.95	1.77

. The speed variation ranges of all five strokes were relatively small, with the speed variation range for the stroke “vertical” being only 0.47 mm/s. Figure 18 shows the writing results of strokes “right-falling” at maximum and minimum speed, because it has the largest range of variation. From Figure 18, we can see the results are close to each other at two boundary values. This indicated that the robot could move at a stable speed and guarantee the quality during the writing process, and there will be no impacts caused by sudden speed changes.

4.3.2. Comparison with Other Model Algorithms

This paper compares the intelligent calligraphy robot system with four advanced robotic calligraphy systems and algorithms, which are the Generative adversarial nets based Chinese calligraphy (GANCC) Chinese calligraphy robot, the generative adversarial network—long short-term memory network (GAN-LSTM) calligraphy robot learning system based on generative adversarial network (GAN) and long short-term memory (LSTM) network, the GAN-AC network model, and the deep deterministic policy gradient (DDPG) algorithm-based generative model. The results of the qualitative comparison are shown in Figure 19. By comparing with the current optimal model and algorithm, the advantages of our proposed writing trajectory control model in stroke generation are demonstrated. Cosine similarity (CSIM) and Structural similarity index measure (SSIM) were used as specific quantitative criteria to evaluate the differences between the different results. The results of 100 experiments written by the robotic end-effector were quantitatively evaluated based on CSIM and SSIM, as shown in

Table 2. CSIM comparison with other model algorithms (%).

System, Model	“Horizontal”	“Short Left-Falling”	“Vertical”	“Long Left-Falling”	“Right-Falling”
Our model	99.52	99.17	99.54	99.46	98.76
DDPG	98.97	97.94	98.95	98.04	98.17
GAN-AC	98.85	98.31	97.61	98.58	97.34
GANCC	94.13	95.15	98.28	96.77	95.87
GAN-LSTM	97.86	98.06	98.37	98.66	97.80

and

Table 3. SSIM comparison with other model algorithms (%).

System, Model	“Horizontal”	“Short Left-Falling”	“Vertical”	“Long Left-Falling”	“Right-Falling”
Our model	97.17	93.44	97.57	97.11	92.55
DDPG	93.56	91.66	93.23	92.14	91.39
GAN-AC	94.43	92.35	89.33	93.32	89.94
GANCC	81.39	82.59	90.90	86.55	83.23
GAN-LSTM	92.18	92.21	91.03	93.41	90.85

. The average values of CSIM and SSIM of basic strokes written by the robot exceed those of other model algorithms, which indicates that the writing ability for basic strokes based on our proposed model conforms to brush the stroke model with better performance. Figure 20 shows the quantitative comparison results between our system and other models and algorithms.

4.3.3. Writing Similarity Evaluation

The results of 100 experiments were quantitatively evaluated based on CSIM and SSIM, as shown in

Table 4. CSIM and SSIM of basic strokes writing experiment (%).

Strokes	CSIM_Max	CSIM_Min	CSIM_Avg	SSIM_Max	SSIM_Min	SSIM_Avg
“Horizontal”	99.81	99.34	99.52	97.79	96.95	97.17
“Short left-falling”	99.31	99.06	99.17	94.32	93.16	93.44
“Vertical”	99.88	99.40	99.54	98.06	97.27	97.57
“Long left-falling”	99.72	99.29	99.46	97.40	96.69	97.11
“Right-falling”	99.01	98.33	98.76	93.56	92.20	92.55

. It records the best, worst and average similarity for each stroke. The average CSIM value for all five strokes exceeds 98%, which indicates that the writing ability for basic strokes is strong. It can be seen from Table 4 that the CSIM of the basic stroke “vertical” is the highest, with a maximum similarity of 99.88%.

According to values of CSIM and SSIM in Table 4, among the five strokes, the robot performs better in writing the strokes “horizontal”, “vertical” and “long left-falling”, followed by the stroke “short left-falling”, and is slightly weaker in writing the stroke “right-falling”.

To verify the generalization and robustness of the robotic calligraphy under the varying Chinese character structures and styles, the robot was controlled to conduct 100 writing experiments on Chinese characters with different structures and styles. The writing results were quantitatively evaluated based on the CSIM and SSIM similarity metrics, as shown in

Table 5. CSIM and SSIM of Chinese characters writing experiment (%).

Characters	CSIM_Max	CSIM_Min	CSIM_Avg	SSIM_Max	SSIM_Min	SSIM_Avg
“bu”	90.19	84.09	86.61	66.51	62.10	63.81
“qu”	95.89	86.08	88.73	73.32	64.94	68.73

. Table 5 recorded the similarity of each Chinese character written by the robotic end-effector in the best case, the worst case and the average similarity of all Chinese character images. As can be seen from Table 5, the average CSIM of Chinese characters written by the robot exceeds 85%. Based on the average values of CSIM and SSIM, it is evident that the robot based on the proposed model possesses a certain level of writing capability for Chinese characters with different structures.

4.4. Ablation Study

The ablation study was conducted to clarify the contributions of “zhuanbitiaofeng” and “zhebi” to the final quality. Figure 21 shows the writing results of strokes “right-falling” before and after “zhuanbitiaofeng”. Figure 22 shows the writing results of strokes “reclining hooks” before and after “zhebi”. Figure 22b shows the writing result without “zhebi”, while Figure 22c shows the result after incorporating “zhebi”. By adding “zhuanbitiaofeng” and “zhebi”, the final writing quality can be improved significantly.

5. Conclusions and Future Works

This paper primarily studied fine-grained control of robotic brush calligraphy and proposed a robotic writing trajectory control model based on a brush stroke model combined with the stroke rules of brush calligraphy. The speed variation ranges of basic strokes were relatively small, with the stroke “vertical” being only 0.47 mm/s. Compared with other existing methods, the average values of CSIM and SSIM of basic strokes and Chinese characters written by the robot exceed those of other model algorithms. The average CSIM value for all five strokes exceeds 98%, with a maximum similarity of 99.54%. For Chinese characters written by the robot, the average value of CSIM exceeds 85%. Furthermore, as the specifications of brush pens are taken into consideration, the generated strokes and characters under the same posture parameters were different when using pens with different sizes, which leads to good generalization and robustness for different pens.

In the future, more diverse control methods with hairy brushes will be further explored for robotic calligraphy to further improve the robot’s writing capabilities and replication effects. The approach with a machine vision system to self-learning a robotic arm using artificial intelligence will be studied to improve the quality of the models.