On the Use of the Humanoid Bioloid System for Robot-Assisted Transcription of Mexican Spanish Speech

Abstract

Within the context of service robotics (SR), the development of assistive technologies has become an important research field. However, the accomplishment of assistive tasks requires precise and fine control of the mechanic systems that integrate the robotic entity. Among the most challenging tasks in robot control, the handwriting task (transcription) is of particular interest due to the fine control required to draw single and multiple alphabet characters to express words and sentences. For language learning activities, robot-assisted speech transcription can motivate the student to practice pronunciation and writing tasks in a dynamic environment. Hence, this paper is aimed to provide the techniques and models to accomplish accurate robot-assisted transcription of Spanish speech. The transcriptor is integrated by a multi-user speech recognizer for continuous speech and the kinematic models for the Mexican Spanish alphabet characters. The Bioloid system with the standard humanoid configuration and no special modifications or tools was considered for implementation. Particularly, the proposed transcriptor could perform the handwriting task with the Bioloid’s two two DOF (degrees-of-freedom) arms. This enabled writing of one-line short and long sentences with small alphabet characters (width <1.0 cm). It is expected that the technique and models that integrate the transcriptor can provide support for the development of robot-assisted language learning activities for children and young adults.

1. Introduction

Within the educational context, robots can take the roles of learning tools, learning companions, teaching assistants and mediators to support group learning [1,2,3]. For language learning activities that involve pronunciation and spelling knowledge, the use of robots has been proposed [2]. In this regard, a system integrated by automatic speech recognition (for assessment of spoken words) and speech-to-text transcription (for the presentation of correct spelling) can provide support for language learning.

The inclusion of a robotic system (particularly a humanoid robot) within the learning task can improve the user experience and increase the learning rate. As mentioned in [4], in comparison with screen-based agents, robotic partners have been shown to improve users’ compliance with tasks [5], maintain more effective long-term relationships [6] and improve learning gains when acting as tutors [7]. Particularly, robots that resemble people (humanoid robots) motivate curiosity and are more engaging [2]: children treat robots that look and move like people as real speakers/listeners. Furthermore, the young students do not have to worry about the consequences (e.g., verbal punishment) of making pronunciation or syntax errors when interacting with the robot. This may lead to reduce students’ anxiety and improve the willingness to participate in the learning activity [2].

Handwriting is one of the main activities considered in language learning activities for children, because it provides the visual representation of spoken words. As discussed in [8], although there has been a relatively low status and lack of attention given to handwriting at the primary and secondary school levels, research has determined that handwriting in the form of automatic letter writing facilitates the higher order composing processes of text production. Hence, difficulties in handwriting skills at an early age often negatively affect academic performance and self-esteem [4,9,10].

In the technological context, handwriting is one of the most complex tasks performed by robots, because it requires fine control of servomechanisms. This control involves the dynamic aspects of the writing/drawing task, such as speed, force and position [11]. Human-like factors, as fatigue, are also explored in the control of robot arms for handwriting applications [12]. The accomplishment of the handwriting task has many important implications in other fields of human-robot interaction (HRI), such as teaching/transfer of handwriting skills [13,14,15], analysis and replication of human motion for calligraphy education [16], education and training of forensic handwriting experts [17], application in rehabilitation therapy [17] and communication support for people with neuromuscular disorders via a brain-machine interface (BMI) [18].

Some examples of robots used for handwriting tasks (calligraphy) of Chinese characters are presented in the literature. The robot presented in [19] was developed with advanced controllers (Mitsubishi Q-02 and Q-172-CPU for logic and drive control), servo-amplifiers (Mitsubishi MR-J2S-20B) and servo-motors (HC-KFS23). The use of a standard seven DOF KUKA light weight robot for the calligraphy task was presented in [20]. In [21], dynamic and kinematic models were developed for the calligraphy of Japanese characters. These models were tested with a simulated three DOF SCARA robot.

Handwriting of the Latin alphabet is commonly performed because it is the basic alphabet for many languages around the world. Some examples of robots performing writing of words with the Latin alphabet are presented in the literature with the SCARA system [22], the NAO system [23], and a CNC GFY 44/48 system [17]. Particularly, the system presented in [17] is able to reproduce human-written characters (signatures) enabling its use for forensic purposes. Research relevant to modeling (kinematics) and control for this task can be found in [11,12,17,24].

Although there are significant works on robot-assisted handwriting, most of these works (e.g., [17,19,20,22]) were performed with equipment that was not intended for natural HRI. A humanoid robot is considered more suitable, as it resembles the human appearance and, thus, can improve its integration within social contexts [2]. Within the context of humanoid robots, NAO is one of the most well-known robots used for HRI and certainly can perform writing tasks, as presented in [23]. However, its cost (approximately 8,000.00 USD as of 2014) makes it affordable just for some users with significant economic resources. On the other hand, the integration of robot-assisted handwriting within the educational context has not been explored in the literature. Nevertheless, the work presented in [4,15] has studied this situation, presenting a “learning by teaching” paradigm to improve handwriting skills in children. The NAO robot was used to perform trajectories that were expected to represent handwritten Latin alphabet characters and single words with different variability levels. However, these characters were not actually “written” by the robot. Instead, these characters were displayed in a tablet, which was synchronized with the NAO’s movements (robot trajectory was synchronized with the tablet trajectory display). The robot intentionally added variability to the trajectories while writing words observed in cards presented by the children. This variability represented typical mistakes of children learning handwriting. Then, children would evaluate the word written by the robot (displayed in the tablet) and present the correct (or most appropriate) writing. With this approach, children could improve their handwriting skills by teaching the robot the correct trajectories.

An issue regarding the handwriting task presented in [4,15] is that physical writing was not actually performed. While robot simulation is an attractive alternative for implementation, it is important to consider the practical conditions to achieve a reliable evaluation of the handwriting task. This is because elements of the real world, as the precision of the servomechanisms, texture of the writing plane and the type of pen, including the ink, affect the dynamics of the writing task.

In comparison with the NAO robot, the Bioloid system is a more affordable equipment for real implementations due to its cost (1200.00 USD as of 2015). Although no work related to handwriting with this system has been found in the literature, it has been widely used for similar purposes. In [25], the Bioloid was used to build a robotic arm for the task of drawing faces. This work is important for the purpose of writing, because it presented the use of the Bioloid to draw visual patterns. An advantage of the Bioloid system is that it can be configured into different architectures, including the humanoid architecture.

This paper is focused on providing the techniques and models to use the Bioloid system for handwriting of the Mexican Spanish language. Speech recognition is considered as the input channel for handwriting. It is expected that the resources presented in this paper can be used for the development of educational applications as those described in [2,4]. The contributions of the present work are listed as follows:

(a)

development of a physical speech transcription system by means of robot-assisted handwriting;

(b)

performance evaluation of the humanoid Bioloid to draw small (1.0 cm width) alphabet characters in curved lowercase format;

(c)

development of the kinematic models for the humanoid Bioloid to perform the drawing of the Mexican Spanish alphabet with both arms;

(d)

development of a two-arm writing scheme for the transcription (handwriting) of large spoken sentences (multiple words).

This paper is structured as follows: In Section 2, an overview of the Bioloid system is presented. Then, in Section 3, the details of the speech-to-text transcriptor are presented. These details include the development of the speech recognizer, the kinematic models of the Mexican alphabet and the graphical user interface (GUI) that integrated all elements. The performance evaluation of the transcriptor is presented in Section 4. Finally, the conclusions and future work are presented in Section 5.

2. The Bioloid System

The Bioloid system is an educational robot kit produced by the South Korean company Robotis [26]. It consists of components and small modular servomechanisms (AX-12A Dynamixel) that can be assembled to construct robots of different configurations. This system had been popular in the RoboCup international robotics competition [27].

In this work, the Premium version of the Bioloid kit (CM-530) was used. Figure 1 presents the Type-B humanoid Bioloid used in this work [28]. The robot was named Y1T0 (“yito”) and was integrated by 16 Dynamixel servomechanisms. Because each Dynamixel is an independent mechanism with independent motions, each servomechanism represents a degree-of-freedom (DOF). Hence, the Type-B Bioloid was a 16 DOF humanoid robot. In Figure 1, the array of Dynamixel servomechanisms (dyn1–dyn8 and dyn11–dyn18) used for the Type-B Bioloid is presented.

Figure 1. Dynamixel array (dyn1–8, dyn11–18) of the B-Type humanoid Bioloid.

Figure 1. Dynamixel array (dyn1–8, dyn11–18) of the B-Type humanoid Bioloid.

Programming Platform

The Bioloid kit includes a CD-ROM with its programming platform (RoboPlus), which is based on C-programming and consists of three main software utilities:

(a)

RoboPlus Task: This utility is used for the development of “task code” (e.g., programming code) for management and control of the Bioloid’s hardware resources. Sequences of motions can be coordinated by specific task codes, and complex functions or actions can be accomplished by the coordination of task codes.

(b)

RoboPlus Manager: This utility is used for the maintenance of the Bioloid’s hardware components. Hence, major functions of this utility are updating and restoration of the controller’s firmware and testing of the controller and peripheral devices (servomechanisms, sensors).

(c)

RoboPlus Motion: This utility is used for the development of “step” motions for the Bioloid. These motions then can be managed by task code.

While the RoboPlus software enables the user to develop applications quickly for demonstration purposes, it presents some limitations for the proposed transcription system:

(a)

It only allows the development of standalone applications.

(b)

There is no direct integration with other programming languages.

(c)

Motions created with RoboPlus Motion can only be accessed via RoboPlus Task code.

(d)

The creation scheme for motions, which is based on “steps”, is suitable for the realization of medium-to-large straight or curved trajectories, such as those described in [28]. However, this scheme is not suitable for the realization of fine movements or trajectories required to draw small and curved lowercase alphabet characters.

MATLAB programming was considered as an alternative for this work. In order to have control over the Bioloid’s Dynamixel servomechanisms (and implement the kinematic models), the toolbox developed by the Incognite research group [29] was used. Particularly, the following functions of the toolbox were considered: setSerialPort for enabling the connection of the Bioloid to the PC and MATLAB, setMovingSpeed for setting the speed of each Dynamixel and setGoalPosition for setting the position of each Dynamixel (position ranges from 0–1024). More details about the use of these functions for the transcription task are presented in Section 3.2.

3. Speech-to-Text Transcriptor

As presented in Figure 2, the speech transcriptor is integrated by the following modules: (1) multi-user continuous speech recognizer; and (2) alphabet transcriptor. These modules are explained in the following sections.

3.1. Speech Recognizer and Alphabet Segmentation

The automatic speech recognizer (ASR system) used in this work is a modification of the base system presented in [30] for Mexican Spanish. The base system enables the user to perform the continuous entry of new words and dynamic speaker adaptation. It can recognize isolated words and sentences with an overall recognition accuracy for male and female speakers over 90% with a vocabulary of less than 500 words [30]. The recognition engine was developed with HTK (Version 3.4) [31], and the acoustic modeling was performed at the phonetic level with the set of phonemes defined in [32].

In Figure 3, the graphical user interface (GUI) of the speech-to-text transcriptor is presented. The interface itself integrates the speech recognizer and the alphabet transcriptor for the decoded speech. The GUI was programmed with the GUIDE toolkit of MATLAB.

Figure 2. Modules of the speech-to-text transcriptor.

Figure 2. Modules of the speech-to-text transcriptor.

Figure 3. Main interface of the speech-to-text transcriptor.

Figure 3. Main interface of the speech-to-text transcriptor.

From Figure 3, the button labeled “Registro y Adaptacion de Usuario” (User Registration and Adaptation) leads to another interface that enables the user to adapt the speech recognizer to his/her voice patterns. This interface is presented in Figure 4 and includes fields to add a new user’s name (“Escriba Nombre de Usuario”) or select an already registered user (“Seleccione Usuario”). As the new user is added, or a previously-registered user is selected, he/she can proceed to perform the speaker adaptation process. For this purpose, the user must record 16 sentences. This is done by pressing the button labeled with the associated sentence (e.g., “MI FAMILIA VIVIO EN MEXICO”). By doing this, the user provides speech samples for the adaptation process, which is realized and finalized by pressing the button “Adaptar” (to adapt). More details of this interface can be found in [30].

Figure 4. Speech adaptation interface of the speech-to-text transcriptor.

Figure 4. Speech adaptation interface of the speech-to-text transcriptor.

The button labeled “Reconocedor y Transcriptor de Voz” (Speech Recognizer and Transcriptor) leads to the interface of the speech-to-text transcriptor. This interface is presented in Figure 5. The first step to use this interface is to select an already registered user by using the pop-up menu “Seleccione Usuario”. Then, the user can proceed to either add new vocabulary by writing it down in the field “Añadir Nuevas Frases o Palabras” or use the recognizer. If new vocabulary is added, the user can use this vocabulary as stimuli to perform additional speaker adaptation: the user records the pronunciation of the added sentence by pressing “Grabar para Adaptacion” and then, by pressing the “OK” button, the interface realizes and finishes the process. The additional vocabulary is updated into the ASR system by pressing the “Crea Modelo de Lenguaje” button. If the user wants to use the recognizer, he/she must first define an adjustment value for the recognizer under “Valor de Ajuste (1-30):”. This value represents the scale grammar factor and controls the influence of the language model (statistical information of the vocabulary) on the recognition process [30,31]. Then, by pressing “Ejecuta Reconocedor”, the recording of the user’s speech is started. When the user has finished this step, he/she just needs to press the button again. By doing this, the ASR starts the decoding process, and the recognized speech is displayed under the field “Frase Reconocida” (Recognized Sentence).

Figure 5. Speech recognition interface of the speech-to-text transcriptor.

Figure 5. Speech recognition interface of the speech-to-text transcriptor.

In order to perform the transcription with the Bioloid robot (e.g., robot-assisted handwriting), the user just has to press the button “Transcripcion de Frase” (Transcription of Sentence). Internally, the interface performs a direct alphabet segmentation from uppercase to lowercase format of the words’ alphabet characters displayed under “Frase Reconocida” (e.g., “LA_MAÑANA_ES_FRIA” → “la mañana es fria”). The symbol “_” is considered as an “empty” (space) alphabet character. When this segmentation is performed, each character’s model is selected from a database to perform its actual writing/drawing with the Bioloid robot (execution of kinematic models). The development of this database is presented in Section 3.2.

3.2. Database of Alphabet Models

3.2.1. Writing Space

The first step to develop the alphabet kinematic models was to define the size of the alphabet characters based on the reachable space of each robotic arm for the writing task. In Figure 6, the writing space (or writing region) of the right robotic arm is presented, identifying a space of 3.0 × 9.0 cm. The same space was identified for the left arm.

Figure 6 was also important to identify the x-y coordinate system for the development of the kinematic models. The length of the robotic arm consists of the links $L_1$ (7.0 cm), which is attached at the shoulder, and $L_2$ (6.0 cm), which is attached at the elbow.

3.2.2. Inverse Kinematics

The second step was to define the alphabet characters to be handwritten. In Figure 7, the alphabet considered in this work is presented. Each character can be drawn over a space defined by a set of points labeled from A–U, as presented in Figure 8. The height of this space is restricted by the limit specified in Figure 6 (3.0 cm).

Figure 6. Reachable x-y coordinates for all ${\theta }_1$ and ${\theta }_2$ combinations using forward kinematics equations.

Figure 6. Reachable x-y coordinates for all ${\theta }_1$ and ${\theta }_2$ combinations using forward kinematics equations.

Figure 7. Mexican Spanish alphabet.

Figure 7. Mexican Spanish alphabet.

Figure 8. Point-based approach for alphabet drawing (writing).

Figure 8. Point-based approach for alphabet drawing (writing).

Any character presented in Figure 7 can be drawn based on a reference (center) coordinate (given by the point K) and a radius (r). The width of any character is equal to 2r, and the height of any character can be up to 4r, as deduced from Figure 7 and Figure 8.

Considering the writing line as defined by the points M-N-O, the height of the writing space can be expressed in terms of the radius as 6r. This is due to some characters having a height of 4r above the writing line (e.g., h, l, k) and some others extending 2r below the writing line (e.g., g, j, p). Curves and lines can be used to draw the alphabet characters by just considering the reference coordinate $\left[x_0,y_0\right]$ (point K) and the radius/width (r). As presented in Figure 8, from $\left[x_0,y_0\right]$, the coordinates for all other points can be computed and used as start-end points for drawing the segments that form each character. Consecutive characters can be drawn (written) by updating the reference coordinate given by K.

For the task of drawing lines, just “start” and “end” points are required [28]. However, for the task of drawing curves, these points are given by angular ranges in terms of π. Table 1 presents the set of points and angles that were defined to draw the segments for each alphabet character presented in Figure 7.

Table 1. Start-end points for the creation of the Mexican Spanish alphabet characters: (1) = x coordinate of a point, (2) = y coordinate of a point.

Alphabet	Straight Lines		Curved Lines-Circumferences
	Start Point	End Point	Center, Radius	Angular Range
a	I, where I(2) = I(2) − r/2	O	K, r	$0\le \theta \le 2\pi$
b	A	M	K, r	$0\le \theta \le 3\pi$
c			K, r	$\pi /4\le \theta \le 7\pi /4$
d	C	O	K, r	$0\le \theta \le 2\pi$
e	J	L	K, r	$0\le \theta \le 7\pi /4$
f	D	M	E, r	$\pi /4\le \theta \le \pi$
	G	I
g	R	I	Q, r	$\pi \le \theta \le 2\pi$
			K, r	$0\le \theta \le 2\pi$
h	O	L	K, r	$0\le \theta \le \pi$
	M	A
i	N	H
	H, where H(2) = H(2) + 0.2	H, where H(2) = H(2) + 0.3
j	R	I	Q, r	$\pi \le \theta \le 2\pi$
	I, where I(2) = I(2) + 0.2	I, where I(2) = I(2) + 0.3
k	A	M
	E	G
	G	O
l	B	N
ll	B, where B(1) = B(1) − 0.3	N, where N(1) = N(1) − 0.3
	B, where B(1) = B(1) + 0.3	N, where N(1) = N(1) + 0.3
m	O	I, where I(2) = I(2) − r/2	K, where K(1) = K(1) + r/2 and K(2) = K(2) + r/2, r = r/2	$0\le \theta \le \pi$
	N	H, where H(2) = H(2) − r/2	K, where K(1) = K(1) − r/2 and K(2) = K(2) + r/2, r = r/2	$0\le \theta \le \pi$
	G	M
n	O	L	K, r	$0\le \theta \le \pi$
	G	M
ñ	O	L	K, r	$0\le \theta \le \pi$
	G	M
	G, where G(2) = G(2) + 0.2	I, where I(2) = I(2) + 0.2
o			K, r	$0\le \theta \le 2\pi$
p	G	S	K, r	$0\le \theta \le 2\pi$
q	I	U	K, r	$0\le \theta \le 2\pi$
r	G	M	K, r	$\pi /4\le \theta \le \pi$
s	I	G, where G(1) = G(1) + r/2	K, where K(1) = K(1) − r/2 and K(2) = K(2) + r/2, r = r/2	$\pi /2\le \theta \le 3\pi /2$
	J, where J(1) = J(1) + r/2	L, where L(1) = L(1) − r/2	K, where K(1) = K(1) + r/2 and K(2) = K(2) − r/2, r = r/2	$3\pi /2\le \theta \le 5\pi /2$
	M	O, where O(1) = O(1) − r/2
t	B	N
	G	I
u	G	J	K, r	$\pi \le \theta \le 2\pi$
	L	I
v	G	N
	N	I
w	G	N, where N(1) = N(1) − r/2
	N, where N(1) = N(1) − r/2	H
	H	O, where O(1) = O(1) − r/2
	O, where O(1) = O(1) − r/2	I
x	G	O
	I	M
y	G	N
	I	S
z	G	I
	I	M
	M	O

Table 1. Start-end points for the creation of the Mexican Spanish alphabet characters: (1) = x coordinate of a point, (2) = y coordinate of a point.

Alphabet	Straight Lines		Curved Lines-Circumferences
	Start Point	End Point	Center, Radius	Angular Range
a	I, where I(2) = I(2) − r/2	O	K, r	$0\le \theta \le 2\pi$
b	A	M	K, r	$0\le \theta \le 3\pi$
c			K, r	$\pi /4\le \theta \le 7\pi /4$
d	C	O	K, r	$0\le \theta \le 2\pi$
e	J	L	K, r	$0\le \theta \le 7\pi /4$
f	D	M	E, r	$\pi /4\le \theta \le \pi$
	G	I
g	R	I	Q, r	$\pi \le \theta \le 2\pi$
			K, r	$0\le \theta \le 2\pi$
h	O	L	K, r	$0\le \theta \le \pi$
	M	A
i	N	H
	H, where H(2) = H(2) + 0.2	H, where H(2) = H(2) + 0.3
j	R	I	Q, r	$\pi \le \theta \le 2\pi$
	I, where I(2) = I(2) + 0.2	I, where I(2) = I(2) + 0.3
k	A	M
	E	G
	G	O
l	B	N
ll	B, where B(1) = B(1) − 0.3	N, where N(1) = N(1) − 0.3
	B, where B(1) = B(1) + 0.3	N, where N(1) = N(1) + 0.3
m	O	I, where I(2) = I(2) − r/2	K, where K(1) = K(1) + r/2 and K(2) = K(2) + r/2, r = r/2	$0\le \theta \le \pi$
	N	H, where H(2) = H(2) − r/2	K, where K(1) = K(1) − r/2 and K(2) = K(2) + r/2, r = r/2	$0\le \theta \le \pi$
	G	M
n	O	L	K, r	$0\le \theta \le \pi$
	G	M
ñ	O	L	K, r	$0\le \theta \le \pi$
	G	M
	G, where G(2) = G(2) + 0.2	I, where I(2) = I(2) + 0.2
o			K, r	$0\le \theta \le 2\pi$
p	G	S	K, r	$0\le \theta \le 2\pi$
q	I	U	K, r	$0\le \theta \le 2\pi$
r	G	M	K, r	$\pi /4\le \theta \le \pi$
s	I	G, where G(1) = G(1) + r/2	K, where K(1) = K(1) − r/2 and K(2) = K(2) + r/2, r = r/2	$\pi /2\le \theta \le 3\pi /2$
	J, where J(1) = J(1) + r/2	L, where L(1) = L(1) − r/2	K, where K(1) = K(1) + r/2 and K(2) = K(2) − r/2, r = r/2	$3\pi /2\le \theta \le 5\pi /2$
	M	O, where O(1) = O(1) − r/2
t	B	N
	G	I
u	G	J	K, r	$\pi \le \theta \le 2\pi$
	L	I
v	G	N
	N	I
w	G	N, where N(1) = N(1) − r/2
	N, where N(1) = N(1) − r/2	H
	H	O, where O(1) = O(1) − r/2
	O, where O(1) = O(1) − r/2	I
x	G	O
	I	M
y	G	N
	I	S
z	G	I
	I	M
	M	O

Knowing the trajectories required to draw the alphabet characters is just part of the writing problem. Knowing how to control the angles of the arm’s links $L_1$ and $L_2$ (${\theta }_1$ and ${\theta }_2$) in order to perform such trajectories is the most important part. This is addressed by solving the inverse kinematics problem that involves converting the position (x-y coordinates) and orientation of a manipulator end-effector from the Cartesian space to the joint space (angular domain) [33].

The human hand has approximately 19 DOF; however, these can be reduced to two translations for the handwriting task [24]. If no wrist joint is considered these translations can be performed by a standard two DOF planar arm. Because the Bioloid’s arms resemble standard two DOF planar arms, the inverse kinematics for the Bioloid’s arms can be solved with the kinematic models of a standard two DOF robot arm. By applying the geometric approach for solving the inverse kinematics problem [33,34], any straight line can be drawn as follows:

(a)

Define the start (s) and end (e) x-y points: $P_s=[x_s,y_s]$, $P_e=[x_e,y_e]$.

(b)

Define the vector X with $N_l$ equally-spaced points between $x_s$ and $x_e$.

(c)

Define the vector Y with $N_l$ equally-spaced points between $y_s$ and $y_e$.

(d)

Because X and Y are equally-sized vectors, each [X(i), Y(i)], where i = 1, ..., $N_l$ represents the coordinate of the $N_l$-th x-y point between $P_s$ and $P_e$.

(e)

For each [X(i), Y(i)], compute the inverse kinematics to obtain the associated ${\theta }_1$ and ${\theta }_2$ angles:

$$c{\theta }_2=\frac{\mathbf{X}{\left(i\right)}^2+\mathbf{Y}{\left(i\right)}^2-L_1^2-L_2^2}{2L_1{L}_2}$$

(1)

For the right arm:

$$s{\theta }_2=+\sqrt{1-c{\theta }_2^2}$$

(2)

For the left arm:

$$s{\theta }_2=-\sqrt{1-c{\theta }_2^2}$$

(3)

Then, ${\theta }_2$ can be computed with the arctangent function as:

$${\theta }_2=Atan2(s{\theta }_2,c{\theta }_2)$$

(4)

${\theta }_1$ is computed as follows:

$$k_1=L_1+L_{2}c\theta 2$$

(5)

$$k_2=L_{2}s\theta 2$$

(6)

$${\theta }_1=Atan2(\mathbf{Y}\left(i\right),\mathbf{X}\left(i\right))-Atan2(k_2,k_1)$$

(7)

By applying the same geometric approach, any curved line can be drawn as follows:

(a)

Define the reference (center) x-y point for the curve (e.g., K): $C_c=[x_c,y_c]$

(b)

Define the vector α with $N_c$ equally-spaced angular points between zero and 2π. This vector defines the angular/rotation range for the radius r, which is used as follows: to draw the “o” character, a complete circle must be performed; thus, α must vary from zero to 2π (angular range = $0\le \alpha \le 2\pi$). For other characters, range fractions may be used: to draw the “c” character, α should vary from π/4–7π/4 (angular range = $\pi /4\le \alpha \le 7\pi /4$). By changing the location of the $C_c$ point, the scale for r and the ranges for α, smaller curves required for characters, such as “m”, can be drawn.

(c)

For each $\alpha \left(i\right)$ where i = 1, ..., $N_c$, a coordinate point [X(i), Y(i)] can be computed as follows:

$$\mathbf{X}\left(i\right)=x_c+rcos\left(\alpha \left(i\right)\right)$$

(8)

$$\mathbf{Y}\left(i\right)=y_c+rsin\left(\alpha \left(i\right)\right)$$

(9)

Because α, X and Y are equally-sized vectors, each [X(i), Y(i)], where i = 1, ..., $N_c$ represents the coordinate of the $N_c$-th x-y point of the curve between zero and 2π with radius r.

(d)

For each [X(i), Y(i)], the inverse kinematics to obtain the associated ${\theta }_1$ and ${\theta }_2$ angles can be computed as in the case for straight lines.

3.2.3. Angular Conversion

For the inverse kinematics, ${\theta }_1$ and ${\theta }_2$ are assumed to be in radians. However, the angular position of the Bioloid’s servomechanisms is set in units within the range of 0–1024 [29,35]. In order to properly apply the kinematic models to the Bioloid’s servomechanisms, an angular conversion from radians to the 0–1024 range was performed.

For this purpose, the first step was to track the valid angular positions of the servomechanisms for each of the Bioloid’s arms and to identify their respective reference axes for ${\theta }_1$ and ${\theta }_2$ (the axes associated with zero radians). In Figure 9, the angular positions within the 0–1024 range for the servomechanisms of both arms and their reference axes are presented.

Figure 9. Positions of the Bioloid’s servomechanisms and reference axes.

Figure 9. Positions of the Bioloid’s servomechanisms and reference axes.

A linear regression approach was considered to perform the angular conversion of the servomechanisms’ axes from the radians domain to the 0–1024 range. The following linear equations were derived for each arm’s servomechanisms, where $V_1$ and $V_2$ are the angular positions within the 0–1024 range, and ${\theta }_1$ and ${\theta }_2$ are the angular positions in radians:

(a)

For the right arm:

$$\mathrm{Dynamixel}3\left(\mathrm{dyn}3\right)\to V_1=-190.985\times {\theta }_1+500.$$

(10)

$$\mathrm{Dynamixel}5\left(\mathrm{dyn}5\right)\to V_2=-190.985\times {\theta }_2+500.$$

(11)

(b)

For the left arm:

$$\mathrm{Dynamixel}4\left(\mathrm{dyn}4\right)\to V_1=-190.985\times {\theta }_1+1100.$$

(12)

$$\mathrm{Dynamixel}6\left(\mathrm{dyn}6\right)\to V_2=-190.985\times {\theta }_2+500.$$

(13)

Figure 10. Pseudo-code of the word definition scheme, adjustment of radius, and execution of kinematics for both Bioloid’s arms.

Figure 10. Pseudo-code of the word definition scheme, adjustment of radius, and execution of kinematics for both Bioloid’s arms.

3.3. Execution of the Kinematic Models

To execute the kinematic models of the alphabet characters, the position values of $V_1$ and $V_2$ were transmitted to the respective servomechanisms (Dynamixels) via the setGoalPosition function [29]. The speed of the servomechanisms was set with the setMovingSpeed function. A value of 50 was found to be appropriate for the arms’ servomechanisms. In order to achieve accurate drawings of straight and curved lines, the number of drawing points were set as $N_c$ = 600 and $N_l$ = 300.

Both Bioloid’s arms have a writing or drawing space (region) of 3.0 cm × 9.0 cm. The Cartesian space of both arms considers as the origin point the joint where the $L_1$ link (Dynamixel 3 and 4) is attached. Hence, by considering this origin, the x-range for each arm varies from −4.0 cm–5.0 cm, as presented in Figure 6. In contrast, the y-range varies from 9.0 cm–12.0 cm. It is important to mention that these ranges, and the estimation of $L_2$, were defined by considering that the pen or marker is to be located within the “palm” of the Bioloid’s “hand”, as presented in Figure 9. In this way, the user can attach the pen more appropriately to the robot’s “hand” with simple tools, such as a small cord.

Knowing the space restrictions is important to properly write a sentence and to avoid “truncated” words or sentences. In order to hand write the sentence within the drawing space, the size of the characters must be adjusted. Size is adjusted by reducing the radius r. Long sentences can be written with both arms if a scheme to define the words that will be written by each arm is established. In such a case, the left arm must write (draw) the initial words of the sentence. Furthermore, the length of the radius of the characters written by both arms must be the same. In Figure 10, the pseudo-code of the word definition scheme and the adjustment of the radius for both arms is presented.

4. Performance

Two standard bullet-point medium markers (black for the left arm, blue for the right arm) and letter-sized acetate (transparency) sheets were used as drawing materials for the transcription experiments. In the first experiment, the alphabet characters were drawn/written as presented in Figure 7. In Figure 11, the characters that were written with the Bioloid’s right arm using the kinematic models and the data of Figure 1 are presented. In this case, the speech recognition module was not used. These characters were drawn with r = 0.5 cm, and for comparison purposes, the same characters were drawn by a human user.

Quantitative assessment of the characters that were written with the Bioloid’s arm was performed with a metric based on the percentage of similarity between these characters and those that were handwritten by a human user. Figure 12 presents a general overview of the steps considered for the assessment of the robot transcription:

(a)

Initially, the characters that were drawn by the Bioloid robot and the human user (A and B, respectively) are coded into binary format.

(b)

Then, an adjustment of the character’s weight is performed to reduce the effect of the weight on the characters’ comparison process. This adjustment was performed with the bwmorph function in MATLAB.

(c)

An integrated character is obtained by performing the sum C = $A+B$. Due to the binary coding, the sum of all values in C ($n\left(C\right)$) represents the total number of pixels in A and B. These pixels include $n\left(c_{pix}\right)$ “correct” pixels (those in A that match B) and $n\left(e_{pix}\right)$ “error” pixels (those in A that do not match B).

(d)

$c_{pix}$ is computed as $and(A,B)$ and $e_{pix}$ is computed as $C-c_{pix}$. Then, the similarity between A and B (i.e., $Sim(A,B)$) is estimated by considering the number of “correct” pixels within C ($Sim(A,B)=n\left(c_{pix}\right)/n\left(C\right)$).

Figure 11. Drawing of the alphabet characters with the Bioloid’s right arm and the kinematics models (r = 0.5 cm).

Figure 11. Drawing of the alphabet characters with the Bioloid’s right arm and the kinematics models (r = 0.5 cm).

Figure 12. General steps of the quantitative assessment of the alphabet characters presented in Figure 11.

Figure 12. General steps of the quantitative assessment of the alphabet characters presented in Figure 11.

Figure 13. Results of the quantitative assessment of the alphabet characters presented in Figure 11.

Figure 13. Results of the quantitative assessment of the alphabet characters presented in Figure 11.

Figure 13 presents the $c_{pix}$ and $e_{pix}$ pixels for the characters presented in Figure 11. The similarity percentages are also presented. By considering the reference value of 0.60 for the legibility metric described in [24] for assessment of complex simulated robot writing, a threshold of 60.00% was set to address the accuracy of the Bioloid’s transcription. As presented in Figure 11, for all characters, the percentage of similarity is above 60.00% with an average of 73.44%. It is important to mention that this is a “strict” metric, as it implies that high similarity is only achieved if the robot draws the character “exactly” in the same way as the human user does (and thus, for all pixels, $A=B$). Hence, although the “g” character is clearly recognizable, it is slightly shorter than the character produced by the human user. This slight reduction in the vertical axis increased $e_{pix}$, leading to a similarity of 60.84%. For other characters, there is a slight variation in the horizontal axis. These variations are caused by practical factors, which are discussed in Section 5. However, the characters drawn by the Bioloid robot can be clearly identified by the human user, and the average similarity of 73.44% supports the accuracy of the robot-assisted transcription/handwriting.

In the second experiment, the transcription of the spoken short sentence “la casa es azul” (the house is blue) was performed. This sentence has four words, 12 alphabet characters and three spaces. The writing task was performed with the right arm. In this case, the radius was reduced in order to fit the whole sentence within the writing space. In Figure 14, the handwriting stages of this sentence are presented.

Figure 14. Transcription of a short sentence with the Bioloid’s right arm and the kinematics models (r = 0.3 cm).

Figure 14. Transcription of a short sentence with the Bioloid’s right arm and the kinematics models (r = 0.3 cm).

Finally, in the third experiment, the transcription of the long sentence “en la noche voy a cenar bien” (at night I am going to have good dinner) was performed. This sentence has seven words, 22 alphabet characters and six spaces. Because there are too many characters to be drawn within the writing space of a single arm, the sentence was split into two short sentences: “en la noche voy” and “a cenar bien”. The first short sentence was written with the left arm, and the second short sentence with the right arm. The length of the radius for each arm is the same, and it was computed based on the arm with the highest number of characters (in this case, the left arm). In Figure 15, the handwriting stages of this sentence are presented.

Figure 15. Transcription of a long sentence with the Bioloid’s left and right arm and the kinematic models (r = 0.3 cm)

Figure 15. Transcription of a long sentence with the Bioloid’s left and right arm and the kinematic models (r = 0.3 cm)

5. Discussion and Future Work

In Figure 11, Figure 14 and Figure 15, examples of characters and sentences written with the Bioloid’s Dynamixels are presented. While for most of the alphabet characters, accurate straight and curved lines were achieved, for some others, oscillations were observed in the drawing trajectories. These oscillations were more evident in smaller alphabet characters (r < 0.5 cm). However, the transcribed sentences could be read and understood properly by the human user, and overall, they were well defined. A quantitative metric, termed as the percentage of similarity, was proposed to evaluate the accuracy of the robot-assisted handwritten characters, and an average of 73.44% was obtained for characters with r = 0.5 cm. Particularly, for drawing of small characters, as those presented in this work, no other similar drawing application with the standard humanoid Bioloid has been found in the literature.

In this work, several practical factors were identified that affect the smoothness and accuracy of the drawing trajectories:

(a)

Stability of the robot’s pose: Several poses were considered for the robot in order to have proper stability for the writing task. Initially, the kneeling position was considered; however, significant instability was observed. Then, the standing position was considered with more stability. In this work, however, the kneeling position was considered for the task, because it is a more identifiable pose for the writing task. More stability was achieved by disabling the knee Dynamixels (dyn13, dyn14) to allow the whole body to “rest” on the “back-feet” Dynamixels (dyn17, dyn18). Then, by enabling the thigh Dynamixels (dyn11, dyn12), a straight pose of the upper body was achieved. This was also helpful to produce a “reaction force” in case of a significant friction between the pen and the writing plane (e.g., paper or transparency sheet). Nevertheless, due to the nature and characteristics of the Bioloid’s different elements and moving parts, the robot’s structure is not rigid, and it is susceptible to vibrations and instability.

(b)

Trajectory accuracy and feedback frequency: The Dynamixels have a relatively low trajectory accuracy and low feedback frequency [25], which reduces the precision of the required movements. As such, there are disparities between the straight and curved lines that were planned and the actual trajectory executed by the robotic arm [25].

(c)

Friction: During the writing task, there is a friction force caused by the texture of the pen and the writing plane, which restricts the smooth movements of the Dynamixels. This may also depend on the type of ink and pen. As a consequence of this situation, the tip of the pen may get stuck in a trajectory point. This leads to the following events: (1) the arm’s Dynamixels continue the movement required to complete the trajectory, which increases the force on the pen; (2) when this force is more significant, the tip of the pen is released abruptly, leading to an alteration in the drawing pattern.

(d)

Axis deviation of the writing line: As presented in Figure 14 and Figure 15, a small deviation on the y-axis was observed in both arms while writing the short and long sentences. This may be caused by the approximate conversion from radians to 0–1024 values and the reference values considered for the coordinate axes on the 0–1024 range (see Figure 9).

However, the experiments presented in this work demonstrated the suitability of the Bioloid system for the complex task of handwriting and speech transcription. More work is required to overcome the practical factors identified in the experiments and to extend this application. Thus, future work is focused on the following points:

(a)

Develop the dynamic models required for the handwriting task [11].

(b)

Integrate a visual feedback system to perform compensatory actions to correct alterations (and deviations) in the trajectories.

(c)

Adapt the speech-to-text transcriptor to other humanoid systems, such as the NAO robot.

(d)

Integrate a sensor to adjust the force of the arm on the writing plane to reduce friction with the tip of the pen.

(e)

Reduce the gap between the writing spaces of the left and right arms.

(f)

Perform transcription of sentences into paragraphs. The visual feedback is fundamental for this implementation.

The author declares no conflict of interest.