Planning of Knotting Based on Manipulation Skills with Consideration of Robot Mechanism/Motion and Its Realization by a Robot Hand System

Abstract

This paper demonstrates the relationship between the production process of a knot and manipulation skills. First, we define the description (rope intersections, grasp type and fixation positions) of a knot. Second, we clarify the characteristics of the manipulation skills from the viewpoint of the knot description. Next, in order to obtain the production process of the knot, we propose an analysis method based on the structure of the knot and the characteristics of the manipulation skills. Using the proposed analysis method, we analyzed eight kinds of knots, formed with a single rope, two ropes or a single rope and an object. Finally, in order to validate the production process obtained by the proposed analysis method, we show experimental results of an overhand knot and a half hitch produced by using a robot hand system.

1. Introduction

Manipulation of flexible linear objects is an important problem in the field of robotics, and the number of studies addressing this problem has been increasing recently. The authors have also developed new methods for the skills required by a robot hand in a knotting task and achieved a so-called overhand knot with a high-speed robot system [1].

However, it has not been demonstrated that the skills proposed in our previous work can produce many different types of knots. Therefore, in this paper, we examine this problem based on an original knotting algorithm. The goal of this paper is to analyze the knotting process and to define the relationship between the knotting process and the individual skills that can be performed by a robot hand.

In this paper, a “task” (knotting) is defined as one manipulation carried out by a robot, and “skill” (manipulation skill) is defined as a minimum required element to achieve the task and a meaningful element that changes the rope state. Therefore, one task is achieved by the combination of a certain number of skills. In order to further develop the manipulation theory and its implementation, it is important not only to achieve many tasks, but also to integrate the skills that are needed for task realization, as shown in Figure 1. This paper is summarized as follows:

• We define a problem setting for knotting manipulation (Section 2).
• We extract several basic manipulation skills for knotting (Section 3.1).
• We propose a knot description based on knot theory (Section 3.2), and we identify the characteristics of manipulation skills using the knot description (Section 3.3).
• We also suggest a method for obtaining a knotting procedure in which several knots can be achieved by synthesizing these skills (Section 4.1).
• We analyzed several types of knots using the proposed method (Section 4.2, Section 4.3 and Section 4.4).
• We realized two kinds of knots (overhand knot and half hitch) using a robot hand system (Section 5).
• We finally conclude this article and discuss future work (Section 6).

1.1. Related Work

Several researchers have focused on the manipulation of linear flexible objects. For example, Inaba et al. proposed a method of knotting a rope with a manipulator [2]. Matsuno et al. used an imaging system to recognize the shape of a rope and demonstrated knotting using dual manipulators [3]. Saha et al. suggested a topological motion planner for manipulating flexible linear objects [4]. Morita et al. proposed a knot planning technique using a system called Knot Planning from Observation (KPO) [5]. Wakamatsu et al. formulated a description of knots and a process of knot manipulation [6]. In their study, they considered the geometry of the knot and proposed a knotting process. In other researches, a knotting task that takes into consideration the robot motion and a rope model was executed by a robot arm [7,8], and knotting using a special mechanism with fixation of the rope and the geometrical relation of the knot was studied [9,10].

However, the main topic in the above studies was the analysis of the knot geometry. In experiments conducted by those authors, a manipulator was typically controlled along a trajectory based on the knot geometry. However, the characteristics of the manipulator and end-effector were not considered. As a result, there was considerable redundancy in the motions of the manipulator. In order to achieve knotting at high speed, it is necessary to develop the skills required by the manipulator and end-effector for the knotting task. The aim of our research is to identify the manipulation skills required by a general robot arm and robot hand to achieve a knotting task, as shown in Figure 2.

1.2. Contribution

In order to achieve knotting manipulation for any reference knot, a knotting procedure is strongly required, as shown in Figure 1 [11,12,13]. In this article, we propose an entirely new method for obtaining the knotting procedure based on manipulation skills for knotting by a robot system, as shown in Figure 2. By using the proposed method, we can perform various knotting manipulations with the robot. In particular, the originality of our proposed method is that we consider not only the knot structure, but also the robot mechanism and motion in the knotting procedure. Moreover, we describe several knotting procedures using the proposed method, and we show actual knotting manipulations of two kinds of knots produced by a high-speed robot system.

2. Problem Setting

The knots to be handled in this research include:

• a knot formed with a single rope,
• a knot formed with two ropes and
• a knot formed with a single rope and an object.

Although a knot has a three-dimensional shape, we consider the projected shape of the knot in a two-dimensional plane. This implies that the conventional knot description based on the intersection description in the knot theory can be introduced [6,14], and the actual knotting performed by the robot is performed in a two-dimensional plane, such as on a tabletop. This plane is called the “knot production plane” in this paper.

Under this setting, the following conditions for high-speed knotting manipulation are assumed:

• As shown in Figure 3, a rope is manipulated by an end-effector with two fingers and a robot arm, where the two fingers perform grasping and manipulation of the rope.
• The knot shape during manipulation can be held at multiple fixation points.
• The knot is produced on the knot production plane, and the robot can approach the knot production plane from only one direction, such as from above.

Knotting manipulation is performed by the following procedure:

• Given a reference knot,
• Describe the knot and analyze the knot while unraveling it,
• Develop a plan to tie the knot based on the analysis result and
• Produce the knot with a robot system.

3. Manipulation Skills and Knot Description

First, we extract manipulation skills for planning the knot manipulation and realizing the knot with a robot. Next, we define the knot description based on knot theory, and we describe the characteristics of the manipulation skills.

3.1. Extraction of Manipulation Skills

Extracting the necessary and appropriate manipulation skills is extremely important for producing the knot with the robot. We first analyzed how a human subject tied a knot (overhand knot, shown in Figure 4) to extract the required manipulation skills. Figure 4a–c shows loop production using hand motion. Figure 4d,e shows rope permutation using finger motion. Figure 4f–h shows rope pulling using hand motion. From these results, we can extract three manipulation skills, namely “loop production”, “rope permutation” and “rope pulling”. In addition to these three manipulation skills, we add one more manipulation skill, “rope moving”, in order to handle the end of the rope. Therefore, the manipulation skills required for knotting are summarized as follows, and these manipulation skills are illustrated in Figure 5: (1) loop production (Figure 5b), (2) rope permutation (Figure 5c), (3) rope pulling (Figure 5d) and (4) rope moving (Figure 5a).

3.2. Knot Description

This section defines descriptions of intersections, grasp types of intersections, and fixation locations of the rope.

3.2.1. Description of Intersections

Figure 6 shows signs assigned to intersections. In the case of a knot in a single rope, the two ends of the rope are represented by $E_l$ and $E_r$; and in the case of knots in two ropes and knots in a single rope and an object, the ends of the ropes and the object are represented by $l_1$, $l_2$, $r_1$ and $r_2$. Descriptions of the intersections $C_i^{\{+,-\}}$ are assigned for all intersections from $E_l$–$E_r$ or from $l_1$–$l_2$, where the subscript i is an intersection number. The sign “$+,-$” shows the intersection sign. The sign “−” is counter clockwise (Figure 6a), and the sign “+” is clockwise (Figure 6b).

3.2.2. Description of Grasp Types of Intersections

In addition to the description of intersections, we introduce a new description of the grasp types of intersections. In the case where an intersection is grasped by two fingers, the relationship between the grasp type and the manipulation skill can be easily identified by distinguishing between grasp types. Thus, we define the relationship between the grasp type and the description as shown in Figure 7. Figure 7 shows a cross-sectional view of the grasp types of intersections, where the grasp types are divided into the two types shown in Figure 7a,b. We consider only the grasp state of the intersection by two fingers, and the angle between the two sections of the rope is not considered.

First, the two directions of the rope at “outlets” of the intersection are summed, which is the gray arrow shown in Figure 7. Second, the line that connects the two fingers grasping the intersection is considered. Finally, the relationship between the summed rope direction and this connecting line is examined. If the relationship is parallel, the description of the intersection shown in Figure 7a is defined as $\overline{C_i^{\{+,-\}}}$. If the relationship is orthogonal, the description of the intersection shown in Figure 7b is also defined as $\widehat{C_i^{\{+,-\}}}$. This description method is used in loop production, in particular.

3.2.3. Description of Fixation Locations of Rope

Let us consider a case where the intersection on the loop is grasped by the robot and the intersection is produced by permutation. However, the whole loop is rotated by permutating the intersection in this case, as shown in Figure 8. As a result, the geometrical structure (production of intersection) cannot be changed by this operation. Moreover, the geometrical structure may be unexpectedly changed by this operation. In order to solve these problems and perform operations appropriately, fixation of the rope is required. Therefore, the necessary fixation locations of rope are considered as follows: (1) end point of rope, (2) rope intersection and (3) one point on the loop.

Fixation at the edge point of the rope is needed so as to fix the initial position of the edge, and fixation of the rope intersection is needed so as to maintain the geometrical relation of the knot. Fixation of one point on the loop is also needed so as not to rotate the whole loop. The descriptions of these fixation locations are indicated by the subscripts f, m and p. In the case of the fixation of the edge point of the rope, the description is $E_{l,f}$ ($l_{1,f}$). In the case of the fixation of the rope intersection and one point on the loop, the descriptions are defined as $C_{i,m}^{\{+,-\}}$ and $C_{i,p}^{\{+,-\}}$, respectively. Concrete descriptions together with manipulation characteristics will be explained in the next section.

Actual methods for fixing the rope include fixing the rope by using another finger or applying tension to the rope.

3.3. Characteristics of Manipulation Skills

This section discusses manipulation skills in order to understand their characteristics based on the description of the knot.

3.3.1. Rope Moving

Rope moving is defined as free motion of the end of the rope on the knot production plane. Rope moving is used to cross the two sections of the rope (Figure 9a) or to set one section close to another (Figure 9b). The dotted line in Figure 9 indicates a simplified structure of the knot for this part.

Starting from $E_l$, we walk along the rope writing down each crossing that we meet together with its sign until we reach $E_r$. During this process, each crossing is met twice. For example, in the case where the two sections are intersected, as shown in Figure 9a, one intersection is created, and the description of the intersection can be written as,

$$E_{l,f}-E_r⟶E_{l,f}-C_1^{\{+,-\}}-C_1^{\{+,-\}}-E_r\left(\mathrm{single}\mathrm{rope}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-l_2⟶l_{1,f}-C_1^{\{+,-\}}-l_2\hfill \\ r_{1,f}-r_{2,f}⟶r_{1,f}-C_1^{\{+,-\}}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{two}\mathrm{ropes}\right).$$

In the case where one section of the rope approaches another, as shown in Figure 9b, an intersection is not made.

In rope moving of a single rope in the case where there are intersections already, fixation at the intersection and the loop is needed to maintain the geometrical relation of the knot, as shown by the blue triangle in Figure 10a. The two fixation locations are indicated by subscripts m and p as follows:

$$E_{l,f}-\cdots -C_{i,mp}^{\{+,-\}}-C_{i,mp}^{\{+,-\}}-\cdots -E_r.$$

In the same way, fixation for rope moving of two ropes is also needed to maintain the geometrical relation of the knot, as shown by the blue triangle in Figure 10b. The fixation is indicated by subscript m as follows:

$$\left\{\begin{array}{c}{l}_{1,f}-C_{i,m}^{\{+,-\}}-\cdots -l_2\hfill \\ r_{1,f}-C_{i,m}^{\{+,-\}}-\cdots -r_{2,f}.\hfill \end{array}\right.$$

3.3.2. Loop Production

Loop production is a special example of rope moving and is an operation that makes one intersection in a single rope, as shown in Figure 11. The description of the intersection can be written as (see Figure 11a),

$$E_{l,f}-E_r⟶E_l-\overline{C_1^{\{+,-\}}}-\overline{C_1^{\{+,-\}}}-E_r$$

or (see Figure 11b)

$$E_{l,f}-E_r⟶E_l-\widehat{C_1^{\{+,-\}}}-\widehat{C_1^{\{+,-\}}}-E_r.$$

The cross sign of the intersection depends on the positioning of the loop, that is which is the overpassing arc and which the underpassing. The grasp type also depends on the loop production method.

3.3.3. Rope Permutation

Rope permutation is divided into three types as shown in Figure 12a–c. Figure 12a represents the rope permutation in the case where the robot grasps two parallel sections that do not possess intersections, and the description of the intersection is as follows:

$$\begin{array}{c}\hfill E_{l,f}-E_r⟶E_{l,f}-C_1^{\{+,-\}}-C_2^{\{-,+\}}-C_2^{\{-,+\}}-C_1^{\{+,-\}}-E_r.\end{array}$$

Figure 12b shows the state after loop production and corresponds to Figure 11a. Here, fixation is needed so as not to rotate the whole loop. In the description, the subscript p is added to indicate this fixation. In this operation, two intersections are created, and the description is expressed as,

$$E_{l,f}-\overline{C_{1,p}^{\{+,-\}}}-\overline{C_{1,p}^{\{+,-\}}}-E_r⟶E_{l,f}-C_1^{\{-,+\}}-\overline{C_{2,p}^{\{+,-\}}}-C_3^{\{+,-\}}-C_3^{\{+,-\}}-\overline{C_{2,p}^{\{+,-\}}}-C_1^{\{-,+\}}-E_r.$$

In addition, Figure 12c also shows the state after loop production and corresponds to Figure 11b. In this case, fixation is also required. The description of the intersection is:

$$E_{l,f}-\widehat{C_{1,p}^{\{+,-\}}}-\widehat{C_{1,p}^{\{+,-\}}}-E_r⟶E_{l,f}-C_1^{\{+,-\}}-\widehat{C_{2,p}^{\{+,-\}}}-C_3^{\{-,+\}}-C_1^{\{+,-\}}-\widehat{C_{2,p}^{\{+,-\}}}-C_3^{\{-,+\}}-E_r.$$

3.3.4. Rope Pulling

Rope pulling is an operation that basically removes the intersection, as shown in Figure 13. This skill is effective when used together with rope permutation.

Rope Permutation and Rope Pulling

By utilizing rope permutation and rope pulling, three skills can be executed, such as the production of continuous multiple intersections (Figure 14), changing the cross sign of an intersection (Figure 15) and crossing one section under the rope on the knot production plane (Figure 16).

In the case where the number of intersections is increased (Figure 14), the description becomes:

$$E_{l,f}-E_r⟶E_{l,f}-C_1^{\{+,-\}}-C_2^{\{-,+\}}-\cdots -C_2^{\{-,+\}}-C_1^{\{+,-\}}-E_r\left(\mathrm{single}\mathrm{rope}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-l_2⟶l_{1,f}-C_1^{\{+,-\}}-C_2^{\{-,+\}}-\cdots -l_2\hfill \\ r_{1,f}-r_{2,f}⟶r_{1,f}-C_1^{\{+,-\}}-C_2^{\{-,+\}}-\cdots -r_{2,f}\hfill \end{array}\right.\left(\mathrm{two}\mathrm{ropes}\right).$$

Moreover, in the case where the sign at one intersection is changed (Figure 15), the description becomes:

$$E_{l,f}-C_1^{\{+,-\}}-\cdots -C_i^{\{+,-\}}-\cdots -C_1^{\{+,-\}}-E_r⟶$$

$$E_{l,f}-C_1^{\{+,-\}}-\cdots -C_i^{\{-,+\}}-\cdots -C_1^{\{+,-\}}-E_r.$$

In addition, in the case where the rope makes a cross under the rope (Figure 16), the description becomes:

$$E_{l,f}-E_r⟶E_{l,f}-C_1^{\{+,-\}}-C_1^{\{+,-\}}-E_r.$$

4. Knot Analysis

Here, we propose an analysis method for knot planning, and we analyze various kinds of knots using the proposed method. In particular, we deal with the following types of knots:

• a knot formed with a single rope,
• a knot formed with two ropes, and
• a knot formed with a single rope and an object.

4.1. Analysis Method

In this section, we describe the analysis method for obtaining the procedure for knotting manipulation by a robot system. To achieve this, we introduce an assumption and summarize the rules for the manipulation skills. We also propose an analysis method including concrete algorithm flows for unraveling and knotting.

4.1.1. Constraint Condition for Knotting Manipulation

First, we define a constraint condition for the analysis. In the knotting manipulation, a production process for the first intersection is not specified, and thus, an intersection with a + or − cross sign is produced. In order to specify that the first intersection has the cross sign −, we introduce the following condition. Here, the reason that the first intersection with the sign − is chosen is that it is convenient, and the first operation is specified as overpassing in the knot production plane.

Case of the knot in single rope (see Figure 17a):

• $E_l$ is fixed ($E_{l,f}$), and $E_r$ is freely moved by the robot.
• The first intersection is produced so as to change the rope direction clockwise from $E_{l,f}$ to $E_r$.
• $E_r$ passes over the rest of the string.

Case of the knot with two ropes (see Figure 17b):

• $l_1$, $r_1$ and $r_2$ are fixed ($l_{1,f}$, $r_{1,f}$, $r_{2,f}$), and $l_2$ is freely moved by the robot.
• The first intersection is produced so as to assign the direction ($l_{1,f}\to r_{1,f}\to l_2\to r_{2,f}$) clockwise.
• $l_2$ passes over the rest of the string.

By introducing this assumption, the first intersection is produced on the rope as shown in Figure 17.

4.1.2. Rules for Manipulation Skills

Here, we discuss rules for the manipulation skills based on the constraint condition described in the previous section.

The first intersection, which is created under the constraint condition, must be $C_1^{-}$. According to this condition, we can summarize the manipulation skills as shown in

Table 1. Rule of manipulation skills.

Production of Intersection	⟶	Manipulation Skill
Add an intersection ‘−’ in case that
there is no intersection in a single rope	⟶	Loop production (T)
Add an intersection ‘−’ : intersection of odd number	⟶	Rope moving (S)
: intersection of even number	⟶	Rope permutation + Rope pulling (UV)
Add an intersection ‘+’ : intersection of odd number	⟶	Rope permutation + Rope pulling (UV)
: intersection of even number	⟶	Rope moving (S)
Add two intersections	⟶	Rope permutation (U)
Change cross sign of intersection	⟶	Rope permutation + Rope pulling (UV)
Delete intersections from one intersection to $E_r$ or $l_2$	⟶	Rope pulling (V)

. Thus, since we know, from the unraveling, the location and the cross sign of the next intersection to be created, we can identify the manipulation skill while following the unraveling sequence in reverse. In Table 1, the manipulation skills are represented as follows: rope moving (S), loop production (T), rope permutation (U) and rope pulling (V). Furthermore, UV means that rope permutation and rope pulling are performed in sequence. These symbols are used in the analysis results presented in Section 4.2, Section 4.3 and Section 4.4.

4.1.3. Overall Procedure

Here, we explain the overall procedure of the analysis method, and then, we describe the concrete algorithm flow in the method. The overall procedure is as follows:

• Represent a knot based on the knot description explained in the previous section.
• Unravel one intersection of the knot, starting from the intersection nearest the end of the rope.
• Iterate Step 2 until all intersections disappear. As a result, a sequence of operations for removing the intersections is obtained.
• Apply appropriate manipulation skills to the sequence, while following the sequence obtained in Step 3 in reverse.

With this process, the location and the sign of each intersection are identified. As a result, it can be determined how to generate the intersection. This analysis can be applied not only to a knot formed with one rope, but also to a knot formed with one rope and one object and a knot formed with two ropes. Although the knot production process obtained by the proposed method may not be optimal, this method can always provide one solution for the knot production process.

The concrete flow for knot analysis is as shown in Figure 18 (unraveling) and Figure 19 (knotting).

4.2. Knots with a Single Rope

Here, we consider the description and production method of a knot with a single rope. Pick one endpoint as the beginning of the rope, and denote it by $E_l$. The end of the rope is represented by $E_r$. As concrete examples of knots, we deal with: (1) an overhand knot, (2) a figure-eight knot and (3) a stevedore knot.

Due to space limitations, we show all analysis results only for the overhand knot, and we describe only the procedure of knotting manipulation for the other two knots.

4.2.1. Overhand Knot

The first example is the “overhand knot”, which is the simplest and most typical knot.

Unraveling of Overhand Knot

By unraveling one intersection, starting from the intersection ($C_3^{-}$) near the end ($E_r$) of the rope, the analysis result can be obtained as follows (Figure 20a–d):

$$E_l-C_1^{-}-C_2^{-}-C_3^{-}-C_1^{-}-C_2^{-}-C_3^{-}-E_r\left(\mathrm{a}\right)$$

$$E_l-C_1^{-}-C_2^{-}-C_1^{-}-C_2^{-}-E_r\left(\mathrm{b}\right)$$

$$E_l-C_1^{-}-C_1^{-}-E_r\left(\mathrm{c}\right)$$

$$E_l-E_r\left(\mathrm{d}\right)$$

The production process of the overhand knot can be obtained by following this process in reverse while considering the description of the intersections.

Knotting of Overhand Knot

First, loop production is performed, and the intersection $\widehat{C_1^{-}}$ is created (Figure 21b). Second, the intersection $C_2$ is produced. However, it is not effective to produce only the intersection $C_2$. By checking the final type of knot, it is found that two intersections should be made after the intersection $C_1$. In addition, the final intersection $C_3$ should pass under the rope. For these reasons, the state shown in Figure 21c can be produced only by performing rope permutation (Figure 12c). Finally, using rope permutation and rope pulling, the overhand knot is achieved, as shown in Figure 21d. The description of the intersections for the overhand knot production process is as follows:

$$E_{l,f}-E_r\left(\mathrm{a}\right)$$

$$E_{l,f}-\widehat{C_{1,p}^{-}}-\widehat{C_{1,p}^{-}}-E_r\left(\mathrm{b}\right)$$

$$E_{l,f}-C_1^{-}-\widehat{C_{2,p}^{-}}-C_3^{+}-C_1^{-}-\widehat{C_{2,p}^{-}}-C_3^{+}-E_r\left(\mathrm{c}\right)$$

$$E_{l,f}-C_1^{-}-\widehat{C_{2,p}^{-}}-C_3^{-}-C_1^{-}-\widehat{C_{2,p}^{-}}-C_3^{-}-E_r\left(\mathrm{d}\right)$$

4.2.2. Figure-Eight Knot

First, loop production is performed, and the intersection $\overline{C_1^{-}}$ is created as shown in Figure 22b. Second, according to the rules of the manipulation skills, rope permutation and rope pulling (Figure 16) are carried out to create an even-numbered intersection with sign − (second intersection), and then, the intersection $C_1^{-}$ is produced (Figure 22c). Next, the state shown in Figure 22d can be obtained by rope moving, in which the fixation of the rope is needed, and the description of this fixation is added. Finally, the intersections $C_3^{+}$ and $C_4^{+}$ are created by performing rope permutation and rope pulling twice, thus completing the figure-eight knot, as shown in Figure 22e,g.

$$E_{l,f}-E_r\left(\mathrm{a}\right)$$

$$E_{l,f}-\overline{C_1^{-}}-\overline{C_1^{-}}-E_r\left(\mathrm{b}\right)$$

$$E_{l,f}-C_1^{-}-\overline{C_{2,mp}^{-}}-\overline{C_{2,mp}^{-}}-C_1^{-}-E_r\left(\mathrm{c}\right)$$

$$E_{l,f}-C_1^{-}-\overline{C_{2,mp}^{-}}-C_3^{-}-C_4^{+}-\overline{C_{2,mp}^{-}}-C_1^{-}-C_4^{+}-C_3^{-}-E_r\left(\mathrm{e}\right)$$

$$E_{l,f}-C_1^{-}-\overline{C_{2,mp}^{-}}-C_3^{+}-C_4^{+}-\overline{C_{2,mp}^{-}}-C_1^{-}-C_4^{+}-C_3^{+}-E_r\left(\mathrm{g}\right)$$

4.2.3. Stevedore’s Knot

First, an intersection $\overline{C_1^{-}}$ is made by loop production (Figure 23a). Then, three intersections ($C_1^{-}$, $C_2^{-}$, $C_3^{-}$) are created by performing rope permutation (Figure 12a) three times in sequence and performing rope pulling (Figure 23e) once. Next, the state shown in Figure 23f is obtained by rope moving, in which the fixation of the rope is needed, and the description of this fixation is added. Moreover, two intersections ($C_5^{+}$, $C_6^{+}$) are produced by performing rope permutation two times and rope pulling one time, thus completing the stevedore knot (Figure 23i).

$$E_{l,f}-E_r$$

$$E_{l,f}-\overline{C_1^{-}}-\overline{C_1^{-}}-E_r\left(\mathrm{a}\right)$$

$$E_{l,f}-C_1^{-}-C_2^{-}-C_3^{-}-\overline{C_{4,mp}^{-}}-\overline{C_{4,mp}^{-}}-C_3^{-}-C_2^{-}-C_1^{-}-E_r\left(\mathrm{e}\right)$$

$$E_{l,f}-C_1^{-}-C_2^{-}-C_3^{-}-\overline{C_{4,mp}^{-}}-C_5^{-}-C_6^{+}-\overline{C_{4,mp}^{-}}-C_3^{-}-C_2^{-}-C_1^{-}-C_6^{+}-C_5^{-}-E_r\left(\mathrm{g}\right)$$

$$E_{l,f}-C_1^{-}-C_2^{-}-C_3^{-}-\overline{C_{4,mp}^{-}}-C_5^{+}-C_6^{+}-\overline{C_{4,mp}^{-}}-C_3^{-}-C_2^{-}-C_1^{-}-C_6^{+}-C_5^{+}-E_r\left(\mathrm{i}\right)$$

4.3. Knots with Two Ropes

In this section, we analyze knots formed with two ropes. Three kinds of knots are considered: (1) a sheet bend, (2) a square knot and (3) a granny knot.

The left end and the right end of one rope are represented by $l_1$ and $l_2$, and the left end and the right end of the other rope are represented by $r_1$ and $r_2$. The description of the intersections on the two ropes is performed in the order from the initial location. In the analysis, the knot is unraveled from $l_2$.

4.3.1. Sheet Bend

The one rope described by $r_{1,f}$, $r_{2,f}$ is fixed at the initial state, as shown in Figure 24a. The intersection $C_1^{-}$ is produced by rope moving (Figure 24b), in which the intersection is fixed, and a description of this fixation is added. To create an even-numbered intersection with sign + (second intersection), rope moving is performed, and the intersection $C_2^{+}$ is created (Figure 24c). Next, the intersection $C_3^{+}$, which is an odd-numbered intersection with sign + (third intersection), is created by rope permutation and rope pulling (Figure 24d). Then, the intersection $C_4^{-}$, which is an even-numbered intersection with sign − (fourth intersection), is made by rope moving, rope permutation and rope pulling (Figure 24e). Moreover, the intersection $C_5^{-}$, which is an odd-numbered intersection with sign − (fifth intersection), is created by rope moving (Figure 24f), and the intersection $C_2^{+}$, which is an even-numbered intersection with sign + (sixth intersection), is also created by rope moving (Figure 24g). Finally, to create an odd-numbered intersection with sign − (seventh intersection), rope permutation and rope pulling are executed, then the intersection $C_7^{-}$ is produced. As a result, a sheet bend knot is completed (Figure 24h).

$$\left\{\begin{array}{c}{l}_{1,f}-l_2\hfill \\ r_{1,f}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{a}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-l_2\hfill \\ r_{1,f}-C_{1,m}^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{b}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{+}-l_2\hfill \\ r_{1,f}-C_2^{+}-C_{1,m}^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{c}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{+}-C_3^{+}-l_2\hfill \\ r_{1,f}-C_3^{+}-C_2^{+}-C_{1,m}^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{d}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{+}-C_3^{+}-C_4^{-}-l_2\hfill \\ r_{1,f}-C_3^{+}-C_2^{+}-C_{1,m}^{-}-C_4^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{e}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{+}-C_3^{+}-C_4^{-}-C_5^{-}-l_2\hfill \\ r_{1,f}-C_3^{+}-C_2^{+}-C_{1,m}^{-}-C_5^{-}-C_4^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{f}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{+}-C_3^{+}-C_4^{+}-C_5^{-}-C_6^{-}-C_2^{+}-l_2\hfill \\ r_{1,f}-C_4^{+}-C_3^{+}-C_{1,m}^{-}-C_6^{-}-C_5^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{g}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{+}-C_3^{+}-C_4^{+}-C_5^{-}-C_6^{-}-C_2^{+}-C_7^{-}-l_2\hfill \\ r_{1,f}-C_4^{+}-C_3^{+}-C_7^{-}-C_{1,m}^{-}-C_6^{-}-C_5^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{h}\right)$$

4.3.2. Square Knot

The initial state is set as shown in Figure 25a, which is the same as the sheet bend knot. By performing rope permutation three times and rope pulling one time, the intersections $C_1^{-}$, $C_2^{-}$ and $C_3^{-}$ are created (Figure 25e). Here, the first intersection is fixed, and the description of the fixation is added. Next, the state shown in Figure 25f is obtained by rope moving. Lastly, by performing rope permutation three times and rope pulling one time, the intersections $C_4^{+}$, $C_5^{+}$ and $C_6^{+}$ are created, thus completing the square knot (Figure 25j).

$$\left\{\begin{array}{c}{l}_{1,f}-l_2\hfill \\ r_{1,f}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{a}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{-}-C_3^{-}-l_2\hfill \\ r_{1,f}-C_{1,m}^{-}-C_2^{-}-C_3^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{e}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{-}-C_3^{-}-C_4^{+}-C_5^{+}-C_6^{+}-l_2\hfill \\ r_{1,f}-C_4^{+}-C_5^{+}-C_6^{+}-C_{1,m}^{-}-C_2^{-}-C_3^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{j}\right)$$

4.3.3. Granny Knot

By creating the intersection $C_{r,m}^{+}$, the initial state is set as shown in Figure 26a. Then, the intersection $C_1^{-}$ is made by rope moving (Figure 26b). Here, the first intersection is fixed, and the description of the fixation is added. Next, the intersection $C_2^{-}$ is created by rope permutation and rope pulling (Figure 26c). Moreover, the intersection $C_3^{-}$ is created by rope permutation and rope pulling (Figure 26d), and the intersection $C_4^{-}$ is also made by rope moving (Figure 26e). The intersections $C_5^{-}$, $C_6^{-}$ and $C_2^{-}$ are created by rope moving in sequence (Figure 26g). Finally, the intersection $C_7^{-}$ is created by rope permutation and rope pulling, thus completing the granny knot (Figure 26h).

$$\left\{\begin{array}{c}{l}_{1,f}-l_2\hfill \\ r_{1,f}-C_{r,m}^{+}-C_{r,m}^{+}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{a}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-l_2\hfill \\ r_{1,f}-C_{r,m}^{+}-C_{1,m}^{-}-C_{r,m}^{+}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{b}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{-}-l_2\hfill \\ r_{1,f}-C_{r,m}^{+}-C_2^{-}-C_{1,m}^{-}-C_{r,m}^{+}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{c}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{-}-C_3^{-}-l_2\hfill \\ r_{1,f}-C_{r,m}^{+}-C_2^{-}-C_{1,m}^{-}-C_{r,m}^{+}-C_3^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{d}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{-}-C_3^{-}-C_4^{-}-l_2\hfill \\ r_{1,f}-C_4^{-}-C_{r,m}^{+}-C_2^{-}-C_{1,m}^{-}-C_{r,m}^{+}-C_3^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{e}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{-}-C_3^{-}-C_4^{-}-C_5^{-}-l_2\hfill \\ r_{1,f}-C_4^{-}-C_{r,m}^{+}-C_2^{-}-C_{1,m}^{-}-C_5^{-}-C_{r,m}^{+}-C_3^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{f}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{-}-C_3^{-}-C_4^{-}-C_5^{-}-C_6^{-}-C_2^{-}-l_2\hfill \\ r_{1,f}-C_5^{-}-C_{r,m}^{+}-C_3^{-}-C_{1,m}^{-}-C_6^{-}-C_{r,m}^{+}-C_4^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{g}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{-}-C_3^{-}-C_4^{-}-C_5^{-}-C_6^{-}-C_2^{-}-C_7^{-}-l_2\hfill \\ r_{1,f}-C_5^{-}-C_{r,m}^{+}-C_3^{-}-C_7^{-}-C_{1,m}^{-}-C_6^{-}-C_{r,m}^{+}-C_4^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{h}\right)$$

4.4. Knot with Single Rope and Object

Here, we discuss a knot formed with a single rope and an object. As representative examples of such knots, we analyze: (1) a half hitch and (2) a clove hitch.

The left end and the right end of the rope are represented by $l_1$ and $l_2$, and the left end and the right end of the object are represented by $r_1$ and $r_2$, like the descriptions of the knots formed with two ropes. The description of intersections on the rope and the object is performed in order from the initial location.

4.4.1. Half Hitch

First, the intersection $C_1^{-}$ is created by rope moving (Figure 27b), and the fixation is performed and the description of the fixation added. The intersection $C_2^{-}$, which is an even-numbered intersection with sign − (second intersection), is created by rope permutation and rope pulling (Figure 27c). Next, the intersection $C_1^{+}$, which is an odd-numbered intersection with sign + (third intersection), is also created (Figure 27d). Finally, rope permutation is performed two times and rope pulling one time to complete the half hitch (Figure 27f).

$$\left\{\begin{array}{c}{l}_{1,f}-l_2\hfill \\ r_{1,f}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{a}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-l_2\hfill \\ r_{1,f}-C_{1,m}^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{b}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{-}-l_2\hfill \\ r_{1,f}-C_{1,m}^{-}-C_2^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{c}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-\widehat{C_1^{+}}-C_{2,m}^{-}-C_3^{-}-\widehat{C_1^{+}}-l_2\hfill \\ r_{1,f}-C_{2,m}^{-}-C_3^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{d}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_1^{+}-\widehat{C_2^{+}}-C_3^{-}-C_{4,m}^{-}-C_5^{-}-C_1^{-}-\widehat{C_2^{+}}-C_3^{-}-l_2\hfill \\ r_{1,f}-C_{4,m}^{-}-C_5^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{e}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_1^{+}-\widehat{C_2^{+}}-C_3^{+}-C_{4,m}^{-}-C_5^{-}-C_1^{+}-\widehat{C_2^{+}}-C_3^{+}-l_2\hfill \\ r_{1,f}-C_{4,m}^{-}-C_5^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{f}\right)$$

4.4.2. Clove Hitch

The first intersection $C_1^{-}$ is created by rope moving, as shown in Figure 28b, and the description of the fixation of the first intersection is added. Secondly, the intersection $C_2^{-}$, which is an even-numbered intersection with sign − (second intersection), is created by rope permutation and rope pulling (Figure 28c). Two intersections $C_1^{+}$ and $C_4^{-}$ are produced by rope moving (Figure 28e). The intersection $C_5^{-}$ is created by rope permutation and rope pulling, and the intersection $C_6^{-}$ is also created by rope moving (Figure 28g). Finally, the intersection $C_5^{+}$ is created by rope permutation and rope pulling, thus completing the clove hitch (Figure 28h).

Although multiple procedures can be obtained and these procedure may not be identified in conventional methods, our proposed method can always obtain one procedure. This is the advantage of our proposed method. In addition, the proposed method considers the robot mechanism and the robot motion required for the procedure, and therefore, it is considered that the procedure obtained by the proposed method can be easily implemented on a robot system.

$$\left\{\begin{array}{c}{l}_{1,f}-l_2\hfill \\ r_{1,f}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{a}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-l_2\hfill \\ r_{1,f}-C_{1,m}^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{b}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_{1,m}^{-}-C_2^{-}-l_2\hfill \\ r_{1,f}-C_2^{-}-C_{1,m}^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{c}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_1^{+}-C_{2,m}^{-}-C_3^{-}-C_1^{+}-l_2\hfill \\ r_{1,f}-C_3^{-}-C_{2,m}^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{d}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_1^{+}-C_{2,m}^{-}-C_3^{-}-C_1^{+}-C_4^{-}-l_2\hfill \\ r_{1,f}-C_3^{-}-C_{2,m}^{-}-C_4^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{e}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_1^{+}-C_{2,m}^{-}-C_3^{-}-C_1^{+}-C_4^{-}-C_5^{-}-l_2\hfill \\ r_{1,f}-C_3^{-}-C_{2,m}^{-}-C_4^{-}-C_5^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{f}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_1^{+}-C_{2,m}^{-}-C_3^{-}-C_1^{+}-C_4^{-}-C_5^{-}-C_6^{-}-l_2\hfill \\ r_{1,f}-C_3^{-}-C_{2,m}^{-}-C_4^{-}-C_6^{-}-C_5^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{g}\right)$$

$$\left\{\begin{array}{c}{l}_{1,f}-C_1^{+}-C_{2,m}^{-}-C_3^{-}-C_1^{+}-C_4^{-}-C_5^{+}-C_6^{-}-C_7^{-}-C_5^{+}-l_2\hfill \\ r_{1,f}-C_3^{-}-C_{2,m}^{-}-C_4^{-}-C_7^{-}-C_6^{-}-r_{2,f}\hfill \end{array}\right.\left(\mathrm{h}\right)$$

5. Knotting Manipulation by a High-Speed Robot System

Here, we show actual knotting manipulation by a high-speed robot system, which consisted of a high-speed multi-fingered hand [15], high-speed tactile sensors [16], a high-speed vision system including image processing [17] and an active vision system for gaze control.

We formed two kinds of knots, namely an overhand knot and a half hitch. Concrete explanations about the robot motion and the sensory feedback control method for performing the manipulation skills were described in a previous paper [1,12].

5.1. Experimental Result for Overhand Knot

Figure 29 shows the experimental setup for overhand knot manipulation, and Figure 30 shows the experimental result. In Figure 30a–e, the loop production was performed using wrist motion. Figure 30f,g shows the rope pushing motion for rope permutation. Figure 30h,i shows the rope permutation, and Figure 30j–l shows rope pulling. As a result, the overhand knot could be achieved successfully by the robot system.

5.2. Experimental Result of Half Hitch

Figure 31 shows an overview of the experimental system for half hitch manipulation. In this experiment, the robot system did not have a translational motion mechanism, and therefore, the state in which the rope moving is finished (Figure 27c) was set at the initial state. Figure 32 shows the experimental result. Figure 32a depicts the initial state; Figure 32b,c illustrate the loop production; Figure 32d illustrates rope pushing for rope permutation; Figure 32e–g shows rope permutation; and Figure 32h,i shows rope pulling. Finally, as shown in Figure 32j–l, a human strengthened the knot in order to make the resulting half hitch more visible.

From these experimental results, we confirmed that the proposed method is effective for knotting manipulation, and two knots were achieved successfully by using our proposed method and the robot system. The results can be seen on our web site [18,19].

6. Conclusions

Our goal was to achieve knotting of a flexible rope as an example of dexterous manipulation of a linear flexible object. A summary of the work described in this article is as follows:

• We analyzed the knotting motion of a human hand to identify the necessary knotting skills. Then, we proposed a knot description methodology and a method of obtaining the production process of a knot based on combinations of these skills.
• We suggested a knotting strategy that does not depend on the flexible characteristics of the rope. A real-time tactile and visual sensory feedback control method was proposed to improve the success rate and robustness for various ropes.
• We demonstrated experimental results achieved by a high-speed multi-fingered hand system.

In the future, we plan to achieve several other knots in addition to the overhand knot and half hitch demonstrated here, according to the procedure obtained from the analysis result described in this paper.