A Tactile-Based Wire Manipulation System for Manufacturing Applications

Abstract

This paper presents experimental results developed within the WIRES experiment, whose main objective is the robotized cabling of switchgears. This task is currently executed by human operators; the WIRES Project tackles the development of a suitably designed sensorized end effector for the wire precise manipulation. In particular, the developed gripper with tactile sensors are shown and a procedure for the implementation of the insertion task is presented and discussed. Experimental results are reported both for quality of wire shape reconstruction and success rate of insertion task implementation.

1. Introduction

Robotic manipulation is a complex task especially when deformable and fragile objects have to be grasped. In these cases, the knowledge of geometrical and physical characteristics of the object to manipulate are fundamental for the successful implementation of the task. To this aim, specific sensing systems are developed to be integrated into robotic systems. This paper presents results of activities developed within the WIRES experiment (http://www-lar.deis.unibo.it/people/gpalli/WIRES/), where the main objective is the robotized cabling of switchgears. Switchgears are basic components in a wide range of applications. Currently, the switchgear wiring is executed by human operators due to the complex manipulation tasks. The WIRES Project tackles the development of a suitably designed end effector equipped with a vision system and a tactile sensor for wire-precise manipulation. Preliminary results have been presented in [1,2,3].

Standard approaches to this kind of problem use vision and/or tactile data. Often vision is used alone due to its efficiency in data collection ([4]). However, this solution may fail in the presence of varying lighting conditions and occlusions. The use of tactile sensors helps to improve the success rate by overcoming some environment limitations. As a consequence, there have been many papers where vision and tactile data are integrated in a single approach ([5,6,7,8,9]. The objective of these approaches is the estimation of object characteristics, such as pose, shape, surface features and so on. Among these, some researchers propose interesting algorithms for edge detection [10] that could be considered in future as alternative approaches with respect to the one proposed here in order to improve the estimation quality. At the moment, the estimation quality reached with the approach proposed here is sufficiently high for the task implementation, with a very simple formalization. Some researchers in recent papers [11] use vision systems directly integrated into fingers to evaluate both tactile and image data at the same time and with the same sensing system. Also, this approach demonstrates how the fusion among tactile and vision data can be a good approaches for manipulation tasks. However, none of these past papers tackle the estimation problem of shape and pose of flexible objects like wires.

In previous papers [2,12], the authors presented details of the tactile sensor design and a possible use of tactile data for the reconstruction of the grasped wire shape and the use of the estimated shape for the implementation of an insertion task. In those papers, the model used for the wire was constituted by a quadratic function for the grasped area and a straight line for the part outside the tactile sensor pad. The sensor was mounted on a commercial gripper and preliminary insertion tests have been carried out on a single hole of an electric component fixed on the workbench, with the same wire grasped from a single position.

This paper presents improvements with respect to the previous system in terms of mechatronic solutions that are integrated and tested in a new scenario much closer to real cases. In particular, for this paper, the tactile sensor has been integrated into the final end-effector designed for the WIRES Project, presented and equipped with an electric screwdriver used to automatically complete the connection task. A simplified solution for the wire shape estimation is considered in order to allow its integration directly into the on-board microcontroller. This solution is a subset of that proposed in [12], but is explicitly formalized to be used with the final end-effector in the current study. The quality of the reconstruction has been re-evaluated with the new model, by redefining the quality metric according to the different model, in order to check if the considered simplification does not strongly affect the expected results. Finally, unlike previous papers, the whole system has been tested in a more complex scenario, by grasping, inserting and connecting a sequence of wires in a testing switchgear as shown in the video as supplementary.

2. The Tactile Sensor and the Gripper

The tactile sensor working principle and its design is detailed in [12]. Here, a brief recalling is reported (related to the integration in the gripper). Figure 1 reports some pictures of the developed sensor where the main components are highlighted. The 16 taxels constituted by the optoelectronic components with the deformable layer bonded above represents the transduction part for the sensor. The optical signals are converted in electric signals by using simple resistors and the obtained voltage signals are acquired with a standard Analogue-to-Digital converter. All details about the components integrated in the PCB are reported in [12]. For the integration into the gripper finger, a second PCB with a microcontroller has been developed and connected to the first one. The second PCB is completed by a voltage regulator and a standard connector, which allows to interface the tactile sensor with a standard USB-TTL serial cable. A suitably designed finger case has been realized in aluminum via a 3D printing technique and the extended PCB has been integrated inside this case. The thickness of the designed case is the smallest in order to allow the insertion of the finger among the switchgear components and wires already connected. The case allows a mechanical connection to the gripper by using two screw.

The end effector developed in the WIRES experiment for the implementation of the whole cabling process can be seen in Figure 2. The end effector integrates a 2D camera providing top view of the scene, an computer-controlled screwdriver (to tight the terminal screws) and a 4-DOFs gripper equipped with the tactile sensor. The end effector is also equipped with an integrated torque/controlled screwdriver with remote PLC control and process data recording capabilities (Kolver PLUTO3CA electric screwdriver + EDU2AE/TOP/E control unit). In the final process implementation, the robot arm is used to position the screwdriver tip on the terminal screw, and the FT sensor will be used to control the contact with the screw during the tightening. Therefore, the end effector will be held in an almost fixed position, just the screw motion during the tightening will be compensated. Consequently, the wire insertion will be performed by using the gripper DOFs only. It results that the FT sensor can be used to estimate the interaction between the screwdriver and the terminal screw, but it cannot be used during the insertion and for the wire tightening check, because the magnitude of the force generated by the wire contact is much lower than the one generated by the contact between the screwdriver and the screw, making the former indistinguishable. For this reason, the use of the tactile sensor installed into the gripper fingertips is fundamental also during the insertion and for the wire tightening check, in order to reach a suitable success rate.

Stepper motors with integrated encoder and lead screws have been adopted for the actuation of the end effector. This solution significantly simplifies the control and reduces the weight, the mechanical complexity and the cost of the end effector. Limit switches have been used for absolute position detection on both sides of all the end-effector movement axes. Each motor is driven by a Hydra servo drive control board, used as HW low-level motor controllers. These control boards are arranged on the end effector itself. The communication between the motor control boards and the high level WIRES controller is implemented through CAN bus. A ROS node has been developed to allow the control of the end effector and to ease the integration with other components of the WIRES system. At low level, the motors are controlled by means of the PLCOpen standard, allowing an easy implementation of the end effector controller. The tactile sensor has been integrated into the jaw tips (fingertips). Several versions of 3D printed fingers have been produced in order to evaluated different configurations during experiments.

3. Wire Shape Estimation

A specific sensor reference frame, ${\Sigma }_s(O_s,x_s,y_s)$, is defined at the center of the tactile sensor pad (see Figure 3) and the wire shape estimation problem is tackled with respect to this frame. The 16 taxels are organized as a matrix, where each cell can be identified by its row and column indices. Hence, for each $c_{ij}$ cell it is possible to associate a couple of coordinates $(x_i,y_j)$, corresponding to the physical distances of the cell mechanical center from the sensor frame origin. In particular, the x-coordinates of the columns are −4.5 mm, −1.5 mm, 1.5 mm and 4.5 mm, from left to right, while the y-coordinates of the rows, are 4.5 mm, 1.5 mm, −1.5 mm and −4.5 mm, from top to bottom. The measured voltage variation corresponding to the $c_{ij}$ cell is indicated as $\Delta v_{ij}$.

In this paper, in order to estimate the shape of the grasped wire, this is locally approximated as a straight line, coincident with the longitudinal axis of the wire (see Figure 3), modelled in the ${\Sigma }_s$ frame as the line with equation

$$y_s=m{x}_s+n,$$

(1)

where m and n are the two parameters to be identified by using the tactile data. Since the section of the grasped wire is considered a priori known, estimating the grasped wire shape means to estimate the m and n parameters characterizing longitudinal axis of the wire. The initial position of the wire implies that the grasped wire has the main direction always mainly aligned with the $x_s$-axis (horizontal direction). In this hypothesis, the procedure for the wire shape estimation is constituted by a first step, where the centroid coordinates for each column are computed, and a second step, where the computation of the model parameters in (1) is implemented via a least squares method applied to the data set constituted by the coordinates of the column centroids. In detail,

 step 1:

the y coordinates $y_j^c$ of the column centroids are computed from tactile data as

$$y_j^c=\frac{{\sum }_{i=1}^4{y}_i\Delta v_{ij}}{{\sum }_{i=1}^4\Delta v_{ij}}j=1,\dots ,4,$$

(2)

where $y_i$ is the mechanical y coordinate of the i-th row. Hence, the data set $\overline{\mathcal{D}}$ is constituted by the coordinates $(x_j,y_j^c)$ of the 4 column centroids (where $x_j$ is the mechanical x coordinate of the j-th column).

 step 2:

the model (1) parameters, m, n, are estimated by using a least squares method applied to the data set $\overline{\mathcal{D}}$ resulting from step 1.

The presented procedure has been experimentally applied by grasping a wire in different configurations. Figure 4 reports a generic grasp: the tactile map shows how the cells on the second and the third rows present higher $\Delta v_{ij}$ values. The column centroids (green stars) have been computed by using Equation (2) and than the wire shape has been computed via least squares method (straight line). To assess the accuracy of the algorithm, a comparison among the estimated shapes and the actual ones has been carried out, superimposing a picture of the corresponding grasp to the measured data and estimated shapes. Figure 5 shows the good matching between the estimated and the actual wire shapes. The quality of the shape reconstruction is fundamental to successfully complete the insertion task, as detailed below.

4. The Insertion Task

As discussed in Section 1, the main objective of the WIRES project is the robotic assembly of electric switchgears. To this aim, a fundamental subtask is represented by the insertion of the wire into the holes corresponding to the pins of the electrical components. The successful execution of the task allows the mechanical connection of the wire. The tactile sensor has been integrated into the gripper, suitably designed for the WIRES project in order to experimentally test its funcionalities during the insertion task. The proposed solution for the insertion task described in the following is based on the assumptions that the relative position between the robot system and the switchgear is known and, additionally, the length of the protruding part of the grasped wire is also known. Note that in real applications, a standard calibration procedure for the robot system allows us to obtain a precision that satisfies the first assumption. For the second assumption, the length of the protruding part of the wire can be estimated by using the camera integrated into the gripper as described in Section 2.

A human operator prepares the wire by placing it in a delimited area (based on the gripper stroke), with a random pose. The robotic system is used to grasp the wire and as a consequence, after grasping, the pose of the wire with respect to the tactile sensor is unknown. Then, the grasped wire shape is estimated by computing the model parameters m, n and applying the wire shape estimation algorithm. Figure 6a reports a sketch of a generic grasped wire, with the estimated shape. Let ${\Sigma }_h(O_h,x_h,y_h)$ be the hole frame, with the origin in the center of the hole and the $x_h$-axis aligned with the hole axis; let ${\Sigma }_w(O_w,x_w,y_w)$ be the wire end frame, with the origin in the end point of the wire actual axis and the $x_w$-axis aligned with the wire actual axis; let the frame ${\widehat{\Sigma }}_w({\widehat{O}}_w,{\widehat{x}}_w,{\widehat{y}}_w)$, with the origin in the end point of the estimated wire axis and the ${\widehat{x}}_w$-axis aligned with the estimated wire axis. On the basis of the assumption described above, the poses of ${\Sigma }_s$ and ${\Sigma }_h$ are perfectly known, while the pose of ${\widehat{\Sigma }}_w$ can be computed from the estimated shape parameters m, n and the protruding part L value of the grasped wire. To this aim, the homogenous transformation matrix ${\mathbf{T}}_{\widehat{w}}^s$ can be computed from Figure 6a with simple geometrical considerations

$${\mathbf{T}}_{\widehat{w}}^s=\left[\begin{array}{cccc}cos\gamma & -sin\gamma & 0& lcos\gamma \\ sin\gamma & cos\gamma & 0& lsin\gamma +n\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right],$$

(3)

where $\gamma =arctan\left(m\right)$ and $l=L+a/cos\gamma$ (a is the half side length of the sensor pad). After the computation of ${\widehat{\Sigma }}_w$, a standard technique can be used to program the robotic system in order to align ${\widehat{\Sigma }}_w$ with ${\Sigma }_h$. After that, the resulting configuration is sketched in Figure 6b, with ${\widehat{\Sigma }}_w\equiv {\Sigma }_h$. From this point the insertion task can be easily completed with a linear movement along the $x_h$-axis. In real working conditions the hole diameter D is typically two times larger than the wire diameter d. Note that in this paper, the insertion has been tackled by considering the task a 2D problem; the z-axes are considered all aligned.

5. Assessment of Wire Shape Estimation and Expected Success Rate

In ideal conditions, when the estimation error of the wire shape is zero (i.e., ${\widehat{\Sigma }}_w\equiv {\Sigma }_w$), under the described assumptions, the proposed procedure allows us to align the wire axis and the hole axis, by maintaining the distances between wire and hole edges (both above ${\delta }_a$ and below ${\delta }_b$) equal to the maximum possible value $\overline{\delta }=(D-d)/2$. Obviously, in this case, the execution of the insertion task is guaranteed with a Success Rate $SR=100\%$. In real working conditions, the estimation error of the wire shape implies ${\widehat{\Sigma }}_w\ne {\Sigma }_w$, and since the alignment can be made only between ${\widehat{\Sigma }}_w$ and ${\Sigma }_h$, when the estimation error increases the insertion task may fail. As a consequence, in real conditions the success rate of the insertion task is $SR<100\%$.

The quality of the estimated grasped wire shape and the maximum SR reachable can be evaluated taking into account both the estimation error and the actual diameters of the hole and the wire. In particular, the estimation error can be quantified by considering the relative poses of ${\Sigma }_w$ and ${\widehat{\Sigma }}_w$. The relative pose of these two frames can be represented by the following homogeneous transformation

$${\mathbf{T}}_w^{\widehat{w}}=\left[\begin{array}{cccc}cos\alpha & -sin\alpha & 0& -\Delta sin\alpha \\ sin\alpha & cos\alpha & 0& \Delta cos\alpha \\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right],$$

(4)

where $\alpha$ is the angle between the estimated wire axis and the actual one, while $\Delta$ is the distance between the origins of ${\Sigma }_w$ and ${\widehat{\Sigma }}_w$. In the ideal case, with a perfect shape estimation it is $\alpha =\Delta =0$ and ${\mathbf{T}}_w^{\widehat{w}}=\mathbf{I}$. In real working conditions ($\alpha \ne 0,\Delta \ne 0$), after the alignment ${\widehat{\Sigma }}_w\equiv {\Sigma }_h$ (see Figure 6b), the distances between wire and hole edges depends on $\Delta$ and $\alpha$. In particular, the estimation error implies that the actual position of the wire presents an offset along $y_h$-axis, which is responsible for any failure in the execution of the insertion task. This offset, computed from ${\mathbf{T}}_w^{\widehat{w}}$, is equal to $\Delta cos\alpha$ and it reduces the space between wire and hole edges. The maximum limit for this offset, in order to avoid the unsuccessful execution of the task, is represented by the value $\overline{\delta }$. As a consequence, the following metric

$$\delta =\overline{\delta }-\Delta cos\alpha$$

(5)

can be computed to evaluate both the quality of the grasping and the expected result (success or not) of the insertion task execution. In conclusion, if $0\le \delta \le \overline{\delta }$ the insertion can be successfully completed, while if $\delta <0$ the insertion task cannot be correctly completed. Moreover, the more $\delta$ is close to $\overline{\delta }$, i.e., $\Delta cos\alpha \to 0$, the better is the quality of the estimated wire shape.

6. Experiments

A number of experiments have been carried out to evaluate the proposed approach. For each experiment, the wire shape has been computed according to the procedure detailed in Section 3. Tens of static experiments, with the sensor fixed on the workbench, have been used to evaluate the shape estimation quality and expected success rate. Additional experiments have been carried out to evaluate the actual success rate of the insertion task in real working conditions, by using the sensorized gripper with standard wires and electrical component.

6.1. Estimation Quality and Expected SR

For the first set of experiments, a standard wire with $d=3$mm has been grasped between the tactile sensor, fixed on the workbench, and a transparent methacrylate plate, in different poses. By considering the diameter of the hole of the electric component $D=2d$, it is $\overline{\delta }=1.5$mm. A calibrated optical microscope has been used to take pictures from the transparent plate side. Hence, the offset errors $\Delta cos\alpha$ between the estimated and the actual wire end points can be directly measured from the pictures. Figure 7 reports two sample cases, where the estimated wire shapes are compared to the actual ones. For each considered case the value of the offset error is reported. By using Equation (5) the metric can be computed, obtaining for the cases in Figure 7 the following values: $\delta =0.72$mm for case (a) and $\delta =-0.02$mm for case (b). From these values it is evident that case (a) allows the correct execution of the insertion task, while case (b) does not guarantee a correct insertion phase ($\delta <0$). Note that case (b) corresponds to a grasp configuration close to the diagonal of the sensor pad (that is quite unlikely). The same procedure has been applied to 20 considered experiments. Finally, all grasping cases have been divided into two sets: the first set corresponding to cases with a computed metric $\delta >0$ (17 experiments) and a second set with $\delta <0$ (3 experiments). The expected success rate for the insertion task has been computed, by relating the number of experiments within the first set with respect to the total number of experiments, by obtaining a $SR=85\%$.

6.2. The Insertion Task Implementation

For the implementation of the insertion task, the sensorized gripper has been used. All measurements are reported with respect to the world reference frame $\Sigma (O,x,y)$, placed at the robot base. Position and orientation of ${\Sigma }_s$ with respect to $\Sigma$ is known in each time instant, by using the robot system kinematics. The pose of the electric component hole is defined by ${\Sigma }_h$, assumed known from the switchgear CAD. Figure 8 reports experimental results for the s pose during an insertion task. After the wire grasping, the $x_s$-axis is aligned to the $x_h$-axis during an approaching phase (see Figure 8a), leaving a distance between $O_s$ and $O_h$ equal to $22$mm in this specific case (it is the estimated length L plus the half side length a of the sensor pad). The reached configuration (at $t=10$s) is reported in Figure 9a, where the estimated wire shape and the frame poses (${\Sigma }_s,{\widehat{\Sigma }}_w,{\Sigma }_h$) are reported with respect to $\Sigma$, together with the tactile sensor pad and the component hole. It is evident that without a correction the insertion cannot be completed correctly. The wire shape has been estimated by applying the wire shape estimation algorithm, and the parameters m and n have been used to compute the homogeneous transformation (3). For the experiment reported in the figures $m=-0.0694$ and $n=-3.4866$. By using the computed homogeneous transformation, ${\widehat{\Sigma }}_w$ has been aligned with ${\Sigma }_h$ during the correction phase. Figure 8b shows a zoom of the rotation and the translation applied during the correction. After this phase, the estimated wire axis is aligned with the component hole axis. The reached configuration (at $t=22$s) is reported in Figure 9b, where it is evident that the insertion can now be correctly completed with a simple translation along the x-axis. Figure 8a shows also the final insertion phase. Figure 10 reports the flowchart, where the connections among all subtasks of the whole insertion sequence are reported. Several checks are implemented by using the tactile sensor data during the insertion execution, in order to evaluate if the task is correctly completed or not. During experiments, the wire shape estimation error will affect the final success of the insertion phase. As discussed in Section 5, the actual wire end point is related to the ${\Sigma }_w$ frame, while the estimated wire end point is identified by the ${\widehat{\Sigma }}_w$ frame. To test how this estimation error affects the insertion phase during experiments in real working conditions, the insertion task has been repeated 40 times starting from different initial grasping conditions for the wire. The same experiment described above has been executed and, for each case, the final correct insertion has been evaluated. The number of successfully completed tasks was 33 with a success rate $SR=82.5\%$. The obtained SR is slightly below the expected SR computed in static conditions (see Section 5), as was foreseeable, since during the experiments, additional errors (e.g., robotic system calibration, electric component position) appears together with the wire shape estimation error. Figure 11 reports a sequence of frames extracted from the video (https://youtu.be/oPxkeeQLKi8) related to the paper in order to show how the designed gripper with the proposed approach allow to correctly complete an insertion sequence. Each frame has been marked with the corresponding procedure subtask. The video shows the effectiveness of the proposed approach during a demo. In the video, the robot is used to fix the screwdriver position for the connection, while the insertion is completely implemented by the designed gripper.

7. Conclusions

This paper presented a sensorized gripper for wire manipulation, and in particular, for their insertion into the electric components of a switchgear. The designed gripper integrates tactile sensors suitably optimized for this task. A specific procedure for the insertion task execution has been proposed and evaluated in terms of expected success rate. Experimental results have been reported to show the effectiveness of the proposed strategy. Future work will pprobably be devoted to using the sensor to estimate contact forces between the gripper and the manipulated wire during the whole assembly process of the switchgear.