The proposed method consists of two phases. First, the vision algorithm detects the presence of unknown objects on the scene, segments the scenes to obtain clusters of each object (each cluster is a point cloud) and then, it computes grasping points on the surface of each of the objects (Figure 4
). The method is flexible to obtain grasping points of objects even changing the scenario providing that objects are located on a table or flat surface. Once the vision algorithm provides the robot with the optimal grasping points of the object, the robot plans the trajectory in order to position the robot hand to grasp the object. Occasionally, the grasping of the object is not optimal. For this reason, the method adds a second phase which is used to plan fine hand robot-object interactions. In this step, EMG-based teleoperation of the robot hand-arm is performed to accomplish a successful and stable grasp without slipping and avoiding damage to the object.
3.1. Grasping Points and Pose Estimation
The algorithm calculates pairs of contact points for unknown objects given a single point cloud captured from a RGBD sensor with eye-to-hand configuration. Firstly, the point cloud is segmented in order to detect the objects present in the scene. Then, for each detected object, the algorithm evaluates pairs of contact points that fulfil a set of geometric conditions. Basically, it approximates the main axis of the object using the major vector obtained by running a Principal Component Analysis (PCA) extraction. Then, it calculates the centroid in the point cloud. With this information, it is possible to find a cutting plane perpendicular to the main axis of the object through its centroid. The candidate contact areas are at the opposite edges of the surface of the object that are close to the cutting plane. A standard grasping configuration consists of one point from each of these two areas. Figure 4
shows all these steps graphically.
These candidate areas, in which the robot hand can be positioned, contain multiple potential points so the vision algorithm evaluates a great variety of grasping configurations for the robot hand, using a custom metric that ranks their feasibility. Thereby, the best-ranked pair of contact points is selected, since it is likely to be the most stable grasp, given the view conditions and the used robotic hand. The algorithm takes into account four aspects: the distance of the contact points to the cutting plane, the geometric curvature at the contact points, the antipodal configurations and the perpendicularity to the contact points.
The first one, distance of the contact points to the cutting plane, is important because it is assumed that the grasping of the object is more stable as the robotic hand grasps closer to the centroid of the object, which is an approximation of its centre of mass. This way, the inertial movements caused throughout the manipulation process of the object are more controllable. The second aspect, the curvature, is considered to avoid the grasps of unstable parts on the object surface. The goal is to place the fingertips on planar surfaces instead of highly curved areas that are prone to be more unstable. Grasping objects on non-planar areas can cause a slip and fall of a grasped object when it is being manipulated, for example, if the robot arm executes a lifting movement. Regarding to the third aspect, contact points should be located on places where the robotic fingers can apply opposite and collinear forces (antipodal configuration). Finally, it is desirable to have contact points that are connected by a line perpendicular to the main axis of the object. That is, the contact points are equally distanced from the cutting plane.
The aforementioned aspects are used to define a quality metric to evaluate the candidate contact point and to propose the best grasp points to carry out a successful grasp of the object on the scene. Accordingly, this quality metric ranks with greater values the grasping configurations that place the robotic hand with its palm point towards the object, its fingertips perpendicular to the axis of the object, parallel to the cutting plane and close to the centroid of the object. Notice that this operation is performed for every detected object. Consequently, the final pose of the robot hand is calculated using the best ranked grasping configuration and the approximated main axis of the object.
Our vision algorithm only computes pairs of contact points. This is assumed to avoid the method being dependent on the type of robotic hand mounted at the end of the robotic arm. Two points are the minimum required for a simple robotic gripper but also, any multi-finger robotic hand can adapt its grasping configuration to two points on the object surface. In the experiments, we use an Allegro hand with four fingers, one of which acts as the thumb. In practice, it is assumed that the grasps will be done with three fingers. This number has been limited to three because the Allegro hand size is often bigger than the object size which will be grasped.
In order to perform three-finger grasps, the algorithm takes into account the following criterion: one of the contact points corresponds to the place the thumb must reach during a grasp, while the other contact point remains between the first two fingers (index and middle). This means that the first and second finger wrap around the second contact point. In this way, the grasp adapts its configuration to only two contact points even though the hand uses three fingers. In addition, the robotic hand is oriented perpendicular to the axis of the object, meaning that it adapts to the pose of the object.
When the human operator has selected the desired object that will be grasped, the robotic system guided by the vision algorithm performs the following steps to reach it:
First, the robotic hand is moved to a point 10 cm away from the object. This is a pre-grasping position which is used to facilitate the planning of the following steps. The pre-grasping position is computed, from location (position and orientation) of contact points on the object surface, by the vision algorithm previously described.
Second, the robotic hand is moved forward facing the object with its palm and the fingers opened. In this step the hand reaches the point in which, after closing, it would place the fingertips on the calculated contact points.
The correctness of this position depends on the calibration of the camera position with regards to the world’s origin as well as lighting conditions and reflectance properties of the objects in the scene. Owing to this, the proposed method performs the correction of the robot hand using the sEMG signals. But also, sEMG can be used to accomplish a proper grasp of objects in a complex manipulation.
3.2. Collaborative System with Both Visual and Electromyography Data
The proposed solution has been implemented using the ROS in order to develop nodes in charge of different responsibilities but keeping a communication framework among them. One node has been created, called pointcloud_listener, where point clouds are read and processed to perform the calculus of the grasp contacts. This node publishes a custom ROS message called GraspConfiguration where the point clouds of the objects and the calculated grasp contacts are stored.
Another node, called allegro_control_grasp
, subscribes to this topic and reads the published contact points to generate a grasp pose for the robotic gripper. Then, it proceeds to plan a trajectory following the steps listed in the previous section. MoveIt! [29
] has been used to perform this trajectory planning. Once it reaches the grasping position, the EMG control starts. To do so, it subscribes to a topic called/emgsensor
where the correcting movements are published.
These corrections are published by a third node called emg_reader, which processes the sEMG signals in order to provide messages of type geometry_msgs/Quaternion. This type of ROS message allows us to describe the direction of movement for the arm that the operator wants to perform in order to correct the position of the robotic gripper. Thus, using one of the axis of the Quaternion, we can specify in which axis we want to move the gripper. The w term is set to 1 when we detect the grasping pattern in the EMG signal so the allegro_control_grasp node closes the gripper and continues to lift and carry the object.
It is important to note that this message is constantly published by the emg_reader
node but the allegro_control_grasp
only reads them after performing a correction. This means that messages published during the physical movement of the robot are ignored and, as soon as it stops, the control returns to wait for a new message in the topic. Figure 5
shows a scheme of the nodes and their interactions through ROS.