Many control strategies have also been proposed for the safe and efficient handling of non-rigid objects. This part of the survey aims to provide a high-level overview of the different techniques and concepts used to accomplish this task in recent works. As such, there is a focus on the selection and planning of the manipulation task and trajectory rather than on the low-level control and driving of the actuators. Most of this section is organized in terms of the type of object being manipulated. Section 4.1
describes the handling of linear objects such as ropes, wires, cables, and flexible beams, in terms of both tying knots and routing, shaping and otherwise moving them. Section 4.2
groups the research that works with planar objects such as metal and plastic sheets, as well as all cloth-like materials. The tasks discussed in this section include sorting and folding laundry, assisted dressing, and cancelling unwanted deformations in an industrial setting. Section 4.3
presents research that is concerned with shaping the outline, or 2D projection of non-rigid objects. The shaping and handling of 3D objects is discussed in Section 4.4
. Recent machine learning approaches are grouped in Section 4.5
and papers that work with more general problems or that can hardly be described by the previous categories are discussed in Section 4.6
. A summary of the surveyed control strategies can be found in Table 3
at the end of the section.
4.1. Linear Objects
Many automation tasks are concerned with the handling of linear non-rigid objects, such as the knotting and routing of ropes and cables. Research exploring the tying of knots includes the work of Wakamatsu et al. [48
], who studied the knotting and unknotting of rope with a topological description of the object’s state as a set of oriented crossings. They define four basic manipulation operations to perform state transitions and use them to derive a high level plan to take the object from its current state to an arbitrary target state. From this plan, they computed a sequence of approximate grasp points and directions of motion to perform the knotting/unknotting operation with a single manipulator. They also established a planning method to determine which parts of the rope should be pulled in order to tighten a knot. Saha and Isto [49
] built upon a similar topological description and added the possibility of crossings of the rope with a rigid object, e.g., a beam, box or other support. Their planner makes use of optional “sliding supports” (knitting needles) to hold parts of the unfinished knot in place during manipulation. The resulting system uses two collaborating manipulators to tie self knots and knots around static objects. Bell [50
] developed a system to tie knots without any sensing, by using static fixtures and a system of tracks to handle stiff linear objects such as steel wire. More recently, Wang [51
] explored multiple techniques to tie knots with minimal sensing. These techniques revolve around the use of automatically-designed mechanical fixtures and a local string control strategy using vector fields. They also studied the different constraints required to tie a knot with this system, such as the required number of contact points and regrasp operations.
Works on routing and shaping linear objects include Moll and Kavraki [52
], who developed a path planning algorithm for shaping flexible wires with robotic grippers holding both endpoints. They defined that the wire is in a “stable” state when its shape matches a minimal energy curve, and restricted the planner to such stable states. This reduces the size of the state space and ensures that the strain on the wire is minimal throughout the manipulation process. It is then possible to explore the entire space of collision-free states in order to bring the wire to a desired configuration. Tavasoli et al. [53
] studied the case of two planar robots cooperatively handling a flexible beam. With the goal of suppressing the unwanted vibrations of the beam, they built upon the observation that the positioning and vibration happen in two different time scales. They decoupled the system into “slow” and “fast” subsystems. The slow system behaves as a fully rigid system and is used for the positioning task while the fast system is only concerned with the vibration of the object. The resulting composite controller is able to perform the position tracking task while suppressing the unwanted vibrations of the beam. Ding et al. [54
] also considered a technique for suppressing unwanted vibrations during the manipulation of a flexible beam. Their approach is to decompose the system into actuated and underactuated parts, and use a position-based control strategy with sliding mode control to quickly dampen the vibrations. Shah and Shah [55
] considered the task of attaching heavy cable bundles with fragile interconnections to an aircraft fuselage. The fragility of the interlinks adds severe constraints to the manipulation and anchoring tasks. Their simulation-based planner incorporates gravity in the computation of the cable shape, and carefully selects a sequence of grasping and anchoring points to ensure that the interlinks are not damaged in the assembly process.
Overall, recent works on knotting involve complex operations with the added challenges of having an arbitrary initial state [48
], interactions with other objects [49
], or limited sensing [50
]. Similarly, works on routing cables add the constraint of minimizing strain on the cable or specific parts of it [52
], while moving a flexible beam requires cancelling its vibrations in an optimal manner. These added constraints on the performed tasks and the small number of recent works indicate that the shaping and handling of linear non-rigid objects is already a topic that is quite well known, and that researchers are moving to more complex manipulation tasks.
4.2. Planar Objects
Most of the research effort in handling non-rigid objects is currently concerned with planar objects, i.e., objects that may be modelled as a bidimensional mesh. This includes fabrics and leather, as well as sheetmetal and other thin sheets of flexible material. These materials are often considered to have negligible stretchability, and the deformation is therefore normal to the plane of the resting object.
An everyday application of the automated handling of cloth is folding laundry. Bell [50
] derived the number of grasp points necessary to immobilize a polygonal cloth and showed that this number could be greatly reduced by simple manipulations that make use of gravity. They used these findings to develop a system to fold a T-shirt with a fixed rod and a manipulator with few degrees of freedom. Maitin-Shepard et al. [56
] used a vision-based geometrical approach to detect and grasp the corners of a towel and fold it with a series of regrasps. Cusumano-Towner et al. [57
] furthered this approach by using a Hidden Markov Model to recognize the behaviour of the cloth and detect its type during an initial “disambiguation” manipulation phase, which enables them to bring various articles into a configuration that is suitable for future operations such as folding. Bersch et al. [58
] presented a system to find possible grasps on a crumpled shirt and evaluate their chances of success with machine learning. While using folds detected through 2D and 3D vision data, this heuristic strategy fits a simplified gripper model around the point cloud of the desired grasp point to compute the valid gripper poses for holding the shirt. A function learned automatically with a support vector machine is then used to evaluate the quality of the potential grasp point and gripper pose based on geometrical features that enhance the success rate. Their system then iteratively moves its grasp points towards the shoulders of the shirt before executing an open-loop folding routine. Miller et al. [59
] presented a motion planning algorithm to perform a user-defined sequence of folds on a cloth laying on a table. Their approach greatly reduces the complexity of the problem by dividing the cloth into two quasi-static polygons: one that lays flat on the table and one that is suspended by the grippers. Willimon [60
] also considered the problem of flattening a crumpled or folded piece of laundry. Their system is able to grab an item from a pile of laundry, classify it using RGBD data and active manipulation, and unfold it before moving on to the next item. Li et al. [61
] used a simulation-based approach to find the optimal end effector trajectory during a single folding motion. Given the start and end positions of the manipulator, an offline simulation explores possible trajectories in order to minimize the error between the result and the user-defined “folded” state. This optimal trajectory avoids errors such as dragging the cloth (if the path is too low) or lifting and piling it up (path is too high). The learned trajectory can then be generalized to garments of similar shapes.
In a series of papers which are summarized in [62
], Doumanoglou et al. developed a complete system to fold a pile of crumpled clothes with a dual-armed robot. The first step of their pipeline, picking up a single garment from the pile, is done by using a depth camera to detect folds in the pile, which are “the most suitable grasping points even for humans”. One of the grippers is then moved to grasp the fold that is the highest in the pile. In the second step, unfolding re-grasps, the pose of the object is defined by the relationship between the current grasp points and the manually selected grasp points which will cause the cloth to unfold due to gravity. This representation allows the grippers to be moved towards the optimal grasps easily. Once the garment is laid on the table, an intermediary step is necessary to completely flatten it. Here, a small brush is used by one of the arms to push the detected folds towards the exterior while the other arm holds the cloth in place. Finally, the folding procedure is performed by matching the detected cloth polygon with a predefined triangular folding move (g-fold).
Sannapaneni et al. [63
] developed an algorithm for learning a folding sequence from the visual detection of special markers attached to key points of the cloth during a demonstration. The learned sequence is then generalized to handle different sizes of clothes which have the same shape as in the demonstration. Yang et al. [64
] preferred to use a deep learning approach with teleoperation training. The learned model is combined with the robot’s sensor data at runtime to generate the folding plan. This combined approach increases the robustness of the system and allows the robot to complete or restart the task when disturbed by an external event. Jia et al. [65
] used a “visual feedback dictionary” to map visual features of the cloth material to velocities of the end effector. Combined with their representation of features as “histograms of oriented wrinkles” with high and low-frequency components, this allowed them to complete complex manipulations with little training data.
Some recent work has also tackled the task of assisted dressing, where a person with limited mobility would be helped by a robotic assistant in the everyday task of dressing up. The system of Yamazaki et al. [66
] helps a sitting person put on a pair of pants. They used the optical flow to estimate the state of the clothing and developed a path planning algorithm that adapts to the length, size and position of the person’s legs. Moreover, the system is able to recover from failures, such as the bottom of the pants getting stuck on a toe, by attempting to revert to the previous state. Gao et al. [67
] used an online iterative optimization algorithm to find a person’s preferred dressing path for putting on a sleeveless vest. The general human pose and movement space is recognized through vision sensors, while force control is used for local optimization. The initially configured dressing path is therefore iteratively driven towards the path of least resistance, which is considered to be the person’s preferred dressing path. Zhang et al. [68
] used a hierarchical task structure in which the path planning task is subordinated to the task of minimizing the interference on the user’s movements. This system is therefore able to help someone put on a top while taking into account their movements as well as their previously estimated range of motion. The interference minimization is accomplished mostly through force sensing as the occlusions from the robot and clothing prevent accurate real-time visual pose estimation.
In an industrial setting, Flixeder et al. focused on the task of layering sheets of cloth-like material over a rigid shape, as it is a common problem for tasks such as fiberglass reinforcement and shaping leather. In [69
], they experimented with multi-arm manipulation and tested various control strategies for position, force-impedance, and parallel position and force control. In [70
], they proposed a mechatronic design as well as control and planning strategies for accurate lay up of flexible strips on a complex mold. The work of Li et al. [71
] is focused on cancelling the unwanted deformation of a flexible PCB prior to soldering it. This is done with adaptive region control that allows an assistive arm to control the deformation from any point in the desired region. Significant effort is also devoted to ensuring that the deformation is stabilized before the soldering is performed, while completing the task in as little time as possible. Another approach to cancel unwanted deformations in large plates of sheetmetal or plastic was explored by Park et al. [73
], who embedded a “smart damper” in the gripper. As the end effector cancels most of the deformation, the rest of the manipulator arm can be controlled as if the object was fully rigid.
Some research work attempted to solve more general problems related to the handling of non-rigid sheets rather than focusing on a specific task. Zacharia et al. [74
] proposed a model-free approach to handling flexible sheets laying on a table by augmenting fuzzy logic with genetic algorithms for visual servoing. Shibata et al. [75
] used simultaneous control of the displacement and deformation of a piece of cloth to perform the “wiping motion”, one of the motion primitives used by humans when handling fabrics. Kinio and Patriciu [76
] showed that the
controller was superior to a PID controller for indirect target point manipulation in a sheet of silicone. Elbrechter et al. [77
] developed a path planning algorithm for anthropomorphic hands with different friction coefficients to fold a piece of paper. This is done using visual and tactile feedback as well as real-time physics modelling of the sheet of paper. In their approach, a hierarchical state machine allows dynamically switching between different controllers, e.g., one to achieve contact with the paper and one to maintain contact. They notice that the motion normal to the surface (to maintain contact) is controlled with tactile feedback while the motion in the tangential direction (to fold and crease the paper) is controlled independently with visual feedback. Bodenhagen et al. [46
] used a learning approach with multiple learning and estimation phases to compute the ideal actions to apply to tasks such as conveyor grasping, peg-in-hole and laying down of a thick silicone sheet. Kruse et al. [78
] used only force and vision feedback for the collaborative human-robot handling of a large sheet of fabric, where the robot follows the motions of the human while keeping the fabric taut. They later expanded this work in [79
] by considering a mobile robot, which allows for a greater variety of tasks such as carrying a sheet around a corner.
Other interesting approaches to the manipulation of non-rigid sheets include the work of Dang et al. [80
], who proposed a solution to control the general shape of a flexible surface with potential field based control of an array of microactuators embedded in the object. Patil and Alterovitz [81
] worked on the problem of automated tissue retraction by surgical assistance robots. This task involves grasping and peeling back a thin layer of tissue to make the underlying area accessible to the surgeon. A sampling-based planner is used to explore the space of possible grasp points and paths to minimize the maximum deformation energy, the maximum stress and the total control effort needed to provide sufficient exposure of the underlying area. Their system relies on a physical simulation of the model and can be used to select an initial grasp point, optimize a human-defined retraction path or compute the entire motion sequence while avoiding obstacles. Inahara et al. [82
] performed the non-prehensile stretching and compression of a thin wheat dough by controlling the acceleration of a vibrating plate on which the object lays. This was later improved in Higashimori et al. [83
] by allowing the object to “jump” from the plate during shaping. From the realm of computer graphics and animation, Bai et al. [84
] developed an algorithm to compute the necessary joint torques of simulated anthropomorphic hands in order for a cloth to follow a defined motion. It is interesting that the task description is given as a path to be followed by any point(s) of the cloth. The forces on the contact points are automatically derived to bridge the state of the cloth and that of the hands, which is formulated as a model-predictive-control problem. Recently, Cocuzza and Yan [85
] explored the shaping of a thin rheological object (fondant icing). They developed a method to identify the material’s properties from tensile tests and used this model to compute an optimal motion path for a serial manipulator to shape the icing over a cake. The manipulation speed is increased by using the object’s elastic deformation properties to control the final plastic deformation.
The handling of planar non-rigid objects is rapidly becoming a well-researched topics, with cloth-like materials receiving the most attention. This is easily explained by the number and variety of applications that they open for industrial and household automation. First, in the timeframe covered by this survey, the automated folding of laundry has moved from open-loop detection and folding for a single towel to full-fledged systems able to autonomously fold a pile of assorted garments. It appears that the main challenge for such a system is the extraction and identification of the item to fold before bringing it into a suitable initial configuration. This is due to the infinite number of poses that clothing may take. Assisted dressing, a second household task involving clothing, has also received some attention. Here, the focus is on eliminating risks of injury and increasing human comfort. This is done by using a combination of vision, tactile and force sensing to deal with occlusions and by developing algorithms that are able to comply with unexpected movements during the dressing session. However, these systems have not yet achieved much flexibility and are only able to handle a single garment and a fixed initial position.
A wider variety of materials is used in non-household environments, with the primary goal of shaping thin sheets of material. The control schemes used in these applications are as different as the tasks to which they are applied, but a common strategy is the use of force control in addition to position control, especially for materials that show some elasticity in the direction of the desired deformation. The use of tactile and force feedback is also a frequent addition to the primarily vision-based systems. Overall, and even though some areas have already been explored, the handling of planar non-rigid objects is an active research field with many unsolved problems and potential improvements.
4.3. 2D Projection of Objects
Another subject of interest with regards to the manipulation of non-rigid objects is the active shaping of the 2D projection of an object. This includes tasks such as controlling the shape of an object’s contour in an image, or indirectly moving internal object points to specific targets. Gopalakrishnan and Goldberg [86
] considered the case of grasping an object with two frictionless contacts. They discussed the concept of “deform closure”, which is the “deformation space” equivalent to holding a similarly shaped rigid object in form closure with contact points in concavities. They computed the optimal jaw separation by balancing the energy needed to release the object with the energy that would cause a permanent deformation.
], Das explored multiple tasks related to the 2D control of non-rigid objects. First, they developed a planning algorithm to shape the contour of an object into a desired curve with an arbitrary number of planar manipulators. This system also computes the location of the contact points on the original contour from their position on the deformed contour. The second controller they designed is for collaborative target point positioning. In this case, multiple fingers or manipulators work together to bring an internal point to a desired position by applying forces to the object contour. Their final task is a path planning algorithm for inserting and positioning a bevel-tip needle through a soft tissue. The needle is only rotated by increments of
so that the deformation of the tissue—and therefore the path of the needle—are kept in the plane. This allows precise positioning of the needle tip.
Higashimori et al. [88
] presented a method to actively shape a rheological object with unknown viscoelastic properties by separating the plastic deformation from the elastic deformation. To observe a large deformation and accurately estimate the material’s properties without causing unwanted deformation, they set the maximum deformation in the “parameter estimation” phase to the desired deformation. Once the elastic response is known, they used integral force control to drive the plastic deformation. This allows for shorter manipulation times than position based control as it is not necessary to wait for the elastic deformation to dissipate. A similar property was exploited by Yoshimoto et al. [89
] to shorten manipulation times. Once the elastic properties of the object are known, they used force control with visual feedback to drive the deformation stress. Once a sufficient force has been applied, the elastic recovery automatically drives the object towards the desired shape without further interaction, even though an excessive deformation was temporarily applied.
], Das and Sarkar presented techniques to shape the outline of a planar rheological object with multiple manipulators. They discretized the object boundary into a number of control points equal to the number of manipulators, and developed an optimized motion planner to pick the initial contact points. This selection is based on minimizing an energy-like parameter between the desired curve and the original one. In [91
], they augmented their controller with the shape Jacobian matrix, which maps the general shape of the object with the local shape changes at the control points. This also enables their planner to compute the optimal intermediate shapes as the object deforms. In [92
], they expanded this work to move internal control points towards target locations by applying minimal forces to the object boundary. The total forces applied to the object are minimized by monitoring energy dissipation in order to select appropriate actuation points.
Alonso-Mora et al. [93
] presented a system to achieve the collaborative manipulation of large flexible objects such as bedsheets with a separate wheeled robot handling each control point. The object is modelled as the triangulation polygon of its 2D projection (top view) with constraints on the minimum and maximum of the variable-length neighbour distances to prevent overstretching and excessive sagging. They used a centralized planner which defines the task as a change of configuration of the object polygon. Low level planning is left to the individual robots, which do not communicate directly. Instead, they observed the position of their neighbours and the force transmitted through the object to determine the object’s current configuration. A receding horizon local planner is implemented on each robot to compute its next position and avoid collisions by solving a velocity optimization problem. Recently, Navarro-Alarcon and Liu [94
] proposed a new representation of the object contour based on a truncated Fourier series. This representation, while being more compact than spatial-domain approaches such as point cloud, allows them to effectively ignore the high frequency components during visual servoing. This provides their system with more speed and reliability when dealing with objects that have a jagged contour, or if the desired shape is defined by a rough sketch. Instead of computing the object’s full deformation model, their algorithm performs the online local estimation of the deformation properties. This information is used to iteratively drive the object towards the desired shape within the possibilities of the predefined control points.
2D control of non-rigid objects covers many use cases where the desired deformation and the applied forces exist in the same plane. As such, it may be a reasonable simplification for handling 3D objects when the deformation along a certain dimension can be safely ignored, or when depth information is not available and would be difficult to collect. Once again, the recent research presents a variety of approaches, but a few trends still emerge. First is the prevalent use of multiple manipulators to handle potentially large objects [87
], where sophisticated collaboration schemes are allowed by the reduced computational complexity of a 2D workspace. Another interesting strategy is the use of force control on the elastic deformation of the object in order to induce a desired plastic deformation in a shorter time [88
]. Overall, it appears that most of the surveyed approaches use 2D control as a computational simplification for handling 3D objects when the depth constraints can be integrated in the 2D space [93
] or ignored.
4.4. 3D Objects
Recently, researchers have turned their attention to the automated handling of 3D non-rigid objects. Earlier works by Navarro-Alarcon et al. focused on the ability to create specific point and angle-based deformation features that can be observed by a 2D vision system. In [95
], they built a dynamic velocity control law for a single control point on the object. They also used an iterative estimation of the deformation Jacobian based on 2D visual feedback only. This technique allows them to avoid the need for prior knowledge and modelling of the object while explicitly controlling the object’s elastic deformation through real-time visual servoing. Multiple types of deformation features were explored, namely moving a given point to a target location, creating a certain angle, and changing the distance between points. They expanded this work in [98
] by presenting an analytical, energy-based solution for active deformation while retaining the adaptive behaviour of their controller and its ability to function in uncalibrated environments. They also developed a solution which does not need to compute the deformation optical flow in real-time by making use of information from offline deformation tests. They also considered curvature-based deformation features. In [99
], they developed a controller which is able to make use of all 6 degrees of freedom available to their manipulator, which was not the case with their previous experiments. This larger range of motion allows for more flexibility in the deformation tasks, in terms of both simultaneously controllable features and the number of reachable configurations. More recently, in [12
], they used stereoscopic vision feedback and a similar control algorithm for controlling 3D deformation features with two manipulators. Once again, no prior knowledge of the object model is required as the vector of deformation parameters is estimated in real time.
In recent work, Delgado et al. focused on in-hand manipulation of elastic objects with tactile control only. In [100
], they started by classifying objects between rigid and non-rigid based on the “sensation of rigidity” and total displacement when varying the contact forces applied by the robotic fingers. Then, they introduced a planning and control system to maintain and adapt the contact forces while performing basic manipulation tasks such as lifting, rotating, squeezing and moving the object. This control strategy is based solely on tactile information and does not depend on an object model nor on knowledge of its weight and friction coefficient. In [18
], they explored similar tasks while taking into account a minimalist spring-based model of the object. An initial exploration phase is used to find the object’s elasticity parameters (Young’s modulus) by varying the applied forces and measuring the displacement and stiffness. These data are then integrated in a basic model which connects each contact point to the object’s centre of mass with a spring. Afterwards, manipulation tasks are planned and performed mostly by readjusting the fingers’ positions and applied forces. In [101
], they used tactile images created by dynamic Gaussians as a common representation for tactile data. This representation allows merging data from sensors with different resolutions and provides a high-level interface for controlling the pressure applied by each finger. Their system allows both the creation of tactile images from observed pressure data and the control of the robot fingers based on a desired tactile image. They tested this control strategy with different bimanual tasks where each hand is equipped with a different sensor technology and must respect a global desired tactile image while the handled object is being bent, folded or otherwise moved by the robot.
The research on control of 3D non-rigid objects is quite recent and focuses on basic shaping tasks and in-hand control. Even though the small number of research groups discovered in this survey does not allow for much generalization, some trends may still be noticed. The control tasks are based on feedback from a single type of sensor, resulting in either visual servoing or tactile control. Moreover, they use only a minimal representation of the object, if any model is used at all. This leads to reactive strategies that must be constantly adjusted based on real-time observations. Overall, the manipulation of 3D non-rigid objects is a topic that is still in its infancy.
4.5. Learned Control
Given the recent popularity of machine learning algorithms, it is interesting to highlight their applications to the handling of non-rigid objects. Recent research works include Li et al. [61
], who used multiple simulation experiments in order for their system to learn an optimal end effector path for a single folding motion. Lee et al. [102
] presented a method for learning force-based manipulation “skills” from multiple demonstrations. Their system warps the demonstrated forces and end effector positions to match the current situation, and applies statistical learning to combine multiple demonstrations in order to perform a new task and automatically select a tradeoff between the error in position and in force. The demonstrations are done using either teleoperation or direct “kinesthetic teaching”. They have tested their system with tasks such as tying knots in ropes of various lengths, flattening a towel and erasing a whiteboard. Tang et al. [103
] presented a new method to warp learned paths to new situations. In their work, the function relating the trained shape to the test shape is derived in the tangent space instead of in the cartesian space. Contrary to the “point cloud” cartesian mapping, this tangent mapping preserves the structural information of the object, therefore eliminating the risk of overstretching the object when warping manipulation paths from the training scene to the test scene. They tested their algorithm by shaping a simulated cable.
Sannapaneni et al. [63
] developed a system that learns a folding task from visual demonstrations. Special markers are attached to key points of the object while it is folded by a human operator. The system learns the marker paths and is able to generalize them for folding articles of different sizes but identical shape. Yang et al. [64
] used teleoperation training with deep learning in a folding task. The robustness of the system is increased by combining the learned model with sensory data in real-time to compute the motion plan and recover from disturbed or interrupted tasks. Langsfeld [104
] developed a system that learns multiple new tasks by observing a human performing the task as well as by iterative approximation. Tasks such as pouring a specific volume of fluid in a container and cleaning a compliant part were successfully learned and generalized to handle different parameters. Hu et al. [105
] performed the online learning of an object’s deformation model by using a Gaussian Process Regression algorithm that selectively ignores uninformative data. This learned model is used to build a visual servoing controller to manage the 3D deformation of various objects. This system is evaluated by performing tasks such as bending a rolled towel or a plastic sheet, folding a towel, and placing a piece of fabric such that pins may be inserted in specific locations.
Machine learning was shown to be a powerful tool when dealing with the complex interactions involved in the control of non-rigid objects. It is especially valuable when it allows to avoid the need to build an explicit control algorithm for tasks that may be difficult to describe formally, or when it is not feasible to capture all of the task parameters. Overall, the most popular learning approach for robotic manipulation of non-rigid objects appears to be learning by demonstration, where the robot learns and generalizes a task by “watching” a human perform it. This is a powerful paradigm as it allows a general-purpose robot to perform multiple tasks with non-rigid objects without the need to build a separate controller for each task.
4.6. Other Control Strategies
Even though the categories presented in previous sections cover most use cases and strategies for controlling robotic manipulators handling non-rigid objects, many interesting approaches are not easily related to any of these specific groups. The research presented in this section discusses these exotic control strategies, solutions to general problems which are not limited to a single object geometry, and some unusual applications.
Smolen and Patriciu [106
] explored surgical applications with a simulation approach to deformation planning. They used the reproducing kernel particle method to simulate a soft tissue that is manipulated by several control points at its boundary. The goal of the planner is to move internal control points to target positions by applying forces to the external manipulation points. Goldman et al. [107
] presented multiple algorithms to enable surgical robots to autonomously map the shape and stiffness of living tissues. Their techniques rely on force and position data to capture the object parameters in the immediate probing region. A hybrid force–motion controller performs the exploration sequence in a user-defined area and uses a recursive algorithm for multiresolution sampling based on the local stiffness differences. They also considered the exploration of “deep” features by following the natural boundaries defined by stiffness segmentation.
Sugaiwa et al. [108
] presented an algorithm to set the grasping force for an initially unknown object by measuring its physical properties. Their in-hand sensing approach uses the deflection of a passive mechanical element to detect the moment of deformation or slippage, which they combined with the applied forces and hand configuration to deduce specific properties. First, they measured the forces required to create a dent in the object as well as to prevent slippage when lifting. All measurements are done by incrementally increasing the force applied to the object until it starts to move. The object’s stiffness (denting force) is measured by checking for discrete deformations, its weight is computed by attempting to roll it on the table, and the friction coefficient between the object and the fingers is measured by pushing the object “into” the table while holding it by its sides. The lift-off force is then computed based on the object’s weight and friction coefficient. The signed difference between the denting force and the lifting force is used to classify the object as rigid, soft or excessively soft, and the grasp force is set to the lifting force for rigid and excessively soft objects, and to the denting force for soft objects. This allows setting the grasping force as high as possible to avoid slippage while minimizing the deformation.
] used the concept of diminishing rigidity to avoid modelling and simulating the objects being manipulated. This principle states that the effect of a force on a non-rigid object diminishes as the distance to the application point increases. They used this property to quickly estimate the deformation Jacobian which is used to drive internal control points towards targets based on external forces. They applied this technique to tasks such as tying a rope around a cylinder, spreading a cloth on a table, and collaborative folding, all while correcting for overstretching and avoiding obstacles. Frank et al. [25
] included the handling of non-rigid objects in the path planning system of a mobile robot. As the robot navigates the environment, it interacts with surrounding objects to estimate their deformation properties and takes them into account when building a model of the environment. This allows the platform to consider passing through a curtain or pushing aside a plant in order to reach an otherwise inaccessible area.
Essahbi et al. [110
] worked with the muscle separation process in the meat industry. The goal is to dynamically generate the pulling and cutting tasks for a multi-arm system to separate the meat from the bone. The workpiece is detected and modelled with input from a structured light vision system and force sensors, and the cutting path is selected based on tissue curvature. The object model predicts the tissue behaviour once it is cut and helps in setting the force applied by the cutting and pulling arms. Langsfeld et al. [111
] automated the bimanual cleaning of a compliant part. Initially, only the geometrical shape of the part is known to the system. The part stiffness is discovered during the cleaning task and integrated as a linear finite element model. The goal is to achieve efficient path planning for the cleaning arm as well as for the grasping arm, as regrasps might be necessary in order to hold the part closer to the area being cleaned and avoid excessive deformation. Wnuk et al. [113
] performed a general analysis of a complete bin-picking scenario where the system could handle non-rigid objects through simulation. They described the different steps of the process, namely object localization, approach, grasping, and subsequent manipulation, as well as the different challenges faced during each step. This work allows them to develop the hardware and software requirements for the successful completion of the task, as well as provide a theoretical system architecture to meet these requirements.