# Singulation of Objects in Cluttered Environment Using Dynamic Estimation of Physical Properties

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## Featured Application

**Singulation of objects in a cluttered living room environment to enhance the grasping and identification capabilities of a robot manipulator. Singulation of objects in autonomous assembly by robot manipulator.**

## Abstract

## 1. Introduction

## 2. Methodology

## 3. Impulse Modeling

#### 3.1. Collision between Manipulator and Object

**n**at the contact point:

#### 3.2. Collision Kinematics between Objects

#### 3.3. Condition for Scattering and Pushing

## 4. Learning Physical Properties

#### 4.1. Coefficient of Friction

#### 4.2. Coefficient of Restitution

#### 4.3. Mass Ratio of Objects

#### 4.4. Exactness of Estimated Parameters

## 5. Scattering Algorithm and Experimentation

- (i)
- Initially, we assumed that all objects are placed together.
- (ii)
- The distance among objects after scattering does not go beyond the given workspace. Maximum and minimum distances $\left({S}_{\mathrm{min}},\text{\hspace{0.17em}\hspace{0.17em}}{S}_{\mathrm{max}}\right)$ are given.
- (iii)
- Hitting strategy: Among objects, the first priority is to hit the largest object in the cluster for an initial collision to scatter all objects. The reason for hitting the largest object is to provide enough impulse to drive out other smaller objects.
- (iv)
- How to hit the largest object: This is based on a central impact to simplify the number of parameters. Other than that, the number of parameters needed to estimate the physical properties are increased.
- (v)
- Magnitude of the velocity: In terms of deciding the magnitude of colliding velocity, it is based on the scattered distances among objects. We conducted simulations several times until all objects satisfied the condition of scattering.
- (vi)
- Direction of the robot motion: The direction of robot motion is selected considering the ability of the manipulator to generate impulse in certain direction. Kim et al. [26] proposed the normalized impact geometry to analyze the impulse generation ability of the manipulator. Figure 10 shows the generalized impulse geometry for the robot manipulator, which is constructed based on the impulse model of Equation (8). Figure 10b,c shows the maximum and minimum impulse direction for four different configurations. The direction of robot motion is decided to apply the maximum impulse. However, in some cases the largest object is surrounded by other objects in clutter and not accessible for initial collision with the robot manipulator. In this scenario, the direction of robot motion is decided to maximize the external impulse without considering the constraint (iii).
- (vii)
- The scattering index is set as the average distance among objects. It can be expressed as:$${s}_{\mathrm{min}}\le {\displaystyle \sum}_{i=1}^{M}{\displaystyle \sum}_{j=i+1}^{M}\frac{\Vert {\mathit{r}}_{i}-{\mathit{r}}_{j}\Vert}{N}<{s}_{\mathrm{max}},where\text{\hspace{0.17em}\hspace{0.17em}}N={}_{M}C_{2},$$$$\Vert {\mathit{r}}_{i}-{\mathit{r}}_{j}\Vert \ge {d}_{\mathrm{min}}\left(i\ne j\right),$$
^{th}object in plane. $N$ and $M$ are the number of objects and the number of connection of objects, respectively. Equation (35) implies the average distance among all objects and Equation (36) implies that distance between two consecutive objects should be grater or equal to ${d}_{\mathrm{min}}$. The additional constraint of Equation (36) is included to make sure that all objects are apart from each other.

## 6. Quantitative Analysis

## 7. Conclusions

## Supplementary Materials

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Caldera, S.; Rassau, A.; Chai, D. Review of deep learning methods in robotic grasp detection. Multimodal Technol. Interact.
**2018**, 2, 57. [Google Scholar] [CrossRef] - Sarantopoulos, I.; Doulgeri, Z. Human-inspired robotic grasping of flat objects. Robot. Auton. Syst.
**2018**, 108, 179–191. [Google Scholar] [CrossRef] - Argall, B.D.; Chernova, S.; Veloso, M.; Browning, B. A survey of robot learning from demonstration. Robot. Auton. Syst.
**2009**, 57, 469–483. [Google Scholar] [CrossRef] - Laskey, M.; Lee, J.; Chuck, C.; Gealy, D.; Hsieh, W.; Pokorny, F.T.; Dragan, A.D.; Goldberg, K. Robot grasping in clutter: Using a hierarchy of supervisors for learning from demonstrations. In Proceedings of the IEEE Conference on Automation Science and Engineering, Fort Worth, TX, USA, 21–25 August 2016; pp. 827–834. [Google Scholar]
- Ross, S.; Bagnell, D. Efficient reductions for imitation learning. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, Sardinia, Italy, 13–15 May 2010; pp. 661–668. [Google Scholar]
- Kloss, A.; Schaal, S.; Bohg, J. Combining learned and analytical models for predicting action effects. arXiv
**2017**, arXiv:1710.04102. [Google Scholar] - Eitel, A.; Hauff, N.; Burgard, W. Learning to singulate objects using a push proposal network. arXiv
**2017**, arXiv:1707.08101. [Google Scholar] - Hermans, T.; Rehg, J.; Bobick, A. Guided pushing for object singulation. In Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and System, Vilamoura, Portugal, 7–12 October 2012; pp. 4783–4790. [Google Scholar]
- Chang, L.; Smith, J.R.; Fox, D. Interactive singulation of objects from a pile. In Proceedings of the 2012 IEEE International Conference on Robotics and Automation, Saint Paul, MN, USA, 14–18 May 2012; pp. 3875–3882. [Google Scholar]
- Dogar, M.; Hsiao, K.; Ciocarlie, M.; Srinivasa, S. Physics-Based grasp planning through clutter. Robot. Sci. Syst.
**2012**, 2012, 78–85. [Google Scholar] - Dogar, M.; Srinivasa, S. Push-grasping with dexterous hands: Mechanics and a method. In Proceedings of the IEEE/IRSJ International Conference on Intelligent Robots and Systems, Taipei, Taiwan, 18–22 October 2010; pp. 2123–2130. [Google Scholar]
- Zhou, J.; Paolini, R.; Bagnell, J.A.; Mason, M.T. A convex polynomial force-motion model for planar sliding: Identification and application. In Proceedings of the IEEE International Conference on Robotics and Automation, Stockholm, Sweden, 16–21 May 2016; pp. 372–377. [Google Scholar]
- Yu, K.T.; Bauza, M.; Fazeli, N.; Rodriguez, A. More than a million ways to be pushed a high-fidelity experimental dataset of planar pushing. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Daejeon, South Korea, 9–14 October 2016; pp. 30–37. [Google Scholar]
- Bicchi, A.; Kumar, V. Robotic grasping and contact: A review. In Proceedings of the IEEE International Conference on Robotics and Automation, San Francisco, CA, USA, 24–28 April 2000; pp. 348–353. [Google Scholar]
- Wu, J.; Lim, J.; Zhang, H.; Tenenbaum, J.; Freeman, W.; Wilson, R.C.; Hancock, E.R.; Smith, W.A.P.; Pears, N.E.; Bors, A.G. Physics 101: Learning physical object properties from unlabeled videos. Br. Mach. Vis. Conf.
**2016**, 2016, 2. [Google Scholar] - Wu, J.; Yildirim, I.; Lim, J.J.; Freeman, B.; Tenenbaum, J.; Cortes, C.; Lawrence, N.D.; Lee, D.D.; Sugiyama, M.; Garnett, R. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. In Proceedings of the Advances in Neural Information Processing Systems 28 (NIPS 2015), Montreal, Canada, 7–12 December 2015; pp. 127–135. [Google Scholar]
- Walker, I.D. Impact configurations and measures for kinematically redundant and multiple armed robot systems. IEEE Trans. Robot. Autom.
**1994**, 10, 670–683. [Google Scholar] [CrossRef] - Imran, A.; Yi, B.-J. Impulse modeling and new impulse measure for human-like closed-chain manipulator. IEEE Robot. Autom. Lett.
**2016**, 1, 868–875. [Google Scholar] [CrossRef] - Imran, A.; Yi, B.-J. Motion optimization of human body for impulse-based applications. Intell. Serv. Robot.
**2018**, 11, 323–333. [Google Scholar] [CrossRef] - Barghijand, H.; Akbarimajd, A.; Keighobadi, J. Quasi-Static object manipulation by mobile robot: Optimal motion planning using GA. In Proceedings of the International Conference on Intelligent Systems Design and Applications, Cordoba, Spain, 22–24 November 2011; pp. 202–207. [Google Scholar]
- Lynch, K.M. Locally controllable polygons by stable pushing. In Proceedings of the IEEE International Conference on Robotics and Automation, Albuquerque, NM, USA, 25 April 1997; pp. 1442–1447. [Google Scholar]
- Goyal, S.; Ruina, A.; Papadopoulos, J. Planar sliding with dry friction. Parts 1: Limit surface and moment function. Wear
**1991**, 143, 307–330. [Google Scholar] [CrossRef] - Wittenburg, J.; Likins, P. Dynamics of Systems of Rigid Bodies; B. G. Teubner: Stuttgart, Germany, 1978. [Google Scholar]
- Imran, A.; Yi, B.-J. A closed-form analytical modeling of internal impulses with application to dynamic machining task: Biologically inspired dual-arm robotic approach. IEEE Robot. Autom. Lett.
**2018**, 3, 442–449. [Google Scholar] [CrossRef] - Choi, J.Y.; Yi, B.-J. Dynamics and impact control of a flying soccer ball. J. Korean Phys. Soc.
**2009**, 54, 75–84. [Google Scholar] [CrossRef] - Kim, J.; Chung, W.K.; Youm, Y. Normalized impact geometry and performance index for redundant manipulators. In Proceedings of the IEEE International Conference on Robotics and Automation, San Francisco, CA, USA, 24–28 April 2000; pp. 1714–1719. [Google Scholar]
- Chen, L.C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A.L. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell.
**2017**, 40, 834–848. [Google Scholar] [CrossRef] [PubMed] - Badrinarayanan, V.; Kendall, A.; Cipolla, R. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell.
**2017**, 39, 2481–2495. [Google Scholar] [CrossRef] [PubMed] - He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2961–2969. [Google Scholar]
- Detectron. Available online: https://github.com/facebookresearch/Detectron (accessed on 1 December 2018).
- Virtual Motion. Available online: http://www.virtualmotion.co.kr/ index.do (accessed on 31 July 2019).

**Figure 1.**Potential applications of singulation: (

**a**) Living room; (

**b**) LEGO parts-based assembly by a robot manipulator.

**Figure 4.**(

**a**) Collision model of two translating and rotating bodies. (

**b**) Object and manipulator collision model by considering friction. (

**c**) Collision model among objects by considering friction.

**Figure 6.**Setup, objects, and observed parameters from collision of two objects. (

**a**) Setup and object involved in scattering application. (

**b**) Observed displacements. (

**c**) Observed velocities. (

**d**) Observed accelerations.

**Figure 7.**Physical properties. Average value and standard deviation of coefficient of friction of four objects shown in Figure 5a.

**Figure 8.**Coefficient of restitution: Average value and standard deviation. M stands for manipulator.

**Figure 10.**Impulse geometry: (

**a**) normalized impulse ellipsoid for the Indy 7 robotics arm; (

**b**) impulse ellipsoid for the four configurations; (

**c**) maximum and minimum impulse direction.

**Figure 12.**Scattering experiments for singulation of two objects: (

**a**) before collision; (

**b**) after collision. Manipulator velocity: 0.4 m/s.

**Figure 13.**Scattering experiments with three objects by considering circular shapes only: (

**a**) before collision; (

**b**) after collision. Manipulator velocity: 0.5 m/s.

**Figure 14.**Scattering experiments for singulation of three objects by considering general shapes: (

**a**) before collision; (

**b**) after collision. Manipulator velocity: 0.4 m/s.

**Figure 15.**Scattering experiments for singulation of four objects by considering circular shapes only: (

**a**) before collision; (

**b**) after collision. Manipulator velocity: 0.6 m/s.

**Figure 16.**Scattering experiments for singulation of four objects by considering general shapes: (

**a**) before collision; (

**b**) after collision. Manipulator velocity: 0.5 m/s.

**Figure 17.**The quantitative analysis between real world and virtual world. A comparison between real/virtual world distances among all objects. (

**a**) Two-object singulation; (

**b**) three-object singulation by considering the circular shape only; (

**c**) three-object singulation by considering the general shape of objects; (

**d**) four-object singulation considering the circular shape only; (

**e**) four-object singulation considering the general shape of objects.

Objects | ma/ma | ma/mb | ma/mc | ma/md | mb/mb | mb/md | mc/mb | mc/md | ma/me |
---|---|---|---|---|---|---|---|---|---|

Error % | 3.47 | 6.15 | 4.8 | 6.9 | 1.7 | 10 | 2.5 | 6.5 | 3.13 |

Objects | me/mb | mc/me | me/md | ma/mf | mf/mb | mc/mf | mf/md | me/mf | |

Error % | 7.7 | 2.91 | 0.79 | 8.75 | 5.98 | 1.05 | 3.14 | 6.17 |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Imran, A.; Kim, S.-H.; Park, Y.-B.; Suh, I.H.; Yi, B.-J.
Singulation of Objects in Cluttered Environment Using Dynamic Estimation of Physical Properties. *Appl. Sci.* **2019**, *9*, 3536.
https://doi.org/10.3390/app9173536

**AMA Style**

Imran A, Kim S-H, Park Y-B, Suh IH, Yi B-J.
Singulation of Objects in Cluttered Environment Using Dynamic Estimation of Physical Properties. *Applied Sciences*. 2019; 9(17):3536.
https://doi.org/10.3390/app9173536

**Chicago/Turabian Style**

Imran, Abid, Sang-Hwa Kim, Young-Bin Park, Il Hong Suh, and Byung-Ju Yi.
2019. "Singulation of Objects in Cluttered Environment Using Dynamic Estimation of Physical Properties" *Applied Sciences* 9, no. 17: 3536.
https://doi.org/10.3390/app9173536