# Development and Performance Evaluation of Image-Based Robotic Waxing System for Detailing Automobiles

^{*}

## Abstract

**:**

## 1. Introduction

## 2. System Description

## 3. Stereo Vision and Path Planning

#### 3.1. Image Processing

#### 3.2. Stereo Vision

_{w}, y

_{w}, z

_{w}), the camera coordinate system (x

_{c}, y

_{c}, z

_{c}), and the pixel coordinate system (u

_{p}, v

_{p}). The conversion relationship is as shown in Figure 3. The world coordinate system refers to a specific origin in the workspace, so any object in the space has coordinates in relation to the defined origin so that the world coordinate system and robotic arm coordinate system can be defined with an identical origin and directions. The camera coordinate system is a coordinate system that describes other objects based on the location of the camera. Generally speaking, there is no deflection in the light that travels through the center of the lenses. For this reason, we set the center of the lenses as the origin of the camera coordinate system. The origin of the pixel coordinate system is the upper left corner of the image plane and describes the location of the target on the image plane. The system must convert the location of the target into world coordinates so that the robotic arm can track the planned path. The coordinate conversion is achieved by first converting the world coordinates into camera coordinates and then converting them into pixel coordinates.

_{p}, v

_{p}) are the target coordinates as described by the pixel coordinate system, and (x

_{w}, y

_{w}, z

_{w}) are the coordinates of the target in the world coordinate system; f, u

_{i}, and v

_{i}are intrinsic parameters of the camera, which are fixed upon manufacturing; f denotes the focus of the camera; (u

_{i}, v

_{i}) are the coordinates of the camera center. Note that these intrinsic parameter values can be obtained via a standard camera calibration process [31].

**R**

_{3×3}and

**T**

_{3×1}are the extrinsic parameters of the camera, including a translation matrix and a rotation matrix. However, extrinsic parameters vary with world coordinate definitions and the position in which the camera is installed. We installed the camera on the second axis of the robotic arm in this study, so the conversion relationships between coordinate systems will change as the robotic arm moves. Thus, a camera coordinate system had to be defined to calculate the extrinsic parameters.

**E**is the extrinsic parameter of the camera;

**P**and

_{r}**P**are the locations of the target as described by the coordinate systems of the robotic arm and the camera, respectively. With this coordination matrix, we can convert the target location estimated by the camera coordinate system into coordinates in the robotic arm coordinate system, perform 3D reconstruction using the depth and color planes obtained using the depth camera, and construct a point cloud plot of the target sheet metal to plan the waxing path:

_{c}#### 3.3. Planning of Waxing Path

## 4. Force Control of Robotic Arm

**F**

_{d}) and an impedance control algorithm. Furthermore, if the sponge at the end-effector of the robotic arm is not completely flat on the surface of the sheet metal, it will affect the evenness of force application, increase the chance of structural damage in the robotic arm, and create an additional moment M

_{x}in the robotic arm, as shown in Figure 7. In view of this, we also considered orientation impedance control to keep the sponge flat on the surface based on the external moment. Figure 8 displays a block diagram of the overall force control system. Considering the position terms p

_{y}and p

_{z}of the end-effector in the y- and z-directions and rotation angle θ

_{x}around the x-axis,

**x**= [p

_{y}p

_{z}θ

_{x}]

^{T}(position of end-effector); $\dot{\mathbf{x}}$ and $\ddot{\mathbf{x}}$ denote the velocity and acceleration of the end-effector,

**x**

_{d}= [p

_{yd}p

_{zd}θ

_{xd}]

^{T}is the desired position of the end-effector. The mass and moment of inertia matrix of the end-effector is

**M**= diag(M

_{y}, M

_{z}, J

_{θ}), and the matrices of damping and stiffness between the robotic arm and the environment are

**B**= diag(B

_{y}, B

_{z}, B

_{θ}) and

**K**= diag(K

_{y}, K

_{z}, K

_{θ}), respectively. The force that comes into contact with the external environment is the external force/torque

**F**

_{ext}= [F

_{y}F

_{z}M

_{x}]

^{T}. As the force sensor is installed on the end-effector, conversion relationships F

_{y}= F

_{s}sinθ and F

_{z}= −F

_{s}cosθ exist between the measured waxing force F

_{s}and the coordinate system of the point cloud. Note that θ can be derived using Equation (3). Let the waxing force being applied to the target be F

_{w}; its components F

_{yd}= F

_{w}sinθ, F

_{zd}= −F

_{w}cosθ, and M

_{xd}= 0 can be expressed using expected force vector

**F**

_{d}= [F

_{yd}F

_{zd}M

_{xd}]

^{T}.

**F**

_{ext}and

**F**

_{d}to change the position responses of the overall system. When the difference is zero, it means that the force applied by the robotic arm to the sheet metal has reached the system's target value. Furthermore, impedance control is used to alleviate the impact of the force created by the end-effector when it comes into contact with the sheet metal.

## 5. Image Based Performance Evaluation

#### 5.1. Selection of External Light Source

#### 5.2. Waxing Assessment Criteria

#### 5.2.1. Image Preprocessing

#### 5.2.2. Establishment of Image Assessment Criteria

## 6. Experiments and Performance Analysis

#### 6.1. Analysis of Force Tracking Performance

**M**,

**B**and

**K**) in Figure 8 all influence the force tracking error and response speed. By adjusting the impedance parameters, we change the response behavior of the system to achieve force tracking performance with stability, a short settling time, and no overshoot. If the damping ratio (ζ) is set as 1 and the settling time as less than 0.5 s, then, with the transient responses of a standard second-order dynamic system [35], we can derive that the natural frequency (ω

_{n}) of the system must be greater than 8 rad/s to fulfill this constraint. In this experiment, we set the natural frequency as 10 rad/s; in terms of F

_{y}, if the mass coefficient selected for the impedance controller is M = 3.5 kg, then the damping and stiffness coefficients are B = 70 N s/m and K = 350 N/m, respectively. We also chose the same set of parameters to achieve the impedance control of components F

_{y}and M

_{x}and used a single target point to perform waxing force tracking of the robotic arm and analyze its response results.

_{s}, the force that is vertical to the sheet metal, reaches the target value F

_{w}set by the system. At the same time, we must suppress the torque M

_{x}created when the end-effector is not vertical to the sheet metal. Figure 14 presents the response results of force tracking when F

_{w}is set as 15 N. As can be seen, the responses resulting from this set of parameters can quickly reach the target value, the settling time is significantly shorter than 0.5 s, no overshoot exists, and the absolute value of the steady state error is less than 0.3 N. The robotic arm showed some posture deviations during the time from 0.1 s to 0.4 s, which prevented the end-effector from being completely flat on the sheet metal surface and created a torque of approximately 3 N mm. However, posture compensation eliminated this after 0.4 s, so it did not affect the results of the waxing experiment. When F

_{w}is set between 10 N and 20 N, the force tracking results of other fixed values present an identical trend.

#### 6.2. Examination of Waxing Parameters

#### 6.2.1. Experiment Conditions and Parameter Settings

**x**. Via inverse kinematics, we obtained the motor position command and then realized point-to-point motion control in the motor with the proportional-derivative (PD) control algorithm. Waxing of each fixed point was then performed from the bottom upwards according to each of the planned paths. The movements between paths were achieved using the xyz platform, coordinated with impedance control to prevent the robotic arm from applying excessive force when it comes into contact with the sheet metal.

_{d}#### 6.2.2. Discussion of Waxing Results

#### 6.3. Waxing Parameter Optimization

#### 6.3.1. Problem Formulation

^{2}= 0.996) indicates that the fitted surface is very close to the distribution of the experimental results, so the quantic equation with two variables served as the objective function of optimization in the proposed system. The various coefficients are as shown in Table 6:

#### 6.3.2. Optimization Results and Verification

## 7. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Kumar, S. Theories of musculoskeletal injury causation. Ergonomics
**2001**, 44, 17–47. [Google Scholar] [CrossRef] [PubMed] - Wagner, H.R. Apparatus for Washing and Waxing Cars. U.S. Patent 3447505, 3 June 1969. [Google Scholar]
- Falls, J.W. Automatic Car Waxer. U.S. Patent 5076202, 31 December 1991. [Google Scholar]
- Hung, V. Car Waxing Machine with Driving Handle. U.S. Patent 6725491, 27 April 2004. [Google Scholar]
- Wu, X.; Wu, X.; Guo, H. Combination Automobile Detailing Machine. U.S. Patent 11518864, 11 September 2006. [Google Scholar]
- Ning, F.; Cong, W.; Wang, H.; Hu, Y.; Hu, Z.; Pei, Z. Surface grinding of CFRP composites with rotary ultrasonic machining: A mechanistic model on cutting force in the feed direction. Int. J. Adv. Manuf. Technol.
**2017**, 92, 1217–1229. [Google Scholar] [CrossRef] - Sallinen, M.; Heikkila, T.; Sirvio, M. Robotic Deburring System of Foundry Castings Based on Flexible Workobject Localization; SPIE: Boston, MA, USA, 2001; pp. 476–487. [Google Scholar]
- Hsu, F.Y.; Fu, L.C. Intelligent robot deburring using adaptive fuzzy hybrid position/force control. IEEE Trans. Robot. Autom.
**2000**, 16, 325–335. [Google Scholar] [CrossRef] - Mohsin, I.; He, K.; Cai, J.; Chen, H.; Du, R. Robotic polishing with force controlled end effector and multi-step path planning. In Proceedings of the IEEE International Conference on Information and Automation, Macau, China, 18–20 July 2017; pp. 344–348. [Google Scholar]
- Chen, F.; Hao, S.; Miao, X.; Yin, S.; Huang, S. Numerical and experimental study on low-pressure abrasive flow polishing of rectangular microgroove. Powder Technol.
**2018**, 327, 215–222. [Google Scholar] [CrossRef] - Han, G.; Sun, M. Compound control of the robotic polishing process basing on the assistant electromagnetic field. In Proceedings of the IEEE International Conference on Mechatronics and Automation, Changchun, China, 9–12 August 2009; pp. 1551–1555. [Google Scholar]
- Sharma, K.; Shirwalkar, V.; Das, A.P.; Pal, P.K. Robotic polishing of pilger-die. In Proceedings of the Conference on Advances in Robotics, Goa, India, 2–4 July 2015. [Google Scholar]
- Jamisola, R.; Ang, M.H.; Oetomo, D.; Khatib, O.; Ming, T.; Lim, S.Y. The operational space formulation implementation to aircraft canopy polishing using a mobile manipulator. In Proceedings of the IEEE International Conference on Robotics and Automation, Washington, DC, USA, 11–15 May 2002; pp. 400–405. [Google Scholar]
- Zhang, D.; Yun, C.; Song, D.; Lan, X.; Wu, X. Design and application for robotic polishing system. In Proceedings of the IEEE International Conference on Mechanic Automation and Control Engineering, Wuhan, China, 26–28 June 2010; pp. 2470–2474. [Google Scholar]
- Guvenc, L.; Srinivasan, K. Force controller design and evaluation for robot-assisted die and mould polishing. Mech. Syst. Signal. Process.
**1995**, 9, 31–49. [Google Scholar] [CrossRef] - Lu, Y.Y.; Dong, J.H. The study of force control with artificial intelligence in ceramic grinding process. In Proceedings of the IEEE International Symposium on Intelligence Information Processing and Trusted Computing, Huanggang, China, 28–29 October 2010; pp. 208–211. [Google Scholar]
- Liu, H.; Wan, Y.; Zeng, Z.; Xu, L.; Zhao, H.; Fang, K. Freeform surface grinding and polishing by CCOS based on industrial robot. In Proceedings of the International Symposium on Advanced Optical Manufacturing and Testing Technologies, Suzhou, China, 26–29 April 2016. [Google Scholar]
- Zhao, T.; Shi, Y.; Lin, X.; Duan, J.; Sun, P.; Zhang, J. Surface roughness prediction and parameters optimization in grinding and polishing process for IBR of aero-engine. Int. J. Adv. Manuf. Technol.
**2014**, 74, 653–663. [Google Scholar] [CrossRef] - Akafuah, N.K.; Poozesh, S.; Salaimeh, A.; Patrick, G.; Lawler, K.; Saito, K. Evolution of the automotive body coating process—A review. Coatings
**2016**, 6, 24. [Google Scholar] [CrossRef] - Placido, F.; Birney, R.; Kavanagh, J. Investigation of automotive detailing products by ellipsometry and contact angle analysis. Acta Phys. Pol. A
**2009**, 116, 712–714. [Google Scholar] [CrossRef] - Gomez, O.; Perales, E.; Chorro, E.; Burgos, F.J.; Viqueira, V.; Vilaseca, M.; Martinez-Verdu, F.M.; Pujol, J. Visual and instrumental assessments of color differences in automotive coatings. Color Res. Appl.
**2016**, 41, 384–391. [Google Scholar] [CrossRef] [Green Version] - Sharifzadeh, M.; Alirezaee, S.; Amirfattahi, R.; Sadri, S. Detection of steel defect using the image processing algorithms. In Proceedings of the IEEE International Multitopic Conference, Karachi, Pakistan, 23–24 December 2008; pp. 125–127. [Google Scholar]
- Besari, A.R.A.; Zamri, R.; Rahman, K.A.A.; Palil, M.D.M.; Prabuwono, A.S. Surface defect characterization in polishing process using contour dispersion. In Proceedings of the IEEE International Conference on Soft Computing and Pattern Recognition, Malacca, Malaysia, 4–7 December 2009; pp. 707–710. [Google Scholar]
- Samtaş, G. Measurement and evaluation of surface roughness based on optic system using image processing and artificial neural network. Int. J. Adv. Manuf. Technol.
**2014**, 73, 353–364. [Google Scholar] [CrossRef] - Khoo, S.W.; Karuppanan, S.; Tan, C.S. A review of surface deformation and strain measurement using two-dimensional digital image correlation. Metrol. Meas. Syst.
**2016**, 23, 461–480. [Google Scholar] [CrossRef] - Simunovic, G.; Svalina, I.; Simunovic, K.; Saric, T.; Havrlisan, S.; Vukelic, D. Surface roughness assessing based on digital image features. Adv. Prod. Eng. Manag.
**2016**, 11, 93–140. [Google Scholar] [CrossRef] - Liu, E.; Liu, J.; Gao, R.; Yi, H.; Wang, W.; Suo, X. Designing indices to measure surface roughness based on the color distribution statistical matrix (CDSM). Tribol. Int.
**2018**, 122, 96–107. [Google Scholar] [CrossRef] - Besari, A.R.A.; Prabuwono, A.S.; Zamri, R.; Palil, M.D.M. Computer vision approach for robotic polishing application using artificial neural networks. In Proceedings of the IEEE Student Conference on Research and Development, Kuala Lumpur, Malaysia, 13–14 December 2010; pp. 281–286. [Google Scholar]
- Luh, J.Y.S.; Lin, C.S. Optimum path planning for mechanical manipulators. J. Dyn. Syst. Meas. Control
**1981**, 103, 142–151. [Google Scholar] [CrossRef] - Tsai, L.W. Robot Analysis: The Mechanics of Serial and Parallel Manipulators; Wiley: New York, NY, USA, 1999. [Google Scholar]
- Zhang, Z. A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell.
**2000**, 22, 1330–1334. [Google Scholar] [CrossRef] - Jung, S.; Hsia, T.C.; Bonitz, R.G. Force tracking impedance control of robot manipulators under unknown environment. IEEE Trans. Control Syst. Technol.
**2004**, 12, 474–483. [Google Scholar] [CrossRef] - Canny, J. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell.
**1986**, 8, 679–698. [Google Scholar] [CrossRef] [PubMed] - Ballard, D.H. Generalizing the Hough transform to detect arbitrary shapes. Pattern Recognit.
**1981**, 13, 111–122. [Google Scholar] [CrossRef] - Norman, S.N. Control Systems Engineering; Wiley: New York, NY, USA, 2011. [Google Scholar]
- Luenberger, D.G.; Te, Y. Linear and Nonlinear Programming; Springer: New York, NY, USA, 2008. [Google Scholar]

**Figure 2.**Mask operation. (

**a**) color image; (

**b**) depth image; (

**c**) mask computing; (

**d**) mask to depth image.

**Figure 4.**Camera coordinate system (Frame 0–1: motor rotation axis, Frame 2: position of camera, Frame 3: center of camera).

**Figure 6.**Posture constraint for the robot end-effector. (

**a**) point cloud 3D plot; (

**b**) slope estimation.

**Figure 7.**Use of robotic arm in waxing force control. (

**a**) normal waxing force control; (

**b**) end-effector not flat on sheet metal.

**Figure 10.**Results of image preprocessing (area waxed by auto detailer). (

**a**) captured RGB image; (

**b**) RGB to grey image; (

**c**) after edge detection.

**Figure 13.**Image analysis results from different camera angles. (

**a**) circle detection results from camera angle 15°; (

**b**) circle screening results from camera angle 15° (130 samples); (

**c**) circle detection results from camera angle 30°; (

**d**) circle screening results from camera angle 30° (135 samples); (

**e**) circle detection results from camera angle 45°; (

**f**) circle screening results from camera angle 45° (137 samples).

**Figure 14.**Force tracking control results of the robotic waxing system (F

_{w}= 15,

**M**

_{3×3}= diag(3.5, 3.5, 3.5),

**B**

_{3×3}= diag(70, 70, 70),

**K**

_{3×3}= diag(350, 350, 350)).

**Figure 17.**Waxing image analysis following system parameter optimization (waxing force 15 N, dwell time 0.5 s/pt). (

**a**) waxing results comparison (proposed system vs. car detailer); (

**b**) captured image with the proposed system; (

**c**) Hough circle transform results (130 circles).

# of Path | Path Function (after Cubic Curve Fitting) |
---|---|

Path 1 | ${z}_{1}(y)=-2.86\times {10}^{-6}{y}^{3}+0.0055{y}^{2}-2.135y-126.0215$ |

Path 2 | ${z}_{2}(y)=-1.02\times {10}^{-5}{y}^{3}+0.0223{y}^{2}-15.037y-3154.08$ |

Path 3 | ${z}_{3}(y)=-5.49\times {10}^{-6}{y}^{3}+0.0117{y}^{2}-6.9988y-1134.38$ |

Path 4 | ${z}_{4}(y)=-7.69\times {10}^{-6}{y}^{3}+0.0167{y}^{2}-10.716y-2051.36$ |

Path 5 | ${z}_{5}(y)=-7.29\times {10}^{-6}{y}^{3}+0.0156{y}^{2}-9.7094y-1746.57$ |

Path 6 | ${z}_{6}(y)=-1.17\times {10}^{-5}{y}^{3}+0.0256{y}^{2}-17.241y-3607.75$ |

Samples in Different Range ($\overline{\mathit{x}}=4.40,\mathit{\sigma}=0.20$) | |||
---|---|---|---|

Camera View (θ) | Total Samples | $\overline{\mathit{x}}\pm \mathit{\sigma}$ | $\overline{\mathit{x}}\pm 2\mathit{\sigma}$ |

15° | 144 | 96 samples | 130 samples |

30° | 144 | 98 samples | 135 samples |

45° | 144 | 97 samples | 137 samples |

Estimation of Radius | Criterion Definition |
---|---|

Mean value ($\overline{x}$) | 4.4 pixel |

Standard deviation (σ) | 0.2 pixel |

Range of circle radius | $\overline{x}\pm 2\sigma $ |

Average result from Table 2 | 134 samples |

Threshold | 10 samples |

Waxing Motor Specifications | Waxing Force (From 10 to 19 N) | Dwell Time | Executing Time for all Paths (Approx.) |
---|---|---|---|

24 V—850 rpm (30 N-m) | 10 N | 0.1 s/pt 0.2 s/pt 0.3 s/pt 0.4 s/pt | 5~6 min 10~11 min 15~16 min 20~21 min |

13 N | |||

15 N | |||

17 N | |||

19 N |

Waxing Force | Dwell Time | Executing Time | Estimated Result (Total: 144 Samples) |
---|---|---|---|

10 N | 0.1 s/pt | 5 min 28 s | 70 samples |

0.2 s/pt | 10 min 35 s | 82 samples | |

0.3 s/pt | 15 min 24 s | 89 samples | |

0.4 s/pt | 20 min 19 s | 93 samples | |

13 N | 0.1 s/pt | 5 min 36 s | 92 samples |

0.2 s/pt | 10 min 21 s | 102 samples | |

0.3 s/pt | 15 min 18 s | 110 samples | |

0.4 s/pt | 20 min 30 s | 115 samples | |

15 N | 0.1 s/pt | 5 min 23 s | 110 samples |

0.2 s/pt | 10 min 29 s | 116 samples | |

0.3 s/pt | 15 min 34 s | 121 samples | |

0.4 s/pt | 20 min 36 s | 127 samples | |

17 N | 0.1 s/pt | 5 min 31 s | 112 samples |

0.2 s/pt | 10 min 33 s | 117 samples | |

0.3 s/pt | 15 min 29 s | 115 samples | |

0.4 s/pt | 20 min 24 s | 110 samples | |

19 N | 0.1 s/pt | 5 min 36 s | 115 samples |

0.2 s/pt | 10 min 21 s | 117 samples | |

0.3 s/pt | 15 min 18 s | 115 samples | |

0.4 s/pt | 20 min 30 s | 113 samples |

p_{00} | −7917 | p_{21} | −76.58 | p_{04} | 18140 |

p_{10} | 3081 | p_{12} | −1923 | p_{50} | 0.01935 |

p_{01} | −3663 | p_{03} | −8127 | p_{41} | −0.008048 |

p_{20} | −471.6 | p_{40} | −1.324 | p_{32} | −4.448 |

p_{11} | 915.7 | p_{31} | 2.224 | p_{23} | 12.24 |

p_{02} | 9295 | p_{22} | 159.7 | p_{14} | −255.9 |

p_{30} | 35.63 | p_{13} | −83.42 | p_{05} | −11600 |

Optimization Parameters of the Waxing System | Lower and Upper Bounds |
---|---|

Force (N) | From 10 N to 17 N |

Dwell time (s/pt) | From 0.1 s/pt to 0.5 s/pt |

Executing Time | Estimated Result (Total: 144 Samples) | |
---|---|---|

Car detailer | About 15 min | 134 samples |

The waxing system | 25 min 16 s | 130 samples |

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Lin, C.-Y.; Hsu, B.-C.
Development and Performance Evaluation of Image-Based Robotic Waxing System for Detailing Automobiles. *Sensors* **2018**, *18*, 1548.
https://doi.org/10.3390/s18051548

**AMA Style**

Lin C-Y, Hsu B-C.
Development and Performance Evaluation of Image-Based Robotic Waxing System for Detailing Automobiles. *Sensors*. 2018; 18(5):1548.
https://doi.org/10.3390/s18051548

**Chicago/Turabian Style**

Lin, Chi-Ying, and Bing-Cheng Hsu.
2018. "Development and Performance Evaluation of Image-Based Robotic Waxing System for Detailing Automobiles" *Sensors* 18, no. 5: 1548.
https://doi.org/10.3390/s18051548