Optimal Realtime Toolpath Planning for Industrial Robots with Sparse Sensing

Abstract

Non-contact surface processing does not involve direct contact between the tool and a worksurface. An industrial robot mostly uses preplanned toolpaths to perform non-contact surface processing. A preplanned toolpath may work well in repetitive conditions but may easily become inaccurate and unsafe if the tool needs to follow unknown worksurface variations. Many industrial processes, e.g., painting, coating, and sandblasting, typically involve worksurfaces with unknown variations. This study proposes an optimal toolpath planning method for an industrial robot equipped with end-of-arm distance sensors to automatically guide its tool motion along unknown worksurface variations. The distance sensors facilitate sparse sensing to acquire sparse data that is just enough for the quick and adequate perception of unknown worksurfaces by requiring fewer measurements and less computing. Optimization facilitates the optimality of multi-objective toolpath planning with a customizable value function, where the multiple objectives comprise adapting to unknown worksurface variations and traveling between known tool targets. To validate the proposed toolpath planning method, this study conducts a simulation experiment on a virtual robot with four end-of-arm distance sensors and a workpiece with unknown surface variations. The experimental results indicate that the proposed method is accurate and near-optimal even in the presence of sensor noises.

1. Introduction

Industrial robots are emerging as major autonomous systems for handling various manufacturing tasks that are highly repetitive, difficult, and unsafe to humans [1,2,3]. The International Organization for Standardization (ISO) [4,5] identifies an industrial robot as an automatically controlled, reprogrammable, multipurpose, manipulative system with several reprogrammable axes, which may be either fixed in place or mobile for use in industrial automation applications. An industrial robot mostly uses preplanned toolpaths to perform non-contact surface processing. A preplanned toolpath may work well in repetitive conditions but may easily become inaccurate and unsafe if the tool needs to follow unknown worksurface variations. Many industrial processes, e.g., painting, coating, and sandblasting, typically involve worksurfaces with unknown variations.

Researchers have proposed theoretical methods for robotic toolpath planning [6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22] as well as their practical applications [6,7,8,9,13,14,16,18,23]. Navigation and the precision of toolpaths are practically important as an industrial robot may exhibit positioning errors [24,25] due to uncertain variations involved in the environment. Non-rigid robots with hyper-elastic, soft bodies experience even bigger challenges in terms of navigation, motion precision, stiffness, and compliance [26,27,28,29]. All these challenges in robotics indicate the importance of sensing, which can provide the needed perception of priorly unknown variations in environments, for better navigation and control of robots and autonomous systems.

For robotic non-contact surface processing, contactless sensing is always preferred as it allows a robot to sense surfaces from a distance. There are various contactless sensors (or non-contact sensors), such as vision cameras [30,31,32,33,34,35,36,37,38,39], laser trackers [24,40], Light Detection and Ranging (LiDAR) scanners [41,42], and so on, used in industrial applications. Vision cameras provide ample perception but require fast communication interfaces and extensive data processing. Laser sensors like range sensors, laser trackers, and LiDAR scanners can provide accurate spatial sensing but produce harmful laser beams. Ultrasonic distance sensors are simpler than machine vision cameras and safer than laser beams. They are not affected by the color or transparency of objects but have limitations in terms of acoustic disturbances and short detection ranges.

Researchers have documented various methods for the sensor-based routing and control of industrial robots [9,13,38,39,43,44], mobile robots [30,42,45,46], and aerial robots or drones [34,47,48]. However, these research outcomes mostly focus on solutions with resource-intensive sensing rather than generic solutions with less resource-intensive sensing for baseline perception. An example of resource-intensive sensing is machine vision, which can easily produce abundant data that is costly to compute in real time. In contrast, sparse sensing with simple distance sensors produces sparse data that is just enough for the quick, adequate perception of environments with less computing. In general, sparse sensing [49] is a sensing technique that leverages fewer measurements that are just enough to reconstruct full-field perception. It requires fewer hardware and software resources to provide quick, baseline perception.

There is a noticeable research gap in realtime toolpath planning with sparse sensing in industrial robotics. To bridge this gap, this study proposes an optimal path planning method with sparse sensing for an industrial robot equipped with end-of-arm distance sensors to automatically guide its tool motion along unknown worksurface variations. The end-of-arm distance sensors facilitate sparse sensing for quick, baseline perception. Optimization facilitates the optimality of multi-objective path planning with a customizable value function, where the multiple objectives of path planning comprise adapting to unknown worksurface variations and traveling between known toolpath targets.

To validate the proposed toolpath planning method with sparse sensing, this study conducted a simulation experiment on a virtual robot equipped with end-of-arm distance sensors and a workpiece with unknown surface variations. The simulation experiment systematically demonstrated the proposed method with and without sensor noises.

2. Materials and Methods

2.1. Problem Formulation

An end-effector is a tool that goes on the end of a robot arm. A tool pose defines the tool position and orientation in the reference frame. A robot manipulates its tool pose within the operational or task space of the robot. A robot tool must have a tool reference frame or simply a tool frame, and the tool frame origin is the tool center point or tool control point (TCP), which is the location where the robotic process takes place. A target is a tool pose, and a toolpath or path is a continuous stream of tool poses.

This problem formulation involves an industrial robot with an end-of-arm tool that has four distance sensors and a tool reference frame for non-contact surface processing as illustrated in Figure 1. Adaptive toolpath planning needs to guide the robot TCP along unknown worksurface variations by maintaining relative tool orientation and alignment, while the tool travels between known start and end tool poses. The end tool pose or target also serves as the frame of reference. A toolpath must adapt to unknown worksurface variations along the XZ plane of reference frame, which means an adapted toolpath must be constrained to the XZ plane of reference frame. The relative tool orientation depends on a manufacturing process, i.e., a welding process requires a certain attack angle between the welding tool and the toolpath. In this problem formulation, the robot tool axis, which is the ${\mathit{z}}_{\mathit{T}}$ axis, needs to be orthogonal to the sensed worksurface for clarity and simplicity.

The four end-of-arm distance sensors have a square arrangement with a diagonal length of $l$ on the sensor plane as shown in Figure 1. This square arrangement is centered at the tool axis, which is the ${\mathit{z}}_{\mathit{T}}$ axis of the tool frame, and the tool axis is orthogonal to the sensor plane. Moreover, the two diagonals of the square arrangement are parallel to the tool ${\mathit{x}}_{\mathit{T}}$ axis and the tool ${\mathit{y}}_{\mathit{T}}$ axis, respectively. All the sensors have the same concentric inclination angle $\phi$, which is assigned to 90 degrees to make each sensor axis orthogonal to the sensor plane for simplicity. Four distance sensors sense the distances—$a_n$, $b_n$, $c_n$, and $d_n$—between the sensor plane points and the sensed worksurface points, respectively, in each timestep $n\in \mathbb{N}$. The sensor plane is the interface plane between the robot end and the robot tool in Figure 1. The four distance sensors are associated with four customizable reference distances—$a_{ref}$, $b_{ref}$, $c_{ref}$, and $d_{ref}$—that signify a desired tool pose relative to a sensed worksurface. These four reference distances define a tool reference plane. In this formulation, let the XY plane of the robot tool frame be the tool reference plane for clarity and simplicity, which implies that all the reference distances are equal to the distance between the sensor plane and the XY plane of the robot tool frame. Note that the sensor plane is parallel to the XY plane of the robot tool frame.

The origin of the robot tool frame is tool center point (TCP), where a non-contact process takes place. The robot tool needs to follow sensed worksurface variations along the XZ plane of the reference frame (${\mathit{p}}_{\mathit{B}}$) by maintaining alignment between the tool reference plane (the XY plane of the robot tool frame) and the sensed worksurface, while traveling between known start and end tool poses.

Toolpath $\mathit{P}\left({\mathit{p}}_{\mathit{A}}, {\mathit{p}}_{\mathit{B}}\right)$ in (1) is a series of adapted toolpath poses in between given or known start and end tool poses, ${\mathit{p}}_{\mathit{A}}\in {\mathbb{R}}^6$ and ${\mathit{p}}_{\mathit{B}}\in {\mathbb{R}}^6$, respectively.

$$\mathit{P}\left({\mathit{p}}_{\mathit{A}}, {\mathit{p}}_{\mathit{B}}\right)=\left\{{\mathit{p}}_{\mathit{n}}\in {\mathbb{R}}^6 | {\mathit{p}}_0={\mathit{p}}_{\mathit{A}}; {\mathit{p}}_{\mathit{N}}={\mathit{p}}_{\mathit{B}}; N\in \mathbb{N};n=0, 1,\dots , N\right\}$$

(1)

In (1), n and N are a timestep index and the finite number of timesteps, respectively. In each timestep n, realtime toolpath planning produces an adapted tool pose ${\mathit{p}}_{\mathit{n}}\in {\mathbb{R}}^6$. Each tool pose or target ${\mathit{p}}_{\mathit{n}}$ consists of the x, y, z coordinates of the TCP and the ${\theta }_x$, ${\theta }_y$, ${\theta }_z$ Euler angles with respect to the reference frame in Figure 1. In every timestep, the adaptive toolpath planner must estimate the next pose ${\widehat{\mathit{p}}}_{\mathit{n}+1}$ using the current tool pose ${\mathit{p}}_{\mathit{n}}$ and a tool pose correction ${∆\mathit{p}}_{\mathit{n}}$ as formulated in (2). The tool pose correction ${∆\mathit{p}}_{\mathit{n}}$ in (2) consists of a correction ${∆\mathit{p}}_{\mathit{n}|\mathit{s}}$ relative to the sensed worksurface and a correction ${∆\mathit{p}}_{\mathit{n}|{\mathit{B}}_{\mathit{n}}}$ relative to an incremental tool target ${\widehat{\mathit{p}}}_{{\mathit{B}}_{\mathit{n}}}$. Let $‖{∆\mathit{p}}_{\mathit{n}|{\mathit{B}}_{\mathit{n}}}‖=∆p$ be a known constant as a uniform increment in the direction of the final tool target ${\mathit{p}}_{\mathit{B}}$. Figure 2 shows the geometry of an incremental tool target ${\widehat{\mathit{p}}}_{{\mathit{B}}_{\mathit{n}}}={\mathit{p}}_{\mathit{n}}+{∆\mathit{p}}_{\mathit{n}|{\mathit{B}}_{\mathit{n}}}$. Moreover, Figure 2 illustrates the vector operations in (2) for estimating the next pose ${\widehat{\mathit{p}}}_{\mathit{n}+1}$.

$${\widehat{\mathit{p}}}_{\mathit{n}+1}={\mathit{p}}_{\mathit{n}}+{∆\mathit{p}}_{\mathit{n}}={\mathit{p}}_{\mathit{n}}+{∆\mathit{p}}_{\mathit{n}|\mathit{s}}+{∆\mathit{p}}_{\mathit{n}|{\mathit{B}}_{\mathit{n}}}$$

(2)

Finally, the problem formulation boils down to finding the tool pose correction ${∆\mathit{p}}_{\mathit{n}}$ and its components, ${∆\mathit{p}}_{\mathit{n}|\mathit{s}}$ and ${∆\mathit{p}}_{\mathit{n}|{\mathit{B}}_{\mathit{n}}}$, to estimate the next tool pose ${\widehat{\mathit{p}}}_{\mathit{n}+1}$ based on the current tool pose ${\mathit{p}}_{\mathit{n}}$ in every timestep. This study proposes a numerical solution to this problem by using realtime error tracking with linear quadratic regulation as outlined in Section 2.2.

2.2. Realtime Error Tracking with Linear Quadratic Regulation

The end-of-arm distance sensors in Figure 1 provide four distance sensor readings—$a_n$, $b_n$, $c_n$, and $d_n$—in every timestep n. The four distance sensors are associated with four customizable reference distances—$a_{ref}$, $b_{ref}$, $c_{ref}$, and $d_{ref}$—that signify a desired tool pose relative to a sensed worksurface. These four reference distances define a tool reference plane. In this study, the XY plane of the robot tool frame is the tool reference plane for clarity and simplicity, which implies that all the four reference distances are equal to the distance between the sensor plane and the XY plane of the robot tool frame. The deviations of four distance readings from their respective reference distances are $a_{n|ref}$, $b_{n|ref}$, $c_{n|ref}$, and $d_{n|ref}$ as derived in (3).

$${\left[\begin{array}{cc}\begin{array}{cc}{a}_{n|ref}& b_{n|ref}\end{array}& \begin{array}{cc}{c}_{n|ref}& d_{n|ref}\end{array}\end{array}\right]}^{\mathit{T}}={\left[\begin{array}{cc}\begin{array}{cc}{a}_n& b_n\end{array}& \begin{array}{cc}{c}_n& d_n\end{array}\end{array}\right]}^{\mathit{T}}-{\left[\begin{array}{cc}\begin{array}{cc}{a}_{ref}& b_{ref}\end{array}& \begin{array}{cc}{c}_{ref}& d_{ref}\end{array}\end{array}\right]}^{\mathit{T}}$$

(3)

Besides tracking worksurface variations, the robot tool must reach the known final tool target ${\mathit{p}}_{\mathit{B}}$ in Figure 1. The final tool target ${\mathit{p}}_{\mathit{B}}$ also serves as the frame of reference or reference frame for convenience. The TCP coordinates of the current tool pose ${\mathit{p}}_{\mathit{n}}$ are $x_n$, $y_n$, and $z_n$. Since the final tool target ${\mathit{p}}_{\mathit{B}}$ is located at a remote distance, an incremental tool target ${\widehat{\mathit{p}}}_{{\mathit{B}}_{\mathit{n}}}$ is used at a uniform incremental distance from the current pose ${\mathit{p}}_{\mathit{n}}$ in the direction of the final tool target ${\mathit{p}}_{\mathit{n}}$ in each timestep as shown in Figure 2. The TCP coordinates of the current tool pose ${\mathit{p}}_{\mathit{n}}$ that is relative to an incremental tool target ${\widehat{\mathit{p}}}_{{\mathit{B}}_{\mathit{n}}}$ are $x_{n|B_n}$, $y_{n|B_n}$, and $z_{n|B_n}$, which are incremental tool target errors, according to (4).

$${\left[\begin{array}{ccc}{x}_{n|B_n}& y_{n|B_n}& z_{n|B_n}\end{array}\right]}^{\mathit{T}}={\left[\begin{array}{ccc}{x}_n& y_n& z_n\end{array}\right]}^{\mathit{T}}-{\left[\begin{array}{ccc}{x}_{B_n}& y_{B_n}& z_{B_n}\end{array}\right]}^{\mathit{T}}$$

(4)

The sensed deviations $a_{n|ref}$, $b_{n|ref}$, $c_{n|ref}$, and $d_{n|ref}$ as well as the incremental tool target errors $x_{n|B_n}$, $y_{n|B_n}$, and $z_{n|B_n}$ constitute an error vector ${\mathit{e}}_{\mathit{n}}\in {\mathbb{R}}^7$ in (5).

$$\begin{gathered} {\mathit{e}}_{\mathit{n}}={\left[\begin{array}{ccc}\begin{array}{cc}{e}_n〈1〉& e_n〈2〉\end{array}& \begin{array}{cc}{e}_n〈3〉& e_n〈4〉\end{array}& \begin{array}{ccc}{e}_n〈5〉& e_n〈6〉& e_n〈7〉\end{array}\end{array}\right]}^{\mathit{T}}= \\ ={\left[\begin{array}{ccc}\begin{array}{cc}{a}_{n|ref}& b_{n|ref}\end{array}& \begin{array}{cc}{c}_{n|ref}& d_{n|ref}\end{array}& \begin{array}{ccc}{x}_{n|B_n}& y_{n|B_n}& z_{n|B_n}\end{array}\end{array}\right]}^{\mathit{T}} \end{gathered}$$

(5)

Figure 3 demonstrates a tool ${\theta }_y$ angle as a pitch angle between the tool XY plane and a worksurface using the sensed deviations, $a_{n|ref}$ and $c_{n|ref}$. Likewise, a tool ${\theta }_x$ angle can also be identified as a roll angle between the tool XY plane and a worksurface using the sensed deviations, $b_{n|ref}$ and $d_{n|ref}$. Adaptive toolpath planning aims to maintain orientation alignment between the tool XY plane and a sensed worksurface using these two tool angles for tool pose corrections. The sensor inclination angle $\phi$ in Figure 1 is $90°$. The sensitivity factor $\eta$ is a heuristic scalar between 0 and 1, where 0 means no sensitivity, and 1 means full sensitivity. It can be used to lessen the amplifying effects of the two sensed deviations on the pivot point, which is the TCP. It can also be used to lessen the effects of sensor noises. The $\eta$ sensitivity factor is 0.5 in this study.

The goal of adaptive toolpath planning is to minimize the error vector ${\mathit{e}}_{\mathit{n}}$ with less effort. Therefore, the proposed approach involves a first-order error tracking model presented in (6).

$${\mathit{e}}_{\mathit{n}+1}={\mathit{e}}_{\mathit{n}}+{\mathit{u}}_{\mathit{n}}$$

(6)

In (6), a control input vector ${\mathit{u}}_{\mathit{n}}\in {\mathbb{R}}^7$ aims to minimize the error vector ${\mathit{e}}_{\mathit{n}}$ in each timestep n. It is derived from the error vector ${\mathit{e}}_{\mathit{n}}$ according to (7), (8), and (9) using a control gain matrix $\mathit{K}\in {\mathbb{R}}^{7\times 7}$ that is a predefined diagonal matrix of the scalar gains $K_s$ and $K_B$ in (8) and (9). The gain $K_s$ is used to regulate the deviations of the four sensor readings from their respective reference distances, and the gain $K_B$ is used to guide the TCP motion in the direction of the known final tool target ${\mathit{p}}_{\mathit{B}}$.

$${\mathit{u}}_{\mathit{n}}=-\mathit{K}{\mathit{e}}_{\mathit{n}}$$

(7)

$$\mathit{K}=\mathit{d}\mathit{i}\mathit{a}\mathit{g}(K_s,K_s,K_s,K_s,K_B,K_B,K_B)\in {\mathbb{R}}^{7\times 7}$$

(8)

$$\begin{gathered} {\mathit{u}}_{\mathit{n}}={\left[\begin{array}{ccc}\begin{array}{cc}{u}_n〈1〉& u_n〈2〉\end{array}& \begin{array}{cc}{u}_n〈3〉& u_n〈4〉\end{array}& \begin{array}{ccc}{u}_n〈5〉& u_n〈6〉& u_n〈7〉\end{array}\end{array}\right]}^{\mathit{T}}= \\ =-{\left[\begin{array}{ccc}\begin{array}{cc}{K}_s{a}_{n|ref}& K_s{b}_{n|ref}\end{array}& \begin{array}{cc}{K}_s{c}_{n|ref}& K_s{d}_{n|ref}\end{array}& \begin{array}{ccc}{K}_B{x}_{n|B_n}& K_B{y}_{n|B_n}& K_B{z}_{n|B_n}\end{array}\end{array}\right]}^{\mathit{T}} \end{gathered}$$

(9)

The tool pose correction ${∆\mathit{p}}_{\mathit{n}}$ in (2) consists of a correction ${∆\mathit{p}}_{\mathit{n}|\mathit{s}}$ relative to the sensed surface and a correction ${∆\mathit{p}}_{\mathit{n}|{\mathit{B}}_{\mathit{n}}}$ relative to an incremental tool target ${\widehat{\mathit{p}}}_{{\mathit{B}}_{\mathit{n}}}$. Hence, the formulas in (10) and (11), respectively, derive ${∆\mathit{p}}_{\mathit{n}|\mathit{s}}$ and ${∆\mathit{p}}_{\mathit{n}|{\mathit{B}}_{\mathit{n}}}$ from ${\mathit{u}}_{\mathit{n}}$ in every timestep n. In (10), ${∆p}_{n|s}〈z〉$ is the average of the tool–worksurface distance corrections, $u_n〈1〉$, $u_n〈2〉$, $u_n〈3〉$, and $u_n〈4〉$ to align the toolpath TCP with a sensed worksurface. The tool orientation corrections, ${∆p}_{n|s}〈{\theta }_y〉$ and ${∆p}_{n|s}〈{\theta }_x〉$, are derived from ${\mathit{u}}_{\mathit{n}}$ by using the approach demonstrated in Figure 3, where the sensed deviations, $a_{n|ref}$, $b_{n|ref}$, $c_{n|ref}$, and $d_{n|ref}$, are replaced by the tool–worksurface distance corrections, $u_n〈1〉$, $u_n〈2〉$, $u_n〈3〉$, and $u_n〈4〉$, respectively, in (10) to align the tool reference plane with a sensed worksurface. The $\eta$ sensitivity factor in (10) is 0.5 in this study. In (11), the TCP corrections, $u_n〈5〉$, $u_n〈6〉$, and $u_n〈7〉$, are applied to the tool pose correction ${∆\mathit{p}}_{\mathit{n}|{\mathit{B}}_{\mathit{n}}}$ to reach the incremental tool target ${\widehat{\mathit{p}}}_{{\mathit{B}}_{\mathit{n}}}$. Figure 4 provides an overall schematic of realtime error tracking with linear quadratic regulation for adaptive toolpath planning with sparse sensing, where ${\dot{p}}_{ref}$ is a reference tool speed that encompasses a predefined TCP speed and a predefined tool angular speed.

$${∆\mathit{p}}_{\mathit{n}|\mathit{s}}=\left[\begin{array}{c}\begin{array}{c}{∆p}_{n|s}〈x〉\\ {∆p}_{n|s}〈y〉\\ {∆p}_{n|s}〈z〉\end{array}\\ \begin{array}{c}{∆p}_{n|s}〈{\theta }_x〉\\ {∆p}_{n|s}〈{\theta }_y〉\\ {∆p}_{n|s}〈{\theta }_z〉\end{array}\end{array}\right]=\left[\begin{array}{c}\begin{array}{c}0\\ 0\\ {\displaystyle \frac{u_n〈1〉+u_n〈2〉+u_n〈3〉+u_n〈4〉}{4}}\end{array}\\ \begin{array}{c}{\tan }^{-1}\left(\eta *{\displaystyle \frac{u_n〈1〉-u_n〈3〉}{l}}\right)\\ {\tan }^{-1}\left(\eta *{\displaystyle \frac{u_n〈2〉-u_n〈4〉}{l}}\right)\\ 0\end{array}\end{array}\right]$$

(10)

$$\begin{array}{ll}{∆\mathit{p}}_{\mathit{n}|{\mathit{B}}_{\mathit{n}}}& ={\left[\begin{array}{ccc}\begin{array}{cc}{∆p}_{n|B_n}〈x〉& {∆p}_{n|B_n}〈y〉\end{array}& \begin{array}{cc}{∆p}_{n|B_n}〈z〉& {∆p}_{n|B_n}〈{\theta }_x〉\end{array}& \begin{array}{cc}{∆p}_{n|B_n}〈{\theta }_y〉& {∆p}_{n|B_n}〈{\theta }_z〉\end{array}\end{array}\right]}^{\mathit{T}}\\ & ={\left[\begin{array}{ccc}\begin{array}{cc}{u}_n〈5〉& u_n〈6〉\end{array}& \begin{array}{cc}{u}_n〈7〉& 0\end{array}& \begin{array}{cc}0& 0\end{array}\end{array}\right]}^{\mathit{T}}\end{array}$$

(11)

The proposed realtime toolpath planning then employs linear quadratic regulation (LQR) to compute the gain matrix $\mathit{K}$ in (8) using a customizable value function $\varphi \left({\mathit{e}}_{\mathit{n}}, \mathit{\pi }\right)$ in (12), where $\mathit{\pi }$ is a control sequence in (13) that steers the error tracking model in (6) from its current state ${\mathit{e}}_{0|\mathit{n}}$ (or ${\mathit{e}}_{\mathit{n}}$) to a final state ${\mathit{e}}_{\mathit{\infty }|\mathit{n}}$ on an infinite time horizon.

$$\varphi \left({\mathit{e}}_{\mathit{n}}, \mathit{\pi }\right)={\sum }_{t=0}^{\infty }\left({\mathit{e}}_{\mathit{t}|\mathit{n}}^{\mathit{T}} \mathit{Q}{\mathit{e}}_{\mathit{t}|\mathit{n}}+{\mathit{u}}_{\mathit{t}|\mathit{n}}^{\mathit{T}} \mathit{R}{\mathit{u}}_{\mathit{t}|\mathit{n}}\right); {\mathit{e}}_{0|\mathit{n}}={\mathit{e}}_{\mathit{n}},\mathit{Q}={\mathit{Q}}^{\mathit{T}}\succcurlyeq 0, \mathit{R}={\mathit{R}}^{\mathit{T}}\succ 0$$

(12)

$$\mathit{\pi }=\left\{{\mathit{u}}_{\mathit{t}|\mathit{n}} {\in \mathbb{R}}^6 | t=0,1, \dots , \infty \right\}$$

(13)

In (12) and (13), $t\in \mathbb{N}$ is a timestep index on an infinite time horizon. $\mathit{Q}\in {\mathbb{R}}^{7\times 7}$ and $\mathit{R}\in {\mathbb{R}}^{7\times 7}$ are customizable state and control weight matrices, respectively. The goal of the optimization problem in (14), which is subject to (6) and (7), is to find an optimal control sequence ${\mathit{\pi }}^{\mathit{*}}$ to steer the error tracking model in (6) from its current state ${\mathit{e}}_{0|\mathit{n}}$ (or ${\mathit{e}}_{\mathit{n}}$) to an optimal state ${\mathit{e}}_{\mathit{\infty }|\mathit{n}}^{\mathit{*}}$ that converges to zero on an infinite time horizon.

$$\left\{\begin{array}{c}\varphi \left({\mathit{e}}_{\mathit{n}}\right)=\underset{\mathit{\pi }}{\min }\varphi \left({\mathit{e}}_{\mathit{n}}, \mathit{\pi }\right)\\ {\mathit{\pi }}^{\mathit{*}}=\underset{\mathit{\pi }}{\mathrm{argmin}}\varphi \left({\mathit{e}}_{\mathit{n}}, \mathit{\pi }\right)\end{array}\right.$$

(14)

To solve the optimization problem in (14) that is subject to (6) and (7), a cost-to-go matrix $\mathit{P}\in {\mathbb{R}}^{7\times 7}$ can be identified using the discrete algebraic Riccati equation (DARE) in (15) on an infinite time horizon.

$$\mathit{P}=\mathit{Q}+\mathit{P}-\mathit{P}{\left(\mathit{R}+\mathit{P}\right)}^{-1}\mathit{P};\mathit{P}={\mathit{P}}^{\mathit{T}}\succcurlyeq 0$$

(15)

After solving (15) for $\mathit{P}$, a time-invariant gain matrix of $\mathit{K}$ can be found according to formula in (16) for the realtime error tracking model in (6) and its control input in (7).

$$\mathit{K}={\left(\mathit{R}+\mathit{P}\right)}^{-1}\mathit{P}$$

(16)

Many researchers including Al Tamimi et al. [50], Rawlings et al. [51], and Scokaert et al. [52] documented distinct variants of optimal control and linear quadratic regulation with in-depth mathematical derivations and proofs on stability and controllability. Their research papers can provide more theoretical details for further reading.

2.3. Experimental Settings

This study involves a simulation experiment on an industrial robot with end-of-arm distance sensors and a workpiece with unknown surface variations to demonstrate the proposed toolpath planning with sparse sensing using RobotStudio^® 2024, which is an industrial robot programming and simulation program [53]. RobotStudio^® 2024 has digital engineering capabilities of robot modeling, programming, and simulation with virtual sensors and devices. The simulation experiment employed a virtual ABB IRB-1200 robot with end-of-arm distance sensors and a workpiece with unknown surface variations as modeled in RobotStudio^® 2024. Figure 1b portrays the setup of the simulation experiment. The robot tool frame with the tool center point (TCP) was arranged at 200 mm from the sensor plane along the tool ${\mathit{z}}_{\mathit{T}}$ axis. The diagonal length l of the sensor square arrangement was 40 mm. The sensor inclination angle $\phi$ was 90 degrees for all four sensors. The robot was tasked to start from a known start pose and then reach a known final tool target by aligning the TCP with unknown worksurface variations and maintaining tool orthogonality relative to the sensed surface. It is assumed that the TCPs of both the start and the end poses are predefined worksurface points. The end pose served as the frame of reference. The ${\dot{p}}_{ref}$ reference tool speed of simulation comprised 100 mm/s as a TCP speed and 1000 deg./s as a tool angular speed. The $∆p$ uniform increment of TCP was 10 mm. The timestep of the simulation was 0.5 s. The data sampling rate was 125 Hz.

This study used sensor noises to test the robustness of the proposed approach. Sensor noises can have various sources, such as vibrations, environmental disturbances, sensor calibration errors, etc. To apply noise to the sensor perception, each sensor reading was subjected to a random scalar from a 4 mm range between −2 mm and +2 mm.

To demonstrate customizable optimal toolpath planning, three optimal regulation cases were considered via customized weight matrices in

Table 1. Experimental cases with customized weight matrices.

Cases	$\mathbf{Error} \mathbf{State} \mathbf{Weight} \mathbf{Matrix} \mathit{Q}{ \in \mathbb{R}}^{7\times 7}$	$\mathbf{Control} \mathbf{Weight} \mathbf{Matrix} \mathit{R}{ \in \mathbb{R}}^{7\times 7}$
Case 1: Reaching the known toolpath target is priority.	$\mathit{d}\mathit{i}\mathit{a}\mathit{g}(1,1,1,1,10,10,10)$	$\mathit{d}\mathit{i}\mathit{a}\mathit{g}(10,10,10,10,1,1,1)$
Case 2: Balancing the Case 1 and Case 3 goals is priority.	$\mathit{d}\mathit{i}\mathit{a}\mathit{g}(10,10,10,10,10,10,10)$	$\mathit{d}\mathit{i}\mathit{a}\mathit{g}(10,10,10,10,10,10,10)$
Case 3: Following sensed surface variations is priority.	$\mathit{d}\mathit{i}\mathit{a}\mathit{g}(10,10,10,10,1,1,1)$	$\mathit{d}\mathit{i}\mathit{a}\mathit{g}(1,1,1,1,10,10,10)$

, where $\mathit{Q}{ \in \mathbb{R}}^{7\times 7}$ and $\mathit{R}{ \in \mathbb{R}}^{7\times 7}$ are error state and control weight matrices, respectively. These weight matrices are used for solving the optimization problem in (14), which is subject to the error tracking model in (6) with the control input in (7). The customizable weights are based on a simple scoring range between 1 and 10, which serves as a handle on intuitively praising or penalizing certain error states and control actions for optimal toolpath planning.

In Case 1, the weights for sensed deviations, which are the first four entries in the diagonal of the error state weight matrix $\mathit{Q}$, are 1s, while the remaining weights for toolpath target errors are 10s. This means toolpath target errors are more costly to endure than sensed deviations. Also, the weights for sensed deviation control, which are the first four entries in the diagonal of the control weight matrix $\mathit{R}$, are 10s, while the remaining weights for toolpath target control are 1s. It means sensed deviation control is more costly to execute than toolpath target control. Overall, the Case 1 weights imply that reaching the known toolpath target is the top priority.

In Case 2, the customized weights are all equal to 10s, which means reaching the known toolpath target and following sensed surface variations are equally important.

In Case 3, the weights for sensed deviations, which are the first four entries in the diagonal of the error state weight matrix $\mathit{Q}$, are 10s, while the remaining weights for toolpath target errors are 1s. It means sensed deviations are more costly to endure than toolpath target errors. Also, the weights for sensed deviation control, which are the first four entries in the diagonal of the control weight matrix $\mathit{R}$, are 1s, while the remaining weights for toolpath target control are 10s. It means toolpath target control is more costly to execute than sensed deviation control. Overall, the Case 3 weights imply that surface tracking is the top priority.

A time-invariant gain matrix of $\mathit{K}$ was computed according to (16) for each case using the weights in Table 1 and the linear quadratic optimization in (14) that is subject to (6) and (7).

Table 2. Time-invariant gains obtained via linear quadratic regulation for the three cases in Table 1.

Cases	$\mathbf{Gains} \mathbf{for} \mathbf{Surface} \mathbf{Error} \mathbf{Tracking}: {\mathit{K}}_{\mathit{s}}$	$\mathbf{Gains} \mathbf{for} \mathbf{Toolpath} \mathbf{Target} \mathbf{Error} \mathbf{Tracking}: {\mathit{K}}_{\mathit{B}}$
Case 1: Reaching the known toolpath target is priority.	0.27	0.92
Case 2: Balancing the Case 1 and Case 3 goals is priority.	0.62	0.62
Case 3: Following sensed surface variations is priority.	0.92	0.27

presents the computed LQR gains. These LQR gains were used to systematically demonstrate optimal toolpath planning with and without sensor noises.

3. Results and Discussion

The simulation experiment involved three optimal realtime toolpath planning cases as specified in Table 1 and Table 2 with and without sensor noises. To apply artificial noise to sensor perception, each sensor reading was subjected to a random number from a 4 mm range between −2 mm and +2 mm. In each case, an industrial robot with end-of-arm distance sensors was tasked to travel between known start and end tool poses by maintaining the tool orthogonality and the TCP alignment with the sensed worksurface as well as the ZX plane of the reference frame. The TCPs of both the start and the end tool poses were worksurface points. The end pose (the final tool target) also served as the reference frame. The ${\dot{p}}_{ref}$ reference tool speed of simulation comprised 100 mm/s as the TCP speed and 1000 deg./s as the tool angular speed. The $∆p$ uniform increment of TCP was 10 mm, and the timestep of simulation was 0.5 s. The data sampling rate was 125 Hz.

Reaching the known toolpath target is the top priority in Case 1. Worksurface tracking is the top priority in Case 3. Balancing the Case 1 and Case 3 goals is the priority in Case 2. Figure 5 presents an image sequence of the Case 3 simulation.

The cycle times of Case 1, Case 2, and Case 3 without sensor noises were about 16 s, 18 s, and 23 s, respectively. The total lengths of the Case 1, Case 2, and Case 3 toolpaths without sensor noises were about 848 mm, 852 mm, and 866 mm, respectively. The cycle times of Case 1, Case 2, and Case 3 with sensor noises were about 16 s, 19 s, and 28 s, respectively. The lengths of the Case 1, Case 2, and Case 3 toolpaths with sensor noises were about 848 mm, 855 mm, and 908 mm, respectively. Sensor noises affected Case 3 more than any other cases since sensor-based surface tracking is the top priority in Case 3.

Figure 6, Figure 7 and Figure 8 present comparisons between the toolpath orientation angles of an ideal case and the three cases with and without sensor noises. The ideal case involves ideal tool poses to achieve along an ideal toolpath defined by the intersection between the worksurface and the XZ plane of the reference frame. Figure 9 and Figure 10 present comparisons between the ideal toolpath and the toolpaths of the three cases with and without sensor noises. Figure 11 presents the toolpath TCP errors of the three cases with and without sensor noises. A positive error in Figure 11 means the TCP is above the worksurface, and a negative error means the TCP is below the worksurface.

The results in Figure 6, Figure 7 and Figure 8 indicate that each experimental case was able to accomplish the anticipated optimal path planning goals in accordance with the customized weights specified in Table 1 and the LQR gains in Table 2. The tool ${\theta }_z$ angle is nicely maintained at 90 degrees in each of the three cases even in the presence of sensor noises. In comparison with the ideal case and Case 1, both Case 2 and Case 3 produced fairly accurate toolpath orientation angles but longer cycle times since tracking sensed surface variations requires more time to accomplish, especially in Case 3. In contrast, Case 1 produced less-accurate toolpath orientation angles but the shortest cycle time since reaching the known tool target is the top priority of Case 1. This urgency to reach the end target causes insufficient time to accurately track sensed surface variations. Added sensor noises barely affected the performances of Case 1 and Case 2 but noticeably affected the Case 3 performances. Since sensor-based surface tracking is the top priority in Case 3, sensor readings play essential roles in Case 3, which makes it more sensitive to sensor noises. This sensitivity can be reduced by reducing the $\eta$ sensitivity factor in (10). Note that this study used a sensitivity factor of $\eta =0.5$. The results in Figure 6, Figure 7 and Figure 8 are summarized in Figure 12, which presents the Root Mean Square (RMS) errors [23] of the tool orientation for the three cases with and without sensor noises. Case 1 produced the largest angular RMS errors in the tool ${\theta }_x$ and ${\theta }_y$ angles in comparison with the ideal toolpath orientation angles shown in Figure 6 and Figure 7, while Case 3 and Case 2 produced much lower angular RMS errors.

According to Figure 9, Figure 10 and Figure 11, the toolpath TCP coordinates of the three cases eventually converge at zero when the robot tool reaches the final tool target at the end of each process cycle. Note that the final tool target is also the reference frame. Figure 9 shows that the TCP alignment with the XZ plane of the reference frame looks flawless in each of the three cases with and without sensor noises as expected. Figure 10 presents comparisons between the ideal toolpath and the toolpaths of the three cases with and without noises. The results in Figure 9, Figure 10 and Figure 11 are summarized in Figure 13. In terms of tracking unknown surface variations, the toolpaths of Case 3 and Case 2 are the most accurate ones with RMS errors of less than 2.5 mm even in the presence of sensor noises. The Case 3 toolpath has some fluctuations in Figure 10 and Figure 11, especially in the presence of sensor noises. Such unwanted fluctuations can be reduced by reducing the $\eta$ sensitivity factor in (10). The Case 1 toolpaths with and without sensor noises exhibit noticeable RMS errors of over 7 mm, which is expected since worksurface tracking has a lower priority in Case 1.

Figure 14 presents box-and-whisker plots of the actual toolpath speeds. Each box-and-whisker plot depicts the upper and lower extremes, the upper and lower quartiles, and the median of each dataset. Since the main emphasis of the proposed method is on enhancing the tool pose accuracy of the toolpaths, the proposed method does not regulate tool speeds by letting the robotic system control its tool speeds based on the commanded or reference tool speeds and kinematic constraints. The commanded or reference tool speeds of the simulation experiment were 100 mm/s as a linear speed of the toolpath TCP and 1000 deg/s as an angular speed of toolpath orientation. The actual toolpath speeds of the simulation experiment were lower than the reference tool speeds as the robot tool needed to slow down to apply new sensor-based changes to the tool pose in every timestep of 0.5 s. This limitation can be resolved by tool speed planning with speed interpolation as well as a specific sensor arrangement that allows the robot to sense the worksurface ahead of the tool along the toolpath. Moreover, Model Predictive Control (MPC) can be instrumental in achieving proper tool speed planning. Needless to say, both LQR and MPC are specific variants of optimal control.

The simulation experiment successfully demonstrated the predefined optimal path planning strategies with and without noises. These findings indicate that the proposed toolpath planning method with sparse sensing has the potential to automate non-contact surface processing. In this study, sparse sensing required fewer measurements and less computing than that for standard machine vision. The computational time complexity of sparse sensing was $O\left(n\right)$, which was much lower than $O(n\times m)$ of a standard image convolution algorithm typically used in machine vision, where n and m are the total number of timesteps and the total number of image pixels, respectively. The big $O$ notation is a math notation that describes the limiting behavior of an algorithm or a function when the input argument tends toward infinity.

A limitation of the proposed path planning approach is an inability to deal with worksurface discontinuities, such as edges, holes, pockets, steep protruding surfaces, or sharp ridges. A possible solution to this problem can be smart sensor arrangements along with an intelligent algorithm that can detect surface discontinuities by examining distance sensor readings. Another limitation of the proposed toolpath planning approach is that it regulates the tool pose but not the tool speed. This limitation can be resolved by adding toolpath speed planning with speed interpolation. The future work will focus on resolving these limitations, establishing benchmark datasets for comparisons with other methods, and implementing artificial intelligence for better robot tool motion planning and control.

4. Conclusions

This study proposes an optimal toolpath planning method for an industrial robot equipped with end-of-arm distance sensors to automatically guide its tool motion along unknown worksurface variations. The end-of-arm distance sensors facilitate sparse sensing to acquire sparse data that is just enough for the quick, adequate perception of unknown worksurfaces. Optimization facilitates the optimality of multi-objective toolpath planning with a customizable value function, where the multiple objectives comprise adapting to unknown worksurface variations and traveling between known tool targets. To validate the proposed toolpath planning method with sparse sensing, this study conducted a simulation experiment on a virtual robot equipped with end-of-arm distance sensors and a workpiece with unknown surface variations. Sparse sensing required fewer measurements and less computing than that for conventional machine vision. The simulation experiment successfully demonstrated the predefined optimal strategies with and without sensor noises. These findings indicate that the proposed toolpath planning method with sparse sensing has the potential to automate non-contact surface processing. The future work will focus on adding tool speed planning, establishing benchmark datasets for comparisons with other methods, and implementing artificial intelligence for better robot tool motion planning and control.