1. Introduction
With the rapidly developing trend of Industry 4.0, industrial robots have been widely used in various applications, such as automotive manufacturing lines [
1], semiconductor production lines [
2], etc. For the goal of autonomous control of industrial robots, various methodologies have been developed to achieve high-accuracy robot manipulation. It has been expected that an Industry 4.0 production line is massively customized. In a mass-customization production line, there exist various kinds of products with different geometrical shapes. There may not be a general solution for automation. Thus, sensing and manipulation with a high degree of freedom became essential for handling these mass-customized products.
In a traditional production line, additional workers or devices may be needed to align or position their products in the conveying system. Inline inspection, therefore, is required to be able to recognize the position and orientation of an inspected object without the need for additional human workers and positioning devices. Moreover, the conveying system usually needs to stop for a couple of seconds to perform stable optical inspections. Stoppage of the conveying system decreases the overall production yield. It is desirable to develop an inline system that is capable of performing inspection of a product in a moving conveyor.
Recently, researchers have developed several robotics systems for picking or inspecting target objects in a moving conveyor or a production line. Han, et al. [
3] determined the optimal path planning of a robot arm to pick randomly placed objects in a moving conveyor in terms of minimizing the pick-and-place time in the reachable range of the robot arm. They found that the effectiveness of object picking is related to how the objects are distributed in the conveyor. Anwar, et al. [
4] developed some visual servo methodologies to move a camera at the end effector of a robot arm to face the inspection surface on an object. The end effector position was determined based on an image-based Jacobian approach. Abbood, et al. [
5] used a camera and some image-processing techniques to identify the shape of an object in a moving conveyor. A low-cost system based on a simple robot arm with plastic gears and servo motors was built to demonstrate the concept of picking objects at a slow moving speed of less than 9 cm/s.
One of the most important features in the inline inspection system for mass-customization production lines is high-accuracy manipulation. Juan and his colleagues [
6,
7] measured the absolute position errors of the end effector of a 6-revolute-joint (6R) robot arm in the working space by laser. Accordingly, the error field of the end effector in the working space was parametrically modeled by kriging and was utilized to compute the corrected coordinates of the end effector. With the high accuracy of the laser measurement (i.e., ~0.001 mm), the absolute accuracy of the end effector was improved from ~1 mm to ~0.05 mm. However, the experimental setup of laser measurement was very time consuming.
To make the 3D position measurement of the end effector more efficient, Juan and his colleagues [
8,
9] measured the distances from the end effector to three different cable encoders. These three length measurements were then used to compute the 3D position of the end effector by trilateration. With a lower accuracy of the cable encoder (i.e., ~0.1 mm), the accuracy of the absolute positioning of the end effector reached the range of ~0.5 mm. Lin, et al. [
10] also showed how multilateration could be used to estimate the end effector position when more than 3 cable encoders were used to measure their distances to the end effector. Shih and Lin [
11] started to investigate how the end effector positioning under various levels of payloads could be improved using trilateration based on three cable encoder measurements and coordinate correction based on kriging models of error fields.
Cable encoder measurements show good potential on 3D positioning of industrial robots but they are not suitable for in situ measurements. Shih, et al. [
12] developed a vision-based correction of end effector positioning in planar motions. A 2D camera was used to compute the transformation between the camera coordinates and the end effector coordinates in a given plane. Any target position of the end effector in the plane could then be computed with its coordinate in the live image. This approach was good for in situ measurement and robot manipulation but it only worked for positioning in a given plane and not suitable for moving target positions.
Wu, et al. [
13] utilized a 3D scanner to capture the geometry and orientation of an object, which was randomly placed in a moving conveyer. Since the moving speed and direction of the conveyer was known, a 6R robot was able to move along with the moving object as it passed through the working range of the robot. The developed stereoscopic object tracking could be utilized for the application of inline inspection of randomly placed objects in a moving conveyer with known speed and direction. Wu, et al. [
14] presented the development and implementation of inline inspection based on stereoscopic object scanning and object tracking by a 6R robot.
In this paper, a method of inline inspection with an industrial robot (IIIR) and its applications in mass-customization production lines are presented.
Section 2 introduces the experimental setup and the required methodologies for matching the coordinates of the industrial robot and the 3D scanner.
Section 3 explains how the camera position could be estimated based on the computer-aided design (CAD) model of a given object and the 3D point cloud of the object captured under the 3D scanner.
Section 4 shows the experimental process and the results. Finally, the conclusion is given in
Section 5.
2. Experimental Setup and Initialization of the Inline Inspection with a 6R Robot
Figure 1 shows the experimental setup of the developed inline inspection system [
13]. A 3D scanner (i.e., stereovision camera with two pixel sensors in a resolution of 2 micron) was installed at the top of the entrance of a conveyer. The moving speed of the conveyer was denoted as
and was given as 33.8 mm/s. The moving direction of the conveyer was along the +X direction of robot coordinate, which could be seen in
Figure 1. As an arbitrarily shaped object entered the conveyer, its 3D geometry and orientation would be captured by the 3D scanner. The 3D point cloud acquired by the 3D scanner would be used to calculate the desired positions and orientations of the end effector of the industrial robot. Accordingly, the end effector was desired to move along with the object and perform optical inspections without stopping the conveyer. This paper will present the required methodologies to estimate the end effector coordinates for the proposed inline inspection.
To match the 3D scanner coordinate,
, and the robot coordinate,
, the transformation from
to
, denoted as
, should be computed. The end effector of the robot was moved to
different positions within the inspection range of the 3D scanner. In this paper,
and
points were uniformly distributed, as shown as the red circles in
Figure 2. The
end effector position observed in
was denoted as
. Accordingly, the 3D scanner was used to captured these end effector positions and
end effector position captured in
was denoted as
. The
transformed 3D scanned position was denoted as
and should satisfy the following equation:
Since the end effector coordinate should be closed to the transformed 3D scanned coordinate , the transformation could then be estimated by least square approximation (LSA).
The Equation (1) can be rewritten as the following matrix form:
where each vector was formulated in the homogeneous coordinate. To determine the unknown parameters,
for
and
, the data from all
experiments was put together producing the following equation:
for
. Equation (3) is rewritten as the following:
where
is a
vector that contains the
coordinate of every
;
is a
matrix that contains all the coordinates of every
;
is a 4 × 1 vector that contains the components of the
vrow of
. Based on LSA, the unknown parameters in
could be estimated by:
The transformed coordinates of the 100 end effector positions measured by the 3D scanner are shown as the blue crosses in
Figure 2. The errors (i.e., coordinates of the end effector positions minus the transformed ones of the 3D scanned positions) along the X, Y and Z directions are shown in
Figure 3,
Figure 4 and
Figure 5, respectively. The averages and standard deviations of the errors along the X, Y and Z directions are listed in
Table 1. Since the 3D scanner is less accurate in the Z direction than the X and Y directions, the standard deviation of errors along the Z direction is greater than those along the X and Y directions.
Figure 6 shows the Euclidean errors between the end effector positions and transformed ones of the 3D scanned positions. Based on the averaged value of Euclidean errors listed in
Table 1, the error between
and
was around 2 mm, which was mostly caused by the lower accuracy of the 3D scanner along the Z direction.
3. Estimation of the End Effector Coordinates for the Inline Inspection
Figure 7 illustrates the position of the camera for the desired inspection. Suppose the CAD model of the investigated object was known, as shown in
Figure 7a. The camera was expected to take a picture facing a target point,
, on the object along the direction of
, where
stands for the normal vector of object surface near
. The CAD model and the desired camera position were saved in the database as the cyber model for later object recognition and positioning of camera. The distance from
to the center of the camera lens,
, was given as
. Therefore,
could be determined by:
The transformation from to could be extended to more complicated conditions. This paper only focuses on the translation along the direction of with a distance of .
As the true object entered the conveyer as shown in
Figure 1, the geometry and orientation of the object was captured by the 3D scanner, as shown as the blue points in
Figure 7b. Iterative closest point (ICP) [
15] determined the transformation,
, from the known CAD model to the point cloud obtained from 3D scanner. ICP, which was first introduced by Besl and McKay [
15] in 1992, iteratively finds the minimal distance between two sets of points, lines, or surfaces. Later on, Fischler and Bolles [
16] used random sample consensus (RANSEC) to improve the accuracy of 3D data sampling. Wahl, et al. [
17] was able to capture the geometrical features from the sampled 3D data and analyze the histogram of the point features. Rusu, et al. [
18] then introduced a fast point feature histogram (FPFH) to speed up the process of point feature analysis. This paper uses the FPFH to estimate the transformation between the known CAD model and the captured 3D point cloud.
Based on the transformation of the model matching, the target point on the object, denoted as
, was then found near the point cloud captured from the 3D scanner and was given by:
The normal vector of the surface near
was determined by LSA and was denoted as
. Therefore, the position of the camera to perform the desired inspection could be computed by:
Consider the object entered the conveyer at time of
, the coordinate of
may not be inside the working range of the robot. It’s necessary to estimate at what time the object would move into the working range of the robot so that the desired inspection could be performed. In our implementation, the conveyer moved along the +X direction of the robot coordinate. Therefore, the Y- and Z-components of
did not change with time. Only the X-component of
, denoted as
, varied with time
and could be written as:
where
and
stand for the X-components of
and
, respectively. Suppose the
entered the working range of the robot and
moved out of the working range. The inspection should be done within the time interval of
. From Equation (9), the limits of the inspection time could be determined by:
where
and
represent the minimal and maximal X-coordinates of the working range of the robot, respectively. Finally, the coordinates of the end effector for following the object and performing the desired inspection were determined, i.e.,
,
, …,
. At each time
, the desired position and orientation of the end effector were computed based on the inverse kinematics in the motor control system.
4. Experiment and Result of the Proposed Inline Inspection
The experimental process of the presented methodology is shown in
Figure 8. An object to be inspected entered the conveyer at the beginning of the experiment, as shown in
Figure 8a. The geometry and the orientation of the object were then captured by the 3D scanner. Suppose the CAD model of the object was known, the transformation between the captured point cloud and the CAD model in the database was estimated by ICP, as explained in
Section 3. The target inspection point on the surface of the object,
, was then calculated as well as the normal vector of the inspected surface,
. Given the desired distance between the camera and the target point,
d, the desired positions of the camera,
, could then be calculated.
Considering the known moving speed of the conveyer and the working range of the industrial robot, the camera could reach the desired positions between time
and time
, as shown in Equations (10) and (11), respectively.
Figure 8b shows the moment of time
where the camera moved to the desired position of
. The industrial robot started to follow with the movement of the object to create near-zero relative movement between the camera and the moving object, as shown in
Figure 8c. The inline spection was then performed before the object moved out of the reachable range at time
.
In the experiments of the inline inspection, a flat plate with a circular mark, as shown in
Figure 9, was to be inspected. The center of the circular mark was the target point for inspection. The normal vector of the plate had a directional angle,
, from the X axis and a tilting angle,
, from the X-Y plane, as shown in
Figure 10. The area of the circular mark was 301.69 mm
2. The area of the circular mark would be measured during the object following and inline inspection to show the accuracy of the optical inspection. The relative movement between the camera and the moving plate would be investigated to show the stability of the inline inspection system.
In this experiment, the directional angle
of the inspection surface was
and the tilting angle
was
. The exact position of the object was randomly placed on top of the conveyer. The scale factor of the inspected image was 0.1865 (mm/pixel). From the beginning to the end of the inspection process, one inspection image of the flat plate was taken for every 0.5 s. The relative movement of the center of the circular mark between each consecutive image was measured and shown in
Figure 11. The average relative movement of center was 0.2460 mm and the maximum relative movement was 0.6521 mm. Taking derivative with time, the absolute relative speed of the center was shown in
Figure 12. The average speed was 0.3358 mm/s and the maximum speed was 0.9420 mm/s. For a camera of 60 frames per second (fps), the relative moving speed between the object and the camera was around 6 micron (around 16 micron for the worst case), which was stable enough for most industrial production inspections. During the inspection process, the area of the circular mark was measured and shown in
Figure 13. Compared to the true circular area, 301.69 mm
2, the error of the area measurement during the inspection process is shown in
Figure 14. The average error was 3.6695 mm
2 (1.21% of error) around and the maximum error was 5.2122 mm
2 (1.73% of error). The results of the presented experiment are listed in
Table 2.
A comparison between the presented IIIR and some other existing methods that were developed in the past few years is given in
Table 3. The investigated systems were developed for either pick-and-place applications [
5,
12,
19,
20] or production inspections (i.e., [
4] and this work). For planar manipulations [
5,
12], image-based object recognition would be enough; on the other hand, 3D scanning was utilized for recognizing the objects that were either randomly placed (i.e., [
20] and this work) or had arbitrary shapes [
19]. The challenges of the robot manipulations increased as the objects were placed in a moving platform (i.e., [
5] and this work). The comparison in
Table 3 showed that the presented IIIR delivered a unique ability to recognize randomly placed objects in a moving platform with good accuracy (i.e., relative speed between camera and object = 0.3358 mm/s = 5.6 micron/frame for a 60-fps camera), which is important for the inline inspection in a mass-customization production line.
5. Conclusions
This paper has presented a method of inline inspection with an industrial robot (IIIR) that carries a camera to move along with moving object and performs the desired optical inspections in mass-customization production lines. The developed inline inspection system integrated multiple technologies, including a 3D scanning, embedded system, coordinate transformation, robot control and machine vision. In our implementation, the relative speed between the moving object and the camera and the speed was around 0.34 mm per second (worst case was around 0.94 mm per second). For a camera of 60 frames per second, the relative moving speed between the object and the camera was around 6 microns (around 16 microns for the worst case), which was stable enough for most industrial production inspections. Furthermore, the average error of the inspection measurement was round 1.21% (worst case was 1.73% of error). The developed system could be extended to various kind of inline operations for automation applications in Industry 4.0.
The main contribution of this paper was the integrated system of a 3D scanner for recognizing the position and orientation of a moving object, an industrial robot for moving to the desired position to perform inline inspection without stopping the conveyor, and a series of numerical methods for matching the coordinates of the 3D scanner and the robot arm, and calculating the desired end effector positions for inline inspections. The developed system was able to perform accurate object tracking without using costly measurement devices such as a high-speed camera or laser tracker. Therefore, it could be widely applied to various kinds of automation production lines but also highly applicable for small- and medium-sized enterprises.