A Remote Controlled Robotic Arm That Reads Barcodes and Handles Products

Abstract

In this study, a 6-axis robotic arm, which was controlled by an embedded Raspberry Pi with onboard WiFi, was developed and fabricated. A mobile application (APP), designed for the purpose, was used to operate and monitor a robotic arm by means of a WiFi connection. A computer vision was used to read common one-dimensional barcode (EAN code) for the handling and identification of products such as milk tea drinks, sodas and biscuits. The gripper on the end of the arm could sense the clamping force and allowed real-time control of the amount of force used to hold and handle the products. The packages were all made of different material and this control allowed them to be handled without danger of damage or deformation. The maximum handling torque used was ~1.08 Nm and the mechanical design allowed the force of the gripper to be uniformly applied to the sensor to ensure accurate measurement of the force.

1. Introduction

Nowadays, robotic arms are often used to perform complex or highly repetitive tasks. The use of a robot eliminates the kind of mistakes that a weary person would make after doing a boring and tiresome task for a long time. Advances in technology have made it possible for devices with a wide range of different kinds of sensors to be devised and these are evident everywhere in our living environment [1,2]. Computer vision, using small video cameras, can identify and control motion in a robotic arm. It is only necessary for settings and parameters to be set for a robot to automatically carry out tasks such as the classification and picking and placing of objects [3,4].

Supermarkets and large retail stores face many challenges: shortages, incorrect placement on shelves and theft. Fierce market competition has led to a decline in profits and high labor costs make it difficult to ensure that there is enough staff to handle all the work in a store [5,6]. Walgreens, for example, has more than 8200 stores in the United States and they sell many tens of thousands of products every day and even the counting of so many items takes staff many hours to complete [7,8]. In 2016, Zhang et al. [9] designed a mobile robot that can automatically move along a preset path, has a radio-frequency-identification (RFID) scanner, and can carry out an inventory and scan and tag products while it is moving. In 2015, the Simbe Robotics Company developed a new robot that moves continuously along shelves to check the contents. When a problem is found, a message is sent automatically to staff for action to be taken [8,10].

The aim of this study was the development of a robotic arm with computer vision identification, as well as a mobile application (APP) for the barcode identification of goods using a WiFi connection [11,12]. The devised system could be installed on a vehicle that moves around the store to scan the bar codes of goods on the shelves. It can notify staff of the need for goods replenishment and can move products that have been incorrectly placed [13,14]. Traditional vision identification applications usually involve the use of a PC and a considerable amount of power and complicated software. However, the small computers now available are powerful, use very little power and the software needed is simple and versatile. The use of these little devices is economical and they can easily be used for such applications as goods inventory [15,16].

2. Methods and Design

2.1. The Design of a 6-Axis Robotic Arm

The 6-axis robotic arm designed and fabricated for this study is shown in Figure 1. The design is modular and each rotating joint is realized using a single joint component that is rotated by a small AX-12A servomotor unit from Dynamixel (Lake Forest, CA, USA). The solid frame and joint parts were fabricated by 3D printing using a polylactide (PLA) polymer. The parts are assembled using small screws and nuts, which makes changes to the structure very easy. The joint elements and motors can be easily exchanged.

2.2. Mechanical Design of the Gripper

In this study we picked and placed several common packaged products. These included some made of paper, a regular aluminum can and a plastic biscuit container, see Figure 2. Their size and shape varied from 59 × 35 × 145 mm (paper package), to 66 dia × 91 mm long (aluminum can), to a 65 dia × 100 mm (biscuit package).

We designed a gripper for the robotic arm with a total length of 165 mm and a maximum moving distance of 97 mm, to cope with the different shapes and nature of the packages, see Figure 3. The actuator used for the sliding motion of the two gripper fingers was a TowerProMG996R servo motor unit produced by Tower Pro Ltd. (Taipei, Taiwan) (www.towerpro.com.tw), which can generate a maximum torque of ~1.08 Nm at an operating speed of 0.14 s/60° and a voltage of 6 V.

Since the three test objects were quite different in nature, one being quite strong and the other two more fragile and subject to crushing. The force exerted by the gripper jaws needed to be adjustable. The one of most common means used for gripper actuation is compressed air [17]. However, the accurate control of air pressure actuation is not easy to achieve and the irregular objects gripped can easily be crushed by either precision or power prehensile grippers [18,19]. In order to cope with this issue, the gripper we used with a precision servo motor was fitted with two force sensors of FlexiForce A301 produced by Tekscan Inc. (Boston, MA, USA) (www.tekscan.com).

2.3. Kinematics

The forward and inverse kinematics of the robotic arm were physically modeled and numerically solved by Denavit-Hartenberg (D-H) matrix [20] based on the Equations (1) and (2). The robot coordinate frames for transformation are shown in Figure 4, and the corresponding D-H parameter of the robotic arm can be obtained in accordance with this model of coordinate frames.

Table 1. The Denavit-Hartenberg (D-H) parameters of the robotic arm.

Axis	èi (deg)	di (mm)	ai (mm)	ái (deg)
0	è1	0	0	90°
1	è2	196.25	0	0°
2	è3 − 90°	0	105.5	0°
3	è4 − 90°	0	100.4	0°
4	0°	98	0	0°
5	0°	165	0	0°

shows the physical parameters of the model that were substituted into the D-H matrix based on Equation (1). This yields a 0~5 axis rotation matrix. Using Equation (2), the rotation matrix of each axis was multiplied to obtain all of the elements of the matrix, as expressed in Equations (3) to (14), where è₂₃₄ = è₂ + è₃ + è₄. With P_x, P_y, and P_z as expressed by Equations (12) to (14), they were used to convert the angles to the coordinate system, where è₂₃ = è₂ + è₃.

The inverse kinematics of the robotic arm, which could be derived from Equation (2), are related to derivation of the forward kinematics as mentioned above. The position of each linkage joint could be obtained by multiplying the matrix of each joint linkage. Therefore, after Equation (2) has been applied, we multiplied both sides of the equation by (⁰A₁)⁻¹, (⁰A₁·¹A₂)⁻¹, (⁰A₁·¹A₂·²A₃)⁻¹, and (⁰A₁·¹A₂·²A₃·³A₄)⁻¹, Equations (15) to (30) will eventually be yielded, where s₂₃ = sin(è₂₃), s₁₂₃ = sin(è₁₂₃), c₂₃ = cos(è₂₃), and c₁₂₃ = cos(è₁₂₃).

$${}^{\mathrm{i}-1}\mathrm{A}_{\mathrm{i}}=\left[\begin{array}{cccc}\cos {\theta }_i& -\cos {\alpha }_i\sin {\theta }_i& \sin {\alpha }_i\sin {\theta }_i& a_i\cos {\theta }_i\\ \sin {\theta }_i& \cos {\alpha }_i\cos {\theta }_i& -\sin {\alpha }_i\cos {\theta }_i& a_i\sin {\theta }_i\\ 0& \sin {\alpha }_i& \cos {\alpha }_i& d_i\\ 0& 0& 0& 1\end{array}\right]$$

(1)

$${}^0\mathrm{T}_5={}^0\mathrm{A}_1·{}^1\mathrm{A}_2·{}^2\mathrm{A}_3·{}^3\mathrm{A}_4·{}^4\mathrm{A}_5=\left[\begin{array}{cccc}{n}_x& o_x& a_x& p_x\\ n_y& o_y& a_y& p_y\\ n_z& o_z& a_z& p_z\\ 0& 0& 0& 1\end{array}\right]$$

(2)

$$n_x=\sin {\theta }_1\sin {\theta }_5+\cos {\theta }_{234}\cos {\theta }_1\cos {\theta }_5$$

(3)

$$n_y=\cos {\theta }_{234}\cos {\theta }_5\sin {\theta }_1-\cos {\theta }_1\cos {\theta }_5$$

(4)

$$n_z=\sin {\theta }_{234}\cos {\theta }_5$$

(5)

$$o_x=\cos {\theta }_5\sin {\theta }_1-\cos {\theta }_{234}\cos {\theta }_1\cos {\theta }_5$$

(6)

$$o_y=\cos {\theta }_1\cos {\theta }_5-\cos \left({\theta }_2+{\theta }_3+{\theta }_4\right)\sin {\theta }_1\sin {\theta }_5$$

(7)

$$o_z=-\sin {\theta }_{234}\sin {\theta }_5$$

(8)

$$a_x=\sin {\theta }_{234}\cos {\theta }_1$$

(9)

$$a_y=\sin {\theta }_{234}\sin {\theta }_1$$

(10)

$$a_z=-\cos {\theta }_{234}$$

(11)

$$P_x=a_4\cos {\theta }_1\cos {\theta }_{23}+a_3\cos {\theta }_2+d_5\sin {\theta }_{234}$$

(12)

$$P_y=a_4\sin {\theta }_1\cos {\theta }_{23}+a_3\cos {\theta }_2+d_5\sin {\theta }_{234}$$

(13)

$$P_z=d_1+a_4\sin {\theta }_{23}+a_3\sin {\theta }_2-d_5\cos {\theta }_{234}$$

(14)

$${\theta }_1=\arctan 2\left(y,x\right)$$

(15)

$${\theta }_2=\arctan 2\left(s_2,c_2\right)$$

(16)

$$c_2=\frac{c_{1}x+s_{1}y+s_{234}{d}_5}{c_3{a}_4+a_3}$$

(17)

$$s_2=\frac{z-d_1-c_{234}{d}_5}{s_3{a}_4+a_3}$$

(18)

$${\theta }_3=\arctan 2\left(s_3,c_3\right)$$

(19)

$$c_3=\frac{{\left(c_{1}x+s_{1}y+s_{234}{d}_5\right)}^2+{\left(z-d_1-c_{234}{d}_5\right)}^2-a_3{a}_4}{2c_3{a}_4}$$

(20)

$$s_3=\pm \sqrt{1-c_3{}^2}$$

(21)

$${\theta }_4=\arctan 2\left(s_4,c_4\right)$$

(22)

$$c_4=-\left(s_{23}{c}_1{a}_x+s_{123}{a}_y-c_{23}{a}_z\right)$$

(23)

$$s_4=-\left(c_{123}{a}_x+c_{23}{s}_1{a}_y+s_{23}{a}_z\right)$$

(24)

$${\theta }_5=\arctan 2\left(s_5,c_5\right)$$

(25)

$$c_5=s_1{o}_x-c_1{o}_y$$

(26)

$$s_5=s_1{n}_x-c_1{n}_y$$

(27)

$$c_{234}=a_z$$

(28)

$$s_{234}=-\left(c_1{a}_1+s_1{a}_y\right)$$

(29)

$${\theta }_{234}=\arctan 2\left(s_{234},c_{234}\right)$$

(30)

3. Results and Discussion

3.1. Robotic Control

In this study, a simple graphical interface program was designed for the numerical simulation of motion of the robot arm, as shown in Figure 5. The motion of the robotic arm could be simulated by entering the coordinate points of the end actuator or the rotation angle of each axis joint. Furthermore, the end actuator of the robot arm (without the pick-up of any one object) was simulated to move along rectangular coordinates X (−100~100), Y (150~250) and Z fixed to 220. The calculated and measured results of the rectangular motion of the robotic arm are shown in Figure 6.

It should be noted that the actual position was determined using the position encoder of the servo motor as the robotic arm moved over time. The angles measured using the encoders were compared to the theoretical angles from the simulation calculation, where the angle of each axis was converted to the corresponding actuator coordinates using forward kinematics, as mentioned earlier. The aim was to verify that the forward kinematics and inverse kinematics were correctly presented in the study.

In Figure 6, the solid red line represents the planned path from the numerical calculation by the computer program, while the solid blue circular dots stand for the measured coordinates of the robot arm end actuator in actual motion. The average errors of the tri-axial coordinates were determined as X = 2.49%, Y = 0.14%, and Z = 0.28%.

Similarly, the simulated and measured results of axis angles for all the servo motors are shown in Figure 7. It can be found that the angle deviation of the third axis is more severe than that of the other axes because the third axis is influenced by the weight of the robotic arm under the load of the servo motor. The average angle errors are 11.11% for the 0-th axis, 14.39% for first axis, 4.85% for the second axis, and 4.90% for the third axis. Tests were also carried out with the gripper carrying an object over the same path. The simulated and measured results are shown in Figure 8. It can be seen that the position of the end-actuator offset path was increased when an object was being carried by the gripper. The average tri-axial coordinates errors are X = 2.49%, Y = 0.14%, and Z = 0.28%.

Figure 9 shows the simulated and measured results of axis angles for all the servo motors. It was found that the actual angles of each axis drifted over time and this may have been due to noisy data from the servo motor position encoders. It was also found that the angular error of the robotic arm changed with different payloads.

The errors between the theoretical and measured angles for the 2nd and 3rd axes became larger with an increased payload. In the experiments, the average angle error for the 0-th axis was found to be 9.82%, the others were 17.41% for the first axis, 4.92% for the second axis, and 5.95% for the third axis.

3.2. Clamping Force and Barcode Identification

The force sensor (sensitivity of 0~111 N, Tekscan FlexiForce A301) installed on the gripper was tested to show the relationship between detected force and electrical resistance and conductivity. The experimental response of the sensor to an applied force in a range of 5~50 N is shown in Figure 10. An approximately linear relationship of the electrical conductance response of the sensor allows it to be accurately calibrated and used to measure the clamping force of the fingers of the gripper.

A regular web camera was used for one-dimensional barcode identification. Some of the EAN barcodes used by most retailers everywhere are shown in Figure 11, illustrating the algorithm of bar code recognition [21,22]. First, A in the figure is the original image captured by the camera, and the identified barcode is outlined by a green rectangle via comparison to preset thresholding values for edge detection; Secondly, B is a transformed binary image in grayscale from original color one, in which the white area represents the position of the barcode; Thirdly, C is the identified barcode ready for dimension reading of binary black and white stripes based on the EAN code; Finally, D shows the content transformation of the barcode displayed in the command bar, successively showing the decoded 13 digital numbers. The experimental results showed that the minimum proportion of 50% of a barcode could be correctly captured and identified at a maximum distance of 19 cm, thereby reaching a success rate of approximately 100% under normal test circumstances.

3.3. AGraphic User Interface (GUI) with an Embedded Computer

The graphics user interface (GUI) of the APP that enables the robot arm to perform computer vision bar code identification and goods inventory is shown in Figure 12. As mentioned earlier, the web camera captures the barcode image and uses the data to find the corresponding information about the object to determine if it was correctly placed. If the placement was wrong, the robot would pick up and remove the object. Please note, thus far, the position of object was pre-determined by the user because of no distance sensor was used here. This could be further implemented in the future. Finally, the overall GUI system was integrated into an embedded computer (Raspberry Pi) with a 10-inch touch screen as shown in Figure 13. Intuitive control of the robotic arm could be achieved by simply touching the command buttons.

4. Conclusions

In this study, we designed and fabricated a 6-axis robotic arm with a gripper for picking and placing products on shelves for inventory control. The gripper had real-time force sensors that controlled the different clamping force that was required for each kind of product to avoid damage by crushing. The mechanical design ensured that the force was applied uniformly to the sensor to avoid the chance of inaccurate measurement. Inverse kinematics coordinate conversion was implemented which showed an error of less than 3.3% caused by insufficient torque servomotor actuation. Camera vision allowed the simple operation of the robot using a graphic user interface (GUI), and product recognition from the barcode was straightforward. We anticipate that this kind of robot will play an important role in commodity management and the inventory of goods in supermarkets and retail stores.