A Novel Disassembly Strategy of Hexagonal Screws Based on Robot Vision and Robot-Tool Cooperated Motion

Abstract

Disassembly plays an important role in the production process. Screw automatic unfastening guided by a robot has been widely used in the fields of industrial manufacturing and maintenance. Different from the previous studies that have used a flexible effector and expensive sensors, this paper presents a novel unfastening strategy based on robot vision for a hexagonal screw with an arbitrary loose state. In a robotic system, an industrial camera and a servo unfastening tool are installed at a robotic end-effector. The main contributions of this work are as follows. A camera pose adjustment method is proposed to obtain high-quality images of a target screw. The hexagonal screw pose calculation method based on the geometric analysis is developed to complete the screw–tool engagement. The cooperated motion of a robot and an unfastening tool is planned for the screw unfastening action. The effectiveness of the proposed control strategy is verified by experiments, and the influence of the motion speed on the unfastening quality is analyzed using the torque data collected by the unfastening tool. The analysis results can provide a significant foundation for the motion parameter selection in the proposed strategy.

1. Introduction

In recent years, robots have been increasingly applied to the field of automated manufacturing, thus improving both working efficiency and quality. In the assembly process of mechanical manufacturing, threaded assembly is one of the most common methods [1]. In particular, the screw-unfastening process is always performed when workpieces require maintenance to extend the product life through re-manufacturing effectively [2]. In modern industry, screw fastening and unfastening tasks have been typically completed in three ways: through manual operation using a wrench or screwdriver, as shown in Figure 1a; through automatic operation using a Cartesian robot, as shown in Figure 1b; and through automatic operation using an articulated robot, as shown in Figure 1c. Disassembly tasks in complex three-dimensional (3D) structured and unstructured environments have been mostly completed by humans. As humans perceive information on environmental parameters and screw pose mainly by eyes, the peg-in-hole, fastening, and unfastening states are typically analyzed through a sense of touch by hands [3,4]. Although humans can perform the loosening of screws under different complex states, this approach has the disadvantages of low efficiency, massive labor, and difficult control of the disassembly effect. To address these problems, automatic operation methods have been developed. Compared to a Cartesian robot, an articulated robot is more applicable to complex 3D environments, for instance, for different loose states located on the same surface and screws located on different surfaces, as shown in Figure 2. A force or torque sensor has been commonly used to perceive interaction information during the operation. Moreover, to improve the automation level, many studies have adopted control methods of automatic peg-in-hole, fastening, and unfastening actions guided by industrial robots. In the following, some of the recent representative studies are presented to show the contributions of this work.

To address the peg-in-hole problem, some studies have installed passive compliance devices at the end effector of a robot to perform the action [5], thus compensating for the location error between the peg and hole. However, the complaint mechanism has a complex, difficult, and high-cost design and cannot be used for the precise control of the force and torque due to high structural flexibility, which limits screw assembly operation. A number of studies used robotic vision sensors to measure relative posture between the peg and hole or other workpieces in a 3D environment. Suszyński [6] presented an industrial robotic system of industrial robot equipped with a triangulation scanner in the assembly process. And accurate positioning of the elements in 3D was recognized based on point clouds. Wang [7] proposed a robotic assembly system under the guidance of the 3D models over the internet, which used a monocular camera to collect a sequence of images in different poses. Yan [8] developed a robot assembly system with two manipulators and a structured light camera. The component pose at the end of a manipulator is calculated in 3D space. The peg-in-hole action is performed using a robotic rigid end effector based on geometric and physical models, as well as the position or force feedback [9,10]. In addition to the peg-in-hole problem, the control of fastening and unfastening operations represents the core limitation to the successful and high-quality system operation. Certain studies have solved this problem using a combination of vision and force and torque sensors, obtaining a good effect in the screw-fastening tasks [11,12].

Compared with the screw-fastening process, a robotic vision-based screw-unfastening process requires measuring the screw pose with a hexagonal shape, not a circular hole. The measurement accuracy is crucial for the success and quality of the unfastening process. Additionally, automatic processes of the assembly and disassembly actions are not completely the opposite. Jia [13] and Jiang [14] analyzed and reviewed a large number of automatic fastening actions, which cannot be directly conducted or adjusted. Some of the related studies focused on operational tools, which are different for the fastening and unfastening actions. Seliger [15] proposed a detachable tool consisting of a modified drill with a movable needle, which can be used to grab the top of fasteners. Chen [16] designed an unfastening tool that can be fixed on the end of a robot and applied to fasteners of the same shape. There are four main methods for screw unfastening, including human operation, artificial teaching control, human–robot cooperation, and sensor-based intelligent control methods. For instance, Vongbunyong [17] proposed a human dismantling platform, which can assist a computer in learning the dismantling process of an LCD display rear cover, including tool locations and the key process. Bdiwi [18] completed the repeated disassembly automation operation of motor bolts by robots using the artificial control-based method. Liu [19] built a hybrid human–robot cooperation unit to complete the automatic disassembly work for the Diaphragm coupling structure. DiFilippo [20] and Schumacher [21] developed a robotic system based on a Kinect sensor, which has three motion axes, and applied it to the notebook disassembly. Experimental results [20] have shown that the success rate of the assembly work guided by this system can reach 96.5 % (251/260). Actually, robots with three motion axes have been widely used in the automated assembly and disassembly tasks in the field of electronic assembly. However, their application is limited for workpieces with high requirements for size and shape requirements, as well as for that with screws located on multiple planes. Li [22] proposed a spiral method for a robot to search for screws and adjust the tool–screw engagement based on sensing data of a force or torque sensor, as well as completing the screw unfastening operation through human–robot cooperation (HRC). Still, the working efficiency and success rate of this method cannot meet practical requirements for key components in the field of aerospace and rail transit.

In summary, the existing methods have been limited in screw removal in complex 3D environments. To address this limitation, this work presents a low-cost and high-adaptability disassembly strategy based on robot vision for a robot with six degrees of freedom (DOFs) equipped with a special screw tightening tool at its end effector, as shown in Figure 3. The comparison of representative recently-proposed methods and the strategy proposed in this study is given in

Table 1. Comparison between the previously reported methods and the proposed method.

Method	Sensor	DOFs	Control Type	Scene Adaptability	Intelligence	Disassembly Quality
Vongbunyong [17]	No	-	Manpower	Any scene	Low	Not considered
Bdiwi [18]	No	6	Robot teaching	Low	Low	Not considered
Liu [19]	No	6	HRC	Low	Low	Not considered
DiFilippo [20]	Camera	3	Automation	2D scene	High	Not considered
Li [22]	F/T sensor	6	HRC	Flat surfaces	High	Considered
This study	Camera	6	Automation	3D scene	High	Considered

.

The main contributions of this work can be summarized as follows:

(1) A camera pose adjustment method is proposed for obtaining high-quality images of a target screw. In this method, the camera orientation is adjusted based on the geometric feature analysis results of the characteristic circle, aiming to make the image plane parallel to the screw head. Meanwhile, the camera position is adjusted based on the rough location of a screw relative to the image center, aiming to make the screw located approximately in the center of the image plane. This can help to improve the measurement accuracy of the screw pose.

(2) For the screw–tool engagement, a calculation method is developed to determine the hexagonal screw pose rapidly and accurately based on image information. In this method, 3D locations of hexagonal vertexes can be obtained through an accurate description of screw edges. Compared with the machine-learning-based method [23], this method reduces the location uncertainty caused by sample training and meets the application requirements of ordinary bolts, which can help to increase the success rate of the screw-unfastening operation significantly.

(3) An unfastening control method is proposed for the cooperated motion of an unfastening tool and a robot to ensure the disassembly quality. Additionally, variation curves of the torque under different motion speeds are analyzed based on sensing data of the unfastening tool, which provides a significant data foundation for the value selection of motion parameters.

(4) A strategy based on vision feedback and methods of the imaging process and coordinated motion control of a robotic system is proposed. Compared with the methods based on force and torque information, it largely reduces the operating cost of a system, as well as the data size and processing complexity.

From the main contributions of this work, the novelty lies in new methods proposed for screw–tool engagement and unfastening action. The new method for screw–tool engagement is composed of a camera pose adjustment method and a screw pose calculation method, which are proposed to improve the engagement success rate and reduce the system cost. The unfastening action is realized by the cooperated control of the axial motion of the robotic end effector and the rotational motion of the unfastening tool with proper speeds, which can be taken according to their influence analysis on the disassembly quality. Based on the above methods, an automatic disassembly strategy of hexagonal screws based on robot vision is finally obtained. Compared with the methods based on force and torque information, it largely reduces the system cost since there is no need for high-cost sensors, such as a six-axis wrist force sensor, as well as the data size and computing complexity.

The rest of this article is organized as follows. Section 2 introduces a disassembly strategy based on robot vision, including four steps—camera pose adjustment, screw–tool engagement, hexagonal screw unfastening, and moving screw away from a workpiece—and describes the hexagon screw pose calculation based on the camera pose adjustment, as well as the geometry analysis. Section 3 presents the experiments conducted to verify the effectiveness of the proposed control strategy and discusses the experimental results in terms of the screw–tool engagement. Section 4 gives the main conclusions and future work directions.

2. Proposed Screw Unfastening Strategy

As introduced above, this study presents an effective screw unfastening system for a six-DOFs robot equipped with an end effector composed of a special screw-tightening tool and a camera, as shown in Figure 4. In this system, a servo unfastening tool consists of a motor, a reducer, inner displacement and torque sensors, and an output rotation shaft. There is a regular hexagonal groove at the end of the rotation shaft, having a slightly larger size than the target screw head and an elastic mechanism, as shown in Figure 5. To realize the disassembly operation, the tool is mounted at the end of the robot so that it can guide it to the target screw. It has its control system, which communicates with the robot through the Beckhoff module and exchanges messages with a PC through the TCP/IP protocol. During the operation, a signal for performing the unfastening operation is sent to the unfastening tool when a robot arrives at the desired location. Namely, the screw-unfastening operation is performed through the cooperated motion of the tool and robot. The signals for stopping motions are sent to the tool and robot at the same time. Finally, the robot finishes the disassembly operation and prepares for the next unfastening task.

In the proposed strategy, the unfastening operation of a hexagonal screw is divided into four steps, as shown in Figure 6. The first step is to adjust the pose of an end-effector camera to obtain high-quality images of a hexagonal screw, which is necessary for the accurate calculation of its pose. In this step, the robotic pose relative to the workpiece is first measured through a characteristic circle to determine the start and target poses of the robotic end effector. Then, based on the measurement result, the robotic motion is planned for the subsequent process of camera pose adjustment. In this study, the robot’s target pose is calculated to ensure that the end effector is parallel and stays at a certain distance above the screw, which is helpful to improve the screw image quality for the second step.

The second step is to calculate the screw pose and then move the robotic end effector forward and lock the screw, which is a pre-action for the third step. In this step, a hexagon screw’s pose in the robotic base frame is calculated based on the geometry analysis results of a hexagonal shape. The calculated pose is used to plan the robotic motion in the locking action of a screw by the end-effector tool. This step makes the proposed strategy adaptable to the disassembly operation of screws with different loose states by providing the correct feedback on the target screw’s pose for the robotic motion.

The third step is to unfasten a screw through the cooperated motion of the robot and unfastening tool. In this step, the disassembly quality can be improved by selecting proper values of the motion parameters, which can be obtained by the analysis of the torque variation over time during the unfastening action.

The fourth step is to move a screw away from the workpiece. In this step, the robot continues its motion along a planned trajectory; the tool stops its rotating motion but maintains the normal force so that the screw does not fall. Finally, the robot puts the screw into a container and proceeds to the next unfastening operation.

The proposed strategy aims to improve the efficiency and quality of screw disassembly, as well as the adaptability of a robotic system at a reduced cost. Particularly, the screw pose calculation method based on a vision system is proposed to provide accurate information for the one-time engagement action of a tool and a hexagonal screw. The engagement duration has a major impact on the bolt disassembly efficiency. In the previously reported methods [24], the screw–tool engagement has been achieved through the location adjustment using a vision sensor and the orientation search using a force sensor. Compared with the previous methods, the proposed vision-based method can avoid repeated trial-and-error actions during the orientation search, thus providing higher operational efficiency.

Except for affecting the efficiency, the reduced disassembly quality may damage the thread and make it impossible to reuse, which can significantly increase the manufacturing cost. Therefore, in this study, the disassembly quality is improved by setting proper values of motion parameters of the robot and unfastening tool, which are obtained through the torque variation analysis over time. The torque variation analysis results indicate that the motion parameters’ values depend on the trade-off between efficiency and quality. Therefore, the torque monitoring aims to determine the proper values of motion parameters during the process planning stage, which is not a necessary step for all disassembly actions, so the actual disassembly speed could be significantly increased. However, in the previous studies, torque monitoring was used for the orientation search for the screw–tool engagement. For instance, Li [22] proposed a hexagon screw spiral searching method for a robot, which monitors the feedback on force or torque to realize the screw–tool engagement, as shown in Figure 7.

In addition to the efficiency and quality, applicability is also an important feature of a disassembly technique. Obviously, the vision-torque-based methods are limited when there exist obstacles between target screws, as shown in Figure 8a. This makes it impossible for a robot to complete the spiral motion smoothly and also makes it less efficient for screws with different loose states because more time is needed to search a screw along the axial direction, as shown in Figure 8b. In addition, some of the latest studies have used vision-based methods to perform the disassembly task using a Cartesian robot when screws are located on a plane [21,25]. Therefore, the existing studies have certain limitations in complex environments. To extend the applicability, this work uses a vision system to determine the 3D pose of a target screw accurately [26]. The identification and positioning of target can be realized by global recognition and positioning using binocular camera system. The screws were searched and matched by the Gaussian pyramid hierarchical method and greedy algorithm, and the optimal matching results were obtained based on vision calibration and picture correction. However, the pose of the screws obtained by this method [23] are not accurate enough, so the strategy proposed in this paper is needed to accurately obtain the pose of screws. Then, the system performs the engagement and unfastening actions based on the calculated pose using a robot, which can effectively avoid problems existing in the previous studies.

For the above steps, the motion planning of a robot is a significant task, which has been presented in the previous research on obstacle avoidance and other motion performance constraints [27]. Moreover, the camera pose adjustment, screw–tool engagement, and screw unfastening are the main problems considered in this study. The advanced methods proposed to address the three listed problems are introduced in the following sections.

2.1. Robotic Motion-Guided Camera Pose Adjustment Method

This work aims to develop an effective vision-based disassembly strategy for screws located in 3D complex structures, including different working planes. The measurement accuracy of a vision system has a major impact on the effectiveness and efficiency of the proposed strategy. This section proposes a camera pose-adjustment method guided by robotic motion, aiming to improve the image quality of a screw. To solve this problem, a characteristic circle needs to be placed on the structure to be disassembled. The image processing of the characteristic circle aims to realize the camera pose adjustment to obtain high-quality images of the target screw, which has a significant impact on the calculation accuracy of a screw pose. In the proposed strategy, there is one characteristic circle on each working plane, and screws on the same plane can be disassembled.

The camera pose adjustment is completed by three steps: the initial camera view adjustment, the orientation adjustment and the position adjustment. These are presented as follows in detail.

(1) Initial camera view field adjustment: The camera view is firstly adjusted by regular searching of the characteristic circle through the robotic motion based on the global recognition and positioning, aiming to make a screw and the characteristic circle located in the same image. According to the imaging principle, the object distance is positively related to the camera view field but negatively to the area covered by the target in an image, which means that the object distance adjustment should consider both the camera view field and the image quality. In this study, the general adjustment is performed by the robotic motion based on the above principle, which makes the characteristic circle and the target screw be located in the same image and their image area as large as possible.

Generally, when the distance between the characteristic circle and a screw is very long, the image quality will be greatly affected, which will reduce the location accuracy of a screw and can result in a failure of the engagement of the screw and unfastening tool. To analyze the allowable distance range of the proposed robotic system, experimental tests were performed. In these tests, the distance between the hole and a screw was set to be in a range from 20 mm to 300 mm. Testing results are given in

Table 2. Test results of the distance: $L_{hts}$ denotes the distance between a hole and a screw, $Rati{o}_A$ denotes the percentage of the image area of a screw, and $Erro{r}_s$ means the positional error of the screw.

Test Number	$L_{\mathit{hts}}$ (mm)	${\mathit{Ratio}}_A$ (%)	${\mathit{Error}}_s$ (mm)
1	20	12.14	0.073
2	40	7.22	0.138
3	80	2.86	0.29
4	120	1.81	0.508
5	150	0.99	0.636
6	200	0.72	0.857
7	250	0.49	0.954
8	300	0.28	1.115

, from which it can be seen that the accuracy was greatly affected by the distance $L_{hts}$ and area percentage $Rati{o}_A$. Therefore, the strategy is only applicable to cases when the distance is less than a certain value, which is defined by a specific requirement for the measurement accuracy.

For realizing the unfastening action successfully, the sizes of the screw and tool end should satisfy the condition of $d_1<d_2$, as shown in Figure 9. Accordingly, the minimal gap g is limited by the following condition,

$$g<\left(e-s\right)/2$$

(1)

where values of two parameters e and s are defined by the nut size specification standard.

In this study, the target screws’ parameters are as follows: $e=17.77$ mm and $s=16$ mm. The minimal gap satisfies the condition of $g<0.885$ mm according to the standard GB/T5783-2000. In the proposed robotic system, $g=0.88$ mm, which indicates that the location error cannot exceed the minimal gap of $0.88$ mm. Therefore, aiming to improve the reliability of the proposed strategy, based on the test results in Table 2, the distance between the hole and a screw cannot be larger than 200 mm. For most industrial robots, such as the KUKA robot used in this work, their positioning accuracy is always high enough considering structural flexibility and control resolution, and can reach ±0.02 mm to ±0.05 mm. The accuracy of the positioning of the robot is far less than the minimal gap when performing screw unfastening. Therefore, the robot positioning accuracy has little influence on screw–tool engagement.

After achieving the camera view field adjustment, the characteristic circle and a screw are located in the same camera view field, as shown in Figure 10. This adjustment can help the subsequent camera pose adjustment.

(2) Camera orientation adjustment: During the camera pose adjustment step, the recognition and positioning of the characteristic circle and a target screw are completed in the image coordinate system first, as shown in Figure 10. The camera pose can be adjusted by the end-effector motion guided by a robot. For accurate adjustment, the motion of the robotic end effector is divided into three rotational motions along the A-axis, B-axis, and C-axis, as shown in Figure 4.

Particularly, when an image of the characteristic circle is taken by a camera under a certain imaging angle $\theta$, the circle edge will be an ellipse curve, as shown in the left scheme in Figure 11. However, when $\theta =0$, the edge of the circle will be a circle, as shown in the right scheme in Figure 11. Therefore, this parameter is used for the camera orientation adjustment based on the robotic motion. In the proposed strategy, the rotational motions along the B-axis and C-axis of a robotic tool are planned to complete this adjustment by establishing the radial basis forward network (RBF) neural network [28]. The RBF neural network has strong nonlinear mapping ability, fast computing speed, and good prediction ability. The RBF network is different from other neural networks, which has only one hidden layer [28,29]. The network is composed of an input layer, hidden layer and output layer, as shown in Figure 12. The activation function of the hidden layer is a Gaussian function. The specific form of RBF neural network algorithm is

$${\phi }_j=exp\left(\frac{\parallel x-c_j{\parallel }^2}{2{\sigma }_j^2}\right)$$

(2)

$$y=\sum _{j=1}^K{\phi }_j{w}_j$$

(3)

where x and ${\phi }_j$ denote the input of the network and the output of the hidden layer. $c_j$ and ${\sigma }_j$ mean the Gaussian basis center, the base width parameter of the hidden layer node. y and $w_j$ represent the output of the network and the connection weight between the $j_{th}$ neuron of the hidden layer and the network output. The good convergence of the results is the premise of the neural network. Therefore, the gradient descent method is used to adjust the center, width and weight of the RBF network so as to speed up its convergence.

The serial number, major axis, and minor axis of characteristic circle are considered the input, and rotational angles $\beta$ and $\gamma$ are considered the output. In this paper, the number of hidden layers is one, and the final number of neurons is three. The experimental data (

Table 3. Parameters of sample.

Serial Number	${\mathit{D}}_{\mathit{c}}$	D	$\mathit{\beta }$	$\mathit{\gamma }$
1	721.777	628.073	−5	−28
1	693.03	618.731	−2	−26
1	395.214	383.869	3	22
2	470.899	468.667	−4	5
2	503.24	501.746	−2	−2
⋯

) used in this paper are from the real screw removal platform. In the data sample, 855 groups were randomly selected, 755 groups were used as the training set, and the remaining 100 groups were used as the prediction set. After network training and testing, linear regression analysis is conducted between the desired target and the actual output of the network model. The weight from the input layer to the hidden layer is $net.I{W}_{3\times 3}=[1,0.924,0.94;0.6,0.901,0.962;0.1,0.873,0.963]$. The weight from the hidden layer to the output layer is $net.L{W}_{2\times 3}=[0.127,-0.156,0.3;1.559,-2.812,1.251]\times {10}^5$. As shown in Figure 13a,b, the regression coefficient between the expected output and the actual output is very close to 1, which indicates that the network has a good prediction effect. Figure 13c,d show the comparison between the predicted results and the experimental results. The maximum relative errors of $\beta$ and $\gamma$ are 0.2% and 1.7%, the average absolute errors are 0.02 and 0.31. Based on the obtained neural network, values of $\beta$ and $\gamma$ can be predicted. Then, the camera orientation adjustment can be completed.

(3) Camera position adjustment: A rough position of a screw relative to the characteristic circle is determined to adjust the camera position relative to the screw. In this step, real distances $X_{cts}$ and $Y_{cts}$ of the end effector travelled in two axial directions of an image can be obtained based on the corresponding image distance and the pixel physical size $d_p$, as follows $X_{cts}=d_p{x}_{cts}$ and $Y_{cts}=d_p{y}_{cts}$. The current object distance $dis$ is calculated based on the ratio of the real size to the image size of the characteristic circle. Then, this distance is used to adjust the camera shooting distance $di{s}_s$ of a screw to achieve a high-quality image, which can be realized by moving the robotic end effector along the direction of the camera optical axis by a distance of $DIS=dis-di{s}_s$. Based on the above adjustments, a screw can appear in the image field of a camera completely and clearly, as shown in Figure 14, which can help the screw location calculation in the next step.

The robotic motions needed for the camera orientation and position adjustments can be obtained based on the elliptical feature of the characteristic circle image and the rough position of a screw relative to the characteristic circle in the image coordinate system. Therefore, a 3D model can help to locate a screw that is difficult to appear in the same camera view with the characteristic circle but is not necessary for all screw disassembly tasks. In the experimental tests of this study, all adjustment motions are performed based on the image processing of an image that includes both the characteristic circle and the target screw.

2.2. Screw–Tool Engagement Method Based on Screw Pose Calculation

The screw–tool engagement includes two steps—the screw pose calculation and the engagement motion guided by a robot—which are presented in detail in the following.

(1) Screw pose calculation: The screw pose is calculated based on feature points of its pixel image, which can be extracted and located using the HALCON [30]. The pose calculation accuracy directly determines the success rate and efficiency of the engagement operation.

In this work, a Canny filter [31] is used to extract the hexagonal edge of a target screw from the whole image. For this algorithm, the filter value, which is in the range of 1∼200, has a crucial impact on the effect of the image noise removal and affects the accuracy of the edge extraction. In this work, the artificial experience is obtained from the previous image processing tests, and it can be described as follows: the more elements in an image exist, the larger the filter value should be, and the more calculations are required. In the experimental tests of this work, there existed only screws and the background with a single feature, so the filter value was set to 20, which could ensure the edge detection accuracy at an acceptable computing cost.

It should be noted that a proper filter value is determined based on the artificial experience mainly because the structure is simple. However, in real applications, there always exist complex assembly structures with bolted connections, so an adaptive adjustment method for the filter value should be selected according to the structural features. Therefore, an adaptive image processing method of screws is further studied to improve the effectiveness of the proposed control strategy.

After the edge extraction, all edges $Edg{e}_i$ of an image can be extracted, and the longest among them ${\left(Edg{e}_i\right)}_{max}$ is selected as an edge of a hexagonal screw head. Edges of an image before and after the edge selection are shown in Figure 15. The length $L_{screw}$ of the target edge can be obtained as follows:

$$L_{screw}=max\{Length\left(Edg{e}_i\right),0<i\le I\}$$

(4)

where i denotes the number of edges extracted from the image, and $Length(·)$ represents the length calculation function of an edge.

The target edge is divided into several parts based on the point number of the edge contour. The maximum and minimum distances between the contour and the approximate line are set to 10 pixels and 5 pixels, respectively. As shown in Figure 16a, the set of lines and curves is used as an approximate contour, and this contour is further divided into sub-contours using the contour length as a feature. Finally, some of the curves are removed to make a screw have at least one line on each edge, setting the lower and upper limits of the contour length to 100 pixels and 1000 pixels, respectively. Hexagon edges of the screw head obtained by the presented procedures are shown in Figure 16b.

The obtained multiple sub-contours are linearly fitted using the ordinary least squares linear fitting operator of Tukey [32]. The abscissa $RowBegi{n}_i$ and ordinate $ColBegi{n}_i$ of the starting point, as well as the abscissa $RowEn{d}_i$ and ordinate $ColEn{d}_i$ of the end point, in the pixel coordinate system are obtained by applying the linear fitting operator. A line between a beginning point and the corresponding end point can be fitted using a linear equation, which can form six sides of a hexagon through extensions. The linear equation includes the following relations:

$$\left\{\begin{array}{c}{y}_i=k_i{x}_i+b_i\hfill \\ y_i=r_i/n_i\hfill \\ b_i=RowBegi{n}_i-k_{i}ColBegi{n}_i\hfill \end{array}\right.$$

(5)

$$\left\{\begin{array}{c}{r}_i=RowBegi{n}_i-RowEn{d}_i\hfill \\ n_i=ColBegi{n}_i-ColEn{d}_i\hfill \end{array}\right.$$

(6)

where $k_i$ and $b_i$ means function parameters of the i-th line.

As shown in Figure 17, an intersection $P_i$ and its 2D location denoted by $P_i\left(P_{ix},P_{iy}\right)$ can be obtained by Equations (3) and (4), based on which the hexagonal center $P_c\left(P_{cx},P_{cy}\right)$ can be determined as follows:

$$\left\{\begin{array}{c}\hfill P_{cx}=\sum P_{ix}/i\hfill \\ \hfill P_{cy}=\sum P_{iy}/i\hfill \end{array}\right.$$

(7)

Based on the above-presented processing and calculations, a 2D location of a screw $O_{xy}^s$ in the camera frame can be obtained. Relative relationships between the centers of the screw, tool frame, and camera frame are shown in Figure 18.

(2) Engagement motion guided by robot: In real situations, the bottom of loose screws is always not tightly close to the top surface of a workpiece, which means that it is difficult to estimate the distance between the bottom of the robotic tool and the top of the screw precisely based only on the above calculation equations. To solve this problem, an estimation method of distance $Z_s$ is proposed to avoid the damage of the end effector on a screw or workpiece. In the proposed method, the length of a hexagonal side is used to represent the screw size, and its value $S_{image}$ in the image plane can be calculated as an average length of six sides based on 2D locations of intersections $P_i\left(P_{ix},P_{iy}\right)$. Then, the distance $Z_s$ between the screw and the camera can be calculated according to the imaging principle of a monocular camera as follows:

$$Z_s=f·S_{real}/S_{image}$$

(8)

where $S_{real}$ and f denote the real length of one side and the camera focus, respectively.

Then, a 3D location of a screw $O_{xyz}^s=[O_{xy}^s,Z_s]$ in the camera frame can be obtained. Based on the conversion matrix ${\mathit{R}}_{scam}^{tool}$ obtained from the hand-eye calibration of the monocular camera [33], the screw position $O_{xyz}^t={\mathit{R}}_{scam}^{tool}·O_{xyz}^s$ in the tool frame can be calculated. Based on the screw position calculation result, the tool center can reach the position of the screw head. However, six vertexes $P_i$ of the screw cannot correspond to points $P_i^t$ of the tool, which is significant for the screw–tool engagement, as shown in Figure 19. To achieve a perfect engagement of the screw head and the tool, an angular offset $\theta$ between the coordinate systems of the screw and tool is introduced and calculated based on the directional vectors of $xoy$ and $x_{t}o{y}_t$, which can be determined by the location of the screw center and one intersection $P_i$, and by the forward kinematics of the robot, respectively. Then, the screw can engage with the tool by rotating $\theta$ along the tool axis. Based on the screw pose $O_{xyz}^t$ and the rotational angle $\theta$, joint angles $\mathit{JA}$ of a robot can be calculated based on its inverse kinematics model, which was presented in detail by Xu [34].

Finally, the following condition is used to determine if the screw–tool engagement is achieved:

$$S_{move}>S_{screw}^{tool}$$

(9)

where $S_{move}$ and $S_{screw}^{tool}$ denote the axial distance the tool travels and the distance between the top of the screw and the bottom of the tool end in the same direction, respectively.

2.3. Screw Unfastening Method Based on Cooperated Motion

As shown in Figure 20, the unfastening motion is realized through the cooperation of the rotational motion of the unfastening tool and the robotic motion along the axial direction of a screw. For the cooperated motion, the angular velocity w of the unfastening tool and the linear velocity $v_r$ of the robotic end effector should be set properly. If the value of $v_r$ is much higher than the normal speed $v_s$ of a screw, the screw will break away from the tool before the screw leaves the hole. In contrast, if the value of $v_r$ is too small, there can be two negative effects. First, the spinning of the hexagonal screw may seriously damage the hole. Second, the normal force F can be generated between the tool and the workpiece, which can severely damage the thread surface quality of a workpiece. In this work, values of w and $v_r$ are set based on Equation (8), which can help to improve the motion stability. Using Equation (8), the cooperated motion of a robot and the unfastening tool is planned based on the screw pitch, which is determined according to the national standard gb/t 5783-2000 and a specific type of screw. Therefore, in the proposed strategy, the screw type needs to be known in advance. Additionally, the proposed system is applicable only to screws with the same size $\left(M10\right)$ since the end size of the unfastening tool is fixed.

$$v_r=w/360\times P$$

(10)

In Equation (8), P denotes the pitch of a screw.

It should be noted that if a target structure includes different types of threaded components, disassembly can be realized using a combination of the two following techniques: fuzzy decision, which is introduced for the adaptive determination of the screw pitch according to the vision measurement result and applicable conditions, and an advanced unfastening tool, which is designed to be applicable to screws with a wide size range.

The rotational angle of the unfastening tool can be determined based on the distance from surface A to surface B, which can be estimated based on values of heights $Z_c$ and $Z_s$, as shown in Figure 18. The length $L_s$ of the engaged thread can be calculated by $L_s=L-\left(Z_c-Z_s\right)$, where L denotes the total length of a screw. To ensure the complete removal of the screw, the unfastening angle ${\theta }_t$ of the tool can be determined by

$${\theta }_t=L_s/P\times 360+{\theta }_s$$

(11)

where ${\theta }_s$ is the safety rotational angle of the unfastening tool, which represents the increased angle based on the calculated angle determined by the cooperated motion planning of the robot and the unfastening tool. The definition angle ${\theta }_s$ aims to ensure complete screw disassembly. In this work, the value of ${\theta }_s$ is set to 960 deg, which means the unfastening tool will rotate three more times after the cooperated motion. Based on the experimental test results, the assembly reliability can be improved based on this strategy.

The rotational orientation of the disassembly screw can be determined based on the screw thread rotational direction, including the left-handed and right-handed cases. In most applicable cases, screws with the same rotational direction are used. Most industrial applications use right-handed screws, which are also used in this work.

3. Experimental Results and Discussion

In this study, real experiments were designed and implemented to validate the effectiveness of the proposed strategy. In the following sections, the platform setup and the flowchart of the disassembly are presented, and the results are discussed in detail.

3.1. Experimental Platform Setup

The robotic system included a 6-DOF industrial robot (KUKA KR16 R1610), a servo screw unfastening tool (AND ENGINGEERING, ACT-64XX) for unfastening $M10$ screws, and a monocular camera (HIKVISION, MV-CE200-10GM), whose the internal parameters were as follows: the focus was 25 mm, $S_x$ was 2.4 μm, $S_y$ was 2.4 μm, and resolution was $5472\times 3648$). The repetitive positioning accuracy of the robot is ±0.04 mm (ISO 9283) [35]. The accuracy has little influence on screw–tool engagement. Therefore, in this work, the positioning accuracy of the robot is not taken into account during the engagement accuracy analysis.

A hexagonal screw removal platform was constructed, as shown in Figure 21. The workpiece to be disassembled denoted a T-shaped plate with seven $M10$ screw holes and a size of $700\times 450\times 300$ mm. Three screws were located on the horizontal plane with a spacing of 50 mm. The other screws were located on the vertical plane with a spacing of 50 mm. The three screws were set up, and distances from their bottoms to the workpiece surface were 15 mm (#1), 10 mm (#2), and 5 mm (#3).

In the proposed strategy, during the camera pose adjustment step, the characteristic circle was used to determine the camera orientation and position relative to a screw, which provided the necessary information for the pose adjustment whose purpose was to improve the screw image quality. This step aimed to increase the calculation accuracy of the screw pose. Therefore, the number of planes determined the number of characteristic circles needed for the pose adjustment. Namely, the proposed system only needed to have one single characteristic circle for each of the surfaces. Therefore, in the experimental platform, characteristic circles were located on the horizontal and vertical planes of the workpiece and were generally far away from obstacle surfaces of the workpiece to avoid collisions during the rotational and translational motions.

The screw-unfastening system used the Visual Studio Platform, and the program was written in the C# programming language.

3.2. Detailed Experimental Unfastening Procedure

The detailed flowchart of the proposed unfastening strategy is shown in Figure 22. In Figure 22, parameters are defined and calculated in the context of the motion control of a robotic system, which clearly shows the use of these parameters and the whole control process. In Figure 22, the left column shows real actions of a robotic system, including image collection using a camera and motions of the robot and unfastening tool. The real actions were controlled by the parameters obtained by computing modules of PC, which are presented in the right column in Figure 22. The modules in the right column denoted imaging processing in HALCON and parameter calculations programmed in the C# environment. The left column provided images as input of the right column.

For the proposed strategy, the working efficiency could be improved by setting the high motion speeds for the unfastening tool and the robotic end effector. However, high speeds could easily cause mechanical vibration, which could further severely affect the stability of thread engagement and reduce the thread quality of the workpiece during the unfastening operation. Therefore, in this experiment, the torque values obtained under different values of the rotational velocity w were collected by sensors mounted on the tool and used to analyze the effect of the motion speed on the unfastening quality.

3.3. Results and Discussion

To analyze the measurement accuracy of the experimental vision system, the screw–tool engagement based on a robot vision was performed 20 times in the same manner for each screw $\left(\#1,\#2and\#3\right)$ in a random position. The desired position $\left(x_{real},y_{real},z_{real}\right)$ and the angular offset ${\theta }_{real}$ of each screw were calibrated by the manually teaching of a robot and used to calculate the corresponding error between the calculated and desired values, i.e., $\left(x_{cal},y_{cal},z_{cal}\right)$ and $\theta cal$. The largest errors are listed in

Table 4. Errors between the calculated and desired screw poses.

Screws	$\mathsf{\Delta }x$ (mm)	$\mathsf{\Delta }y$ (mm)	$\mathsf{\Delta }z$ (mm)	$\mathsf{\Delta }$ (mm)	$\mathsf{\Delta }\theta$ (deg)
$\#1$	0.483	0.376	2.136	0.612	0.41
$\#2$	0.265	0.814	2.469	0.856	0.73
$\#3$	0.327	0.659	0.196	0.735	0.65

, where $\Delta x=\mid x_{real}-x_{cal}\mid$, $\Delta y=\mid y_{real}-y_{cal}\mid$, $\Delta z=\mid z_{real}-z_{cal}\mid$, $\Delta =\sqrt{\Delta x^2+\Delta y^2}$, and $\Delta \theta ={\theta }_{real}-{\theta }_{cal}$. Error $\Delta$ indicates a displacement offset between the calculated and desired positions of the screw center on the plane it was located on, which directly demonstrated whether the engagement succeed or not.

According to the obtained results, the largest value of the error $\Delta$ was 0.856 mm. It should be noted that there must exist the measurement error due to affecting factors such as uncertainties during the procedures of camera calibration and image processing. However, there was a gap of 0.88 mm between the bottom of the unfastening tool and the screw head, which could guarantee the next engagement and unfastening actions. In addition, the results in Table 4 show that errors in the normal direction were generally larger than in the other directions, which was mainly because the normal position acquisition based on robot vision did not completely rely on image processing, and also the object distance estimation of the monocular camera.

Variations in the torque of the three screws with the time under different settings are shown in Figure 23. The screws were disassembled at different rotating velocity values of 90 rpm, 120 rpm, 150 rpm, and 180 rpm. After engaging between the tool and the screw, it took 1.778 s, 1.418 s, 1.067 s, and 1.0 s to unfastening $\#1$ screw under the cooperation motion of the manipulator and tool. The curve of $\#1$ screw in Figure 23a shows that the torque firstly reached 4.9 N·m after the tool had started working and then rapidly decreased to approximately 2.4 mm/s when w = 90 rpm. The average unfastening torque increased from 2.44 mm/s to 3.27 mm/s when the rotating speed of the tool increased from 90 rpm to 180 rpm.

For the curves of $\#2$ and $\#3$ screws, variations in the torque showed a similar trend. Since the wear of holes and screws differed, curves obtained under three cases were not identical.

The thread contact between the screw and the hole was intensified due to higher motion speeds, so a larger torque was generated. Also, a greater vibration was generated under the higher speeds, which could cause higher instability of the torque. The standard deviation was used to evaluate torque fluctuations. As shown Figure 24, the standard deviation of the torque of the three screws increased with the rotating speed. For instance, the standard deviation of $\#1$, $\#2$, and $\#3$ screws increase by 20.49%, 30.37%, and 25.85%, respectively, when the speed increase from 90 rpm to 180 rpm. The results indicated that the torque fluctuation had a negative effect on the thread quality of the workpiece, which is important for some reusable and long-serving workpieces. The results indicated that the torque fluctuation had a negative effect on the thread quality of the workpiece, which is important for some reusable and long-serving workpieces. According to the results, the motion speed of the unfastening work can be determined considering the actual requirements for the working efficiency and unfastening quality, and the variation curve of the torque can provide a useful data foundation for the value selection of the motion parameter.

4. Conclusions and Future Works

(1) Conclusions: This work proposes an automated disassembly strategy based on robot vision for a hexagonal screw with arbitrary loose states. The robotic system is composed of a 6-DOF robot, a monocular camera, and a servo unfastening tool. In the proposed strategy, advanced methods are employed to complete tasks of the camera pose adjustment, screw–tool engagement, and screw unfastening, which increase system adaptation and reduce the hardware and software costs. Finally, real experiments are performed to verify the effectiveness of the proposed strategy. Based on the experimental results, the following conclusions can be drawn:

(1) The screw–tool engagement is achieved using only a vision system, which can significantly reduce the system cost. However, achieving a successful engagement requires high certain measurement accuracy of the vision system. Through the experiment, the largest positioning error on the plane where the screw is located is calculated as 0.856 mm, which is less than the preset gap between the screw head and tool sleeve. This result indicates that the measurement accuracy is high enough to ensure successful screw disassembly.

(2) The unfastening operations of the three tested screws are successfully completed through the cooperation motion of the manipulator and unfastening tool. The experimental results show that the proposed strategy can be used to perform the screw disassembly work effectively and reliably.

(3) Variations in the torque values of the three screws under different settings are similar, which indicates the unfastening process at the micro level. Results show that the torque and its standard deviation increase with the rotating speed, which demonstrates that the motion speed of the unfastening work can be determined considering both the actual requirements of the working efficiency and the unfastening quality. Moreover, the variation curve of the torque can provide a data foundation for the value selection of the motion parameter.

(2) Limitations and future works: This work mainly aims to develop a whole control strategy based on a vision system for screw disassembly, including the camera pose adjustment strategy for high-quality images of a screw, the screw–tool engagement strategy, and the unfastening strategy with properly set motion parameters. However, there exist cases with a poor contract between the target screw and the background. In these types of cases, the effectiveness of the proposed strategy will be reduced since the accuracy and robustness of the screw edge detection can be significantly affected by a poor contract. In this work, a screw is recognized based on the geometric features of a hexagonal shape of a screw head, which can be totally distinguished from other markings surrounding the screw but cannot effectively solve the above-mentioned problem. To overcome this problem, the machining learning-based algorithms can be used to improve the robustness of the screw edge detection under a poor contract, which could be costly during the training but may not increase the running cost during the screw disassembly. In future work, the machining learning-based algorithms will be introduced to improve the proposed strategy further, and more complex cases with high reliability will be discussed.