Manufacturing Technology on a Mechatronics Line Assisted by Autonomous Robotic Systems, Robotic Manipulators and Visual Servoing Systems ^†

Abstract

This paper proposes the implementation of an assisting technology to a processing/reprocessing mechatronics line (P/RML), comprising the following: two autonomous robotic systems (ARSs), two robotic manipulators (RMs) and three visual servoing systems (VSSs). The P/RML has four line-shaped workstations assisted by two ARSs—wheeled mobile robots (WMRs): one of them equipped with an RM, used for manipulation, and the other one used for transport. Two types of VSSs—eye to hand and eye in hand—are used as actuators for precise positioning of RMs to catch and release the work-piece. The work-piece visits stations successively as it is moved along the line for processing. If the processed piece does not pass the quality test, it is taken from the last stations of the P/RML and it is transported to the first station where it will be considered for reprocessing. The P/RML, assisted by ARSs, RMs and VSSs, was modelled with the synchronized hybrid Petri nets (SHPN). To control the ARSs, we propose the use of trajectory-tracking and sliding-mode control (TTSMC). The precise positioning that allows the picking up and releasing of the work-piece was performed using two types of VSSs. In the case of the first one, termed eye to hand VSS, the cameras have a fixed position, located at the last and the first workstations of the P/RML. For the second one, named eye in hand VSS, the camera is located at the end effector of the RM.

1. Introduction

Central to the idea of the paper is the overall proposed approach: a technology that works on a laboratory system and integrates several systems: processing/reprocessing mechatronics line (P/RML), autonomous robotic systems (ARSs), robotic manipulators (RMs) and visual servoing systems (VSSs). The technology involves several concepts: task planning, hybrid modelling, simulation, sensors, actuators, monitoring and real-time control. The technology allows the recovery of products through reprocessing and works completely automated without the intervention of the human operator. The main elements of originality and contributions are concentrated in the following areas: tasks assignment, planning and synchronization of P/RML assisted by ARSs, RMs and VSSs, hybrid modelling and simulation with synchronized hybrid Petri nets (SHPN), modelling and implementation eye to hand and eye in hand VSS and real-time control, so that the whole system becomes fully automated. Through this article, we extended the degree of automation and efficiency of these production lines by ARSs, RMs and VSSs [1,2,3,4,5].

A flexible manufacturing line represents all workstations and cells, actuators, acquisition equipment, ARSs, RMs, VSSs, transport, storage, monitoring and control systems, all of which are capable of performing tasks for processing operations, in a reconfigurable way that gives reversibility, repeatability and, last but not least, flexibility [4,6,7,8,9,10]. The general structure of a P/RML allows the highlighting of the system’s general functions: the function of automatic processing of parts or subassemblies; the function of automatic storage, transport and handling; the function of automatic control of all the components of the system and of supervision, control and automatic diagnosis—realized with the help of one or more PLCs in different configurations, centralized or distributed; the automatic processing function that is performed within the technological subsystem of the P/RML, comprising of the workstations (cells); the function of automatic storage, transport and handling. The processing/reprocessing (P/R) planning process involves more complex requirements such as geometric relationships, performance measurement and evaluation, resource scheduling, kinematics control, and system planning. P/R plans are operation sequences (drilling, reaming, extrusion, bending, etc.), executed in serial or parallel so that the product meets the quality requirements. To a technology on a P/RML assisted by ARSs, RMs and VSSs, a planning strategy oriented to the characteristics of the system is often more effective than techniques derived from domain-independent methods, [10,11]. If the fully processed product fails the quality test, the task planner provides the best sequence for reprocessing the product, [12,13].

The P/RMLs assisted by ARSs and RMs have hybrid characteristics, consisting of continuous dynamics and discrete events behaviours, [7,8]. Hybrid Petri nets (HPNs) are instruments used to model such complex systems [13,14,15,16,17]. Development of knowledge based on an HPN model integrated with a sequence generation algorithm was successfully applied to the modelling and planning of a flexible process and system at a high level, [18]. In this paper, a synchronized hybrid Petri nets (SHPN) model for technology on a P/RML assisted by ARSs, RMs and VSSs is presented. The aim is to assign the tasks to the P/RML workstations for reprocessing the work-piece that fails the quality test. The reprocessing operations are performed on the same line. The new technology concept of P/R is illustrated in an SHPN model, which complies with both aspects: the discrete approach P/R operations and the continuous approach for the displacement of the ARSs. The P/RML is assisted by two ARSs, two RMs, one of them mounted on an ARS, and three VSSs. The P/RML dynamics are determined by events, supplied by the control sequences of the automatic system, and by interaction with the ARSs, which represents the continuous-time part of the system. The considered system is a hybrid one and requires specialized tools for modelling. The hybrid model is elaborated using the dedicated modelling tool, HPN, described in [15,16,17]. An SHPN model results from the combination of the discrete-event model of the P/RML with the cyclic and continuous-time of the ARSs, [5].

Visual servoing systems are known as flexible and robust actuators used to enhance the accuracy and versatility of a control architecture based on visual feedback, [19]. The task of a servoing system is to control the movement of a robotic manipulator from an initial position to the desired configuration. The task of a VSS is defined in relation to how the robot can interact with the working environment using visual sensors. The visual features extracted from the images acquired using the visual sensor represent the inputs for the control architecture. Selecting visual features is a crucial point in a VSS, as these visual features are relevant to performance and accuracy. The minimum number of visual features used in the control architecture to control the movement of an RM depends on the number of degrees of freedom of the robot. Therefore, it is desirable to have a correlation between the visual features used and the movements of the visual sensor [5,6,7,8,9,19].

The rest of the paper is organised as follows: the description of the P/RML assisted by ARSs, RMs and VSSs with task planning useful preliminary assumptions, for developing SHPN model, are laid out in Section 2; model structure and SHPN model formalism, in generalised and customised forms, is presented in Section 3; simulation results are also presented for the customised SHPN model associated with P/RML, assisted by ARSs; modelling and control of the eye to hand and eye in hand VSS, based of the moment of the images, are presented in Section 4; in Section 5, the implementation and real-time control of the P/RML assisted by two ARSs, two RMs, two eye to hand and one eye in hand VSSs based on the SHPN model and signals from sensors are presented; comments on the robustness to the uncertainties of the proposed technology and the need for the use of VSSs can be found in Section 6; some final remarks can be found in Section 7.

2. P/RML Assisted by ARSs, RMs and VSSs

2.1. Hardware Description

P/RML FESTO MPS-200, shown in Figure 1, is a configurable laboratory mechatronic system, which can be assimilated to a flexible manufacturing line that performs several operations. It is composed of four workstations (cells), each performing different operations. These workstations are controlled by separate Siemens S7 300 PLC and each of them ensures the operations in different stages: buffer, handling, processing and sorting. ARS Pioneer 3-DX is used for the recovery operation which is a WMR with two driving wheels and one free wheel. ARS PeopleBot is used for the transport operation, which is similar to the WMR ARS Pioneer 3-DX. The control of the ARSs is carried out by a specialized application developed and executed on a remote PC that transmits commands via Wi-Fi Link, [20,21].

On the ARS Pioneer 3-DX, an RM Pioneer 5-DOF (degrees of freedom) is mounted, also controlled via Wi-Fi. For controlling ARSs and RM Pioneer 5-DOF are used dedicated functions from ARIA (Advanced Robotic Interface for Applications) package. In order to return the piece to the line is used 7-DOF manipulator, RM Cyton 1500 connected via USB with the process PC. In the synchronization of the P/RML with the ARSs, RMs are used as signals from sensors and high definition video cameras, two of them integrated as eye to hand and the third as eye in hand VSS. P/RML, ARSs, RMs, cameras and work-pieces are shown in Figure 1 and Figure 2.

2.2. Eye to Hand and Eye in Hand VSSs

The eye to hand VSS is defined by the mounting of the video sensor in a fixed position relative to the work environment, [8,9,19]. The eye in hand VSS refers to a system where the video sensor is located to the end effector of the RM, [8,9,19]. For the image-based VSSs, 2D image information is used directly to estimate the desired motion of the robot. Typical tasks such as tracking and positioning are accomplished by minimizing the error between the visual features extracted from the current image and the corresponding visual features of the desired image.

For image-based architectures, the visual sensor used to extract visual information about the work environment can be mounted either directly on the eye-in-hand effector of the robot, in which case the motion of the robot induces and the movement of the camera, or the camera can be mounted somewhere in the workspace (eye-to-hand) so that it can observe the robot’s movement from a fixed point. The general representations of the two architectures are presented in Figure 3. The most commonly used features in object classification and form recognition are the features called moments of the image which, because of their benefits, are also used for robotic and artificial intelligence purposes. Considering an image as a two-dimensional intensity distribution, the moments of the image contain information about the image area, the orientation of the image and the coordinates of the centre of gravity.

2.3. Task Planning of P/RML

The P/R operations can be broken down into a sequence of elementary tasks coupled in parallel with work-piece positioning tasks along the work station, as in [6,8,12]. The hybrid processing/reprocessing strategy is based on the hierarchical model proposed in [6,7,8,22], which uses a block diagram representation shown in Figure 4. ARSs carry the work-piece that fails the quality test from the place where it is stored (last station of P/RML to the initial position (first station of P/RML) for reprocessing. Figure 4 shows the task planning for processing a work-piece and is transporting by ARSs.

2.4. Assumptions

Specialized technology on this type of P/RML represents the base of a flexible industrial production line that gives a high range of standard products. Additionally, through reprocessing, the products can be recovered and brought to the required quality standards. The technology on P/RML, further developed, depends on aspects like operation modes, operation lengths and types of finished products [23]. Therefore, for P/RML, ARSs, RMs and VSSs, some assumptions have to be made for controlling the whole system:

Assumption 1.

P/RML is deterministic and with a single SHPN model, that allows the obtaining of suitable quality products and recoverable products, through reprocessing;

Assumption 2.

The number of the P/RML work stations involved in P/R is previously known and will remain unchanged;

Assumption 3.

Only one type of work-piece, but of different colours, is processed or reprocessed, because the proposed technology involves processing operations specific to a certain type of product;

Assumption 4.

All conditions and parameters of the technology are initially known, including task durations, costs and quantity of work-pieces that will take part in the process;

Assumption 5.

The workstations of the P/RML have a linear distribution: transport, handling, processing and storing workstation;

Assumption 6.

The P/R operations are executed on the same line and work-pieces can be processed at the same time in different system stages;

Remark 1.

As a result of the positioning errors or tool misalignments, it is possible for the processing operations not to be carried out properly. If these errors are in the meantime eliminated by performing the operations through reprocessing, the work-piece should be brought to the required quality standard.

Assumption 7.

A red work-piece will mean that the quality test is not passed and reprocessing is needed;

Assumption 8.

In the deposit, the first level from the top is for rejected work-pieces;

Assumption 9.

Two ARSs assisted the P/RML, one of them having mounted an RM, Pioneer 5-DOF, used for picking up the work-piece, and the other used for transport;

Assumption 10.

Two eye to hand VSS cameras are mounted on the P/RML, one on the last and another on the first work station;

Assumption 11.

One eye in hand VSS camera is mounted on the second RM, Cyton 1500.

The use of ARSs is justified for collaborating with the P/RML in order to recover the rejected work-pieces and transport them for reprocessing or for definitive rejection at the beginning of the line. The path and distances that the ARSs should pass are shown in Figure 3.

3. SHPN Model of the P/RML with Integrated ARSs

3.1. Structure of the SHPN Model

The hybrid aspect of the model (P/RML assisted by ARSs) is given by the variables associated with the distances covered by ARSs. These distances are covered by the ARSs between the last workstation (storage workstation) and the first station (transport workstation) of the P/RML. For modelling, we will use the SHPN tool, [12,16], which integrates the discrete aspect of P/RML with the continuous aspect of ARSs displacement likes is shown in [6,7,8,9]. The global model is of the SHPN type because it is interfaced with external events for synchronization, the events being signals coming from sensors and VSSs. The SHPN structure, from Figure 5, corresponds to the discrete modelling of the P/R processes and the continuous dynamics of the ARSs one of them equipped with RM, which assist the P/RML to bring the work-piece from the last to the first workstation, to be reprocessed. The internal structure of the SHPN model integrates two PN models, each of them having a specific typology: TPN (Timed Petri Network) and SPN + TPN (Synchronized Petri Network + Timed Petri Network) and THPN (Timed Hybrid Petri Network), [15,16,17]. These models describe the following operations that are performed automatically: processing (TPN modelling), reprocessing (TPN modelling) and assistance of the ARSs for the work-piece fastening and bringing it for reprocessing. Adding the apparition of an external event, the signals from sensors used for synchronization of the P/RML and the ARSs and RMs, the final model will have an SHPN structure that is shown in Figure 5. Each of the models represents a sequence in the real-time control operation, as follows: TPN type for P/R operations and SPN + TPN type for work-piece retrieving, transport and manipulation.

3.2. Formalism of the SHPN Model Associated to P/RML Assisted by Two ARSs

SHPN model from Figure 6, associated with the P/RML assisted by two ARSs is a triplet,

$$SHPN=\langle \begin{array}{ccc}THPN,& E_d,& Sync\end{array}\rangle ,$$

(1)

THPN is a septuplet

$$THPN=\langle P,T,Pre,Post,m_0,h,tempo\rangle ,$$

(2)

$E_d$ is a set of external events, signals from sensors.

$Sync$ maps the elements for the set of the discrete transitions in the set of the external events. External events are signals from sensors, used for synchronization.

$$Sync:T\to E_d.$$

(3)

P is a set of finite places,

$$P=P^D\cup P^C,$$

(4)

where:

$$P^D={\left\{P_{d_i}\right\}}_{i=\overline{1,15}}\cup {\left\{P_{d\_C_i}\right\}}_{i=\overline{1,14}}\cup {\left\{P_{r_j}\right\}}_{j=\overline{1,8}},$$

(5)

with ${\left\{P_{d_i}\right\}}_{i=\overline{1,15}}$ the set of the discrete state corresponding to the processing operations on the P/RML. (buffering, handling, catching, drilling, boring, sorting, releasing, etc.), ${\left\{P_{d\_C_i}\right\}}_{i=\overline{1,14}}$ are the discrete variables of the control system and ${\left\{P_{r_j}\right\}}_{j=\overline{1,8}}$ the discrete states that define the ARSs displacements, with ${\left\{P_{r_j}\right\}}_{j=\overline{1,4}}$ discrete states of the ARS Pioneer 3-DX, while ${\left\{P_{r_j}\right\}}_{j=\overline{5,8}}$ the discrete states of ARS PeopleBot,

$$P^C={\left\{P_{c_k}\right\}}_{k=\overline{1,5}},$$

(6)

with ${\left\{P_{c_k}\right\}}_{k=\overline{1,2}}$ the continuous places of ARS Pioneer 3-DX displacement, while ${\left\{P_{c_k}\right\}}_{k=\overline{3,5}}$ defines the continuous places of the ARS PeopleBot displacement.

$T$ is the set of finite transitions

$$T=T^D\cup T^C$$

(7)

where:

$$T^D={\left\{T_{d_i}\right\}}_{i=\overline{1,14}}\cup {\left\{T_{r_j}\right\}}_{j=\overline{1,14}},$$

(8)

with ${\left\{T_{d_i}\right\}}_{i=\overline{1,14}}$ the set of the discrete transitions for executing work-piece processing, ${\left\{T_{r_j}\right\}}_{\begin{array}{l}j=\overline{1,7}\\ \end{array}}$ the set of the discrete transitions set of ARS Pioneer 3-DX displacement while returning the work-piece for reprocessing and ${\left\{T_{r_j}\right\}}_{\begin{array}{l}j=\overline{8,14}\\ \end{array}}$ the set of the discrete transitions of ARS PeopleBot while returning the work-piece for reprocessing,

$$T^C={\left\{T_{c_k}\right\}}_{k=\overline{1,4}},$$

(9)

where: ${\left\{T_{c_k}\right\}}_{k=\overline{1,2}}$ is the set of the continuous transitions of ARS Pioneer 3-DX displacement for transporting the work-piece to the ARS PeopleBot and ${\left\{T_{c_k}\right\}}_{k=\overline{3,4}}$ the set of the continuous transitions of ARS PeopleBot displacement for transporting the work-piece to the RM Cyton. The sets of the places and of the transitions are disjoint $P\cap T=\varnothing$.

$Pre:P\times T\to Q_{+}$ is the input incidence function;

$Post:P\times T\to Q_{+}$ is the output incidence function;

$m_0:P\to R_{+}$ is the initial marking.

For $Pre$, $Post$ and $m_0$, in the case where $P_i\in P_d$, the functions take values in $N$ (natural number set) and, in the case where $P_i\in P_c$, the functions take values in $Q_{+}$ (positive rational number set).

$h$ is a hybrid function

$$h:P\cup T\to \left\{D,C\right\},h:P^D\cup T^D\to \left\{D\right\};h:P^C\cup T^C\to \left\{C\right\},$$

(10)

that indicates for every node whether it is a discrete node (sets $P^D$ and $T^D$) or a continuous node (sets $P^C$ and $T^C$), where $\left\{D\right\}$ and $\left\{C\right\}$ are the sets of the discrete or continuous node, respectively

$$\left\{D\right\}=P^D\otimes T^D\subseteq \left(P^D\times T^D\right)\cup \left(T^D\times P^D\right),\left\{C\right\}=P^C\otimes T^C\subseteq \left(P^C\times T^C\right)\cup \left(T^C\times P^C\right);$$

(11)

$tempo$ is the function of the timings associated to the transitions: $T^D={\left\{T_{d_i}\right\}}_{i=\overline{1,14}}\cup {\left\{T_{r_j}\right\}}_{j=\overline{1,14}}$ $T^C={\left\{T_{c_k}\right\}}_{k=\overline{1,4}}$ in Figure 6 and is defined as:

$$tempo:T\to Q_{+}\cup \left\{0\right\}.$$

(12)

If $T_j\in T_d$, then $d_j=tempo\left(T_j\right)$ is known as the timing associated with $T_j$. If $T_{cr}\in T_c$ then

$$U_r=\frac{1}{tempo\left(T_c\right)}$$

(13)

is the flow rate associated with $T_c$. For $T_c={\left\{T_{c_k}\right\}}_{k=\overline{1,2}}$, $U_{c_k}=U_r;U_{r\max }=V_r$, where $U_c$ is the variable flow of ARS displacement between continuous places, with $V_r=94\mathrm{mm}/\mathrm{s}$.

Let $ED\left(T_j,m\right)$ be the ED-enabling degree of a C-transition, $T_j$, for a marking $m$, denoted as after all the arcs, from a $C-$ place to a $C-$ transition, were erased.

$$ED\left(T_j,m\right)=\underset{P_i{\in }^0{T}_j\cap P^D}{\min }\lfloor \frac{m_i}{Pre\left(P_i,T_j\right)}\rfloor ,$$

(14)

Let the maximum firing speed be of transition $T_c$ as the product of its flow rate $U_r$ by its ED- enabling degree. As is shown in Figure 6, will have

$$ED\left(T_{c_j},m_{c\left(j+1\right)}\right)=\left\{0,1\right\},$$

(15)

where:

$$m_{c\left(j+1\right)}=V_j\cdot w\left(T_c{}_j\times P_c{}_{\left(j+1\right)}\right),$$

(16)

is the mark associated with a continuous place and

$$w\left(T_c{}_r\times P_c{}_{r+1}\right)=D\left(W_{N+1-j},N_{d_{j+1}}\right)/D\left(N_{d_j},W_{N+1-j}\right)$$

(17)

is the weight of the arc connecting a continuous transition to a continuous place of the ARS.

For $N=2$ the arches $\left(P_i\times T_j\right)$ have a weight equal to one, where:

$$P_i=\left\{\mathrm{ARS}state1,ARSstate2\right\}{\in }^o\left\{T{c}_k\right\}{}_{k=\overline{1,10}}\cap P_d,$$

(18)

For an SHPN, a transition is enabled if each of its input places has enough tokens.

The SHPN model, shown in Figure 6, is an oriented graph described with the Petri Nets (PN) formalism, according to the discrete events system (DES). The SHPN model describes the dynamics of the P/RML assisted by ARSs, as a hybrid process. For making the model, the following assumptions were made:

${\left\{P_{d_i}\right\}}_{i=\overline{1,15}}$ are the discrete states of the work-piece under processing. The actions that determine the modification of these states correspond to the operations: processing (drilling, boring, etc.), transporting, quality tests (QT1 and QT2) and handling;

${\left\{P_{r_j}\right\}}_{j=\overline{1,8}}$ are the discrete states of the two ARSs, determined by the running of the fixed point displacement and manipulation;

${\left\{P_{d\_C_i}\right\}}_{i=\overline{1,14}}$ are the control/synchronization signals necessary for controlling the sequence of tasks and synchronization at the P/RML level, served by ARSs;

$P^C={\left\{P_{c_j}\right\}}_{j=\overline{1,5}}$ are the continuous state variation of the position of the two ARSs, in relation to the endpoint of the displacement, corresponding to a complete service cycle (see Figure 3, with the workstations and the successive positioning scheme of the two ARSs). The synchronization signals are intended for the synchronization of the P/RML, ARS Pioneer 3-DX and ARS PeopleBot actions. In the SHPN formalism, they correspond to the set of events, which in the SHPN model are represented by:

$$E_d={\left\{Ed{d}^i\right\}}_{i=\overline{1,5}}\cup \left\{e\right\},$$

(19)

where: $Ed{d}^1=Sync\left(P_{d\_C}{}_1\right)$, $Ed{d}^2=Sync\left(P_{d\_C}{}_{11}\right)$, $Ed{d}^3=Sync\left(P_{d\_C}{}_{12}\right)$, $Ed{d}^4=Sync\left(P_{d\_C}{}_{13}\right)$, $Ed{d}^5=Sync\left(P_{d\_C}{}_{14}\right)$;

$\left\{e\right\}$ represents the neutral event that is considered to “synchronize” transitions $T\backslash \left\{T_{r14},T_{d14},T_{r7},T_{r14}\right\}$ that are neutral in terms of synchronization with other sub-processes;

$E_d={\left\{Ed{d}^i\right\}}_{i=\overline{1,5}}\cup \left\{e\right\}$ are the signals injected by changing the states, in the controlled sub-processes, as follows:

$Ed{d}^1$ synchronization signal for START_P/RML with END_ARS PeopleBot and START_Piece_Delivery to P/RML for reprocessing;

$Ed{d}^2$ synchronization signal for END_QT2 with START_ARS Pioneer 3-DX actions;

$Ed{d}^3$ synchronization signal for END_ ARS PeopleBot and START_ARS Pioneer 3-DX actions;

$Ed{d}^4$ synchronization signal for END_ARS Pioneer 3-DX and START_ARS PeopleBot actions;

$Ed{d}^5$ synchronization signal for END_Piece_Delivery to P/RML for reprocessing with START_ARS PeopleBot actions;

$Sync=Syn{c}_0\cup Syn{c}_1$ represents synchronization signals corresponding to each transition in the SHPN model with other sub-processes, so that the SHPN model, implicitly its simulation, is consistent with the control strategy of the whole process:

$$Syn{c}_0:T\backslash \left\{T_{r14},T_{d14},T_{r7},T_{r14}\right\}\to \left\{e\right\},$$

(20)

$$Syn{c}_1:\left\{T_{r14},T_{d14},T_{r7},T_{r14}\right\}\to {\left\{Ed{d}_i\right\}}_{i=\overline{1,5}}.$$

(21)

The paper presented here focuses on the hybrid aspect of the process, both in modelling and in control, taking into account the scenarios of P/RML assistance by ARSs. The hybrid control system of the P/RML serviced by two ARSs, is able to control in real-time the entire process, according to the control strategy.

3.3. Simulation of SHPN Model

Following the testing and simulation of the SHPN model described using the Sirphyco package shown (Figure 6), the simulation results are shown in Figure 7, Figure 8 and Figure 9. The advantage of using common simulation for both the discrete and continuous aspects in the model is that the interaction between the two parts can be obtained and analysed. Additionally, as a result of the hybrid model simulation, the maximum speed of the two ARSs that assist P/RML was reached. The speed was computed so that the cycle time for work-piece recovery and transporting is minimum. In addition, the hardware constraints of the ARSs have to be taken into account in order to obtain the best speed. By using the SHPN model, a unique and precise connection is achieved between the two models: the TPN which models the P/RML tasks and the THPN which models the operations of two ARSs while retrieving and transporting the work-pieces. In the complete SHPN model, represented in Figure 6, each task performed by the P/RML (such as buffering, handling, drilling, boring, etc.) has a very precisely determined time, together with the duration of ARSs displacement. Figure 7 and Figure 8 present simulation results of the model and it can be observed that the time ranges correspond to the given reality.

Firstly, ARSs are initialised and then the ARS Pioneer 3-DX waits for an external event that is described through the existence of a work-piece that could be reprocessed. The video system used for synchronization detects the appearance of the work-piece and creates this external event which will trigger the robot for retrieving. Next, the ARS Pioneer 3-DX will put the recovered work-piece on the PeopleBot for transporting. Two synchronization signals will be needed:

-$Sync1$-synchronization signal End QT2 and START ARS Pioneer 3-DX;

-$Sync2$-synchronization with P/RML for reprocessing of the work-piece, END ARS PeopleBot action and START Pioneer 3-DX action.

4. Modelling and Control of the VSSs for Moments of the Image

4.1. Modelling and Control Structure

The structure of a VSS is based on the following components: an autonomous system composed of a mobile robot equipped with a 5 or 6-DOF manipulator, a controller and a video sensor (web camera). The fundamental part of the architecture, the image-based controller requires a priori information about the behaviour of the system in order to minimize the error between a current configuration of visual features, $f$, and a desired configuration, $f^{*}$. To model the behaviour in the open loop of the servoing system, the two entities that form the fixed part must be analysed separately: the manipulator robot and the visual sensor. It is still considered a configuration of the eye-in-hand type for the visual sensor-robot assembly. The main idea for modelling a VSS is to minimize the error between the real and desired features extracted from the video sensor, [9,19,23,24]. The control structures of two types of VSSs are shown in Figure 10 and Figure 11. Let ${\upsilon }_c^{*}$ be the signal associated to the control input of the ARS. This signal represents the reference speed of the camera and has the following structure: ${\upsilon }_c^{*}={({\upsilon }^{*},{\omega }^{*})}^T$, ${\upsilon }^{*}={[{\upsilon }_x^{*},{\upsilon }_y^{*},{\upsilon }_z^{*}]}^T$ and ${\mathsf{\omega }}^{*}={[{\mathsf{\omega }}_x^{*},{\mathsf{\omega }}_y^{*},{\mathsf{\omega }}_z^{*}]}^T$ defined as the linear and the angular speed, respectively. The signal ${\upsilon }_c^{*}$ is expressed in the Cartesian space and requires a transformation to be able to be applied to the manipulator robot. If we note with $s={[s_1,s_2,s_3,s_4,s_5,s_6]}^T$ the posture obtained by ${\upsilon }_c^{*}$ integration, then we define the robot Jacobian as:

$$J_r=\lfloor \partial s_i/\partial q_j\rfloor ,i,j=1,\dots ,6.$$

(22)

where $q_j,j=1,\dots ,6$ represents the state of the robot’s joints. Thus, the signal transformation of ${\upsilon }_c^{*}$ from Cartesian space to robot joint space will be achieved by $J_r^{-1}$ and the interaction matrix. The interaction matrix needs to fulfil a series of properties in order to obtain the optimal behaviour for VSS. It has to be non-singular and diagonal. Considering the moments $m_{ij}$ as a set of visual features then, the analytical form for its variation in time,${\dot{m}}_{ij}$, corresponding to the moments of order $(i+j)$, depending on the speed of the camera ${\upsilon }_c^{*}$, is:

$${\dot{m}}_{ij}=L_{m_{ij}}{\upsilon }_c,$$

(23)

where $L_{m_{ij}}=[m_{v_x},m_{v_y},m_{v_z},m_{w_x},m_{w_y},m_{w_z}]$ is the interaction matrix.

Based on the theory presented in [9,19,23], the interaction matrix corresponding to a set of moments of the image $f={[x_n,y_n,a_n,\tau ,\xi ,\alpha ]}^T$ , for $n$ points, can be computed as below:

$$L_f=\left[\begin{array}{cccccc}-1& 0& 0& a_n{e}_{11}& -a_n(1+e_{12})& y_n\\ 0& -1& 0& a_n(1+e_{21})& -a_n{e}_{11}& -x_n\\ 0& 0& -1& -e_{31}& e_{32}& 0\\ 0& 0& 0& {\mathsf{\tau }}_{{\mathsf{\omega }}_x}& {\mathsf{\tau }}_{{\mathsf{\omega }}_y}& 0\\ 0& 0& 0& {\mathsf{\xi }}_{{\mathsf{\omega }}_x}& {\mathsf{\xi }}_{{\mathsf{\omega }}_y}& 0\\ 0& 0& 0& {\mathsf{\alpha }}_{{\mathsf{\omega }}_x}& {\mathsf{\alpha }}_{{\mathsf{\omega }}_y}& -1\end{array}\right],$$

(24)

4.2. Control Input

The most used method to generate the control signal for robots is the use of proportional control. The visual servoing system can be viewed as a minimization problem that finds the path of the visual sensor by minimizing the cost function attached to the error vector. Note with $f^{*}$ the desired feature vector, $f$ with the current feature vector, and $r(t)$ the relative position between the camera and the object at time $t$. Note with $f\left(r\right(t\left)\right)$ the features variation reported to the relative movement between the workspace and the video sensors:

$$\dot{f}=\frac{\partial f}{\partial r}\frac{dr}{dt}+\frac{\partial f}{\partial t}=L_f{\upsilon }_c+\frac{\partial f}{\partial t},$$

(25)

Because the time variation of the features, reporting to the motion of the object, is zero, then $\frac{\partial f}{\partial t}=0$ for a static object. Therefore, (25) becomes

$$\dot{f}=L_f{\upsilon }_c,$$

(26)

where ${\upsilon }_c$ is the vector describing the relative speed between the object and the video sensor $L_f$ and is the interaction matrix from (24). To describing the control law, it is required to define an error function between the target features $f^{*}$ and the current features $f$:

$$e=f-f^{*}$$

(27)

Because most implementations of visual servoing systems do not take into account the dynamics of the robot (the dynamics of the robot is considered one), then ${\upsilon }_c={\upsilon }_c^{*}$ and (26) becomes:

$$\dot{f}=L_f{\upsilon }_c^{*}$$

(28)

From (27) and (28) the time variation of the error can be expressed as:

$$\dot{e}=L_f{\upsilon }_c^{*}$$

(29)

Since the robot control input is defined by ${\upsilon }_c^{*}$ and wishing an exponentially negative decrease in the error, $\dot{e}=-\lambda e$, $\dot{e}=L_f{\upsilon }_c^{*}=-\lambda e$, the following control law is obtained from (29):

$${\upsilon }_c^{*}=-\lambda L_f^{+}e,$$

(30)

where $L_f^{+}$ is the pseudo inverse of the interaction matrix and can be computed with the expression:

$$L_f^{+}={(L_f^T{L}_f)}^{-1}{L}_f^T,$$

(31)

Because in real-time VSSs, the distance $Z$ between the points of interest and the reference system attached to the camera is not precisely known, it must be estimated, noted with $\stackrel{\wedge }{L_f^{+}}$. The estimation of this matrix is based on the pseudo-inverse of the target features interaction matrix $\stackrel{\wedge }{L_f^{+}}=L_f^{*}{}^{+}$ with $\stackrel{\wedge }{L_f^{+}}=\frac{1}{2}{(L_f+L_f^{*})}^{+}$. Because this matrix remains constant during the entire control algorithm, the control law is:

$${\upsilon }_c^{*}=-\frac{1}{2}\lambda {(L_f+L_f^{*})}^{+}e.$$

(32)

5. Real-Time Control of P/RML with Integrated ARSs, RMs and VSSs

To control the assistive technology of the four stations P/RML Festo MPS 200, three robotic systems were integrated: ARS Pioneer 3-DX equipped with RM Pioneer 5-DOF, for picking up the work-piece from the last station; ARS PeopleBot for transporting and RM Cyton 1500 for picking up and releasing the work-piece from ARS PeopleBOT AND on the first work station of P/RML, respectively.

Real-time control of the assisted technology is based on five control loops:

• control loop for P/RML with SIMATIC STEP 7 and SIEMENS 300 PLC;
• control loop for ARS Pioneer 3-DX equipped with RM PIONEER 5-DOF and eye to hand VSS1. Video camera is mounted on the last station of P/RML;
• control loop for ARS PeopleBot based on trajectory tracking sliding-mode control (TTSMC). Additionally, an obstacle avoidance method based on sonars and ultrasounds was implemented [5,9,10];
• control loop for RM Cyton 1500 based on eye in hand VSS for precise positioning for picking up work-piece from ARS PeopleBot. The eye in hand VSS camera is mounted on the end effector of RM Cyton 1500;
• control loop for RM Cyton 1500 based on inverse kinematic for transporting work-piece and eye to hand VSS2 to release it on P/RML. The eye to hand VSS2 cameras is mounted to first station of P/RML.

All five control loops communicate through two remote computers one of them runs the graphic user interface (GUI) application and controls P/RML, ARSs, VSS1 and VSS2, another controls RM Cyton 1500 and eye in hand VSS. Specific programming packages, SIMATIC STEP 7, Microsoft Visual Studio and MATLAB, are used to control the whole system. As shown in Figure 12, the communication between the GUI application and ARSs is done based on a TCP/IP protocol. Using the specialized image processing toolbox from MATLAB and the control input defined in equations (30) and (32), the synchronization between all control loops is implemented.

Figure 13 illustrates a series of image and frames captured during real time-control based on the eye to hand VSS located to the last station of P/RML. Additionally, Figure 13 shows the real-time steps with the precise positioning, based on eye in hand VSS, of the ARS Pioneer 3-DX equipped with RM Pioneer 5-DOF: (a) Displacement of the ARS Pioneer P3-DX and positioning to retrieve the work-piece from the P/RML, (b) catching the work-piece, (c) displacement of ARS Pioneer P3-DX to the ARS PeopleBot, based on real-time, trajectory-tracking, sliding-mode control; (d) releasing the work-piece on ARS PeopleBot, (e) return to the starting position of the ARS Pioneer P3-DX. Real-time control of ARS Pioneer 3-DX equipped with RM Pioneer 5-DOF for picking up work-piece using eye to hand VSS. Some frames of real-time control of ARS Pioneer 3-DX equipped with RM Pioneer 5-DOF for picking up work-piece using eye to hand VSS, are presented in Figure 14.

Figure 15 describes the trajectory with obstacle avoidance of ARS PeopleBot assisting P/RML based on TT-SMC and sensitivity bubble method (sonars located on bottom platform), [10,25,26,27]. The sensitivity bubble, shown in Figure 16, depends on the speed of movement of the robot and allows a good detection of obstacles. The components for the sensitivity bubble,$Boundry=\left[b_1{b}_2\dots b_8\right]$, are computed using:

$$b_i=k_i\cdot v\cdot T_s$$

(33)

where: $i$—number of readings; $b_i$—component of the sensitivity bubble; $k_i$—the safety factor,$v$—robot speed; $T_s$—the sample time. An obstacle is detected when the distance measured by the sonars, up to the obstacle, is smaller than the sensitivity bubble.

In Figure 17, it can be observed that there is an obstacle on the desired trajectory (black line) and it is desired to find a point holding the property that allows the robot to bypass the obstacle using the bypass trajectory (blue line). Knowing the width of the robot, we checked if the sonar detects another nearby obstacle in the chosen bypass direction.

Figure 18 describes the following: the desired and the real trajectory of the ARS PeopleBot obtained with TT-SMC in a closed loop. A small deviation of the trajectory can be observed when ARS PeopleBot executes the return of ${90}^{\circ }$. Figure 19 shows the tracking error when moving on the X axis and the tracking error when moving on the Y axis. When the ARS is moving on the X axis, there are two deviations due to connection loss, but they tend immediately to 0. Additionally, Figure 19 illustrates the angular error, where it can be seen that the maximum error is $\pm 6^{\circ }$.

Figure 20 shows the real-time control of RM Cyton based on invers kinematics, [20], eye in hand and eye to hand VSS for handling the work-piece: in the left initial position, in the middle, caching the work-piece from ARS PeopleBot, in the right, releasing the work-piece on P/RML.

Real and desired trajectories are shown in: Figure 21, corresponding to the movement of RM Cyton from initial position to the ARS PeopleBot position for picking up the work-piece from upper platform of ARS PeopleBot, using inverse kinematics control and eye in hand VSS for precision positioning; Figure 22, corresponding to the movement of RM Cyton for transporting work-piece from ARS PeopleBot to P/RML, using inverse kinematics control and eye to hand VSS for precision positioning; Figure 23 corresponding to the movements of RM Cyton from P/RML to the parking position using inverse kinematics control.

Some frames of real-time control of RM Cytor for picking up work-piece from ARS PeopleBot, based on eye in hand VSS and releasing it on P/RML, based on eye to hand VSS, are presented in Figure 24 and Figure 25, respectively.

6. Discussion

We mention in [5] that the SHPH model and its simulation in autonomous mode (HPN model) are based on two reasons. The first one is to make compatible the discrete dynamics of P/RML with the continuous dynamics of ARSs. The compatibility is needed because the P/RML and ARSs have characteristics and physical constraints that should be considered. The second one is due to the uncertainty introduced by the precise positioning of the RM for picking up or releasing the work-piece, because VSSs have not been implemented. Additionally, we state in [5,9,28] that the introduction of VSSs is a research objective in the near future. Assisting P/RML by VSSs, RMs and VSSs led to the elimination of an important part of the uncertainty during a complete cycle. One of the objectives of the control and monitoring of this manufacturing technology on P/RML assisted by ARSs, RMs and VSSs is the minimization of the cycle time. By using the TTSMC to control ARSs and sensitivity bubble method to avoid obstacles, the problem of system robustness was partially addressed. The following have to be considered as uncertainties: faulty sensors/actuators, route/storage space blockage and payload variation. In future research, these will also be taken into account.

7. Conclusions

The assistive technology implemented to the four workstations P/RML, FESTO MPS 200, allows the recovery, through reprocessing, of products that do not correspond qualitatively. To successfully implement this eco-technology, ARSs, RMs and VSSs are used for assistance, with the last ones as actuators for the movements of the RMs. Although it is a technology that works at the laboratory level, it can be extended to the real industry, especially in those areas where positioning and precision handling are required. Based on the SHPN model, SIMATIC STEP 7, Visual C ++ and MATLAB platforms, real-time control structures were implemented, allowing fully automatic, processing/reprocessing, assisted by ARSs, RMs and VSSs. The use of the combination of Eye in hand and Eye to hand VSS as actuators allowed the robotic systems to make precise positioning and manipulation. The signals from the sensors were used to synchronize tasks and operations. The SHPN model and its simulation were useful in combining the discrete dynamics of P/RML with the continuous one of robotic systems. This technology can be applied not only to mechatronics lines but also in manufacturing processes in metallurgy, ceramics, glass and the automotive industry. In most cases, robotic manipulators that assist these processes have fixed locations. Through this study, we extended the degree of automation and efficiency of these production lines using ARSs, RMs and VSSs.