A Computer Vision-Based Navigation and Localization Method for Station-Moving Aircraft Transport Platform with Dual Cameras

Abstract

In order to develop equipment adapted to the aircraft pulse final assembly line, a vision-based aircraft transport platform system is developed. This article explores a guiding method between assembly stations which is low-cost and easy to change routes by using two-dimensional code and two complementary metal oxide semiconductor (CMOS) cameras. The two cameras installed on the front and back of the platform read the two-dimensional code containing station information to guide the platform. In the process of guiding, the theoretical position and posture of the platform at each assembly station are known, but there is a difference between the actual and theoretical values due to motion errors. To reduce the influence of the deviation on the navigation route, a localization method is proposed based on the two-dimensional images captured by the cameras. Canny edge detection is applied to the processed image to obtain the position of the two-dimensional code in the image, which can measure the angle/distance deviation of the platform. Then, the computer can locate the platform precisely by the information in the two-dimensional code and the deviation measured by the image. To verify the feasibility of the proposed method, experiments have been performed on the developed platform system. The results show that the distance and angle errors of the platform are within ±10 mm and ±0.15° respectively.

1. Introduction

In aircraft manufacturing, the assembly has great impact on the production cost, which is estimated to account for almost 30% of the total cost [1,2]. To reduce the manufacturing cost, many aviation companies have introduced the pulse final assembly line for its high efficiency [3]. The aircraft pulse final assembly line is composed of several different assembly stations which are made up of dozens of jobs. During each takt time, an unassembled aircraft enters from one end of the line and stops at the first assembly station. After completing the corresponding assembly jobs at a station, the aircraft is required to move to the next station.

There are two ways to transport the aircraft between the assembly stations [4,5]. The first one is based on the overhead cranes. The aircraft is hoisted by a crane and moved to the next station. As it is dangerous and time-consuming, it is gradually replaced by the second way. The second way adopts the rail and positioners. The aircraft is supported by the positioners and can move together along the rail [6,7]. However, it needs to build the rail and it is difficult to change routes once the rail is built. AGVs (Automatic Guided Vehicle) have been used for repeated transportation tasks in factories for decades [8,9]. For small-sized aircraft, like fighters, it can be more flexible to transport by AGVs. This paper aims to develop an automatic guided platform system for the transportation of small-sized aircraft between assembly stations.

In recent years, various types of indoor navigation and localization methods using magnetic, infrared, laser, and vision sensors have been proposed [10,11,12]. Cameras are cheaper and able to provide more information than other sensors. Besides, vision navigation does not require costly transmitters, receivers, and amplifiers to transmit signals. Therefore, vision navigation is adopted in this paper.

Vision navigation technique can be mainly divided into four categories: lane-tracking navigation, landmark-recognizing navigation, navigation using appearance-based matching, and simultaneous localization and mapping (SLAM) [13]. Landmark-recognizing navigation based on artificial landmarks is easy to change routes, and the artificial landmarks are easy to produce, manipulate, and maintain. Jeisung et al. [14] used fiduciary markers with a capital letter or triangle indicating direction in it to guide a vehicle. Kim et al. [15] applied Artificial Neural Networks (ANNs) to recognize letters of the alphabet on the marker to locate the AGV. Petriu et al. [16] proposed a method of recognizing bit codes on the floor to give the AGV direction. Based on the idea of Petriu et al. [16], some studies [17,18,19] used two-dimensional code to guide the vehicle. A two-dimensional code or QR (Quick Response) code is a graphic marker of a two-dimensional matrix which can store position information. In addition, QR code is small, low cost, and easy to implement. Therefore, QR codes are utilized as landmarks for navigation and localization in this paper.

Rostkowska et al. [20] addressed the application of QR codes as landmarks for mobile robot navigation, but this method of displaying big QR codes on every box is not widely used. Luca et al. [21] used QR codes mounted on a ceiling for smart wheelchair navigation. However, the camera will fail to detect the QR code if the ceiling is too high and it requires at least one QR code in the field of view of the camera at any time. Nguyen et al. [22] attached the QR codes on gaps between tiles on a floor. The robot gets position and orientation information from the QR code and follows the gap line to the next QR code. Unfortunately, there is no tiles on the workshop floor and no gap lines to follow.

Moreover, the previous studies did not take the vehicle or robot size into account. The experiments were all carried out on small prototypes. For long-sized vehicle, such as the platform in this article, a small angle deviation will result in a large distance deviation at the end of the vehicle. Also, the angle deviation may be too small to be detected by the QR code image. As a result, the vehicle can stop at the right position but the distance deviation at the end of the vehicle body is large.

In this paper, QR codes are stuck on the floor at each assembly station. Two CMOS (Complementary Metal Oxide Semiconductor) sensor cameras are installed at the front and back of the platform to capture the QR code images. Therefore, the small angle deviation of the platform can be calculated by the images in the two cameras and the length of the platform. In addition, the drive wheels of the platform are equipped with rotary encoders. In this way, the position of the platform can be calculated by the encoder data when the QR code is not in the field view of the camera.

The main contribution of this paper is to provide a new solution for the transportation of small-sized aircraft between stations in the final assembly line. In addition, the work presents a new method for measuring the small angle deviations of long-sized vehicles in QR code navigation.

This paper organization is as follows: Section 2 describes the system architecture of the transport platform and its workflow in the aircraft final assembly line. The navigation mathematical model and localization method are introduced in Section 3. In Section 4, experiments are presented in detail, including analysis and discussion of the results. Finally, Section 5 collects the conclusions of the work.

2. Materials

2.1. System Architecture

The aircraft transport platform is a flexible and numerically controlled system. It consists of a supporting system, a navigation system, and a control system. The mechanical structure of the transport platform is shown in Figure 1, while

Table 1. Dimensions and specifications of the transport platform.

Description	Quantity
Length × Width × Height	10,500 × 2500 × 1300 mm
Mass	10.2 kg
Max. Linear Velocity	1.67 m/s
Drive Wheel Diameter	410 mm
Diameter of the Wheel for Steering	300 mm

shows the main dimensions and specifications of it.

The supporting system is composed of four parking mechanisms and four bracket mechanisms. The bracket mechanisms are distributed over the transport platform according to the shape and dimensions of the aircraft fuselage, as shown in Figure 1a. They can move up and down singly or synchronously to support the aircraft fuselage. Each bracket contains two pin-axis force sensors, which can monitor the bearing of the bracket mechanism in real time. The parking mechanisms are located at the four corners of the transport platform, as shown in Figure 1b. The supporting foot of the parking mechanism is connected to the screw elevator through a ball head, which can adapt to uneven ground.

The navigation system consists of two drive wheels, four wheels for steering, four industrial casters, and two CMOS cameras. They are used to navigate and locate the platform. The four industrial casters have a swivel design and play an auxiliary supporting role to prevent deformation of the middle part of the platform body. Each drive wheel has two servo motors, which control the forward and backward directions and the rotational direction. Each wheel for steering contains only one servo motor which controls the direction of rotation. The layout of the wheels is shown in Figure 1b. The front and rear CMOS cameras of which the centers are on the central axis of the platform face the floor. A CMOS sensor is a kind of light-emitting pulse sensor, which can convert scene information into a color information matrix for computer processing.

The control system has an industrial computer and an embedded PLC (Programmable Logic Controller CX1020, Beckhoff, Will, Germany), and the data interaction between the two devices is carried out through the TCP/IP communication protocol. The control software is developed with TwinCAT NC (Beckhoff, Will, Germany) and Visual C++ (Microsoft, Redmond, WA, USA). The control of the whole system includes the lifting movement control of the supporting system and the omnidirectional movement control of the platform.

Figure 2 shows the control loop of the supporting system. The speed and position are set by the user through the control software. Then, the computer will generate the motor commands of the movement according to the input and transmit them to the embedded PLC. These commands trigger the servo motors through the drives, generating velocity and moving the bracket mechanisms or parking mechanisms. The servo motors operate in a closed-loop system, which includes three types of control loops: the position loop, the velocity loop, and the current loop. Both the current loop and the velocity loop are PI controllers with proportional and integral gains, while the position loop is a proportional controller. The velocity loop and the current loop are implemented in the servo drives, and the position loop is implemented in the PLC.

Figure 3 illustrates the navigation control system which needs the user to input a target station number and select a motion mode. The design theoretical position and posture of the aircraft for each assembly station are stored in the computer and are sorted by the station numbers. The station numbers are encoded into QR codes, which are placed on the floor at corresponding stations. After processing the captured QR code image, the position of the QR code in the image is obtained by edge detection, while the station number is obtained by decoding the QR code. Through the proposed localization method, which will be introduced in detail in the next section, the deviation of the platform can be calculated by the QR code images. Then, the actual position and angle can be calculated by the theoretical value and the obtained deviation. Finally, based on the user input and the actual position and angle, the computer calculates the navigation route and generates the servo motor commands. The wheels receive the commands and drive the platform to the target station. In this paper, for the drive motor of the drive wheel, the proportional gain of the position loop is set to 3.5.

2.2. Workflow in Aircraft Final Assembly Line

As is mentioned in the introduction, the platform is used to transport an aircraft in a pulse final assembly line. Figure 4 shows an example of an aircraft pulse final assembly line. According to the assembly jobs, the production line is divided into seven stations. Station No. 0 is the preparation station, which is used to hoist the fuselage onto the platform and to wait for assembly. The installation of conduits and cables is carried out at No. 1 station. The wings are butted at station No. 2, while the vertical tail is installed at station No. 3. At station No. 4, docking of nose and final test are carried out. Station No. 5 and 6 are used for research and maintenance.

As shown in Figure 4, at each assembly station, QR codes are placed on the floor directly below the cameras of the platform. Then, the transport platform can work in the assembly line, and the specific process is as follows:

(a)

Hoist the aircraft onto the platform. Drive the platform below the aircraft fuselage hoisted by the workshop crane via the equipped wireless infrared remote. Control the bracket mechanisms to rise to the specified height. Then, place the aircraft fuselage slowly on the bracket mechanisms of the platform. Drive the platform into station No. 0, and park it at the position where both cameras can capture the QR codes on the floor.

(b)

Carry out the assembly jobs. After the platform arrives at the designated position, the wheels will brake automatically. The parking mechanisms will move synchronously to raise the platform and to bear the weight of the aircraft and the platform. Then, the assembly jobs can be carried out on the platform. During the assembly process, the aircraft can be adjusted to the specified height and be stably supported.

(c)

Transport the aircraft to the next station. Move the parking mechanisms back to the original position synchronously through the control software. Enter the number of the target station and select the motion mode. Confirm that the navigation parameters are correct and click the start button. Then, the platform will automatically transport the aircraft to the target station and park in the design posture.

3. Navigation and Localization Study

3.1. Navigation Mathematical Model

If the unevenness of the workshop floor is neglected and the transport platform is regarded as a rigid body, the motion of the platform in the workshop can be regarded as the plane motion of rigid bodies. The local coordinate system $\left(x,o,y\right)$ of the transport platform is established by taking the geometric center of the platform as the origin and the forward direction as the positive direction of the x axis, as shown in Figure 5. In Figure 5, wheel No. 1–4 are wheels for steering and No. 5–6 are drive wheels. Four industrial casters located in the middle of the platform only play an auxiliary supporting role and do not affect the motion of the platform, so they have been neglected in the model. The transformation equation between local and global coordinates is as follows:

$$\left[\begin{array}{c}{X}_i\\ Y_i\\ 1\end{array}\right]=\left[\begin{array}{ccc}\cos \alpha & \sin \alpha & x_0\\ -\sin \alpha & \cos \alpha & y_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_i\\ y_i\\ 1\end{array}\right]$$

(1)

where $x_0,y_0$ is the global coordinates of the origin of the local coordinate system, $\alpha$ is the angle between the local coordinate system and the global coordinate system, and $\left(x_i,y_i\right)$ and $\left(X_i,Y_i\right)$ $\left(i=1,2,3\dots 8\right)$ are the local and global coordinates of the wheels or cameras.

Table 2. Local coordinates of the wheels and cameras.

No.	Components	Local Coordinates	No.	Components	Local Coordinates
1	FL Wheel	(4970, −925)	5	Front Drive Wheel	(4815, 0)
2	FR Wheel	(4970, 925)	6	Back Drive Wheel	(−4815, 0)
3	BL Wheel	(−4970, −925)	7	Front CMOS Camera	(5346, 0)
4	BR Wheel	(−4970, 925)	8	Back CMOS Camera	(−5346, 0)

Note: FL, FR, BL and BR are short for front left, front right, back left and back right respectively. CMOS is short for complementary metal oxide semiconductor.

describes the local coordinates of the platform components designed in this work.

The position and angle of the platform are expressed by $\left[x_0,y_0,{\alpha }_0\right]$ which are all functions of time. The motion of the platform can be modelled as follows:

$$x_0(\mathrm{t})=f_1(\mathrm{t}),y_0(\mathrm{t})=f_2(\mathrm{t}),{\alpha }_0(\mathrm{t})=f_3(\mathrm{t})$$

(2)

The motion of the platform between stations can be regarded as the process from the initial position and posture $\left[x_0,y_0,{\alpha }_0\right]$ of a certain station to the target position and posture $\left[x_0^{\prime },y_0^{\prime },{\alpha }_0^{\prime }\right]$ of the next station through a certain motion mode in time t. The motion mode of the platform in this paper includes self-rotational motion, linear motion, and fixed-axis rotation.

3.1.1. Self-Rotational Motion

The self-rotational motion is used to adjust the forward direction of the platform, which is achieved by turning the six wheels to 90 ° and by giving the two drive wheels opposite speeds. The angle of the rotation can be controlled by the distance of the drive wheels, which can be obtained by the following:

$$\mathsf{\Delta }\alpha =\frac{S}{R}=\frac{S}{y_5}$$

(3)

where $\mathsf{\Delta }\alpha$ is the angle of rotation, which is positive for clockwise motion and negative for counterclockwise motion, and where $S$ is the distance of the drive wheels.

3.1.2. Linear Motion

Linear motion is the most used motion mode. Its process includes three phases: self-rotation to adjust the direction, linear motion to reach the target position, and self-rotation to reach the target posture, as shown in Figure 6.

Since both the first and the third phases are self-rotating motions, the distances $S_1$ and $S_3$ of the drive wheels can be calculated by Equation (3). The corresponding rotation angles $\mathsf{\Delta }{\alpha }_1$ and $\mathsf{\Delta }{\alpha }_2$ can be obtained from the angle $\omega$ between the straight line $o{o}^{\prime }$ and the $X$ axis, which is as follows:

$$\begin{array}{cc}\mathsf{\Delta }{\alpha }_1={\alpha }_0-\omega & \mathsf{\Delta }{\alpha }_2=\omega -\end{array}{\alpha }_0^{\prime }$$

(4)

where $\omega$ can be obtained from the start and end positions of the platform:

$$\omega =\{\begin{array}{l}-\begin{array}{cc}\frac{\pi }{2}& x_0^{\prime }=x_0,y_0^{\prime }<y_0\end{array}\\ \begin{array}{cc}\frac{\pi }{2}& x_0^{\prime }=x_0,y_0^{\prime }>y_0\end{array}\\ \begin{array}{cc}\arctan \frac{y_0^{\prime }-y_0}{x_0^{\prime }-x_0}& x_0^{\prime }\ne x_0\end{array}\end{array}\omega \in \left[-\frac{\pi }{2},\frac{\pi }{2}\right]$$

(5)

The distance $S_2$ of the second phase can be calculated by the distance formula between two points. Apply the coordinates of the initial and target positions to the distance formula, which is as follows:

$$S_2=\sqrt{{(y_0^{\prime }-y_0)}^2+{(x_0^{\prime }-x_0)}^2}$$

(6)

3.1.3. Fixed-Axis Rotation

Figure 7 presents the two phases of the fixed-axis rotation, which is rotation around a fixed axis to the target position and self-rotation to the target posture.

The path of the first phase is an arc with the start position and the target position as the two endpoints and the heading of the initial posture as the tangent direction. Therefore, it is required that the target position point is not on the initial heading direction line, which can be expressed as follows:

$$y_0^{\prime }-y_0\ne \left(x_0^{\prime }-x_0\right)\tan {\alpha }_0$$

(7)

Assuming that the global coordinates of the center $P$ of the arc path are $\left(X_P,Y_P\right)$, then $\left|oP\right|=\left|o^{\prime }P\right|$ can be described as follows:

$${\left(x_0-X_P\right)}^2+{\left(y_0-Y_P\right)}^2={\left(x_0^{\prime }-X_P\right)}^2+{\left(y_0^{\prime }-Y_P\right)}^2$$

(8)

$P$ is on the line $oP$, which means the following:

$$Y_P=X_P\cot {\alpha }_0+y_0-x_0\cot {\alpha }_0$$

(9)

Based on Equations (8) and (9), we can calculate the global coordinates $\left(X_P,Y_P\right)$ of the center $P$. Then, we plug $\left(X_P,Y_P\right)$ into Equation (1) and we can obtain the local coordinates $\left(x_P,y_P\right)$ of the center $P$. Let ${\theta }_i$ denote the angle need to rotate of No. i wheel, $\left(i=1,2,3\dots 6\right)$. According to the geometrical properties of a circle, ${\theta }_i$ is equal to the corresponding central angle ${\beta }_i$, which can be calculated by the following:

$$\begin{array}{c}\tan {\theta }_i=\tan {\beta }_i=\frac{y_i}{x_p-x_i},\\ {\theta }_i=\begin{array}{cc}\arctan \frac{y_i}{x_p-x_i}& {\theta }_i\in \left(-\frac{\pi }{2},\frac{\pi }{2}\right)\end{array}\end{array}$$

(10)

where ${\theta }_i$ is also positive for clockwise motion and negative for counterclockwise motion.

It is assumed that the central angle of the arc path is $\gamma$, and then, the distance of the first phase can be obtained by the following:

$$S_1=\sqrt{{(y_5-y_p)}^2+{(x_5-x_p)}^2}\times \gamma$$

(11)

where $\gamma$ is equal to the angle between the line $oP$ and the line $o^{\prime }P$. $\gamma$ also depends on which side of the initial orientation the target position is located on, which means $\gamma$ needs to be expressed in the form of piecewise functions:

$$\gamma =\{\begin{array}{l}\begin{array}{cc}\frac{\pi }{2}+{\alpha }_0-\arctan \frac{y_0^{\prime }-Y_p}{x_0^{\prime }-X_P}& y_0^{\prime }-y_0<\left(x_0^{\prime }-x_0\right)\tan {\alpha }_0\end{array}\\ \begin{array}{cc}\frac{\pi }{2}-{\alpha }_0+\arctan \frac{y_0^{\prime }-Y_p}{x_0^{\prime }-X_P}& y_0^{\prime }-y_0>\left(x_0^{\prime }-x_0\right)\tan {\alpha }_0\end{array}\end{array}$$

(12)

The second phase of the fixed-axis rotation is a self-rotating motion, and the distance can be calculated by Equation (3). Correspondingly, the $\mathsf{\Delta }\alpha$ term in this case is related to $\gamma$, the initial angle ${\alpha }_0$, and the target angle ${\alpha }_0^{\prime }$. According to the geometric relationships, $\mathsf{\Delta }\alpha$ can be obtained by the following:

$$\mathsf{\Delta }\alpha =\{\begin{array}{l}\begin{array}{cc}{\alpha }_0-\gamma -{\alpha }_0^{\prime }& y_0^{\prime }-y_0<\left(x_0^{\prime }-x_0\right)\tan {\alpha }_0\end{array}\\ \begin{array}{cc}{\alpha }_0+\gamma -{\alpha }_0^{\prime }& y_0^{\prime }-y_0>\left(x_0^{\prime }-x_0\right)\tan {\alpha }_0\end{array}\end{array}$$

(13)

3.2. Localization Method

In theory, the platform will park at the assembly station in the design posture, which means that the QR code will be in the center of the image captured by the CMOS camera. However, in actual navigation, because of the existence of motion errors, the actual position and posture will deviate from the theoretical values, which leads to a deviation between the QR code and the image center. As the cameras are fixed on the platform, there must be a quantitative relationship between the two deviations which can help us to obtain the deviation of the platform from the images and locate the platform precisely.

3.2.1. Image Acquisition

To measure distance and angle deviation of the platform through the QR code images, it is necessary to explore the correspondence of the image with the captured scene. Figure 8 illustrates a simplified camera image acquisition model with the camera components in detail.

Let the length of each pixel be $\lambda$ $\mathrm{mm}$. According to the relationship between the real scene distance and the image distance shown in Figure 8, we can calculate $\lambda$ by the triangle similarity theorems:

$$\mathsf{\Delta }\alpha =\{\begin{array}{l}\begin{array}{cc}{\alpha }_0-\gamma -{\alpha }_0^{\prime }& y_0^{\prime }-y_0<\left(x_0^{\prime }-x_0\right)\tan {\alpha }_0\end{array}\\ \begin{array}{cc}{\alpha }_0+\gamma -{\alpha }_0^{\prime }& y_0^{\prime }-y_0>\left(x_0^{\prime }-x_0\right)\tan {\alpha }_0\end{array}\end{array}$$

(14)

where $\phi$ is the field of view angle, $W,L$ is the CMOS size, $f$ is the focal length, $Q_x,Q_y$ is the image resolution, and $h$ is the installation height defined by the platform project.

Table 3. Parameters description and values of the cameras used in this work.

Description	Quantity
Brand Model	VISIONGO SGIB1280-60 gm/gc
Sensor Technology	CMOS image sensor
Max. Frame Rate	60 fps
Pixel Resolution	1280 × 1024 pixels
CMOS Size	7.716 × 5.319 mm
Focal Length	20 mm
Installation Height	455 mm

describes the parameters of the camera used in this work.

3.2.2. Image Processing

The origin images captured by the cameras need to be denoised and preprocessed to posterior information extraction, as shown as an example in Figure 9a. The image processing algorithm initiates with a Gaussian filter to remove salt-and-pepper noise and to smooth the images. Equation (15) gives the two-dimensional Gaussian function:

$$G\left(x,y\right)=\frac{1}{2\pi {\sigma }^2}\exp \left(-\frac{x^2+y^2}{2{\sigma }^2}\right)$$

(15)

where $\sigma$ is the standard deviation that controls the smoothness of the image. The smaller the value, the higher the positioning accuracy of the filter but the lower the signal-to-noise ratio. On the contrary, a higher $\sigma$ value leads to a lower positioning accuracy but a higher signal-to-noise ratio. After many tests, the value of $\sigma$ in this paper is set to 2, and the filtered image is shown in Figure 9b.

For the RGB (Red, Green, and Blue) color image, each pixel contains three components and each component has 256 values [23], which results in a large amount of computation. In order to reduce the computational complexity, the filtered image is converted to a grayscale image which contains only brightness information. Figure 9c gives the converted image.

In order to describe the accurate position of the QR code in the image, the image coordinate system $\left(x_c,o_c,y_c\right)$ is established by taking the center of the image as the origin and the forward direction as the positive direction of x axis, as shown in Figure 9e. Canny edge detection is applied to obtain the coordinates of the edges in the image. Canny edge detection algorithm is a classical edge detection algorithm based on image gradient calculation developed by John F. Canny in 1986 [24,25]. Figure 9d is the binary image with clear Canny edge in white and background in black.

Since the QR code is a standard square, it is easy to obtain its position by the coordinates of its four corners. Based on Canny edge detection results, coordinates of four corners of the QR code are extracted, as shown in Figure 9e. Then, we can use the four coordinates to calculate the position of the QR code and to measure the distance and angle deviation of the platform.

3.2.3. Deviation Calculation

For ease of description, let $\left(x_A,y_A\right)$, $\left(x_B,y_B\right)$, $\left(x_C,y_C\right)$, and $\left(x_D,y_D\right)$ denote the coordinates of the four corners obtained by the abovementioned image processing. In this paper, the coordinates of the center point $\left(x_{qr},y_{qr}\right)$ of the QR code are used to represent its position. For reducing the error of edge detection, the four-point average method is used to calculate the coordinates of the center point:

$$x_{qr}=\frac{x_A+x_B+x_C+x_D}{4},y_{qr}=\frac{y_A+y_B+y_C+y_D}{4}$$

(16)

In order to calculate the deviation of the platform, it is necessary to convert the coordinates in the image coordinate system $\left(x_c,o_c,y_c\right)$ into the current platform coordinate system $\left(x_r,o_r,y_r\right)$. Based on Equation (15), the length of each pixel $\lambda$ and the local coordinates of the front camera $\left(x_7,y_7\right)$ and back camera $\left(x_8,y_8\right)$, we can derive the transformation equation between the two coordinate systems:

$$\begin{array}{cc}\left[\begin{array}{c}{x}_r\\ y_r\\ 1\end{array}\right]=\left[\begin{array}{ccc}\lambda & 0& x_i\\ 0& \lambda & y_i\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_c\\ y_c\\ 1\end{array}\right]& \left(i=7,8\right)\end{array}$$

(17)

Figure 10 presents the QR code images in the front and rear cameras when the platform is parked at a certain assembly station. The actual position and posture of the platform $\left[x_{r0},y_{r0},{\alpha }_{r0}\right]$ can be modelled as follows:

$$\begin{array}{ccc}{x}_{r0}=x_0+e_x& y_{r0}=y_0+e_y& {\alpha }_{r0}={\alpha }_0+\delta \end{array}$$

(18)

where $\delta$ is the angle deviation, and where $e_x$ and $e_y$ are the distance deviations in the x and y directions, respectively.

Let $q_{r1}$ and $q_{r2}$ denote the center points of the QR code in the front and back cameras, respectively. According to Figure 10, the slope of the connection between $q_{r1}$ and $q_{r2}$ is equal to the angle deviation $\delta$, and the midpoint of $q_{r1}$ and $q_{r2}$ is the theoretical position of the platform in the current platform coordinate system $\left(x_r,o_r,y_r\right)$. Based on this relationship, we can calculate the angle deviation $\delta$ and the distance deviation $e_x$ and $e_y$ through the coordinates of these two points in the current platform coordinate system. Let $\left(x_{qr1},y_{qr1}\right)$ and $\left(x_{qr2},y_{qr2}\right)$ denote the coordinates of the two center points in their image coordinate systems, which can be obtained by Equation (16). Then, plug $\left(x_{qr1},y_{qr1}\right)$ and $\left(x_{qr2},y_{qr2}\right)$ into Equation (17) and we can obtain the coordinates of the two points in the current platform coordinate system, which are $\left(\lambda x_{qr1}+\frac{L}{2},\lambda y_{qr1}\right)$ and $\left(\lambda x_{qr2}-\frac{L}{2},\lambda y_{qr2}\right)$, respectively. By substituting the two coordinates into the mentioned geometric relationships, the final deviation calculation formula is derived as follows:

$$\begin{array}{c}\delta =\begin{array}{cc}\arctan \frac{y_{qr1}-y_{qr2}}{x_{qr1}-x_{qr2}+L/\lambda }& \delta \in \left(-\frac{\pi }{2},\frac{\pi }{2}\right)\end{array}\\ e_x=\left(-\frac{y_{qr1}+y_{qr2}}{2}\times \sin \delta -\frac{x_{qr1}+x_{qr2}}{2}\times \cos \delta \right)\times \lambda \\ e_y=\left(\frac{x_{qr1}+x_{qr2}}{2}\times \sin \delta -\frac{y_{qr1}+y_{qr2}}{2}\times \cos \delta \right)\times \lambda \end{array}$$

(19)

The actual position and posture of the platform can be obtained by substituting the results of Equation (19) into Equation (18). Based on the corrected position and posture, the input target station number, and the selected motion mode, the computer can calculate the accurate navigation path and generate the required motor commands for the wheels to drive the platform to the target station.

4. Experimentation

To verify the feasibility and effectiveness of the navigation and localization method, experiments were performed on a design path containing seven stations. Figure 11 shows the experimental path schematic. The design position and posture of each station were stored in the computer and sorted by the station numbers. The station numbers were encoded into the QR codes, which were placed on the floor for the cameras to read.

To reproduce the condition closest to the actual working conditions, the platform was loaded with weights by using an auxiliary rack. The weight of the weights was slightly heavier than the target aircraft, and the weights were distributed according to the weight distribution of the target aircraft. In this paper, the total weight of the loading weights was 22 T, as shown in Figure 12a. After loading, the platform was driven to station No. 1 of the test path by the equipped wireless infrared remote and parked at the position where both of the front and back QR codes can be captured by the two cameras, as shown in Figure 12b. Then, the remote control was switched to the navigation mode. Through the control software, according to the design path, station No. 2 was input as the target station and linear motion was selected as the motion mode. The subsequent station number and motion mode were input into the control software when the platform arrived at the target station. In addition to the user input, the software interface, as shown in Figure 12c, also displays real-time loading information of the four bracket mechanisms, fault information of the wheels, and the QR code information obtained by the cameras, which is helpful for the user to monitor the status of the platform in the navigation process.

When the platform was parking at each station, the actual position and angle were measured and recorded and the deviation between the actual and theoretical values was calculated. The results of the first navigation test are summarized in

Table 4. Theoretical and actual positions and postures at each station of the first test.

No.	Theoretical Value (mm, mm, °)	Actual Value (mm, mm, °)	Deviation (mm, mm, °)
1	(10,000.00, 20,000.00, 90.000)	(10,051.23, 20,031.36, 88.620)	(51.23, 31.36, −1.380)
2	(20,000.00, 30,000.00, 0.000)	(19,994.13, 29,991.11, −0.132)	(−5.87, −8.89, −0.132)
3	(30,000.00, 20,000.00, −90.000)	(30,003.53, 20,008.11, −89.994)	(3.53, 8.11, 0.006)
4	(50,000.00, 20,000.00, 45.000)	(49,995.73, 19,995.01, 45.056)	(−4.27, −4.99, 0.056)
5	(70,000.00, 20,000.00, −90.000)	(70,005.14, 20,006.32, −89.858)	(5.14, 6.32, −0.142)
6	(85,000.00, 20,000.00, 0.000)	(84,997.24, 19,999.76, 0.099)	(−2.76, −0.24,0.099)
7	(95,000.00, 20,000.00, 90.000)	(94,994.02, 20,004.35, 89.875)	(−5.98, 4.35, −0.125)

. From Table 4, one can observe that the deviation of station one is much larger than that of other stations. This is because the platform is driven by the wireless infrared remote into station one, which makes it difficult to adjust the position and posture accurately. Still, at station two, the position deviation is reduced to within 10 mm and the angle deviation is reduced to 0.13°, which means that the transport platform can obtain the correct path to the second station and can drive into the second station accurately. It is verified that, through our algorithm, the theoretical position and the posture of station one were corrected to the actual values. From the deviation of subsequent stations, it can be summarized that, in the first navigation experiment, the maximum distance deviation in the X direction is −5.87 mm, the maximum Y-directional deviation is −8.89 mm, and the maximum angle deviation is −0.142°, which preliminary demonstrates that the proposed method is a feasible and effective solution for aircraft transportation between assembly stations.

In order to analyze the accuracy of the transport platform, repeated tests were carried out. Likewise, the position and posture of the platform at each station were measured and recorded. As cited, the data of station one was independent of the navigation accuracy and was removed in the analysis. As a summary of trials results, Figure 13 displays a line chart of the calculated position and posture deviation of the other stations. In Figure 13, one can visually perceive the overall performance of the platform. The platform runs stably, and the deviation is randomly distributed within a certain limit.

Compared with the positive and negative directions of the deviation, the value is more important for the accuracy of the platform. Therefore, we used the absolute deviation of the test results for quantitative analysis.

Table 5. Total trial data summary for the repeated tests.

Statistical Items	Average Value	Max. Value
Deviation in X direction	4.82 mm	9.21 mm
Deviation in Y direction	4.32 mm	9.99 mm
Distance Deviation	4.57 mm	9.99 mm
Angle Deviation	0.079°	0.149°

summarizes the statistics analysis of the processed data. Observing Table 5, the deviations in the X and Y directions are basically the same, which indicates that the distance deviation of the platform in this paper does not have directionality. Thus, the average of the deviations in the two directions was considered as the overall distance deviation of the platform, as shown in the third line of Table 5. Generally, the position deviation of the platform is less than 10 mm, with 4.57 mm on average, while the angle deviation is less than 0.15°, with 0.079° on average. The acceptable position and angle deviations of the aircraft at each assembly station are mainly determined by the size of the aircraft, and in this paper, they need to be within $\pm 25\mathrm{mm}$ and $\pm 0.2°$ respectively. Therefore, the platform can meet the position and posture requirements of the target aircraft at the assembly station in this paper.

To explore the relationship between the deviation and motion mode, the experimental data were divided into two groups: one was linear motion, while the other was fixed-axis rotation. Therefore, data of two, three, and seven stations and four, five, and six stations were extracted and analyzed, respectively.

Table 6. Comparison between the linear motion and the fixed-axis rotation.

Statistical Items	Distance Deviation Mean	Distance Deviation Max.	Angle Deviation Mean	Angle Deviation Max.
Linear Motion	4.43 mm	8.98 mm	0.085°	0.147°
Fixed-axis Rotation	4.72 mm	9.99 mm	0.084°	0.149°
Difference	0.29 mm	1.01 mm	0.001°	0.002°
Proportion	6.54%	11.2%	1.17%	1.36%

summarizes and compares the two groups data. From Table 6, it can be observed that, both on average and at maximum, the distance deviation in linear motion mode is better than that in fixed-axis rotation while the angle deviation is almost the same. It might be interpreted that the distance deviation is mainly the result from the navigation route, which is more complicate in fixed-axis rotation mode, and that the angle deviation is caused by the final angle adjustment, which is the same in the two motion modes. Thus, the platform in this work has a better position accuracy in linear motion mode than in fixed-axis rotation mode.

5. Conclusions

In this paper, an efficient and feasible navigation and localization method has been proposed for the transport platform for aircraft final assembly. The guiding approach is based on two CMOS cameras installed on the front and back of the platform. The platform obtains the theoretical position information by decoding the QR code images captured by the two cameras. Compared with other navigation methods in literature, the proposed approach has some advantages in terms of the cost of installation and changing routes. To obtain the precise position and angle of the platform, a localization method based on the QR code images is proposed. Canny edge detection is applied to the processed image to obtain the position of the QR code in the image. By constructing the relationship between the captured real scene and the image, the angle/distance deviation of the platform can be obtained by the image. Then, the actual position and posture of the platform can be calculated based on the obtained deviation. The proposed method has been integrated into the platform and has been tested on the design path in a workshop. The results suggest that the position accuracy of the platform is within $\pm 10\mathrm{mm}$ and that the angle error is within $\pm 0.15°$, which can meet the position and posture requirements of the target aircraft at the assembly station in this paper.

As future work, in terms of error reduction, the work points in two directions. The first point concerns the fixed-axis rotation. As discussed, the performance of the fixed-axis rotation mode is worse than that of the linear motion mode. It suggests that we can explore a simpler control strategy for the fixed-axis rotation mode. The other direction refers to the sensor and image which affect the accuracy of localization. In this sense, in the future, we plan to investigate the relationship between the parameters of CMOS sensor camera and the deviation of the platform and to improve the edge detection algorithm according to the characteristics of the acquired images.