A Novel Vision- and Radar-Based Line Tracking Assistance System for Drone Transmission Line Inspection

Abstract

This paper introduces a position controller for drone transmission line inspection (TLI) utilizing the integral sliding mode control (SMC) method. The controller, leveraging GNSS and visual deviation data, exhibits high accuracy and robust anti-interference capabilities. A deviation correction strategy is proposed to capture high-voltage transmission line information more robustly and accurately. Lateral position deviation is calculated using microwave radar data, attitude angle data, and deviation pixels derived from transmission line recognition via MobileNetV3. This approach enables accurate and stable tracking of transmission lines in diverse and complex environments. The proposed inspection scheme is validated in settings with 10-kilovolt and 110-kilovolt transmission lines using a drone with a diagonal wheelbase of 0.275 m. The experimental process is available in the YouTube link provided. The validation results affirm the effectiveness and feasibility of the proposed scheme. Notably, the absence of a high-precision positioning system in the validation platform highlights the scheme’s versatility, indicating applicability to various outdoor visual-based tracking scenarios using drones.

1. Introduction

Electric resources are irreplaceable in diverse fields, such as safeguarding industrial production, agricultural production, military defense, national defense, and people’s livelihoods. Electric transmission lines transport electrical energy produced by power plants to various regions. Many transmission lines are built in harsh natural environments, and they can easily malfunction or be damaged. Regular inspection is essential to ensure the safety of transmission lines and equipment. In addition to traditional walking checks, some advanced robots, such as climbing robots [1], drones [2,3,4,5,6], and combination robots [7,8], are also designed to help complete the tasks more effectively [9].

Electric power inspection tasks can be categorized into four types, including the inspection of power components, transmission lines, power towers, and defects [10]. Various comprehensive inspection methods for components, towers, and defects have been proposed. Typically, personnel manually gather waypoint data to generate the path automatically [11]. In the approach outlined in this thesis, drones equipped with image acquisition systems are employed for path tracking. Image data are collected and analyzed simultaneously [12,13], ensuring that the drone remains above the wires during TLI.

In recent years, researchers have devoted significant attention to TLI. One approach, presented in [14], utilized the Hough transformation for transmission line detection and employed a fuzzy C-means clustering algorithm for subsequent selection. Building on this, ref. [15] enhanced the Hough transform’s accuracy and efficiency. Another study, ref. [16], proposed imaging characteristics and a knowledge-based detection method. Additionally, ref. [17] exploited parallel features between cables for rapid and precise transmission line identification, while [18] introduced a deep learning (DL) method for accurate line extraction. In dynamic settings, ref. [19] determined the pixel velocity based on recognition results and integrated it with a Kalman filter to estimate the cable positions. Notably, ref. [20] employed event cameras for the accurate, timely, and continuous tracking of transmission lines. It is important to highlight that these works primarily focus on wire detection and may require verification in real-world environments.

Recent advancements in the field of TLI have seen diverse approaches addressing challenges in drone-based inspection systems. For instance, an intelligent inspection system proposed in [21] utilized DJI M600 Pro and maintained a distance of 25 m from the transmission line to mitigate GNSS distortion and geomagnetic interference. Similarly, ref. [22] introduced a histogram method for directional line segments to determine the transmission line direction, while [23] combined dual-antenna GNSS with laser data to track transmission line locations. However, these solutions have limitations, such as challenges in gathering rich power line surface information over long distances or losing close-range transmission line details.

To address these challenges, ref. [24] designed a multi-rotor drone capable of landing on the transmission line, calculating the attitude angles during landing using monocular and laser radar data. While effective, this method required manual operations during the landing process, imposing high requirements on the personnel. Subsequent works, such as [25,26], proposed autonomous inspection frameworks using vision-lidar and dual-lidar, respectively. Despite their advancements, these solutions were primarily verified in simulation or unenergized transmission line environments, often overlooking the information within the drone system.

In contrast, the present work, building upon the foundation laid by [27], not only improves transmission line recognition with the introduction of MobileNetV3 but also enhances the drone’s deviation control capabilities through integral sliding mode control. The culmination of these improvements results in the proposal of a line-tracking assistance system, a novel contribution validated in an actual high-voltage tower environment. This work provides a unique and comprehensive solution, addressing gaps left by the existing methods. The contributions of this work are threefold:

• A line tracking assist system (LTAS) is proposed as an independent module, distinct from the drone’s control system. The LTAS is designed to enhance the drone’s tracking capabilities along the transmission lines significantly.
• The proposed system employs DL techniques, a MobileNetV3 scheme, to process the raw images effectively, addressing challenges posed by background disturbances. Additionally, the implementation includes an SMC-based deviation controller. This controller aids the drone in consistently maintaining its position relative to the transmission lines, leveraging real-time line recognition results. The goal is to ensure that the drone follows the target line seamlessly throughout its trajectory.
• The developed drone and the proposed scheme were tested through actual flight validations in energized 10-kilovolt and 110-kilovolt transmission line environments. These validation results can be accessed at https://youtu.be/Nbmx3v0LHX4 (accessed on 4 January 2024). The LTAS played a crucial role in ensuring the drone’s accurate tracking along the transmission lines.

The remainder of this paper is structured as follows: Section 2 outlines the challenges associated with TLI and introduces the proposed LTAS. Section 3 explains the transmission line recognition process. Section 4 details the designed deviation control system. The simulation and the practical experiments are presented in Section 5.

2. Challenges in TLI and System Overview

2.1. Challenges in TLI

In existing TLI methodologies, two primary approaches are prevalent: waypoint-based inspection and line-recognition-based inspection.

Waypoint-based inspection involves planning a predetermined trajectory in the ground station, often called the waypoint mission. Subsequently, the drone autonomously follows this trajectory and captures information during flight. However, since the route is pre-specified, there is the potential for deviation between the planned course and the actual transmission line. Existing solutions typically rely on higher-accuracy devices to measure the transmission line’s position and the drone’s localization. Moreover, maintaining a safe distance between the drone and the transmission line is crucial throughout the inspection process to avoid geomagnetic interference around the transmission line.

Waypoint-based inspection has high equipment costs and may not effectively capture more detailed transmission line information. This limitation is evident in studies such as [2,28].

The second approach relies on recognition results to guide the drone along the transmission line, often utilizing radar sensors and cameras for detection. However, this method introduces challenges related to geomagnetic interference around the transmission line. The small size of the transmission line makes it difficult to capture data from a long distance, mainly when relying on cameras as the primary sensing device. External wind interference and the complexity of the environment may contribute to sensor failure or misidentification, requiring careful consideration.

In cases where unexpected situations arise, such as sensor failures or misidentification issues, the drone may deviate from the transmission line, potentially failing to pursue the inspection mission. These challenges highlight the importance of robust and resilient methodologies for line-recognition-based inspection.

The two approaches mentioned above have specific areas that warrant improvement. The waypoint-based inspection scheme is susceptible to trajectory deviations and often relies on high-precision equipment for long-range inspection. On the other hand, the second scheme based on line recognition faces challenges related to sensor failure or misidentification.

In the field of TLI, there is a pressing need for drones to fly along transmission lines at close range without depending on high-precision equipment. Moreover, these drones should be resilient to wind interference during flight. Addressing these requirements is crucial for developing effective and reliable TLI methodologies.

2.2. Deviation Control System Overview

Considering the above points, this work addresses the design of a deviation control scheme for line inspection based on the waypoint mission. The method integrates data from cameras, radars, and GNSS sensors. It is important to note that the proposed scheme operates independently of the standard cascade control system in the drone control system. The overall structure of the proposed method is depicted in Figure 1.

During inspection, drones fly above the transmission lines and utilize cameras to capture information. Line recognition, however, faces challenges due to interference from the complex ground environment, and the imaging characteristics of the transmission line can vary under different lighting conditions. Ensuring the stability of traditional line recognition schemes using fixed parameters for extracting transmission lines becomes difficult. An improved deep learning method is proposed to address this challenge. The transmission line’s position is also estimated by considering the drone’s lateral velocity. The estimated results are then used in the line recognition section to assist in selecting the transmission lines.

The LTAS is employed as an auxiliary system independent of the UAV primary control system, facilitating stable and precise control of the UAV directly above power transmission lines, and visual information is utilized within a visual serving approach in the control system. The inputs to the line assistance system include images captured by the camera $\mathsf{\Delta }pix$, the relative height $\mathsf{\Delta }h$ between the UAV and the power lines obtained from radar measurements, the UAV’s roll angle $\theta$ (neglecting this if the camera is fixed at the bottom of the aircraft and maintained level through the use of a gimbal), the UAV’s heading angle $\varphi$, the roll angle target value ${\theta }_r$, and the lateral velocity $v_l$ of the UAV (computed by the UAV’s navigation algorithm, with the raw horizontal velocity input derived from the measurements of the GNSS sensor).

The output of the line assistance system consists of two components: one representing the additional target value for the roll angle ${\theta }_{Ar}$ and the other indicating the target value for the heading angle deviation ${\varphi }_{Ar}$. They are computed based on the lateral deviation distance $d_{lr}$ and the heading angle deviation ${\varphi }_{lr}$ (between the UAV and the power lines) estimated by a deviation state observer.

The deviation state observer takes into consideration the lateral deviation distance $\mathsf{\Delta }d$ between the UAV and the power lines obtained from visual data, the influence of the roll angle $\theta$, the UAV’s heading angle $\varphi$, the roll angle target value ${\theta }_r$, and the lateral velocity $v_l$ of the UAV. The observed results serve as input to the deviation controller and are utilized to estimate the probable position and orientation of the power lines in the image at the next time step. This information is then employed to allocate weights for line filtering during image recognition dynamically. This design offers two benefits: reducing misidentification and obtaining an accurate deviation state.

The lateral deviation distance $\mathsf{\Delta }d$ is calculated using the pixel deviation $\mathsf{\Delta }pix$ obtained from visual data, the camera optical parameters, and the relative height $\mathsf{\Delta }h$ between the UAV and the power lines measured by radar, employing the pinhole imaging principle.

The ground station provides the target heading angle ${\varphi }_r$, the altitude $al{t}_r$, and waypoints $P_r$ for the UAV. The UAV estimates the attitude using data from the three-axis angular velocity, the three-axis acceleration, and the three-axis magnetometer sensors. The resulting attitude angles (pitch angle $\phi$, roll angle $\theta$, heading angle $\varphi$) serve as feedback for the inner-loop angular controller. The GNSS data undergo fusion through integrated navigation, yielding the position in the NED coordinate system. The latitude and longitude coordinate P serves as feedback for the position controller, the altitude $alt$ as feedback for the altitude controller, and the horizontal velocity (obtained through coordinate transformation for the lateral UAV velocity $v_l$ in the body coordinates) as feedback for the velocity controller.

The three deviation control values output by the LTAS will act on the inner attitude controller (roll angle target value = cascade controller roll angle target value ${\theta }_r$ + LTAS correction controller roll angle output ${\theta }_{Ar}$; the heading angle target value = cascade controller heading angle target value ${\varphi }_r$ + LTAS correction controller heading angle output ${\varphi }_{Ar}$), and the altitude controller (altitude target value = integrated navigation feedback altitude $alt$ + UAV set distance from power line height − radar height data $al{t}_{Ar}$), respectively.

With the assistance of the LTAS, the drone can fly close to the transmission line without the need for expensive high-precision sensors, all while collecting detailed information. This scheme presents a novel and practical reference solution for TLI using drones.

3. Transmission Line Recognition

3.1. Line Recognition Description

GNSS coordinate drift poses a challenge due to factors like the ionospheric refraction error, the tropospheric refraction error, and the receiver clock error. Maintaining the drone’s relative position with the transmission line becomes difficult, especially in scenarios with known start and end points obtained from the ground station. In TLI, the camera is positioned below the drone, allowing the observation of lateral deviation between the drone and the transmission line through captured images. A radar sensor measures the relative vertical distance between the transmission line and the drone, while the horizontal deviation distance is calculated using images from the camera.

The obtained deviation distance enables the drone to perceive its deviation from the transmission line. Image processing aids the drone in predicting the position and angle of the transmission line in the image at the next moment. This allows for a more accurate selection of the line from the multiple lines detected in the image.

The line recognition system comprises a camera, a microcomputer, and a flight control system. The camera serves a dual purpose: capturing raw images of the transmission line and generating a binarized image corresponding to the raw image. Both sets of images are then transferred to the microcomputer.

The raw images are directly stored on the microcomputer, while the binarized image undergoes further processing through the Hough transform. This process extracts the position and heading deviation about the recognized transmission lines in the images. The microcomputer then matches the recognition results with the estimated results from the autopilot. The resulting transmission line is marked in the images with red stripes, facilitating viewing at the ground station. An overview of the entire process is illustrated in Figure 2.

3.2. Image Segmentation Based on Canny Edge Detection

The old scheme employs traditional edge detection for transmission line recognition in this program. In the given application scenario, Raspberry Pi is utilized as the image processing unit, necessitating consideration of its computational limitations. The preprocessing stage involves three primary steps: grayscale conversion, contrast enhancement, and denoising.

The raw RGB image is first grayscaled to alleviate the computational burden. The weighted average method (1) is deemed the most suitable choice among the classic and effective grayscale conversion methods. Grayscaling, calculated by adjusting the weights based on the characteristics of the target object, is expected to be more conducive to enabling the subsequent processes.

$$G\mathrm{ray}\left(x,y\right)=W_r\ast R\left(x,y\right)+W_g\ast G\left(x,y\right)+W_b\ast B\left(x,y\right)$$

(1)

In the weighted average method (1), $W_r$, $W_g$, and $W_b$ represent the weights assigned to the R, G, and B channels, respectively, with the constraint that $W_r+W_g+W_b=1$. Considering the different sensitivities of human vision to the primary colors, being highest for green and lowest for blue, the weight parameters should adhere to the condition $W_r>W_g>W_b$.

While grayscaling the entire image reduces the computational burden on the processor, it may result in the loss of specific grayscale details. Therefore, contrast enhancement is deemed necessary. Given that the features of the detected transmission lines exhibit consistency, grayscale enhancement is achieved using the segmented linear transformation scheme (2).

$$g\left(x,y\right)=\left\{\begin{array}{cc}\frac{M_g-d}{M_f-b}\left[f\left(x,y\right)-b\right]+d& b\le f\left(x,y\right)\le M_f\\ \frac{d-c}{b-a}\left[f\left(x,y\right)-a\right]+c& a\le f\left(x,y\right)\le b\\ \frac{c}{a}f\left(x,y\right)& 0\le f\left(x,y\right)\le a\end{array}\right.$$

(2)

In Equation (2), the three-stage segmented linear transform divides the gray range of the raw image $f\left(x,y\right)$ into three segments: $[0,a]$, $[a,b]$, and $[b,M_f]$, with corresponding transformations to $[0,c]$, $[c,d]$, and $[d,M_g]$, resulting in $g\left(x,y\right)$.

When the transformed interval is larger than the original interval, it is labeled “stretching”; conversely, if the gap becomes smaller, it is marked “compression”. It is essential to note that various factors, such as the image sensors and the environmental conditions, can introduce noise to the image during image acquisition and transmission. Image filtering is necessary to address this issue. In this approach, the median filtering scheme is chosen for this task. While it effectively reduces noise, it is crucial to acknowledge that this filtering process may lead to the blurring of fine details, especially at the edges of the transmission line.

To accurately identify the features of transmission lines, the Canny operator is employed to detect changes in edges. This operator exhibits strong anti-interference characteristics and can retain edge information if it is the target. However, the extracted edge information often interferes, as pylons are built in highways, fields, and rivers. The collected images contain a lot of invalid backgrounds in addition to the transmission lines. After the edges of the image are extracted by the Canny operator, the edges of the interference backgrounds need to be removed, a process achieved by combining mask processing and morphological operations.

Mask processing is equivalent to superimposing an image with a transparent and opaque area on the image to be processed. The part of the raw images under the delicate area remains unchanged, while the opaque area is removed. In this way, practical regions of the images can be selected. Assuming that the image to be processed is $g\left(x,y\right)$, and the mask is $b\left(x,y\right)$, the expression formula of the final processed image is $M\left(x,y\right)$, as shown in Equation (3).

$$M\left(x,y\right)=\left\{\begin{array}{c}0 ,b\left(x,y\right)=0\\ g\left(x,y\right)\begin{array}{cc}& ,b\left(x,y\right)=1\begin{array}{cc}& \end{array}\end{array}\end{array}\right.$$

(3)

The color feature of the wire was chosen as the primary basis of the mask, and the color threshold of the RGB of the wire was adjusted according to the collected top view of the wire. Subsequently, this adjusted threshold was used for binarization of the raw color map of the transmission line.

At this point, the preprocessing of the raw image has been completed. The entire processing process is depicted in Figure 3.

3.3. Image Segmentation Based on MobileNetV3

As the backbone network, MobileNetV3 comprises multiple convolutional layers and extended convolutional layers, enabling it to extract rich feature information from the input image. Its lightweight architecture contributes to high computational and parametric efficiency, reducing the model’s complexity while maintaining accuracy. This characteristic makes it well suited for deployment in the drone ontology for line recognition.

Additionally, combining the context module, the channel-wise module, and the arm module enhances the efficiency and accuracy of the semantic segmentation tasks. This integrated approach results in more accurate segmentation outcomes while reducing the computational and parameter complexity concurrently.

Figure 4 illustrates the network structure of MobileNetV3. Firstly, new nonlinear activation functions, h-swish and h-sigmoid, effectively reducing the computational costs, are introduced. Second, the block module is updated, and the squeeze-and-excitation (SE) module is incorporated, assigning a more considerable weight to specific channels to enhance accuracy. Third, the time-consuming layer structure is redesigned by eliminating convolutional kernels that contribute little to accuracy. The last stage is streamlined to improve the inference speed. Finally, leveraging the neural architecture search (NAS) for parameter optimization automates the search and selection of the best network structure, enhancing the performance and efficiency of the model.

3.4. Transmission Line Selection Strategy

After the Hough transform is finished, all the lines that satisfy the threshold in the image are extracted, containing single or multiple parallel lines and interference lines. In the actual inspection scenes, the ground line in the middle of numerous transmission lines is the primary target to be tracked. The remaining lines on both sides are still in the acquisition area. Therefore, extracting the desired line from the multiple recognition results must be considered.

After completing the Hough transform, all lines that meet the specified threshold in the image are extracted. These lines may include single or multiple parallel lines and interference lines. In practical inspection scenarios, the primary target for tracking is often the ground line in the middle of multiple transmission lines. The remaining lines on both sides are still within the acquisition area. Therefore, it becomes crucial to address the task of extracting the desired line from the multiple recognition results.

Two strategies are proposed to address this issue:

• Choose a forward straight line from the image’s center within the polar angle range of (−30, 30 degrees). Take note of the image dimensions as ($Pi{x}_X$, $Pi{x}_Y$). Ensure that the Y coordinate of the line’s midpoint falls within the specified threshold ($Pi{x}_Y/2\pm Pi{x}_Y/20$).
• Record the pixel deviation and the polar angle of the previously recognized line in the preceding frame as $r_0$ and ${\theta }_0$, respectively. Constrain the recognized results in the current frame within the ranges $r_0\pm r_m$ and ${\theta }_0\pm {\theta }_m$, where $r_m=Pi{x}_Y/20$ and ${\theta }_m=10$ degrees serve as the specified limits.

After aligning the drone directly above the transmission line, ensuring the drone’s heading matches that of the line, Strategy 1 is applied. This strategy eliminates pairs of straight lines with angles significantly deviating from the drone’s heading and located far from the center of the screen. The selected pairs are then categorized into a set of straight lines. The optimal result is determined by choosing the group with the most significant consecutive lines within the collection. If the drone is not directly above the transmission line, the optimal output is determined by selecting the nearest straight line to the pixel point ($Pi{x}_X/2$, $Pi{x}_Y/2$).

The drone initiates flight along the line after reaching the mission’s starting point and successfully recognizing the target transmission line. Simultaneously, Strategy 2 is activated. The corresponding weight changes are illustrated in Figure 5.

In the grayscale diagram, weight is inversely proportional to the grayscale; smaller grayscale values indicate higher weights, and vice versa. The yellow line in Figure 5 signifies the straight line with the highest weight.

The weights are partitioned into two components: the angle and the position, with the angle taking precedence over the position. For instance, under Strategy 1, the weight assignment is illustrated in Figure 5a. The yellow line positioned in the center of the screen receives the highest weight. Any tilt or deviation from the screen’s center reduces its weight. The straight line with the maximum weight among the recognized lines is selected.

Figure 5b illustrates a scenario where the drone deviates to the right or tilts left. In this case, the weights favor positions closer to the x-axis on the screen. The polar angle from the previous moment increases, impacting the polar angle of the straight line on the screen. Similarly, Figure 5c depicts the opposite situation, with the yellow straight line representing the position with the highest weight.

4. Deviation Control System Design

4.1. Control System Structure

The proposed LTAS, as discussed in Section 2.2, utilizes the line recognition results as feedback to compute the auxiliary roll target. This target is integrated into the standard cascade control system, enhancing the drone’s lateral control to manage route deviations and external wind interference effectively.

For deviation control, ${\theta }_{Ar}$ is implemented using SMC. Additionally, a Kalman filter is employed to process the feedback information. The structure of this system is illustrated in Figure 6.

In Figure 6, the Kalman filter is employed to estimate the actual motion of the drone using inputs $d_{lr}$ and $v_{la}$. Considering the continuous and stable nature of $v_{la}$, the filter proves effective in line recognition failure or misrecognition scenarios. The filter’s output includes $y_K$ and the process state $X_K$. Additionally, $y_K$ is a component of the in-line selection, as introduced in Section 3.

Once the deviation feedbacks $X_K$ and $y_K$ are determined, the tracking target error $e_{iK}$ and the state error $E_K$ are calculated with the corresponding state targets $R_d$ and the position target $r_{dp}$. Finally, an integral SMC method calculates the auxiliary roll target ${\theta }_{Ar}$.

The subsequent sections will present a detailed discussion of the Kalman filter and the controller’s calculation process.

4.2. Deviation States Estimation

The outcome of line recognition, representing the real-time distance between the drone and the transmission line, is crucial for the efficacy of deviation control. However, environmental factors may cause this information to fail or be misrepresented, resulting in incorrect controller outputs and potentially impacting normal inspection operations. Consequently, the possibility of line recognition failure needs to be addressed.

A steady-state Kalman filter is employed to estimate the deviation distance to handle such situations, taking into account the lateral velocity of the drone.

Implementing the steady-state Kalman filter can be conceptualized as a linear system with multiple inputs. To realize this, knowledge of the system model is essential. Therefore, we first need to outline the dynamics of the deviation control system.

Given the context of inspection, where there is no significant attitude change (within 45 degrees) [29], and considering the system inputs ${\omega }_r$ and outputs, a second-order transfer function (4) proves sufficient for characterizing its properties [30,31,32].

$$G\left(s\right)=\frac{b_0}{s^2+a_{0}s+a_1}$$

(4)

The parameters $a_0$, $a_1$, and $b_0$ must be determined through fitting using actual flight data. This involves utilizing the target roll $\theta r$, the lateral velocity $vla$, and the line recognition result $d_{lr}$ data. Once obtained, these parameters ensure that $\theta r$ and $vla$ meet the required criteria.

$${\ddot{v}}_{la}=-a_0{\dot{v}}_{la}-a_1{v}_{la}+b_0{\theta }_r.$$

(5)

Let $X_K={[\dot{v_{la}},v_{la},p_{la}]}^T$, where ${\dot{p}}_{la}=v_{la}$ and $u={\theta }_r$. The deviation dynamics (5) can then be expressed in matrix form. Additionally, the integral of the lateral velocity is employed as the output.

$$\left\{\begin{array}{c}{\dot{X}}_K=A{X}_K+Bu,\\ y_K=C{X}_K.\end{array}\right.$$

(6)

with

$$\begin{array}{c}A=\left[\begin{array}{ccc}-a_0& -a_1& 0\\ 1& 0& 0\\ 0& 1& 0\end{array}\right],B=\left[\begin{array}{c}{b}_0\\ 0\\ 0\end{array}\right],C=\left[\begin{array}{ccc}0& 0& 1\end{array}\right].\hfill \end{array}$$

(7)

Considering the Gaussian noise w on the input and the output measurement noise v, the discrete deviation system can be expressed as:

$$\left\{\begin{array}{c}{X}_d(t+1)=\mathsf{\Phi }{X}_d\left(t\right)+\mathsf{\Gamma }u\left(t\right)+\mathsf{\Gamma }w\left(t\right),\\ Y_d\left(t\right)=H{X}_d\left(t\right)+v\left(t\right).\end{array}\right.$$

(8)

Here, $\mathsf{\Phi }$, $\mathsf{\Gamma }$, and H correspond to the discretized A, B, and C. The design of the steady-state Kalman filter involves state and measurement update processes, as illustrated in (9) and (10).

$${\widehat{X}}_d(t+1|t)=\mathsf{\Phi }{\widehat{X}}_d\left(t\right|t-1)+\mathsf{\Gamma }u\left(t\right),$$

(9)

$$\begin{array}{c}{\widehat{X}}_d\left(t\right|t)={\widehat{X}}_d\left(t\right|t-1)+M({\widehat{Y}}_d\left(t\right)-H{\widehat{X}}_d\left(t\right|t-1)),\\ {\widehat{Y}}_d\left(t\right|t)=H{\widehat{X}}_d\left(t\right|t),\end{array}$$

(10)

where M represents the optimal innovation gain utilized to minimize the estimation errors under the noise covariances $Q=E\left(w\left(t\right)w{\left(t\right)}^T\right)$ and $R=E\left(v\left(t\right)v{\left(t\right)}^T\right)$. Finally, ${\widehat{X}}_d$ represents the estimated deviation state, and ${\widehat{Y}}_d$ denotes the estimated deviation distance.

4.3. Integral SMC-Based Deviation Control Design

After completing the deviation distance estimation, a continuous and stable deviation feedback, $Y_K$, is obtained. This information is the basis for designing a deviation controller using an integral sliding mode control algorithm. The objective is to maintain the deviation distance at 0.3 m throughout the inspection, ensuring that the drone remains directly above the transmission line to capture information.

The controller design process assumes that the deviation state can be accurately estimated, i.e., $X_K={\widehat{X}}_K$ and $y_K={\widehat{y}}_K$. Given the inspection task’s goal of keeping the drone stably above the line at all times ($r_{dp}=0$ and $R_d={[0,0,0]}^T$), the tracking error is defined as $E_K=X_K-R_d=X_K$. The integral of deviation is introduced to enhance the accuracy of deviation control as an extended state of the deviation dynamics, as depicted in (11).

$$\begin{array}{c}{\dot{E}}_{ex}=\left[\begin{array}{c}{\dot{E}}_K\\ {\dot{e}}_{iK}\end{array}\right]=\left[\begin{array}{cc}A& 0_{3\times 1}\\ C& 0\end{array}\right]\left[\begin{array}{c}{E}_K\\ e_{iK}\end{array}\right]+\left[\begin{array}{c}B\\ 0\end{array}\right]u\hfill \\ =A_{ex}{E}_{ex}+B_{ex}u\hfill \end{array}$$

(11)

Design the sliding mode surface $\sigma =S{E}_{ex}$, and its derivative form can be expressed as

$$\dot{\sigma }=S{\dot{E}}_{ex}=S{A}_{ex}{E}_{ex}+B_{ex}u.$$

(12)

Here, $S=[s_0,s_1,s_2,s_3]$ represents the parameters of the sliding surface corresponding to $E_{ex}$. Assuming that the deviation state converges to the surface and remains on it, $\sigma =\dot{\sigma }=0$. Consequently, the equivalent control $u_{eq}$ can be calculated.

$$u_{eq}=-{\left(S{B}_{ex}\right)}^{-1}S{A}_{ex}{E}_{ex}=K_l{E}_{ex}.$$

(13)

To mitigate the chattering phenomenon, one can replace the traditional sign function with a smoother function

$$f\left(\sigma \right)=\frac{\sigma }{\left|\sigma \right|+\delta },$$

(14)

where $\delta$ is a small positive constant for avoiding singular results. The switching control $u_{nl}$ can be designed as

$$u_{nl}=K_{n}f\left(\sigma \right).$$

(15)

In (15), the switching gains A and B adjust the strength of the switching control. Finally, the output of the integral SMC-based deviation control is

$$\begin{array}{c}u=u_{eq}+u_{nl}\hfill \\ =K_l{E}_{ex}+K_{n}f\left(\sigma \right).\hfill \end{array}$$

(16)

5. Simulation and TLI Verification

5.1. Inspection Platform

To assess the effectiveness of the proposed scheme in practical applications, we developed a small quadcopter with a 247 mm wheelbase. The main sensors include an RGB camera and a GNSS receiver, as depicted in Figure 7.

In the lower part of Figure 7, the specific distribution of each component is illustrated in

Table 1. Primary relevant sensors.

Device	Model	Description
GNSS	M8N series [33]	The GNSS receiver is positioned atop the platform to maximize satellite signal reception and to mitigate the internal sensors’ geomagnetic interference.
Main Control Board	STM32F4 [34], IAR environment [35]	The main control board is positioned at the central location of the airframe.
Camera	OAK-1 MAX [36]	The RGB camera is mounted on the bottom of the drone.
Radar	PLK-LC2001l [37]	The mm-wave radar is mounted on the bottom of the drone near the camera.
Raspberry Pi	Compute Module 4 [38]	The Raspberry Pi core board is integrated beneath the main control board.

.

The GNSS receiver (a) provides the UAV with the raw position and velocity measurements (including the UAV lateral velocity to the LTAS).

The main control board (b) incorporates fundamental devices, such as the IMU, barometer, compass, and data logging module, enabling essential flight functionalities. It incorporates two STM32F4 chips, the primary chip handling communication and computation, while the secondary chip is responsible for data storage (flight logs and path recognition results recorded in the data logging module at 10 Hz).The proposed LTAS is developed in C within the IAR environment. It runs in conjunction with the flight control program (including the cascade control system) on the main chip of the control board.

The RGB camera (c) has the processing power of 4 TOPS and features onboard AI capabilities for image recognition.

The mm-wave radar PLK-LC2001l (d) detects the relative vertical distance between the drone and the transmission line. The measured distance is then used to convert the pixels and the actual altitude.

The Raspberry Pi (e) we selected has 4 GB of RAM and 32 GB of eMMC storage. To validate the feasibility of image processing, during experimentation, we displayed the Raspberry Pi’s processing frame rate, CPU utilization, and memory usage on the processed image output, as illustrated in Figure 8.

In the upper section of Figure 7, the inter-device data/control flow is elucidated, with both the LTAS and the cascaded control system implemented on the main control board. Various sensors, such as the GNSS and the IMU (inertial measurement unit), communicate via the CAN protocol with the main control board. These real-time data streams to the main control board undergo integrated navigation processing, yielding real-time feedback for the UAV, including information such as the UAV position coordinates, the horizontal body lateral velocity, the roll angle, the heading angle, etc.

Communication between the ground station and the remote controller signals is established through the data link and the SBus link, facilitating remote communication with the main control board. The ground station transmits waypoint information to the main control board while the remote controller adjusts the UAV waypoints using the SBus interface. The camera communicates with the Raspberry Pi through a serial port, transmitting the processed line information and original images to the Raspberry Pi.

The millimeter-wave radar communicates with the main control board through a serial port on the Raspberry Pi, transferring raw deviation data to the LTAS system within the main control board. The deviation control input is integrated into the cascaded control system following deviation state estimation and control by the deviation controller. The cascaded control system, through a multi-level conversion of control inputs, ultimately assigns corresponding PWM signals to each motor, empowering the propellers to propel the UAV.

5.2. Line Recognition Verification

During power transmission line inspections, it becomes challenging to capture detailed information if the camera is positioned too far from the transmission lines. Even if the system can identify the straight lines representing the transmission cables, the significance of the inspection is compromised. It hinders detecting risks like bird nests or ice accumulation on the lines. Consequently, the system cannot implement effective mitigation measures.

The LARGE version of MobileNetV3 is selected; its specific parameter settings are detailed in

Table 2. Training parameters.

Input	Operator	Exp	#out	SE	NL	s
${224}^2\times 3$	conv2d	–	16	–	HS	2
${122}^2\times 16$	bneck, 3×	16	16	–	RE	1
${122}^2\times 16$	bneck, 3×	64	24	–	RE	2
${56}^2\times 24$	bneck, 3×	72	24	–	RE	1
${56}^2\times 24$	bneck, 5×	72	40	✓ 1	RE	2
${28}^2\times 40$	bneck, 5×	120	40	✓	RE	1
${28}^2\times 40$	bneck, 5×	120	40	✓	RE	1
${28}^2\times 40$	bneck, 3×	240	80	–	HS	2
${14}^2\times 80$	bneck, 3×	200	80	–	HS	1
${14}^2\times 80$	bneck, 3×	184	80	–	HS	1
${14}^2\times 80$	bneck, 3 × 3	184	80	–	HS	1
${14}^2\times 80$	bneck, 3 × 3	480	112	✓	HS	1
${14}^2\times 112$	bneck, 3 × 3	672	112	✓	HS	2
${14}^2\times 112$	bneck, 5 × 5	672	160	✓	HS	1
$7^2\times 160$	bneck, 5 × 5	960	160	✓	HS	1
$7^2\times 160$	bneck, 5 × 5	960	160	✓	HS	1
$7^2\times 160$	conv2d, 1 × 1	–	960	–	HS	1
$7^2\times 960$	pool, 7 × 7	–	–	–	–	1
$1^2\times 960$	conv2d, 1 × 1	–	1280	–	HS	1
$1^2\times 1280$	conv2d, 1 × 1	–	k	–	–	1

¹ This symbol indicates that SE module has been enabled.

.

The dataset is augmented before training to enhance the segmentation effect, involving random rotation, random blurring, size adjustment, and image normalization. Random rotation aids in adapting to various wire orientations, and random blurring effectively addresses camera focusing issues, while size adjustment and normalization processing contribute to improved training efficiency.

During the model training process, a learning rate warm-up strategy is employed. Initially, the learning rate is set to a low value for a few epochs to facilitate a gradual convergence of the model towards stability. Subsequently, the predetermined learning rate is selected after ensuring the model has reached a relatively stable state. This approach accelerates the convergence of the model, leading to improved training effectiveness.

Stochastic gradient descent (SGD) is selected as the optimizer, and a hybrid loss function comprising the CrossEntropyLoss and the Lovasz Softmax loss is chosen. Throughout the training process, metrics such as the accuracy (ACC), the mean intersection over union (mIoU), and the loss values are monitored, as illustrated in Figure 9.

The parameters obtained from training are utilized for inference validation. Five scenarios were selected to assess the detection of transmission lines in different environments from varying perspectives. The results are presented in five columns, as shown in Figure 10. The first column displays the raw image, the second column showcases the result after processing the raw image with a median filter, and the third column illustrates the outcome of processing using the DL method.

The final results of different methods are presented in the fourth and fifth columns. The fourth column displays the processing outcomes using the median filtered, the Hough transformed, and the masked. The fifth column reveals the results obtained through the proposed solution.

Examining the red background images in Figure 10, it is evident that utilizing the DL method for wire extraction offers undeniable advantages, effectively excluding interference in the background compared to median filtering. The black background reveals the final wire extraction results. For scenarios 1, 2, 3, and 4, the proposed scheme exhibits a more vital ability to filter out interference, as background interference has minimal impact on the method. Even for scenario 5, where the transmission line is nearly integrated into the background, the proposed scheme can still accurately extract the target wire segments.

In Figure 11, a closer examination of scenarios 1, 2, and 4 from Figure 10 is presented, focusing on the recognition details. Scenario 1 in Figure 11 showcases the superior accuracy of the DL scheme compared to the edge detection methods. Scenario 2 highlights the potential issue with edge detection, where numerous straight lines unrelated to the transmission line are recognized, a challenge effectively addressed by the new scheme. Scenario 3 contrasts the limitations of the old method, which may identify the edge features of a house while overlooking the presence of the transmission line, in contrast to the new scheme’s accurate identification of the target.

Hence, the proposed line recognition method recognizes transmission lines and demonstrates superior anti-interference capabilities against disturbances such as road marking lines.

5.3. Deviation Controller Verification

After completing the line recognition test in a static environment, further validation of the deviation control in the LTAS is essential in a dynamic environment before the actual inspection task. The process involves acquiring the lateral motion dynamics of the drone and designing the estimator (8) and controller (16).

Referring to 5, where ${\theta }_r$ and $v_{la}$ are the inputs and outputs, respectively, the lateral velocities are obtained by allowing the drone to operate freely in an open outdoor environment. The required states ${\theta }_r$ and $v_{la}$ are recorded at 50 Hz, consistent with the control frequency.

Utilizing the Ident toolbox in MATLAB, a fitting operation is performed on the recorded data to obtain the dynamics. Subsequently, the estimator and controller can be designed. For the designed inspection platform, the relevant parameters in the deviation control are listed in

Table 3. Parameters in deviation control.

Components	Parameter
Lateral dynamics (5)	$a_0$ = 2.48, $a_1$ = 3.901, $b_0$ = 31.96
Estimator (8)	$\begin{array}{c}Q=0.1,R=\left[\begin{array}{cc}0.001& 0\\ 0& 0.01\end{array}\right]\hfill \end{array}$
Integral SMC (16)	$\begin{array}{c}{K}_l=[-0.0052,-0.3815,-0.4804,-0.1331],\hfill \\ K_n=-0.0407,\delta =0.05\hfill \end{array}$

.

For safety considerations, a 10-kilovolt transmission line environment with a lower height is chosen to verify the deviation control, as depicted in Figure 12. The inspection process aligns with previous work [27]: The drone autonomously takes off and elevates above the tower. Subsequently, the pilot operates the drone to maintain a position directly above the transmission line at the task’s starting point. The inspection flights commence, and the proposed LTAS is activated simultaneously. The entire experiment is conducted under the supervision of electric utility personnel.

The drone’s flight trajectory is illustrated in Figure 13, with the effect of the LTAS highlighted by a red curve. Throughout this process, the estimator’s estimate of the lateral state is depicted in Figure 14. The pilot initially maneuvers the drone to the starting position. Subsequently, after reaching the endpoint, the drone returns to its take-off position.

Figure 14 displays the estimation of the lateral state throughout the inspection process, encompassing the dynamic acceleration, the velocity, the real-time position, and the position with delay (line recognition result). Specifically, Figure 14b,d present the estimation of the lateral velocity and recognition results, respectively, allowing for a comparison with the actual measurement for verification. In Figure 14b, it is evident that the estimated velocity closely aligns with the actual measurement result $v_{la}$. The position estimation results in Figure 14d align with the actual situation, effectively mitigating the influence of noise on the measurement results. Consequently, the implemented Kalman filter realistically responds to the lateral dynamics, and the estimated real-time position deviation $Y_K$ can be further utilized for line selection in line recognition.

If the effective field of view is defined as the short edge from the center of the power line to the image edge, multiplied by 2, divided by the total image width, the value during normal operation should be no less than 30%, and during stable operation, it should be no less than 70%. Based on the selected camera’s horizontal field of view (68°), the inspection height (2 m), and the intended allowable free roll angle (15°), considering the deviation distance of the unmanned aerial vehicle from the power line, the normal working range should be within ±0.48 m, and the stable working range should be within ±0.21 m.

Using integral SMC, the drone maintains a consistent position directly above the transmission line with a target deviation of 0.1 m during flight. The tracking effect of the drone is illustrated in Figure 15.

Figure 15a illustrates the deviation distances observed during the tracking process. It is evident that the distance between the UAV and the transmission line consistently remains within 0.2 m, ensuring continuous visibility of the entire transmission line for the camera. The outcomes of the controller computations are presented in Figure 15b, where the blue segment represents the total output, and the light blue color designates the calculated nonlinear component. This validation affirms that the designed integral sliding mode control (ISMC) maintains the UAV directly above the transmission line. Consequently, it can be inferred that the proposed LTAS enhances the UAV’s performance in inspection tasks.

5.4. Verification in TLI

The inspection flight validation occurred near a pole tower in the Tomitsu Prefecture, Japan, depicted in Figure 16. The tower’s height is 41 m, and the ground line connecting the electricity pylon is the inspection target. Throughout the validation process, the total length of the transmission line under inspection is 137 m, with a geographic direction of 3 degrees. Notably, in this wild and open environment, the ground wind reaches a maximum of 4.6 m/s, presenting actual interference to the drone during inspection. Importantly, the entire validation process was conducted safely under the supervision of power practitioners.

The inspection process mirrors the procedure outlined in Section 5.3. Following the pilot’s operation to position the drone directly above the transmission line, line recognition and deviation control mechanisms come into play, aiding in the precise execution of TLI. The drone’s recorded flight trajectory and the recognized deviations are presented in Figure 17.

In Figure 17, the drone’s flight trajectory is depicted in Figure 17a, with the blue curves representing the drone’s flight directly above the transmission line and after the inspection is completed. The red segment illustrates the trajectory during the inspection above the transmission line. Throughout the inspection process, the angle and position deviations between the drone and the transmission line are recorded as in Figure 17b and Figure 17c, respectively. The recorded results indicate that the ${\varphi }_{lr}$ deviation reaches a maximum of only 3 degrees and consistently stays within $\pm 2$ degrees from the transmission line throughout the process. Notably, between 254.2 and 261.4 s, a change in the drone’s heading angle occurred due to external interference, promptly corrected, as demonstrated in Figure 18. The corresponding time points and the deviation data for each image are detailed in

Table 4. Corrective performance under strong interference.

Serial Number	Time (Second)	${\mathit{\varphi }}_{\mathit{l}\mathit{r}}$ (Degree)	${\mathit{Y}}_{\mathit{K}}$ (Meter)
1	254.2	1	0.2
2	255.2	0	0.19
3	256.9	−1	0.14
4	257.9	−2	0.0
5	258.9	−3	−0.08
6	259.1	−2	−0.104
7	259.8	−1	−0.128
8	260.4	0	−0.16
9	261.4	0	−0.2

.

The real-time correction of the positional deviation between the drone and the transmission line is evident in Figure 17c. Although the performance is somewhat degraded compared to the inspection of the 10-kilovolt transmission line, primarily due to wind interference at higher altitudes, the recorded results demonstrate that the deviation $Y_K$ is consistently maintained within the range of $\pm 0.2$ m throughout the inspection process. The mean value of the distance deviation is −0.01 m, ensuring that the tracked target transmission line remains consistently within the camera’s capture range.

The successful verification of the proposed line-tracking assistive system in a real high-voltage tower environment underscores its practical applicability to actual TLI work. Moreover, its adaptability to any inspection platform, leveraging low-cost sensors for proximity information acquisition of the transmission line, makes it a versatile and cost-effective solution.

6. Conclusions

This work focuses on the inspection tasks of transmission lines in UAV power inspections, utilizing RGB cameras, millimeter-wave radar, and GNSS receivers. We designed a novel line tracking assistance system consisting of three components: transmission line recognition, a deviation state estimator, and a deviation controller. To address the challenge of complex backgrounds in transmission line images, we employ the MobileNetV3 network for line recognition. Furthermore, leveraging millimeter-wave radar, we calculate the relative horizontal distance between the line and the UAV. Combining this information with navigation data from the GNSS receiver, the deviation state estimator provides more accurate deviation information. Lastly, based on the deviation information, the deviation controller, using a sliding mode algorithm, computes auxiliary attitude targets to ensure that the UAV consistently follows the transmission line during the inspection. The proposed solution is deployed on our in-house UAV platform and validated through flight tests in energized environments with 10-kilovolt and 110-kilovolt power towers. The experimental results confirm the effectiveness and practical feasibility of the proposed LTAS.

Additionally, the experimental outcomes reveal some limitations of the proposed solution. Variances in the along-line flight performance are observed in different working environments due to the simplistic linear model used in the deviation controller design, which does not account for external disturbances. Consequently, our future focus will be on improving the deviation controller to enhance the system’s versatility in diverse environments.