Airborne Streak Tube Imaging LiDAR Processing System: A Single Echo Fast Target Extraction Implementation

Abstract

In this paper, a ground target extraction system for a novel LiDAR, airborne streak tube imaging LiDAR (ASTIL), is proposed. This system depends on only a single echo and a single data source, and can achieve fast ground target extraction. This system consists of two modules: Autofocus SSD (Single Shot MultiBox Detector) and post-processing. The Autofocus SSD proposed in this paper is used for object detection in the ASTIL echo signal, and its prediction speed exceeds that of the original SSD by a factor of three. In the post-processing module, we describe in detail how the echoes are processed into point clouds. The system was tested on a test set, and it can be seen from a visual perspective that satisfactory results were obtained for the extraction of buildings and trees. The system mAP^IoU=0.5 is 0.812, and the FPS is greater than 34. The results prove that this ASTIL processing system can achieve fast ground target extraction based on a single echo and a single data source.

1. Introduction

Airborne LiDAR is an active ground observation system, which has the advantages of round-the-clock, strong penetration, accurate range finding, and short production cycle [1,2]. Since its ability to quickly collect 3D terrain data in local areas [3], it has been widely used in urban 3D modeling [4,5], forestry resources survey [6,7], power facility monitoring [8,9], disaster assessment [10,11], and other applications [12,13,14]. Airborne streak tube imaging LiDAR (ASTIL) is a novel LiDAR that was originally applied for underwater object detection [15], and is still not widely available.

It is generally known that the data format obtained by conventional LiDAR based on single-point scanning is generally a point cloud. Most post-processing algorithms based on point clouds generally demand a certain scale of points to obtain reliable results. Taking the filtering algorithm based on adaptive TIN (triangular irregular network) proposed by Axelsson [16] as an example, the local lowest point within a user-defined grid is selected as the seed point, and the grid size should not be smaller than the size of the largest structure. This requirement for the number of points makes the real-time processing of a single echo signal from conventional airborne LiDAR a difficult problem to solve.

Compared with the conventional LiDAR, the laser footprint from ASTIL is shaped into a strip that is hundreds of meters long, and a streak tube is used to collect the echo signal [17]. These configurations result in the ASTIL having the advantages of wide field of view and high info acquisition efficiency. Because of the idiosyncratic working mechanism of ASTIL, its raw echo signal is a two-dimensional single-channel digital image. This kind of echo signal is rich in semantic information and is capable of reflecting the cross-section information of surface objects in the irradiated area [18]. Therefore, it has the potential to directly identify the target only by relying on a single echo and a single data source. However, the direct application of the ASTIL raw echo for ground objects extraction has not received sufficient attention [17,18].

In terms of data processing, S. Zhang [19] presented the solution that geospatial artificial intelligence applies deep learning techniques to help solve complex detection and classification problems. Currently, the state-of-the-art deep learning benchmark frameworks in the field of object detection mainly include Faster RCNN, YOLO (You Only Look Once) [20], SSD (Single Shot MultiBox Detector) [21], and their derivatives [22]. Faster RCNN is the first deep learning framework to implement end-to-end object detection [23]. The most prominent contribution of this framework is the proposed RPN (Region Proposal Network), which associates the proposal region generation and the convolutional network through the anchor mechanism, with high accuracy [24,25]. This framework with RPN is called two-stage algorithms, and the prediction accuracy is higher because of the introduction of RPN, however, at the cost of a decrease in inference speed [26]. One-stage algorithms abandon the time-consuming component RPN and treat the detection task as a regression problem, such as YOLO, SSD [27]. SSD is a new object detection framework proposed by W. Liu et al. [21] in 2015. It is considered as the second one-stage object detection framework in the deep learning era [28]. SSD predicts objects of different scales from feature maps of different scales, achieving high detection accuracy. It uses small convolutional filters applied to feature maps to predict category scores and box offsets for a fixed set of default bounding boxes. On the Pascal VOC2007 test dataset, the inference speed of the SSD300 model is as high as 59 FPS, which significantly outperforms YOLOV1 in terms of speed and accuracy. However, the above frameworks were all proposed for RGB real scene images. For ASTIL echo signals, the semantic information is simpler and has fewer categories, and ASTIL has higher real-time requirements for signal processing. As a result, the original SSD network cannot be suitable for the fast processing of ASTIL echo, and needs to be structurally optimized (see Section 3.1).

In this paper, we propose an ASTIL processing system. This system only needs a single echo and a single data source to achieve fast ground target extraction, which gives ASTIL the potential for real-time ground target extraction. This system consists of two modules: an object detection module and a post-processing module. In the object detection module, we structurally optimize the original SSD according to the ASTIL signal characteristics, and propose a Autofocus SSD to speed up the prediction speed. In the post-processing module, we show in detail how the signal is processed into a point cloud. Then, we conducted an experiment to compare the performance of the Autofocus SSD and other state-of-the-art networks. An ablation experiment was executed, and the optimal base network structure was explored. Finally, we tested the overall performance of the system.

2. Background Knowledge

2.1. Airborne Streak Tube Imaging LiDAR

ASTIL adopts the “pendulum” scanning mode, as shown by the yellow dotted line in Figure 1a, and the scanning trajectory is zigzag along the flight direction of the flight vehicle. Figure 1b shows the scanning trajectory of the ASTIL laser footprint in a certain scene. Some trees and buildings are located within the scanned area. Here, F₁ to F₄ respectively represent laser footprints irradiated on different objects. Echo signals of footprint F₁ to F₄ are shown in Figure 2.

As can be seen from Figure 2, the echo morphological features from buildings and trees in the ASTIL echo signal are obviously different. The echo from a building is generally stronger and has two scarps adjacent to the echo from the ground. The echo from trees is usually low in intensity and appears as a diffuse point cloud. Therefore, two kinds of ground objects in the ASTIL echo signal can be extracted according to their different echo morphological features.

2.2. Data Collection and Annotation

The data used for training and test sets were collected near Hanzhong City, Shaanxi Province, China. Figure 3 shows a typical scene of the location. In this scene, many buildings and trees are irregularly staggered, and this scene can represent the distribution pattern of ground objects in most urban low-rise residential areas. As a result, the collected data has a good representativeness for the urban low-rise residential area.

At the phase of data collection, the aircraft was flying at an altitude of about 3000 m above sea-level, and ASTIL’s laser repetition frequency was 1000 Hz. More details of the data collection are shown in

Table 1. Data collection details.

Item	Configuration Info
acquisition platform	Harbin Y-12 (fixed-wing aircraft)
laser wavelength	532 nm
echo type	waveform sampling
training set acquisition time	2014.8.18 14:00:00
test set acquisition time	2014.8.18 13:41:38

. The ASTIL irradiated laser footprint on the ground surface was about 130 m long and 0.5 m wide. After data collection, we annotated these data using the tool, LabelImg. The specific details of various data sets are shown in

Table 2. Details of various data sets.

Objects	Raw Training Set	Training Set	Test Set
tree	9249	11,223	18,537
building	28,124	11,086	14,018

. The class-imbalance is caused by the large difference in the number of categories of buildings and trees in the raw training set. To address this, we flipped the echo signal containing only trees up and down, and down sampled the signal containing buildings to get the final training set.

3. ASTIL Echo Signal Fast-Processing System

The overall framework of the ASTIL fast-processing system is shown in Figure 4, which mainly includes two modules: object detection and post-processing. More details are given in the following sections.

3.1. Autofocus SSD Network

3.1.1. Hierarchical Setting of Default Box Size Based on K-Means

In the SSD network, default boxes with suitable sizes can not only shorten the regression time, but also improve the prediction accuracy of the model. The default box size applied in the original SSD is specifically designed for real scene images. However, the objects in these real scene images are not the same size as those in the ASTIL echo signals. The k-means clustering was used to calculate default box sizes. We write $g_i^u$ as an indicator for the u-th parameter of the i-th ground truth box size. Here, ground truth box parameters only contain height and width, $n=2$. The distance measure is Euclidean distance, as defined in (1).

$$dist(g_i,g_j)=\sqrt{{\displaystyle \sum _{u=1}^n{\left|g_i^u-g_j^u\right|}^2}}$$

(1)

Ground truth box sizes is clustered into eight clusters, and the results are shown in Figure 5. After rounding the clustering results, sizes of the default boxes are (30, 32), (49, 49), (56, 75), (96, 67), (63, 111), (123, 101), (83, 302), and (71, 174).

3.1.2. Architecture

Different from real scene images, ASTIL echo signals have the characteristics of concentrated echo distribution and straightforward semantic information. In view of these characteristics, we improve the SSD network and propose a Autofocus SSD network. Figure 6 shows the overall architecture of the network. Here, “IRB” represents the inverted residual module in the MobileNetV2 [29]. ‘CONV’, ‘Maxpool’, ‘FC’, and ‘BN’ denote convolutional layer, maximum pooling layer, fully connected layer, and batch normalization layer, respectively. F1 to F4 represent feature maps. ‘LOC’ and ‘CONF’ denote the location and confidence prediction modules of the model, respectively. ‘SRE’ is the signal region extractor and ‘SRE backbone’ is the feature extraction network, and these two modules constitute the signal region extraction unit proposed in this paper.

In order to meet the demand of fast processing of echo signals, we improved the original SSD by adding a signal region extraction unit and simplifying the base network, which improves the prediction speed.

3.1.3. Signal Region Extraction Unit

The original SSD network extracts feature maps by the base network and some CBR modules, and then predicts position and confidence on each cell of each output feature map. Write the convolution filter as $\mathrm{Conv}(c_{\mathrm{input}},c_{\mathrm{output}},k,s,p)$, where $c_{\mathrm{input}}$, $c_{\mathrm{output}}$, $k$, $s$ denote the input channel, output channel, kernel size, and stride, respectively, and $p$ refers to the number of zeros padded around the input tensor.

For input channel $c$, the number of default boxes on the feature map $n_{\mathrm{dbox}}$, the location prediction module ‘LOC’ is written as $ℒ$, can be expressed as (2). Here, ${𝒯}_{\xi }$ is defined as a tensor dimensional transformation mapping shown in (3). Suppose the single-scale feature map (such as F1 in Figure 6) ${\mathit{F}}_{\mathrm{s}}\in {ℝ}^{N\times c\times h\times w}$, because there are four regression parameters for ground-truth box, here $\xi$ is set to 4. $ℒ({\mathit{F}}_{\mathrm{s}})$ is also considered as a linear transformation as shown in (4).

$$ℒ=\left[\mathrm{Conv}(c,4\cdot n_{\mathrm{dbox}},(3,3),1,1)\circ {𝒯}_4\right]$$

(2)

$${𝒯}_{\xi }:{ℝ}^{N\times c\times h\times w}\to {ℝ}^{N\times \xi \times \frac{c\cdot h\cdot w}{\xi }}$$

(3)

$${ℝ}^{N\times c\times h\times w}\to {ℝ}^{N\times 4n_{\mathrm{dbox}}\times h\times w}\to {ℝ}^{N\times 4\times hw{n}_{\mathrm{dbox}}}$$

(4)

Similarly, for the categories number $cls$, the confidence prediction module ‘CONF’ $\mathcal{C}$, can be expressed as (5). $\mathcal{C}({\mathit{F}}_{\mathrm{s}})$ is a linear transformation as shown in (6).

$$\mathcal{C}=\left[\mathrm{Conv}(c,(cls+1)\cdot n_{\mathrm{dbox}},(3,3),1,1)\circ {𝒯}_{cls+1}\right]$$

(5)

$${ℝ}^{N\times c\times h\times w}\to {ℝ}^{N\times (cls+1)\cdot n_{\mathrm{dbox}}\times h\times w}\to {ℝ}^{N\times (cls+1)\times hw{n}_{\mathrm{dbox}}}$$

(6)

Therefore, for multi-scale feature maps $\mathit{F}=({\mathit{F}}_1,{\mathit{F}}_2,{\mathit{F}}_3,{\mathit{F}}_4)$, ${\mathit{O}}_{\mathrm{loc}}$, ${\mathit{O}}_{\mathrm{conf}}$ denotes the predicted output tensor with respect to the location and confidence, then (7) and (8) are established.

$${\mathit{O}}_{\mathrm{loc}}\in {ℝ}^{N\times 4\times n_{\mathrm{dbox}}\cdot {\displaystyle \sum _{i=1}^4{h}_i{w}_i}}$$

(7)

$${\mathit{O}}_{\mathrm{conf}}\in {ℝ}^{N\times (cls+1)\times n_{\mathrm{dbox}}\cdot {\displaystyle \sum _{i=1}^4{h}_i{w}_i}}$$

(8)

The raw echo signal (Figure 2) size of ASTIL is 500 × 1000 (height × width). However, as the signal region extraction in Figure 6 shows, those streaks indicating the signal, called signal streaks, are concentrated in only one part of the image (yellow box), and the information in other areas is not substantially useful for the processing of the echo signal. In view of this, the signal region extraction unit (SREU) was proposed.

The SREU is composed of a signal region extraction backbone (SRE backbone) and a signal region extractor (SRE). The SRE backbone is intercepted from the first three layers of the MobileNetV2 network, however, we adjusted the stride of the third layer to (2,2). It is specifically composed of CBR, IRB1_1, and IRB1_2 (See the SRE backbone in Figure 6). In the other hand, the SRE (see SRE in Figure 6) was set up with two channels, the input of channel a is the feature map extracted from the SRE backbone, and the input of channel b is the raw echo signal. The feature map entering channel a is extracted features by two convolutional and pooling layers, after which the predicted signal center is output by a fully connected layer and a sigmoid activation function. Then, the SRE expands 150 pixels left and right based on the predicted signal center, and the signal region is obtained.

Table 3. SRE specific configuration.

Operator	${\mathit{c}}_{\mathrm{input}}$	${\mathit{c}}_{\mathrm{output}}$	$\mathit{k}$	$\mathit{s}$	$\mathit{p}$
CONV	24	16	(3, 3)	1	none
Maxpool	--	--	(3, 3)	--	--
CONV	16	8	(3, 3)	1	none
Maxpool	--	--	(3, 3)	--	--
FC	624	1	--	--	--

shows the specific configuration of the SRE. The signal center value is clamped to [150, 850] to prevent cropping beyond the image.

After the input image is processed by SREU, the size is cropped from raw 500 × 1000 to 500 × 300 with almost no information loss. After that, suppose the input $\mathit{I}\in {ℝ}^{N\times c\times h\times w}$ , output ${\mathit{O}}_{\mathrm{loc}}^{\mathrm{SREU}}$ and ${\mathit{O}}_{\mathrm{conf}}^{\mathrm{SREU}}$, which are processed by SREU, become (9) and (10).

$${\mathit{O}}_{\mathrm{loc}}^{\mathrm{SREU}}\in {ℝ}^{N\times 4\times \frac{3}{10}{n}_{\mathrm{dbox}}\cdot {\displaystyle \sum _{i=1}^4{h}_i{w}_i}}$$

(9)

$${\mathit{O}}_{\mathrm{conf}}^{\mathrm{SREU}}\in {ℝ}^{N\times (cls+1)\times \frac{3}{10}{n}_{\mathrm{dbox}}\cdot {\displaystyle \sum _{i=1}^4{h}_i{w}_i}}$$

(10)

According to the SSD working mechanism, ${\mathit{O}}_{\mathrm{loc}}^{\mathrm{SREU}}$ and ${\mathit{O}}_{\mathrm{conf}}^{\mathrm{SREU}}$ are used for post-processing processes such as non-maximum suppression. Compared to and ${\mathit{O}}_{\mathrm{loc}}$, and ${\mathit{O}}_{\mathrm{conf}}$, the data size of ${\mathit{O}}_{\mathrm{loc}}^{\mathrm{SREU}}$ and ${\mathit{O}}_{\mathrm{conf}}^{\mathrm{SREU}}$ is reduced to 3/10 of original outputs.

3.1.4. Streamlining the Base Network

The original SSD is mainly applied to real scene images, and its base network is the truncated VGG16. However, the ASTIL echo signal possesses simple semantic information and small pattern space compared with real scene images, and the use of complex VGG networks does not achieve fast prediction. Therefore, the base network of the framework proposed adopted the truncated MobileNetV2 with simple structure and low number of parameters for fast signal processing. The specific configuration is shown as the base network in Figure 6. We only intercepted the first 11 layers of MobileNetV2 and adjusted the stride of the convolutional filter in the second CBR in the seventh inversed residual structure to (1, 1) to get feature maps at the appropriate scale. For the exploration of the base network structure, see Section 4.3.3.

3.1.5. Loss Function

Let $x_{ij}^k=\left\{0,1\right\}$ denotes the $i$-th default box matching to the $j$-th ground truth box of category $k$. When $x_{ij}^k$ is 1, it means that the both have a matching relationship and vice versa. In this research, the overall objective loss function is a weighted sum of the localization loss, the confidence loss, and the SREU prediction loss:

$$\begin{array}{l}L(x,s,l,g,{\widehat{c}}^{\mathrm{sig}},c^{\mathrm{sig}})\\ =\frac{1}{N_{\mathrm{dbox}}}(L_{\mathrm{loc}}(x,l,g)+L_{\mathrm{conf}}(x,s))+10\cdot L_{\mathrm{SREU}}({\widehat{c}}^{\mathrm{sig}},c^{\mathrm{sig}})\end{array}$$

(11)

where $L_{\mathrm{loc}}(x,l,g)$ is the localization loss between the predicted bounding box ($l$) and ground truth box ($g$), and $L_{\mathrm{conf}}(x,s)$ is the confidence loss. $s$ denotes the confidence score. ${\widehat{c}}^{\mathrm{sig}}$ and $c^{\mathrm{sig}}$ are the signal center predicted by SREU and the ground truth signal center, respectively. $N_{\mathrm{dbox}}$ is the number of matched default boxes. Similar to SSD, we regress the offsets for the center $(\mathrm{cx},\mathrm{cy})$ of the default bounding box ($d$) and for its width ($\mathrm{w}$) and height ($\mathrm{h}$), refer to Equation (12). $l_i^m$ is the regression parameters predicted by the i-th bounding box. ${\widehat{g}}_j^m$ is the regression parameters generated by the i-th bounding box matching to the j-th ground truth box.

$$\begin{array}{l}{L}_{\mathrm{loc}}(x,l,g)={\displaystyle \sum _{i\in \mathrm{Pos}}^{N_{\mathrm{dbox}}}{\displaystyle {\sum }_{m\in \left\{\mathrm{cx},\mathrm{xy},\mathrm{w},\mathrm{h}\right\}}{x}_{ij}^k{\mathrm{smooth}}_{\mathrm{L}1}}}(l_i^m-{\widehat{g}}_j^m)\\ \begin{array}{cc}{\widehat{g}}_j^{\mathrm{cx}}=(g_j^{\mathrm{cx}}-d_i^{\mathrm{cx}})/d_i^{\mathrm{w}}& {\widehat{g}}_j^{\mathrm{cy}}=(g_j^{\mathrm{cy}}-d_i^{\mathrm{cy}})/d_i^{\mathrm{h}}\end{array}\\ \begin{array}{cc}{\widehat{g}}_j^{\mathrm{w}}=\log (g_j^{\mathrm{w}}/d_i^{\mathrm{w}})& {\widehat{g}}_j^{\mathrm{h}}=\log (g_j^{\mathrm{h}}/d_i^{\mathrm{h}})\end{array}\end{array}$$

(12)

Especially, the confidence loss of object recognition is

$$L_{\mathrm{conf}}(x,s)=-{\displaystyle \sum _{i\in \mathrm{Pos}}^{N_{\mathrm{dbox}}}{x}_{ij}^k\log ({\widehat{s}}_i^k)-{\displaystyle \sum _{i\in \mathrm{Neg}}\log ({\widehat{s}}_i^0)}}$$

(13)

where ${\widehat{s}}_i^k=\exp (s_i^k)/{\displaystyle {\sum }_k\exp (s_i^k)}$.

The prediction loss of the SREU is shown in Equation (14).

$$L_{\mathrm{SREU}}({\widehat{c}}^{\mathrm{sig}},c^{\mathrm{sig}})=\left|{\widehat{c}}^{\mathrm{sig}}-c^{\mathrm{sig}}\right|$$

(14)

3.2. Post-Processing of the Echo Signal

The ASTIL echo signal only reflects the cross-sectional outline of the surface of the ground object within the laser irradiation area (150 m × 0.5 m). For common ground objects, multiple echo signals are required to complete the detection of the object as a whole. In order to correctly map the complete surface morphology and geographic location of objects, it is necessary to convert the echo signals into a point cloud format. This section proposes a pipeline for converting ASTIL echo signal to a point cloud.

3.2.1. Signal Centroid Extraction

The horizontal coordinate of each pixel of the signal streak in the ASTIL echo signal reflects the distance information of the target, whereas the vertical coordinate of the pixel reflects the spatial location information of the target. Only by accurately obtaining the distance and position information of the pixels representing the target can they be inverted into point clouds using the imaging model combined with pos information.

Signal centroid extraction is similar to the skeleton of signal streak, as shown in Algorithm 1. The method of extracting the signal centroid for each row is as follows. First, find the position ($inde{x}_{max}$) and its value ($valu{e}_{max}$) of the pixel with the largest value in that row. If there is more than one equal maximum value, the leftmost pixel is used as the maximum pixel. Second, the start and end positions ($po{s}_{start}$ and $po{s}_{end}$) of the signal streak larger than the signal threshold ($threshol{d}_{sig}$) are found by traversing to the left and right with the $inde{x}_{max}$, and the signal width ($widt{h}_{sig}$) is calculated. If the $valu{e}_{max}$ is greater than the $threshol{d}_{sig}$ and the $widt{h}_{sig}$ is greater than the $threshol{d}_{width}$, the signal centroid in this row can be calculated as $(po{s}_{start}+po{s}_{end})/2$. Otherwise, no signal exists in this row.

Algorithm 1: Signal centroid extraction

Input: Extracted echo signal $\mathit{I}\in {ℝ}^{500\times 1000}$   Output: Echo streak centroid $\mathit{O}\in {ℝ}^{500}$ function calc_line(line_data, thresholdsig, thresholdwidth):   indexmax $\leftarrow$ argmax(line_data)   valuemax $\leftarrow$ max(line_data)   indexstart $\leftarrow$ 0   indexend $\leftarrow$ 1000   for index = indexmax to indexstart do     if line_data[index] > thresholdsig then       posstart = index     else       break     end if   end for   for index = indexmax to indexend +1 do      if line_data[index] > thresholdsig then       posend = index     else       break     end if   end for   widthsig = posend − posstart + 1   if valuemax > thresholdsig and widthsig > thresholdwidth then     out_x = (posstart + posend)/2     return out_x   end if function main($\mathit{I}$, thresholdsig, thresholdwidth):   $\mathit{O}$$\leftarrow$ [ ]   foreach line_data $\in$ $\mathit{I}$ do     $\mathit{O}$.append(calc_line(line_data, thresholdsig, thresholdwidth))   end foreach   return $\mathit{O}$

3.2.2. Calibration

Due to the structural characteristics of STIU, the signal it collects is distorted. In application scenarios, it is necessary to use a calibration matrix to reduce the errors introduced by the distortion. The implementation details of calibration are as Equation (15). Here, ${\mathit{I}}_{}^{\mathrm{cali}}\in {ℝ}^{\iota \times 2}$ is the input matrix, ${\mathit{O}}_{}^{\mathrm{cali}}\in {ℝ}^{\iota \times 2}$ is the corresponding output matrix, $\iota \in \left[0, 500\right)$. ${\mathbf{A}}_{}^{\mathrm{cali}\_\mathrm{x}},{\mathbf{A}}_{}^{\mathrm{cali}\_\mathrm{y}}\in {ℝ}^{500\times 1000}$ are the horizontal and vertical calibration matrices, respectively. $A_{i,j}$ denotes the $i,j$ elements of matrix $\mathit{A}$, and ${\mathit{A}}_{i,:}$ denotes the i-th row of matrix $\mathit{A}$.

$$\begin{array}{l}{\mathit{O}}_{\phi ,:}^{\mathrm{cali}}=\{\begin{array}{cc}\left({\mathrm{A}}_{\mathrm{index}}^{\mathrm{cali}\_\mathrm{y}},{\mathrm{A}}_{\mathrm{index}}^{\mathrm{cali}\_\mathrm{x}}\right)& \lceil I_{\phi ,1}^{\mathrm{cali}}\rceil =\lfloor I_{\phi ,1}^{\mathrm{cali}}\rfloor \\ \left({\mathrm{A}}_{\mathrm{index}}^{\mathrm{cali}\_\mathrm{y}},\mathrm{lerp}({\mathit{I}}_{\phi ,:}^{\mathrm{cali}},{\mathbf{A}}^{\mathrm{cali}\_\mathrm{x}})\right)& \lceil I_{\phi ,1}^{\mathrm{cali}}\rceil \ne \lfloor I_{\phi ,1}^{\mathrm{cali}}\rfloor \end{array}\\ \mathrm{index}={\mathit{I}}_{\phi ,:}^{\mathrm{cali}}\end{array}$$

(15)

$$\mathrm{lerp}((j,i),\mathit{A})=A_{j,\lfloor i\rfloor }+(A_{j,\lceil i\rceil }-A_{j,\lfloor i\rfloor })\cdot \frac{i-\lfloor i\rfloor }{\lceil i\rceil -\lfloor i\rfloor }$$

(16)

3.2.3. Data Fusion

Data fusion refers to the operation of fusing echo signals with global positioning system (GPS), inertial measurement unit (IMU), and scan angle data. This information is an indispensable part of the point cloud inversion. However, these data originate from devices with different acquisition frequencies. In this experiment, the acquisition frequency of GPS and IMU is 200 Hz, and the counterpart of ASTIL is 1000 Hz. For the data fusion, POS were interpolated to coincide with the ASTIL acquisition frequency. The POS data acquired per second can be expressed as:

$$\begin{gathered} \mathit{D}={\left({\mathit{d}}_{}^{{\mathrm{t}}_0},\cdots {\mathit{d}}_{}^{{\mathrm{t}}_{99}}\right)}^{\mathrm{T}},\hspace{0.17em}\hspace{0.17em}\mathit{D}\in {ℝ}^{200\times 7} \\ {\mathit{d}}^t=(t,d_t^{\mathrm{longitude}},d_t^{\mathrm{latitude}},d_t^{\mathrm{altitude}},d_t^{\mathrm{roll}},d_t^{\mathrm{pitch}},d_t^{\mathrm{yaw}}) \end{gathered}$$

where $t$ indicates the acquisition moment and the meanings of other variables are shown in the superscripts. The interpolation method is as in Equation (17). Here, $t^{\mathrm{echo}}$ is the acquisition moment of the echo signal, ${\mathit{D}}_{\phi ,i}$ denotes the $i$-th element of the $\phi$-th element in $\mathit{D}$. The relationship between $t^{\mathrm{echo}}$, ${\mathit{D}}_{\phi }$, and ${\mathit{D}}_{\phi +1}$ is as in Equation (18).

$$\mathrm{lerppos}({\mathit{D}}_{\phi },{\mathit{D}}_{\phi +1},t^{\mathrm{echo}})=\frac{t^{\mathrm{echo}}-{\mathit{D}}_{\phi ,0}^{}}{{\mathit{D}}_{\phi +1,0}^{}-{\mathit{D}}_{\phi ,0}^{}}\cdot \left({\mathit{D}}_{\phi +1}^{}-{\mathit{D}}_{\phi }^{}\right)+{\mathit{D}}_{\phi }^{}$$

(17)

$$t^{\mathrm{echo}}\in \left[{\mathit{D}}_{\phi ,0},{\mathit{D}}_{\phi +1,0}\right)$$

(18)

4. Experiment and Results

The ASTIL echo signal fast-processing system was implemented with Python, PyTorch, OpenCV, NumPy, and Open3D. The experimental platform configuration is Windows 10, AMD Ryzen 5 5600G, 32 GB of RAM, and Nvidia RTX A4000.

4.1. Network Training Strategy

4.1.1. Autofocus SSD

The base network for the Autofocus SSD accepts the MobileNetV2 loaded with pre-trained weights provided by PyTorch. Unlike original SSD, no data augmentation techniques were used during training. Two default boxes were set per feature layer. Specifically, default boxes sizes in the first feature layer are (30, 32) and (49, 49), and the second layer are (56, 75) and (96, 67), the third layer are (63, 111) and (123, 101), and the fourth layer are (83, 302) and (71, 174). The training was divided into two stages. First, the base network and extra feature layers were frozen, and the SREU branch was trained for 10 epochs. At this time, the loss is only the $L_{\mathrm{SREU}}$ component in (11). After that, the branch SREU, base network, and extra feature layers were trained for 50 epochs. The total loss at this stage is (11). At each stage, the Autofocus SSD was trained end-to-end using the stochastic gradient descent algorithm with a batch size of 32. The momentum was fixed to 0.9, and the weight decay was selected to be 0.0005. The initial learning rate is set to 0.005, and learning rate becomes half of the original after every five epochs.

4.1.2. Other Networks

For Faster RCNN, original SSD and YOLOV5s, the training batch was set to 16, and the size of the fed image was (3, 500, 1000). Their backbones adopted all feature extraction layers in MobileNetV2. The learning rate decreasing schedule was consistent with Autofocus SSD. Data augmentation and other hyperparameter configurations were left as-were.

4.1.3. Evaluation Metrics

Some evaluation metrics were adopted to analyze the extraction performance of the proposed system: mean average precision ($\mathrm{mAP}$), frames per second (FPS), multiply-accumulate computations (MACs), and precision of SREU (${\mathrm{precision}}_{\mathrm{SREU}}$). ${\mathrm{mAP}}^{\mathrm{IoU}=\delta }$ represents the $\mathrm{mAP}$ of the network when the $\mathrm{IoU}$ between the predicted bounding box and the ground truth is greater than $\delta$ as the positive sample. MACs consist of one multiplication operation and one addition operation, which is approximately equal to two floating point operations [30]. In this paper, we employ it to evaluate the computational cost of the model. ${\mathrm{precision}}_{\mathrm{SREU}}$ is defined as follows:

$${\mathrm{precision}}_{\mathrm{SREU}}=\frac{\left|\left\{{\mathit{x}}_i|\mathrm{IoU}({\widehat{S}}^i,S^i)>0.85\right\}\right|}{\left|\chi \right|}$$

(19)

where $\chi =\left\{{\mathit{x}}_1,{\mathit{x}}_1,\cdots {\mathit{x}}_m\right\}$ is the validation set, ${\widehat{S}}^i$ and $S^i$ are the predicted signal region and the ground truth signal region of $x_i$. $\left|\hspace{0.17em}\cdot \hspace{0.17em}\right|$ represents the number of elements in a set.

4.2. Echo Signal Detection Result

The ASTIL echo signals were predicted using the Autofocus SSD we proposed, and the results are shown in Figure 7. From a visual perspective, this model is able to detect objects effectively. It’s ${\mathrm{mAP}}^{\mathrm{IoU}=0.5}$ on the validation set is 0.832 and the FPS is 84.54 (

Table 4. Network performance comparison.

Method	${\mathbf{mAP}}^{\mathbf{IoU}=0.5}$	Params	FPS	${\mathbf{precision}}_{\mathbf{SREU}}$
Faster RCNN	0.827	82.32 M	45.35	--
SSD	0.842	13.57 M	26.59	--
YOLOV5s	0.787	2.92 M	87.72	--
Autofocus SSD	0.832	0.32 M	84.54	0.933

). ${\mathrm{precision}}_{\mathrm{SREU}}$ is 0.933, indicating that SREU can accurately extract the signal region. The parameter quantity of this model is only 2.36% of the original SSD, and the FPS is 317.94% of the original SSD.

4.3. Autofocus SSD Analysis

4.3.1. Compared with Baseline Methods

Some state-of-the-art object detection networks and the network we propose were tested, as shown in Table 4. It can be seen here that among these networks, although YOLOV5s has the fastest prediction speed, its mAP is the worst. The SSD has the best mAP, but the FPS is only 26.59. Faster RCNN also has a satisfactory detection result, with a mAP of 0.827. However, the parameters of the model are enormous, with params of 82.32 M. Combining the trade-off between prediction speed and prediction accuracy, Autofocus SSD is the optimal framework. Its prediction accuracy is only 1% lower than the original SSD, but it has as high as 84.54 FPS, which is comparable to the prediction speed of YOLOV5s. In addition, its parameter size is 0.32 M, only about one tenth of YOLOV5s counterpart, which is more conducive to its deployment on many mobile and embedded applications. It is important to note here that the speed of inference for the first three models in Table 4 is not consistent with our common sense. It is well known that we always think that the prediction speed of one-stage networks is always faster than that of two-stage networks. However, the prediction speed of Faster RCNN is higher than that of SSD, mainly due to the following reasons:

• The feature extraction networks of these models were replaced with the same networks;
• The input image sizes for these models were all set to (3, 500, 1000);
• The data enhancement techniques for these models were retained and were not set to be identical;
• ASTIL has fewer foreground targets, and Faster RCNN extracted only a small number of proposals, reducing its prediction elapsed time.

4.3.2. Ablation Study

Table 5. Ablation study of Autofocus SSD.

Components	SREU	✕	✓	✓
	Depth	19	19	11
Metrics	${\mathbf{mAP}}^{\mathbf{IoU}=0.5}$	0.683	0.740	0.832
	${\mathbf{mAP}}^{\mathbf{IoU}=0.75}$	0.148	0.205	0.364
	Params	2.99 M	2.49 M	0.32 M
	FPS	72.93	74.10	86.31
	MACs	3.40 G	1.28 G	1.13 G

shows the ablation studies of Autofocus SSD. Depth indicates how deep the MobileNetV2 framework is used as the base network. The maximum depth of the feature extraction section in the MobileNetV2 network is 19. The study shows that under the same base network conditions, the SREU module can improve mAP by 5.7% with reducing the computational cost by 62.35%. In the circumstance that both have SREU modules, the depth of the base network will also have an impact on the network prediction performance. The mAP at network depth 11 is 9.2% higher than that at depth 19. The parameters and computing cost of the former are only 12.85% and 88.28% of the latter, respectively, and the FPS from the former is 16.48% higher than the latter.

4.3.3. Base Network Structure Selection

Here we conduct two groups of controlled experiments to explore the effects of network depth and feature map size on model prediction performance. The first group maintains a consistent feature map size by adjusting the stride of the second convolutional layer of the 7th and 14th inverse residual blocks of MobileNetV2 to 1. These models are called Modified MobileNetV2. The second group does not change any parameters in the original MobileNetV2, they are called original MobileNetV2. Their specific details are shown in

Table 6. Relationship between depth and Autofocus SSD prediction performance in Modified MobileNetV2 group.

Network Depth	${\mathbf{mAp}}^{\mathbf{IoU}=0.5}$	${\mathbf{mAp}}^{\mathbf{IoU}=0.75}$	Feature Map Size	Params (M)	FPS
5	0.658	0.184	(63, 38), (32, 19), (16, 10), (14, 8),	0.098	82.61
6	0.733	0.241		0.113	95.57
7	0.781	0.292		0.128	90.57
8	0.811	0.316		0.154	88.38
9	0.821	0.338		0.209	87.25
10	0.831	0.350		0.263	86.45
Autofocus SSD	0.832	0.364		0.317	86.31
12	0.831	0.358		0.389	83.74
13	0.836	0.365		0.507	82.82
14	0.835	0.358		0.625	80.64
15	0.813	0.324		0.791	80.08
16	0.824	0.340		1.110	79.20

and

Table 7. Relationship between depth and Autofocus SSD prediction performance in Original MobileNetV2 group.

Depth	${\mathbf{mAp}}^{\mathbf{IoU}=0.5}$	${\mathbf{mAp}}^{\mathbf{IoU}=0.75}$	Feature Map Size	Params (M)	FPS
9	0.799	0.297	(32, 19), (16, 10), (8, 5), (6, 3)	0.209	93.13
11	0.804	0.300	(32, 19), (16, 10), (8, 5), (6, 3)	0.317	87.91
13	0.799	0.295	(32, 19), (16, 10), (8, 5), (6, 3)	0.507	84.18
15	0.725	0.191	(16, 10), (8, 5), (4, 3), (2, 1)	0.791	80.69

.

Figure 8 shows the relationship between the depth of the base network and the prediction performance of Autofocus SSD. In terms of Modified MobileNetV2, it can be seen that both ${\mathrm{mAP}}^{\mathrm{IoU}=0.5}$ and ${\mathrm{mAP}}^{\mathrm{IoU}=0.75}$ have been improved as the network depth increases, and the performance reaches saturation when the network depth is 11. Naturally, FPS tends to decrease with the network depth increasing. Comparing the two groups of data, Modified MobileNetV2 and Original MobileNetV2, the latter’s ${\mathrm{mAP}}^{\mathrm{IoU}=0.5}$ and ${\mathrm{mAP}}^{\mathrm{IoU}=0.75}$ are both lower than the former. In the case where the weight parameters are not changed (Figure 8c), the main factor of performance degradation is the reduction in the feature map size from the (63, 38) to (32, 19). This performance degradation is even more pronounced when the depth is 15, when the feature map size is scaled from (32, 19) to (16, 10). In terms of prediction speed, when the network depth is greater than 10, the FPS of the two groups of models is roughly the same.

Therefore, according to the above results, the network with a depth of 11 in the Modified MobileNetV2 group is the optimal base network for the Autofocus SSD.

4.4. ASTIL Fast-Processing System Evaluation

In this section we used the framework shown in Figure 4 to process ASTIL raw echo signals. We employed nine processes to accelerate the data processing and applied Open3D to display the data in real time.

We tested the system on the test set and the results are shown in Figure 9. Two regions ((a), (d)) were tested, and it is evident that the system is able to roughly extract targets from a visual perspective. Comparing (a) and (b), it can be seen that the system can even extract the rough outlines of trees under complex scene conditions. However, the extraction of buildings is not satisfactory. It can be seen from (c) and (f) that there are some interstices in the extraction results of buildings, which may be partly due to incomplete echo signals of ASTIL, and partly due to insufficient representation of the training set.

See

Table 8. ASTIL fast processing system performance evaluation.

$\mathbf{Sys} {\mathbf{mAP}}^{\mathbf{IoU}=0.5}$	$\mathbf{Sys} {\mathbf{mAP}}^{\mathbf{IoU}=0.75}$	Model FPS	Sys FPS
			Building	Tree
0.812	0.295	86.31	34.82	39.22

for more statistical information. The gap between the ${\mathrm{mAP}}^{\mathrm{IoU}=0.5}$ obtained on the test set and that obtained on the validation set is within an acceptable range, indicating that the model, Autofocus SSD, has good generalization ability. However, the system FPS is lower than the model FPS, which is only 40.34% to 45.44% of the model, mainly because part of the time overhead is spent on the post-processing. In addition, the extraction efficiency of the system for different targets is also different, which is mainly due to the difference in the number of ground objects in the measured area, and the prediction rate is higher with fewer ground objects.

5. Conclusions

In this paper, we propose a fast-processing system for the novel LiDAR ASTIL. This system has the advantage of only depending on a single echo and a single data source to achieve fast ground target extraction, which contributes to ASTIL having the potential for real-time ground target extraction.

The system mainly includes two modules: object detection and post-processing. For object detection, in order to achieve the purpose of fast extracting echo signals, we have carried out structural optimization on the SSD, and proposed an Autofocus SSD, which can achieve ${\mathrm{mAP}}^{\mathrm{IoU}=0.5}$ up to 0.832 and FPS up to 84.54. The prediction speed is more than three times faster than the original SSD. For post-processing, we show in detail how to process ASTIL echo signals into point clouds for real-time display.

The system is tested on the test set, system ${\mathrm{mAP}}^{\mathrm{IoU}=0.5}$ is 0.812, system FPS is greater than 34, which shows that the system can satisfactorily achieve fast ground target extraction from ASTIL echo signals.

Nevertheless, there are still some potential improvements that can be made. First, for network prediction, we have not adopted any hardware optimization techniques, such as ONNX Runtime inference accelerator, TensorRT inference optimizer. Second, for the implementation of the system, we only enabled multi-process acceleration, but the utilization rate of each CPU is only about 10%, and there is still a lot of potential for improvement. Consequently, in our future work, we are going to apply the above two techniques to accelerate our system for real-time processing of ASTIL echo signals.