Implementation of MIMO Radar-Based Point Cloud Images for Environmental Recognition of Unmanned Vehicles and Its Application

Abstract

High-performance radar systems are becoming increasingly popular for accurately detecting obstacles in front of unmanned vehicles in fog, snow, rain, night and other scenarios. The use of these systems is gradually expanding, such as indicating empty space and environment detection rather than just detecting and tracking the moving targets. In this paper, based on our high-resolution radar system, a three-dimensional point cloud image algorithm is developed and implemented. An axis translation and compensation algorithm is applied to minimize the point spreading caused by the different mounting positions and the alignment error of the Global Navigation Satellite System (GNSS) and radar. After applying the algorithm, a point cloud image for a corner reflector target and a parked vehicle is created to directly compare the improved results. A recently developed radar system is mounted on the vehicle and it collects data through actual road driving. Based on this, a three-dimensional point cloud image including an axis translation and compensation algorithm is created. As a results, not only the curbstones of the road but also street trees and walls are well represented. In addition, this point cloud image is made to overlap and align with an open source web browser (QtWeb)-based navigation map image to implement the imaging algorithm and thus determine the location of the vehicle. This application algorithm can be very useful for positioning unmanned vehicles in urban area where GNSS signals cannot be received due to a large number of buildings. Furthermore, sensor fusion, in which a three-dimensional point cloud radar image appears on the camera image, is also implemented. The position alignment of the sensors is realized through intrinsic and extrinsic parameter optimization. This high-performance radar application algorithm is expected to work well for unmanned ground or aerial vehicle route planning and avoidance maneuvers in emergencies regardless of weather conditions, as it can obtain detailed information on space and obstacles not only in the front but also around them.

1. Introduction

Radar is most commonly used in vehicles and aircrafts to detect and track long-range targets. Unlike a camera-based system, radar can provide information in backlight or dark conditions at night. Unlike Lidar, it can provide more than 200 m forward information even in foggy, snowy or rainy weather. It also can measure the relative radial velocity of the detected target via the Doppler effect exactly [1,2,3]. Radar systems used in the military, aircraft, ships and automotives play a key role in detecting and tracking moving objects, and emergency braking for ADAS [4]. As the unmanned vehicle technology has developed, much research has been conducted on the implementation of additional functions such as three-dimensional sensing of stationary objects, providing a point cloud image, detailed classification and parking lot and empty space detection [5]. In this work, a point cloud imaging algorithm is implemented based on our radar system. Normally, the point cloud image aligns the information of the detected targets to the current position value and draws them to three-dimensional coordinates [6,7]. The GNSS is generally located in the center of the vehicle drive shaft, and the radar is mounted on the front bumper or roof. In order to improve the quality of the point cloud image, an algorithm for correcting the position coordinate value of GNSS and the alignment error of the sensor that collects actual data is also proposed. The actual improvement is obtained by comparing with the error value of a detected points image for the front single target and the three-dimensional point cloud image realized and compared directly through the actual parked vehicle detection. By overlapping this improved image with the navigation map image, the algorithm that can accurately indicate the location of the vehicle is implemented. In addition, we also fuse the camera image and the three-dimensional point cloud image to achieve redundancy of object detection. This paper is organized as follows. Section 2 details the principle of the Frequency Modulated Continuous Wave (FMCW) radar, a fabricated radar system and algorithm and a misalignment correction method. Section 3 details the implemented point cloud image and sensor fusion image. And finally, conclusions are given in Section 4.

2. Materials and Methods

2.1. Frequency Modulated Radar

FMCW radar is one of the representative systems suitable for application such as an altimeter in unmanned aircraft and transportation collision avoidance involving vehicles and ships. It transmits continuously modulating signals at a pre-defined rate over a fixed time period and measures the distance and velocity of targets. The frequency of the transmitted signal f(t) can be calculated as:

$$f\left(t\right)=f_c+\frac{B}{T}t,-\frac{T}{2}\le t\le \frac{T}{2}$$

(1)

where $f_c$ (Hz) is the center frequency, $B$ (Hz) is over the frequency bandwidth, and $T$ (s) is the sweep time.

The angular argument of a transmitted signal, also known as the phase angle, is defined as the integral of the function $f\left(t\right)$ which is calculated over a given time interval and represents the total area under the curve of the waveform.

$$\arg \left(t\right)=\underset{0}{\overset{t}{\int }}\left(f_c+\frac{B}{T}\tau \right)d\tau =f_{c}t+\frac{B}{2T}{t}^2$$

(2)

Then, the transmitted signal $S_T\left(t\right)$ can be written with the above two equations:

$$S_T\left(t\right)=A_T\mathrm{c}\mathrm{o}\mathrm{s}(2\pi \left(f_c+\frac{B}{2T}t\right)t+{\phi }_T)$$

(3)

where $A_T$ is the amplitude and ${\phi }_T$ (rad) is the phase.

The transmitted signal will be reflected from the objects. The reflection coefficient is $\rho$ and the value of materials is $\rho$ < 1. The received signal $S_R\left(t\right)$ will be a time delayed transmitted signal with frequency and phase shifted. FMCW radar extracts the beat signal $f_{\mathit{beat}}\left(t\right)$ by mixing transmitted and received signals as follows:

$$f_{\mathit{beat}}\left(t\right)=S_T\left(t\right)S_R\left(t\right)=S_T\left(t\right){\rho S}_T\left(t-{\tau }_d\right)$$

(4)

where ${\tau }_d=\frac{2r}{c}$ (s) is the delay of the received signal, $r$ (m) is the distance with object, and $c$ (m/s) is the speed of light.

The transmitted signal and ${\tau }_d$ time delayed receiving signal Equation (3) are substituted in the beat signal Equation (4), and then,

$$\begin{gathered} f_{\mathit{beat}}\left(t\right)=A_T^2\rho \cos \left(2\pi \left(f_c+\frac{B}{2T}t\right)t+{\phi }_T\right) \\ \cos \left(2\pi \left(f_c+\frac{B}{2T}\left(t-{\tau }_d\right)\right)\left(t-{\tau }_d\right)+{\phi }_T\right) \end{gathered}$$

(5)

The formula cos($\mathsf{\alpha })$ cos($\mathsf{\beta }$) = $\frac{1}{2}$ (cos($\mathsf{\alpha }+\mathsf{\beta })$ + cos($\mathsf{\alpha }-\mathsf{\beta })$) is adapted to Equation (5),

$$\begin{gathered} =\frac{\rho A_T^2}{2}(\mathrm{c}\mathrm{o}\mathrm{s}(2\mathsf{\pi }(2f_{c}t+\frac{B}{2T}\left(2t^2-2{\tau }_{d}t+{\tau }_d^2\right) \\ +\mathrm{c}\mathrm{o}\mathrm{s}\left(2\mathsf{\pi }\right(\frac{B}{T}{\tau }_{d}t+f_c{\tau }_d-\frac{B}{2T}{\tau }_d^2\left)\right)) \end{gathered}$$

(6)

After multiplying two signals, the high-frequency term of Equation (6) can be filtered out via a band pass filter, and then, the last term remains.

$$f_{\mathit{beat}}\left(t\right)=\frac{\rho A_T^2}{2}\mathrm{c}\mathrm{o}\mathrm{s}\left(2\mathsf{\pi }\left(\frac{B}{T}{\tau }_{d}t+f_c{\tau }_d-\frac{B}{2T}{\tau }_d^2\right)\right)$$

(7)

If terms change to the following:

$$\begin{gathered} A=\frac{\rho A_T^2}{2}, K\left(r\right)=\frac{B}{T}{\tau }_d=\frac{2B}{cT}r, \\ {\Phi }_r\left(r\right)=2\mathsf{\pi }\left(f_c-\frac{B}{2T}{\tau }_d\right){\tau }_d=\frac{4\pi }{c}\left(f_c-\frac{B}{Tc}r\right)r \end{gathered}$$

(8)

The beat signal can be represented in the form of a sinusoidal signal whose frequency changes with distance r:

$$f_{\mathit{beat}}\left(t\right)=\mathrm{A}\cos \left(2\mathsf{\pi }K\left(r\right)t+{\Phi }_r\left(r\right)\right)$$

(9)

Through this method, the FMCW radar could accurately measure the distance between the radar and the target.

2.2. Radar System

The radar system consists of four 79 GHz Silicon Complementary Metal Oxide Semiconductor (CMOS) Radio Frequency Integrated Circuits (RFICs). Within the system, four RFICs are synchronized with each other using 20 GHz local oscillator (LO) signal and they operate like a single RFIC chip system. Multi-Input Multi-Output (MIMO) is a technology that effectively increases the total number of antennas by introducing a virtual antenna concept [8,9]. The orthogonality of each transmitting signals can be obtained through specific methods such as time division multiplexing and MIMO, which improves angular resolution through an increased antenna aperture with a virtual array [10,11]. Combined with an appropriate antenna array arrangement, high-resolution point cloud information of the surroundings can be obtained from the radar. MIMO’s matrix M, consisting of intermediate frequency (IF) signals, is defined as follows:

$$M=\left[S_1^{IF}, S_2^{IF},\cdots ,S_{N_v}^{IF}\right]$$

(10)

where $N_v=N_{TX}·N_{RX}$, $S_i^{IF}$ is the complex time-domain chirp signal and $N$ represents sampled points of each virtual element.

The MIMO transceiver antennas are configured to fill the front of the RF board in a complex repetitive pattern along the length rule of wavelength. Therefore, we place the RF chips on the back of the RF board for flexible wiring design and layout on the board. Although there may be a loss in terms of via-interconnection, the freedom of antenna placement can be greatly increased and the line length minimized, which helps to improve the RF performance. Even in the overall design change, MIMO antenna arrangement and layout change can be minimized. The 20 GHz LO signal is allocated to the master and slave chips using the Wilkinson power divider. In order to minimize the deviation among LO synchronization lines and the line length from antennas to transceivers, they are arranged point-symmetrically. As developed in the previous equation, the radar radiates FMCW signals over time and the beat frequency is obtained through down conversion with the received signal. A fast-chirp FMCW radar transmits a large number of chirps with a chirp duration (T chirp) of tens of microseconds in a single frame, and has the advantage of avoiding multiple-target-identification problems over the classical modulation methods [12,13,14]. The main parameters of the radar are summarized in

Table 1. Main parameters of our tested radar system.

Center Frequency (GHz)	Bandwidth (GHz)	Velocity Resolution (ms)	Range Resolution (cm)	Angle Resolution (deg.)	Field of View (deg.)
79	3.5	0.1	4.3	2.5	160

. The fabricated system is mounted on the bumper of the test vehicle with basic detection functions for on-road scenario verification. It is confirmed that the vehicle at a range of more than 100 m could be stably detected. Basic performance of the system is measured by placing 10 dBsm corner reflectors in front of the vehicle. The maximum field of view of the radar is 160 degrees with an angle resolution of 2.5 degrees. The system is configured in a 3.5 GHz bandwidth, and the maximum detection distance is shortened because the total number of range bins are the same, but the distance resolution is improved to 4.3 cm.

2.3. Radar Signal Processing

After the sampling process of analog-to-digital converter in RF chips, data are transferred to the processor via gigabit Ethernet line. Received signals are organized and stored in the form of data cubes in memory. In order to extract range and velocity information from data cubes, a 2D FFT is carried out. It is necessary to reduce the processing time by simplifying the dimension of the Range-Doppler Map or computational complexity. Our experiments apply a matrix compression method that intentionally omits some antenna channels to configure inter-element spacing larger than a half wavelength. A Doppler coherent focusing Direction of Arrival (DoA) method operates at a relatively small amount of computation by adjusting the distribution of the scanning area [15].

2.4. Misalignment Correction

Figure 1 shows positions of GNSS and the radar sensor. Due to different positions of the radar mounted on the front bumper of the vehicle and the GNSS mounted at the center of the rear shaft of the vehicle, the position error of the points occurs when the point cloud image is generated during rotation and curve driving. In straight driving, data are collected with a certain delay between radars and GNSS and thus are relatively less distorted. However, when driving outside a straight line, such as curving or rotating, the positions of the radars and GNSS in the vehicle are different. so, the detected single point targets become blurred and appear as an object in space. This blurring can be minimized by applying the misalignment offset modification algorithm.

The location of the target viewed from radar sensor is described as:

$$\begin{gathered} x_{radar}=r_{radar}\times {sin\theta }_{radar} \\ {y}_{radar}=r_{radar}\times {cos\theta }_{radar} \end{gathered}$$

(11)

The location of the target viewed from GNSS is described as:

$$\begin{gathered} x_{radar}^{ldc}=x_{radar}+∆\mathsf{\alpha } \\ {y}_{radar}^{ldc}=y_{radar}+∆\mathsf{\beta } \end{gathered}$$

(12)

where $x_{radar}^{ldc}$ and $y_{radar}^{ldc}$ are $x$ and $y$ coordinates with local displacement, and $∆\mathsf{\alpha }$ and $∆\mathsf{\beta }$ are $x$ and $y$ axis distances between GNSS and the sensor.

The misalignment correction coefficient matrix based on the sensor is calculated as:

$$\begin{gathered} {\left[x_{radar}^{ldc} y_{radar}^{ldc}\right]}^T= \\ {R}_{MC}\times {\left[x_{radar} y_{radar}\right]}^T+{[∆\mathsf{\alpha }∆\mathsf{\beta }]}^T \end{gathered}$$

(13)

where $R_{MC}$ is the correction distance for misalignment.

Finally, the movement of the platform observed in global position can be defined as follows:

$${\left[x_g y_g\right]}^T=R_{HA}\times {\left[x_{radar}^{ldc} y_{radar}^{ldc}\right]}^T+{[x_{gnss} y_{gnss }]}^T$$

(14)

where $x_g y_g$ is the global coordinate of $x$, $y$ and $R_{HA}$ is the heading alignment distance between GNSS and radar sensor.

2.5. Camera-Radar Calibration

To calibrate the distortion caused by the lens of the camera, the internal calibration is performed by obtaining the focal length and optical center radial distortion coefficients of the camera. One point on the camera image $[{\begin{array}{ccc}u& v& 1\end{array}]}^T$ is converted into a camera homogeneous coordinate $[{\begin{array}{cccc}{X}^{\prime }& Y^{\prime }& Z^{\prime }& 1\end{array}]}^{\mathrm{T}}$, and the relationship between point and coordinate is formulated as follows:

$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_u& s& u_0\\ 0& f_v& v_0\\ 0& 0& 1\end{array}\right]$$

(15)

where $f_u$ and $f_v$ are focal lengths, $u_0$ and $v_0$ are optical centers, and $s$ is the skew coefficient.

Skew coefficient indicates the degree of tilt in the Y-axis direction of the cell array of the image sensor. Nowadays, cameras have few skew errors, so this value is generally set to the camera module. To convert a point in the radar’s coordinate $[{\begin{array}{cccc}X& Y& Z& 1\end{array}]}^{\mathrm{T}}$ into a point in the camera coordinate $[{\begin{array}{cccc}{X}^{\prime }& Y^{\prime }& Z^{\prime }& 1\end{array}]}^{\mathrm{T}}$, the rotational matrix R and translation factor values $p$ must be introduced.

$$\left[\begin{array}{c}X\\ Y\\ Z\\ 1\end{array}\right]=\left[\begin{array}{cc}{R}_{3\times 3}& p_{3\times 1}\\ 0& 1\end{array}\right]\left[\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\\ Z^{\prime }\\ 1\end{array}\right]$$

(16)

This matrix is referred to as an extrinsic parameter, and eight variable values are determined by performing more than eight tests using the evaluation board and the corner reflector. It is expressed as the following equation:

$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_u& s& u_0\\ 0& f_v& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& p_1\\ r_{21}& r_{22}& r_{23}& p_2\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\\ Z^{\prime }\\ 1\end{array}\right]$$

(17)

3. Results

3.1. Point Cloud Image

A Point cloud image is a group of point images in which signals reflected from objects are produced in the form of three-dimensional points, and have been widely applied to implement a digital twin of buildings or objects using laser scanners [16,17,18,19,20,21].

As the performance of the radar has been enhanced, more and more functions such as road course estimation, classification and localization from the existing simple obstacle detection and tracking-oriented functions have been developed. Radar-based point cloud images are a set of data points in space and accumulate radar detection points with X, Y and Z coordinates from temporal and spatial perspectives [6,22,23,24,25,26]. Points of the image adopt the GNSS localization sensor to enhance the accuracy for both temporal and spatial domains. The implementation is relatively straightforward, but if the exact location value is not found, image performance will be greatly deteriorated. However, only when the position alignment and compensation between the GNSS sensor and the radar sensor are exactly determined can the image be generated properly without blur. Figure 2 shows the images before and after misalignment correction for the front single point target collected in vehicle driving, and Figure 3 shows the compared point cloud images of the parked vehicle before and after misalignment correction. Before modification, the signal detected by the radar mounted on the front bumper of the vehicle is transformed based on the GNSS sensor coordinates on the center of the rotation axis of the rear wheel. However, the image generated after applying the coordinate modification algorithm shows that the detected points are more clustered around the vehicle. Since radar can measure the Doppler velocity for the detected object, it can also output the relative speed of the target around an ego-vehicle. In our experiment, we use an ego-velocity estimation algorithm that can estimate the speed of an ego-vehicle and moving objects such as vehicles and stationary objects such as curbs, walls and street trees, which are accurately distinguished and represented using the general characteristics of such a radar implemented with our algorithm.

Figure 4 shows the results of constructing a three-dimensional point cloud radar image around the road via our radar mounted on the vehicle. This figure also shows that the curbstone and the outer wall of the sidewalk appear clearly in the point cloud image.

3.2. A Sensor Fusion Image Based on Camera and Radar

A radar system with a bandwidth of 1 GHz can acutely measure the distance from the radar to the detection object with accuracy up to 20 cm based on the flight time of characteristics of the RF signal. By virtue of an accurate measurement, point cloud images that are well aligned and combined with cameras and radar images become increasingly useful as various usage scenarios have been found in the unmanned vehicle. In our experiment, the radar-based point cloud image overlapping with the navigation road map image can accurately represent the current position of the ego vehicle. As shown in Figure 5, the QtWeb based road map Application Programming Interface (API) is displayed in real time and overlaps with the two-dimensional point cloud radar map image so that the current position of the vehicle can be accurately indicated. The stationary curbstones around the vehicle are represented in blue, and the detected vehicles and moving pedestrians in the front are represented in red.

Image fusion with a camera and a three-dimensional point cloud radar image is also implemented in our work, because improving target-detection performance by combining a camera image and a radar image has attracted much attention. In order to fuse images, three-dimensional radar point coordinate information needs to be converted into the coordinates of a two-dimensional camera. After calibration, Intersection over Union (IoU) is applied. IoU is an evaluation metric used to measure the accuracy of an object detector on a particular dataset. As shown in

Table 2. Misalignment value of tested camera and radar.

Index	Pixel Value (pixels)	RMSE (m)
Misalignment	2.2	0.05

, through calibration, the alignment is improved to have an error value within 2.2 pixels and a Root Mean Square Error (RMSE) less than 0.05 m. Figure 6 shows the fusion of the camera image and three-dimensional radar point cloud image. The median barrier in the middle of the road, building walls, street trees and the metal box on the sidewalk can be observed in this fusion image.

4. Discussion

Previously, a 79 GHz high-resolution radar system for unmanned vehicles was developed, and basic detection performance was verified through road driving tests. In order to expand the application of this radar system, we propose to implement a point cloud image by using GNSS sensors and our radar as existing detection and tracking functions. Due to the different mounting positions of the radar and GNSS (i.e., different reference points of coordinates), alignment errors may occur when the vehicle rotates or turns, which causes distortion of the point cloud image. To solve this problem, we develop a correction algorithm for errors. It is confirmed that the error of 2.7 m when driving over 20 m was reduced to less than 0.2 m for vehicles applying the correction algorithm. The accurate measurement can meet the needs of various application scenarios of unmanned vehicles. We present several typical scenarios in the experiments. An algorithm that enables overlapping a point cloud image detected by a radar in real time with a QtWeb-based navigation image is implemented. By aligning and matching road information on navigation and edge information of roads detected via radar, the accurate location of current unmanned vehicles can be estimated in shaded areas where a GNSS signal is cut off or when driving slowly in the city center. Another scenario is to overlap the radar’s image information with the camera’s image information under the same coordinate reference system but with different modality information for the same target. Extrinsic calibration, a correction for different positions of camera lens and sensor, is further performed. This can guarantee not only camera information but also the radar’s location, angle and speed information for objects in front of the vehicle, and thus greatly improve detection accuracy compared to using the single-sensor system. The application and sensor fusion functions of these radars are expected to truly make contributions in unmanned vehicles such as Unmanned Aerial Vehicles (UAVs) and Urban Air Mobility (UMA), helping them successfully reach their destinations.