Real-Time Information Fusion System Implementation Based on ARM-Based FPGA

Abstract

In this study, an information fusion system displayed fusion information on a transparent display by considering the relationships among the display, background exhibit, and user’s gaze direction. We used an ARM-based field-programmable gate array (FPGA) to perform virtual–real fusion of this system as well as evaluated the virtual–real fusion execution speed. The ARM-based FPGA used Intel^® Realsense^TM D435i depth cameras to capture depth and color images of an observer and exhibit. The image data was received by the ARM side and fed to the FPGA side for real-time object detection. The FPGA accelerated the computation of the convolution neural networks to recognize observers and exhibits. In addition, a module performed by the FPGA was developed for rapid registration between the color and depth images. The module calculated the size and position of the information displayed on a transparent display according to the pixel coordinates and depth values of the human eye and exhibit. A personal computer with GPU RTX2060 performed information fusion in ~47 ms, whereas the ARM-based FPGA accomplished it in 25 ms. Thus, the fusion speed of the ARM-based FPGA was 1.8 times faster than on the computer.

1. Introduction

Augmented reality (AR) in large-area direct-view transparent displays is a type of information fusion system. This type of information fusion system can be used by an observer to interact with and directly access the corresponding information of their real-world target of interest. Such systems can be applied in retail, manufacturing, medical applications, service industries, etc. [1,2,3,4] and are thus expected to be implemented in various fields in the near future. These systems can merge information by considering the relationships among the screen, background object, and the user’s gaze direction to overlap the corresponding virtual information on a screen. Moreover, direct-view displays are more user-friendly and intuitive for users. As a result, research on the development and applications of direct-view transparent displays, which are considered an indispensable part of smart life applications, has gained traction [5,6]. Such systems rely on artificial intelligence (AI) recognition technology to record the object and the user’s gaze direction. Thus, the computing power capacity as well as the tradeoff between computing power and power consumption are the most critical issues for these systems.

In AI recognition technology, GPU-based computers are primarily employed to establish and train deep learning models. However, GPU-based computers are not suitable for performing models in some field applications. Field-programmable gate array (FPGA)-based embedded systems, whose speed, power consumption, size, and flexibility surpass those of GPU-based computers, are advantageous for real-time deep-learning identification and classification models.

FPGAs, consisting of programmable logic (PL) units, built-in memory, and floating point units, perform high-speed data exchange with peripheral hardware. As a result, a deep learning model can be run on the FPGA entity in a chip form [7]. The execution speed can be more than three to five times that of GPUs, and the average power consumption is also lower than that of GPUs. Therefore, FPGAs are more suitable for deep model identification or classification in prediction or inference extraction operations [8].

The implementation and performance summary of FPGA-based accelerators [9] has two discussion directions for accelerating convolutional neural network (CNN) model computations using FPGAs. The first approach is to compress the CNN model. Reducing memory footprint is realized by singular value decomposition, whereas filter clustering methods [10] simplify the connection numbers of convolution or fully connected layers through network pruning techniques [11,12,13] to reduce model complexity and prevent over-fitting. The method of enforcing weight sharing [14] is also used to reduce memory footprint. The other direction focuses on implementing CNN based on the architecture of FPGA hardware, including the hardware parallelization of data types and arithmetic units.

In deep learning model computations, addition and multiplication operations are required for convolution, pooling, and fully connected layers. Typically, floating-point arithmetic is used to avoid precision sacrifice. However, the number of floating-point arithmetic units inside an FPGA is limited, whereas the amount of data to be temporarily stored is substantial, depending on the number of layers and parameters in the model. To date, numerous ways have been explored to reduce the number of bits or data operations without sacrificing precision. In previous studies [15,16,17], fixed-point formats, with 8 to 16 bits as the basic arithmetic format, were used to improve the data computation speed and reduce the amount of data without reducing prediction accuracy. However, this method requires additional optimization and adjustment or the addition of floating-point arithmetic in the data conversion step. VITIS AI is a development platform from Xilinx which is designed for drawing AI-based inferences on Xilinx hardware platforms. It consists of an optimized IP, tools, libraries, models, and example designs, and supports mainstream frameworks as well as the latest models. This platform is capable of executing various deep-learning tasks. VITIS AI also provides a range of pretrained models that can be retrained for specific applications.

CPU- and GPU-based computers are convenient for rapidly implementing fusion information systems. However, the computing speed, power consumption, size, and flexibility of these computers are inferior to those of FPGA-based computing systems. Notably, the AI computation power capacity and tradeoff between the computing power and power consumption are the most critical limitations of information fusion systems. Thus, in this study, we used an ARM-based FPGA to integrate complex works and increase the computation speed of fusion information systems. In addition, we compared the computation speed of the information fusions performed by an ARM-based FPGA and a computer with a GPU RTX206. Figure 1 shows the hardware of the information fusion system used in our study. In this paper, Section 2 introduces the architecture and method employed by the information fusion system. Section 3 elucidates the implementation of the information fusion system on an ARM-based FPGA. Section 4 presents the experimental details and results obtained using the fusion information system implemented by the ARM-based FPGA. Finally, Section 5 provides a comprehensive discussion on the obtained results and highlights the major conclusions and inferences drawn from them.

2. Information Fusion Method

The information fusion system can generate an image in the line of sight of an observer that sees the image overlapped on their exhibit of interest. The side view of the information fusion system is shown in Figure 2. The system detects the positions of the observer and exhibit relative to that of the transparent display. Thus, the system employs two depth-color cameras to capture the color and depth images of the observer and exhibit. The system receives these images and feeds them into neural network models, which locate the observer and exhibit in the color images. The identified positions of the observer and exhibit in the color images are then used to determine their positions in the depth image corresponding to the color images. Thus, the positions of the observer and exhibit in the real world can be derived by mapping their positions in the image coordinates from the real world coordinates. Subsequently, the system calculates the displayed position and size of the fusion information according to the relative positions of the observer and exhibit. Finally, the system displays the fusion information on a transparent display to provide additional details for the exhibit. The workflow of the information fusion system is illustrated in Figure 3.

The system performs two essential processes: recognition and position acquisition of the fusion information. In the position-acquisition process, the system performs image registration between the depth and color images. The depth-to-color image coordinate maps are derived using Equations (1)–(3), shown below:

$$\left[\begin{array}{c}{X}^D\\ Y^D\\ Z^D\end{array}\right]=d{\left[\begin{array}{ccc}{f}_x^D& 0& p_x^D\\ 0& f_y^D& p_x^D\\ 0& 0& 1\end{array}\right]}^{-1}\left[\begin{array}{c}{u}^D\\ v^D\\ 1\end{array}\right]$$

(1)

$$\left[\begin{array}{c}{X}^C\\ Y^C\\ Z^C\end{array}\right]=\left[\begin{array}{cccc}{r}_{11}^{D\mathrm{C}}& r_{12}^{D\mathrm{C}}& r_{13}^{D\mathrm{C}}& t_1^{D\mathrm{C}}\\ r_{21}^{D\mathrm{C}}& r_{22}^{D\mathrm{C}}& r_{23}^{\mathrm{DC}}& t_2^{D\mathrm{C}}\\ r_{31}^{D\mathrm{C}}& r_{32}^{D\mathrm{C}}& r_{33}^{\mathrm{DC}}& t_3^{D\mathrm{C}}\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}^D\\ Y^D\\ Z^D\\ 1\end{array}\right]$$

(2)

$$Z^C\left[\begin{array}{c}{u}^C\\ v^C\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x^C& 0& p_x^C\\ 0& f_y^C& p_y^C\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}^C\\ Y^C\\ Z^C\end{array}\right]$$

(3)

Equation (1) is the transformation from the two-dimensional depth image coordinate (u^D, v^D) to the three-dimensional depth camera coordinate (X^D, Y^D, Z^D). f_x^D and f_y^D are the focal lengths of the depth camera; p_x^D and p_y^D are the central positions of the depth image; d is the depth data of the depth image. Equation (2) is the transformation from the depth camera coordinate to the three-dimensional color camera coordinate (X^C, Y^C, Z^C). r₁₁^DC to r₃₃^DC are the rotation parameters, and t₁^DC to t₃^DC are the translation parameters. Equation (3) is the transformation from the color camera coordinate to the two-dimensional color image coordinate (u^C,v^C).

After detecting the observer or exhibit in the color image, the position of observer or exhibit in the three-dimensional color camera coordinate can be derived from the inverse transformation of Equation (3). Next, the positions are mapped to the three-dimensional transparent display coordinate (X^M, Y^M, Z^M) according to Equation (4), expressed below:

$$\left[\begin{array}{c}{X}^M\\ Y^M\\ Z^M\end{array}\right]=\left[\begin{array}{cccc}{r}_{11}^{CM}& r_{12}^{CM}& r_{13}^{CM}& t_1^{CM}\\ r_{21}^{CM}& r_{22}^{CM}& r_{23}^{CM}& t_2^{CM}\\ r_{31}^{CM}& r_{32}^{CM}& r_{33}^{CM}& t_3^{CM}\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}^C\\ Y^C\\ Z^C\\ 1\end{array}\right]$$

(4)

Here, r₁₁^CM to r₃₃^CM are the rotation parameters, and t₁^CM to t₃^CM are the translation parameters.

The positions of the observer and exhibit in the transparent display coordinate are (X_o^M, Y_o^M, Z_o^M) and (X_e^M, Y_e^M, Z_e^M), respectively. Equation (5) is used to calculate the position of the fusion information in the transparent display coordinate system. The coordinate (X_m^M, Y_m^M, Z_m^M) represents the position of the fusion information in Figure 4.

$$\left[\begin{array}{c}{X}_m^M\\ Y_m^M\end{array}\right]=\frac{Z_e^M}{Z_e^M-Z_o^M}\left[\begin{array}{c}{X}_o^M\\ Y_o^M\end{array}\right]+\frac{-Z_o^M}{Z_e^M-Z_o^M}\left[\begin{array}{c}{X}_e^M\\ Y_e^M\end{array}\right]$$

(5)

Finally, the position and size coordinates of the transparent display are mapped to the transparent image coordinate for display according to Equation (6).

$$\left[\begin{array}{c}{u}^M\\ v^M\end{array}\right]=\left[\begin{array}{ccc}{s}_x& 0& p_x\\ 0& s_y& p_y\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}^M\\ Y^M\\ 1\end{array}\right]$$

(6)

3. Implementation

The information fusion system mainly comprised a transparent display, two depth cameras, Realsense D435i, and a Xilinx Zynq UltraScale+ MPSoC ZCU102 development board. The depth camera provided depth and color images for object detection. The development board contained an ARM-based FPGA XCZU9EG chip with a quad-core Arm^® Cortex^®-A53 (PS), dual-core Cortex-R5F real-time processors, Mali™-400 MP2 graphics processing unit (MGPU), and a PL. The PS side received image data and sent images to the PL. Image registration between the color and depth images was performed on the PL instead of on the PS, because the image registration was faster on the PL than on the PS. The PL contained a deep learning processor unit (DPU) that executed the exhibit and observer identification. The image registration and detection results were fed back to the PS. Next, the PS calculated the size and position of the information displayed on the transparent display according to the coordinates and depth values of the observer’s eyes and objects. The MGPU received the display information from the PS and updated the information to the transparent display. The computation architecture of the system is shown in Figure 5.

The PS performed multi-thread tasks using a total of four threads, including DecodeThread, DpuThread, SortingThread, and GuiThread. DecodeThread received the color and depth images and sent them to the PL for image registration. Then, DecodeThread received the image registration results and pushed the color images into a queue. DpuThread popped the images in the queue to the DPU for detecting the observer and exhibit and received the detection results, including the coordinates, sizes, and labels. SortingThread sorted the images in the queue according to the frame indices. GpuThread displayed the images at a specific position on the transparent display.

The information fusion system employed the RetinaFace and YOLO models to detect the face of the observer and exhibit, respectively. The RetinaFace model was pretrained and supported by the VITIS AI, which could rapidly apply the RetinaFace model on the DPU. The YOLO model for detecting LEGOs was self-trained and converted into a DPU-executable xmodel format; the conversion flow is shown in Figure 6. The parameters of the trained YOLO model were modified, as shown in

Table 1. Modified parameters of the self-trained YOLO model for the Xilinx DPU.

Parameters	Origin	For Xilinx DPU
Input image weight	608	512
Input image height	608	512
Activation function	mish	leaky
Max-pooling size 1	9	6
Max-pooling size 2	13	8

, to run it on the DPU. The size of the image sent as an input to the DPU was 512 × 512 pixels. Moreover, because the DPU did not support the mish activation function, the leaky rectified linear unit activation function was used. The max-pooling sizes were reduced, because the maximum maxpool size of the DPU was limited to eight. Furthermore, the computing number format of the model was converted from floating-point to integer by the VITIS AI quantizer to increase the computation speed.

The image registration function was programmed using C++, and the VITIS™ High-Level Synthesis tool was utilized to synthesize the function into the registration level (RTL) module for the PL implementation. The code is displayed in

Table A1. C++ code of image registration for the High-level RTL synthesis.

1	void align_depth_to_color_2(float intrinsics_color[4], float intrinsics_depth[4], float extrinsics_depth_to_color[12], float depth_scale ,unsigned short int depth[307200], float depth_of_color[640*480]) {
2	float pd_uv[2];
3	float Pd_uv[3];
4	float Pc_uv[3];
5	unsigned short int pc_uv[2];
6	for(int i = 0; i < 480; i++) {
7	//#pragma HLS pipeline II = 1
8	for(int j = 0; j < 640; j++) {
9	//#pragma HLS pipeline II = 1
10	pd_uv[0] = j;
11	pd_uv[1] = i;
12	Pd_uv[0] = (pd_uv[0]-intrinsics_depth[2]) * depth[i*640+j]*depth_scale/intrinsics_depth[0];
13	Pd_uv[1] = (pd_uv[1]-intrinsics_depth[3]) * depth[i*640+j]*depth_scale/intrinsics_depth[1];
14	Pd_uv[2] = depth[i*640+j]*depth_scale;
15	Pc_uv[0] = extrinsics_depth_to_color[0] * Pd_uv[0] + extrinsics_depth_to_color[1] * Pd_uv[1] + extrinsics_depth_to_color[2] * Pd_uv[2] + extrinsics_depth_to_color[3];
16	Pc_uv[1] = extrinsics_depth_to_color[4] * Pd_uv[0] + extrinsics_depth_to_color[5] * Pd_uv[1] + extrinsics_depth_to_color[6] * Pd_uv[2] + extrinsics_depth_to_color[7];
17	Pc_uv[2] = extrinsics_depth_to_color[8] * Pd_uv[0] + extrinsics_depth_to_color[9] * Pd_uv[1] + extrinsics_depth_to_color[10] * Pd_uv[2] + extrinsics_depth_to_color[11];
18	pc_uv[0] = (unsigned short int)((intrinsics_color[0] * Pc_uv[0]/Pc_uv[2] + intrinsics_color[2]));
19	pc_uv[1] = (unsigned short int)((intrinsics_color[1] * Pc_uv[1]/Pc_uv[2] + intrinsics_color[3]));
20	if(pc_uv[0] < 640 && pc_uv[0] ≥ 0 && pc_uv[1] < 480 && pc_uv[0] ≥ 0 )
21	depth_of_color[pc_uv[1] * 640 + pc_uv[0]] = Pd_uv[2];
22	}
23	}
24	}

. The input image size is 640 × 480 pixels, and the input parameter includes the intrinsic and intrinsic parameters of the color and depth cameras. Lines 10 to 19 in Table A1 describe the coordinate transformation from the depth camera coordinate to the color camera coordinate. Lines 20 and 21 in Table A1 describe the depth data alignment in the color camera coordinates.

4. Experimental Setup and Results

The total utilization of XCZU9EG is described in

Table 2. Utilization of XCZU9EG resources. The first row of each module presents the total number of units used by this module, and the second row shows the resources used by the module as a percentage of the total resources.

Module Name	LUT	LUTAsMem	REG	BRAM	DSP
DPUCZDX8G_1	51,082	5680	97,940	257	690
	(19.69%)	(3.98%)	(18.86%)	(30.82%)	(27.38%)
DPUCZDX8G_2	51,308	5680	98,128	257	690
	(19.78%)	(3.98%)	(18.90%)	(30.82%)	(27.38%)
Image registration	11,568	981	18,923	15	74
	(4.46%)	(0.69%)	(3.64%)	(1.80%)	(2.94%)
sfm_xrt_top	9638	540	8142	4	14
	(3.71%)	(0.38%)	(1.57%)	(0.48%)	(0.56%)
Total Utilization	138,219	14,139	251,993	612	1468
	(50.00%)	(10.00%)	(46.00%)	(67.00%)	(58.00%)

. The first row of each module presents the total number of units used by this module, and the second row shows the resources used by the module as a percentage of the total resources. The image registration model used 4.46% of the look-up-table (LUT), 3.64% of the register (REG), and 2.94% of the digital signal-processing (DSP) unit in XCZU9EG. Two DPU units were used for the RetinaFace and YOLO models. DPUCZDX8G_1 used 19.69%, 18.86%, 30.82%, 27.38% of the LUT, REG, Block RAM (BRAM), and DSP, respectively. In contrast, DPUCZDX8G_2 used 19.78%, 18.90%, 30.82%, and 27.38% of the LUT, REG, BRAM, and DSP, respectively. Thus, the two DPU units used 39.47%, 37.76%, 61.64%, and 54.76% of the LUT, REG, BRAM, and DSP, respectively. Evidently, most resources were used for the DPU processing. The total utilization of the information fusion accounted for 50–60% of the entirety of the resources in XCZU9EG.

A Linux system on the ARM side executes the application program to execute image reading, detecting object frame selection, and name marking, as well as to obtain the image output. The interface includes functions to open advertisement video, select the target object to watch, and jump to the relevant introduction interface after clicking the display. The position of the human eye and exhibit are recognized using the front and rear depth cameras, respectively. The position of the object on the screen seen by the eyes is calculated using the above-mentioned relative position formula. The directional touch function realized in the ARM-based FPGA platform is shown in Figure 7. After clicking the target, the corresponding introduction interface is displayed, as shown in Figure 8.

5. Conclusions and Discussion

Lyu et al. [18] implemented a YOLOv4 model on the FPGA platform to identify citrus flowers; the computing speed of this model was approximately 33.3 ms, and the power consumption of the FPGA was 20 W. Pérez et al. [19] used an FPGA with a CNN model for image classification and achieved a speed of 24.6 frames per second. Previous studies indicate that the FPGA can accelerate AI computations. In this study, we successfully used an ARM-based FPGA to implement a fusion information system with a transparent display. The ARM-based FPGA accelerated the information fusion processes, including image registration, face recognition, and object recognition. The PL and PS executed image registration in approximately 3 ms and 1.1 s, respectively, indicating that the image registration speed of the PL was higher than that of the PS. Consequently, the PL side used approximately 46.2% of the total resources to fuse the information. The ARM-based FPGA and the computer with GPU RTX2060 performed information fusion in 25 ms and 47 ms, indicating that the information fusion speed of the ARM-based FPGA was 1.8 times more than that of the computer with GPU RTX2060.