3D Reconstruction Framework for Multiple Remote Robots on Cloud System

Abstract

This paper proposes a cloud-based framework that optimizes the three-dimensional (3D) reconstruction of multiple types of sensor data captured from multiple remote robots. A working environment using multiple remote robots requires massive amounts of data processing in real-time, which cannot be achieved using a single computer. In the proposed framework, reconstruction is carried out in cloud-based servers via distributed data processing. Consequently, users do not need to consider computing resources even when utilizing multiple remote robots. The sensors’ bulk data are transferred to a master server that divides the data and allocates the processing to a set of slave servers. Thus, the segmentation and reconstruction tasks are implemented in the slave servers. The reconstructed 3D space is created by fusing all the results in a visualization server, and the results are saved in a database that users can access and visualize in real-time. The results of the experiments conducted verify that the proposed system is capable of providing real-time 3D scenes of the surroundings of remote robots.

1. Introduction

Nowadays, various innovative technological systems, such as multimedia contexts [1,2,3,4,5,6], video codecs [7,8], healthcare systems [9,10], and smart indoor security systems [11,12] are being actively researched. In the area of cloud computing, a failure rules aware node resource provision policy for heterogeneous services consolidated in cloud computing infrastructure has been proposed [10]. Various applications for human pose estimation, tracking, and recognition using red-green-blue (RGB) cameras and depth cameras have also been proposed [13,14,15,16,17,18,19,20,21,22,23,24]. A depth camera gives an image in each frame that contains information about the distance from the camera to other objects around the camera’s position. In contrast, an RGB camera gives a color image in each frame. The security problems in distributed environment and cloud database as a service (DBaaS) have also been studied [25,26,27].

Cloud computing [28,29] provides shared computer processing resources and data to computers and other devices on demand. In this study, we employed the cloud model to create a three-dimensional (3D) reconstruction framework for multiple remote robots. The 3D reconstruction [30,31,32,33,34,35,36] has been widely studied as an approach to enhance the sense of reality and immersion for users. However, accomplishing this technique in real-time and with high quality is challenging, particularly with multiple remote robots. In many tasks, such as rescue or tracking terrain, multiple remote robots are required to expand the working area. Each robot often has a light detection and ranging (Lidar) sensor, several cameras, and an inertial measurement unit-global positioning system (IMU-GPS) sensor attached. A Lidar sensor returns a point cloud that senses the surface of the terrain around the robot. The cameras are either 2D or stereo, and the images obtained from them are used to compute color points or texture coordinates. The IMU-GPS sensor is used to check the position and direction of each robot. An enormous amount of data is obtained from the multiple sensors attached to each robot. Processing these bulk data in real-time is impossible using a single computer with limited power. Thus, we propose a cloud system that optimizes the reconstruction of these large amounts of sensor data from multiple remote robots. In this framework, we create a master-slave server model in which the master server distributes the data to be processed to multiple slave servers to increase the computing ability. The segmentation servers use our previously proposed algorithm [37] to separate each frame of the point cloud data into two groups: ground and non-ground. The first group consists of ground points of terrain that the robots can traverse. The second group consists of non-ground points that the robots cannot traverse, such as buildings, walls, cars, and trees. The reconstruction servers process the ground and non-ground data differently. Furthermore, the non-ground data are modeled using colored particles [35]. On the other hand, the ground data are reconstructed by meshing and texture mapping [36]. The results from all reconstruction servers are then sent to a visualization server to save in a database. We employ a file database where users can access and visualize the 3D scene around all the robots.

2. Related Works

Reconstruction of 3D scenes from point clouds is an important task for many systems such as autonomous vehicles and rescue robots. Several studies have been conducted on reconstruction in both indoor and outdoor environments [30,31,32,33,34,35,36]. In the indoor environment, Izadi et al. [30] and Popescu and Lungu [31] employed a Microsoft Kinect sensor. Khatamian and Arabnia [32] reviewed several 3D surface reconstruction methods from unoriented input point clouds and concluded that none of them are able to run in real-time. In [33,34,35,36], the authors used one 3D laser range scanner combined with cameras and an IMU-GPS sensor for the outdoor environment. Because the features of ground and non-ground data are different, they have to be segmented and reconstructed separately. Over the past few years, many segmentation methods [38,39,40,41,42,43] have been proposed. However, these methods have disadvantages related to accuracy and processing time. We previously introduced a new ground segmentation algorithm [37] that works well with various terrains at high speed.

Paquette et al. [44] employed orthorgonal lines to determine the orientation of the mobile robot. In [45,46], the authors proposed a method for navigating a multi-robot system through an environment while obeying spatial constraints. Mohanarajah et al. [47] presented architecture, protocol, and parallel algorithms for collaborative 3D mapping in the cloud with low-cost robots. Jessup et al. [48] proposed a solution for refining transformation estimates between the reference frames of two maps using registration techniques with commonly mapped portions of the environment. Another study [49] investigated a visual simultaneous localization and mapping (SLAM) system based on a distributed framework for optimization and storage of bulk sensing data in the cloud. In [50,51], the authors supplemented the insufficient computing performance of a robot client using a cloud server for collaboration and knowledge sharing between robots. Sim et al. [52] proposed a calibration scheme for fusing 2D images and 3D points. For fast transfer data among the computers in the network, we use the UDP-based data transfer protocol (UDT) [53] which UDP stands for user datagram protocol. This method can transfer data at high speeds (10 Gbit/s). In our research, the data are compressed to decrease the bandwidth for transmission via the network and reduce storage space in the database. The compression is an important task in many studies [54,55,56,57]. There are two types of compression: lossy and lossless. For 2D images, we use a lossy compression method, and a lossless compression method is employed for other data. We employ a fast and high ratio compression method library [58].

The studies cited above involve server-based data processing technology for processing bulk sensor data generated from a number of clients. The insufficient computing performance of the client was obviated by employing a distributed processing environment using a cloud server. However, one common feature of the studies cited above is reliance on batch-processing methods, rather than real-time processing. In contrast, we propose a real-time platform for the reconstruction of 3D scenes from point clouds and 2D images acquired by sensors attached to multiple robots.

3. Framework Design

The cloud-based 3D reconstruction framework acquires data from multiple sensors attached to multiple remote robots. The data are used to reconstruct the 3D scene in real-time. Figure 1 gives an overview of the framework. Suppose that we have n remote robots working simultaneously, each with k sensors comprising several 2D cameras, a 3D range sensor, and an IMU-GPS. The 2D cameras return color images of the environment around each robot. In each frame, the 3D range sensor releases numerous laser rays and returns a point cloud. The IMU-GPS sensor is used to detect the position and navigation of each robot. We therefore construct a framework that creates a working environment for these n robots in real-time. The framework contains one master server, one visualization server, m segmentation servers, and m reconstruction servers (m ≥ n). Each robot has one computer that acquires all the data from multiple sensors. The data are compressed before sending to the master server via wireless communication.

Figure 1. Overview of our three-dimensional (3D) reconstruction framework based on the cloud for multiple remote robots.

After considering the computing resources of all slaves, the master distributes the data to the m segmentation servers. In each segmentation server, the data are decompressed, then, by using IMU-GPS data, and all points are converted from local to global coordinates. We separate the point cloud into two parts: ground and non-ground. The segmentation result is again compressed before being sent to its corresponding reconstruction server. In each reconstruction server, all of the data are first decompressed and then used to reconstruct the virtual 3D scene. Then, the results from all reconstruction servers are compressed again and transferred to the visualization server to be saved in the database. Users can then access the database and obtain the 3D scene of the working area of all the remote robots.

3.1. Ground Segmentation Algorithm

We employ our ground segmentation algorithm [37]. The main idea of this algorithm is to divide a point cloud into small groups, each called a vertical-line. The pseudo-code representing our idea is shown in Algorithm 1. First, we decompress the point cloud and IMU-GPS data. As the frame data gained from the range sensor are in the local coordinates, we have to convert them into global coordinates. We employ the method presented in [35] to change the coordinates of all 3D points in each frame dataset with small distortions. For example, robot #k starts collecting 3D points at time $t_{\mathrm{i}}$. At time $t_{\mathrm{j}}$, the local coordinate system of a 3D point $P_{\mathrm{C}}$ is centered at robot position $L_{\mathrm{j}}$. To change the coordinate system of all points, we use the following formula:

$$P_{\mathrm{G}}=R_{\mathrm{j}}\left(P_{\mathrm{C}}+L_0\right)+L_{\mathrm{j}}-L_{\mathrm{i}}$$

(1)

where $L_0$ is the 3D vector from the 3D sensor location to the GPS sensor location. $L_{\mathrm{i}}$ is the 3D vector from the starting position of robot #k to one fixed point, which is chosen as an origin of the global coordinate. $R_{\mathrm{j}}$ is the rotation matrix of robot #k at time $t_{\mathrm{j}}$. The error of this method is only a few centimeters. In addition, this method is capable of merging point clouds captured by many robots. After segmentation, we compress ground and non-ground groups separately before sending them to a corresponding reconstruction server.

Algorithm 1 Ground segmentation

Convert all points into global coordinate()

FOR EACH vertical-line IN a frame data

WHILE NOT (all points in the vertical-line are labeled)

Assign a start-ground point()

Find a threshold point()

Assign points’ label()

END

END

3.2. Terrain Reconstruction Algorithm

The method used to reconstruct the 3D terrain is based on the ideas presented in [35,36]. The architecture running in each reconstruction server is illustrated in Figure 2. Following the ground segmentation step, each set of frame data is divided into two groups. We apply a projection method using texture buffers and non-ground points. A set of colored points is obtained using the calibration method [52]. These colored points are split into several nodes based on the x–z coordinates. Then, the non-ground result of each node is compressed and sent to the visualization server. The ground points are used to update the ground mesh, which is a set of nodes. The size of each ground node is equal to the size of the non-ground node outlined above. We then implement texture mapping from the ground mesh and texture buffers. The ground surface result is also compressed and sent to the visualization server. We employ the zip library (ZLIB) [58] for compression and decompression because it is lossless and fast. Each block in Figure 2 is described in detail in [35,36]. However, sending the ground and non-ground reconstruction result data is a challenging task, even after compressing the data. More details of the sending result data are given in the ensuing section.

Figure 2. Terrain reconstruction framework in each reconstruction server.

3.3. Reconstruction Data Processing

In previous studies [47,48,49,50,51], colored points were used to make particles. Thus, the size of the resulting data was relatively small. We also do not need to consider any overlapping problem. In our system, we model ground data via triangular meshes and texture mapping to enable better reconstruction quality. We employ a high speed, accurate, and stable method [53] for transferring big data via the network. However, sending the ground surface result in every frame would result in two problems. First, the size of the data is significant and the network cannot satisfy the transfer. Second, it is difficult to avoid overlapping data in a visualization server. To solve these problems, we send the reconstruction results of the ground and non-ground data in different ways. For colored non-ground points, we count the total number of points. If the number of colored points is larger than a maximum value, we compress all node data and send the data to the visualization server. For the ground surface data, we propose a new transmission algorithm. As mentioned above, the ground data are divided into a set of mesh nodes. Each mesh node is processed independently; hence, we also send the surface data of each node separately using Algorithm 2. The Threshold is one of the values used to consider sending ground surface data. In general, if the Ratio is smaller than the Threshold, the ground surface data are not sent. Never_Send_Data is a global variable with an initial value TRUE. If the function returns FALSE, it means no data should be sent. Otherwise, we compress the data and send them to the visualization server. Figure 3 represents the working area of each robot. The sending data of each robot are independent. Each node is represented by a cell and the blue line demonstrates the orthogonal line mentioned in Algorithm 2. Each robot has one orthogonal line in the x–z surface. The orthogonal line has two characteristics. First, it goes through each robot’s position. Second, it is perpendicular to the robot’s direction.

Algorithm 2 Sending ground surface data

Ratio ← Number_Of_Triangles/Maximum_Triangles

Estimated_Ratio ← Number_Of_Triangles/Estimated_Number_Of_Triangles

Threshold ← Medium_Value

Intersection_Status ← Check intersection between orthogonal line and Node()

Dynamic_Increasing_Delta_1 ← (1 − Previous_Ratio)/2

IF Intersection_Status = TRUE THEN

IF Never_Send_Data = TRUE THEN

Threshold ← Small_Value

ELSE IF (Ratio − Previous_Ratio) < Dynamic_Increasing_Delta_1 THEN

RETURN FALSE

END

END

IF Never_Send_Data = TRUE THEN

IF (Ratio < Threshold) OR (Estimated_Ratio < Threshold) THEN

RETURN FALSE

ELSE

Previous_Ratio ← Ratio

Never_Send_Data ← FALSE

END

ELSE

Dynamic_Increasing_Delta_2 ← (1 − Previous_Ratio)/4

IF (Ratio − Previous_Ratio) < Dynamic_Increasing_Delta_2 THEN

Dist ← Distance between Node’s center and orthogonal line()

IF (Dist < Size_Of_Node) AND (Ratio > Previous_Ratio) THEN

Previous_Ratio ← Ratio

ELSE

Return FALSE

END

END

Previous_Ratio ← Ratio

END

END

Compress Data()

Send Data()

RETURN TRUE

Figure 3. Example of working area for each robot from the top view.

To calculate the distance from each node’s center to the orthogonal line in Algorithm 2, we employ Equation (2). In this equation, V and P are the current direction and the position of the mobile robot, respectively, and $C_{\mathrm{i}}$ is the center of node i:

$$d_{\mathrm{i}}=\frac{\left|V_{\mathrm{x}}{C}_{\mathrm{i},\mathrm{x}}+V_{\mathrm{z}}{C}_{\mathrm{i},\mathrm{z}}-V_{\mathrm{x}}{P}_{\mathrm{x}}-V_{\mathrm{z}}{P}_{\mathrm{z}}\right|}{\sqrt{V_{\mathrm{x}}^2+V_{\mathrm{z}}^2}}$$

(2)

We use two diagonal lines to check the intersection between the orthogonal line and each node, as shown in Figure 3. We also calculate it using the four functions illustrated in Equation (3). If $f_{1,\mathrm{i}}{f}_{3,\mathrm{i}}<0\mathrm{or}{f}_{2,\mathrm{i}}{f}_{4,\mathrm{i}}<0$, we conclude that the orthogonal line cuts node i. In Equation (3), s is the size of each node.

$$\{\begin{array}{l}{f}_{1,\mathrm{i}}=V_{\mathrm{x}}(C_{\mathrm{i},\mathrm{x}}-\frac{s}{2})+V_{\mathrm{z}}(C_{\mathrm{i},\mathrm{z}}-\frac{s}{2})-V_{\mathrm{x}}{P}_{\mathrm{x}}-V_{\mathrm{z}}{P}_{\mathrm{z}}\\ f_{2,\mathrm{i}}=V_{\mathrm{x}}(C_{\mathrm{i},\mathrm{x}}+\frac{s}{2})+V_{\mathrm{z}}(C_{\mathrm{i},\mathrm{z}}-\frac{s}{2})-V_{\mathrm{x}}{P}_{\mathrm{x}}-V_{\mathrm{z}}{P}_{\mathrm{z}}\\ f_{3,\mathrm{i}}=V_{\mathrm{x}}(C_{\mathrm{i},\mathrm{x}}+\frac{s}{2})+V_{\mathrm{z}}(C_{\mathrm{i},\mathrm{z}}+\frac{s}{2})-V_{\mathrm{x}}{P}_{\mathrm{x}}-V_{\mathrm{z}}{P}_{\mathrm{z}}\\ f_{4,\mathrm{i}}=V_{\mathrm{x}}(C_{\mathrm{i},\mathrm{x}}-\frac{s}{2})+V_{\mathrm{z}}(C_{\mathrm{i},\mathrm{z}}+\frac{s}{2})-V_{\mathrm{x}}{P}_{\mathrm{x}}-V_{\mathrm{z}}{P}_{\mathrm{z}}\end{array}$$

(3)

To save the reconstruction result in the visualization server, we create a database with the data structure shown in Figure 4. This database is organized in a set of nodes, each containing data about a cube area, which is equal to the size of each node in the reconstruction step. We divide each node’s data into four parts. The first part is colored non-ground points. For visualization in the user’s application, we perform particle rendering from these colored points. The other parts are the texture image, vertex buffer, and index buffer containing data about the ground surface. In order to speed up writing and save the storage space, all data are stored in compressed format.

Figure 4. Database structure in the visualization server.

4. Experiments and Analysis

We conducted two experiments and evaluated the proposed framework. We employed multiple datasets comprising data captured from a Lidar sensor (Velodyne HDL-32E, Velodyne Inc, Morgan Hill, CA, USA), an IMU-GPS sensor (customized by us), and three 2D cameras (Prosilica GC655C, Allied Vision Inc, Exton, PA, USA) to simulate multiple robots. Table 1 gives a listing of the computers used in the system. We used three segmentation servers and three reconstruction servers. In each reconstruction server, we employed an NVIDIA graphics card (NVIDIA Inc, Santa Clara, CA, USA), each of a different type and with different amounts of memory, for graphics processing unit (GPU) computing. The user’s application was executed on the same PC as the visualization server. Our experiments utilized three simulation robots. The robots sent the IMU-GPS data, point clouds, and images to the master server in real-time and each robot returned approximately 60,000 points per second. As the frame rate of the Velodyne sensor is 10 fps, we therefore captured 10 images per second from each 2D camera. Using three cameras, we obtained 30 images per second. The resolution of each 2D image was 640 × 480 pixels. To reduce the size of the image data, we needed a fast and high ratio compression. In our experiments, we employed a lossy compression method with JPEG standard. For storing data in the visualization server, we used a file database. We created four folders to contain the colored non-ground points, texture images, vertex buffers, and index buffers, with the corresponding data saved in four files in corresponding folders. Each file was separated by name, which is the coordinate of the node’s center. For future work, we will consider employing another database management system (DBMS). For high-speed data transfer, we employed the UDT protocol [53] which is a UDP-based method.

Table 1. Configuration of the computers used in the experiment.

PC	Central Processing Unit (CPU)	Random-Access Memory (RAM) (GB)	Graphics Processing Unit (GPU)	Video RAM (VRAM) (GB)
Master server	Core i7, 2.93 GHz	12	-	-
Segmentation sever 1	Core i7-2600K, 3.4 GHz	8	-	-
Segmentation sever 2	Core i7-2600K, 3.4 GHz	8	-	-
Segmentation sever 3	Core i7-2600K, 3.4 GHz	8	-	-
Reconstruction sever 1	Core i7-6700, 3.4 GHz	16	NVIDIA GTX 970	12
Reconstruction sever 2	Core i5-4690, 3.5 GHz	8	NVIDIA GTX 960	6
Reconstruction sever 3	Core i7-6700HQ, 2.6 GHz	8	NVIDIA GTX 960M	4
Visualization server	Core i7-6700, 3.4 GHz	16	-	-
User	Core i7-6700, 3.4 GHz	16	-	-

4.1. Experimental Results

After running the experiments, the user was able to see the results in real-time. Figure 5 and Figure 6 show the results of the first experiment. Figure 5a,b demonstrate the 2D and 3D scenes of the user viewer after running the system for a few seconds. Figure 5c,d show the results after the three robots worked for approximately one minute. The three robots scanned a mountain road approximately 800 m long. Each blue cell in the 2D map denotes one node from the top view. The x–z size of the node is 12.7 m. We also captured four 3D screenshots, as shown in Figure 6. These screenshots were taken at the positions illustrated in Figure 5c. Using the same method, Figure 7 and Figure 8 show the results of the second experiment. The results show that our framework is able to reconstruct 3D scenes from multiple remote robots at the same time. As shown below, we successfully generated visualization results using the proposed framework in real-time.

Figure 5. Real-time visualization results viewed by the user in the first experiment: (a) 2D scene after three robots ran for a few seconds; (b) 3D scene after three robots ran for a few seconds; (c) 2D scene after three robots worked approximately 1 min; and (d) 3D scene after three robots worked approximately 1 min.

Figure 6. Screenshots of the 3D scene: (a–d) at positions 1, 2, 3, 4 in Figure 5c, respectively.

Figure 7. Real-time visualization result viewed by the user in the second experiment: (a) 2D scene after three robots ran for a few seconds; (b) 3D scene after three robots ran for a few seconds; (c) 2D scene after three robots worked approximately 50 s; and (d) 3D scene after three robots worked approximately 50 s.

Figure 8. Screenshots of the 3D scene: (a–d) at positions 1, 2, 3, 4 in Figure 7c, respectively.

4.2. Experimental Analysis

To evaluate the proposed system, we measured the processing time and network usage. The segmentation and reconstruction time per frame in each server are depicted in Figure 9 and Figure 10. Table 2 shows the average segmentation time of each server. The average processing time per frame in each segmentation server of the first and second experiment are 2.69 and 2.84 ms, respectively. Each segmentation server, therefore, can run at a rate of 352 fps. Table 3 shows the average reconstruction time of each reconstruction server. The average reconstruction time per frame is 34.57 ms for the first experiment and 37.41 ms for the second. Hence, each reconstruction server is capable of running at a rate of 27 fps. The frame rate of the Velodyne sensor is 10 fps; hence, our system can process data in real-time. Figure 11 and Figure 12 show the total network usage for each robot per second. Table 4 presents the average network usage of each robot in megabytes per second. In the first experiment, 4.28 MB/s is needed for each robot to transfer data via the network. The average bandwidth for each robot is 3.99 MB/s in the second experiment. In both experiments, the required bandwidth is higher than the bandwidth requirement of [47] (500 KB/s) and [49] (1 MB/s). However, whereas we utilized ground mesh and texture mapping, previous studies used only color points. In addition, we employed different sensors and our method has better visualization quality in outdoor environments than [47,49]. The results show that our system can perform suitably in the environment using wireless communication.

Figure 9. Segmentation and reconstruction time in milliseconds in each server per frame in the first experiment: (a,c,e) Segmentation time per frame in segmentation servers 1, 2, and 3, respectively; and (b,d,f) reconstruction time per frame in reconstruction servers 1, 2, and 3, respectively.

Figure 10. Segmentation and reconstruction time in milliseconds in each server per frame in the second experiment: (a,c,e) Segmentation time per frame in segmentation servers 1, 2, and 3, respectively; and (b,d,f) reconstruction time per frame in reconstruction servers 1, 2, and 3, respectively.

Table 2. Average processing time in each Segmentation server.

Server	Processing Time (ms)
	Experiment 1	Experiment 2
Segmentation server 1	2.33	2.91
Segmentation server 2	2.78	2.75
Segmentation server 3	2.97	2.87
Average Time	2.69	2.84

Table 3. Average processing time in each Reconstruction server.

Server	Processing Time (ms)
	Experiment 1	Experiment 2
Reconstruction server 1	31.55	36.14
Reconstruction server 2	34.41	37.04
Reconstruction server 3	37.74	39.06
Average Time	34.57	37.41

Figure 11. Network usage (in megabytes per second) for each robot in the first experiment.

Figure 12. Network usage (in megabytes per second) for each robot in the second experiment.

Table 4. Average network usage for each robot.

Robot	Network Usage (MB/s)
	Experiment 1	Experiment 2
Robot 1	4.11	3.97
Robot 2	4.40	3.82
Robot 3	4.34	4.17
Average Value	4.28	3.99

5. Conclusions

In many systems, such as rescue or tracking terrains, multiple remote robots working simultaneously result in more efficiency than a single robot. However, reconstruction for multiple remote robots is challenging. This paper proposed a 3D reconstruction framework for multiple remote robots on the cloud. The proposed framework utilizes a master-slave server model, thereby enabling distributed processing of bulk data generated by multiple sensors. To evaluate our proposed framework, we conducted experiments in which three remote robots were simulated. The input data were processed on segmentation servers and reconstruction servers. Then, the resulting data were fused in a visualization server. The experimental results obtained confirm that our system is capable of providing real-time 3D scenes of the surroundings of all remote robots. In addition, users need not consider the computing resource. Further study is needed to expand the proposed framework to reduce the network usage and apply it to real robots. We plan to increase the quality of the 3D reconstruction by meshing and texture mapping non-ground data. We also plan to research special cases such as limited bandwidth and varying ability of the master server.