Performance Investigation and Repeatability Assessment of a Mobile Robotic System for 3D Mapping †

Preliminary Comparison between Handheld and Mobile Robotic Mapping Systems. In 2021. Abstract: In this paper, we present a quantitative performance investigation and repeatability assessment of a mobile robotic system for 3D mapping. With the aim of a more efﬁcient and automatic data acquisition process with respect to well-established manual topographic operations, a 3D laser scanner coupled with an inertial measurement unit is installed on a mobile platform and used to perform a high-resolution mapping of the surrounding environment. Point clouds obtained with the use of a mobile robot are compared with those acquired with the device carried manually as well as with a terrestrial laser scanner survey that serves as a ground truth. Experimental results show that both mapping modes provide similar accuracy and repeatability, whereas the robotic system compares favorably with respect to the handheld modality in terms of noise level and point distribution. The outcomes demonstrate the feasibility of the mobile robotic platform as a promising technology for automatic and accurate 3D mapping.


Introduction
Nowadays, acquiring and recording high-resolution 3D information of internal and external environments is crucial in the study and analysis of both buildings and human settlements [1]. This need is highlighted by the United Nations 2030 Agenda for Sustainable Development, which has underlined the importance of making cities and human settlements inclusive, safe, resilient and sustainable [2]. In detail, Sustainable Development Goal 11 (SDG11) indicates the targets of enhancing inclusive and sustainable urbanization as well as protecting and safeguarding the world's cultural and natural heritages. At the same time, SDG11 outlines the aim of substantially increasing the number of cities and human settlements adopting and implementing integrated policies and plans toward resource efficiency, adaptation to climate change and resilience to disasters. In this framework, the availability of efficient and cost-effective surveying techniques for semi-structured or unstructured environments is essential, especially where external methods for localization, such as a Global Navigation Satellite System (GNSS), are not always available.
The surveying of civil structures, usually performed by means of classical mapping technologies, such as Photogrammetry and Terrestrial Laser Scanning (TLS), has been revolutionized in recent years by using portable Mobile Mapping Systems (MMSs) [3]. Trolley, backpack and handheld devices [4] can be easily carried by a person acquiring accurate 3D data of surrounding environments by simply walking through the area of interest, as shown in [5]. The core of this surveying technology is Simultaneous Localization and Mapping (SLAM) algorithms, thanks to which 3D models of the environment can be obtained without requiring a GNSS signal. Initially developed by the robotics community to allow the autonomous navigation of a mobile platform in an unknown environment [6], SLAM methods are now also employed to estimate the trajectories of moving devices (RGB cameras, laser scanners and other active sensors) while acquiring accurate maps of an area of interest [7].
Several studies examined the advantages and disadvantages of portable MMSs in diverse test fields, as demonstrated by flourishing literature on the topic [8]. For example, in [9], a wearable mobile laser system was evaluated for the indoor 3D mapping of a complex historical site, whereas in [10], different handheld mobile solutions were tested for the 3D surveying of underground built heritage. Moreover, in [11], the performance of a handheld laser scanner was investigated in different outdoor scenarios, such as the surveying of a building façade and a mountain torrent. These studies reported accuracies ranging from a few centimeters to 10 cm for the surveying of indoor environments and urban areas with portable MMSs.
Mapping sensors are also adopted to inspect constructions or manufactured parts and identify possible discrepancies between as-built workpieces and their nominal specifications [12]. The data acquired by portable mapping devices are essential for creating a Building Information Model (BIM) according to the as-built condition of a structure [13][14][15]. For example, in [16], an approach for the semi-automated generation of a parametric BIM for steel structures based on TLS data was presented. Furthermore, surveyed data recently proved to be fundamental in supporting functional and occupancy analyses of buildings in challenging situations, such as the COVID-19 pandemic [17].
The reliability and completeness of 3D information on buildings and human settlements could be improved, and acquisitions could be automated by mounting mapping devices (e.g., laser scanners and RGB-D cameras) on robotic platforms. Such a platform can be steered remotely or can autonomously perform the required surveying of an area of interest, optimizing the acquisition process.
Today, mobile robots are being increasingly applied to mapping, and the importance of surveying automation is confirmed by a wide variety of applications in several different fields. Mobile platforms are indeed suitable for explorations and inspections in hazardous and challenging environments [18,19] and are used for inspections in disaster situations and rugged environments [20] instead of humans, ensuring safety and repeatability. In the agricultural field, both wheeled and tracked mobile vehicles are employed for orchard mapping and precision farming [21,22], harvest supporting and 3D mapping in greenhouses [23] as well as the detection and segmentation of plants and the ground [24]. A survey of localization and mapping by robots in agriculture and forestry is available in [25].
Mobile robots are also employed for the mapping of archaeological and cultural heritage sites, as shown in [26,27]. Finally, the digitalization of buildings is another relevant application of mobile systems based on robotic platforms and autonomous vehicles [28]. For instance, in [29], a large-scale 3D model of buildings was obtained by using multiple cooperative robots and by considering several criteria, such as visibility between robots, error accumulation and efficient traveling. A review on autonomous mobile systems for the digitization of buildings can be found in [30].
A fundamental aspect of autonomous mobile scanning systems is the quantitative evaluation related to a resulting 3D model. Indeed, the quality of a final point cloud is a pillar in regards to extracting the metric information of a scene and, possibly, a BIM model. Moreover, a realistic model is not guaranteed if the precision of a 3D map is not evaluated properly [30]. By considering handheld MMSs, and more generally standard geomatics surveying techniques, the acquired data are usually compared with already available ground truths obtained with more accurate surveying technologies, such as TLS (e.g., in [8,31]).
When moving to mobile robotic systems, only a paucity of works deal that with robot-based 3D mapping report comparisons with ground truths. For instance, in [32], previously acquired scans were assumed as ground truths, and the deviation angle from each reference axis in degree and the root mean-square error (RMSE) in meters were measured. Furthermore, in [33], rotational and translational estimation errors verified the accuracy of 3D mapping performed by a simulated mobile robot in multiple outdoor environments. However, as reported in [30], a high percentage of mobile scanning methods available in the literature did not provide reports concerning the accuracy of 3D maps obtained against ground truths.
Another important aspect that requires proper attention is survey repeatability, defined as the variation that can be expected in a final 3D model when mapping the same area under similar conditions (i.e., using the same scanner device, moving platform and trajectory) within a short time interval [34]. While comparing a single resulting map with a ground truth provides an estimate of accuracy and systematic errors, assessing repeatability allows one to quantify the model precision and uncertainties that can significantly influence change-detection analyses [35].
A preliminary comparison of handheld and mobile robotic mapping systems was discussed in [36]. However, in that work, no comparison with a ground truth was shown and only one survey did not guarantee a repeatability and reliability assessment.
In this paper, we present the performance and repeatability assessment of a mobile robotic system for 3D mapping. In detail, we used the commercial HERON Lite system composed of a 3D laser scanner coupled with an Inertial Measurement Unit (IMU), which realizes the high-resolution mapping of a surrounding environment and can provide realtime results. Experimental tests were carried out, performing a survey of the ground floor of the Rizzi building of the University of Udine (Italy) and considering two approaches: • Case 1: the acquisition device installed on a mobile robot (referred in the following as robotic mode (R)). • Case 2: the acquisition device attached to a telescopic pole and carried manually (referred to as handheld mode (H)).
Since the main contribution of this work is a comparison of the robotic and handheld mapping modes, for a fair analysis in both cases, the acquired data were processed with the same SLAM algorithm and software parameters, i.e., the proprietary HERON Desktop package provided by the manufacturer of the 3D mapping system.
An analysis of the results shows that the robotic mode compares favorably with respect to the handheld one and confirms the feasibility of using the robotic system for future automatic surveys with the aim of supporting renovation projects for sustainable cities.
This paper is an evolved version of a preliminary conference work in [36] and extends that work as follows: • The method is validated by comparing the point clouds obtained in both Cases 1 and 2 with a ground truth based on a previously acquired TLS survey. • The repeatability of the mapping is assessed by performing multiple scans and evaluating the consistency of the results. • A new experimental setup for the laser scanner installed on the mobile robot is developed to improve the point of view of the sensor during the robotic mapping. The higher point of view provides advantages in cluttered environments, limiting occlusions and data gaps in the model caused, e.g., by furniture.
The remainder of the paper is organized as follows: In Section 2, the materials and methods used in this work are described, whereas in Section 3, the experimental results are presented. A discussion of the results is given in Section 4. Finally, the paper is concluded in Section 5.

Materials and Methods
In this section, we first present the system setup and the sensor characteristics, then we describe the procedure for the experimental data acquisition and processing. Finally, the acquisition of the ground-truth point cloud is illustrated.

System Setup and Sensor Characteristics
The experimental evaluation was carried out using the portable MMS HERON Lite by Gexcel srl [37]. This device is composed of a Velodyne Puck LITE laser scanner [38] coupled with an Xsens MTi inertial sensor [39], the data from which were used for the system trajectory estimation. Sixteen channels characterize the scanning head and emit infrared laser beams at a wavelength of 903 nm, allowing one to acquire 300,000 points/s in a single return mode at a maximum distance of 100 m and to cover a 360 • horizontal field of view and a 30 • vertical field of view. The sensor head is connected to a battery and a control unit for data storage and real-time processing. According to the manufacturer's specifications [37], the employed system ensures a precision of 3 cm and a final global accuracy of 5 cm. However, this can be downgraded to 20-50 cm depending on the followed trajectory and the geometry of the scanned environment, which can significantly influence the results of the SLAM algorithm, as shown in [11].
In its standard configuration, the HERON Lite system is a handheld device; thanks to a telescopic carbon-fiber pole, it can be manually carried by a surveyor during mapping operations, as shown in Figure 1a (Case 2). In our experiments, the HERON Lite device was also installed on a mobile robotic platform with an angle of approximately 30 • between the vertical direction and the laser rotation axis (Case 1). Unlike the previous work [36], the scanning head was fitted on a custom aluminum support at a height of about 1 m from the robot base ( Figure 1b). This new setup ensured a more adequate point of view of the scanning head, limiting unreliable measurements (at a range of <1 m) of the floor and providing advantages in cluttered indoor environments. Regarding the mobile platform, we adopted the MP-500 robotic system by Neobotix GmbH [40], which is equipped with two drive wheels, an undriven castor wheel and a SICK 2D laser scanner used for safety purposes. The platform has dimensions of 81.4 × 59.2 × 36.1 cm, a payload of 80 kg and maximum speed of 1.5 m/s. This mobile robot is usually employed for autonomous transportation operations in industrial environments, surveillance and part handling within large areas and measurement of physical data (e.g., temperatures and gas concentrations) in indoor environments. During the surveys, the mobile robot was steered remotely using a wireless joystick.
With the HERON Lite system, a preliminary map of a scanned environment can be obtained in real time and visualized on the screen of the control unit. However, for an accurate and dense 3D model, data post-processing via the software HERON Desktop [41] is required. Indeed, the raw data of this system are saved in only proprietary file formats and cannot be exported for processing with alternative algorithms.
In detail, with the proprietary package, the trajectory and the map are computed in a three-stage pipeline. The first step, called Odometer, implements an online SLAM algorithm to retrieve a preliminary estimate of the sensor trajectory. This approach computes the pose of each cloud (a single 360 • scan) by aligning it to the previous ones via the Iterative Closest Point (ICP) algorithm [42]. In this stage, IMU data provide a rough tentative pose of the scanner. Since this method is prone to drift accumulation, a subsequent refinement is needed. To this end, the Create Maps step (named in the proprietary software interface) splits the full trajectory into Local Maps, which are subsequently treated as internally rigid point clouds. The latter represent the inputs for the final stage, known as Global Optimization, which corresponds to a full SLAM method [43]. With this approach, it is possible to consider loop closures (if any) along the trajectory (i.e., resurveyed areas) that are treated as consistency constraints in the global registration process among all the local maps. In this way, error accumulation along the trajectory is minimized, resulting in more accurate 3D point clouds of the surveyed environment. Finally, the obtained complete map can be automatically filtered to remove moving objects (e.g., walking people), applying a process called Clean Data.

Experimental Data Acquisition and Processing
As a case study, the ground floor of the west-wing corridor of the Rizzi building of the University of Udine (Italy) was surveyed. It is characterized by a square-based plant with dimensions of 80 m × 80 m and was mapped following a closed-loop trajectory (shown in blue in Figure 2). Please note that the south part of the corridor was surveyed twice-at the beginning and at the end of the trajectory. As already mentioned, both the roboticand handheld-mode mapping were performed, repeating the survey three times with each platform. In the following, we refer to the robotic cases as R-1, R-2 and R-3; similarly, the handheld ones are called H-1, H-2 and H-3. Table 1 summarizes the main characteristics of the data acquisitions. Please note that for a fair comparison, all the surveys were processed using the same software parameters and the same SLAM algorithm (provided by the manufacturer of the system) previously described. Figure 2 shows an X-ray orthophoto of the mapped corridor extracted from the R-1 model by projecting the point cloud on the horizontal plane.

Ground-Truth Acquisition
For a quantitative assessment of the obtained 3D point clouds, we employed a "static" terrestrial laser scanner (TLS) to acquire a reference model. The instrument was the RIEGL Z390i system (Figure 3a), a time-of-flight laser scanner with an angular measurement resolution of 1 mgon, a declared accuracy of 6 mm and a repeatability of 4 mm for single shots and 2 mm for averaged measures. Thanks to the high accuracy and precision that can be reached with this scanning device, the TLS point cloud could be considered the ground-truth model.
In detail, a TLS data acquisition was performed on the south part of the corridor (corresponding to the dashed rectangle in Figure 2), which has dimensions of 42 m × 7 m. The TLS instrument was vertically fixed on three positions (2V, 3V and 4V, where V stands for vertical) along the corridor axis; 2V and 4V were symmetrically placed with respect to 3V, located in the corridor's center (Figure 3b). From each scan position, once fixed to 0.10 • the horizontal/vertical angular step, a sequence of three panoramic scans was performed, recording on average 2.84 × 10 6 points. Moreover, photogrammetric images taken with a Nikon D200 integrated metric camera were panoramically acquired to color the TLS scans with RGB values. Each scan sequence was then resampled, keeping the same angular acquisition resolution; the obtained "mean scans" had a noise, verified in the correspondence of planar surfaces, of about 2 mm, starting from scan sequences characterized by a noise of 3 mm. In order to register the three obtained scans, 16 reflecting disks 5 mm in diameter and 18 reflecting cylinders 10 mm in diameter and height were suitably placed in the corridor, mainly along the walls (Figure 3b). The 3D coordinates of the reflecting disks were measured using a Leica TCRA 1201+ total station. This is a high precision topographic instrument characterized by a nominal accuracy of 1 (0.3 mgon) for angular measurements and 1 mm + 1.5 ppm for distance measurements. The topographic survey of the TLS targets was conducted from three vertices roughly coinciding with the TLS stations in order to ensure the best visibility of the targets. Angular and distance measurements were taken on both the left and right faces of the instrument and then averaged. In order to refer the topographic measurements of the three stations to the same datum, in addition to the targets for the scans, two additional stable references were installed consisting of high-accuracy Leica GPR121 topographic round prisms, which were accurately measured from each station. The topographic measurements were least squares adjusted using MicroSurvey STAR*NET v. 7.1 [44], an advanced software solution used in the topographic field to estimate the coordinates of surveyed points together with their precision and reliability, allowing one to identify possible systematic errors and to verify that the results meet the accuracy requirements.
The uncertainties obtained in the final coordinates of the reflecting circular targets ranged from 0.5 to 4.1 mm along the three axes. Exploiting these points as control points, the three TLS clouds 2V, 3V and 4V were registered in a global point cloud with RIEGL RiSCAN PRO v. 1.5, the companion software for RIEGL TLS systems [45], obtaining residuals with a standard deviation of 4.5 mm along the X axis (corresponding to the longitudinal axis of the corridor), 2.4 mm along the Y axis and 3.4 mm along the vertical component Z. These values confirmed that the global TLS cloud could be adopted as a reference model.
As a last remark, since the robotic and handheld point clouds were expressed in arbitrary reference frames, the subsequent comparison with the ground truth required a preliminary alignment to the reference model that was performed with the software JRC 3D Reconstructor (v. 4.3.1) [46] using the ICP algorithm. In addition to the various tools for point-cloud processing, the chosen software allowed us to directly import the results from HERON Desktop. Figure 4 shows the point clouds of the analyzed corridor obtained with the robotic mapping platform (Figure 4a) and with the TLS (Figure 4b).

Experimental Results
As an initial quantitative evaluation of the point-cloud accuracy obtained with the mobile robotic mapping platform and with the handheld survey modality, the cloud-tocloud absolute distances (C2C) with respect to the ground-truth model were estimated for each survey using the open-source software CloudCompare (v. 2.11.3) [47]. Figure 5a,b show the outcomes for R-1 and H-1, respectively, whereas Table 2 reports the mean and standard deviation values for the six analyzed cases. Moreover, graphical representations of the results in the forms of box plots are shown in Figure 6a. From a visual inspection, it can be noticed that higher residuals were located in the central parts of the walls for both the robotic and the handheld surveys, indicating the presence of a relative deformation with respect to the reference model in this area. In any case, the computed 3D distances did not exceed 6 cm and were compatible with the accuracy of the instrument according to the manufacturer's specifications.    H-1). For grey points, the C2C distance could not be computed due to data gaps in the TLS model.  As highlighted in the previous section, the studied corridor is mapped at the beginning and at the end of the trajectory; the low-distance values revealed by the C2C analysis represent a confirmation that no relevant residual misalignment is present at the loop closure. Moreover, the results are consistent among the three data acquisitions with each platform, indicating a high survey repeatability.
To further investigate the repeatability, C2C distance values were evaluated between pairs of surveys carried out with the same mapping mode (i.e., R-1 vs. R-2, R-1 vs. R-3 and R-2 vs. R-3 and H-1 vs. H-2, H-1 vs. H-3 and H-2 vs. H-3). The results are reported in Figure 7, which shows the histograms of the C2C distance distribution for all the comparisons. The outcomes are also summarized in Table 3. The values of mean and standard deviation always lower than 1 cm further demonstrate the high repeatability of both survey modalities.
Observing, instead, the point-cloud density that characterizes the two acquisition modes, it can be noticed from Figure 8 that the mobile robotic platform guarantees a more uniform point distribution, whereas the handheld survey is characterized by low density values on the floor. Moreover, the statistics reported in Table 4 highlight an overall higher surface density for the robotic mapping cases (mean values ranging from 20,300 points/m 2 to 21,043 points/m 2 ), showing a minor variability among the three survey repetitions (as confirmed by the box dimensions in Figure 6b). This behaviour can derive from (i) the lower speed of the robot with respect to the walking operator (see the acquisition time reported in Table 1) and (ii) the constant speed maintained during the robotic acquisitions. Table 3. Cloud-to-cloud absolute distances computed between pairs of robotic and handheld surveys for repeatability assessment.

R-1 vs. R-2 R-1 vs. R-3 R-2 vs. R-3 H-1 vs. H-2 H-1 vs. H-3 H-2 vs. H-3
Mean (     In addition to a more consistent point density, the point cloud acquired with the robotic system shows a slightly lower level of noise, confirming the results reported in our preliminary work [36]. As suggested in [5], this feature was quantitatively assessed by extracting three patches of the point clouds (one on each wall and one on the floor) and fitting a plane to each subset. Table 5 reports the Root Mean Square (RMS) of the distances of the points from the corresponding estimated plane. An average RMS of 1.7 cm was estimated for the robotic mapping cases, whereas an RMS of 2.0 cm resulted from the handheld-mode surveys. As expected, the TLS significantly outperformed the mobile systems. These outcomes can be explained by the fact that the robot performed the survey at a lower speed and, above all, avoided sudden movements and oscillations to which the laser scanner was subjected when it was manually carried by a walking person. In this regard, Figure 9b clearly shows the oscillations that characterized the trajectory followed by the scanning sensor in the handheld-mode survey that were not present when the device was installed on the robotic platform ( Figure 9a). Moreover, the data acquired by the IMU are influenced by the characteristics of the trajectory. Indeed, the IMU data are subsequently employed in the SLAM process and thus play an important role in the creation of the final point cloud. Table 5. Results of the plane-fitting procedure. The RMS of the distances between the points and the corresponding estimated plane are reported for each patch, giving an indication of the sensor noise.  Finally, in Figure 10, we report a detail of the point clouds acquired with the two survey modalities. Thanks to the higher density, the more uniform point distribution and the slightly lower noise level, in the point cloud retrieved via the robotic mapping ( Figure 10a) the objects visually appear sharper (see, e.g., the baskets and the items hanging on the walls). This feature could be helpful for the digitization and vectorization processes that usually follow the point-cloud acquisition.

Discussion
The results reported in the previous section clearly show that, overall, a tight correspondence with the ground-truth model can be noticed for all the obtained point clouds (mean distances ranging from 2.0 to 2.3 cm), with no significant differences between the two survey modes. The computed C2C distances are in line with the outcomes provided by other works in the literature that assessed the accuracies of portable devices carried manually. In [11], e.g., the HERON Lite system was tested in two different outdoor scenarios, reporting an average C2C distance of 5.4 cm for the survey of a torrent reach, estimated with respect to a photogrammetric ground-truth model. Discrepancies higher than 7 cm were revealed when comparing the point cloud of a building façade provided by the portable laser scanner with the TLS model. Accuracies ranging from a few to 10 cm for the surveying of buildings and urban areas with different commercial portable mapping systems were declared in [5,8,48].
As discussed in detail in [5,11,49], one of the main drawbacks of portable scanning devices is a high level of noise that affects the resulting point cloud. Experiments carried out in previous works revealed a sensor noise of 1-2 cm for all the tested commercial systems, a result that also emerged from the experimental evaluation reported in the previous section (Table 5). In this regard, the slightly lower level of noise reached with the robotic platform, which guaranteed a smoother trajectory and limited oscillations, appears to be of significant importance.
To the best of the authors' knowledge, no similar works performing a comparison between handheld and robotic mapping platform can be found in the current literature, making a direct comparison with the presented results unfeasible. Furthermore, as highlighted in [30], several approaches based on robotic platforms for 3D mapping did not provide any quantitative evaluations of the accuracies of the obtained models, with some reporting only non-metric information, such as the number of unobserved cells in a scan position [50]. A limited number of studies provided quantitative details regarding accuracy and precision. For instance, in [29], the accuracy of a 3D model was evaluated by comparing the positions of only six corners, obtaining an average error of 23.1 mm, whereas the work in [32] reported a centimeter accuracy in a registration process between consecutive scans. More in-depth analyses and comparisons with ground-truth models should therefore be provided in the future in the field of robotic mapping to promote the evaluation of methods and platforms.

Conclusions
The increasing demand for updated and detailed 3D models of buildings, essential in any renovation project, is fostering the development of flexible and efficient surveying techniques. To this end, robotic mapping systems could provide a fundamental aid to reduce and automate the time-consuming manual work usually required for mapping operations. In this paper, we investigated the performance of a mobile remote-controlled robot equipped with a 3D laser scanner for indoor surveys, focusing on the accuracy and repeatability that characterized the tested system. Quantitative evaluations showed a high repeatability and a tight correspondence between point clouds obtained from robotic mapping and a ground-truth model acquired with a TLS. Moreover, the robot-based acquisition platform compared favorably with respect to the handheld mode. Indeed, a higher surface density, more uniform point distribution and lower noise level were shown in the former case. These outcomes represent a further confirmation that robotic mapping could be a viable alternative to the well-established surveying mode, reducing the manual work usually required and, above all, guaranteeing, at the same time, a high quality of final results.
In the future, our research will focus on automating the surveying carried out with robotic platforms in order to perform the autonomous 3D mapping of both more-complicated inner spaces and outdoor environments without relying on human interventions. Furthermore, we will consider the adoption of different scanning sensors capable of providing raw data to be processed with open-source SLAM algorithms. In this context, different SLAM approaches will be compared to evaluate their performance in the reconstruction of a reliable 3D map. Moreover, additional sensors, such as RGB cameras and GNSS receivers, will be integrated for further applications in outdoor scenarios, e.g., 3D mapping in agriculture or in cluttered environments. Funding: This research was partially funded by the Laboratory for Advanced Mechatronics-LAMA FVG, the international research center for product and process innovation of the three Universities of the Friuli Venezia Giulia Region (Italy).

Data Availability Statement:
The data presented in this study are available on request from the corresponding author.