A Comparison of Low-Cost Sensor Systems in Automatic Cloud-Based Indoor 3D Modeling

: The automated 3D modeling of indoor spaces is a rapidly advancing ﬁeld, in which recent developments have made the modeling process more accessible to consumers by lowering the cost of instruments and o ﬀ ering a highly automated service for 3D model creation. We compared the performance of three low-cost sensor systems; one RGB-D camera, one low-end terrestrial laser scanner (TLS), and one panoramic camera, using a cloud-based processing service to automatically create mesh models and point clouds, evaluating the accuracy of the results against a reference point cloud from a higher-end TLS. While adequately accurate results could be obtained with all three sensor systems, the TLS performed the best both in terms of reconstructing the overall room geometry and smaller details, with the panoramic camera clearly trailing the other systems and the RGB-D o ﬀ ering a middle ground in terms of both cost and quality. The results demonstrate the attractiveness of fully automatic cloud-based indoor 3D modeling for low-cost sensor systems, with the latter providing better model accuracy and completeness, and with all systems o ﬀ ering a rapid rate of data acquisition through an easy-to-use interface.


Introduction
Indoor 3D modeling has a large number of uses, including the planning of construction [1], the preservation of cultural heritage [2,3], and providing a basis for a virtual reality applications [4]. While a model can of course be constructed manually, efficient measurement can be utilized for reality-based modeling [5]. Terrestrial laser scanners (TLSs) provide dense and accurate geometric information, though they are expensive [6], require the careful planning of scan locations, and can be time-consuming to use [7]. In recent years, an increasing number of low-cost sensor systems for 3D modeling using different operating systems have entered the market [8,9]. Easy-to-use laser scanners for projects with moderate requirements for accuracy are available, including the Leica BLK360 [3].
Panoramic multi-camera systems are widely available as a lightweight, low-cost option [10]. A number of low-cost consumer-grade panoramic cameras have entered the market, allowing the user to capture a 360-degree view at once, thus reducing the number of images required to cover a scene [11]. Panoramic cameras have also been used for the photogrammetric modeling of indoor spaces, with 3D models being possible to obtain through automatic processing, with camera calibration and the use of an optimized projection improving the model quality [12].

Leica RTC360
The Leica RTC360 terrestrial laser scanner is applied for obtaining a reference point cloud. Like the BLK360, its function is based on time-of-flight [52]. With a data collection rate of two million points per second, a range of 130 m, and an accuracy of 2.9 mm at 20 m [59], it is a suitable choice for obtaining a reference point cloud. The measurement data are processed with Leica Register360. In the scanning process, no targets are used, requiring the point clouds to be combined using automated cloud-to-cloud matching in Register360.

Test Sites and Data Acquisition
Two test sites, shown in Figure 1, with different characteristics were used for testing the described instruments. The Tetra Conference Hall at the Hanaholmen Swedish-Finnish Cultural Centre in Espoo, Finland is a 173.5 m 2 hall with an irregular hexagonal shape and a roof height of 3.20 m. The interior is quite large, and the curtains along the walls give it an unorthodox shape for the Matterport processing. Lecture hall 101 at the Aalto University Department of Mechanical Engineering in Espoo, Finland is rectangular with solid walls and, at 69 m 2 , is significantly smaller than the Tetra hall. It also has a lower roof height at 2.50 m.
The characteristics of the applied instruments are presented in Table 1, as well as their end products. The applied instruments are the Matterport Pro2 3D (M), Ricoh Theta V (T), Leica BLK360 (BM for Matterport-processed data, BL for Leica-processed data), and Leica RTC360 (R), which produce point clouds (PC) and meshes (M). described instruments. The Tetra Conference Hall at the Hanaholmen Swedish-Finnish Cultural Centre in Espoo, Finland is a 173.5 m 2 hall with an irregular hexagonal shape and a roof height of 3.20 m. The interior is quite large, and the curtains along the walls give it an unorthodox shape for the Matterport processing. Lecture hall 101 at the Aalto University Department of Mechanical Engineering in Espoo, Finland is rectangular with solid walls and, at 69 m 2 , is significantly smaller than the Tetra hall. It also has a lower roof height at 2.50 m.   Data acquisition was performed with each sensor in immediate succession for the conditions between the sensors to remain as close as possible. In order to minimize outliers, the RTC360 scans measured all points twice. The sensor systems were used as is to evaluate their performance as purchased, though the results could potentially have been improved by calibration [3,[60][61][62]. Table 2 shows an overview of the scanning process of both test sites, with the total scanning time comprising the full scanning process from the start of the first scan to the end of the last, including the time spent moving the instrument. For the BLK360, a low resolution corresponds to a 15 × 15 mm resolution at 7.5 m, while a medium resolution corresponds to 10 × 10 mm at 7.5 m. For the RTC360 data, the instrument's medium resolution corresponding to 6 × 6 mm at 10 m was used, and all points were scanned twice, denoted as 2x.

Comparison Methods
The processed point clouds and meshes were compared to the reference point cloud in CloudCompare 2.10.2 [63] using cloud-to-cloud (C2C) and cloud-to-mesh (C2M) distance analyses. For the distance analysis, point clouds and meshes were co-registered using the ICP algorithm following an approximate manual alignment. In order to ensure an optimal registration, points outside the examined area were removed, eliminating the possibility of stray points affecting the registration. The algorithm was set to finish once the root mean square error (RMSE) difference between two iterations was less than 10 −5 m, as subsequent iterations would provide negligible benefit. The meshes were sampled, and subsequently aligned in the same manner as the point clouds.
CloudCompare offers four methods for calculating C2C distances; direct Euclidean distance, distance calculation with a least squares best fitting plane, using a 2.5D Delaunay triangulation of the projection of the points on the plane, or with a quadratic function [64]. In this paper, the quadratic function is used for the C2C calculation, as it is capable of representing smooth and curvy surfaces in addition to straight planes. For C2M calculations, CloudCompare only supports the calculation of Euclidean distances from the point cloud to the closest point on the mesh.
The analyses were conducted both on the room geometries with the furniture and all objects between floor and roof level removed, and on a small, detailed segment of the room with furniture present. In both cases, the segment consisted of a table with chairs, featuring small objects on the table. This makes it possible to separately analyze the performance of the sensor systems in large spaces with sparse features, as well as their capability to reconstruct detailed objects. In order to evaluate the presence of outliers in the data, the 90th and 99th percentile deviations were calculated. Additionally, the share of points below 1 cm was noted as a general, singular value of performance.
According to the ISPRS Benchmark on Indoor Modelling evaluation framework presented by [65], the quality of indoor modeling can be assessed in terms of completeness (the extent of the reconstruction of the reference), correctness (the extent of constructed elements present in the reference), and accuracy (the geometric distance between the elements of the source and reference). While this framework specifically concerns reconstructed mesh models, these principles can also be applied to point clouds, though with differing calculations. The ISPRS benchmark defines accuracy as the median of the unsigned distances between the reconstructed vertices of a geometric model and a reference surface. The points of the compared point cloud can directly be used in place of mesh vertices, however, in this paper, the closest point of the reference point cloud is used to measure C2C and C2M distances, rather than measuring the distance to a reference surface.
For evaluating the accuracy of the tested sensor systems, we use the mean and standard deviation, as the distribution of the errors between the point clouds and meshes displays the consistency of the sensor systems' performance throughout the data set better than the singular value of the median.
According to the benchmark, completeness is defined as the intersection between the source and reference model, while the correctness is calculated as the area of intersection between the source and reference summed over all surfaces. As point clouds do not contain surfaces, we used the 90th and 99th percentile distances to measure completeness and correctness. As stray points outside the examined space have been eliminated manually a priori, the 99th percentile distance is used to find any notable inconsistencies, with a distance greater than a few centimeters (which could be explained by the limited function of the sensor system) indicating an incomplete representation of elements in the compared point cloud or mesh. The purpose of the 90th percentile value is twofold; an unexpectedly large value points to significant inconsistencies in the elements, evaluating completeness and correctness, and the value itself provides a measure of the accuracy. Finally, the share of points or vertices within 1 cm of any point in the reference point cloud is examined. This threshold was selected as it provides a singular value of reference for evaluating the performance, with any deviations smaller than 1 cm assumed to be located in an area with high correctness and completeness.
Additionally, the characteristics of the Matterport processing can be evaluated by examining the point and triangle counts of the generated point clouds and meshes, with the numbers demonstrating the impact of the system on the point cloud density.

Results
From the Matterport processing system, a mesh and point cloud were obtained for each instrument. The processing times required for Matterport (MP) and Leica processing are provided in Tables 3 and 4, along with the point and triangle counts for each point cloud (PC) and mesh (M). In Tables 5 and 6, the results of the C2C and C2M analyses of the Tetra test site are shown, first for the room geometry in Table 5 and followed by the detailed segment in Table 6. The results are further divided into the analysis of point clouds and meshes in both cases.  Figure 2a,b demonstrates the difference between the Matterport and Theta V point clouds of the room geometry, with the Theta V exhibiting notable error propagation towards the left of the image. Figure 2c) highlights the inability of the BLK360 to obtain full coverage of the desk in the segment, while the Matterport has captured the scene to a larger extent, with the exception of some parts of the objects in the scene close to floor level. For C2M distances, the results are always shown as projected onto the reference point cloud; the desk has been fully covered in the reference, but is colored red in the C2M analysis of the BLK360 data due to holes in the mesh.
Remote Sens. 2020, 12, x FOR PEER REVIEW 8 of 20  Figure 2a,b demonstrates the difference between the Matterport and Theta V point clouds of the room geometry, with the Theta V exhibiting notable error propagation towards the left of the image. Figure 2c) highlights the inability of the BLK360 to obtain full coverage of the desk in the segment, while the Matterport has captured the scene to a larger extent, with the exception of some parts of the objects in the scene close to floor level. For C2M distances, the results are always shown as projected onto the reference point cloud; the desk has been fully covered in the reference, but is colored red in the C2M analysis of the BLK360 data due to holes in the mesh. The results of the distance analysis of the hall 101 test site room geometry are shown in Table 7, and Table 8 displays the results of the segment. The results of the distance analysis of the hall 101 test site room geometry are shown in Table 7, and Table 8 displays the results of the segment. Similarly to what can be seen in the Tetra data, Figure 3a,b shows that the Theta V suffers from error propagation, which is found to a lesser extent in the BLK360 data. Figure 3c,d displays the difference between the ability of the Matterport and Theta V to accurately reconstruct small details, with the Theta V only being able to capture the general shape of the table.
Remote Sens. 2020, 12, x FOR PEER REVIEW 9 of 20 Similarly to what can be seen in the Tetra data, Figure 3a,b shows that the Theta V suffers from error propagation, which is found to a lesser extent in the BLK360 data. Figure 3c,d displays the difference between the ability of the Matterport and Theta V to accurately reconstruct small details, with the Theta V only being able to capture the general shape of the table.

Analysis of Data Acquisition and Processing Times
In terms of data acquisition times, longer-range instruments, such as the BLK360, become advantageous in larger open spaces, requiring fewer scans to cover the entire space. For the spaces used in this study, however, this is offset by the shorter scan duration of the Theta V and Matterport, which also enables them to cover occluded areas quicker than the BLK360. While the intention of the comparison was to study the capabilities of the sensor systems as is, in the state they are delivered to a user, the calibration of the sensor systems could potentially improve the results. The Matterport can be calibrated using a set of TLS-scanned targets and a comparison of their coordinates, which has been shown to improve performance and eliminate the possibility of systematic errors in the sensor system [3]. As a dual fisheye lens camera, the calibration of the Theta V is challenging [60], but the successful calibration of its predecessor Theta S has been conducted [61]. The self-calibration of the TLS can be conducted using either point or planar targets to ensure adequate data quality, though correlation between the calibration parameters, particularly the scanner rangefinder offset and the position of the scanner, must be accounted for. The use of an asymmetric target field and the measurement of tilt angle observations has been found to reduce correlation, thus improving the calibration [62].
The processing times differed notably between the sensor systems, as shown in Tables 3 and 4. The choice of sensor system may even be impacted by the processing time; for example, the processing of the Theta V data of the Tetra hall initially failed after 36 h, and required more than ten hours to complete on a second attempt. Processing times in excess of 12 h may already complicate the daily operation of systems. The Theta V data also took the longest to process of the hall 101 data sets, but the difference was significantly smaller, and at slightly under two hours, it is a comparatively quick process. For smaller spaces, the Theta V can be a viable option in terms of the required time resources, though the 173.5 m 2 Tetra hall proved difficult to process, to the point of limiting the feasibility of the Theta V in large spaces.
The geometric accuracy of the tested sensor systems with automatic processing should be adequate for measurements being conducted on the resulting point clouds and models. In addition, sufficiently dense point clouds or mesh models are required for accurate reconstruction and detailed visualization, though a high density does not necessarily reflect the quality of the point cloud or model.

Point Cloud Density and Mesh Triangle Counts
As seen in Tables 3 and 4, the point count difference between the Matterport-processed point clouds is smaller than the difference between the Matterport-and Leica-processed BLK360 point clouds, as the Matterport processing limits the point clouds to a 1-cm grid by default unless the algorithm detects details smaller than 1 cm. This detection differs depending on the sensor system used, and can also lead to false positives, where the processing has detected details in the Matterport point cloud that do not exist in reality, as shown in Figure 4. Conversely, however, small objects may not be present in the Theta V point cloud. The Leica processing does not limit the point cloud to a perceivable grid, creating a significant difference in the point counts of the BLK360 point clouds depending on the processing system. The Leica processing produces a dense point cloud, containing 2175% and 742% of the points of the Matterport-processed BLK360 data for Tetra and hall 101, respectively. The markedly lower density of the Matterport-processed point clouds may impact their feasibility in applications requiring the visualization of point clouds, as the textures of the space or small details may not be visible in a sparse point cloud. The Leica processing does not limit the point cloud to a perceivable grid, creating a significant difference in the point counts of the BLK360 point clouds depending on the processing system. The Leica processing produces a dense point cloud, containing 2175% and 742% of the points of the Matterport-processed BLK360 data for Tetra and hall 101, respectively. The markedly lower density of the Matterport-processed point clouds may impact their feasibility in applications requiring the visualization of point clouds, as the textures of the space or small details may not be visible in a sparse point cloud.
When comparing the mesh triangle counts to point cloud point counts, we can see that the mesh triangle counts vary less by the room size. This is likely a result of decimation in the Matterport processing. Most notably, the BLK360 triangle count is lower for the larger Tetra hall than for the smaller hall 101, this being notable with the curtains lining the walls containing complex shapes to reconstruct. The Theta V is an outlier in both cases, however, with a triangle count several times that of the other systems despite the Theta V point clouds having a lower point count.

Distance Analysis of Indoor Space Geometries
As shown in Tables 5 and 7, the C2C distances of the Matterport and BLK360 point cloud of the room geometries are similar, with an improvement in accuracy for the BLK360 when Leica processing is used. Figure 2 shows the difference between the sensor systems' capabilities in a large space, with the Matterport point cloud showing moderate deviations in the roof and the curtain along the walls, while the Theta V point cloud is increasingly skewed towards the left parts of the image, also having areas with clearly higher deviations along the curtain. Figure 5 shows the deviation of the Theta V point clouds along the walls, where the shape for hall 101 is skewed, though it follows the general shape of the wall, while the shape of the curtain in the Tetra hall has not been taken into account.  When comparing the mesh triangle counts to point cloud point counts, we can see that the mesh triangle counts vary less by the room size. This is likely a result of decimation in the Matterport processing. Most notably, the BLK360 triangle count is lower for the larger Tetra hall than for the smaller hall 101, this being notable with the curtains lining the walls containing complex shapes to reconstruct. The Theta V is an outlier in both cases, however, with a triangle count several times that of the other systems despite the Theta V point clouds having a lower point count.

Distance Analysis of Indoor Space Geometries
As shown in Tables 5 and 7, the C2C distances of the Matterport and BLK360 point cloud of the room geometries are similar, with an improvement in accuracy for the BLK360 when Leica processing is used. Figure 2 shows the difference between the sensor systems' capabilities in a large space, with the Matterport point cloud showing moderate deviations in the roof and the curtain along the walls, while the Theta V point cloud is increasingly skewed towards the left parts of the image, also having areas with clearly higher deviations along the curtain. Figure 5 shows the deviation of the Theta V point clouds along the walls, where the shape for hall 101 is skewed, though it follows the general shape of the wall, while the shape of the curtain in the Tetra hall has not been taken into account. The Leica processing does not limit the point cloud to a perceivable grid, creating a significant difference in the point counts of the BLK360 point clouds depending on the processing system. The Leica processing produces a dense point cloud, containing 2175% and 742% of the points of the Matterport-processed BLK360 data for Tetra and hall 101, respectively. The markedly lower density of the Matterport-processed point clouds may impact their feasibility in applications requiring the visualization of point clouds, as the textures of the space or small details may not be visible in a sparse point cloud.
When comparing the mesh triangle counts to point cloud point counts, we can see that the mesh triangle counts vary less by the room size. This is likely a result of decimation in the Matterport processing. Most notably, the BLK360 triangle count is lower for the larger Tetra hall than for the smaller hall 101, this being notable with the curtains lining the walls containing complex shapes to reconstruct. The Theta V is an outlier in both cases, however, with a triangle count several times that of the other systems despite the Theta V point clouds having a lower point count.

Distance Analysis of Indoor Space Geometries
As shown in Tables 5 and 7, the C2C distances of the Matterport and BLK360 point cloud of the room geometries are similar, with an improvement in accuracy for the BLK360 when Leica processing is used. Figure 2 shows the difference between the sensor systems' capabilities in a large space, with the Matterport point cloud showing moderate deviations in the roof and the curtain along the walls, while the Theta V point cloud is increasingly skewed towards the left parts of the image, also having areas with clearly higher deviations along the curtain. Figure 5 shows the deviation of the Theta V point clouds along the walls, where the shape for hall 101 is skewed, though it follows the general shape of the wall, while the shape of the curtain in the Tetra hall has not been taken into account.  While [9] present a TLS-Matterport comparison where the roof height forces the Matterport to exceed its recommended range, our test sites have a roof height well within the range of the Matterport, and the deviations of the area within its range appear similar. In our results, the deviations of the area within the range of the Matterport appear similar for both test sites. The Matterport-processed BLK360 point cloud and Matterport point cloud exhibit similar characteristics in all metrics, while the Theta V point cloud has larger deviations with a broader spread. With 53 percent of points within 1 cm of the hall 101 room geometry reference point cloud, as noted in Table 7, it falls well below the Matterport, which has the worst performance of the remaining sensors. Figure 6 shows the distribution of points in the hall 101 room geometry, displaying the difference between the points located below the 90th and 99th percentile of deviations, as well as the mean deviations. The mean indicates the spread of deviations when used in conjunction with the 90th and 99th percentile deviations, as a mean close to the 90th percentile deviation may point to outliers skewing the mean upwards. For the meshes, the deviation is calculated as the distance from a point in the reference point cloud to the closest point on the mesh. Both distributions are a measure of the ability of the sensor system to reconstruct geometry; a high 99th percent deviation indicates inconsistencies in the presence of elements in comparison to the reference, affecting completeness and correctness, while the 90th percent deviation measures accuracy, and in the case of a high value, also indicates severe deficiencies in completeness and correctness. The Theta V shows such values, while the Matterport and BLK360 with Matterport processing provide better, similar looking results for point clouds and meshes alike. In contrast, the Leica-processed BLK360 point cloud is superior to all other data sets, including the Matterport-processed BLK360 data.
Remote Sens. 2020, 12, x FOR PEER REVIEW 12 of 20 The deviations of the Matterport point cloud are consistent with the findings of [4], where the errors in an indoor setting generally fell within the one-percent deviation stated by the manufacturer. While [9] present a TLS-Matterport comparison where the roof height forces the Matterport to exceed its recommended range, our test sites have a roof height well within the range of the Matterport, and the deviations of the area within its range appear similar. In our results, the deviations of the area within the range of the Matterport appear similar for both test sites. The Matterport-processed BLK360 point cloud and Matterport point cloud exhibit similar characteristics in all metrics, while the Theta V point cloud has larger deviations with a broader spread. With 53 percent of points within 1 cm of the hall 101 room geometry reference point cloud, as noted in Table 7, it falls well below the Matterport, which has the worst performance of the remaining sensors. Figure 6 shows the distribution of points in the hall 101 room geometry, displaying the difference between the points located below the 90th and 99th percentile of deviations, as well as the mean deviations. The mean indicates the spread of deviations when used in conjunction with the 90th and 99th percentile deviations, as a mean close to the 90th percentile deviation may point to outliers skewing the mean upwards. For the meshes, the deviation is calculated as the distance from a point in the reference point cloud to the closest point on the mesh. Both distributions are a measure of the ability of the sensor system to reconstruct geometry; a high 99th percent deviation indicates inconsistencies in the presence of elements in comparison to the reference, affecting completeness and correctness, while the 90th percent deviation measures accuracy, and in the case of a high value, also indicates severe deficiencies in completeness and correctness. The Theta V shows such values, while the Matterport and BLK360 with Matterport processing provide better, similar looking results for point clouds and meshes alike. In contrast, the Leica-processed BLK360 point cloud is superior to all other data sets, including the Matterport-processed BLK360 data. In the meshes, the Theta V was an even further outlier than in the point clouds, with only 17 percent of the points in the Tetra room geometry reference cloud at a distance below 1 cm from the mesh. While the Theta V mesh contains more triangles than the other two meshes combined, this does not reflect positively in the C2M distance analysis, as seen in Figures 2 and 3. Unlike for the point clouds, the Matterport-processed BLK360 clearly outperforms the Matterport.  In the meshes, the Theta V was an even further outlier than in the point clouds, with only 17 percent of the points in the Tetra room geometry reference cloud at a distance below 1 cm from the mesh. While the Theta V mesh contains more triangles than the other two meshes combined, this does not reflect positively in the C2M distance analysis, as seen in Figures 2 and 3. Unlike for the point clouds, the Matterport-processed BLK360 clearly outperforms the Matterport.  The mean deviations do not significantly differ, though the Matterport exhibited a higher variance. The Theta V remains the outlier of the three tested sensor systems, showing larger C2M distances than the BLK360 and Matterport in every metric. In [10], panoramic photogrammetry is used to automatically generate a model from Samsung Gear360 images. With the default projection, the errors are larger than those we obtained with the Theta V, though by using a custom projection, the authors of [10] were able to achieve a higher degree of accuracy in photogrammetric modeling.
In a detailed segment, the results of Tables 6 and 8 show a clearer difference between the Matterport and the Matterport-processed BLK360 data than is the case for the room geometry. As in the room geometry, the Theta V remains the clearly worst-performing sensor system. The choice of processing system for the BLK360 data clearly impacts the quality of the point cloud, with the Leica processing providing more accurate results, though the Matterport-processed BLK360 point cloud remains superior to the Matterport point cloud. Figure 8 displays the difference between the Matterport point cloud and the two BLK360 point clouds of the hall 101 segment. The mean deviations do not significantly differ, though the Matterport exhibited a higher variance. The Theta V remains the outlier of the three tested sensor systems, showing larger C2M distances than the BLK360 and Matterport in every metric. In [10], panoramic photogrammetry is used to automatically generate a model from Samsung Gear360 images. With the default projection, the errors are larger than those we obtained with the Theta V, though by using a custom projection, the authors of [10] were able to achieve a higher degree of accuracy in photogrammetric modeling.
In a detailed segment, the results of Tables 6 and 8 show a clearer difference between the Matterport and the Matterport-processed BLK360 data than is the case for the room geometry. As in the room geometry, the Theta V remains the clearly worst-performing sensor system. The choice of processing system for the BLK360 data clearly impacts the quality of the point cloud, with the Leica processing providing more accurate results, though the Matterport-processed BLK360 point cloud remains superior to the Matterport point cloud. Figure 8 displays the difference between the Matterport point cloud and the two BLK360 point clouds of the hall 101 segment.
The authors of [9] show a TLS-Matterport comparison, where the Matterport cannot accurately represent detailed areas. The choice of processing method makes a larger difference for the BLK360 in the detailed segment than for the room geometry, as the Leica-processed BLK360 point cloud exhibits the best results in every metric. A benefit of the Matterport processing for the BLK360, however, is the possibility to use an online walkthrough, and the measurement data captured by operating the scanner in Matterport's Capture app can also be processed with Leica's proprietary processing system. The authors of [9] show a TLS-Matterport comparison, where the Matterport cannot accurately represent detailed areas. The choice of processing method makes a larger difference for the BLK360 in the detailed segment than for the room geometry, as the Leica-processed BLK360 point cloud exhibits the best results in every metric. A benefit of the Matterport processing for the BLK360, however, is the possibility to use an online walkthrough, and the measurement data captured by operating the scanner in Matterport's Capture app can also be processed with Leica's proprietary processing system.
In the meshes of the segments, the Matterport outperforms the BLK360 in both spaces, having a larger share of points within 1 cm of the reference point cloud. The Theta V still exhibits the largest deviations, though it performs slightly better in the segment mesh than in the point cloud, as shown in Figure 9. The chairs by the desk have not been reconstructed in either Theta V data set, while the other sensor systems are capable of doing so.  In the meshes of the segments, the Matterport outperforms the BLK360 in both spaces, having a larger share of points within 1 cm of the reference point cloud. The Theta V still exhibits the largest deviations, though it performs slightly better in the segment mesh than in the point cloud, as shown in Figure 9. The chairs by the desk have not been reconstructed in either Theta V data set, while the other sensor systems are capable of doing so. The authors of [9] show a TLS-Matterport comparison, where the Matterport cannot accurately represent detailed areas. The choice of processing method makes a larger difference for the BLK360 in the detailed segment than for the room geometry, as the Leica-processed BLK360 point cloud exhibits the best results in every metric. A benefit of the Matterport processing for the BLK360, however, is the possibility to use an online walkthrough, and the measurement data captured by operating the scanner in Matterport's Capture app can also be processed with Leica's proprietary processing system.
In the meshes of the segments, the Matterport outperforms the BLK360 in both spaces, having a larger share of points within 1 cm of the reference point cloud. The Theta V still exhibits the largest deviations, though it performs slightly better in the segment mesh than in the point cloud, as shown in Figure 9. The chairs by the desk have not been reconstructed in either Theta V data set, while the other sensor systems are capable of doing so.  The results of the hall 101 segment follow the ones of the Tetra segment, with similar differences between the sensor systems. Among the segment point clouds, the BLK360 outperforms the Matterport regardless of processing system, as shown in Figure 8. The opposite is true for the meshes, where the Matterport is ahead of the BLK360 in every metric. This may point to Matterport's processing system having the best compatibility with their proprietary sensor system, despite also supporting the Leica BLK360, which optimally provides more accurate point clouds, particularly if processed through Leica's processing system. If a higher-quality mesh is desired, Leica-processed point clouds must be processed with third-party software. The Theta V performs at a similar level to the Tetra segment in the point cloud, but its accuracy drops further in the mesh processing. The meshes of the hall 101 segment are presented in Figure 10.
Remote Sens. 2020, 12, x FOR PEER REVIEW 15 of 20 The results of the hall 101 segment follow the ones of the Tetra segment, with similar differences between the sensor systems. Among the segment point clouds, the BLK360 outperforms the Matterport regardless of processing system, as shown in Figure 8. The opposite is true for the meshes, where the Matterport is ahead of the BLK360 in every metric. This may point to Matterport's processing system having the best compatibility with their proprietary sensor system, despite also supporting the Leica BLK360, which optimally provides more accurate point clouds, particularly if processed through Leica's processing system. If a higher-quality mesh is desired, Leica-processed point clouds must be processed with third-party software. The Theta V performs at a similar level to the Tetra segment in the point cloud, but its accuracy drops further in the mesh processing. The meshes of the hall 101 segment are presented in Figure 10. In both segments, it can be noted that the Theta V performs poorly in reconstructing details, and even larger objects, causing significant completeness deficiencies in the data. If comparing the RGBcolored hall 101 segment point clouds of RTC360 and Theta V in Figure 11a,b, significant differences can immediately be seen. The computer on the table is missing entirely, and the furniture is poorly defined. Figure 11c shows the points on the RTC360 point cloud at a distance larger than the 90th percentile deviation to the Theta V point cloud, as the RTC360 point cloud contains more details, allowing the areas not captured by the Theta V to be easily distinguished. The points above the 99th percentile deviation in Figure 11d show the areas with the most significant completeness issues. Figure 11e shows the points with a deviation of larger than 1 cm, which corresponds to 43 percent of all points. These points are found throughout the point cloud, though less so on the surface of the table. In both segments, it can be noted that the Theta V performs poorly in reconstructing details, and even larger objects, causing significant completeness deficiencies in the data. If comparing the RGB-colored hall 101 segment point clouds of RTC360 and Theta V in Figure 11a,b, significant differences can immediately be seen. The computer on the table is missing entirely, and the furniture is poorly defined. Figure 11c shows the points on the RTC360 point cloud at a distance larger than the 90th percentile deviation to the Theta V point cloud, as the RTC360 point cloud contains more details, allowing the areas not captured by the Theta V to be easily distinguished. The points above the 99th percentile deviation in Figure 11d show the areas with the most significant completeness issues. Figure 11e shows the points with a deviation of larger than 1 cm, which corresponds to 43 percent of all points. These points are found throughout the point cloud, though less so on the surface of the table.
In all tested scenarios, the Matterport performs reasonably well, with the share of points below 1 cm falling within a maximum of eight percentage points in comparison to the BLK360 with Matterport processing, and the Matterport performing better by 73 to 64 percent in the Tetra segment. While the BLK360 mostly provided the best results for the Matterport-processed data, the Leica-processed BLK360 point cloud showed the lowest C2C distances up to the 90th percentile in every case. The room geometry mean deviations for of the Leica-processed BLK360 point cloud were 8 mm in the Tetra conference hall and 2 mm in the hall 101 test site, supporting the accuracy described in [3,40], though the deviations of the Matterport-processed point clouds and meshes were larger, and the difference was more pronounced in the detailed segments. The BLK360 uses a unique tripod, and is incompatible with most tripods used for other instruments, with the scanner base set to a height of 110 cm, which barely enables the scanner to reach the surface of a table of average height. In the Matterport-processed mesh of the Tetra segment displayed in Figure 2d, the top of the desk is mostly absent, causing large discrepancies between the mesh and the reference point cloud. Should the BLK360 be used from a taller height than the proprietary tripod, the results would be expected to improve, given its high accuracy in the areas in full view of the instrument. The other sensor systems can be set up on tripods of multiple types, and can be used from any height. In all tested scenarios, the Matterport performs reasonably well, with the share of points below 1 cm falling within a maximum of eight percentage points in comparison to the BLK360 with Matterport processing, and the Matterport performing better by 73 to 64 percent in the Tetra segment. While the BLK360 mostly provided the best results for the Matterport-processed data, the Leicaprocessed BLK360 point cloud showed the lowest C2C distances up to the 90th percentile in every case. The room geometry mean deviations for of the Leica-processed BLK360 point cloud were 8 mm in the Tetra conference hall and 2 mm in the hall 101 test site, supporting the accuracy described in [3,40], though the deviations of the Matterport-processed point clouds and meshes were larger, and the difference was more pronounced in the detailed segments. The BLK360 uses a unique tripod, and is incompatible with most tripods used for other instruments, with the scanner base set to a height of 110 cm, which barely enables the scanner to reach the surface of a table of average height. In the Matterport-processed mesh of the Tetra segment displayed in Figure 2d, the top of the desk is mostly absent, causing large discrepancies between the mesh and the reference point cloud. Should the BLK360 be used from a taller height than the proprietary tripod, the results would be expected to improve, given its high accuracy in the areas in full view of the instrument. The other sensor systems can be set up on tripods of multiple types, and can be used from any height.
The Theta V is the clear outlier in every scenario, consistently showing the weakest performance of the tested sensor systems. While featureless areas are difficult for image-based sensors to model due to a lack of points to connect the images with, the processing has extracted sufficient geometry for the Theta V to create a full model of the room geometries. The largest deviations can be found in the details of the segment, as the Theta V is incapable of modeling small objects, with the objects on the tables not being present in either model. While small objects are missing entirely, there are also issues in modeling furniture, with objects being reduced to horizontal or vertical surfaces, e.g., the Figure 11. The RGB-colored RTC360 (a) and Theta V (b) point clouds of the hall 101 segment, with the clearest visual differences noted; the points on the RTC360 point cloud with a distance to the Theta V point cloud larger than the 90th percentile deviations (c), 99th percentile deviations (d), and 1 cm (e), with points exceeding the threshold shown in red. Significant completeness issues can be seen, particularly in the 99th percentile deviations.
The Theta V is the clear outlier in every scenario, consistently showing the weakest performance of the tested sensor systems. While featureless areas are difficult for image-based sensors to model due to a lack of points to connect the images with, the processing has extracted sufficient geometry for the Theta V to create a full model of the room geometries. The largest deviations can be found in the details of the segment, as the Theta V is incapable of modeling small objects, with the objects on the tables not being present in either model. While small objects are missing entirely, there are also issues in modeling furniture, with objects being reduced to horizontal or vertical surfaces, e.g., the chairs by the table in the hall 101 segment, as seen in Figure 3d. Thus, the Theta V is unsuited for projects in which the shapes of objects within the space are to be reconstructed with a reasonable degree of accuracy. It remains a feasible alternative for obtaining the general geometry of a space, however.

Conclusions
The increasing availability of cloud-based software systems for automatic 3D modeling of indoor spaces and their compatibility with a number of differing sensor systems makes these systems an increasingly attractive alternative for indoor modeling. In this study, the results of the automatic, cloud-based processing of 3D point clouds and mesh models using data from different low-cost sensor systems were evaluated.
We have conducted a comparison between point clouds and meshes produced by the Matterport processing system based on data from the Ricoh Theta V panoramic camera, the Matterport Pro2 3D RGB-D camera, and the Leica BLK360 laser scanner, using a high-quality Leica RTC360 point cloud as a reference. As a black-box system, the exact function of the Matterport processing system is not publicly disclosed. Therefore, the results reflect both the performance of the sensors and the ability of the processing system to utilize scan data with no user input. As the BLK360 is compatible with both Matterport processing and Leica's proprietary processing system, with the latter producing a point cloud, the impact of the Matterport processing on the TLS data was also examined.
A full model of the room geometry could be obtained from the automatic processing in all tested cases. Issues with error propagation, difficulty in finding sufficient features in the walls, and in the case of the Tetra hall, the irregular structure of the walls, affected the results, though all results fell well within the tolerances stated by the manufacturer. In a detailed segment, the ability of a laser scanner to capture small details was highlighted, with the Leica-processed BLK360 cloud showing the best accuracy. Using Matterport processing, the low scanning height and consequently low point density of objects located on a similar height to the scanner affected the results, as the Matterport outperformed the BLK360 with Matterport processing in both segments. As with the room geometry, the accuracy of the modeled areas fell within the tolerance stated by the manufacturer, with larger deviations stemming from the inability of the sensor system to capture data from particular areas, thus producing a model with limited completeness.
While the results demonstrate issues both with the performance of the sensor systems and the ability of the processing system to utilize the scan data, the tested processing system offers a low-cost solution for modeling indoor environments, where centimeter-level precision is not required and a visually pleasant model is desired. With the tested sensor systems ranging from a consumer-grade panoramic camera to a professional-grade laser scanner, the sensor system should be selected depending on the needs of the user. The automatic cloud-based processing of indoor scan data and panoramic images provides a viable alternative for the rapid modeling of indoor spaces, with a high rate of data acquisition and low time and resource requirements.