Real-Time 3D Reconstruction Pipeline for Room-Scale, Immersive, Medical Teleconsultation

Abstract

Medical teleconsultation was among the initial use cases for early telepresence research projects since medical treatment often requires timely intervention by highly specialized experts. When remote medical experts support interventions, a holistic view of the surgical site can increase situation awareness and improve team communication. A possible solution is the concept of immersive telepresence, where remote users virtually join the operating theater that is transmitted based on a real-time reconstruction of the local site. Enabled by the availability of RGB-D sensors and sufficient computing capability, it becomes possible to capture such a site in real time using multiple stationary sensors. The 3D reconstruction and simplification of textured surface meshes from the point clouds of a dynamic scene in real time is challenging and becomes infeasible for increasing capture volumes. This work presents a tightly integrated, stateless 3D reconstruction pipeline for dynamic, room-scale environments that generates simplified surface meshes from multiple RGB-D sensors in real time. Our algorithm operates directly on the fused, voxelized point cloud instead of populating signed-distance volumes per frame and using a marching cube variant for surface reconstruction. We extend the formulation of the dual contouring algorithm to work for point cloud data stored in an octree and interleave a vertex-clustering-based simplification before extracting the surface geometry. Our 3D reconstruction pipeline can perform a live reconstruction of six incoming depth videos at their native frame rate of 30 frames per second, enabling the reconstruction of smooth movement. Arbitrarily complex scene changes are possible since we do not store persistent information between frames. In terms of mesh quality and hole filling, our method falls between the direct mesh reconstruction and expensive global fitting of implicit functions.

1. Introduction

Immersive telepresence has been a research domain for many decades, driven by the vision of enabling experts to collaborate over distance. Steuer [1] defined telepresence as “the experience of being present in an environment using a communication medium”. The concept of telepresence encompasses a multitude of different topics, ranging from social, perceptual, and behavioral to technical aspects of interacting within a remote environment. An important part of telepresence is digitizing and transmitting a physical space over a communication medium to allow remote users to join that space virtually.

The computational complexity of reconstructing a dynamic environment in real time and the bandwidth of the communication medium are typically limiting factors when designing immersive telepresence systems. Some previous works proposed the transmission of static scene reconstructions to save bandwidth and reduce complexity [2,3] or limited the transmission to individual participants [4]. Maimone et al. [5] presented a telepresence system capable of transmitting dynamic environments using multiple RGB-D cameras in a lab environment.

Researchers explored various methods to capture a physical space using multiple static cameras and then reconstruct that scene in 3D to transmit it to remote users. Initial approaches used a sea of cameras [6] to reconstruct the environment using multiview stereo (MVS) reconstruction methods. Later, RGB-D cameras were used to improve the capturing of environments in real time with higher fidelity [7,8]. Initially, the environment is captured as separate clouds of points per camera, either through direct acquisition of depth images or by processing color image pairs using multiview stereo algorithms. These individual point clouds are then fused into a common reference frame using intrinsic and extrinsic parameters [9]. Based on the fused point cloud, either direct visualization methods [10,11] can be used to display the reconstructed scene, or surface reconstruction algorithms are used to transform the unordered point cloud into a surface mesh [12]. Depending on the mode of visualization and the processing steps, remote clients receive point clouds, surface meshes, depth images, and corresponding texture information from the color video. The distant physical space is rendered and displayed at the remote clients using some form of virtual or mixed reality display.

Interestingly, several initially published telepresence systems considered medical consultation a strong use case [6,13,14]. When considering the need for highly specialized experts for the various domains of medical interventions that often come with timely treatment, it becomes evident that enabling medical experts to collaborate efficiently over distance would be very beneficial. Medical intervention is a dynamic environment with considerable interaction between stakeholders, patients, and medical equipment. Therefore, reconstruction methods that assume a static environment do not provide a sufficient context for remote experts during most medical interventions. To support the local team with their decision-making process, remote experts would require not only a high-quality view of the patient but also relevant health indicators, access to imaging data and availability of instruments, and medical devices such an intraoperative X-ray [13]. Therefore, for medical consultation, dynamic, room-scale 3D reconstruction in real-time with sufficient detail in regions of interest is an essential building block for modern telepresence solutions.

In this work, we present a real-time 3D reconstruction pipeline for room-scale, immersive medical teleconsultation (see Figure 1). Similar to FusionMLS [15], our pipeline performs stateless mesh reconstruction from multiple depth images. Instead of populating a signed distance volume and applying the marching cubes algorithm [16], our reconstruction method initially generates an octree from the fused point clouds. It directly extracts the mesh using an extended version of the recursive dual contouring algorithm [17] that works on voxelized point clouds instead of signed distance fields. With a voxelized point cloud, fewer surface samples have to be generated than volumetric signed distance samples by at least a factor of two using an optimal sparse data structure, where only corners of sign-changing cells are considered because the vertex generation for the cell interior is skipped. We can also further exploit the octree structure to efficiently simplify the resulting geometries with vertex clustering and connect mixed voxelization resolutions with dual contouring.

Our contributions are as follows: First, we extend the recursive dual contouring algorithm by Ju et al. [17] to include all diagonal connections for cells, which enables the algorithm to be used on point clouds stored in an octree. We define the hole-filling and connectivity criteria for the point-cloud-based algorithm to build correct surface meshes. Furthermore, we adapt vertex clustering simplification to include filtering conditions that preserve the topology and silhouette of nonmanifold, initial surface meshes. We evaluate the proposed method by comparing the reconstruction quality with existing methods and provide real-world performance measurements that show that the proposed pipeline can reconstruct room-scale environments in real time.

The remainder of this document is structured as follows. In Section 2, we review related work and discuss differences between the proposed and existing methods. In Section 3, we present our proposed pipeline step by step. In Section 4, we introduce our extensions to apply dual contouring on voxelized point clouds and showcase common issues (Section 4.2) that generate interior meshes. Based on this observation, we present the filtering conditions to prevent interior mesh generation during simplification and a method to preserve nonmanifold edges in Section 5. In Section 6, we compare our reconstruction to existing offline direct triangulation and implicit surface reconstruction methods and present performance results for reconstructing a room captured with a ring of six RGB-D cameras. We finally conclude in Section 7 and present future work in Section 8.

2. Related Work

Within the research domain of immersive telepresence, we review relevant literature for real-time 3D reconstruction from point clouds, as shown in Figure 2. Over several decades, researchers explored ways to capture a physical space and transmit it to a distant location [1] to enable immersive, mixed reality telepresence [6,7,14,18,19]. Most previous works depend on some form of visual depth sensing. Multiple view stereo algorithms can be used with calibrated camera pairs [20] to extract surface information by determining the disparity between two rectified images. Such methods require sufficiently textured surfaces for the stereo matching and are computationally demanding for real-time processing. Alternatively, specialized depth sensors with projected infrared patterns [21] or a time-of-flight camera can directly acquire 2.5D surface point clouds [2,5,8,22]. Structured-light capture systems project static infrared textures onto surfaces and, therefore, enable the accurate acquisition of surfaces also from textureless objects and environments. However, such systems can be quite expensive, can have a limited acquisition frame rate, and require extensive calibration. Sensors that measure the time of flight (ToF) are typically small and work well in indoor scenarios. Sufficiently high resolution and frame rates can be captured with modern ToF sensors such as the Microsoft Azure Kinect camera at a relatively low price.

Two distinct concepts have emerged for capturing the local environment: a dynamic camera for static scenes [2,3,23] and multiple stationary cameras for dynamic scenes [5,8,15,24,25]. A large body of work exists for capturing static scenes with moving cameras. Surfaces can be extracted with high quality from moving color images with known camera poses using the structure-from-motion method [26]; however, the process is typically offline. Alternatively, simultaneous localization and mapping [27] combines camera pose estimation and map creation to enable real-time operation. If the camera sensor also captures depth images, surface reconstruction can be performed by fusing the acquired point clouds using the estimated camera poses [28].

If multiple static, calibrated depth cameras capture a scene, the acquired images can be easily projected into a common reference frame to form an unstructured surface point cloud. A vast amount of work [29] for the reconstruction of implicit surfaces from point clouds is available. One popular formulation is radial basis functions [30,31], which are continuous interpolation functions representing a single smooth and manifold object. Different approaches, such as Poisson reconstruction [32,33], Fourier-based reconstruction [34,35], smooth signed distance surface reconstruction [12], or multilevel partition of unity [36], present different behaviors in terms of robustness, smoothness, and run-time complexity.

Our work focuses on setups consisting of multiple stationary cameras for dynamic environments to capture a room from multiple perspectives as input for surface reconstruction algorithms that work in real time. We further aim to simplify the resulting surface mesh to lower the bandwidth requirements for the communication network while keeping sufficient surface detail to enable medical teleconsultation.

2.1. Real-Time Dynamic Surface Reconstruction

A real-time reconstruction of dynamic scenes from multiple static RGB-D cameras is a challenging task [37]. Generally, existing work can be discerned into three general ideas for generating the surface.

The first family of approaches is based on efficiently performing 2.5D mesh generation by connecting each backprojected vertex with its neighboring depth pixels, which is effectively the 2D Delaunay triangulation [38] of the image plane. Overlapping surfaces need to be combined gracefully with multiple depth cameras like Maimone et al. [5] do during rendering with quality weights for each depth and RGB camera. Alexiadis et al. [39] and Meerits et al. [25] also proposed view-independent approaches that purge redundant triangles and fix the boundary with retriangulation or triangle stitching. Meerits et al. [25] do not disregard redundant data on surfaces but readjust all vertices using a projection onto the local least squares surface.

Second, a more complex class of techniques is based on reconstructing a deformable, canonical model, whose deformations are updated at each frame to facilitate dynamic scene changes. Finding a model that enables arbitrary scene changes is challenging, and complex models will be computationally expensive. In the initial work from Zollhoefer et al. [40], the canonical model was a mesh captured ahead of time. Still, subsequent works [4,41,42] moved to a more flexible TSDF, reconstructed the canonical model live, and supported more dynamic deformations. For human performance capture [43,44], limited topological changes, and small motion or a single RGB-D-camera [45,46,47], these methods provide a high-quality reconstruction since knowledge is transferred, and occluded parts can be reconstructed if seen prior.

Suppose arbitrarily large and complex scene changes are required. In that case, the third family of approaches discards the idea of the deformation model and builds a new surface representation from the current set of depth images from scratch. While many methods for 3D surface reconstruction from point samples [12,32,33,35,48,49] exist, few are efficient enough to be performed in real time. Maimone et al. [24] presented an adapted depth image integration of KinectFusion [23], which segments static and moving objects. FusionMLS by Meerits et al. [15] has shown a very efficient TSDF reconstruction, where all SDF samples are generated for each frame and voxel, using neighborhood-based MLS surface reconstruction [50] using a very efficient neighborhood query that collects neighbors in each camera’s image space. Subsequently, the mesh is generated using marching cubes [16]. Our work has similar goals as FusionMLS and provides a stateless efficient mesh reconstruction using a volumetric domain. Compared with FusionMLS [15], our approach can generate a simplified mesh and mix different voxel resolutions because we use dual contouring on a regular, sparse octree instead of marching cubes [16] on a less sparse two-layer hierarchy with shared boundary voxels.

2.2. Isosurface Extraction

Many of the approaches highlighted in Section 2.1 generate an implicit surface, and most use marching cubes (MC) on GPUs [16,51] to create the output mesh quickly. Still, many different methods [52] exist to handle the problem of isosurface extraction.

While the original MC method places vertices at the edges of the signed distance voxels, dual methods, such as SurfaceNets by Gibson et al. [53], place the vertices of the final mesh in the cell interior. Dual contouring (DC) allows a better reconstruction of sharp details because the vertex placement is not restricted to cell edges. Connecting mixed resolutions is possible using MC but easier with DC because the algorithm can be directly executed on nonuniform hierarchical data structures like Ju et al. [17] presented for octrees without any modifications, and like triangle stitching [54]. However, dual contouring does not guarantee that the resulting surface will be manifold, and the fixes in subsequent work [55,56] require a significantly more complex algorithm. Since the correct topology is sometimes more important, the dual marching cubes method of Schaefer and Warren [57] combines the topology guarantees of MC with sharper detail by placing an additional dual feature vertex into faces generated with MC. The MC method was recently implemented using a data-driven approach [58], highlighting how many perspectives exist for solutions.

We chose DC for our work because the strict run-time requirements and the recursive traversal by Ju et al. [17] are almost as easy to parallelize as regular MC while providing seamless connections between multiple detail levels. Furthermore, the lack of manifold guarantees is insignificant since the geometry of multiple depth cameras is highly nonmanifold, and many edge cases of nonmanifold geometry cannot be captured.

2.3. Mesh Simplification with Dual Contouring

Since the DC method nicely operates on sparse data structures with mixed resolution, Ju et al. [17] demonstrated simplification via octree-based bottom–up vertex clustering with quadratic error functions [59] before mesh extraction, which is much more efficient than other mesh simplification techniques that generally require the high-resolution mesh in memory [60]. However, this approach limits simplification opportunities to the octree structure, which is addressed by enhancing the number of representative vertices per cell using a more advanced encoding as performed by Zhang et al. [61], who increased the limit to two vertices per cell. Schaefer et al. [55] later completely removed this limitation by performing the top–down dual mesh traversal before bottom–up simplification, allowing a list of representative vertices for different surfaces to be contained in each cell with accompanying conditions for manifold preservation. However, their procedure is no longer tail recursive because the results obtained by top–down traversal are required during bottom–up traversal, causing significantly more data dependencies that reduce opportunities for parallelization.

Since performance is our most important requirement and we aimed to implement simplification in GPU kernels [62], we used the octree-based vertex clustering with a single representative vertex, similar to the original work of Ju et al. [17] due to the simpler encoding as a proof of concept.

3. Pipeline Overview

Our processing pipeline for mesh reconstruction consists of five steps (Figure 3). We only accept a synchronized set of depth images as input, which either requires all images to match in a certain time window or an exact match if synchronization is handled beforehand using hardware synchronization. Initially, we use an image-based temporal filter as found in Realsense SDK [63] to reduce noise on the static parts of the depth images.

3.1. Octree Generation and Voxelization

In our proposed pipeline, the first step uses the depth images and camera calibration data to project pixels into an octree of all potentially nonempty voxels of configurable voxel size. Instead of building the octree using bottom–up sorting [64], we build a pointer-based octree from top–down using a heuristic to determine whether a certain arbitrarily sized voxel is occupied. For each of the voxel corners of a potentially occupied child node, the position is projected into each depth camera’s image space and compared with the depth value of the residing pixel. If any backprojected depth pixel of one of the eight corners matches up to a query radius threshold defined by the voxel size, it is deemed occupied. For efficiency, the last leaf occupancy query uses only a circumsphere for the voxel center instead of querying all eight corners. This method allows false positives and will generate false negatives if all eight corners of a voxel project into an empty pixel, which is not a significant issue in our testing.

Since we use a pointer-based octree instead of a hash-map-based data structure, such as Zhou et al. [64], data locality is better enforced blockwise using stream compaction [65].

Our approach allows mixing arbitrary voxel sizes. For demonstration purposes, we allow configuring two different low- and high-detail regions separated using an axis-aligned, user-defined bounding box. The subsequent processing steps for normal estimation and simplification also allow separate configurations for the different areas.

3.2. Surface Sample and Normal Estimation

For each potentially nonempty leaf of the octree, all points in a configurable neighborhood radius of the voxel center are collected in image space, using the same image space projection optimization, such as FusionMLS [15]. A surface sample and normal are generated for each voxel using a neighborhood-based local surface estimation method for point clouds. In this work, we utilize principal component analysis (PCA) [66] as a high-speed method and second-degree moving least squares (MLS) [50] for better quality. We apply statistical outlier removal to filter noise points and false positives by discarding every voxel, which does not contain enough points in the neighborhood.

The result of the first two steps is equivalent to performing voxelization, statistical outlier removal, and normal estimation, on the whole, merged point clouds of all depth cameras, which is part of many open-source libraries for point cloud processing (e.g., Open3D [67]). Our optimizations that exploit the compute resources of GPUs and the 2.5D structure of point clouds generated from depth images, which were also used in previous works [15,25], make the process efficient enough to perform in real time.

3.3. Simplification via Octree-Based Vertex Clustering

As Ju et al. proposed initially [17], the mesh simplification for dual contouring is a vertex-clustering-based preprocessing step that replaces interior nodes with a leaf containing the representative sample that is determined by the quadratic error function (QEF) [59], which minimizes the sum of least squared distances of all given input planes (Section 5). This function is ill-conditioned in coplanar cases and, therefore, requires robust linear solvers or high numeric precision. In this work, we use the recently introduced probabilistic extension for QEF by Trettner and Kobbelt [68] for better robustness.

Compared with their work, the conditions for our simplification are required to be more strict to preserve manifoldness and silhouette (Section 5). Like previous steps, the bottom–up simplification is implemented in CUDA using dynamic parallelism to reduce CPU/GPU overhead by dynamically launching a kernel for all current simplification candidates.

3.4. Dual Contouring for Point Clouds

The final step extracts the mesh from our data structure using recursive octree traversal initially introduced by Ju et al. [17], which we extend to include diagonal connections (Section 4.1). This part is performed on the CPU using work-stealing-based simplification (Section 6.3.1). Since dual contouring generates quad meshes, we perform triangulation, duplicate removal, and vertex list compaction in a final series of GPU kernels called Finalization.

3.5. Textured Rendering

Our texturing method Figure 1 and Figure 4) projects each fragment into the image space of the color cameras to gather the optimal pixel and into the depth cameras to check for occlusion, like in shadow mapping. Since we generate a simplified mesh, choosing a camera per face causes more inconsistencies and discontinuities than dynamically choosing a camera per pixel (Figure 5). If multiple cameras see the same surface, a selection of strategies is available. One strategy, to maximize resolution, chooses the camera with the shortest distance to the observed surface. Since we record the physical shading observed at the color camera, another strategy can use the camera with the smallest viewing angle difference between the current eye position and physical camera to prioritize correct shading.

4. Dual Contouring for Voxelized Point Clouds

4.1. Extension for Diagonal Connections

Dual contouring is an isosurface extraction method for signed distance fields. In the work of Ju et al. [17], the signed-distance samples and gradients are located at the corners of each cell. Therefore, a dual quad face is generated for each sign-changing cell edge with four incident cells. From the perspective of the sign-changing cell, each dual edge connection is formed across the 6-point neighborhood. The dual vertex is already present in the cell with voxelized point clouds, but it is not guaranteed that all occupied cells will neighbor across a shared edge (Figure 4a). Thus, the neighborhood must be extended to include all sets of four neighboring occupied cells, which share a common corner, i.e., the 26-point neighborhood of a 3D grid (Figure 4b).

To accommodate the larger neighborhood that must be checked for point clouds, we extend the recursive procedures of Ju et al. [17], named cellProc, faceProc, and edgeProc, with two new procedures, which we also name after the shared primal element, diagonalEdgeProc and cornerProc.

For cellProc, up to 12 subprocedures of diagonalEdgeProc (Figure 6a) and 32 permutations of cornerProc (Figure 7) are added to original 8 cellProc, 12 faceProc, and 6 edgeProc subprocedures. In faceProc, 8 subprocedures of diagonalEdgeProc (Figure 6b) and all 32 combinations of cornerProc (Figure 8a) are added, since they additionally cross through both nodes. For edgeProc, 14 possible cornerProc combinations that involve all input nodes are added to the 2 faceProc subprocedures. If 1 of the 4 nodes in cornerProc is not a leaf, a single recursive case is generated (Figure 8c).

Our diagonalEdgeProc procedure contains at least both involved nodes marked in green (Figure 6) and the 2 additional nodes neighboring this cell edge, if available because only 14 of 32 cornerProc cases are handled in edgeProc. The remaining 18 are handled here since some do not require 4 parent nodes like in Figure 8b. This representation will cause duplicate recursive calls to cornerProc if all 4 nodes in a primal edge are present, which can be solved by not calling the recursions on one of the two directions of diagonalEdgeProc. We chose this encoding for consistency with the original recursion [17]. More optimal encodings without duplicate cases may exist; e.g., if edgeProc is modified to have optional nodes, it accommodates all cases of diagonalEdgeProc.

4.2. Connectivity Properties and Interior Mesh Generation

With the extensions for diagonal connections, all combinations of four voxels sharing a common corner in the immediate neighborhood are connected. Thus, our method will only generate holes, where the voxel mesh of all leaf cells would contain holes. One disadvantage of this approach is the lack of discernability between interior and surface mesh parts, which is particularly significant if mixed voxel sizes are combined. For a power of two resolution border $2^c$, at least $2^c+1$ empty, high-resolution voxels must be present between two surfaces to prevent the closing of the wrong gaps (Figure 9a).

If the voxel size is uniform, faulty connections still occur, when, for example, a corner is smoothed and hidden by the additional diagonal face (Figure 9b). Discerning these cases is challenging because if the exterior connections were not indicated as in the example of Figure 9b, this case may also be a small manifold object. We omit to filter these cases since the internal faces in a uniform voxel are limited to crossing a single diagonal voxel.

If two layers of voxels are supplied as input, a wasteful interior mesh is also generated. For two close-by opposing surfaces (Figure 10a), which are physically unlikely to capture with depth cameras, we only allow connections between vertices with normals that are pointing in the same direction (i.e., only vertices where the angles between all normals of a face are smaller than e.g., 120°), like that used in other direct meshing methods, such as greedy projection triangulation by Marton, Rusu, and Beetz [69]. The case in Figure 10b will happen regardless of noise, if the splitting plane of the octree is badly placed and generates many insignificant interior triangles. We propose solving this problem by merging vertices with normals $n_1,n_2$ across a dual edge of unit direction d, if

$$n_1·n_2>cos\left({\delta }_n\right)\mathrm{and}|\frac{n_1+n_2}{||n_1+n_2||}·d|>cos\left({\delta }_d\right)$$

(1)

Thus, ${\delta }_n$ specifies the maximum angle between merged vertices, and ${\delta }_d$ denotes the maximum angle between the average normal and the dual edge direction. The two vertices are merged by moving one to the average position and marking the other as deleted with reference to update all face indices later. With this step, we can significantly reduce the number of duplicate triangles if low-quality voxelization is used (Figure 11).

Figure 10. Interior Mesh Generation Causes. Two examples of layers of voxels being generated: (a) shows two close-by opposite surfaces, and (b) visualizes an unfortunate voxel-grid placement for the surface.

Figure 10. Interior Mesh Generation Causes. Two examples of layers of voxels being generated: (a) shows two close-by opposite surfaces, and (b) visualizes an unfortunate voxel-grid placement for the surface.

5. Topology and Silhouette-Preserving Simplification

With vertex-clustering-based simplification, which is popular in tandem with dual contouring [17,55,61], the resulting mesh is both simplified and requires less computational resources during extraction.

We use the probabilistic extension of quadratic error functions introduced by Trettner and Kobbelt [68] for our bottom–up simplification due to better robustness and expose a single standard deviation ${\sigma }_n$ to configure the normal variance ${\Sigma }_n={\sigma }_n^{2}I$. The plane position variance ${\Sigma }_p$ is set to 0 because it does not influence the position of the representative vertex and only increases the expectancy value of the simplification error.

The formulation for the QEF components $A\in {\mathbb{R}}^{3\times 3},b\in {\mathbb{R}}^3$, and $c\in \mathbb{R}$ for a single input plane with position $\overline{q}$ and normal $\overline{n}$ as introduced more generally in [68] is therefore

$$\begin{array}{cc}\hfill A& =\overline{n}{\overline{n}}^T+{\sigma }_n^{2}I\hfill \\ \hfill b& =\overline{n}{\overline{q}}^T\overline{n}+{\sigma }_n^2\overline{q}\hfill \\ \hfill c& ={\left({\overline{q}}^T\overline{n}\right)}^2+{\sigma }_n^2{\overline{q}}^T\overline{q}\hfill \end{array}$$

(2)

In our implementation, calculating the QEF solution $x^{*}=A^{-1}b$ was sufficiently stable with single-precision floating-point numbers and pivoted LU decomposition with nonzero ${\sigma }_n$, which is also equivalent to solving the generalized Tikhonov regularization of the original problem [68].

5.1. Preventing Interior Faces

Simplifying all internal nodes that do not exceed the simplification error is too aggressive since our approach to dual contouring that greedily connects all immediate neighbors will generate nonmanifold geometry and internal faces (Figure 9). We prevent the generation of false internal faces with a simple, but overreaching, heuristic. For each set of empty child nodes of an internal node (red in Figure 12), the external neighbors (blue in Figure 12) must be empty to allow simplification. Since the node would be treated as fully occupied after simplification, the loss of an empty voxel could cause newly formed connections to neighbor nodes. If the blue nodes of Figure 12 are internal nodes, they are also treated as occupied to simplify the query. Moreover, suppose an immediate parent node exists at the preceding depth. In that case, the node is also not simplified because simplification is performed in parallel in a GPU kernel where the filtering condition cannot consistently detect it due to a race condition.

5.2. Silhouette Preservation

Vertex-clustering-based simplification is not designed to preserve the topology, and the QEF error is only correct in a valid 2-manifold. Geometry captured from depth images is highly nonmanifold since many objects will only be seen from one side. If simplification via vertex clustering is applied trivially on nonmanifold edges and corners, they will be pulled in, significantly changing the perceived silhouette in rendering (Figure 13).

To address this issue, each point is classified as interior or exterior to detect which octree nodes must be excluded from simplification. We query the entire 26-point neighborhood for each leaf cell in parallel in a GPU kernel and detect all potentially generated incident faces. To simplify this query, we use the approximation that four incident faces per vertex indicate that the vertex is placed in the mesh interior in a quad face, which is invalid, and exceptions exist (Figure 14a). Furthermore, if three faces are located in the same 2 × 2 × 2 octant of the neighborhood, the face is also in the interior (Figure 14b). All permutations of three incident faces are covered with two conditions: if less than three nodes are empty in the neighborhood described in Figure 14b, the node is an interior node.

6. Evaluation

We demonstrate the capabilities of our system using a recording of six ceiling-mounted Azure Kinect DK cameras [70] arranged as a ring, using the NFOV_UNBINNED mode on the depth camera and the maximum resolution of 640 × 576. The recording captures a room of 5.88 m × 7.15 m for 59 s, where three persons walk around a surgical table with a dummy head placed on top, and all persons enter and leave the captured domain multiple times. All performance measurements were recorded on an Ubuntu 18.04 server with 2× Intel Xeon Gold 5120 14 core CPUs and an Nvidia RTX A6000 GPU. Our build was compiled using GCC 7.5 and CUDA 11.1.

For our testing, the configuration parameters of the temporal filter [63] are $\alpha =0.15$, $\delta =3$ cm, and persistence at 3 (valid in last 2 of 4). For mesh extraction, we use 20 CPU threads and allow a maximum angle difference for vertex normals of 120°, and the parameters to control vertex merging across a splitting plane are ${\delta }_n={\delta }_d={30}^{\circ }$. While these parameters are constant, we will evaluate four different sets of voxelization and simplification settings. The parameter set unsimplified-low-detail uses a 2 cm voxel size and reconstructs the surface samples using principal component analysis across a neighborhood radius of 1.5 cm. The preset simplified-low-detail adds simplification with a QEF error threshold of 1 × 10⁻³ and normal standard deviation ${\sigma }_n=0.15$. The third preset, simplified-high-quality, changes the local surface estimation method to moving least squares [50], fitting a second-degree bivariate polynomial surface to the local neighborhood. Finally, the preset LOD uses MLS reconstruction with a neighborhood size of 1.5 cm only for the OR table in an axis-aligned box with dimensions of 95 × 38 × 180 cm and a simplification threshold of 1 × 10⁻⁴. The region outside the box uses voxels that are twice as large with 4cm and performs local surface estimation using PCA with 2.5 cm neighborhood radius and a simplification threshold of 1 × 10⁻³.

6.1. Comparison with Existing Offline Methods

We compare our implementation with a limited selection of existing, publicly available implementations for mesh reconstruction. As a primary, direct triangulation method of the input point cloud, we compare it with ball pivoting [71], which is implemented in Open3D [67] using ball radii of 1, 2, 3, 4, and 8 cm (Figure 15a). The more optimized greedy projection triangulation [69], which is part of the PCL point cloud processing library [72], greedily connects all points in the projected 2D neighborhood using the point’s normals (Figure 15b). For this method, we used a neighborhood radius of 5 cm, a maximum surface angle of 45°, a minimum triangle angle of 10°, and a maximum triangle angle of 120° to find as many connections as possible. For an implicit surface reconstruction method, we compare it with smooth signed distance surface reconstruction (SSD) [12] using the implementation of Michael Misha Kazhdan [73] with an octree depth of 10 and SurfaceTrimmer for level 8 or higher to remove all surfaces without many underlying samples. All methods use the unsimplified, voxelized point cloud generated using the first frame of our recording and the unsimplified-low-detail preset as their input. Other implicit surface reconstruction techniques, such as variants of Poisson reconstruction [32,33,49], are visually very similar to SSD in our particular dataset.

Our method generates fewer holes than the other general purpose direct meshing techniques (Figure 14b and Figure 15a) but cannot fill arbitrarily large gaps like implicit surface reconstruction (Figure 15c), which also inadvertently fills gaps, such as between the legs of the person in the background. Since our method also uses the voxelized samples directly, the resulting surfaces are not as smooth as the reconstruction with SSD since noisy points are still connected if they are in the immediate neighborhood. Therefore, our mesh extraction method heavily relies on correct, trustworthy samples from voxelization. Due to the simplification of flat surfaces, our method generates significantly fewer triangles.

6.2. Simplification Characteristics

Simplification is a key aspect of our proposed method since it saves bandwidth and subsequent time spent for mesh extraction if many flat surfaces are present in our geometry. We compare the loss of quality with respect to varying QEF error thresholds (Figure 16). The probabilistic normal standard deviation ${\sigma }_n$ is fixed at 0.15. As expected and consistent with previous work on QEF [59,68], the mesh error from simplification is small, even if the number of triangles is significantly reduced, particularly since many flat surfaces are present.

The simplification process reduces the total uncompressed memory requirement for mesh storage in this scene by approximately 40% (

Table 1. Simplification Benchmarks. Simplification run time and resulting mesh size for the first frame of our recording (Section 6). The memory requirement is calculated by assuming a simple indexed mesh representation requiring 24 bytes per vertex for position and normal and 12 bytes per triangle.

QEF Threshold	GPU (ms)	Vertices	Triangles	Memory (MiB)
				rel.	abs.
0	0	117,228	414,820	100%	7.43
1 × 10−5	10.88	117,055	409,582	99%	7.37
5 × 10−5	10.65	87,338	295,169	72%	5.38
1 × 10−4	10.97	72,736	239,102	59%	4.4
1 × 10−3	11.16	57,611	169,441	44%	3.26
1 × 10−2	12.3	53,933	157,055	41%	3.03

); thus, an uncompressed streaming application would only require 91 MiB/s instead of approximately 223 MiB/s net bandwidth at 30 meshes per second. The total run time for simplification is at least 10 ms due to the expensive neighborhood query at the lowest octree level to detect nonmanifold edges (Section 5.2).

6.3. Performance Characteristics

6.3.1. CPU Scaling

While most of our pipeline is performed on GPUs, we perform the extended recursive mesh extraction (Section 4.1), which has significantly more procedures to perform than the original work by Ju et al. [17] on CPU using task-based parallelization. We use a simple work-stealing-based scheduler with randomized victim selection based on the work-stealing queue of Lê et al. [75] with the modification to steal tasks in batches of 32 to reduce overhead.

Even though the scheduler has significant overhead (Figure 17a), we can hit our target run time of 31 ms with 14 cores for extracting an unsimplified mesh and 9 cores using a simplified mesh. We chose 31 ms rather than 33 ms for 30 FPS to accommodate CPU/GPU memory copies and mesh finalization, which takes less than 1 ms of GPU time in our measurements (

Table 2. Pipeline Benchmarks. Run time of each processing stage using the simplified-low-detail preset. GPU times captured using the NVIDIA Nsight Compute gpu__time_active metric. Approximately 1500 samples recorded for our entire recording (Section 6).

Processing Stage	Run Time (ms)
	Min.	Avg.	Max.
Temporal filter (6 images)	0.065	0.125	15.147
Octree generation	0.527	5.691	5.839
PCA surface sample reconstruction	0.012	11.642	41.65
Simplification	0.096	10.814	11.677
mesh extraction (14 CPU cores)	30.347	37.809	55.196
Mesh finalization	0.783	0.825	0.882

).

6.3.2. Full System Performance

To process a stream of depth images in real time, our method must keep up with the recording speed of 30 frames per second. Our playback system feeds images into the reconstruction pipeline as fast as possible, allowing observed frame rates above 30 FPS, which we achieve on average through our recording using the LOD preset, and only misses by 2 FPS for unsimplified- and simplified-low-detail (

Table 3. System Performance. Full system performance using a selection of preset settings (Section 6.3.2).

Preset Name	Frames per Second			Latency (ms)
	Min.	Avg.	Max.	Min.	Avg.	Max.
unsimplified-low-detail	26.1	28.3	31	110	272	305
simplified-low-detail	26.1	28.3	31.3	185	274	303
simplified-high-detail	14.5	24.6	29.6	99	111	192
LOD	17.3	31.8	51.2	45	68	140

), which is due to a slight CPU bottleneck (Table 2) in mesh extraction.

The latency figures (Table 3) are larger than necessary due to our usage of a ring buffer (Figure 3) that queues up to 2 frames to fully saturate the mesh extraction process. If the mesh extraction is not saturated and, therefore, no frames are queued up, like in the LOD preset, latency is the sum of all serial processing steps (Table 2) with some communication overhead.

7. Discussion

Instead of using a signed distance field as an intermediate representation, we directly generate each voxel cell’s representative vertices and surface normal using well-understood point cloud voxelization and local surface estimation [50,66]. With this decision, we effectively only reconstruct data ultimately used in meshing. However, since dual contouring on an SDF does not consider diagonal neighborhoods in the voxel grid, we must extend the recursive traversal to include all 32 diagonal quad face cases (Section 4.1). Furthermore, to utilize simplification, we require an exhaustive condition for the preservation of the connectivity (Section 5.1) and a nonmanifold edge detection (Section 5.2) to preserve the original silhouette.

However, extending the neighborhood also causes unwanted, accidental interior faces constrained by the voxel size (Figure 9b). A manual fix-up of poorly aligned voxel layers is also necessary (Figure 10b), which would not be an issue with implicit surfaces encoded as a signed distance field.

While our method is not intended to compete with general-purpose surface reconstruction methods due to limited hole filling, which requires dense input point clouds to generate a voxelization without holes, our performance results, room size, and number of cameras are on par with the FusionMLS [15] stateless reconstruction system while also performing the simplification of flat surfaces. The efficiency of the task-based CPU parallelization (Figure 17a) of our current prototype can be further improved using better, state-of-the-art lightweight task scheduling systems [76].

8. Conclusions and Future Work

We presented a stateless surface reconstruction method, which is scalable and efficient enough to reconstruct a full room in real time with sufficient quality for immersive teleconsultation. Our method generates a mesh directly from voxelized point clouds. We extended the formulation of Ju’s recursive dual-mesh traversal [17] for cases of the diagonal neighborhood and showed how to encode these cases. Like in the original dual-contouring meshing, manifoldness is not guaranteed, which appears in our application if surfaces are too close to resolve correctly in the voxel grid. Holes are only filled if the voxelization itself does not contain holes, and we presented a way of merging or separating close-by surfaces if the grid for voxelization is suboptimally aligned. We also specified additional conditions for safe simplifications, such that no interior faces are generated.

Since we use a dual-meshing method, multiple detail levels with mixed resolutions for voxels are possible, and seamless connections are formed for the final mesh, which makes the method very versatile. Due to the voxel-clustering-based simplification of the octree, the output mesh is much smaller compared with other mesh reconstruction techniques, such as marching cubes or direct 2.5D meshing of all depth pixels [5,15], particularly if the geometry consists of many flat surfaces. Moreover, since simplification is a predecessor of mesh generation, the workload for the subsequent mesh generation is significantly reduced. Given the right parameters, we can provide a reconstruction of a simplified mesh from geometry captured with six fixed, partially overlapping depth cameras in real time directly, without a canonical model or more expensive out-of-order processing steps. The quality of our reconstruction is better than applying an existing direct-meshing method directly after voxelization, but not as smooth and hole-free as implicit surface reconstruction because we still use the points obtained by voxelization directly.

Currently, we use a per-pixel screen space texturing method for the final mesh to demonstrate our system (Section 3.5), which is prone to errors with slight misalignments of the calibration and requires high-quality depth images. The integration of existing work for consistent texturing [77,78,79] would be very beneficial to increase visual fidelity and consistency, which is vital for medical telepresence since many features and tools are too small to resolve with geometry using depth cameras. The enhancement of depth images [80,81,82,83] could also significantly increase the quality of our visual result and the time warping procedure presented in FusionMLS [15] to merge depth images at different recording timestamps better.

While our method generates a simplified mesh, we currently do not compress the representation during transport, requiring us to use a gigabit connection. Mesh compression [84] libraries, such as Google/Draco [85], add a lot of latency for the large-scale meshes we use. Exploiting the spatiotemporal coherence [86] would yield best compression ratios, and since our mesh is not arbitrary, but generated from an octree, the data structure could be potentially exploited to more efficiently find the relevant frame-to-frame deltas in future works. Furthermore, DNN-based image segmentations for static and dynamic sections, like in Yang et al. [87], have the potential to further reduce the load of dynamic updates through the system by either completely splitting the pipeline, such that no connecting faces between the static and dynamic parts (e.g., feet on the floor) are formed, or through intelligently combining the octrees before mesh extraction to generate seamless connections.

Currently, our filters for simplification are very restrictive since only one representative vertex is contained in each cell. Encoding extensions for multiple vertices per cell, like Zhang et al. [61] presented, would further extend the opportunities for simplification if they are adapted to the highly parallel GPU execution model.