Fused Transparent Visualization of Point Cloud Data and Background Photographic Image for Tangible Cultural Heritage Assets

Abstract

Digital archiving of three-dimensional cultural heritage assets has increased the demand for visualization of large-scale point clouds of cultural heritage assets acquired by laser scanning. We proposed a fused transparent visualization method that visualizes a point cloud of a cultural heritage asset in an environment using a photographic image as the background. We also proposed lightness adjustment and color enhancement methods to deal with the reduced visibility caused by the fused visualization. We applied the proposed method to a laser-scanned point cloud of a high-valued cultural festival float with complex inner and outer structures. Experimental results demonstrate that the proposed method enables high-quality transparent visualization of the cultural asset in its surrounding environment.

1. Introduction

Digital archiving, which involves measuring, recording, and preserving tangible and intangible cultural assets, utilizing digital information technology, has attracted increasing attention worldwide [1,2,3]. Targets of digital archiving have expanded from planar materials, such as documents, paintings, and photographs, to three-dimensional (3D) objects, such as sculptures, buildings, and archeological artifacts, as well as intangible cultural assets, such as dances, plays, and cultural events [4,5,6]. To make digital archives of 3D objects, particularly large-scale cultural assets, simply capturing an image with a camera is insufficient. Over the past decade, with the development of laser scanning, photographic measurement, and unmanned aerial vehicles, archiving of large-scale 3D objects became possible [7]. A vehicle-based mobile mapping system that integrates a camera, laser scanner, inertial measurement unit, and global positioning system provides an efficient way to generate a 3D point cloud [8]. Precise measurement of large-scale 3D objects has advanced digital archiving of large-scale cultural heritage assets [9,10,11,12,13,14].

Laser-scanned point clouds of large-scale cultural heritage assets often contain complex structures and a large number of points. For example, the numbers of 3D points acquired in our laser-scanning projects were as follows: 3 × 10⁸ points for Khentkawes’ Tomb (Egypt), 3 × 10⁸ points for Machu Picchu (Peru), and 9 × 10⁸ points for Hagia Sophia (Turkey). The most straightforward visualization strategy for a point cloud is point-based rendering [15,16]. Point-based rendering techniques do not require a pre-processing to transform the raw point cloud data into a polygon mesh or a voxel-based presentation. Thus, the precision and density of the original data can be preserved. Discher et al. proposed a real-time rendering approach for 3D point clouds that combines point-based and image-based rendering techniques [17,18,19]. Their studies focused on the applications to virtual reality or web-based environments. However, transparent visualization is required to understand the complex shape and internal structures of valuable cultural heritage assets. Seemann et al. proposed a transparent visualization technique that combines traditional surface splatting with semi-transparent spheres for complex point clouds with different qualities [20]. Conventional point-based rendering suffers from the computational cost required by depth sorting, which makes it unsuitable for large-scale point clouds. In our previous research, we proposed a stochastic algorithm, i.e., Stochastic Point-Based Rendering (SPBR), for precise transparent visualization of large-scale complex point clouds [21]. SPBR achieves noise-robust, high-speed, transparent visualization without requiring depth sorting. We applied SPBR to transparent visualization of large-scale cultural heritage assets [22]. In these applications, we used single-color (black or white) backgrounds to avoid the effects of colors on the visualization results. However, in some cases, scale, position, and surrounding environment information are essential to present a cultural heritage asset. Visualization of cultural heritage assets, as well as their surrounding environments, is required. The simple idea is to scan the entire surroundings and visualize them with the target cultural asset. However, this is not practical because it requires a great deal of manual effort and time, as well as computational time for visualization. Tanaka et al. proposed a method that generates 3D point clouds of surrounding environments using panoramic images [23]. Discher et al. integrated context-providing geographic data, such as 2D maps and 3D terrain models, for visualization and exploration of 3D point clouds [19]. However, to the authors’ knowledge, no previous study has reported fused transparent visualization of a large-scale laser-scanned point cloud at interactive speed.

Therefore, we proposed a fused visualization method that transparently visualizes a point cloud of a cultural heritage asset in an environment using a photographic image as the background. We also proposed lightness adjustment and color enhancement methods to improve the visibility of the cultural heritage asset visualized by the proposed fused transparent visualization method.

2. Fused Transparent Visualization of Point Cloud Data and Background Photographic Image

In this section, we describe the proposed method for precise fused transparent visualization of large-scale complex point clouds acquired by laser scanning of cultural heritage objects. The proposed method is an extension of SPBR, which was developed in our previous research.

2.1. Fused Transparent Visualization Procedure

Similar to the conventional SPBR visualization procedure [21], the fused transparent visualization also consists of three steps.

STEP 1. Creation of point ensembles

We adjusted the number of points according to the desired opacity $\alpha$ and then randomly divided the points into multiple groups that we refer to as “point ensembles.” Each point ensemble is statistically independent and has the same point density. Hereafter, we refer to the number of point ensembles as the “ensemble number” and denote it by L.

STEP 2. Point projection and background fusion

For each point ensemble, we created an intermediate image by projecting the 3D points onto a 2D image plane. In this process, we considered point occlusion per pixel. In other words, if a pixel is projected by multiple points, its value is determined by the nearest point along the sight direction. If a pixel is projected by none of the points, its value is given by the background photographic image. This step differs from the conventional SPBR. Thus, each intermediate image is a fused result of the point ensemble and the background photographic image.

STEP 3. Intermediate image averaging

The output fused transparent image of the point cloud data and the photographic image was created by averaging the L intermediate images. Thus, an SPBR-based fused transparent visualization of the point cloud and background was achieved.

The difference between the proposed method and the conventional SPBR [21] is that rather than using a specific fixed color for the background in STEP 2, the proposed method used the pixel color of the selected photo as the background image. Therefore, the background of the transparently visualized target object is its surrounding scene captured in a photograph rather than a single-color. Scheme of the proposed fused transparent visualization is shown in Figure 1.

2.2. Application to a Laser-Scanned Point Cloud

We applied our method to a laser-scanned point cloud of the Hachiman-Yama float in the Gion Festival (Figure A1 and Figure A2) and fused it with a corresponding background photographic image. The point cloud acquired with a RIGEL laser scanner (VZ-400) contains 2.62 × 10⁷ points. Figure 2 shows the comparison result of the conventional SPBR based method and the proposed method $\left(L=100,\alpha =0.3\right)$. Figure 3 shows the local details of the visualization results of the two methods.

As can be seen in Figure 2c, the scale and surroundings of the float can be perceived, as well as its inner structure, using the proposed method. However, with the effect of the background, the visibility of the float decreased compared to the visibility obtained using the conventional visualization method without the background. Thus, we proposed a visibility enhancement method to solve this problem.

3. Visibility Enhancement for Fused Transparent Visualization

As described in the previous section, the proposed method achieved fused transparent visualization of the point cloud and the background photographic image. However, the visualized point cloud, particularly parts with a lower opacity, becomes unclear after fusing with the colorful background. In this section, we analyze the problems and propose solutions to enhance the visibility of the fused transparent visualization.

3.1. Causes of the Problem

Two issues are considered to cause a lack of clarity. (1) For a pixel in the visualized image, when the colors of the projected points and the background are similar, the difference between the object and the background is difficult to distinguish. (2) When a small number of points are projected to a pixel, after averaging the intermediate image, the color of the pixel will be dominated by the background color.

First, we verified the color difference between the visualized image after averaging the ensembles of the point cloud and the background image shown in Figure 2 in the CIELAB color space. For two colors, $\left(L_1^{*},a_1^{*},b_1^{*}\right)$ and $\left(L_2^{*},a_2^{*},b_2^{*}\right)$, the color difference $\Delta E_{ab}^{*}$ is defined as:

$$\Delta E_{ab}^{*}=\sqrt{{\left(L_2^{*}-L_1^{*}\right)}^2+{\left(a_2^{*}-a_1^{*}\right)}^2+{\left(b_2^{*}-b_1^{*}\right)}^2}$$

(1)

The normalized color difference $\Delta E_{ab}^{*}{}^{\prime }\left(i,j\right)$ on a given pixel can be calculated as follows:

$$\Delta E_{ab}^{*}{}^{\prime }\left(i,j\right)=\frac{\Delta E_{ab}^{*}\left(i,j\right)-\Delta E_{ab\min }^{*}}{\Delta E_{ab\max }^{*}-\Delta E_{ab\min }^{*}}$$

(2)

where $\Delta E_{ab}^{*}\left(i,j\right)$ is the color difference on pixel $\left(i,j\right)$, and $\Delta E_{ab\min }^{*}$ and $\Delta E_{ab\max }^{*}$ are the minimum and maximum color difference values in the whole image, respectively.

We mapped the normalized color difference to a rainbow color map with 256 steps. The visualization result is shown in Figure 4a. As can be seen, the color differences in many regions, particularly the contours of the lanterns in the upper part, were not significant. This color similarity would cause low visibility in these regions on the application of fused visualization.

Then, we verified the number of projections on each pixel. The normalized number of projected points on a given pixel $p^{\prime }\left(i,j\right)$ is calculated as follows:

$$p^{\prime }\left(i,j\right)=\frac{p\left(i,j\right)-p_{\min }}{p_{\max }-p_{\min }}$$

(3)

where $p\left(i,j\right)$ is the number of projected points on pixel $\left(i,j\right)$, and $p_{\min }$ and $p_{\max }$ are the minimum and maximum numbers of projected points in the whole image, respectively.

We also mapped the normalized number of projected points to a rainbow color map with 256 steps. The visualization result is shown in Figure 4b. As can be seen, more points were projected to the pixels that belong to the framework of the float. However, fewer points were projected to the lanterns; thus, visibility enhancement on these regions is required.

3.2. Lightness Adjustment

For the regions where the colors of the point cloud and the background were similar, we applied lightness adjustment to the background photographic image.

For each pixel in the visualized image, its value is given by the alpha blending formula [24]:

$$B=\alpha C_{pt}+\left(1-\alpha \right)C_{bg}$$

(4)

where $C_{pt}$ is the average color of the projected points, and $C_{bg}$ is the color of the background. We converted the background image to the HSV (hue, saturation, value) color space and introduced a parameter β ($0\le \beta \le 1$) to Equation (4). Thus, Equation (4) can be rewritten as follows:

$$B=\alpha C_{pt}+\beta \left(1-\alpha \right)C_{bg}$$

(5)

Note that β is only applied to the V channel in the HSV color space. When $\beta =1$, Equation (5) is identical to Equation (4), in which lightness adjustment does not apply. For each pixel on the projection plane, lightness adjustment only applies when color difference $\Delta E_{ab}^{*}\left(i,j\right)$ is smaller than a threshold of 0.2. Therefore, lightness adjustment was applied adaptively. Figure 5 shows the results of lightness adjustment at different β values ($\beta =0.9,0.7,\mathrm{and}0.4$). As can be seen, when β was 0.9, the visibility of the lantern in the upper left was still low. However, if β was set to 0.4, the visualized image became unnatural due to the darker background. According to our experiments, 0.7 is the proper value for β in this application.

3.3. Color Enhancement

When a pixel has fewer projected points, its color will be dominated by the background color, even when the colors of the projected points and the background differ significantly. This problem can be solved by setting a higher opacity $\alpha$ in transparent visualization using SPBR. However, higher $\alpha$ requires a very large number of projected points, which is extremely difficult in practice. Therefore, we proposed a color enhancement method by replacing the background color with the average point color according to a certain probability.

For the regions only contained in the background, the pixel color in the final image was defined only by the background photographic image. Otherwise, we compared the normalized number of projected points $p^{\prime }\left(i,j\right)$ of each pixel to a threshold. If $p^{\prime }\left(i,j\right)$ was greater than or equal to the threshold, we considered that a sufficient number of points were projected to the pixel and did not perform any extra processes. On the other hand, if $p^{\prime }\left(i,j\right)$ was smaller than the threshold, color enhancement was required for this pixel. Because the final image was generated by averaging the intermediate images, we performed a subsequent process on the pixels of the intermediate images. For each pixel that needed to be color-enhanced, we examined its corresponding pixels in the intermediate images. If the pixel in the intermediate image was not projected by any points, rather than the background color, its color was set to the average color of the projected point $C_{pt}$ by a probability $\mu$.

Figure 6 shows the results of color enhancement at different $\mu$ values ($\mu =0.3,0.6,\mathrm{and}0.9$). In this experiment, we set the threshold for the normalized number of projected points to 0.3, which is consistent with opacity $\alpha$. As can be seen, with a higher $\mu$ value, the color of the point cloud had been enhanced. However, higher $\mu$ led to unnatural enhancement on the visualized image. We considered 0.6 to be an appropriate value for $\mu$ in this application.

4. Experimental Results and Evaluations

We implemented the proposed method using C++ with several libraries, including Open Graphics Library (OpenGL), Point Cloud Library (PCL), and Kyoto Visualization System (KVS). All tests were performed on an iMac with an Intel Core i-7-5960X CPU, 16 GB memory (DDR3, 1600 MHz), and an NVIDIA GeForce GT 750M GPU (1024 MB). We combined lightness adjustment and color enhancement and applied the combined processes to fused transparent visualization of the point cloud and background photographic image. Average computation time for each process is shown in

Table 1. Average computation time for each process of the proposed method (in seconds).

Lightness Adjustment	Color Enhancement	Fused Transparent Visualization
0.09	2.38	3.19

. Figure 7a shows the improved fused visualization result using the proposed method $\left(L=100,\alpha =0.3,\beta =0.7,\mu =0.6\right)$. Figure 7b shows the local details of the visualization result of Figure 7a. Compared to the visualization results without lightness adjustment and color enhancement in Figure 2c and Figure 3b,d, the proposed method achieved better visibility, particularly in the regions that contain lanterns.

To evaluate the effectiveness of the proposed method, we applied the Sobel filter [25] to extract the edges in the images. The Sobel filter measures the spatial gradient on an image by applying a pair of 3 × 3 convolutional kernels to detect horizontal and vertical edge components. We assumed that regions with high visibility result in clear edges. Figure 8 shows the edge extraction results by applying the Sobel filter to the visualization results of the conventional and the proposed methods. Figure 9 shows the local details of the upper left part of the corresponding images in Figure 8.

As shown in Figure 8a and Figure 9a, the edges of some objects, particularly the lanterns in the upper left, have not been extracted successfully. This indicates that the visibility of the objects is low in these regions. Figure 8b,c and Figure 9b,c show the results of applying lightness adjustment and transparent visualization individually to improve the visibility of the fused transparent visualization. It is evident that lightness adjustment successfully emphasized the contours of the lanterns, whereas color enhancement successfully emphasized the patterns on the lanterns. The edge extraction result of the proposed method that combines lightness adjustment and color enhancement is shown in Figure 8d and Figure 9d. We can see that both the shapes and patterns of the lanterns are extracted successfully. The experimental results show that the proposed method has improved the visibility of the point cloud object in the fused transparent visualization.

An overview of the advantages and disadvantages of the conventional SPBR and the proposed method is shown in

Table 2. Overview of the conventional SPBR (Stochastic Point-Based Rendering) and the proposed method.

	SPBR	Proposed Fused Transparent Visualization	Proposed Method with Lightness Adjustment	Proposed Method with Color Enhancement	Proposed Method with Lightness Adjustment and Color Enhancement
Transparent visualization	Yes	Yes	Yes	Yes	Yes
Fused visualization	No	Yes	Yes	Yes	Yes
Visibility in regions with similar object-background colors (e.g., counters of the lanterns)	Poor	Poor	Good	Poor	Good
Visibility in regions with few projected points (e.g., patterns of the lanterns)	Poor	Poor	Poor	Good	Good

.

5. Conclusions

We have proposed a fused transparent visualization method to visualize laser-scanned point cloud data of cultural heritage assets with their surrounding environments. We extended our SPBR method to adapt to the background fusion task. Compared to conventional transparent visualization with a single-color background, fused transparent visualization with a photographic image as the background often results in low visibility of foreground objects, which means it is difficult to distinguish the valuable cultural heritage from the background. Therefore, we proposed two solutions to improve the visibility of the fused transparent visualization result: (1) lightness adjustment for regions with similar colors of the point cloud and the background, and (2) color enhancement for regions with fewer projected points. The experimental results have confirmed the effectiveness of the proposed method.

In the current process, the size of the point cloud on the projection plane was determined manually according to the background image. In the future, we will investigate the automatic adjustment of the size and angle of the point cloud for projection based on the background information. Besides, we plan to extend the proposed method to fused transparent visualization with panoramic photos, which will provide a 360-degree view for the audience to appreciate the cultural heritage in a natural surrounding environment.