Contextualization of Archaeological Information Using Augmented Photospheres, Viewed with Head-Mounted Displays

Abstract

Photospheres, or 360° photos, offer valuable opportunities for perceiving space, especially when viewed through head-mounted displays designed for virtual reality. Here, we propose to take advantage of this potential for archaeology and cultural heritage, and to extend it by augmenting the images with existing documentation, such as 2D maps or 3D models, resulting from research studies. Photospheres are generally produced in the form of distorted equirectangular projections, neither georeferenced nor oriented, so that any registration of external documentation is far from straightforward. The present paper seeks to fill this gap by providing simple practical solutions, based on rigid and non-rigid transformations. Immersive virtual environments augmented by research materials can be very useful to contextualize archaeological discoveries, and to test research hypotheses, especially when the team is back at the laboratory. Colleagues and the general public can also be transported to the site, almost physically, generating an authentic sense of presence, which greatly facilitates the contextualization of the archaeological information gathered. This is especially true with head-mounted displays, but the resulting images can also be inspected using applications designed for the web, or viewers for smartphones, tablets and computers.

1. Introduction

In archaeology, as well as in other scientific fields where information derived from the environment is of primary importance, the physical presence of researchers at the site under investigation is often beneficial for the quality of interpretations. Knowledge of the environmental context (e.g., topography, vegetation cover, presence of anthropogenic structures, geomorphological or geological peculiarities, etc.) beyond the zone studied generally contributes to better understanding of the processes and potential interactions that have presided over the development of the site. Archaeologists therefore seek to describe context, far beyond the restricted geographical extent of excavated areas. Despite these attempts, without physical presence at the site, it is often difficult to apprehend space (by fixing references, and estimating scales, distances, and volumes), since spatial perception is closely related to personal sensory experience [1]. For example, Henri Poincaré stated that locating an object in space is simply figuring out the physical movements that would be necessary to reach it [2]. Even the best photographs, which incorporate scale, can only convey some of the relationships that exist between objects and/or structures, because perceived distances and angles cannot be fully described by a single perspective [3]. The field of view covered by photographs is limited by the focal length of the lens, and the world is generally rendered at eye level (i.e. from one to two meters above the ground). Geographical information systems (GIS) are efficient for the manipulation and analysis of spatial information layers [4], but generally fail when attempting to render any physical sensation of space. Innovative solutions can nowadays be found in the field of virtual reality (VR). The concept of virtual archaeology was first proposed by Paul Reilly in 1990 [5], when he originally referred to the use of three-dimensional models of buildings and artefacts for documentation purposes. The considerable advancements in the field of computer graphics during the past few decades have led to the development of virtual tours staging archaeological information in virtual worlds. These tours are often based on web viewers, where 360° photos (also known as spherical panoramas or photospheres) are used for navigation, while trigger buttons provide access to specific archaeological information in 2D (plans, etc.) or 3D form [6,7,8,9,10]. Three-dimensional models may also be rendered in interactive virtual 3D scenes, enhancing the feeling of presence [11], or for research purposes, such as testing hypotheses [12]. The interest of such virtual tours, integrating informational modeling, is obvious for heritage preservation [13]. Virtual environments can thus be created, providing documentation, interpretation, and information about the conservation state of the archaeological remains [6]. Interestingly, virtual tours also convey valuable information for education [14,15]. As suggested by Barcelo [16], scientific visualization should no longer be restricted to “presentation” techniques, but should also include explanatory tools. Incorporating archaeological documentation directly into the 360° scene (and separately as often the case) is expected to reinforce the feeling of immersion, more particularly when tours are experienced via wide field-of-view, immersive, head-mounted displays (HMDs) [17,18]. Such devices provide a better sense of scale and depth than any printed document or image displayed with standard monitors and projectors [19], even though absolute egocentric distances may be slightly underestimated by HMDs, compared to the real world [1]. Given an interpupillary distance of about 6 cm, we are not able to apprehend a 3D scene solely by stereoscopic vision for objects further away than 10–20 m [20]. To estimate depth and scale, our brain therefore uses the presence of familiar objects, assumed to be of normal size, interprets the possible overlay of object contours, and the distribution of highlights and shadows, and analyzes linear perspectives [21]. Therefore, a full 3D rendering (i.e., the production of two different, shifted images, one for each eye) is not always necessary, and simple 2D photospheres may suffice to apprehend space. The ability to augment, on demand, 2D representation as a photosphere of an archaeological site with pertinent information (e.g., maps, 3D models, virtual reconstruction, etc.) would be advantageous for research purposes, for disseminating discoveries to a broader audience, and even to promote tourism. For scenes modelled entirely in 3D, registering any additional 3D models is relatively easy, using specialized game engines, such as Unity, Blender, or others [22,23], because 3D coordinates on which new documents must be attached are available. After registration, the scene can then be rendered using a virtual 360° camera. Projecting the same documents on 2D photospheres may be much more problematic. Photospheres acquired from the air or from the ground are neither georeferenced nor oriented, nor do they contain information about depth. Equirectangular projection also suffers from polar deformation (e.g., North and South Poles are stretched over the entire width of the image, at the top and bottom edges, respectively). Accurate registration of objects close to polar regions may be far from straightforward, as these regions appear highly distorted. Some authors therefore simply place the 3D model inside a rendered virtual scene, and then manually translate, rotate, and scale-up, to reach the appropriate alignment [24]. When integrating 2D documents produced by orthoprojection, severe visual mismatches with the photosphere may occur due to parallax issues. In such cases, there is no chance that good results will be achieved manually, even after many attempts. To the best of our knowledge, there is no simple procedure available nowadays for accurate registration of 2D or 3D archaeological information on equirectangular projections, whether in game engines, or in packages or libraries of the main programming languages. The aim of this study is therefore to propose practical workflows to overcome these drawbacks, with data from the archaeological sites of Loropéni (Burkina Faso) and Tsatsiin Ereg (Mongolia).

2. Materials and Methods

2.1. Study Sites

The photospheres and archaeological information presented here were acquired from two archaeological sites. The first is the Loropéni ruins, in Burkina Faso, probably dating from the fifteenth century [25], and listed as a World Heritage site by UNESCO since 2009 [26,27]. This imposing enclosure, with laterite stone walls up to 6 m high, built to protect buildings, courtyards, and alleys, covers about 1 ha, and is surrounded by wooded savanna. The second site, at Tsatsiin Ereg, Mongolia, is composed of large burial structures, with decorated stelae belonging to the Khereksuur and Deer Stone cultures [28]. These stelae were raised at the end of the second to early first millennium BC. Standing about 0.5–5 m tall, these megaliths were hand-carved with graceful symbols, including stylized deer [29].

2.2. Photosphere Assembly

The ground-based spherical panoramas were acquired using a Roundabout-NP Deluxe II 5R panoramic tripod head (Roundabout-NP, Rosenheim, Deutschland), to which was attached a DSLR Nikon D800 (Nikon Corporation, Tokyo, Japan), equipped with a NIKKOR 24 mm prime lens (Nikkor, Tokyo, Japan); see Supplementary Materials, Text S1 for more details about principles of equirectangular projection. The use of a panoramic head allows the nodal point of the lens to be maintained on the rotation axis, avoiding stitching failure, in particular for the nearest objects. Two sets of 12 pictures, using a horizontal rotation of 30° between pictures, were taken at +30° and −30° from the horizontal plane. Two additional pictures were acquired, one for nadir and one for zenith. For aerial views, a Phantom 3 PRO from DJI was used, in combination with the Litchi application for Android. This combination allows the seamless automatic acquisition of a set of pictures, covering slightly more than the entire southern hemisphere. Three rows of 13 pictures with different inclinations were necessary for appropriate coverage and overlapping. The images were stitched together with Autopano Giga Pro 4, Kolor/Gopro (Kolor, Francin, France), a software program also used to produce the final equirectangular images, limited here to 6000 × 3000 pixels. The northern hemisphere, depicting the sky, which cannot be captured by drone, contained no archaeological information. It was therefore manually completed for natural rendering. For ground acquisition, the tripod visible at the nadir was simply patched with a logo.

2.3. Programming

The code snippets allowing the registration of archaeological information were written, for images, in Python 3.6, with the help of the opencv, numpy, png, and matplotlib libraries, and a set of functions written for the CellTool software [30]. For 3D models, the script used the R language [31] in combination with the Morpho, mesheR, rgl, Rvcg, and png packages. Codes are provided as Supplementary Materials (Codes S1 and S2).

3. Method Implementation

3.1. Registration of Planar Documentation on Approximately Horizontal Surfaces

With aerial photospheres taken at a sufficient elevation above the ground, there is a good chance that an archaeological structure located close to the South Pole can be considered to be approximately horizontal. This approximation means that its projection from the position where the viewer stands, on any horizontal plane situated below the viewer, produces an image of the structure possessing the same shape as the orthographic map to be registered (Figure 1). In this case, registration can easily be accomplished by Procrustes transformation, using the sum of errors between input and target landmark coordinates as a goodness-of-fit criterion [32,33,34]. Expressed more formally, let P and Q be two configurations of n points on input and target images. The aim is to find a rigid-body transformation T so that:

$$\underset{\mathit{T}}{min}||\mathit{P}\mathit{T}-\mathit{Q}||$$

(1)

with T a matrix corresponding to translation, orthogonal rotation, and uniform scaling (see [29] for details of optimization methods).

Once registered, the image can be projected onto the sphere, and then transformed by equirectangular projection. The problem is how to select with precision at least three (preferentially more) unambiguous and not collinear landmarks on the equirectangular image, given that the area of interest lies in the distorted part of the southern hemisphere with aerial photospheres. For this step, the sphere is first rotated to make the region of interest coincide with the South Pole. The southern hemisphere is then stereographically projected from the North Pole (0, 0, 1) onto a plane below the sphere, here tangent at the South Pole (0, 0, −1) [35]. Such a projection is not recommended for mapping large parts of the southern hemisphere due to the high distortion appearing close to the equator, but it performs well close to the South Pole [36], where the site of interest lies. Archaeological structures become clearly visible and landmarks can be precisely positioned. Note that the workflow would have been simplified if the first stereographic projection used to define the landmark’s position had been performed directly from the center of the sphere, instead of from the North Pole. Tests nonetheless proved that, for large structures, a projection from the sphere center produces more distortion, increasing further from the nadir, and making landmark positioning more difficult. The example presented here uses an aerial view of Loropéni, on which is projected a high-resolution hill-shaded digital elevation model (DEM), produced by photogrammetry from more than 4000 pictures taken at eye level (Figure 2).

The workflow is summarized below:

• Determine the coordinates of the center of the area of interest on the equirectangular image (Figure 2A).
• Rotate the sphere in order to place the center of interest at the South Pole (Figure 2B).
• Project the area surrounding the South Pole stereographically on a tangent plane, using the North Pole as reference (Figure 2C₁).
• Position a set of landmarks manually on the document to be registered (input) and on the stereographic projection (target) (Figure 2C₁,C₂).
• Back-project the target landmarks onto the unit sphere (North Pole as reference) (Figure 2D).
• Rotate the landmarks together with the unit sphere to return to the sphere’s original position (Figure 2E).
• Project the landmarks on a plane tangent to the South Pole, taking the center of the unit sphere as reference.
• Calculate the rotation matrix, scaling factor, and translation vector, from input to target coordinates by rigid Procrustes registration (cf. Equation (1)).
• Position the transformed input image in the tangent plane coordinate system.
• Back-project the registered image onto the unit sphere (sphere center as reference).
• Perform an equirectangular projection of the image to be registered (Figure 2F), and add as background the original image (Figure 2G).
• The final image is ready to be seen in HMD (Figure 2H).

The registered image perfectly fits the archaeological structures (Figure 2G,H; see also in Supplementary Material, Figures S1 and S2 for the original and augmented jpeg images), strengthening the feeling of immersion for the viewer, particularly when the augmented image is viewed with the HMD. Note that any other map could be projected following the same procedure.

3.2. Registration of Planar Documentation on Irregular Surfaces

It is also quite common for the topographical surface to be irregular, or for the viewpoint not to be elevated enough to consider this surface as reasonably horizontal. Figure 3 illustrates the problems related to parallax when points of interest are located on an irregular surface. Let A’, B’, C’ be the apparent position on the horizontal plane of points A, B, C from viewpoint 1, and A’’, B’’, C’’ from viewpoint 2, while A’’’, B’’’, C’’’ are the orthographic projection of A, B, C (e.g., a map of the area). The distances between points of interests A’, B’, C’ and A’’, B’’, C’’ projected onto the plane are no longer proportionate to their counterparts, A’’’, B’’’, C’’’, on the map to be registered (cf. Figure 3). This is especially true from viewpoint 1: B’ appears much closer to C’ than to A’, while B’’’ is approximately in mid-position between A’’’ and C’’’. Note that from viewpoint 2, the ratios of distances between points are in greater conformity with those observed on the map, because of the approximately zenithal position of viewpoint 2, and its high elevation in relation to variations in relief. In such circumstances, especially for the scene observed from viewpoint 1, any rigid registration of the map is not recommended, because serious mismatches are expected if the map is registered using the Procrustes procedure described above. Practical solutions can rather be found in non-rigid techniques, such as thin plate splines (TPS), a method popularized by image morphing. This technique is based on surface interpolation, using a curve-smoothing spline function over scattered control points. As constraints, the input surface is deformed using minimal bending, and all landmark pairs belonging to both input and target must coincide [37]. Warping with TPS requires more point placements than Procrustes registration, because deformation may produce some mismatches, especially close to the edges (if not well constrained by several control points). In practice, the proposed workflow is comparable to the one described above, except that TPS warping is applied in place of Procrustes registration, and that the stereographic projection from the center of the sphere is unnecessary, since TPS accounts for all deformation. The example provided here concerns the burial structures of Tsatsiin Ereg. An orthomosaic was produced from aerial images over a 16 ha area, very rich in funeral monuments made of dry stones. A supervised machine-learning algorithm was applied to delineate the stones automatically on the basis of RGB channels, texture parameters, and elevation (more details about the procedure will be provided in [38]). The aim is to project this information onto the photosphere, to make the structures more visible for the spectator.

A total of 87 control points were therefore placed on both input and target images. The workflow is as follows:

• Determine the center of the area of interest on the equirectangular image.
• Rotate the unit sphere in order to place the center of interest at the South Pole (Figure 4B).
• Project the area surrounding the South Pole stereographically onto a tangent plane, using the North Pole as reference (Figure 4B₁).
• Place a set of control points manually on the stereographic projection (target), and on the document to be registered (input) (Figure 4B₁,B₂).
• Perform TPS deformation of the input image (Figure 4C)
• Project the deformed input image onto the unit sphere.
• Rotate the sphere to return it to its original position.
• Perform equirectangular projection (Figure 4D), and add the original image as background (Figure 4E).
• The final image is ready to be seen in HMD (Figure 4F).

As expected, the image resulting from TPS deformation appears to fit the archaeological structures well, except perhaps close to the edges (Figure 4E; see also in Supplementary Material, Figures S3 and S4 for the original and augmented jpeg images). The results are much more accurate than those obtained by Procrustes (not shown here), because the topography of the surface on which the monument lies is somewhat irregular.

3.3. Registration of 3D Models on the Photosphere

Advances in acquiring 3D models and the treatment of their geometry have also been adapted for archaeological and cultural heritage purposes [39,40,41,42,43]. Such information might also efficiently augment photospheres. The alignment of the 3D model can easily be achieved by first positioning two sets of landmarks on both the equirectangular image and the 3D model (Figure 5).

The optimal position of the 3D model is obtained by minimizing the sum of squared distances, d_i, between landmarks placed on the models and the lines linking their corresponding counterparts on the photosphere to the sphere center:

$$\underset{a,b,c,\phi ,\theta ,\rho }{\min }\sum d_i^2,$$

(2)

with

$$d_i=\frac{||\stackrel{\rightharpoonup }{O{P}_i^{’}}\times \stackrel{\rightharpoonup }{O{P}_i}||}{||\stackrel{\rightharpoonup }{O{P}_i}||},$$

(3)

where O is the center of the sphere; P_i and P’_i the landmarks on the photosphere and their counterparts on the 3D model; a, b, c the translation parameters; and φ, θ, ρ the rotation parameters around the three axes. Note that the size is fixed. The visible parts of the 3D model are then projected onto the sphere and transformed by equirectangular projection. The funeral stela #22 from Tsatsiin Ereg, decorated with symbols including many deer, is used to illustrate the projection capabilities of 3D models on photospheres (Figure 6A). The carved parts of the rock are not clearly visible to the naked eye in the original image, but they become clearer after applying an algorithm of ambient occlusion, which darkens the areas with difficult access to light (for an application in archaeology, see [44]). The aim here is to project this artificial texture obtained from the 3D model onto the photosphere. The complete projection workflow is summarized below:

• Place a set of landmarks manually on the 3D model and on the equirectangular image (if possible, otherwise use appropriate rotation and projection, as described above).
• Translate and rotate the 3D model to minimize the cost function (Equation (2)).
• Determine the part of the 3D model visible from the sphere center.
• Project the color of visible vertices onto the unit sphere (sphere center as reference).
• Perform an equirectangular projection of the sphere (Figure 6B).

The augmentation of the photosphere is undoubtedly effective for better understanding of the monument, especially for apprehending the spatial organization of symbols at the surface of the stela (Figure 6A,B; see also in Supplementary Material, Figures S5 and S6 for the original and augmented jpeg images). Note that several photospheres could have been captured to cover the other sides of the stela, allowing visitors to change their point of view, while maintaining their perception of environmental scale and space.

4. Concluding Remarks

Simple procedures are provided to project different types of materials onto 2D photospheres, whether plane (maps, orthomosaics, etc.) or 3D meshes. To the best of our knowledge, although common mathematical tools based on Procrustes registration, thin plate splines, and simple optimization are involved, such implementations were not available in popular game engines, or in packages and libraries of the main programming languages. The final equirectangular images can be used as supplementary layers over the original photospheres, and incorporated into virtual archaeological tours [45,46], or rendered individually by dedicated web viewers, or stand-alone VR browsers for Windows, Mac, and Linux, and for tablets and smartphones based on Android or iOS (Photo Sphere Viewer, Panorama Viewer, Insta360 Player, PhotoSphere Viewer). The HMDs should; however, be privileged for better immersion, almost real-life experiences, greatly enhancing spatial perception. An Android application, build using Unity and designed for Google Daydream HDMs and compatible smartphones, is provided as Supplementary Material (see Supplementary Materials). It displays the three examples described in this paper, including interaction, and presents photospheres with and without augmentation. Note that it could easily be ported to other popular systems, such as Samsung Gear VR (Facebook technologies LCC, Irvine, USA), Oculus GO (Facebook technologies LCC, Irvine, USA), Google Cardboard (Google LCC, Mountain View, USA), etc. In most virtual tours, photospheres are simply used for navigating, and the documentation is presented separately, using trigger buttons, so that the feeling of immersion, or at least of contextualization, is broken. By contrast, augmented versions of photospheres maintain this feeling, especially with HDMs. The general public can be transported to the site almost physically, with a real sense of being there, which greatly enhances the contextualization of archaeological information. Augmented photospheres may facilitate interpretations by research teams, especially for remote sites, where repeated access on demand may be impossible, due to cost, schedules, seasonal constraints, etc. In rescue archaeology, this new medium could be very valuable to preserve a visual record of interpreted remains, after their destruction. Augmented photospheres are therefore expected to be beneficial for the sustainability of archaeological information, including documentation, visualization, and interpretation, and more generally for scientific mediation. They may also be an asset for tourism and for education, by improving learning performance. Although the entire 3D modeling of a site offers extended capability, such as to build a virtual world, and to allow physical walking through it via a virtual camera, implementation in the field and later in the laboratory is complex, costly, time-consuming, and demanding in terms of storage and computing resources [6]. The acquisition of 2D photospheres does not suffer from these handicaps, as it takes only a few minutes in the field and is inexpensive. Considering the numerous advantages already mentioned above, archaeological use of photospheres should be encouraged, especially when they are augmented by the abundant documentation produced by research, all the more in that the proposed workflow can easily be reproduced for almost every archaeological site.