Mixed Reality in Multiuser Participatory Design: Case Study of the Design of the 2022 Nordic Pavilion Exhibition at the Venice Biennale

Abstract

The case study documents the design process of the physical and digital versions of the heritage-valued Nordic pavilion at the Venice Biennale. The case facilitated a multiuser collaboration in mixed reality (MR), studying the technologies’ influence upon user interactions and design decision making. Retitled as the ‘Sami Pavilion’ in tribute to the Sami artists from Norway, Sweden, and Finland, which the exhibition featured, the case study took place between 2019 and 2022, primarily during the COVID-19 pandemic. The context of the case study prompted a need to explore MR methods to overcome travel restrictions. While MR has shown some interesting utility in design research, the literature indicates the need for more concrete case work. It also was necessary to design a custom solution for multiuser collaboration. As the transferability of predictive design decisions in MR to the physical building relied upon replication between user experiences, the case embodied interesting challenges to prevalent Nordic architectural theory, particularly that of ‘genius loci’ or the ‘spirit of place’, which was a strong component of the heritage value of the building and, thus, an important design narrative. The case study documents how artworks and positions of artworks were tested in various configurations within the MR model by curators and designers to simulate the spatial experience of the design options. Several key design decisions were made based on the unique vantage points offered in MR. The MR model was then used to generate 2D technical documentation and installation instructions, which were installed on site. Studies to check the relationship between the MR model and the finished, physical result were conducted. Findings depicted a high degree of transferability between the MR model and the physical exhibition while noting discrepancies between the field of view (FOV) in the MR and physical spaces in which objects felt smaller in the real-life, built result. Possibilities and limitations for future MR implementation in the design and construction of complex projects in heritage situations are discussed in addition to implications for current architectural theory regarding place and experience from a non-dualistic perspective.

1. Introduction

Architectural design and exhibition curation rely upon dialogue between stakeholders, users, and authorities during the design process. Architectural design is a process of solving multifaceted, ill-defined problems, requiring a constant dialogue between internal representations (resulting from cognitive processes) and external representations based on the use of different media [1], and, therefore, effective communication between individuals is deemed to be the key to delivering a successful construction project [2]. Mixed reality (MR) is a developing technology that allows digital information to be integrated into physical space through smartphones, headsets, and smart glasses, allowing for new forms of hybrid communication. Current scholarship claims there are benefits to introducing MR to design and planning, but also identifies the need to determine the best practice of MR technologies in design curricula [3,4,5]. Following this prompt, the team earlier documented the use of MR in architecture and design education [6] and architectural heritage, leading to virtual models of several sites in Venice, Italy, including the Nordic pavilion [7]. The team also researched the application of MR in various design and planning stages such as competitions, planning, and construction [8]. The case study in this paper arose from a continuation of this research and previous findings in addition to the practical necessities of stakeholders working on the Venice Biennale of Art 2021 (delayed to 2022 due to the COVID-19 pandemic) who expressed the need to facilitate a design process within an MR model.

The work of Norwegian architect Sverre Fehn, and, in particular, the Nordic pavilion in Venice (1962), plays a canonical role in representing the shift from international style modernism to the late modernist or phenomenological tradition in Nordic architecture. This shift was in response to what was perceived as a lack of contextual awareness in international style modernist architecture. Here, Nordic architects within the 1960s context—particularly within the Oslo School of Architecture and among scholars such as Christian Norberg-Schultz [9]—began to view architecture as a human response to problems coming from the environment. Thus, architecture was tasked with ‘milieu creating’ through ordering and improving relations within the external environment (ibid). The Nordic pavilion is exemplary within this canon due to several design decisions concerning the relationship between the architecture and the local site within the Giardini in Venice, particularly the decisions relating to the 13 trees on the site, which Fehn determined to be of particular importance. Fehn’s design decisions involved, therefore, a grid roof structure which allowed for the trees to pass through the roof in a natural manner, uninterrupted by the building. This created a particularly vulnerable relationship with the exterior environment because the building is continuously exposed to the elements, and several trees have died.

Due to its unique architectural features and its prominent location, the Nordic pavilion is one of the most famous buildings in the Nordic late modernist canon and one of the foremost venues for displaying Nordic art and architecture in the world. The Venice Biennale of Art is arguably the most prestigious event in the international visual arts calendar [10], and, as a preeminent, international exhibition, the Biennale is considered a significant force in the globalization of the arts through its increased interconnectedness and the flow of people between nations and cultures [11]. The Biennale exhibition campus is divided into group exhibitions and individual national exhibitions. One of the most significant changes to the artistic programming in the Biennale in recent decades is the tendency for national representations to challenge the ‘symbolic and historical connotations of national pavilions’ [12]. For the Giardini section of the exhibition, countries exhibit representative artists inside freestanding, individual, national pavilions placed throughout the area. The national pavilions, such as the Nordic pavilion, within the Giardini area feature a selection of artists from the pavilion host countries.

High-end arts installations compete at an international level to deliver impactful visitor experiences, so it is natural to value any process which can increase quality and alleviate additional risk in design and project management. It is important to note that many construction project teams are not fully capable of delivering projects to the client in a reliable way [13], which is one of the driving factors of the increasing digitization of the construction industry. Additionally, when arts installations are placed within heritage buildings such as the Nordic pavilion, complex planning and approval processes can be required to ensure that heritage sites are protected. Heritage value, as identified within the Burra Charter, states that the cultural significance of a place should be protected for present and future generations. The pavilions at the Biennale are often designed by famous architects and can have heritage value and are, thus, forced to undergo significant checks and regulations concerning alterations and installment of structural or spatial works. This creates a need for detailed design processes because buildings which can have heritage value are often protected legally from alterations. Installing within heritage buildings is, thus, a complex task which requires planning and communication across disciplines to deliver a project with the intended artistic and curatorial vision alongside the respective heritage approvals.

For the 2021/2 curatorial year, Norway was assigned responsibility for the curation of the Nordic exhibition, which was conducted by the Office of Contemporary Art (OCA). The curatorial concept focused on Sami artists, the indigenous people from northern Norway, Sweden, and Finland (Figure 1). Large artworks were commissioned for the exhibition which needed to be installed within the heritage protected building. For the project, the curatorial team decided they wanted to simulate and design the exhibition in MR initially due to the COVID-19 pandemic, which hindered international travel. Upon discussing the project, the team formatted the research question of whether, in addition to allowing digital site visits in MR, MR would influence design decisions for the exhibition and whether MR would help the stakeholder group understand the artists’ works and curatorial intentions in a more precise manner than traditional drawings and images. The team was then engaged to conduct this research and to help with the planning of the pavilion and its exhibition. Thus, the combination of the stakeholder needs for the Venice Biennale alongside the research theme created the following research question:

How can the research team facilitate a process for multiuser collaboration in the design of art installation within a heritage building in MR, and how will MR influence interactions and design decision making?

The research question prompted a literature review to understand the current state of similar research concerning collaboration and participation in MR design and to understand some of the historical precedent concerning the Nordic pavilion, its context within late modernism and phenomenology, and how this may apply to the history and the state of the art of MR. The team was particularly interested in understanding manners in which MR can be applied to design to better understand design decisions, user feedback, and facilitate participation among stakeholders. Eventually, the team sought to understand how various methods may help gain insight into the relationship between virtual and physical spatial experiences and forms of collaboration. The literature review also helped us frame the required use of several novel methods and techniques to better understand the current state of the art for spatial exhibition design and placement of artworks in MR. Because it was decided that it was necessary to test the curatorial concept inside MR with multiple users from the curatorial team, it was also important to understand the theory, methodology, and current state of the art of multiuser MR.

MR emerged from the cybernetics culture of the 1960s, particularly from the 1965 IFIP Congress in which Sutherland [14] challenged developers of computer graphics to look beyond ‘the picture in the window’ toward creating virtual environments in which ‘the viewer is immersed, and where he sees, feels, and manipulates the virtual objects as if they were real’. This challenge was taken up by Brooks [15,16], who viewed interactive computer graphic simulation as a ‘potent tool... in man’s on-going scientific enterprise in the understanding of the physical universe’, developing interactive 3D graphical simulations to represent the interior of buildings as ‘a tool for an architect and his client to use for rapid prototyping of buildings by visually walking through them to refine specifications’. Later, Airey et al. [17] built a see-through HMD for various architectural applications. To classify data flows of mediated experience between the world, models, and humans, Robinett [18] developed a taxonomy of ‘synthetic experiences’ (Figure 2) which outline the way virtual environments facilitate a feedback loop between human experience and the world, implying that a transmitted experience allows humans to modify the world through a virtual model. Notice, here, the difference between a simulated experience, which entails no action upon the world, and a transmitted experience, which implies human action upon the world through the model.

From this understanding of MR as a model for both simulated and transmitted experience, the architecture industry has also been transitioning from the traditional use of hand sketches to mediated design tools [19], including BIM [20] and virtual reality immersive experiences [21]. Collaborative MR in design is claimed to enhance user feedback and experience to support communication [22,23,24,25], and recent research offers detailed accounts of how design teams have engaged in viewing, evaluating, and discussing designs [26,27]. For example, Billinghurst et al. [28] tested an adaptive avatar for enhancing MR remote collaboration between a local augmented reality user and a remote VR user. By 2018, a Billinghurst lab project, SharedSphere [29], had investigated how to create live, panorama-based collaborative experiences. These recent cases gave us reason to believe that creating a multiuser MR case study was possible and was worth giving a try, while also demonstrating that the transition from a simulated experience to a transmitted experience, as would be required to see if design decisions in MR impact a physical, built result, required the development of tailored materials and methods for the case study.

An interesting thread within neuroscience research documents a range of studies on the transmissibility between physical user experience and MR user experience. In 2012, MR was tested in several spatial navigation studies in which all subjects described immersive experiences, which were confirmed through fMRI monitoring the neuronal activation of the parahippocampal areas, cuneus, and occipital regions [30]. In 2014, Clemente Bellido et al. [31] found proof that users can feel presence similar to physical space inside a MR environment using fMRI, speculating that this could lead to interactions between computer-generated environments and humans in a natural manner as the technology becomes more ‘invisible’ to the user. While providing reason to assume some experiential transmissibility between a digital MR model and a constructed project, the data from such studies also seemed to conflict with the predominant architectural theory regarding the embedded nature of architectural experience within the specific materiality and location of objects and places such as that articulated by Norberg-Schultz [9]. This was important because genius loci or ‘spirit of place’ has been the dominant theoretical lens through which the experience of the Nordic pavilion has been described since its construction. This range of scholarship therefore made it very important to incorporate methods to study the transmissibility of user experience between the physical pavilion, the digital MR model, and its design processes, and then between those and the physical built result, and, additionally, to monitor how the implications of any data influence theory regarding the late modernist Nordic architectural canon. This prompted the secondary research question:

How will findings from the case study influence existing theory on user experience of architectural space in relation to the late modernist Nordic architectural canon?

2. Materials and Methods

To study the use of MR in comparison to a typical design and construction process, the case study research approach followed a typical design review process. The Otto et al. 2005 study [32] provided a good template for comparing the use of MR in design and construction to a typical design review process. Design review is the process of milestones within a design process whereby a design is evaluated against its requirements to verify the outcomes of previous activities and identify issues before commitment to construction. Involving clients, practitioners from other project disciplines, and end users with diverse knowledge backgrounds in the design review process relies on their ability to accurately visualize and understand information and provide meaningful feedback [33]. Here, research has documented difficulties in groups working together effectively with a common understanding of needs and requirements [34] and a lack of communication among key stakeholders as potential causes [35]. However, if requirements and construction prerequisites can be understood in the early phase of a project among the major stakeholders, changes can be significantly reduced with an improved level of trust in project relationships [36].

As noted in the introduction, design review was made more difficult during the COVID-19 pandemic, when restrictions on travel made traditional physical travel and site visits often impossible. At the construction stages, MR has been seen to reduce redesign work and construction delays, reviewing the installation sequence, facilitating concurrent engineering processes, and increasing the quality of the product [37]. Recent discourse has highlighted the capacity to simulate spatial experiences in MR among project stakeholders and users to facilitate evidence-based design decisions [38]. Here, studies have explored methods for distributed, web-based design development and design reviews in shared MR-based environments [39,40]. Additionally, in 2013, a study which utilized VR in architectural design studios during the design process in the stage of proposing and exploring the structural system found few technical problems, suggesting that the technology has overcome initial shortcomings [41]. More recently Lisio and Manuch [42] designed a collaborative, local multiplayer virtual reality game, and Oh et al. [43] designed a multiuser interactive simulation using AR glasses. There have been few studies involving the Venice Biennale directly, but a Jung et al. [44] study of an arts exhibition setting found that more authentic VR and AR environments, in which visitors can be fully immersed, require high quality of resolution.

Based on the literature review, the team realized that it was necessary to prototype a custom design solution for the case study. For researching MR applications in complex design processes such as arts installations in heritage buildings, it is important to understand whether multiuser communications and interactions are possible in MR. Most studies reviewed focused on single-person experiences, and some research highlighted multiplayer use of MR. Therefore, the team looked at tools that could facilitate multiplayer participation in MR. MR software choices were somewhat bound to the ecosystems of the VR headset, for example, the way the HTC Vive headset uses Steam VR software. It was also found that Unity or Unreal technology was the basis for most virtual reality and augmented reality experiences. As there is little established research on how to perform multiplayer MR, the team performed a taxonomy of available methods for setting up the case study. Particularly, the Pietroszek and Moore [45] (

Table 1. AHMED toolset recommendations Source: Pietroszek and Moore, 2019 [45].

Step	Description	Tools and Methods
1	Reconstruction of exhibit artifacts	Microsoft Kinect
2	Volumetric recording	Microsoft Kinect
3	Design and building of the exhibit	Magic Leap

) AHMED toolset recommends a workflow consisting of three phases: reconstruction of exhibit artifacts; volumetric recording of a narrator; and design and building of the exhibit.

As the artworks varied in size, materiality, technical solutions, sound, and lighting needs, in order to curate the works together it was decided that a full 3D model would be created and simulated in various iterations. Based on this information it was assumed that it would be required to develop custom software. However, in the summer of 2021, the alpha version of the platform Arkio was released for testing, and we were able to secure a test license through dialogue with the developers. Arkio is marketed as “a collaborative design tool for architecture that uses virtual reality and can be connected with several Oculus Quest 2 headsets to create a multiplayer simulation”, stating that it allows users to ‘sketch and review designs in VR, AR and on mobile devices then export to Revit or other 3D tools’ while allowing ‘real-time revisions to 3D designs live in collaboration with others’. This was followed by testing a promising version of the software on Oculus Quest with both Oculus Touch controllers and controller-free hand tracking. The interface was very interesting, with tools that snap and slide for careful sizing of walls and buildings, and the team could easily view models at full 1:1 scale to get a look from a specific perspective. Based on this testing, the team expanded on the AHMED framework by adding the multiplayer design sessions as a feedback loop between the design stages (

Table 2. Case study steps.

Step	Description	Tools and Methods
1	Scan of point cloud model of the Nordic pavilion	BLK360 Point cloud scanner, iPad Pro
2	Converting point cloud to mesh and decimating points	Autodesk Recap and Cintoo
3	Importing model into Arkio	Rhino, Arkio
4	Design review: Placing artworks and implementing any changes into model with curator instruction	Autodesk Recap, Photoshop, Arkio
5	Facilitating MR multiplayer workshops	Arkio, Oculus Quest 2
6	Feedback from participants, return to step 4 as needed	Go-along interviews, observation
7	Construction drawings: Translating model into construction and installation drawings for permit	Arkio, Rhino, Photoshop
8	Physical construction	Physical tools and drawings
9	Comparative analysis	On-site interviews, go-along interviews, observation

, Figure 3).

For the workshops with the curator and owner teams, the team needed to find methods to incorporate their feedback into the design process in MR while also collecting observational data. Participatory design allows designers to better assess client constraints and protect the interests of end users [46]. As users tend to evaluate projects differently to designers, it is often important to ask project occupants at any given stage about how a design meets their needs, so the team decided to incorporate interviews with the users during the design stages. Recent research suggests that designers create more innovative concepts and ideas when working within a co-design environment with others than they do when creating ideas on their own. Steen et al. [47], documenting a study between a user-centered group and participatory design group, found that the participatory design group was able to think more systematically about potential solutions and generate proposals that were more holistic. Their findings suggest that participatory design offers benefits as a process for idea generation and a more holistic perspective on the problem and potential solutions. As it was stated that ‘close collaboration between in-house professionals and lead users appears to facilitate the exploration of latent user needs allowing the development of novel design concepts for possible future use situations’, it was found that it was important to monitor how MR affected the design review process and, ultimately, the result.

3. Results

The case study followed the steps described in Table 2 and Figure 3. The results were based on interviews and observational data, including screen recording and go-along interviews, describing how comparative analysis between the MR model and the physical, built result were experienced by users. We also collected several photos and films from the studies and received photos and descriptions during the process from collaborators. Here, it is perhaps noteworthy that, due to the mixed methods approach, and particularly the use of screen recording and observations, the case study generated a huge amount of data. This correlates with a 2018 study [48], which noted the strengths of such methods while noting that one must strike a balance in terms of the volume of data and the unit of analysis and the existence of the usual technical challenges that emerge when one works with technology. Based on these data, this text first provides a brief overview of the case study steps. Following the overview of the case study steps, the results are discussed in more detail.

3.1. Step 1. Scanning of Pavilion (Figure 4)

The first step of the case was the initial 3D scanning of the pavilion. This was conducted in 2019, when the team was given access to the site on the Giardini of Venice, Italy. The team used this opportunity to 3D scan the entirety of the interior and exterior of the Nordic pavilion building using the Leica BLK 360 Lidar scanner and several iPad Pro tablets. This process was carried out over two days and consisted of 29 individual scans. Each scan took approximately 10 min for the scan itself and an additional 5 to 15 min to transfer data to the iPad Pro, select a new scan position, and initiate the next scan sequence. The use of 29 scans was necessary due to the geometry of the building, especially the two sets of roof beams, as seen in Figure 4. To collect scan data from both sides of each roof beam, many interior scans needed to be performed. Additionally, the geometric complexity of the trees, both indoor and outdoor, and the relationship of the pavilion to the exterior environment, required many scans so that the building and its exterior would be one continuous model and so that users in MR could walk in and out of the building, approaching it from the Giardini main avenue from the west or from the US pavilion to the north.

It is important to note that site access was secured through the authorities of the Biennale during a time in which access to the site was closed to the public. This is a very important aspect because, in 3D scanning technology, any persons within a scanning area are captured within the scan, creating ghostlike figures which need to be manually deleted afterwards. Therefore, clearing out as many people as possible is essential to securing useful data. Note the importance in understanding both the 3D scanning hardware and software for securing useful data for future point cloud assembly to MR and the need to deviate from the standard methodology of using printed scanning targets when operating within a heritage context where targets would clutter the result. In post-scan processing, it is necessary to match individual scans to each other, meaning that it becomes necessary to define alternative cross-referencing positions through scan targets, such as exit signs and other prominent features, in advance when choosing scan positions.

Figure 4. Case study step 1. Three-dimensional scanning the pavilion site. Source: Author.

Figure 4. Case study step 1. Three-dimensional scanning the pavilion site. Source: Author.

3.2. Step 2. Model Conversion (Figure 5)

The 3D scans from the Venice trip were brought back to Oslo. They were then collated into a single model within the software Autodesk Recap, which was utilized for creating a point cloud model. Upon testing the raw points in MR, the team quickly found that this initial model was too heavy to be used in MR with the hardware available. For this reason, portions of the model were decimated to a lower point count using the point cloud software Cintoo. Some especially complex components of the point cloud were also 3D modelled manually into 3D mesh models using Cintoo. An important point to consider is the amount of point decimation necessary to achieve the appropriate balance between resolution and framerate in MR. Multiple variations of point clouds were tested to achieve the optimal balance between the density of points and framerate. The team found that 4 mm spacing between points provided the optimal balance between headset framerate and model realism. The point cloud was then placed as the background for the MR model.

Figure 5. Case study step 2. Point cloud model of the site in MR after point cloud decimation to 4 mm point spacing. Source: Author.

Figure 5. Case study step 2. Point cloud model of the site in MR after point cloud decimation to 4 mm point spacing. Source: Author.

3.3. Step 3. Artwork Scans (Figure 6)

The third step consisted of collecting individual artworks from the three artists for the exhibition. This consisted of images, 3D models, 3D model instructions, films, sketches, and other mixed media which were imported and converted to be interactive models within MR. This component of the case required a variety of modelling techniques in Rhino for meshes and textures to function correctly. All of these models were then imported into the Arkio MR scene and tested for functionality within the team. A series of multiuser MR sessions in Arkio were conducted internally to make sure the sessions functioned properly between multiple users. The team found that it was important to have a stable connection hosted by an external PC that opened the session and imported the models. After the session was hosted, users could be invited into the session from multiple MR devices. The team used Oculus Quest 2 headsets for the testing and decided that, based on cost, availability, and resolution, it was the best-suited headset for the multiuser design reviews.

Figure 6. Case study step 3. Placing scanned models within the site in mixed reality. Source: Author.

Figure 6. Case study step 3. Placing scanned models within the site in mixed reality. Source: Author.

3.4. Step 4. Design Review (Figure 7)

After preparations up to step 3, the model was presented to the curatorial team and the artist teams to design the exhibition. In the first four workshop sessions, artworks were placed in various positions to test the curatorial and artistic vision from a first-person perspective through screenshots and walkthroughs in the model. Each session lasted about one hour. There were normally three to six people present in each workshop. These participants were curators and exhibition designers from the team. Artworks and positions of artworks were tested in various configurations within the model to simulate the spatial experience. The curatorial team expressed excitement and interest in the ability to see the exhibition space in such a medium and noted that it was easier to make decisions with the curation inside of the MR model. Following each session, a range of suggestions and amendments were proposed over email or text messages. These suggested changes were then prepared for the next session. Other amendments were made during the workshops while moving elements such as artworks and furniture around inside the model in real time.

Figure 7. Conducting design review in mixed reality. Source: Author.

Figure 7. Conducting design review in mixed reality. Source: Author.

3.5. Step 5. Multiplayer Workshops (Figure 8)

After the initial sessions, the curatorial team went inside the multiplayer model in MR to obtain a better understanding of the model and to generate feedback. To facilitate this, the team used three Oculus Quest 2 headsets connected via the hosted Arkio session inside a studio space inside the Oslo School of Architecture and Design. Once inside the common MR model, participants tested the artwork models and moved them interactively. Participants communicated with each other in the model and viewed various placements and options for artworks. The team found, in general, that testing in MR first allowed for several vantage points to be better understood by curators and artists and had a significant impact on the final design. Without the process, it is difficult to see the final design ending up like it did, because the team witnessed several key decisions being performed during the workshop sessions which influenced the final design. For example, key artworks were moved to entirely new positions due to vantage points within the virtual model, and artworks were moved from wall positions to the floor in response to view corridors.

Figure 8. Conducting design review in mixed reality. Source: Author.

Figure 8. Conducting design review in mixed reality. Source: Author.

3.6. Step 6. Feedback from Participants (Figure 9 and Figure 10)

Sessions within the MR environment generated feedback which was documented through interviews, go-along interviews in MR, and observations, including screen recording. The feedback regarded placement of artworks, view corridors between the Giardini site, the building, and artworks, mounting options between the artworks and the building, etc. This feedback was then used to generate alternative proposals for step 4, which was then reviewed again. It is important here to note that the design reviews were highly confidential due to the nature of the exhibition and the artworks. Using the ecosystems of Arkio allowed for anonymity of data, and password protection allowed for sessions to be securely private. However, Oculus (Meta) ecosystems especially created privacy difficulties. Due to Facebook/Meta requiring utilization of your personal social media profile, it was nearly impossible to secure anonymity and privacy of the data during the process. Thankfully, the team secured the required contractual agreements to process personal data for the case study, but, without such an agreement, any research would have been very difficult. It is unclear how this ecosystem is supposed to be used in a research format with requirements for anonymity other than through making fake accounts, which is against the TOS (Terms of Service) for this type of private MR infrastructure.

In general, the team also found the need to be patient when setting up sessions and reviewing participants’ feedback. This is because multiuser MR is in alpha/beta testing, signaling that such a process is potentially becoming more mainstream but is still outside of any standard practice. In the current state of the art, these types of applications require hands-on bug fixing and high technical competence in addition to dialogue with application developers to understand use potentials and to communicate bug fixes. The team believes that future case studies need to consider the amount of technical competence involved in each individual step of a case before recommending similar cases be performed primarily due to the current instability of the software and hardware not allowing for any straightforward process during the feedback and design review stages.

Figure 9. Conducting design review in MR inside a studio at the Oslo School of Architecture and Design, Norway. Source: Author.

Figure 9. Conducting design review in MR inside a studio at the Oslo School of Architecture and Design, Norway. Source: Author.

Figure 10. Finalizing artwork placement in MR. Source: Author.

Figure 10. Finalizing artwork placement in MR. Source: Author.

3.7. Step 7. Construction Drawings (Figure 11)

Steps 4 to 6 were iterated over many months, with decisions being made between artists, curators, and installers. Once final placement and design were determined, a set of construction drawings were needed to be applied for the technical installation on the physical site. The MR model was then used to generate documentation and installation instructions, which were submitted to heritage and Biennale authorities. Following approval, the drawings were sent to builders and fabricators. Notice here that, while an experimental case study can use new media such as MR and 3D modelling, the industry in general still relies on two-dimensional drawings and media to communicate and apply for permitting and approval. Thus, to transfer the MR model to a physical installation, the team still needed to make 2D drawings for the approval processes. This meant the team had to work backwards from MR to construction drawings. This made for more rather than less work, which appeared counterintuitive at first.

It is worth considering how the MR sessions were transferred to 2D documentation. The team found it easiest to export the 3D model after a workshop session and to generate 2D drawings based on this model. One can also imagine a scenario in which the MR session is integrated into an approval process by inviting authorities into the MR model. While this seemed unfeasible for the case, it could be an interesting aspect to consider for a future case or when considering MR adoption into design and construction more broadly.

Figure 11. Drawing creation from the MR model for site install. Source: Author.

Figure 11. Drawing creation from the MR model for site install. Source: Author.

3.8. Step 8. Physical Construction

Based on the technical construction drawings, the exhibition was constructed on site. This component of the case study was conducted outside of the research scope.

3.9. Step 9. Comparative Analysis

With the physical exhibition in place (Figure 12), the team conducted interviews and observations to compare the relationship between the MR model and the finished physical result. Many of the same participants from the MR sessions in Oslo were present in Venice. The data between the multiuser MR sessions were compared to the physical built output, as seen in Figure 5, to understand the relationship between the digital and physical exhibition spaces. The comparative analysis offered interesting insights. Broadly, participants reported a high degree of correlation between the virtual and physical, built result. This was exciting because the team was unsure how the transferability between the digital and physical models would be experienced. However, the field of view (FOV) in MR was reported to make both spaces and art objects slightly larger, as users reported broadly that objects “felt” smaller in the built result. This created some discrepancy between expectations and the resulting exhibition. The team recommends further studies of FOV in MR headsets when comparing physical to virtual spaces. At least for the Oculus Quest 2 headset, it seems the FOV may be narrower than in real life, which created a first-person scale difference. This could also have been due to the camera angle, lens width, or other aspects between software and hardware. However, it seems noteworthy to report that the experience of the physical space felt smaller than in MR for reasons the team believes has something to do with the FOV in MR.

More interesting findings regard the case study’s relationship with prevalent theories of genius loci, or “spirit of place”, as described in the introduction, which are a strong component of the architectural tradition of the building and its heritage value. This theory claims, in broad terms, that the phenomena of architectural experiences are mediated through the objects and places themselves in a specific place and then experienced by humans. In the findings section, it is discussed how it is possible to experience these phenomena regardless of the actual objects and places because, in short, they can be experienced in a digital MR simulation in a completely different location—for example, a scanned version of a pavilion in Italy being experienced through MR in Oslo or Geneva and even in multiple locations at the same time over a multiuser session. This aspect creates some interesting points of discussion regarding the state of architectural research in relation to the sciences and its current discourse which are discussed in the following section.

4. Discussion

The case investigated how new technological media and forms of visualization and simulation influence the architectural and arts installation design process. The literature review highlighted how, in the design process, MR continues to be used most widely for design review in the later phases. This is slowly changing, and the team believes that researchers can expect collaborative design to become a more prevalent use for MR as the link between design authoring applications and MR becomes more direct, reliable, and flexible, allowing for a bi-directional flow of information. The team found that involving clients, practitioners from other project disciplines, and end users with diverse knowledge backgrounds in the design review process relies on their ability to accurately visualize and understand information and provide meaningful feedback. The study found correlating data to Otto et al. [32] and White and Nikolic [37], who both noted that designers need to exercise judgment when deciding which elements are critical to include in MR models to both ensure useful feedback and maintain the level of control they have with standard rendered images or animated walkthroughs.

By analyzing the first diagram of Robinett’s [18] ‘technologically mediated experience’ taxonomies (Figure 2 and Figure 13) it can be seen that technological mediation is placed between direct perception and action between humans and the world. The mediation is then facilitated between the world through sensors and humans through display systems. Humans then influence the world through actions imposed upon the technological mediation system, which, in turn, performs actions upon the world. Note (Figure 13) that, where a recorded or simulated experience merely instantiates human participation with a technologically mediated model, either with or without active input from the real world, a transmitted experience instantiates the active influence of humans upon the world through a technologically mediated model. It is, therefore, important to understand the conceptual roots of MR and technologically mediated experience not only as visualization technology, but as a way humans enact real changes upon the real world as a transmitted experience.

Comparing this idea to the review of the phenomenological tradition of Nordic architecture and the history of the Nordic pavilion, some interesting points of discussion emerge. As was seen in the literature review, the theory of architectural phenomena and experience, and, later, of genius loci or ‘spirit of place’, has met some recent criticism. Perez-Gomez [49] points out how genius loci rely upon a dualistic philosophical approach which posits a difference, or separation, between the external and internal world. Particularly, the theory relies on the idea that places and objects ‘themselves’ mediate phenomena from their objective world, which, through the human sensory apparatus, are experienced by the individual.

As the team began to point out in step 9 of the Results section, the case findings asked us to explain the way the digital MR model of a building in Italy, when removed from its ‘real’ location in physical space, could produce similar phenomenological effects somewhere else in the world, in this case, in Oslo. Based on the findings in the case, it appears timely to speculate on whether it is not more plausible to posit that the experience of architectural space, and space in general, is rather non-dualistic, i.e., that the experience of architectural phenomena is found solely within the human brain rather than embedded within the material of an object or place. In such a scenario, the experience of objects and places consists of properties solely within the human sensory apparatus in contact with any environment, be it physical, digital, a dream, and so on.

In the literature review, it was found that most state-of-the-art MR research was performed outside of the Nordic context. If one takes a step back to consider the context of the Nordic pavilion and the tradition of the ‘Oslo School’, it is important to note a few aspects both practically and theoretically which help us to understand some of these theoretical aspects further. The interweaving of built examples such as the Nordic pavilion with practical architectural theory related to a Nordic theoretical canon created a somewhat holistic approach to architectural education, research, and practice which has held relevance up to this day. For this reason, researching within this context provides an opportunity to explore and build upon this model, but also creates opportunity for new discovery when performing novel case work. Within this tradition, a specific emphasis on ‘place’ draws on a variety of sources spanning phenomenology, perceptual psychology, and architectural history which seek to establish a connection between a perceptual or phenomenological understanding of architecture (i.e., that of human experience) and that of an objective or noumenal world. This stance led eventually to the formulation of the theory of ‘genius loci’ which claims that objects, and particularly places, ‘mediate’ certain phenomena which can thus be experienced through human sensory faculties [50].

It is important to note that, in recent discourse, the idea of genius loci has met criticism due to various inconsistencies in the theory. The historical background of the shift in interest to place in the postwar period was based on a critique of modernism’s focus on abstract spaces and, thus, involved a critique of modernism as an architectural ideology. As can be seen in Norberg-Schultz’s study [50], the primary empirical observation to support the theory was the rough sketching of various landscapes, such as Northern Europe’s ‘romantic landscape’, the desert region’s ‘cosmic landscape’, and the ‘classical’ Mediterranean landscape, which are implied to embody location-specific experiential phenomena. Harvey [51] saw the use of place in this manner primarily as a way to create ‘authenticity’ as a form of commodification which was driven paradoxically by global market interests rather than any specific spatial properties. Here, Hvattum [52] pointed out that ‘place’ was understood as a qualitatively ‘charged’ dimension which posited that various ‘places’ had specific qualities based on naturally given conditions and that this led to both a form of regional determinism, i.e., that specific places embody specific architectural phenomena, and material fetishization, i.e., that materials ‘in and of themselves’ have specific phenomenological qualities. Hvattum argued that these two aspects of Norberg-Schultz’ theories function as a ‘tyranny’ in contemporary architecture.

How, thus, to interpret the data from the case beyond a critique of genius loci, regional determinism, and material fetishization? Work being produced at the intersection of architecture, philosophy, and neuroscience has begun to construct sophisticated and empirically grounded facts about the human experience of place, and Edelstein and Macagno [53] state that neuroscience and architecture should be seen as correlated disciplines in understanding spatial experience and the influence of environments upon humans. The data reported within recent fMRI (functional magnetic resonance imaging) and MR lab work have been used by neuroscientists as methods to further develop a scientific understanding of spatial experience [54]. In this work, spatial phenomena are reported to be experienced in the participants’ brains whilst they are inside of a MR simulation in an enclosed fMRI ‘tunnel’, navigating through MR space whilst being recorded by fMRI technology.

For example, with the use of VR and fMRI, Kim and Maguire [55] found 3D, grid-like signals in the human entorhinal cortex of the hippocampus, which was seen as evidence for 3D grid codes in the human brain. With the help of brain imaging, researchers can see that the way humans navigate in the world relies upon the use of the same parts of the brain, regardless of whether this world is physical or digital. The initial data from such studies thus find that the brain regions which are utilized to navigate within MR space are the same as those utilized in physical space. Thus, the experience of spatial phenomena needs not be based on physical materials in a strict sense; it needs merely to represent physicality in a manner which is interpreted by the participant. As Garfield [56] puts it, within the school of ‘mind only’ philosophy, as far as the brain is concerned, there is only sensory experience. This can help us understand how a place can ‘feel’ like it does, i.e., exhibit the experiential phenomena it may inhabit, without having to be bound to any specific regionality or materiality, as also pointed out by Hvattum [52].

Regardless of these philosophical matters, the real-world implications of these methods seem to be that MR can offer the ability to make predictive design decisions based on phenomenological data before construction. This assumes of course that such MR design review models can be standardized to some degree and implemented in a coherent and systematic manner in the design process. Having noted how curating exhibitions requires the ability to visualize various separate works into a compositional whole, note, again, that White and Nikolic [37] argue that MR can be used to coordinate the work of different professionals involved in project construction and to support the shared understanding of project information through visualizing the underlying engineering data and enabling users to visualize spatial, temporal, or other information about the designed product in a given context. This can lead users to see MR as a potential tool to reduce risks, increase technological innovation, and improve business practice.

5. Conclusions

Returning to the research questions for the case: what can be said of how to facilitate a process for multiuser collaboration in the design of art installation within a heritage building in MR, and how does MR influence interactions and design decisions between artists, stakeholders, and authorities? Additionally, how has the case influenced thinking on existing theory regarding the user experience of architectural space? In the conclusions chapter, the primary and secondary research question are shown to be interrelated.

In the results section, the case study steps were displayed, and the workflow was provided. The team recommends similar cases attempt to follow the same basic diagram and to modify the choice of hardware and software to the research budget, keeping in mind that technology is likely to make significant progress in the future, making workflows easier and more predictable. It is recommended that some time is spent investigating the relationship between the density of the point cloud model and the intended degree of realism in relation to the MR headset. Here, it is likely that, in future iterations, it will be possible to live render MR directly within a full-resolution point cloud without a need for the decimation of points, and this holds very exciting possibilities for the future. The team found, for this case, that a decimation of 4 mm between points provided the optimal balance between realism in MR and hardware capacity to render points in real time. Regarding multiuser collaboration, the use of Arkio software provided unprecedented ability to connect multiple users within the same MR in real time. Such capabilities are likely to continue to be developed, with promising avenues for continued research in the relationship between multiuser MR and collaborative design across areas of expertise in the design review process.

Importantly, the team documented several moments where concrete design decisions were made which embodied significant changes to the initial design within the MR model. These decisions seemed to be uniquely facilitated by the MR technology and the workshop setup, particularly since the decisions were made within the MR model from vantage points that were not easily determined without the use of MR. This regards the main artwork especially, which was studied from the vantage point of the approach from Giardini Avenue at the intersection point between corner of the Danish pavilion and the view corridor to the artwork. This correlated to how White and Nikolic [35] documented that clients often make different choices when they see the impact of their decisions and tend to reject lower–cost solutions when they can visualize them, as the difference in quality is seen between proposals.

The case relied upon methods such as observation and interviews to determine how the decision-making process was influenced by MR. While expert interviews allowed for specific insights into the research questions, and interviews with research participants helped us to understand the user experience, these interview techniques have little specificity to the technicalities of MR. Here, the team found that go-along interviewing was an effective tool for eliciting contextual, phenomenological accounts of virtual environments, providing a contextualized understanding of a participant’s experience which correlated with Vindenes and Wassen’s pioneering work on this methodology [57]. Being in MR when conducting interviews enabled participants to demonstrate interactions spontaneously, and by providing a guided tour, showcased relevant design suggestions and view corridors within the space and the exterior environment.

Participatory design seems to be a popular, contemporary topic. For architects and designers, their role in such a case study becomes facilitation of a process rather than proposing a design. This requires different skill sets such as user research competence and the ability to make observations. Sanders and Stamper [58] argue that the implications of this shift towards participatory design for the education of designers and researchers are enormous and are changing the landscape of design practice as well, creating new domains of collective creativity. The team notes that using MR in a participatory design process does have many of these effects, and the process for the case study did create moments of “collective creativity”, as was suggested in the literature. However, not everyone has a MR headset. Democratic participation and technical standardization are still far from optimal, and technology investments are critical.

Based on the case study and similar research, the team sees promise in MR technology for participatory design for a variety of reasons, including risk alleviation, better planning, the use of various perspectives and expertise in decision making, and the control and execution of design and construction processes. This is primarily due to the way the MR process can be used to predict phenomenological experiences on behalf of the human participants and the way this theory can be integrated into similar design processes. As noted in the discussion section, contemporary architectural theory regarding the spirit of place has critiqued the notion of regional determinism and material fetishization in this canon. In addition, recent neuroscience experiments have provided data to help situate phenomenological theory within MR and fMRI techniques. These experiments provided earlier evidence in support of a non-dualistic understanding of architectural phenomenology, i.e., that physical and digital MR spaces create spatial experience in similar regions of the human brain rather than physical experience having a separate epistemology. However, it is also important to note that the activation of certain areas of the brain does not necessarily imply stimulation of the sense of presence [59], and that spatial orientation and navigation rely to a large extent on locomotion and its accompanying activation of motor, vestibular, and proprioceptive system [60], and, therefore, there is still plenty of work to do in studying the relationship between fMRI data, MR, and navigation in physical space.

Following on from this discussion, it is notable how MR opens interesting avenues of philosophical, and particularly phenomenological, considerations regarding the human experience of spatial phenomena and the methods designers and architects employ when working towards design goals. While digitization certainly offers a variety of new challenges to the design professions, it may also open avenues to challenge predominant theories and methods concerning the way researchers understand human interaction in the world and how architecture may better life and the environment. With future research, the team hopes to further develop inquiry into how architecture is experienced, practiced, and discussed across laymen, practitioners, and researchers within a variety of disciplines related to human experience and the built environment. It seems that digital technologies such as MR, when used within architecture to predict some of its experience in the real, physical world, show that the digital/physical and mind/body divides may not be as divided as one’s intuition may assume.

To an extent, it is possible to speculate on MR adoption into the AEC industry on a large scale, however, based on the case, it is possible to have doubts about the implementation MR in a holistic system of methods or regulations anytime soon. For example, when encountering the public planning and heritage approval system, the process was still cemented in a 2D, drawing-based process with no requirements for participatory predictive design review data or even a 3D model. Whereas the incentives may be in place for a more digital and predictive design review process, the ecosystems and infrastructure necessary to develop MR into fully fledged systems would likely require resources and competence far outside of the realm of the current architecture context. Furthermore, the coordination of such an effort would require substantial shifts in methodology with an emphasis on participation, quantification, and data-driven decision making.

Innovation is moving quickly, and the interesting and groundbreaking cases on the use of MR in architecture and arts exhibitions in recent years are certainly impressive. At the same time, the gap between MR technology development and the design and planning of the physical environment articulates a situation in which two closely related fields of inquiry are at risk of separation. For the team, this highlights the importance of the broad inclusion of users in both planning and technology development moving forward, but also the ability for practitioners to work on the fly with developing technology in piloting new use cases and developing theory and methodology to help understand the data and to unite findings with the broader scientific community. The team hopes to continue this work in future cases and will pursue these endeavors with creativity and optimism.