A Semantic Mixed Reality Framework for Shared Cultural Experiences Ecosystems

Abstract

This paper presents SemMR, a semantic framework for modelling interactions between human and non-human entities and managing reusable and optimized cultural experiences, towards a shared cultural experience ecosystem that might seamlessly accommodate mixed reality experiences. The SemMR framework synthesizes and integrates interaction data into semantically rich reusable structures and facilitates the interaction between different types of entities in a symbiotic way, within a large, virtual, and fully experiential open world, promoting experience sharing at the user level, as well as data/application interoperability and low-effort implementation at the software engineering level. The proposed semantic framework introduces methods for low-effort implementation and the deployment of open and reusable cultural content, applications, and tools, around the concept of cultural experience as a semantic trajectory or simply, experience as a trajectory (eX-trajectory). The methods facilitate the collection and analysis of data regarding the behaviour of users and their interaction with other users and the environment, towards optimizing eX-trajectories via reconfiguration. The SemMR framework supports the synthesis, enhancement, and recommendation of highly complex reconfigurable eX-trajectories, while using semantically integrated disparate and heterogeneous related data. Overall, this work aims to semantically manage interactions and experiences through the eX-trajectory concept, towards delivering enriched cultural experiences.

1. Introduction

Cultural applications are increasingly used for the development and delivery of cultural experiences to users. To this end, research and development activities have targeted the design and implementation of applications and related systems to support all of the phases of the creation and operation of cultural applications, including content creation and organization, application development [1], application operation within venues [2,3], and in broad Internet of Things (IoT) environments [4,5,6].

Insofar, however, each cultural application is designed, implemented, and deployed separately, increasing the associated development costs (content development, code creation and testing, infrastructure deployment, and maintenance), while, at the same time, limiting the opportunities for sharing and reusing cultural experiences to the level of recommending isolated points of interests (PoIs) or coarse-grained routes [7,8,9]. The impact of these challenges is more pronounced in augmented, virtual, and mixed reality (AR/VR/MR) systems, for which content development, code implementation and deployment infrastructure are more complex and demanding. In addition, the range of the required hardware and software systems [10] poses further data integration and reuse issues.

The SemMR semantic framework proposes an integrated multi-technology and multi-entity approach towards addressing these challenges and supports current, as well as future, interactive technologies that are of low effort and cost, being accessible to all businesses. In particular, SemMR is inclusive towards technologies that are based on MR (including AR/VR). The main ingredients of the SemMR approach are the use of semantic technology for the utilization/integration of data and information discovered on Web sources, the cloud of Linked Open Data (LOD), and the IoT. Through this approach, the SemMR framework promotes experience sharing at user level, as well as data/application interoperability and low-effort implementation at the software engineering level. The framework is based on the notion of a shared cultural experience ecosystem (SCEE) in order to enable and support the management of enhanced user experience in the cultural domain. IoT is the key factor of future interaction [11] and the prominent source of the immense amounts of semantically linked data/information. In order to do so, the user behaviour must drive and be driven by the semantically integrated data/information/knowledge, thus creating a new world of seamless and immersive MR interaction between the real-world entities and the virtual entities. When specialized in the domain of MR, the SCEE ecosystem is denoted in this paper as MR-SCEE.

SemMR is based on two main concepts; the concept of the shared cultural experience ecosystem (SCEE) and the concept of the cultural experience as a semantic trajectory (eX-trajectory):

• The eX-trajectory notion in SemMR is used for the representation of the mapping of a semantic trajectory to an MR experience. A trajectory concerns segments of connected traces/points that represent the movement of entities. A semantic trajectory is a trajectory that has been annotated with additional information that is related to those segments, usually to add knowledge related to moving entities in time and space and their experiences within those dimensions. Such experiences may involve multiple episodes or scenes within the segments of traces, where interconnected and interacting entities are moving and acting in the MR world, following spatiotemporal paths of special application-specific interests, exhibiting diverse behaviours (virtual or real).
• SCEE is the eX-trajectory ecosystem within the SemMR semantic framework, generating, semantically integrating, and managing open and reusable cultural heritage content (cultural heritge experiences, data, and information), cultural applications, and methods. Multiple entities (human and non-human) may interact with each other at different time and space, thus creating a multi-dimensional space of shared cultural user experiences. The SemMR ecosystem maintains the shared eX-trajectories, where the length of each eX-trajectory (cultural experience) might vary. The SCEE is the enabler for advanced interaction and sharing, since expert and non-expert users may interactively author new experiences. On the other hand, along the user interaction timeline, those experiences may dynamically intersect and interchange, resulting to unseen, but relevant, eX-trajectories.

SemMR is a semantic framework for modelling interactions between human and non-human entities and managing reusable and optimized eX-trajectories. It comprises of methods for: (1) creating and managing open, reusable, and optimized eX-trajectory content, applications and tools, (2) tracking, monitoring, and analysing user behaviour during interactions with the environment and with other entities, (3) optimizing (via reconfiguration) eX-trajectories at runtime or at development time, and (4) synthesizing eX-trajectories into new, but still reconfigurable, eX-trajectories, which are augmented by exploiting semantically integrated related data/information that is sourced from diverse and heterogeneous resources.

The contribution of this paper is outlined in the following three points:

• The specification of the SemMR framework for enabling low-effort multi-entity interactions towards creating and managing reusable and optimized eX-trajectories.
• The system architecture for implementing and realizing the SemMR framework.
• The simulated performance evaluation for the deployment of the SemMR in an international cultural site.

The structure of the paper is outlined, as follows: Section 2 presents the related work. Section 3 details the proposed eX-trajectory concept. Section 4 introduces the SCEE, while Section 5 presents the system architecture for SemMR. Section 6 presents the instantiated implementation of SemMR for cultural experiences and Section 7 presents the user behaviour modelling. Section 8 presents the evaluation on the scalability of the proposed framework for a sizable cultural site, an archaeological museum, and, finally, Section 9 discusses the proposed framework and outlines future work.

2. Related Work

As cultural application development and use proliferates, researchers have developed a number of approaches that underpin and facilitate different parts of the cultural application lifecycle. Amato et al. [12] present SNOPS, a system that consolidates participatory sensing, IoT platforms, and recommendation systems under an instantiation of the Service-Oriented Architecture, targeting the collection of information from data sources, which are then exploited for the formulation of context-aware recommendations for users. The context of the recommendations is represented as an upper-level ontology, which encompasses classes for modelling users, objects/places of interest, time intervals, activities (either explicitly modelled or deduced), environmental conditions, and devices (both user access devices and sensors). Chianese et al. [2] describe the design and implementation of a system that is able to leverage cultural spaces into smart cultural environments following the concept of Single Smart Spaces (S³), which result in enhanced user experience. The system that is proposed in [2] retrieves (a) data from sensors that perceive the real world, (b) information from structured and unstructured data sources, and (c) knowledge from users moving into the smart cultural environment, and processes the input to deliver knowledge to users to facilitate a number of tasks, including navigation and information finding. The concept of S³ is also adopted in [12,13], where a context-aware framework for cultural heritage applications is presented. The framework presented in [12,13] captures contextual information under a Context Dimension Tree, which represents six dimensions of the contextual information: users interacting with the system; items within the smart space; activities performed on items; situations within which activities are carried out; locations of activities; and, times when activities were performed. From this information, the system continuously learns usage patterns and propagates the resulting knowledge to users.

The exhiSTORY approach [14,15] integrates IoT and semantic technologies, together with clustering and personalization techniques to leverage exhibits within cultural venues to smart, self-organizing exhibits that cooperate with each other and provide visitors with comprehensible, rich, diverse, personalized, and highly stimulating experiences. In more detail, within the exhiSTORY approach, each exhibit maintains an amount of self-descriptive data and semantic information, and communicates with both (a) neighbouring exhibits and (b) the smart space, to create multiple meaningful collections of items. Each collection tells a story about a specific subject. Subsequently, personalization technologies are employed to select the most prominent stories to be told to visitors, after consulting their profiles.

In the following paragraphs, we elaborate on the research work and technologies related to the main axes of the SemMR framework, namely (i) semantic data management, (ii) virtual entities and IoT, (iii) user profiling, and (iv) mixed reality.

2.1. Semantic Data Management: Link Discovery and Data Integration

Link discovery (LD) is the process of identifying relations (links) between data/information objects of different provenance (i.e., that that have been retrieved from different sources). The identified relations are then used to support several tasks, including data/information integration and deduplication. In the case of spatial datasets, the goal of LD is to identify pairs of spatial objects for which a given set of relations is satisfied. Existing works in this area have mainly targeted the discovery of topological relations (intersects, contains, crosses, meets, etc.) between spatial objects, while the recent work on maskLink [16] has been employed for discovering proximity relations, as well as in trajectory reconstruction and semantic enriching of trajectory segments.

RADON [17] is a recent topological relation discovery approach for relations between data sources of areas and it can discover efficiently multiple relations while using space tiling. Smeros and Koubarakis [18] studied link discovery on spatiotemporal RDF data by examining several topological relations that are defined on polygons. Finally, the maskLink approach [16,19] tackles both topological and proximity relations. It has been implemented in a flexible framework, which includes features, such as:

• streaming and archival data access,
• efficient blocking technique for LineString geometries (minimizing the computational overhead produced by MBRs of such geometries), and
• a suite of generic and “ready-to-use” functions that can be exploited for domain-agnostic trajectory enrichment, demonstrated for the support of complex event recognition [20].

Going beyond the state-of-the-art methods in LD and data integration, SemMR develops LD algorithms for discovering spatiotemporal relations (as well as other well-defined semantic relations) between the eX-trajectories, supporting the meaningful exploitation of similar and related trajectories.

2.2. Virtual Entities: IoT Management and Trustful Interactions

The interaction with objects in MR worlds requires sensing from physical space, or even sensing of user parameters to be able to provide high value user experience. To achieve this, a full IoT infrastructure for collecting important data for VR/AR space reconstruction, as well as device virtualization, device management, and trustful interaction must be provided. In device virtualization, there are several commercial IoT-related products that aim to aggregate all of the data that IoT can generate in cloud storage and then expose them to developers through RESTful interfaces and libraries for enhanced service creation. However, these initiatives remain bound to an information-centric view, where the main value of the things is more on the information they can generate and less on the possibility to include augmented AR/VR interaction with an object and between users, offering services and actuation on it.

In IoT management paradigm, trust-related issues need also be addressed, for instance, ways to manage trust between entities without the existence of a central authority. These issues may be addressed while using clear and simple semantics. As trust management mechanisms have been widely studied in various research fields [21], it is now commonly accepted that the seamless integration of trust management mechanisms in IoT is needed [22,23]. The recommendation and standardization of a well-defined trust negotiation language supporting the semantic interoperability of IoT context is a challenging and open IoT-trust modelling and management topic [24,25].

SemMR delivers an integrated framework for: (a) capturing and virtualizing human and non-human (mobile and smart) entities (users, smart rooms, smart phones, smart bands, smart tags, etc.) and their interconnections, supporting their automated identification and recognition, and their open (re)use by cultural experience authoring environments, and (b) modelling and computing trust for the interaction of the virtualized entities, based on principles, such as friendship, ownership, collaboration, as well as on contextual information, such as environmental conditions that are sensed by the smart tag.

2.3. User Profiling

SemMR uses profiling methods to adapt user interfaces and interactions to the specific characteristics of users, particularly their age, gender and cultural background, their physical and cognitive abilities, their level of engagement, and their preferences. Consequently, it is necessary to model the user profiles at different levels: their intrinsic characteristics (physical characteristics, identity, age, disabilities, behaviour, emotional state, skills, etc.), their physical environment (location), their social environment (job position, tasks), and their needs and preferences. User profiles can be explicitly built by inquiring the users for direct information, or implicitly by deducing their profile from their interaction with the system. Implicit and explicit profiles are complementary aspects. It is important to keep in mind that user profiles change over time and that, in that context, a dynamic user profile is fundamental for successful personalisation. Other works utilise the user personality traits to deduce work leadership profiles and construct harmonious and effective teams [26].

User profile creation and maintenance can be supported through a multitude of mainstream methods. Fine-grained tracking of facial expressions and body movement on the visible spectrum can be achieved using hardware, such as Intel^®RealSense, 3D Kinect and eye-trackers (e.g., Tobii Glasses). Moreover, biometric signals can be recorded and tracked while using sensors, such as NeXus EXG and Blood Volume Pulse. These allow for multimodal interaction, a very natural social form of interaction that has been shown to improve human learning and the treatment of medical conditions. Learning and user experience and acceptance may be enhanced by immersive interactive environments [27]. Learning might be reinforced by multi-sensory approaches that may be used for the personalisation of the assessment and reflection phases for improved user experience.

User profiles may be associated with, or abstracted to, user behaviour models. There are several paradigms for user behaviour modelling and action planning for domains of varying complexity with most prominent concerning Finite State Machines (FSM), Agent-Based Modelling approaches, Social Force models and Activity-Based Models [28]. In FSM, each user action leads to a new state. Simple algebraic structures relate internal states to input and output sequences offering a general model of user behaviour. FSMs were successfully used to model human-robot interactions and dialogue behaviour [29,30]. Agent-based systems are developed for simulating (virtual) human behaviour in a variety of disciplines, from knowledge building in collaborative online communities, like wikis [31,32] to task assignment in crowd work environments [33,34,35] to the way people select which exhibits to see in the physical space of a museum [36]. Users are represented as intentional rational agents. An agent model includes perception, beliefs, desires, planning/reasoning, commitment, intentions, and acting, and represents a comprehensive model of user behaviour simulation.

Social or behavioural forces specify the degree of behavioural change (e.g., changes in acceleration or in direction), as reaction to external forces that are exerted by the environment or other agents. These forces have a stimulating or repelling effect on the motivation of humans to perform certain activities [37].

SemMR models adaptive user and system behaviour in dynamic non-sequential interactions. For this, cognitive models that produce detailed simulations of human (multi-)task performance are designed in order to implement simulated artificial agents to play a role in a multi-agent (multi-entity) setting.

2.4. Mixed Reality

MR refers to environments where real world and virtual world objects are presented together in a single display. The two most common methods for creating such MR environments are AR and Augmented Virtuality (AV). AV blends elements from the real world to the Virtual Environment (VE), while AR works by superimposing computer-generated objects upon the Real Environment (RE). Virtual Reality (VR) is an alternative approach that constructs and displays entirely synthetic worlds that may simulate the physical properties of the real world, where users can be totally immersed in [38,39]. In most AR applications, the RE is streamed through the camera feed of a device, such as a smartphone or a camera-equipped Head Mounted Display (HMD), with the virtual objects being superimposed on the RE by either using computer vision with fiducial markers, or sensors, to properly adjust their position and rotation. However, recently, marker-less AR received significant attention and it is now widely used in popular AR applications development platforms.

Nowadays, most of the popular VR and AV application development platforms utilize sophisticated sensors to support room-scale applications, allowing for hand presence in the virtual world and full body motion support [40], along with wireless support. Eye tracking is also exploited in some high-end HMD platforms to provide better experience in AV.

Wireless HMDs feature high quality inside-out tracking, allowing for developers to seamlessly blend real and virtual environment in AV and AR applications. Finally, state-of-the art technology enables brain activity and eye movement detection, which allows for user behaviour tracking for real time personalization and enhancement of user experience in MR applications.

SemMR enhances MR development systems/platforms by (a) integrating a graphical drag-n-drop code-free authoring environment for synthesizing open and reusable MR experiences, (b) recommendation methods for automatically suggesting related external data/info to be attached to MR content for enhancing it, and for automatically suggesting new eX-trajectories to support the reconfiguration of existing (towards optimization), (c) focusing on IoT to allow for seamless and ‘live’ interaction of interconnected trustworthy deployed entities (human and non-human ones), (d) properly understanding human behaviour and cognition while experiencing MR worlds, (e) semantically integrating external heterogeneous and disparate information in order to enrich the content of the MR experiences, improving their quality and, thus, the quality of the user experience, and (f) appropriate models and methods for the reuse of connections between virtual and real objects, in more than one MR world, enlarging, this way, the MR environment

3. Experience as a Trajectory (eX-trajectory)

A movement track represents the ability to capture the movement of an object or entity moving in a geographical space over some period of time. This temporal sequence of the spatiotemporal positions is represented as pairs of ‘instant’ and ‘point’. Additional data (depending on the capabilities of the movement recording device) may also be recorded, e.g., the instant speed or stillness, acceleration, direction, and rotation. Such captured data are the raw data. In some applications, there is no interest in keeping and analysing continuous non-stop records of raw movement data. Instead, segments of interest may be selected, i.e., a specific movement track within a ‘start’ and ‘stop’ (Begin and End) point. Trajectories are the segments of an object movement track that are of interest for a given application. For instance, when considering an application that is required to track and analyse tourist movement and cultural activities in the city of Athens. For this (big) data recording example, the application identifies a trajectory for the whole track that is left by an individual tourist in Athens (e.g., ‘inside Athens’ trajectory), but also another trajectory tracking a specific daily cultural experience track of this individual (e.g., ‘a tourist in Athens on a Sunday tour’ or ‘a tour in the Museum of Acropolis on Friday morning’).

In some cases, the application processes require using contextual data or information to complement and augment raw data. For instance, to be able to interpret the trajectories of people in a city, additional information regarding the city (e.g., the map or the PoIs of the city) is required. Spatiotemporal data (coordinates) can be reverse geocoded and transformed into names/identities of PoIs (e.g., monuments, parks, or shop names) or names of streets and squares. For example, information about events of any nature, from football games to concerts and protest marches, enables traffic monitoring applications to distinguish among normal and exceptional traffic conditions, leveraging the interpretation of spatiotemporal data (positions). Adding information to raw trajectories is called semantic enrichment of trajectories. Such enrichment requires the process of complementing existing data with additional data/information, i.e., annotations. Additional data/information is connected either to parts of (segments, points) of a trajectory, or to the trajectory as a whole. In this context, a semantic trajectory is a trajectory that has been augmented with annotations that add context. In a tourist example, recording the movement pattern of a museum visitor (e.g., ant, grasshopper) [41] is a trajectory-level semantic annotation. On the other hand, marking the person’s presence at a location, as a visit to a temporary modern art exhibition, is a semantic annotation at the location level.

3.1. eX-Trajectory Modelling

Trajectories are widely defined as the segments of the object’s movement track that are of interest for a given application. Semantic trajectories are trajectories that have been enhanced with semantic annotations and/or one or more complementary segmentations. One might superimpose a structure of homogeneous segments that are meaningful for the particular application to enhance the knowledge captured by trajectories. These homogeneous segments are called episodes. Episodes are defined as a maximal subsequence of a trajectory, such that all of its spatiotemporal positions comply with a given predicate [42].

Existing semantic trajectory modelling and representation approaches impose limitations on the structure or the elements of trajectories. More specifically, they either:

• represent relationships to other entities using plain textual annotations [43,44] instead of semantic links, considerably limiting the accuracy and exploitability of entity relationship information. On the other hand, the use of semantic links to model (a) the relationships between trajectories and other data or (b) the relationships between trajectories and semantic resources that are associated with the behaviour of moving objects would provide a fully-fledged, accurate, and exploitable representation of the same information;
• impose restrictions on the components that can be used for structuring a trajectory, by only allowing a limited set of event types to be included [45]; and,
• require that the ingredients/parts of trajectories adhere to several implicit or explicitly expressed rules [43,46,47].

For example, in some works, semantic trajectories are sequences of sub-trajectories [43], while, in others, they are sequences of episodes [46]. For the datAcron ontology [48], the representation of trajectories at multiple and interlinked levels of analysis is supported. In related works, a rich set of constructs for the representation of semantic trajectories is presented as sequences of episodes, each being associated with raw trajectory data and (optionally) with a spatiotemporal model of movement, although without a fine association between raw movement data and abstract models of movements [46]. In Bogorny et al. [43], semantic trajectories are defined as lists of sub-trajectories, and each sub-trajectory as a list of points. Events and episodes are connected to specific resources at specific levels of analysis: events that are mostly related to the environment are only connected to points [43], while episodes concern things occurring at trajectory-level and they can be linked to specific models of movement [46].

3.2. eX-Trajectory Management and Analytics

A trajectory behaviour is defined as a set of specific characteristics that can be used to identify a particular connection or link to a moving object or to a set of moving objects. The behaviour is defined by a predicate that expresses whether a given trajectory (or a given set of trajectories) shows the corresponding behaviour [42]. For instance,

• a “Tourist” behaviour might concern a daily trajectory that shows the Tourist behaviour, if (a) its departure (Begin) point P1 is a place of type “Accommodation”, (b) it makes at least one stop at a place of type “Museum” or “Tourist Attraction”, (c) it makes one stop at a place of type “Eating Place”, and (d) its arrival (End) point is in the same P1 place as its departure point.
• A “Meet” behaviour might concern a set of trajectories that show the specific (meet) behaviour, if every trajectory of the set roughly ends at the same space (point) and at the same time (instant).

A trajectory can be characterized by several behaviours. For instance, a trajectory can show both a “Speeding” and a “Tourist” behaviour and simultaneously be part of a group of trajectories showing the “Meet” behaviour. For each behaviour, the predicate relies on different characteristics. The trajectories of tourists visiting a city may be analysed for (a) creating tourist profiles and recommending personalized itineraries and services, (b) the flow regulation of visitors and tourist either within cultural venues or at city level, etc. Processes that analyse trajectories to identify similarities and dissimilarities among them (including a feature-based comparison between trajectories), classify the trajectories into types based on their similarity, and extract the salient features that differentiate one trajectory group from another underpin all of this. A set of distinguishing features (called patterns or behaviours) forms a summary description of the group of trajectories.

Several systems for trajectory data management and analysis exist. SpatialHadoop [49] and Simba [50] enable distributed spatial analytics based on the MapReduce paradigm. Nevertheless, they do not exploit the characteristics of trajectory data for efficient data management and analytics. A cloud-based system by Bao et al. [51] and Elite [52] provide distributed solutions for big trajectory data. They utilize specific partitioning strategies in distributed environments in order to support data retrieval. Other systems that offer distributed storage and computing also exist. SnappyData [53] integrates Apache Spark and Apache Geode to support efficient streaming, transactions, and interactive analytics. Although these systems provide solutions that enhance Spark and eliminate inefficiencies of heterogeneous systems, they do not provide flexible operations and optimizations for trajectory data analytics. A recent related work on UlTraMan [54] adopts a flexible framework that supports customizable data formats, partitioning strategies, index structures, processing methods, and analysis techniques, which offer better support to realize optimizations and complex analytics. UlTraMan also adopts a unified engine that supports efficient trajectory data management and analytics.

4. Results Mapping Cultural Experiences to Semantic Trajectories

SemMR integrates key technologies, such as semantic and IoT technologies, for user interaction, in order to advance cultural experiences within a new shared cultural experience ecosystem (Figure 1), accommodating provisions for supporting MR-based interaction.

The key concept in SemMR is the handling of real-world information as part of virtual entities (VE) that change and evolve, either individually or as part of related groups. The VEs are in collaboration to create experiences and they may interact with human experiencers or with each other in scenarios or stories. Such interaction (that can be seen as a dialogue between multiple types of entities, human, and virtual) might be dynamically shape when trajectories of previous and recently authored experiences coincide or become relevant. Multiple users may interact with multiple entities at different times, thus creating a shared experience user space in cultural or other domains. The SemMR ecosystem maintains the shared experiences that are complete, allowing for variable length (number of visiting points of interest/events/activities and their duration). User behaviour is monitored by employing interaction metrics and sensor data from wearables, such as smart bands, on the users. Through the analysis, the shared experiences are reconfigured and then presented to the SemMR entities for selection and interaction. The SCEE is the enabler for advanced interaction and experience sharing, since experts may author interactive experiences that may intersect and interchange in a dynamic fashion during the interaction, offering new relevant cultural interaction potential to the users.

5. System Architecture

Based on the SemMR framework, this section elaborates on the architectural design of a SemMR-based system, which constitutes its first realization/instantiation. The proposed system in Figure 2 is considered to be a sample instantiation of the framework, without only limiting SemMR to the proposed architectural design that is presented in this paper. However, the architectural design presented below is based on one-to-one mapping of its components (methods and tools) to the concept that is specified in SemMR.

The main components of the proposed SemMR-based system are the following:

• eX-trajectory management and analytics,
• management of IoT/VEs,
• management of semantic data and information,
• monitoring and analysis of user behaviour, and
• the cultural application authoring environment.

The functionality of each component is detailed in the following paragraphs.

5.1. eX-Trajectory Management and Analytics

This component is responsible for managing and analysing eX-trajectories while using methods for representing, storing, querying, linking, synthesizing, and enriching eX-trajectories. It identifies similarities between eX-trajectories, which are the types of relations between one or more eX-trajectories. It supports a multitude of analytic tasks for eX-trajectory behaviour, compilation of statistics, and advanced methods for mining co-movement pattern from eX-trajectories. It provides a number of services, by exploiting data from other architectural components and, more specifically: (a) it generates recommendations for the reconfiguration of eX-trajectories, utilizing user behaviour analytics data that are sourced from the user behaviour analysis component; (b) it supports the integration of VEs into eX-trajectories through the use of data that were obtained from the IoT/VE management component; and, (c) it enriches the eX-trajectory data and information by utilizing data that were sourced from the data/information management component.

5.2. IoT/VEs Management

This component is responsible for (a) promoting connectivity between human and non-human entities, (b) enabling abstract representation (virtualisation) of those entities using the semantics of the SemMR ontology, (c) managing VEs by offering services for the efficient storage and querying of data/information created by VEs in the context of their involvement in eX-trajectories, and (d) supporting trustful interactions in the context of cultural spaces, through the realization of a relevant trust model and the computation of trust between VEs.

5.3. Semantic Data/Information Management

This component facilitates: (a) search and discovery of disparate and heterogeneous domain-specific data and information related to eX-trajectories, (b) transformation of data to a common syntax and model (RDF) and data integration under a common view, based on the SemMR ontology as well as suitable domain-specific ontological models, and (c) the enhancement of eX-trajectories, through the computation of offering recommendations containing unified and integrated data and information; to this end, the semantic data/information management employs semantic matchmaking methods.

5.4. User Behaviour Monitoring and Analysis

This component is designed for facilitating (a) the tracking and monitoring of human entities during interaction, (b) analysing user behaviour to identify confusion, boredom, uncertainty, frustration, etc., (c) examining the user affective mental state and preferences to reach decisions regarding intervening actions that must be taken, including offering of comments, the generation of recommendations for path changing, (d) identification and presentation of additional content, (e) augmenting the initial user profile, already present in SemMR by contributing physiological aspects that are derived from the monitor and analysis of user movement, and (f) realizing an integrated interactive interface, through which feedback might be provided.

5.5. The Cultural Authoring Environment

This component is responsible for authoring eX-trajectories and developing cultural apps with low effort. It provides the following: (a) a graphical drag-n-drop code-free authoring interface of eX-trajectories, (b) methods for synthesising open and reusable eX-trajectories, (c) methods for integrating VEs into eX-trajectories, and (d) methods for enhancing eX-trajectories through semantic enrichment. This environment allows for the creation of domain-specific cultural applications that are delivered to the intended audience for immediate use. The cultural authoring environment interfaces with the MR devices and, in general, the infrastructure that is required for the MR application delivery and makes these interfaces available to developers as objects and APIs for a higher level of abstraction. In this fashion, the MR application development is disassociated from the idiosyncrasies and peculiarities of the hardware and, therefore, can be more efficiently developed and with better portability.

In the heart of SemMR, there is a triple store for storing and querying integrated data in Resource Description Framework (RDF) data model [55] and the SemMR ontology encoded in the OWL W3C ontology language [56]. The SemMR ontology is designed to encompass representations for eX-trajectories, VEs, and all additional domain-specific data and information that is required, i.e.,

• data regarding the human movement, behaviour and interactions; these may include raw data sourced from sensors through the user monitoring module, or results of the analysis of these data, as determined by the user behaviour analysis module, and
• data sourced from the Web/LOD/RDBMS, which will be utilized for the semantic enrichment of eX-trajectories.

The design of concepts and properties in the SemMR ontology regarding semantic trajectories is based on the definitions provided by the datAcron ontology for the representation of semantic trajectories of moving objects [48] and the semantic trajectories design patterns provided by Zhang et al. [57].

The SemMR ontology development process will follow a collaborative workflow, such as the one specified in the HCOME methodology [58], while using the collaborative ontology engineering tool WebProtégé and discussion threads via e-mail and Google docs/groups. In the architecture diagram that is illustrated in Figure 2, three stores are depicted as a conceptual approach to the organization of the SemMR data/information management; however, this is not necessarily the case for implementing three different physical stores.

Figure 3 depicts the SemMR key offerings and supported technologies.

6. SemMR Example Ontological Specifications

In the following subsections, we present the cultural heritage domain considerations and the ontology specifications that implement the semantic trajectory for the domain semantics.

6.1. Cultural Heritage Domain

The CrossCult ontology (IRI: http://kb.crosscult.eu/) organizes cultural object semantics in multiple layers, aiming to underpin the representation of connections between cultural heritage data. The CrossCult ontology extends the basic CRM modelling of CH data semantics, by accommodating the modelling of users (of applications) and visitors (of venues), as well as venues (sites, buildings, rooms, etc.). Furthermore, it defines semantics such as interest and review (subclasses of E73 Information Object in CRM). Moreover, the CrossCult ontology reuses elements from the namespaces of FOAF (e.g., for modelling persons, their activities and relationships between persons), Dublin-Core (to describe periods of time and specialised datatypes), and SKOS (to model relationships between concepts), as well as DBpedia (primarily for enriching instances of the Upper-level ontology with links to DBpedia concepts) to offer a more comprehensive coverage of concepts of interest.

A modelling choice of CrossCult ontology, adopted in the SemMR ontology modelling, is the reuse of the specialised CIDOC CRM [59] classes, such as E22.Physical Man Made Object and E24.Physical Man Made Thing, which provide a unified view to a wide range of concepts. Artefacts, paintings, museum exhibits, monuments, are modelled as instances of those CRM classes. This design choice also enables the use of standard semantics for modelling spatial, temporal, geometrical, and other semantic relationships.

Figure 4 depicts a description of an exhibit (PoI), while Figure 5 illustrates the exhibit (PoI) modelling within SemMR. The CrossCult definition of a Visit (and related classes i.e., Visitor and VisitingStyle) in its User-Model namespace has a central role in the representation of the knowledge in SemMR.

In the example that is depicted in Figure 6, we illustrate how individual routes followed by users are linked with a particular visiting style (e.g., ant, grasshopper, butterfly, etc. [36]). Note that a relationship between users and visiting styles is also accommodated at the user profile level, indicating which visiting style a user typically follows. However, it is possible to represent differentiations from the typical user behaviour, occurring at the individual route level, as shown in the example. The visiting style of each individual visit is determined by analysing the user traces against the locations of the exhibits.

6.2. Semantic Trajectories

Motivated by real-life needs in critical domains, such as aviation and maritime, the datAcron ontology provides a coherent and generic representation of semantic trajectories for moving objects [48]. It reuses the ontologies: DUL, SimpleFeature, NASA Sweet, and SSN. A semantic trajectory consists of a sequence of temporally non-overlapping trajectory parts, as shown in the ontology definitions in OWL syntax depicted in Figure 7; each of these parts may be either (a) a semantic node, (b) a raw position obtained from some sensing device, or (c) a trajectory segment. Each trajectory part can be associated with spatiotemporal information regarding its occurrence; in this context, the point or region of the trajectory part occurrence can be expressed via a specific geometry, while the temporal dimension of the trajectory part occurrence can be expressed through a time instant or interval.

Similar to the above definitions, but in a more simplified and generic way, the semantic trajectory design pattern [60] presented in the OWL snapshots that are depicted in Figure 8, defines a semantic trajectory as a number of segments and fixes (synonym to points in other vocabularies). According to OWL, it defines a number of interfaces to incorporate additional geographic information, domain knowledge, and device data.

In SemMR, we link/connect any of the two semantic trajectories representation to the CrossCult ontology, in order to be able to formally represent the eX-trajectory, as defined in this work. To do so, we propose a number of simple design patterns in the SemMR namespace, as described below. In the description of the design patterns, the ontologies listed in

Table 1. Ontologies used for the framework evaluation.

Prefix	IRI	Ontology
traj	http://descartes-core.org/ontologies/trajectory/	Semantic trajectory design pattern
cros	http://kb.crosscult.eu/	CrossCult
crm	http://erlangen-crm.org/160714/	CIDOC/CRM

are used; Table 1 also lists the mapping between the prefixes used in the examples and the full ontology IRIs:

• Define new concept: eXtrajectory as subClassOftraj:SemanticTrajectory

  $$eXtrajectory⊑SemanticTrajectory$$

  (1)
• Link/connect traj:SemanticTrajectory to cros:Visit (which represents a specific visit by a particular user) via an object property (:visitMade)

  $$eXtrajectory⊑\exists visitMade.Visit$$

  (2)
• Link/connect traj:SemanticTrajectory to crm:E28_Conceptual_Object, to allow the explicit representation of the semantics that are associated with a trajectory. Similarly, traj:Segment and traj:Fix are linked/connected to crm:E28_Conceptual_Object, to allow for the representation of semantics that are associated with trajectory segments and individual points. Under this arrangement, all representational levels of eX-trajectories may bear relationships to semantics. However, it should be noted that the semantics appearing at the highest level, used in any context, are the ones conveying the actual meaning of the trajectory more accurately, overriding the lower-level ones. For instance, an eX-trajectory exTr0001 may be comprised of the segments “visit to a museum” (associated with cultural semantics), “launch at a restaurant” (associated with food service and local cuisine semantics), and “shopping at a flea market” (associated with street market and local products semantics). However, the eX-trajectory exTr0001 might be associated with tourist behaviour semantics, modelling the overall behaviour, and not the lower level details of the constituent parts.
• Link/connect cros: Visit to crm:E28_Conceptual_Object, to allow for the explicit representation of the semantics targeted by a particular user visit. This is required, since different users may be following a specific semantic eX-trajectory (which is linked to the cros:Visit object), focusing on the diverse semantics of the eX-trajectory. For instance, a visitor may follow the “Acropolis of Athens” eX-trajectory focusing on the ritual aspects of monuments, while another visitor may follow the same eX-trajectory, focusing on the architectural aspects of monuments. By capturing the point of view of each individual user, more elaborate analytics can be produced, and more accurate matching can be performed, leading to better recommendations.

7. User Behaviour Modelling

Modelling the adaptive user and system behaviour in dynamic non-sequential interactions is a core advantage of the SemMR framework. This is of key importance for a number of settings, including (a) cultural sites and cities with tourist attractions, since their visitors are free to roam and view the site points of interest in no particular order and (b) learning environments, where learners are able to choose between learning paths or access and use learning material in distinct sequences, collaborating with other learners or instructors. Towards this, cognitive models that produce detailed simulations of human (multi-) task performance were designed and used to implement simulated artificial agents in a multi-agent (multi-entity) setting. AI agents compute the most plausible task action(s), given their understanding of the context, actions of others, their preferences and goals, provide alternatives and plans, roll out possible outcomes, and, therefore, are able to adapt their behaviour to their partners. They also know why they select a certain action and can explain why the choices made lead to each specific outcome (explainable AI). Agents can be built while using rather limited real or simulated expert and/or interactive data: an agent is supplied with initial state-action templates encoding domain knowledge (as eX-trajectories enriched with VE and IoT information), the user’s profile, and preferences. Over time, the agent learns from the collected interactive experiences. Suggestions and optimizations are performed by finding prior experiences (instances) that are the most frequently or most recently used and/or are most similar to the current situation (contextual parameters, user affective state, user’s goals, and preferences), blending the instances together to the extent that they match the interactive state.

The advantage of the SemMR approach is that it requires far less experiences for the system to be able to interact with the user in a sensible way and that it incrementally improves as its set of instances increases in size. It also allows for utilizing experiences of others to guide and enrich the experiences for new users. Additionally, growing data from the eX-trajectories store, paired with increased computational power, can be used to apply modern powerful Artificial Neural Networks (ANNs) and Deep Learning (DL) approaches, which showed a significant impact on many AI and HCI applications [61,62]. Finally, the SemMR model accounts for real-time user interaction errors, which supports the modelling of dialogue repair in user understanding, and the integration of a memory and interest model, reflecting the individual and changing configurations of the user’s mind.

8. Evaluation

We ran a set of simulation experiments with artificial users in order to evaluate the feasibility of the approach, the appropriateness of the recommended eX-trajectories, and the scalability of the proposed framework. In this set of experiments, eX-trajectory recommendations were formulated and served to (artificial) users with diverse profile characteristics. Subsequently, the suggestions were evaluated for suitability against the relevant user profiles. Furthermore, the simulation process entailed concurrent formulation of recommendations to measure the execution time that is required for different levels of concurrency and quantify the response time of the SemMR instantiation.

For these experiments we used two machines. The first one was equipped with 2x Intel Xeon 8-Core CPUs, 64GB of RAM, and 480GB SSD with a transfer rate of 550MBps; the total estimated cost of this machine is less than 1.5K Euros, refurbished. This machine hosted the proposed framework, as well as the SemMR instantiation (database and MR eX-trajectory services). It also managed the database items, exhibit locations and semantic descriptions (already pre-processed as VEs), user profiles and trajectories, new content from IoT links (already retrieved but not pre-processed), and recorded and analysed the user paths. The second machine created the pool of concurrent users-visitors. For each user/visitor, the respective trajectory was built, when compared to existing trajectories and synthesized new, suggested trajectories, randomly, between 2–5 trajectories and 4–7 suggestions for each user tour. The machines were connected through an 1Gbps local area network.

For the cultural experience validation, we simulated scenarios for the visitors of the Acropolis Museum in Athens, which contains nearly 4250 works of art and welcomes an average of 4.5K visitors per day (nearly 1.5M per year) and while assuming that less than half of them visit it at the same time.

The user profiles that were utilized in the validation experiment were synthetically generated and included the following aspects: age, music choices, game choices, art preferences, museum theme preferences, mood, visiting style, gender, place of origin, and a returning visitor indicator, following the results of the study presented in Alexandersson et al. [63]. Each user profile could be matched to one of the six personas (predetermined user stereotypes) identified for the Acropolis museum in [64], however the degree of similarity for the best match varied from an absolute match to 5/8 attributes (the “mood” and the “returning visitor indicator” were not part of the stereotype modelling, only appearing in the user profile). Similarly, the eX-trajectory database was synthetically populated. Descriptions of landmark exhibits were crafted according to the permanent collection descriptions offered in the museum’s website and, subsequently, areas of the museum were also modelled while using information from the Acropolis Museum application in the Google Arts and Culture website complemented with information from in-situ visits. Both landmark exhibit and museum area descriptions were semantically tagged. The semantic tags included the thematic area of the objects (e.g., religion, everyday life, wars, death, mythology, etc.), the artefact era, and the type of the artefacts (statues, household objects, buildings, grave goods). Subsequently, eX-trajectories with varying duration and spatiotemporal patterns were created and inserted into the eX-trajectory database, observing the profiles of the users. For instance, visitors with ant-type visiting style move sequentially along the areas of the museum, increasing their speed when the theme of the museum area does not intersect with their theme preferences, while moving more slowly in areas having content they are interested in. On the other hand, grasshopper visitors only approach certain exhibits falling within their interests spending a significant amount of time in front of them, crossing empty spaces and moving at a fast pace in other cases [36]. Physical fatigue was also considered in the generation of spatiotemporal sequences, with the effect being more significant in spatiotemporal sequences that are associated with older persons, as asserted in [63]. Overall, 100 user profiles and 620 eX-trajectories were generated and inserted into the virtual entities and eX-trajectory data store.

The eX-trajectories that were generated by the system were parsed and evaluated for appropriateness to each user profile and visitor path. They were found to be in alignment with the user’s visiting style, thematic preferences, and associated stereotype. For approximately 70% of the user/path combinations, 1–3 of the suggestions contained exhibits beyond the user profile thematic preferences, thus fostering novelty and serendipity [65]. As far as performance is concerned, Figure 9 indicates the overhead of formulating the eX-trajectory recommendations under varying degrees of concurrency.

In Figure 9, we can clearly see that the overhead per execution is small (approximately from 0.18 to 0.3 s, depending on the level of concurrency) and scales linearly with the number of concurrent executions (users). However, for concurrency levels that are higher than 200, the machine was saturated, and performance dropped rapidly, indicating that a second machine (of the same specifications, as the one used in our experiment) must be added.

Overall, the cost required for the machines hosting the SemMR framework is estimated to be less than 17K Euros, which is deemed to be reasonable and affordable, in order to fully support a museum of the size of the Acropolis Museum.

9. Discussion and Conclusions

This paper presented SemMR, a framework for creating a new ecosystem of shared and optimised cultural experiences, offering the potential to accommodate MR interaction. The SemMR approach employs semantic technology for the utilization/integration of data/information discovered on the current Web sources, the Linked Open Data (LOD) cloud, and the Internet of Things (IoT), promoting experience sharing at user level, as well as data/application interoperability and low-effort implementation at the software engineering level. Sustainability is demonstrated by the validation of the framework in real-world use cases involving both end-users, as well as paired technology integrators from industry that not only aim to deploy SemMR, but also to create the experiences per situation and lead the formative user studies for usability evaluation, focusing on the integration and long-term impact of the integrated technologies to the specifics of the application environment.

This paper presents the SemMR framework, along with the proposed architectural design for its implementation and the scalability metrics based on a use case instantiation. The presented work introduces methods and tools for multi-entity interactions between entities of different types, to create reusable and optimized cultural experiences with low effort, towards a shared cultural experience ecosystem. Specifically, the SemMR framework introduces methods and interfaces for code-free implementation and the deployment of shared and reusable MR content, applications, and tools, emphasizing the notion of the eX-trajectory. The SemMR instantiations (implementation) of the framework deliver high quality cultural experiences, facilitating the interaction between a variety of entity types that interact in a virtual and fully experiential world. In addition, SemMR proposes methods for tracking, monitoring, and analysing user behaviour and the user interaction with the environment and other users, towards optimizing MR experiences by recommending their reconfiguration in two modes, which is at run-time (dynamically) or at development time (statically).

In the current work presented in this paper, the following SemMR components have been implemented: the ontology and the data stores, the MR eX-trajectory services for the management and analytics of trajectories, and, partially, the methods for user monitoring and user behaviour analysis for the evaluation of the framework with the experimental use case.

For future work, the SemMR system implementation that includes all components presented in Figure 2 that correspond to all key technologies that are described in this paper will enable the system deployment to real users for formative evaluation. Such an endeavour is expected to create new content that might be evaluated by both users and experts, such as museum curators. For the latter, an open research challenge is the automatic selection of the most relevant content from external sources and its integration to the authored content, for a seamless, yet enhanced, cultural experience.