Envisioning Digital Practices in the Metaverse: A Methodological Perspective ^†

Abstract

Researchers are exploring methods that exploit digital twins as all-purpose abstractions for sophisticated modelling and simulation, bringing elements of the real world into the virtual realm. Digital twins are essential elements of the digital transformation of society, which mostly benefit manufacturing, smart cities, healthcare contexts, and in general systems that include humans in the loop. As the metaverse concept continues to evolve, the line separating the virtual and the real will progressively fade away. Considering the metaverse’s goal to emulate our social reality, it becomes essential to examine the aspects that characterise real-world interaction practices and explicitly model both physical and social contexts. While the unfolding metaverse may reshape these practices in distinct ways from their real-world counterparts, our position is that it is essential to incorporate social theories into the modelling processes of digital twins within the metaverse. In this work, we discuss our perspective by introducing a digital practice model inspired by the theory of social practice. We illustrate this model by exploiting the scenario of a virtual grocery shop designed to help older adults reduce their social isolation.

1. Introduction

The metaverse is a collective virtual shared space that is created by the convergence of physical and virtual reality [1] and allows for social and economic activities. The metaverse represents an opportunity to liberate companies and end-users from screen-bound experiences by offering more instinctual experiences. It can offer advantages and opportunities by enabling activities that in the real world could not be possible and promoting social inclusion: provide opportunities to participate in social activities remotely and propose controlled, low-stimulating, and comfortable virtual environments for individuals who have difficulties with social interaction and are particularly sensitive to visual and auditory stimuli [2].

The metaverse can mirror real physical places, creating an extension of our social, connected world through extended reality (XR) technologies. The XR is considered a catch-all term to refer to augmented reality (AR), virtual reality (VR), and mixed reality (MR). Among them, the MR is intended to allow physical and virtual elements to co-exist and users to interact in real time on both by exploiting recent advancements in computer vision, graphical processing, display technologies, input systems, and cloud computing. Devices for entering the MR are typically in the form of head-mounted displays (HMDs) already available in the consumer market. However, in the future, the metaverse is expected to become more accessible as advancements in technology lead to the development of less intrusive and highly wearable MR devices, gaining wider acceptance among users and taking their experiences to the next level. For example, one of the most intriguing possibilities will be the ability to shop in the metaverse, allowing an experience that transcends place and time. With the help of MR technologies, a metaverse shop replays the experience of a real store visit, including even the natural communication with the staff; at the same time, it can revolutionise the customer experience with more tailored recommendations based on their activity, purchase history, taste, and demographic profiles. To model the interconnection between the real and virtual world, this work follows one of the two possible development directions indicated in [3]. The considered development direction is the one that moves from the real to the virtual world, leading to the realisation of a synthetic environment that imitates the real world and where the user perception of the physical experience will be enhanced by building immersive digital experiences.

Given that the metaverse blends physical and digital world, creating a place where real and virtual objects can co-exist and interact, the concept of a digital twin [4,5,6,7,8] is particularly helpful. A digital twin is a virtual replica of a physical object, place or person. The term, elicited in manufacturing systems, has gradually assessed towards the following definition: a “virtual representation of a physical system (and its connected environment and processes) that is updated through information exchange between the physical and virtual systems” [9]. This can be used to provide a high-quality representation of physical counterparts.

Blending aspects of real life into highly immersive virtual environments will lead to virtual experiences that can be compared to real society, which will consequently be affected by various social implications already studied in recent research [10,11]. Virtual environments must adopt social context and practices to make the experience as close as possible to reality. For instance, the virtual grocery should reflect rules that are held in public places, such as greeting others, respecting personal spaces, being polite, responding to others’ requests, waiting in line, etc. To this purpose, social practice theory [12] seeks to determine the link between practice and context within social situations. Computer scientists identified in social practices a sort of contextual behavioural pattern, i.e., a shortcut to determine an action to do, also incorporating means to address it, that can simplify the deliberation processes of virtual agents [13,14,15]. In  [16], a social practice model is the basilar component in an architectural framework aimed at incorporating social non-player characters (NPCs) into the metaverse. The proposed architecture, named MET-iquette, is composed of modules that enable an artificial agent to analyse the context, recognise the ongoing social practice, and choose the most suitable verbal and nonverbal actions by respecting social norms and pursuing virtue-ethics behaviours.

The paper proposes a methodology for engineering MR applications in the metaverse with a focus on the social perspective that is fundamental in virtual environments. The contribution of the paper lies in the adoption of the social practice theory for creating a virtual ecosystem where humans and autonomous digital entities can participate collaboratively while adhering to established social consensus.

The methodology is based on the concept of social practice, which is used to analyse the requirements of the domain and structure them in a semi-formal notation. During the design phase, the methodology moves to a concept of digital practice through the use of digital twins and suggests exploiting software agents as autonomous and proactive decision-makers.

Furthermore, the paper addresses the issue of incorporating social NPCs into the metaverse by proposing an architectural framework that enables an artificial agent to recognise ongoing practices and choose the most appropriate verbal and nonverbal actions while respecting social norms and pursuing virtue-ethics behaviours.

The methodology offers guidelines to map agents, digital twins, and digital practices into a specific architecture conceived for connecting the physical and digital worlds practically.

The methodology has been applied to engineer a virtual grocery shop. This represents a possible application of MR to help older adults reduce their social isolation, have more fun, and even improve their health. The case study has been inspired by  [17], where authors created a virtual social centre framework for socialising older adults by exploiting metaverse-related technologies and human–computer interaction tools (the preliminary results of their study demonstrated benefits in terms of mental wellness). In our case study, we suppose an older person with physical disabilities wants to buy groceries. They could use a 3D HMD to virtually walk into a digital store that is the exact digital replica of a real shop (including employees and other customers). Being represented by a digital avatar, the person can talk with other customers, ask for help or recommendations, talk with clerks, etc.

The paper is structured as follows: In Section 2, we provide the background information on the problem we aim to address. In Section 3, we introduce the concept of social practices and explain their relevance in analysing the domain and collecting requirements. In Section 4, we describe the design phase of our methodology, which is based on digital twins and software agents. We also present the new concept of digital practice. In Section 5, we suggest a general architecture and a set of technologies to implement MR applications with agents, digital twins, and digital practices. In Section 6, we discuss the case study of a virtual grocery store that was developed using our methodology. Finally, in Section 7, we summarise our findings and outline future research directions.

2. Motivation

To motivate and define the aim of this work, we focus on applications of MR by considering the example of a virtual grocery shop in the metaverse, which was discussed in our previous work [16].

Mary is an older person who is unable to visit her preferred grocery store by herself. She uses a 3D visor to enter the metaverse and accomplish her objective of buying food for dinner. Mary would feel more comfortable in a virtual representation of the grocery store that is similar to the reality where she used to go. This way, she can meet other customers who exhibit proper behaviours within the store. The representation of shelves where she can pick items to buy will help her repeat the same habits she had when shopping.

An important aspect of this scenario is that Mary expects certain norms to be followed in this situation. She will be satisfied with her experience if everyone adheres to the traditional principles of customer service, such as receiving assistance when requested and engaging in brief conversations with acquaintances. Of course, some social conventions must be observed, such as maintaining a safe distance from others and giving priority to the cashier when necessary.

Therefore, to model interactions and synergies in such a virtual scenario, let us take inspiration from sociological and cultural theories, in particular, social practice theory (SP)  [12,13]. In accordance with this theory, social order is interwoven with shared structures of knowledge, which are moulded by cultural values. This enables a symbolic configuration of reality taylored to the specific situation. Social practices are a set of interconnected elements that enable people to make sense of the world and behave in a specific manner. Specifically, in SP, individuals act within a social setting that involves physical and mental activities, objects, knowledge, emotions, skills, and so on. SPs relate to everyday tasks and how people usually and routinely do them in a society, like going to work, cooking, and so forth.

An example of social practice in the grocery shop is “requesting assistance about a product” that involves a customer and an employee. However, even this simple social practice may grow in complexity in a virtual scenario such as the grocery example. Figure 1 shows different situations in which the social practice applies.

The first category relies on the classical interpretation of social interaction that occurs in the physical world. It is the case of a customer physically present in the shop that asks for help from an employee. The first category is based on the traditional understanding of social interaction that takes place in the physical world. For instance, when a customer is physically present in a shop and seeks assistance from an employee. A great deal of literature exists on how to represent and enforce these social structures.

The second category of social practice occurring in such a scenario is the remote customer avatar who interacts with a virtual assistant avatar to obtain information about a product. We already treated this category in previous work, MET-iquette [16], in which we enabled an artificial agent to analyse the context, recognise the ongoing social practice, and choose the most suitable verbal and nonverbal actions by respecting social norms and pursuing virtue-ethics behaviours.

Finally, the focus of this paper is to deal with the third category of social interaction that may occur in a virtual space. It is the case of an employee who interacts both with physical and virtual customers. It is an extended interaction that occurs in a twofold way: physically (speaking) and mediated by technology (through an avatar).

Figure 2 shows components of the physical world and their projections within the virtual reality. Some virtual representations are mirrors of the physical component (the physical space, shelves and products). Clearly, there are elements in virtual reality that do not correspond to physical reality, such as the virtual cart. The virtual customer will be represented through an avatar and will interact with virtual agents (ready to assist her) but also with physical employees (when they enter the metaverse).

• Research Objectives

Our approach involves leveraging digital twins to seamlessly integrate physical and social aspects into the virtual environment. By adopting this software engineering paradigm, we can enable effortless integration between IoT and data analytics while also involving humans in the loop. This approach leads to a more cohesive relationship between individuals and technology, empowering them to leverage cutting-edge technologies that augment their capabilities [7].

We approach system design by identifying and modelling the system as a group of interconnected digital entities. Key to this process are digital twins [4,6,9] and software agents [15,18,19], which serve to bridge the gap between design and reality and encapsulate business intelligence, respectively. To accomplish this, we take a highly detailed approach to modelling the domain, constructing small digital twins to represent physical objects and gradually piecing them together to form the complete system. This incremental methodology ensures that all relevant details are captured.

In the realm of MR applications, interactions between people and assets can be quite intricate. Given the dynamic nature of such contexts, anticipating all possible interactions among physical elements can be a daunting task. To address this challenge, we suggest a methodology that relocates the interaction logic among these entities into specific structures. Such an approach mitigates the strict coupling between digital entities, thereby boosting their reusability and flexibility. Our approach is inspired by social practice theory. To create shared social interactions in dynamic MR environments, we have developed a primary abstraction of our methodology called digital practice.

Moreover, studying the social practices for a domain of interest is a means for analysing and understanding the domain by taking a social perspective. The analyst can examine the physical or virtual world and identify entities such as people, responsibilities, roles, places, and resources. By focusing on meaning and expectations, the analyst can collect and organise user requirements, which can then be used to modify the existing system or introduce a new digital system. Finally, studying the activities can help in understanding how things work currently and how the digital part can improve them.

3. Analysis of the Domain through Social Practices

The initial step in creating MR applications involves analysing and modelling social norms within the particular domain of interest. This requires examining both the virtual and physical world to identify entities such as people, responsibilities, places, roles, and resources. The primary goal is to comprehend the users’ expectations and meanings, allowing the analyst to gather and organise requirements. This process can result in modifications to the current system to introduce the digital system (system-to-be). Finally, studying the activities contributes to understanding how they occurred before the system-to-be and how the digital aspect can enhance them.

3.1. Social Practices

As pointed out by  [20], SPs could serve as a foundation for systems that need to take context into account. For this reason, SP theory served as inspiration for devising a model for cognitive agents [15]. It has been exploited to model agents’ behaviour, striking a balance between personal and social aspects. The model, illustrated in Figure 3, is characterised by the following key components:

• Context: Describing the physical aspect of the environment, it includes the Resources used in the practice, the involved Actors, Affordances enabling social actions, the Time during which the practice occurs, and Places where objects and actors are typically situated.
• Meaning: Encompassing the Purpose of actions, Promotes values supported by the Practice, and Counts-as rules, which denote the interpretation of facts within the context.
• Expectations: Within the practice, these involve potential Plan Patterns, Norms, Strategies), as well as Start and End conditions for the practice.
• Activities: Encompassing the Competences that an agent needs to possess and Possible Actions that can be performed within the practice.

3.2. Identifying Social Practices

During this phase of the methodology, the main objective is to gather valuable data from the domain. It is essential to comprehend the users and their organisational environment to establish the MR application’s goals and to identify non-functional requirements. This type of research usually involves user-centred design techniques like interviews, questionnaires, and focus groups, as described in [21]. By merging the theory of social practices with a user-centred design approach, designers can gain a more profound understanding of the users and their practices, leading to the creation of more efficient and user-friendly products.

In the field of user-centred design, gaining insight into users and their needs is paramount to creating effective products or services. A useful tool for gaining this insight is an understanding of social practices. One influential theory in this area is Reckwitz’s theory [12], which asserts the crucial role that practices play in shaping our social world.

In order to apply this theory to a user-centred design approach, it is necessary to gather information about the practices of users. This includes their routines and how they interact with products or services. This can be achieved through various research methods such as interviews, surveys, and direct observations.

Another important aspect of Reckwitz’s theory is the exploration of mental activities of understanding and knowing in a complex of doings. In the context of user-centered design, this can help with identifying the needs and goals of the users. Understanding these needs is crucial in designing products or services that meet these needs.

By analysing the interconnectedness of bodily and mental routines, the design process can be guided to ensure that the product or service fits seamlessly into the user’s practices. This attention to detail can result in a more successful design that truly meets the needs of the user.

Social practices allow the analysts to create a semi-formal description of the interactions that could happen. However, the domain is sometimes too complex to be schematised in a unique practice. The schema is flexible enough to be refined into sub-practices, generating hierarchical structures. We believe that allowing recursion in creating these structures increases software qualities like flexibility, reusability, adaptivity and scalability.

During the other phases of the methodology, it is helpful to maintain the same hierarchical structure of social practices into digital entities. By following a top–down approach, the design phase can be executed more smoothly and efficiently. This approach ensures that the design objectives are broken into smaller, more manageable ones. This not only makes the process more traceable but also aids in achieving the desired outcomes, especially in the implementation phase.

4. Designing and Coordinating Digital Entities

To design the system, we integrate insights gleaned from the analysis phase with the limitations imposed by the implementation phase. Our approach to implementation relies on using agents to carry out the application’s logic and decision-making abilities. To ensure that the physical features of MR and the social dynamics specific to our field are taken into consideration, we utilise digital twins. This leads to the creation of an innovative modelling element called the digital practice, which further enhances our system’s capabilities.

4.1. Digital Twins

Digital twins represent a two-way flow of data between a physical object and its digital counterpart. Changes made to the physical object are automatically reflected in the digital twin and vice versa [9].

Digital twins have become a widely used concept in advanced manufacturing and networked contexts [4], allowing for remote monitoring and control with advanced AI techniques [4]. The ability of the digital twins to provide synchronised logical counterparts of physical entities has been leveraged to engineering self-organising cyber-physical systems (CPSs) [22]. This virtual representation of physical entities is incredibly valuable in the digital transformation of our society [6], aiding development in smart cities and modern healthcare. By facilitating information exchange between physical and virtual systems, digital twins enable the prediction and optimisation of complex CPSs, making them an essential software engineering paradigm [7].

Digital twins are now being considered as software entities that follow the entire lifecycle of their physical counterparts, which requires systematic DT engineering and management [23]. Recent works discuss state-of-the-art methodologies for designing and implementing digital twins, considering various phases, such as requirements definition, architectural planning, integration and verification, existing protocols and standards [24]. As an example, in  [23], the authors introduce a DT lifecycle model, named the $F{A}^{3}ST$ approach, by putting forth methodologies and tools for effective DT management by contextualising their proposal within the framework of Industry 4.0.

Managing the relationships between all the embedded entities can be overwhelming when dealing with dynamic environments. Our scenario, on the other hand, involves multiple stakeholders with diverse behaviours, resulting in numerous interactions within the virtual/physical space. A single digital twin with a massive model would be too domain-specific, making it challenging to reuse in different circumstances. Conversely, adopting a fine-grained approach in which everything (that is relevant) is modelled through a different digital twin was shown to be successful in designing complex scenarios like smart cities [25].

Our design approach is based on a detailed level of granularity. It allows us to model customers, employees, and resources in the shop as different digital twins in our scenario. The system’s digital twins are used to represent various physical spaces. Three digital twins are dedicated to different areas of the grocery shop. The first digital twin provides a real-time description of the layout of the entire building. Separate digital twins are created for each aisle, shelf, cash desk, and customer assistant desk. For instance, the shelf digital twin reflects the real layout of products, price signs, and nearby people. The product inventory is also a digital twin that shows the availability of products in the shop.

A specific case of digital twins is the human digital twin, representing a copy (or counterpart in cyberspace) of a real person [26]. The concept has been raised in socio-technical systems to put humans into the loop. In medicine, for example, human digital twin models the fact that “the physical system is the patient, or a particular organ or physiological system to be managed” [27]. In surgery, having an updated model of the patient’s body could be of capital importance for the surgeon before and during an operation. In MR applications, human digital twins allow alignment of the avatar to the user, reflecting the user’s mood, interests, and behaviour.

In our example, human digital twins represent employees, mirroring their competencies and roles within the market. Finally, digital twins also represent in-person and virtual customers, reflecting their habits, preferences, and behaviour.

Now, the task at hand is to effectively merge these multiple digital twins in a collaborative manner. It is not uncommon for digital twins to be formed by combining smaller ones, which ultimately leads to a Systems-of-Systems (SoS) approach. Such aggregated digital twins may also operate in a hierarchical manner, with each parent seamlessly orchestrating its respective children. However, there is currently no standardised approach for consolidating independent digital twins to construct a larger system. To address this issue, Broo et al. [28] suggest implementing an integration layer that establishes static connections between digital twins that are required to collaborate.

In the literature, the role of coordinating digital twins is often assigned to software agents [29], where agents enable intelligent social behaviours by mediating digital twin interactions. In our methodology, this role is played by social practices and their digital counterpart.

4.2. Digital Practice

We introduce the concept of digital practice for developing a dynamic social perspective for the actors of our MR application: humans and agents. A digital practice may be considered as ‘a digital twin of a social practice’, adding a digital perspective to the original concept of social practice.

A digital practice is defined by the following:

• A set of Roles for human and software agents. In particular, humans are stakeholders, whereas agents are responsible for enacting autonomous and proactive behaviours within the practice (e.g., virtual assistants).
• Context: Digital twins are central for representing entities of the physical setting (humans, resources and spaces); they are used to define the activation rule of the practice, i.e., the specific situation in which the practice activates.
• Time and Space: Essential to delineate the practice’s activation rule.
• Domain Knowledge: Allows disambiguating the meaning in the practice; it is typically represented as a set of beliefs and rules concerning a domain of interest, allowing the software system to perform automatic reasoning and pattern matching.
• A Goal Model: It represents the final purpose of the practice, merging meaning and social expectation (the feeling that something will or should happen in a context); it includes goals and soft goals linked together by relationships. Goals represent states of the world to be addressed, whereas soft goals represent qualities and values the practice promotes.
• A set of Capabilities: What the digital entities can do (are expected to) do in the digital practice.
• Digital Affordances: Expressing ways of conveying to end-users what actions are possible in the context and how to interact with the system.
• An Orchestration Plan: It describes a usual flow of actions to address the final purpose. This is not necessarily a static plan.
• Norms/Conditions: They hold in the social setting and describe what is considered acceptable and not.

Figure 4 is a conceptual map representing some of the elements that identify the customer assistance digital practice. Place and time are at the centre because they represent triggering conditions from the context in which the practice unfolds. Involved digital elements are the remote customer asking for help (a digital twin), an employee that is expected to answer (a digital twin) or, in the alternative, a virtual assistant (mainly occurring in peak times when all the employees are busy). Interactions are expected to happen according to the store’s policy and social norms. Finally, the involved digital entity should own specific capabilities to be useful in the scenario: an affordance to call for assistance, the employee’s ability to find a product or to provide product information, and finally, the affordance for providing positive/negative feedback on the received service, or the overall virtual experience.

4.3. Agents and Intelligent NPCs

From now on, we will adopt two distinct terms to distinguish the general concept of an agent typically used in software engineering, which we will refer to simply as an agent, and the concept of a human-like agent or virtual actor which we will refer to as an intelligent NPC (in alignment with gaming and metaverse terminology). We define an intelligent NPC as an autonomous, proactive and social virtual entity, making decisions to achieve goals based on the information perceived in the environment where it is situated, where the decision-making processes are also influenced by ongoing social interactions with other agents, both virtual and human. It differs from software engineering agents since, beyond their reasoning abilities for specific tasks, an intelligent NPC encompasses additional aspects of human behaviour, including social interactions, personalities, and motivations. NPCs employed to simulate social contexts in virtual environments should be modelled to achieve behaviour that is socially understandable, credible, and ethically accepted. This entails replicating verbal and nonverbal expressions, including facial expressions, postures, and gestures, in a socially compliant manner that satisfies social expectations. For an agent to truly exhibit social behaviour, it must comprehend various situations, recognise social practices, identify inappropriate behaviours from other agents, and act in accordance with relevant social norms. This is a complex task given the intricacies of social interactions. In modelling an intelligent NPC, we draw on the architecture outlined in MET-iquette, where the concept of social practice and in particular the model proposed in [20] play a key role in driving the agent’s deliberation and behaviour.

Dignum has formalised the social framework to address interactions in complex social scenarios, achieving a balance between proactive, goal-oriented actions and reactive, context-driven responses  [20]. He discussed the main methodologies for shaping agents’ deliberation, comparing them with respect to the SP framework by means of relevant examples [15,20]. Specifically, goal-oriented agents might face challenges in selecting the most suitable plan under circumstances where multiple plans exist or when information is limited or uncertain. Opting for a social practice-based approach provides an advantage, enabling goal-oriented plus context-driven deliberation. As an example, in [30], a new method for integrating social practices into BDI agents is explored, introducing a meta-deliberation plan and a Jason meta-interpreter to support expanded functionality tailored for a simulated care robot scenario. Conversely, purely reactive agents face limitations as they rely on rules or cases to fully describe situations. The author examines the similarities and differences between social practices and other approaches like Work Practice Modelling and Agent Organisation. While both share similarities with the social practice model, the former necessitates the complete filling of a frame, and the latter fails to specify crucial information such as times, places, and objects available in the context of an interaction [15,20]. In [31], the formalisation of social practice presents a more extensive approach to modelling the social world compared to traditional deontic formalisations. Rather than solely concentrating on prescribing actions, social practices encompass shared connections between actions, competencies, values, context elements, and other actions within specific social situations. In [32], the authors assess existing agent models, including reasoning models, normative models, and social–psychological models, in terms of their alignment with three critical aspects of social practices: habituality, sociality, and interconnectivity. Their findings indicate that none of these models fully encompass or explicitly address all the intricacies of these crucial features, providing valuable insights for an improved incorporation of social practices into agents. Social practices have been proposed in various application contexts, leveraging their unique ability to serve both social and functional purposes. These applications range from patient-centred planning [33] to modelling the behaviour of NPCs in the metaverse [16] and creating models for social chatbots and robots [14].

Particularly, the primary focus lies on comparing Belief–Desire–Intention (BDI) [18] and Case-Based Reasoning [34]. BDI allows for modelling proactive and goal-oriented agents; however, these agents might face challenges in selecting the most suitable plan under circumstances where multiple plans exist or when information is limited or uncertain. Opting for an SP-based approach in contrast to a BDI approach provides an advantage, enabling goal-oriented plus context-driven deliberation. In [30], a new method for integrating SPs into BDI agents is explored, introducing a meta-deliberation plan and a Jason meta-interpreter to support expanded functionality tailored for a simulated care robot scenario. Conversely, purely reactive agents, exemplified by CBR, face limitations as they rely on rules or cases to fully describe situations. The author also examines the similarities and differences between SPs and other approaches like Work Practice Modelling and Agent Organisation. While both share similarities with SP models, the former necessitates the complete filling of a frame, and the latter fails to specify crucial information such as times, places, and objects available in the context of an interaction [15,20]. In [31], the formalisation of SP presents a more extensive approach to modelling the social world compared to traditional deontic formalisations. Rather than solely concentrating on prescribing actions, SPs encompass shared connections between actions, competencies, values, context elements, and other actions within specific social situations. In [32], the authors assess existing agent models, including reasoning models, normative models, and social–psychological models, in terms of their alignment with three critical aspects of SPs: habituality, sociality, and interconnectivity. Their findings indicate that none of these models fully encompass or explicitly address all the intricacies of these crucial features, providing valuable insights for an improved incorporation of SPs into agents.

Drawing inspiration from the work presented in [30], in our opinion, integrating digital practices into a BDI framework could constitute a strategic choice, aiming to develop agents capable of pursuing goals while adapting to the specific social context.

5. Implementation Phase

In this phase, we propose a general architecture and a set of technologies for implementing MR applications.

The architecture with its layers is depicted in Figure 5. At the highest level is the Organisational level, which establishes a social structure comprising objectives, norms, and rules that the entire system lifecycle must adhere to. The Agent level contains the system’s business logic and intelligence, which is delegated to autonomous, proactive, and socially active entities called agents. The Artifact level models the physical and virtual resources required during the system lifecycle, including digital twins. Finally, the Virtual level encompasses everything necessary for creating the virtual reality, such as avatars, forms, and graphical style.

Figure 5 shows also the components, software libraries and languages we selected to avoid code from scratch and facilitate the implementation task.

• The virtual environment will be developed by using the Unity framework and official libraries (like Barracuda).
• We selected JaCaMo [19] as the common platform for agent-based applications. JaCaMo includes
  • Jason [35] as a BDI language to program rational agents;
  • CARTAGO [36] to define sharable artefacts agents can manipulate; and
  • MOISE [37] to define multi-agent organisations in terms of roles, goals and plans.
• Even if not mandatory, we suggest the WLDT library [38] for speeding up the definition of digital twins.
• To handle the dynamic nature of digital practices, we also exploit results obtained in  [39,40]. In particular, it is a self-adaptive framework based on:
  • GoalSPEC [40] as a language to specify goals-models; and
  • Capability Language [39] to describe abstract and concrete services.

• Mapping Digital Practices into the Architecture

Social practices serve as an analysis tool, whereas digital practices are digital entities that need to be integrated into the architecture. However, we choose to view digital practices as a cross-layer structure, as they possess a multifaceted nature. They have goals and norms, which belong to the Organisation layer, are dynamic and adaptive, which is a typical feature of the Agent level, and possess capabilities and affordances that can be located in the Artifact and Virtual levels, respectively. Figure 6 summarises this mapping.

In order to ensure that digital practices are able to operate autonomously, we rely on agents to carry out their operational tasks. These agents are essential as they collect data from the context or other digital entities and use an activation model to establish dynamic connections between agents and digital twins. A head agent serves as the primary control centre that encapsulates the business logic and oversees the environment’s context and other agents, ensuring that everything is running smoothly and efficiently. With the help of these agents, digital practices are able to function independently and seamlessly.

Goals, norms and in general the social organisation could be defined via the MOISE language. The aim of such language (and its framework) is to develop agents proficient in multi-agent coordination, enabling effective collaboration with both humans and other systems.

In MOISE, an organisation is viewed as a group of roles. When an agent takes on a role, it agrees to follow a specific set of behaviours that work together to accomplish an overall purpose. The functional definition of the organisation allows for the creation of missions, which are essentially collections of goals. Each role in the organisation has specific responsibilities towards these missions. During run-time, the MOISE framework ensures that the agents’ actions are coordinated towards achieving the system’s overall goals [37].

6. A Case Study

The rapid development of MR technologies has opened up new possibilities for enhancing the quality of life of people who face various challenges in their daily activities. One such challenge is the difficulty of accessing physical grocery stores, which may be due to mobility/health issues or geographical constraints. In this case study, we present a scenario where a virtual grocery shop in MR can provide a satisfying and engaging shopping experience for an older person who cannot visit her preferred grocery store by herself. From the analysis of requirements, we face the social implications of creating a realistic and immersive representation of a grocery store in the metaverse, where different norms and expectations may apply. Then, we describe the design features and will show some expected benefits of implementing it through the proposed methodology.

6.1. Identifying the Social Practices for the Grocery Shop

We exploited goal-oriented requirement engineering (GORE) techniques to collect the involved users’ needs [41] in the form of social practices. In the following, we report an example of social practice for the grocery shop scenario.

Context

Actors

• Physical: Customers and employees physically present in the grocery shop.
• Virtual: Avatar of remote customers or employees in the grocery shop.

Roles

• Customers: Digital or real users purchasing food and/or other household items.
• Employees: Digital or real users assisting customers and maintaining the store.
• NPC agents have the role of providing support.

Resources

• Physical: The products in the store, shopping carts or baskets, cash registers, and other tools or equipment used to maintain the store.
• Virtual: Virtual representations of products, shopping carts or baskets, and other tools or equipment used to navigate and interact within the virtual environment.

Places

• The main places are the grocery shop and its virtual representation in the metaverse.
• Both the physical and virtual shop may be specific areas such as aisles, checkout lanes, and customer service desks where interactions between customers and employees are more likely to occur.

Affordances

• The store layout might make it easy for customers to find employees when they need help or to interact with items.
• Signage and other visual cues might also help guide customers to areas where they can interact with employees.

Time

• The time aspect of this context might include the hours of operation of the grocery shop and any peak times when the store is particularly busy.
• The length of time customers spend in the store and the frequency of their visits might also be relevant.

Meaning

Purpose

• Physical and Virtual: Receiving help finding an item, asking a question about a product, providing feedback, or making a complaint.
• Only Virtual: Providing feedback on the virtual shopping experience or participating in interactive experiences.

Promoted Values

• Physical and Virtual: The values promoted by this social practice include helpfulness, politeness, and respect. For example, customers might value knowledgeable employees willing to help them find their needs. Employees might value customers who treat them with respect and follow the store’s policies.

Counts-as

Successful interaction between customers and employees might depend on the specific purpose of the interaction:

• If the purpose is for a customer to receive help finding an item. In that case, a successful interaction might involve the employee providing clear and accurate information about where to find the item.
• If the purpose is for a customer to make a complaint, then a successful interaction might involve the employee listening attentively and taking appropriate action to address the issue.

Expectations

There are certain expectations that both customers and employees might have when interacting in a grocery shop. For example, customers might expect employees to know the store’s products and provide helpful and polite service. Employees might expect customers to treat them respectfully and follow the store’s policies.

Plan Patterns

The plan patterns for interactions between customers and employees in a grocery shop might vary physically and virtually.

• Physical and Virtual: If a customer needs help finding an item, the plan pattern might involve the customer approaching an employee and asking for help. The employee might then provide information or guide the customer to the item’s location.
• Virtual: Using virtual assistants to request help or to find an employee.

Norms

• Customers are expected to treat employees respectfully and follow the store’s policies.
• Employees are expected to provide helpful and polite service to customers.

Start condition/Duration/End Condition

• The interaction might begin when a customer approaches an employee and asks for help or when an employee greets a customer as they enter the store.
• Some interactions might be brief, such as when a customer asks for directions to a particular item. Other interactions, such as when a customer has a complex question or complaint, might be longer.
• The interaction might end when the customer’s question has been answered or their issue has been resolved. Alternatively, the interaction might end when the customer leaves or moves to another store area.

Activities

Possible Actions

• Physical and virtual: Asking for help finding an item, making a complaint, providing feedback, or simply engaging in small talk while checking out.
• Only Physical: Other possible actions include employees restocking shelves, assisting customers with self-checkout machines, or helping customers bag their items.
• Only Virtual: Using virtual tools or menus to request assistance, participating in interactive experiences or games, and providing feedback on the virtual shopping experience.

Competences

• Good communication skills, knowledge of the store’s products and layout, and the ability to handle customer complaints or issues professionally.
• Customers also need the ability to navigate the store and locate items independently.

6.2. Implementing the Virtual Grocery Shop

This section includes an example of the methodology and architecture applied to the case study of the metaverse grocery shop. When social practices are identified, we can focus on roles’ expectations and exploit goal-oriented requirement engineering (GORE) techniques to generate the corresponding digital practice. Figure 7 provides the model for the Assist/Serve Customer in Peak Time goal. In this case, we are supposed to be at a peak time when employees are very busy; thus, virtual assistants can reduce their workload by being more proactive in assisting remote customers.

This is an interesting use case because it is a mixed scenario involving agents (the virtual assistant) and digital twins (customer digital twin and inventory digital twin). The main abstraction of this diagram is the goal (rounded rectangles), i.e., a functional objective to be addressed. The root goal (Assist/Serve Customers in Peak Time) is refined via AND/OR decomposition relations with the intuitive meaning of providing simpler sub-goals. The process is recursive, generating a decomposition tree. The goal model, for instance, describes how there are two cases in which a virtual assistant materialises near a customer: the customer asks for help, or the virtual agent autonomously understands she requires assistance. The goal model also reports some norms (triangles) that hold during the operations (e.g., employee provides polite service).

An example of a MOISE specification for the Assist/Serve Customers in Peak Time social practice is reported below:

<organisational-specification>

  <scheme>

    <goal id="assist_customers_in_peaktime" type="and">

      <goal id="virtual_assistant_comes_to_customer" type="or">

        <goal id="virtual_assistant_understands" type="and">

          <goal id="scene_analysis"/>

          <goal id="get_data_from_customer_dt"/>

        </goal>

        <goal id="customer_asks_for_help"/>

      </goal>

      <goal id="virtual_assistant_solves_customer_issue"/>

    </goal>

  </scheme>

  <role id="virtual_assistant">

    <mission id="assist_customer">

      <goal id="scene_analysis"/>

      <goal id="get_data_from_customer_dt"/>

      <goal id="customer_asks_for_help"/>

      <goal id="virtual_assistant_solves_customer_issue"/>

    </mission>

  </role>

  <norm>

    <role id="virtual_assistant"/>

    <mission id="assist_customer"/>

    <obligation>polite_service</obligation>

  </norm>

  <norm>

    <role id="customer"/>

    <obligation>follow_store_policy</obligation>

  </norm>

</organisational-specification>

This analysis allows the engineer to reason about the kind of digital entities that must be conceived. The first, trivial one, is the Virtual Assistant who is in charge of helping virtual customers, thus reducing the employee border. This assistant is an autonomous NPC that will take charge of some of the goals of Figure 7. Given its nature, this assistant will be implemented through a Jason agent. Analysing the context for this agent, we discover that it will monitor customers in the scene to understand if they need assistance and the employees in the shop to check if they are too busy to help. Therefore, customers and employees are part of the context, and we need to introduce two digital twins to have a real-time representation. Other elements defining the context, that we omit for the sake of space, are time (is it a peak time?) and space (is the customer in front of a shelf for a long time?).

Figure 8 provides an abstract representation of the identified agent and the two digital twins. The agent is represented as a box with a name and three compartments: Plan, Capabilities and Knowledge (Facts and Rules). This is just a conceptual representation that highlights how the agent can act in the scene. Similarly, the two digital twins are represented as two boxes with three compartments: Models, Capabilities and Views (coherently with Section 4.1). This kind of view is useful because the engineer can establish a mapping with the social practice. Indeed, Plan, Capabilities, Knowledge and Context are all elements of a digital practice, facilitating its creation.

Now that we have all the elements, we can exploit the presented architecture (Figure 5). We follow the prescribed separation in levels (Virtual, Artifact, Agent and Organization), adding the Physical level to resemble the link with the real world. Figure 9 is an example of how our elements are located in the architecture and how they interact.

The higher level, the organisational, comprises stakeholders’ goals. These will be formalised using GoalSPEC and generate a dynamic MOISE organisation [42]. Its objective is to orchestrate the participating agents by assigning them roles and responsibilities. The choice to generate the MOISE structure from the chosen goal is strategic for adapting the system behaviour according to the context. Indeed, the agents’ organisations cannot be statically defined: different goals will demand different behaviours.

The Agent level describes which autonomous and proactive digital entities live in the system. Relying on the chosen agent framework, we may have three categories of agents:

• Virtual assistant (or personal agents) sensing and acting in the environment, providing services and interacting with humans or other agents in the environment;
• Digital-twin-manager agents, in charge of encapsulating the autonomy and proactivity of some complex digital twins; these agents respond to the need to have a clear separation of concerns (as clearly discussed in [36], where digital twins are used to mirroring some aspects of physical things, whereas business logic, reasoning, and in general intelligent and proactive behaviour are more pronounced for an agent;
• Digital-practice-agents that are in charge of ensuring the activation and fulfilment of digital practices.

In this example, we preferred limiting intelligent behaviour to the Virtual Assistant, which is the unique agent of Figure 9. It can observe the scene (by accessing digital twins) and decide when and how to assist customers.

The Artifact level mainly comprises digital twins. Even if, for clarity, Figure 9 reports only Employee, Customer and Product, the Artifact level also contains all the digital twins of the scenario.

The Virtual level relies on the Unity framework (version 2022.1), which is one of the most popular cross-platform graphics engines for developing VR applications. Unity is easy and intuitive to use and has a strong community of developers. The main problem of integrating Jason agents into Unity is that this latter does not natively support Java programming. To overcome this problem, an HTTP REST server has been developed to ensure greater flexibility in the interaction of the virtual environment with agents and digital twins and vice versa (we used the UnityWebRequest library). The virtual level is mostly a 3D reproduction of a grocery shop populated with real persons’ and virtual assistants’ avatars.

Whereas the virtual scene was created directly within Unity, the 3D modelling software Blender was used to sculpt human-like avatars. Blender is open-source software that is particularly complete for complex shapes like human bodies and faces.

7. Conclusions and Future Works

This paper proposes a methodology as a software engineering paradigm to design MR applications with an important physical component, particularly where humans are in the loop. The design paradigm has been extended to create a virtual ecosystem where humans and autonomous digital entities can cooperate adhering to established social consensus. The requirements of the domain have been analysed according to the social practice theory to obtain a semi-formal notation that led to the concept of digital practice and involves digital twins and software agents as autonomous and proactive decision-makers. The proposed methodology aims to address the need to include in the metaverse social NPCs able to recognise practices and modify their behaviour accordingly. The paper discusses the application of the proposed methodology to a case study scenario consisting of engineering a virtual grocery shop as a possible application of MR to help older adults reduce their social isolation. In particular, we choose a fine-grained perspective in which the system is seen as a set of interacting digital twins. The interactions and synergies among the different digital twins has been modeled exploiting the concept of digital practice allowing the integration of the individual perspective of a digital twin with the social perspective and explaining how context influences individual behaviours. Moreover, the preliminary application of the proposed methodology to the case study allowed us to show how to model digital practices and how these can increase the flexibility, scalability and adaptivity of software systems running in dynamic environments.

The application of the methodology we propose to a single case study represents one of the limitations of this paper. However, the application of this modelling methodology to different application domains that include a strong social component as well as the comparison with other modelling methodologies will be part of future works.