Mixed-Reality Demonstration and Training of Glassblowing

Abstract

Traditional crafts exhibit tangible and intangible dimensions. Intangible dimensions include the practitioner’s gestural know-how in craft practice and have received smaller attention than tangible dimensions in digitization projects. This work presents the process of representation and presentation of the glasswork and is exemplified in the re-creation of a historical object. Following an articulated pipeline approach for data collection, annotation, the crafting process is represented visually and semantically in a way that can be meaningfully presented and utilized in craft training and preservation. The outcomes of the proposed approach were used to implement a Mixed Reality installation. The installation targets craft presentation through an exploration of the workspace, as well as craft training through an interactive experience where users re-enact gestures of a glass master holding a tool and receiving audiovisual feedback on the accuracy of their performance. Preliminary evaluation results show high acceptance of the installation and increased user interest.

1. Introduction

Traditional crafts (TCs) are a Cultural Heritage (CH) and exhibit tangible and intangible dimensions. Tangible dimensions regard craft articles and products, materials and tools, as well as natural resources, built workshops, and workplaces. The tangible aspect of TCs is evident in their practice, where materials are transformed with the use of tools, but also skill and knowledge. According to UNESCO, crafts are probably the most tangible of intangible heritage [1]. Comparatively to other forms of Intangible Cultural Heritage (ICH), TCs have received smaller attention by digitization projects [2]. Intangible dimensions regard technical knowledge as well as the socio-historic content of the communities and regions in which they are, or were, practiced. In this work, our effort is to represent these dimensions for the TC instance of glassworking, using an articulated approach to the digitization, representation, and preservation of TCs.

Glasswork is a TC that combines hand and body gestures and a thorough understanding of the material. Glasswork is a challenging craft because, during production, the material changes states from liquid to solid. Such transitions pose requirements in terms of material handling. In this work, we are challenging the technical means for capturing and conveying these sensory aspects of glasswork. The artistic skill and technical knowledge needed for the production of (luxury) glass means some production methods, decoration techniques, and styles are strongly linked to a specific geographical location.

In this work, we focus on glasswork to study the requirements of dexterous aspects and tool manipulation in craft presentation and preservation. The Conservatoire national des arts et métiers (CNAM) hosts a museum of technological innovation in Paris and contains objects related to both the artistic and more industrial production of glass. Furthermore, CNAM hosts historic archives regarding the artifacts and techniques under study and of course the original objects to learn from and digitize. We further study the personal stories of the practitioners to show the intricate interweaving of local traditions and individual lives in cultural transmission.

The requirements of the signal and semantic representation of TCs for their preservation along with proposed presentation techniques are laid out in [3]. In this paper, possibilities of representing and making accessible both tangible and intangible aspects of glassblowing as cultural heritage (CH) are presented. Glassblowing involves craft artefacts, materials, and tools and encompasses craftsmanship as a form of Intangible Cultural Heritage. Intangible CH dimensions include dexterity, know-how, and skilled use of tools. The outcomes of this effort are made available to the public through the set-up of an immersive MR exhibition at the premises of the museum of CNAM (see Figure 1).

2. Background

Approaches to the digital representation and presentation of TCs are relatively new because past research efforts have been focused mainly on the representation of tangible heritage. Of course, in the social sciences and humanities, such representation has received more attention. In this work, social science and humanities research is combined with interactive technologies for CH under a solid theoretic basis for the representation and presentation of TCs [2]. Furthermore, on the representation part, studies on the usage of mixed reality installations in the museum content provide valuable insight regarding the technical possibilities for exploiting the representation.

2.1. Craft Representation and Craft Presentation

TCs bring together the intangible dimensions of dexterity, skill, tradition, and knowledge, with tangible elements such as tools, machines, and materials to be transformed into craft articles. The documentation of TCs through ethnography is a common practice, e.g., [4,5], and there is significant work towards generic approaches to craft representation [6]. From the intangible dimensions, some relate to the cultural, economic, or religious contexts that identify communities. Others relate to the technical context of a specific craft and the corresponding skills, knowledge (or “know-how”). In [7], social context, group membership, and social relations relevant to crafts are reviewed. Guidelines for preservation projects [8] stress the representation and consent of the pertinent community. In addition, intangible dimensions regard the resources of the environment that a craft flourishes in and, as such, regard environmental and climate context [9].

A systematic approach to the digital representation of tangible and intangible dimensions is outlined in [2]. In [10], the requirements towards such a representation are investigated so that the represented content is sufficient to support the authoring of accurate narratives, curation of educational material, and experiential presentations. In [3], an approach towards the specification, acquisition, and representation of pertinent data and knowledge for HC preservation is proposed. In [11], Motion Vocabularies are proposed to associate recordings of craft practice to craft actions and remap the recorded human motion to pieces of virtual machinery whose function is simulated. As a complement to Mingei, systems which leverage semantic information include automatic image annotation [12] and recommendations for users [13].

2.2. Interactive Technologies for CH

Cultural Heritage Institutions seek new ways to attract and engage new visitors. One of the ways to obtain competitive advantage is the investment and implementation of interactive experiences on site [14]. Over the years, several technologies have emerged, each of which provides forms of interaction and various levels of immersion but also poses requirements in terms of space, setup, and deployment. In this subsection, the most prominent of these technologies are presented.

Virtual Reality (VR): An example of a high immersion VR environment for CH is Kivotos [15], a VR environment that uses the CAVE^® [16] system, in a room of 3 m by 3 m, where the walls and the floor act as projection screens and in which visitors take off on a journey thanks to stereoscopic 3D glasses.

Augmented Reality (AR): AR offers more advantages to museum visitors considering that information can be overlaid upon video frames captured by a camera. AR has been applied to make it possible to visualize incomplete or broken real objects as they were in their original state by the superimposition of the missing parts [17]. The ARCO system [18,19] provides customized tools for Virtual Museum (VM) environments, ranging from the digitization of museum collections to the tangible visualization of both museum galleries and artworks. In [20], Virtual Humans (VHs) are used in an AR setting to present narratives on the everyday life of workers in a mastic factory narrated by the workers themselves appearing virtually within the exhibition.

Mixed reality (MR): MR relies on a combination of VR, AR, and the real environment. According to Milgram and Kishino’s virtuality continuum, real-world and virtual world objects are presented together on a single display (e.g., the screen of a mobile phone) [21] that displays the visual representation of both the real and the virtual space [22]. An example of the use of MR techniques in a museum environment is the Situating Hybrid Assemblies in Public Environments (SHAPE) project [23] that uses hybrid reality technology to enhance users’ social experience and learning in the museum and other exhibition environments, concerning cultural artworks and to their related contexts.

X-Reality: AR, VR, and MR applications, when coexisting in a physical context, are referred to as X-Reality (Extended Reality) or XR applications [24]. The use of such technologies has the potential to enrich the information of cultural heritage artifacts and museum exhibits and turn passive visitors into active participants engaged in an interactive and immersive blend of physical and virtual as if it was a single, unified world [25].

2.3. Mixed Reality Installations for Education and Training

The work presented in this paper falls into the category of MR applications for the CH context. According to [26], most of the MR methods support partial immersion, and few of them support full immersion. VR Head Mounted Displays (HMDs) thus support full immersion, whereas AR and holographic AR support partial immersion.

In this context, several attempts have been made over the years to integrate MR technology into the museum both as a standalone experience and by blending technology with the museum exhibitions. In [27,28], a Computer Vision approach is proposed to support non-instrumented, location-based interaction of multiple users with digital representations of large-scale artifacts. A similar approach is followed in [29], where information displays and educative applications are combined with infotainment systems and interactive games for children. Furthermore, other attempts aim not only to provide information but also to personalize content provided in the museum based on the interactive device used and the characteristics of the user accessing an interactive application (e.g., children, families, adults, experts, etc.) [30,31]. Previously presented approaches are focusing on the existence of a physical museum exhibition. Lately, with the emergence of Virtual Reality, such experiences can be provided in a purely virtual form both on the Web and through VR-based virtual museums [32].

The majority of technologies presented above focus on the information provided by museums. TCs and their preservation entails also the need to enhance knowledge on craft practice and techniques through interactive demonstrations and hands-on practice.

2.4. Importance of Intangible Cultural Heritage for Sustainable Tourism

The wealth and variety of expressions and forms of ICH is steadily becoming a principal motivation for travel around the world. Many forms of ICH tourism are associated with longer duration of stay than tangible CH sites (i.e., archaeological sites), which is required for appreciating and experiencing ICH [33]. The UN World Tourism Organization (UNWTO) recognizes that an important challenge lies in identifying, protecting and safeguarding ICH by investing in sustainable tourism development, in consultation with local communities and other stakeholders [34]. In this context, community participation and engagement through inclusive co-design and co-curation activities will increase the acceptance and participation by local communities, which are central in the success of tourism services. As recommended, stakeholders will “enjoy the benefits of tourism development” and “establish projects with communities” [34]. Trends in the Cultural and Creative Industries [35] show that availing content electronically, on site and via the WWW results in increase of interest and financial impact.

In line with the Faro Convention, technical approaches for ICH should demonstrate the value of HCs in “sustainable development, cultural diversity and contemporary creativity” [36], meaning that profits contribute and motivate preservation of HCs. By providing HC representations and competitive, educational, and compelling tourism experiences, it is possible to increase visibility of HCs and preserve this form of ICH as a “shared source of remembrance, understanding, identity, cohesion and creativity” [36]. These properties follow the UNWTO recommendation to actively support “initiatives that follow international best practice for documentation, the use of information technologies and the communication of ICH values” [34].

2.5. Cultural Context

This research work focuses on a unique item of the collection of the museum of the Conservatoire des arts et métiers, a glass carafe. This work entails a challenge which was the reconstruction and representation of the process of its creation through an experimentation approach similar to experimental archaeology. Fourteen carafes were crafted at the premises of CERFAV (Centre européen de recherches et de formation aux arts verriers) in the process of studying and capturing the process of recreating the studied artifact.

2.5.1. About CNAM

With origins dating back to 1794 and the French revolutionary period, the Conservatoire des arts et métiers has a long and complex history. It was very clearly established as a place of emulation, where a repository of objects, a library, and a draughtsman office would be open to the public so that innovation would be encouraged by all means. It inherited from prestigious collections, such as the royal cabinets (including Vaucanson’s cabinet at the Hotel de Mortagne), the repository of the Academy of science as well as objects seized from the émigrés and soon started collecting models, tools and objects. With the ambition of serving the mechanical arts, it was also a place for applied science. In 1819, three chairs were established: industrial economics, chemistry applied to the arts, and mechanics. Soon others would be created to cater to the need of the developing industry. Workers or small entrepreneurs willing to improve their trade were the intended public because all courses were free and lectured after working hours. The emphasis was on applied science and practical knowledge.

2.5.2. Rationale for the Selection of the Carafe as a Study Material

Professor Eugène Péligot [1], a chemist with a keen interest in glass manufacturing, asked one of the most prominent glassmakers of the day, Georges Bontemps, when it came to selecting objects for the galleries regarding his lessons on glassmaking.

In 1842, Bontemps delivered more than fifty pieces, both tools (molds and blowing apparatuses) and finished objects. All pieces were dated from 1842 and originated from the Choisy-le-Roy glassworks. In 1868, Bontemps will make a further donation, in his name, objects relating to his famous Guide du Verrier, published that same year. Among them were two sets of objects, revealing the process in a sort of «frozen state», a pattern much favored in pedagogical dimension at the Musée des Arts et Métiers.

The aim in selecting the carafe, in this work, was to document each manufacturing step, as exemplified by the «frozen» glass artifacts—the rough out foot and the finished one, the handle, etc.—with a recording of the corresponding gestures and bring to life the objects. Furthermore, it was interesting to compare and document the historical tools with those still in use today. To this end, some research was needed in the archives because little was known of the actual teaching of Péligot, although he published two books [37] on the subject. The displays at the museum of CNAM related to the carafe and glassblowing tools are presented in Figure 2 and Figure 3.

2.6. Direct versus Indirect Experiences for Presenting Cultural Context

To date, little work has been done regarding the effects of direct and indirect experiences for presenting cultural context. This fact is more urgently discussed today because CH is complemented by novel digital means, and the COVID pandemic has put great pressure on CHIs to create online digital encounters. To this end, CHIs made an effort to enhance their online presence, and online activity became the main way of approaching the public [38].

In a broader context, research has studied the effects of direct and indirect experience on the production of affective and cognitive responses. In these studies, it is hypothesized that direct experience with an object would tend to produce more affective reactions than indirect experience with the object. Alternatively, indirect experience produces more cognitive reactions [39,40]. Furthermore, the effect of such experiences in learning has been studied in the context of team creativity by examining the effects of task experience acquired directly and task experience acquired vicariously from others on the team. The studies show that direct task experience leads to higher levels of team creativity and more divergent products than indirect task experience [41]. Similarly, on learning, research findings suggest that the direct experiences catalyzed learning outcomes into a stronger motivating force than it had been during the indirect learning experiences [42].

Today, exposure to virtual experiences is becoming common in various contexts. In some cases, these precede both indirect and direct experiences. Research outcomes support that virtual experiences from 3D product visualizations are more similar to direct experience than to indirect experience [43].

A more complex study was done on the combination of experiences of sequential combinations of consumer experiences. Four kinds of sequential combinations of consumer experiences were designed with the results indicating that Virtual Experience followed by a Direct Experience produces the highest product knowledge and brand attitude [44]. Taking into account the aforementioned qualities of experience in this work, we follow a mixed approach where virtually experiencing the craft is followed by user participation in hands-on training experiences.

2.7. Contribution of This Work

Despite their cultural significance efforts for HC representation and preservation are scattered geographically and thematically. In this work, we present an approach to establish HC representations based on digital assets, semantics, existing literature and repositories, as well as mature digitization and representation technologies. These representations aim to capture and preserve tangible and intangible dimensions of HCs.

In the case of glassblowing, central to craftsmanship is skill and its transmission from master to apprentice. In this work, the motion and tool usage of glassblowers is captured in order to preserve and illustrate skill and tool manipulation. The represented knowledge is availed through compelling experiential presentations, using storytelling and educational applications and based on MR.

3. Crafting Process Modeling and Representation

An articulated approach to the documentation and representation of traditional craft processes is proposed in [45]. In this subsection, we report on following the steps of this approach for glasswork. The technical implementation of the approach for the formal and digital representation of the crafting process is supported by the Mingei Online Platform (MOP) [46], where digital assets and semantic meta-data are organized in a formal representation compatible with contemporary digital preservation standards. The methodology is summarized in Figure 4.

3.1. Craft Understanding

Craft understanding follows ethnographic principles and includes background research of secondary sources. We start through a study on the literature resources as background research before the ethnography increases its efficacy, saving time from the comprehension of basic vocabulary and notions.

The ethnographic work involved interviews with the practitioners and fieldwork observation to dive deep into the individualities of the crafting process. This understanding is supported by audiovisual recordings and documentation of interviews and demonstrations. The detailed ethnography enables understanding and supports documentation of gestures, techniques, and steps of a crafting process.

Following [47], a craft understanding has two outputs. The first is a vocabulary of terms with verbal definitions and visual descriptions of the involved objects (nouns) and actions or activities (verbs). The second is a thick description that enables the study of the activity beyond the content of a visual demonstration.

3.2. Data Collection

In this step, the components of the crafting process are digitally recorded, post-processed, and curated in an online platform.

These digital assets record objects and actions, acquired from documenting photographically and in 3D materials, tools, products, and workspaces and recording practitioner crafting actions, in recording sessions. The organization of these sessions is facilitated by the vocabulary and storyboard, in identifying the objects, sites, and practitioner actions to be digitized. Moreover, some of the photographic and video assets may be recorded in combination with ethnographic fieldwork.

A requirement relevant to the glasswork, but also relevant to other crafts, is that the viscous nature of hot glass prohibits the pause of the crafting process during some sequences of tasks. As such, motion files may contain multiple process steps and actions.

3.2.1. Audiovisual Recordings

Pertinent assets regard conventional audiovisual data acquired from the ethnography such as audio and video interviews, photographic documentation and documentation to be used for craft representation including photographs and video documentation of objects, spaces, and demonstrations.

Audiovisual recordings of the crafting process are important for overview and ethnographic study, as well as for documenting significant aspects of the process with a local spatial expression, such as tool gripping postures, detailed and minute manipulations, and so on. Moreover, multiple and/or wide-angle bird-view cameras can be employed to capture the motion of multiple practitioners and their movement within. In the case of glass workshops, the latter is particularly important because glass workshops are shared by multiple practitioners and because the arrangement of workshop machinery is such that the practitioners have to walk from one locus to another during the crafting process.

3.2.2. Documentation of Tools, Machinery, and Workspaces

The documentation of tools, machinery, and workspaces is both photographic and in 3D. We refer to [48] for the photographic documentation of static artifacts and scenes, and to [49] for their 3D documentation. The 3D documentation of metallic tools and glassware can be challenging. We distinguish between the potential historic significance of an object and its utility as a tool, whereas in the latter, solely their geometrical structure can be sufficient. In some cases, the 3D model of tools can be easily modeled or even provided by the manufacturer. The use of synthetic models can simplify the digitization tasks and significantly reduce scanning costs. The collection of tools for glasswork was developed in [50] and is summarized in Figure 5.

Similarly, for the creation of 3D models of machinery, the acquired audiovisual documentation acquired during the ethnographic research was used, and the machines were created from scratch using 3D modeling software. The outcomes of the modeling process are presented in Figure 6.

The digitization of transparent objects is still an open problem in 3D digitization. As such, for those, we rely on photographic documentation and conventional by measurement and technical drawing.

3.2.3. Motion Capture

The applicability of Motion Capture (MoCap) and video modalities depends on the type of environment. Inertial MoCap [51] is more suitable than optical [52] in the cluttered space of workshops, due to reduced installation requirements and independence to occlusions. Nevertheless, inertial MoCap is not sufficiently sensitive to minute motions. A practical issue was that it was difficult for the practitioner to manipulate objects with the gloves of the suit. To solve that, latex gloves were worn on top of the suit’s gloves to add friction. Markerless methods exhibit the least accuracy [53,54] but require only a camera. We found markerless motion recording suitable mainly in obtaining key hand postures and body gestures.

In the context of this research work, Motion Capture occurred during the ethnographic fieldwork at CERFAV, in September and October 2019. During this, fourteen carafes were produced in the workshop to observe and capture social and body interactions that take place during glasswork. Apart from the audiovisual documentation acquired, the process was recorded using MoCap equipment and, in particular, the Nansense MoCap suit and gloves.

Table 1. Tasks recorded at CERFAV.

Insert the glass into the furnace	Moving the blowpipe	Shaping glass with the hand	Rotating the blowpipe	Blowing through the blowpipe	Shaping the glass with the tweezers
Burn the base with the torch	Blowing through the blowpipe	Shaping the glass with the block	Pressing the glass on a metal base	Cutting the glass with the shears	Shaping the glass with the block
Shaping the base of the carafe	Shaping the glass with the paddle	Shaping the glass with the jacks	Shaping the base of the carafe	Adding the handle to the jug	Shaping the glass with the tube

illustrates the tasks that were recorded for glassblowing.

3.2.4. Data Curation

The activities presented in the previous subsection create a large amount of data that should be rationalized, post-processed, and curated to be exploitable for future research. In this context, photographic documentation, video recording of the creation process, 3D objects, and 3D reconstructions are uploaded to the Mingei repository and documented in MOP. Motion recordings are also stored in the repository and documented in MOP to formulate Motion Vocabularies. All digital assets receive unique IRIs for semantic interoperability.

Photographic, audiovisual, and 3D assets that are ingested in the MOP can be browsed by file properties and by thumbnail preview, as shown in Figure 7.

3.3. Knowledge Representation

The objects and actions of the crafting process are semantically represented in this step. The representation uses a few classes, called basic knowledge elements, which contain links to semantic metadata provided by the user and links to the digital assets formed in the previous step.

From motion recordings, reference postures and gestures are identified by the user. To associate frames and segments from motion recordings with postures and gestures respectively, the AnimIO annotation editor [45] was employed, which facilitates body-member-specific annotation of motion recordings. To represent tool and machine usage, motion recordings and 3D models are combined. The recorded gripping postures come into use in this task.

The recordings are combined under the context of the Event knowledge entity, which contains links to the representations of the Location, Participants, Tools, Materials, and (intermediate) Products pertinent to the event. Conceptually, events align with the steps of the process, and both can be hierarchically analyzed in subevents and substeps.

In MOP [46,55], all knowledge elements are created through simple form filling operations. Each type of element has a dedicated Web form where the relevant meta-data are edited. Furthermore, facilities to identify links with other knowledge elements are provided. Links may be provided in the form of a Universal Resource Identifier for external resources or in the form of semantic links for digital items curated in MOP. Such elements could be, for example, the linked media objects that are relevant to the knowledge element.

3.4. Workflow Representation

To encode craft understanding, activity diagrams are borrowed from Unified Modeling Language (UML) [56,57] and used in the following sense. While UML representing computational actions that transform data are represented, in this work, physical actions that transform materials are represented. The transition types Transition, Fork, Merge, Join, and Branch are adopted and denoted as in UML.

Activity diagrams can be refined hierarchically, allowing the increase in representation detail at later stages. Moreover, their visual nature was found to support the collaboration with practitioners. Activity diagrams are essential in representing transitions between process steps, in which distinct tools are employed and actions take place—and, in particular, decision points and parallel tasks in the process.

The progression of sequential steps is modeled by a Transition link. Forks are used to represent the initiation of two parallel tasks. In Merge transition, two or more control paths unite and Join connects steps that should be completed before the transitions to the next step. Merge and Join transitions are structurally similar, but a Join is a synchronization across a set of parallel flows, while in a Merge, only a single flow is active. Finally, Branch transitions connect a step with a decision step that accepts tokens on one incoming edge and selects one outgoing alternative. Branch nodes control the flow of a process by selecting one of several alternatives, based on the outcome of a condition evaluation.

3.5. Process Schema Representation

In this step, the activity diagram is transcribed into a transition graph. The MOP UI facilitates the structure and enables the instantiation of process schemas and their steps. Data fields are used to enter appellations, informal descriptions, and step order. Transitions are instantiated via dynamic UI components that adapt to transition type. An example of a represented process schema is presented in Figure 8.

3.6. Process Representation

Processes representations account for the events that took place during the execution of a process schema. Intuitively, a process is an individual flow of events, out of those possible in the activity diagram. The MOP UI enables the instantiation of processes representation via the entry and chronological ordering of events, accompanied by the recordings that document them. An example of a process representation in MOP is presented in Figure 9.

4. Craft Presentation and Preservation

4.1. Digital Preservation

The digital assets hosted in the MOP repository are provided online in conventional and open formats. Each asset has a unique IRI to be directly integrated by third parties. Our knowledge is available to the Semantic Web via the MOP and the SPARQL endpoint exposed. Furthermore, to ensure compatibility with online knowledge sources, definitions of terms are imported to MOP through linking to terms from the Getty Arts and Architecture thesaurus and the UNESCO thesaurus. For further exploitation of semantic knowledge encoded in MOP, an EDM export facility has also been implemented, allowing (a) export of data in semantic compatible to EDM format and (b) formulation of SPARQL queries to the MOP SPARQL endpoint to receive EDM formatted results.

4.2. Craft Documentation

The represented knowledge network is available through the WWW and the MOP in hypertext format. Semantic links are implemented as hyperlinks that lead to the pages of cited entities. Contents are also organized and presented thematically, per class type. A keyword-based search is also provided. Documentation pages contain links to digital assets, textual presentation of metadata, and previews of the associated digital assets. For locations and events, specific UI modules are provided. For locations, embedded, dynamic maps are provided through OpenStreetMap [32]. Timeline and calendar views are available for events.

The vocabularies formulated in the first step of the craft representation are provided as illustrated vocabularies of tools, which bring together verbal descriptions and visual recordings. In the same way, the steps where a specific tool is used can be retrieved, along with video recordings of such actions—and similarly for the tools and materials required for a certain process.

Processes are presented containing links to the recordings of the knowledge elements for the tools and materials involving the participating practitioners, the date, the tools employed, and the location of the recording. If the process follows a process schema, a link to that schema and its preview are also provided. The hierarchy of process steps is presented using insets, each one presenting textual information and previews of the available digital assets. To present step organization, insets are dynamically unfolded to any depth of the process hierarchy, associated with image previews and embedded videos. Variations include images and textural descriptions.

In some cases, through the post-processing of the original digital assets, we can provide simplified visualizations for illustrative purposes. Such visualizations reduce the information provided, for example, by a video recording to the essential parts of the craft to be presented. A collection of enhanced iconic abstractions of tool usage gestures is presented in Figure 10. For example, in the first row of Figure 10, the picking up of hot glass material is presented in the first two figures while the other two present the creation of form using a hand-held tool. Using the same rationale, several steps are also visualized in this figure in a form of abstraction that presents only the tools and their interaction with matter.

4.3. Craft Demonstration

4.3.1. Workshop Implementation in 3D

For the implementation of the workshop in 3D, the High-Definition Rendering Pipeline (HDPR) offered by Unity3D [58] game engine was used. HDPR offers various features that contribute heavily to how realistic a scene can look, like physically accurate lighting, multiple out-of-the-box material types (Subsurface Scattering, Anisotropy, Iridescence, Specular Color, and Translucency), and several post-processing effects (Ambient Occlusion, Auto-exposure, and Screen Space Reflections). Lighting in HDRP is physically accurate and uses lighting units such as lumens, lux, and candela.

The environment was set up to use an HDRI Sky for the skybox and the ambient lighting. The fog was also added to give some density to the atmosphere and interact with the sunlight within the workshop. Machinery modeled as presented above was also imported and instantiated in the scene based on reference photographic material.

The lighting plan is using lightmaps for all static geometry with the use of Mixed Lights, which provide real-time direct lighting, but its indirect is baked. For dynamic geometry, e.g., characters, Light Probes will be used in the next iteration. Extra care was put in creating semi-transparent materials. Reflections were also added in the scene with the use of Reflection Probes. The resulting workshop is presented in Figure 11.

4.3.2. Implementation of Virtual Humans and Animation

The virtual human bodies and clothes are created to obtain one unified and optimized model, enhancing the visual impact of the characters with texture mapping and material editing. The 3D generation of the virtual bodies also must take into consideration the total number of polygons used to create the meshes in order to keep a balance between the 3D real-time simulation restrictions and the skin deformation accuracy of the models.

Avatars are created with a combination of different software: Adobe Fuse CC/Mixamo [59] is used for creating the body character, the clothes, the hair, and the rigging. The generated model is then imported into Autodesk 3DS max [60] for mesh geometry optimization. Manual methods, by using the editable poly tools, is preferred because it allows us to keep the regularity of the topology while the automatic methods generate a mess geometry which is not suitable for skin deformation nor a regular UV texture map generation.

The processing of the files has been done on Autodesk MotionBuilder software, Version 2019. 0. 1 [61] which is dedicated to animation and the direct integration of motion capture technologies. The process requires the following steps as presented in Figure 12:

• Creation of an “actor” in MotionBuilder with skeleton definition corresponding to the Biovision Hierarchy (BVH).
• Transposition of the received animations (.bvh files) on the actor
• Synchronization of the avatar with the actor by adjusting the the models so that the measurements match and the animations are correctly reproduced (retargeting).

4.3.3. Tool Usage

Tool Handling during animations is implemented using animation rigging, which enables users to animate a mesh with the use of a skeleton, as well as other use cases like runtime rigging. Runtime rigging is when a skeletal animation is modified during gameplay using constraints as a post-process. Useful scenarios include attaching hands to props or aiming the head in reaction to a gameplay event like a character passing by. Rig Constraints are also used to affect objects in reaction to the skeleton’s motion.

4.4. Workshop Demonstration

For craft presentation and to facilitate the installation presented in the next section three applications were implemented. These applications present synchronized content, and thus they integrate a software mechanism for their interconnection. The main application (see Figure 13) is a simulation of the crafting process for creating a glass carafe as modeled in the previous sections, and it combines the knowledge on the crafting process with the modeling of the workshop, the tools, and the machines. Furthermore, it integrates the implemented VHs and their animations as recorded during the ethnographic fieldwork at CERFAV.

The second application (see Figure 14) presents close-up views of the gestures of the glassblowing VH to enhance the understanding of the audience regarding specific crafting gestures used in the making process.

The third application presents the tools used in each step of the process to enhance the understanding between craftsmanship, gestures, and tools. For example, as shown in Figure 15 for the shaping and glassblowing process steps, a blowpipe, newspaper, and a block are used. The tools are presented on the top of the application’s screen while the abstraction of the process is presented in Section 4.3.3, presented on the bottom side of the screen.

4.5. Craft Training

To experience glassblowing gestures, an interactive installation was created, with the use of MAX/MSP 8, a visual programming framework, in combination with python programming, as well as an RGB-D Intel Realsense camera for the real-time tracking and recognition of the gestures of the user. The information that was used from both videos from the gesture recognition engine was the upper body joints (shoulders, elbows, wrists, neck), as extracted from the OpenPose [62] framework. OpenPose is a system for real-time human joints detection, with the use of Deep Learning architectures.

In the screen (see Figure 16), a big frame with the video of the expert glassblower appears, while a smaller one with the video of the visitor places on the top-left of the screen. The user can either start experimenting directly or choose the question mark sign on the top of the installation screen, to see the instructions before starting imitating. An instance of this instruction screen is shown in Figure 4. On that screen, the user can see a video of each one of the gestures before starting experimenting. When the imitation starts, the grey bar right up the expert’s video becomes red. The speed of this bar becoming red concerns how good the gesture recognition results are.

4.6. Exhibition

The results of this research work were deployed in the context of a periodic exhibition at the premises of the museum of CNAM, which is where the worlds of academics and professional activity come together. It is the only higher education establishment dedicated to life-long professional training.

In this context, a dedicated space at the cathedral which is part of the museum was dedicated for the exhibition. The space is a chapel of the cathedral located within the main exhibition of the museum. The installation is comprised of a special construction capable of hosting three large displays created through back projection in thin synthetic fabric. Furthermore, for craft training, a bench has been installed in front of the main display together with a glassblowing pipe to be used by visitors. The installation has integrated hosting spots for the glass carafes and pieces of the carafe created during the ethnographic fieldwork. Special lighting was integrated within to present the interaction of glass with light and to create atmosphere. Figure 17 presents the building of the installation structure.

After the completion of the physical part of the installation, the software was installed in the computers hosting each projection, and their intercommunication was set up. Furthermore, the computers were programmed to communicate with the projectors to open and close automatically thus saving resources, and protecting the projection equipment. Figure 18 present the craft presentation applications installed and running within the installation space.

In the computer hosting the glassblowing presentation, a second application was installed which is dedicated to crafting training. This application allows users to mimic craft gestures using the tools and the bench. The scenario that was set up was that the presentation application sequence is executed first and then the main projection switches to the craft training applications so that users test what they learned from the craft presentations. To do so, an application was used to control the order of execution of the craft presentation and craft training applications. Examples of users interacting with the craft training application are presented in Figure 19. On the left side, a back view of the application and user is presented, while on the right side, we present how users are experiencing the interaction.

4.7. Preliminary Evaluation

After the technical validation of the installation, a short preliminary evaluation with end-users provided by the museum was conducted. The evaluation was focused on museum personnel while a wider evaluation with end-users is planned to happen later on during the lifetime of the installation. The objective of the preliminary evaluation was to identify whether the educational goals set up by the museum are facilitated through the installation and identify possible usability issues.

The first part of the preliminary evaluation was conducted with users from the education department of the museum invited to experience the installation and mimic the craftsperson actions using the bench and tools provided. The results of this evaluation resulted in changes to the UI of the training application in order (a) to provide real-time help to users to guide them through the training process and (b) to enhance the feedback of the system during user operation to enhance the understanding of both successful and nonsuccessful interaction. More specifically, an identified issue regarded the feedback that should be provided when the user is starting to perform a gesture where there was a time frame with no additional information from the system, resulting in users quitting an operation with the feeling that they were not doing the correct gestures.

The second part of the evaluation was done through inspection methods on all users. A user experience evaluator was monitoring the interaction of the users with the system and recorded identified usability issues. More of them were minor details of the UI that were improved. The most severe issue identified was a cold start effect resulting from the lack of sufficient information from the system when moving from the craft demonstration part to the craft training part. For this reason, introductory screens were designed and installed to assist users when the training session started. Furthermore, another issue that regarded the time scheduling of the installation regarded the occurrence of switches between apps while the users were interacting with the training. This was considered a major usability issue and was resolved on site.

After the conclusion of this preliminary evaluation and the resolution of the identified issues, the installation was inspected by a user-experience expert. The formal user-based evaluation is planned for January 2022, where further feedback will be collected by actual users of various age groups.

5. Conclusions

This paper presents the process for the representation and presentation of the craft of glassblowing. The result is the documentation of the process and the creation of a Mixed Reality exhibition at the premises of the museum of CNAM. This research work can be considered not just a case study for the specific craft instance but also as a guide regarding how similar scientific goals can be pursued in the future.

Regarding future improvements of the presented methodology, several directions can be followed. Initially, we acknowledge that the process presented is time and resource-demanding considering that the objective is a valid representation that could lead to multiple presentations of a craft. As such, improvements in the process can significantly improve the uptake of the proposed methodology. To that respect, future improvements could regard the data curatorial platform by simplifying and automating parts of the process to minimize the time spent in data curation in conjunction with the time spent for the scientific exploitation of data. Furthermore, simplification could also regard additional actions required for the post-processing of input data by integrating more automated tools and data processing wizards. Furthermore, another direction regards the further validation of the methodology through new craft instances. Finally, further presentations and visualizations of the represented crafts will enhance the value of representation, and in this field, further research could concert computer-aided facilities for semi-automated creation of craft representations.