Multimodal Narratives for the Presentation of Silk Heritage in the Museum

Abstract

In this paper, a representation based on digital assets and semantic annotations is established for Traditional Craft instances, in a way that captures their socio-historic context and preserves both their tangible and intangible Cultural Heritage dimensions. These meaningful and documented experiential presentations are delivered to the target audience through narratives that address a range of uses, including personalized storytelling, interactive Augmented Reality (AR), augmented physical artifacts, Mixed Reality (MR) exhibitions, and the Web. The provided engaging cultural experiences have the potential to have an impact on interest growth and tourism, which can support Traditional Craft communities and institutions. A secondary impact is the attraction of new apprentices through training and demonstrations that guarantee long-term preservation. The proposed approach is demonstrated in the context of textile manufacturing as practiced by the community of the Haus der Seidenkultur, a former silk factory that was turned into a museum where the traditional craft of Jacquard weaving is still practiced.

1. Introduction

Τhe success of assisting Cultural Heritage (CH) valorization through digital technologies (availability, accessibility, and engaging and interactive presentations) is due to the captivating content of stories, memories, and meaning that CH offers. Traditional Crafts (TCs) have a paramount cultural value and offer a thematic thesaurus of stories and narratives related to the origins of modern societies, history, and culture, the origins of industrial development and design, economic factors and societal outcomes, as well as, personal and family memories, all of which general audiences can relate to.

TC representation and preservation comprise a very demanding topic, as they cover a broad spectrum of tangible and intangible dimensions. In this work, we aim to capitalize on the existing literature and repositories and integrate mature approaches and technologies for the modeling, representation, and digital preservation of CH in a way that is meaningful to the contemporary needs of this domain. Furtherermore, the ability to teach a TC is central to its preservation. Perhaps, the biggest threat to TC viability is due to the declining numbers of practitioners and apprentices [1].

In this context, this paper provides the methodology and tools for the representation and preservation of TCs. This representation captures and represents the knowledge related to TCs, from their objects and the ways of making them, their required materials and skills, and their societal culture, economic impact, and historical significance. Through the representation, we aim to provide learning tools to support TC education, raise interest, and provide business motivation. Motion-driven representations and the semantic organization of content will provide the basis for introductory educational applications for tourism, as well as community and museum education.

The provided methodology is demonstrated in the case of textile manufacturing. For centuries, the German city of Krefeld was known as the “silk city”, and is the location where the Haus der Seidenkultur (HdS), a former silk factory that was turned into a museum, still practices the traditional craft of Jacquard weaving. While one might associate TCs with dexterity, hand tools, and cottage industries, the case of textile manufacturing takes place in a more industrialized environment, making use of large, noisy machinery, as well as a unique form of technology-enabled craftsmanship.

2. Related Work

2.1. Digital Representation of Traditional Crafts

A staggering amount of research on the digitization of artifacts and archives has taken place in the last two decades [2,3,4,5]. This enormous effort has led to sophisticated digitization techniques for artifacts and documents, as well as, guides of good practice for digitization streamlining, i.e., [6]. Moreover, efforts in the preservation of digital assets have produced standards in digitization formats, and enabled the production of knowledge bases open to the public for general knowledge or application development, i.e., Europeana. More recently, the CH community has had an advanced interest in capturing, modeling, and digitally preserving the intangible aspects of CH.

TCs are recognized as a form of CH by prominent organizations. In 2003, UNESCO adopted the Convention for the Safeguarding of Intangible Cultural Heritage (ICH), and in its authoritative text [1], it enumerates traditional craftsmanship as an independent category of ICH. TCs comprise a scientifically challenging domain of CH because they have both tangible and intangible faces, qualifying TCs as “the most tangible manifestation of intangible cultural heritage” [1]. TCs exhibit a wide thematic range of CH topics of historical, societal, anthropological, and ethnographic interest.

To date, research efforts on TCs have been scattered through materials and places of production, and very few have treated the topic of TCs as a source for the renovation and innovation of knowledge. Despite its importance in the mid-19th century, the scientific literature on the preservation and curation of TCs started to emerge recently; only a few studies are treating the topic in an integrated manner, given its multifaceted nature. Efforts towards appropriate treatment have emerged, through the collaboration of a wide range of experts by UNESCO [1], providing a theoretic basis towards this effort. Besides case studies, there has been no effort devoted to the representation of all TC dimensions and the curation of the corresponding digital assets.

The multifaceted nature of TCs has provided a range of definitions that are explored in seminal attempts to thereotically define the notions and contexts of TCs [7,8]. Though a crisp definition is elusive, it is conceded that TCs are characterized, as a minimum, by (a) traditional materials and technologies of their manipulation, (b) a certain type of product, (c) the dextrous use of tools and/or hand-operated machinery to make or repair useful things, and (d) a type of making that involves the knowledge and application of traditional designs.

TCs have both tangible and intangible dimensions. The tangible dimensions regard artifacts and buildings, clothing, tools and machinery, documents and archives, as well as the materials and the physical environment of practice. Their digitization has been addressed by past digitization projects, and a breadth of methods and best practice guides are now available. The intangible dimensions of TCs also have multiple faces that include the required skills, the learning process, as well as the cultural, religious, social, economic, and creative faces [8].

There exists a theoretical and technological gap that calls for: (a) a formalized process for the representation of TCs that copes with the challenges that stem from the representation of the multidimensional nature of TCs and the diverse range of experts involved, and (b) a technological infrastructure that supports these goals, copes with the large data volume, and avails collaboration between these experts.

2.2. Digital Storytelling Technologies

Our approach is rooted in the idea that humans exhibit a limited capacity for memorizing inventories, such as a list of events, as opposed to stories imbued with meaning or causal dependencies [9]. In this work, an approach is proposed that binds the cultural and socio-historic context of TCs with engaging stories that enhance the museum-visiting experience. As such, among the available technologies, Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) exhibit a high capacity to empower experiences in multiple contexts of use, and can be gracefully used to augment physical items and physical spaces, or their digital representation [10], with information.

2.2.1. Augmented Reality for Overlaying Information on the Real World

AR refers to an interactive experience in a real-world environment where the objects that reside in the real world are enhanced by computer-generated information [11,12] with or without the help of markers. According to [13], the evolution of the capabilities of mobile devices, combined with affordable Internet access and advances in networking, computer vision, and cloud computing, transformed AR from science to reality. In 2016, the mobile game “Pokémon Go," which essentially made AR technology popular throughout the world, became a great example of how mobile gaming can take place in an augmented world [14].

One of the manifestations of AR regards the augmentation of Cultural Heritage (CH) objects and sites, aimed at improving the visitors’ experience in these spaces. Mobile museum guides use AR technology to enrich exhibits with information [15,16,17,18], giving rise to new forms of digitally augmented tours [19,20,21]. In the same vein, mobile AR has been used to enhance archaeological sites with virtual and real scenes [22,23,24] where virtual avatars are employed to present contextual information. Today, museums are starting to realize that AR can be an effective way to build user interest in museum collections and exhibits.

VR is “a very powerful and compelling computer application by which humans can interface and interact with computer-generated environments in a way that mimics real-life and engages all the senses” [25]. As such, VR has gained the attention of the CH sector for reviving the past [25], and thus opening a window in time and space where users can travel and experience the past with all of their senses. VR achieved this by combining computer graphics, interaction, and novel approaches towards the digitization of CH components such as artifacts and monuments of material heritage. In this context, VR can be exploited for bridging the gap between the virtual representation of an artifact and the visitor, allowing multiple forms of interaction and storytelling with CH in the virtual or physical museum setting [26,27].

2.2.2. Mixed Reality

MR refers to the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real-time. According to [28], several studies demonstrate that the use of new and combined media enhances how culture is experienced. In this regard, CH uses such technologies for different purposes, including education, exhibition enhancement, exploration, reconstruction, and virtual museums. Among the multiple manifestations of MR for CH most relevant to this work is the usage of augmented artifacts to access and interact with information and artifacts (e.g., [29]). Previous approaches include multimodal interfaces to augment physical CH artifacts with information [30,31], CH-inspired games that employ physical items and digitizations of physical items and sites [32,33,34], informative art displays [35,36,37,38], and immersive mixed reality environments for CH [39].

2.3. Virtual Humans

Virtual Humans (VHs) have been recently employed in CH applications [40,41]. For example, in [42] the effective potential, persuasiveness, and overall emotional impact of VHs with different professional and social characteristics was studied. The authors underline the importance of aligning and fine-tuning narrative styles and content to VHs, which should correspond to their roles in terms of appearance, and highlight the importance of effective components in their storytelling approach. In [43,44], the authoring of multimodal narratives through an online platform is presented. Furthermore, a study of alternative storytelling solutions [45] concludes that VHs presents an empathic medium capable of engaging with new audiences. In the same work, narratives are highlighted as a “new way to discover hidden treasures from the past.” In other works [46], VHs are used for guiding users in CH environments. VHs have also been used for preserving and simulating cultures [47] and teaching crafts. For example, [48] utilizes VHs in a virtual environment for teaching the craft of printmaking, [49] utilizes VR as a tool for communicating the craftsmanship of engraving, and [50,51,52,53] present solutions developed for craft training using tools and machines manipulated by VHs. In [54], Danks et. al. combine interactive television storytelling and gaming technologies to immerse museum visitors into the artifacts on exhibition, engaging the user into the physical space using virtual stories, while [55] describes the use of a VH as a means for providing interactive storytelling experiences at a living history museum.

Research has shown that VHs can affect the virtual experience and stimulate attention and involvement [56,57,58]. Furthermore, their contribution to achieving the suspension of disbelief has been recognized [59]. Narratives, when successfully used for guiding the user through a Virtual Museum, motivate visitors to stay longer and see more [60]. VHs can bring these experiences to a wider audience, including people with disabilities [61], as well as provide a welcome invitation for discovery [62,63]. As such, several approaches have been presented for the implementation of realistic VHs, including segmentation of the motion files acquired through the recording of craft practitioners and narrators (e.g., [59,61,62]).

2.4. Socio-Historic Context

Pure silk, one of the oldest known natural fibers, is still highly fashionable even after thousands of years. The history of Krefeld is closely linked to this magical material. Today, there is a small museum called Haus der Seidenkultur, which shows how the history of silk has shaped the development of the town over the past three centuries. At the beginning of the 17th century, Krefeld came under the rule of the Netherlands, and the town became an island of religious tolerance. Consequently, in a period in which the denomination of the population was determined by the denomination of the ruler, Mennonites from near and far came to Krefeld and settled there. This immigration had far-reaching consequences that have shaped the profile of the town right up to the present day.

The religious refugees brought with them linen-processing skills, and as they were also mostly successful businessmen, they laid the foundation for economic growth and prosperity. The von der Leyen family, immigrants from Radevormwald, also contributed significantly to the development of the “Town like Silk and Velvet.” Originally linen weavers, they increasingly changed the emphasis of their business to silk weaving.

In 1702, Krefeld became Prussian and silk weaving became the most important economic factor, with sales to the Prussian court in Berlin flourishing. In this period the silk weavers were out-workers who received orders to weave fabrics from merchants and traders. The looms were set up in front of the light window in the typical small cottages, some of which still exist today. The head of the household was normally the weaver, and other family members helped with tasks such as reeling the thread onto the bobbins for the shuttle. On one of the main avenues of the town, there is a monument to the weavers, “Meister Ponzelaar.” He wears a frock coat (his Sunday best)—called a “Laakesserock” in the local dialect—and a high-necked waistcoat, a small collar with a silk scarf, and a “Jraduutkapp” (a black cap). At the end of the week, he takes the finished fabric on the beam to the merchant’s office, together with a bag containing any leftover thread. There, he was paid and received a new prepared warp beam and thread for the week ahead. Such weavers were a typical sight in the town until the beginning of the 19th century. Their craft required a rapid comprehension and rhythmic movement of hand and foot.

In 1785, Edward Cartwright invented his first mechanical loom, and he continued to make improvements to it. The enhanced looms then went on sale in 1820. With the advent of mechanization, the silk entrepreneurs started to build factories where all the machines were powered by one source of energy and the workers were responsible for more than one loom.

3. Methodology

An articulated approach to the documentation and representation of crafts is proposed in [64,65,66]. In this subsection, we report on following the steps of this approach for textile manufacturing. The technical implementation of the approach for the formal and digital representation of the crafting process is supported by the Mingei Online Platform (MOP) [43], where digital assets and semantic metadata are organized in a formal representation compatible with contemporary digital preservation standards. The proposed approach is articulated in a sequence of phases, as illustrated in Figure 1. Following this methodology, the representation maintained by the MOP includes knowledge that is represented in (conceptual) layers that correspond to representation tasks, as outlined below.

• Craft understanding identifies the workflow of the crafting processes, the location of practice, the involved objects, and its actors.
• Data collection is the recording of physical objects and the human actions leading to the transformation of materials into articles of craft. The collected data are acquired from media objects, which are an abstraction for digital recordings such as images, video, 3D scanners, and Motion Capture (MoCap), and their technical metadata.
• Craft representation refers to the definition of semantic metadata, or basic knowledge elements. These are the tools and affordances used in the crafting process, as well as the related objects, places, events, and materials.
• Process representation refers to the representation of the crafting process in terms of the steps, and the interrelating conditions between them, that define the crafting process. Process execution representation refers to the semantic representation of an executed crafting process. The executed process steps are linked with recordings of the activity.
• Craft presentation and preservation refers to the experience built on top of this rich semantic representation.
• The exhibition refers to the integration of experiences in a physical space from where visitors can experience the represented craft instance.

3.1. Craft Understanding

3.1.1. Knowledge Collection and Literature Review

Craft understanding is initiated with the collection of existing knowledge in the following categories:

• Basic knowledge: (a) curated text, (b) curated material, and (c) formal representation.
• Existing content: (a) archives, (b) bibliographic research outcomes, (c) audiovisual material, and (d) digital representations.
• Contextual information: (1) brief description of the craft, (2) geographical location and range of the craft, (3) selected and representative communities concerning the craft, (4) craft workers/skilled workers/handicrafts education, (5) gender roles, (6) equipment, (7) craft products, (8) craft traditions, incuding oral traditions/story telling/work songs/myths and legends, (9) social practices/social dimensions, (10) economic dimensions, (11) design dimensions, (l2) artistic dimensions, (13) rituals, (14) festive events, (15) religious dimensions, (16) learning/education/transmission of knowledge, (17) geography of workshops, (18) emic (inside) presentation narrative interviews with craft practitioners.

In the case of textile manufacturing, the literature was studied, including the curated text of the museum, and pertinent essays from museum collaborators and personnel were provided by HdS. The original texts were in German, some published and some from the museum’s records, and were translated by HdS in the context of the project. In almost all cases, both the English and German texts were provided. Furthermore, three documentaries were studied that are productions of HdS. The two documentaries contain demonstrations of the weaving process, performed by the community of volunteers of HdS. The third is a guided walkthrough in the city of Krefeld at places of relevance to the silk workshops of Krefeld.

3.1.2. Understanding Silk Weaving

Weaving is a way to fasten multiple parallel threads that are extended by tension to a perpendicular, interweaved, and much longer thread. Most types of weaving require a minimum of equipment. Machinery introduced over time aims to ease and accelerate the process of weaving.

The yarn that comes from the spinning mill is rewound onto the warp beam of the loom before being used for weaving. In warping, the warp threads from the warping creel that have been sorted by the gathering reed are wound onto the warping drum. After warping, the spooled warp threads are wound onto a large metal roll, the warp beam of the loom.

A loom is any device that holds the warp threads at a reasonable tension and facilitates the interlacement of yarn. A shed is the area formed when some, but not all, of the warp threads are lifted. When the weft thread is passed through the shed, it is over some threads and under the rest. The basic mechanism that forms this shed is the heddle. A heddle is a hole that the warp threads pass through. When the heddle goes up, the warp thread associated with that heddle also does.

Passing a string through a heddle is a task similar to passing a string through a needle. Thus, setting up the threads to pass through the heddles is a tedious and time-consuming task. It is thus important to perform it as scarcely as possible. Estimation of thread quantities is an important part of loom preparation. The weaver must be able to repair a broken thread so that passing the string through the heddle is avoided.

Depending on the way of creating tension for the warp threads, different kinds and types of looms have been invented, i.e., the blackstrap loom, the drawloom, or the conventional handloom, where bars are attached to each thread to create tension.

Up to the invention of the heald, individual warps were lifted by fingers, to insert the weft perpendicularly. The heddle mechanized the lifting of warps. The upper heald frames are connected by a cord passing over a roller, and the lower heald frames are connected to treadles. By depressing one treadle, one heald frame is raised and the other lowered, separating the warp threads. This separation creates an opening, or the shed, that facilitates the insertion of the weft. Depending on which heddles are lifted in each warp, different structures, or weaves, can be woven.

One way to introduce the weft thread was by hand. The weft is wound around a rod that is called a pirn and, while it is interwoven through the warp, it is unraveled from the pirn. The idea of winding the weft yarn onto a stick that could carry it faster through the shed from side to side led to the development of shuttles.

Pressing the weft into place, firmly and evenly, across the width of the fabric, is not easily performed by hand. This tool is required to be: (a) flat, to be entered into an open shed, (b) smooth, to glide easily along the warp threads, (c) firm, so as not to bend under pressure, (d) long, to reach across the warp and beyond and be held, and (e) blade-like and blunt on one side to reach deeply into the angle of the open shed.

These requirements led to the beater-in or batten, which, for its sword-like appearance, is called the “weavers sword” [67]. Weaving is summarized as a repetition of three actions:

• Shedding: warp threads are separated by raising or lowering healds to form a clear space where the weft should pass.
• Picking: the weft is passed across the loom. This is implemented by hand, shuttle, air-jet, or rapier.
• Beating-up or battening: the weft is pushed up against the fell of the cloth by the beater.

3.1.3. Weaving Jacquard-Patterned Fabrics

The Jacquard attachment is a device fitted to a loom that simplifies the process of manufacturing textiles with complex patterns. The machine was controlled by a “chain of cards, that is, some punched cards laced together into a continuous sequence. Multiple rows of holes were punched on each card, with one complete card corresponding to one row of the design.

To create patterned fabrics using a Jacquard loom, the fabric design is first sketched and transferred to squared paper (see Figure 2a). A skilled worker translates the design into punched cards, using a machine for punching cards (Figure 2b). Each complete card represents one row of the weft pattern. The holes created in the cards encode the selection of threads to be lifted when the shuttle passes for that particular row, for the implementation of the pattern. Technically, the holes allow the hooks to pass through, which, in turn, triggers the heddles to be lifted. Cards are laced together in a chain and introduced to the loom. In Figure 3, punched designs for the Jacquard attachment are shown. The act of lifting some of the warp threads, so that the weft passes only below those, can be better understood when looking at a woven fabric, from both sides.

3.1.4. Silk Loom Preparation

Extensive instructions are provided in the HdS film project “Einrichten des Golwebstuhles”(“Setting up the Gold Loom”), where the preparation of the silk or gold loom is presented. In this documentary, visual documentation of the required tasks was studied. In the 2018 HdS film project ”Stadtspaziergang auf Seiden Pfaden” (“Hanging by a thread”), guidelines were studied relevant to the estimation of the amount of thread required depending on the size of the fabric to be manufactured (see Figure 4).

3.1.5. Formulation of Historic Narratives

The major outcome of the craft understanding activities is the creation of text-based narratives produced as the result of the rationalization of knowledge. Rationalization has been useful for identifying several topics of interest. For each of these topics, a text-based narrative has been created. The narratives include technical and contextual knowledge—historic, economic, societal, traditional, gender roles, teaching methods, etc.—relative to the textile manufacturing at Krefeld. Narratives transmit craft knowledge through stories that contain both the tangible and intangible dimensions of a craft, along with the information that highlights its historical and societal significance in conjunction with major historical events. Overall, eighteen narratives have been defined: (1) From Jacquard weaving to computer science, (2) History of haus der seidenkultur: construction of the building, (3) Krefeld from its origins to town like silk and velvet, (4) Hubert Gotzes weaving workshop, (5) The name of Krefeld, (6) History of the family Hubert Gotzes Jr., (7) The story of the Jacquard invention, (8) Silk pilot context, (9) Vocational training in Krefeld, (10) History of the company Hubert Gotzes Inc. in Chicago, (11) The story of the cloth in the shrine of Charlemagne and its motif, (12) Ecclesiastical fabric weaving in Krefeld, (13) Ecclesiastical textiles, (14) Chronicle of the Casaretto family, (15) Founding of the association of friends, (16) In the house with tricky nooks and crannies, (17) Origins of the “Crown prince district”, and (18) A foundation for 100 years.

The objective of the creation of these narratives regards the identification of the information (persons, events, locations) represented and linked to the acquired digital documentation. Ultimately this process leads to the formulation of a semantic representation of these narratives. Pertinent data collection activities relevant to the use case under consideration are presented in Section 3.2.

3.2. Data Collection

The collection of data was a timely process involving various scientific disciplines and technical tools, including photographic documentation, 3D digitization, video documentation of craft practitioners, MoCap of craft processes and narrators, etc. The following sections provide a summary of these activities.

3.2.1. Photographic Documentation and Ultra-High-Resolution Textile Scanning

Photographic documentation focused on museum exhibits that have unique historical value. More specifically, we took photos of patterns in books, paintings, manuscripts, garments, and fabrics, as well as close-up photos of the various parts of the looms. The photographs collected comprise about 20 textile manufacturing documents with instructions, 110 overview photographs of looms and HdS workshops, 80 loom detail photographs, 10 photographs of silk threads, 50 photographs of plants used in the manufacturing of color pigments, and 5 sets of photographs of detailed ecclesiastical garments (about 800 photographs). For the acquisition of ultra-high-resolution scans of textiles, a contactless scanner was used, [68], that is a cost-efficient alternative to off-the-shelf solutions and provides micrometer scans with a resolution of 19.8 Kppi. Results can be seen in Figure 5.

3.2.2. Digitization of Ecclesiastical Vestments

Reputed artifacts of HdS such as ecclesiastical garments were digitized. In Figure 6a, the acquired photographic documentation is presented, while Figure 6b presents the 3D digitization results of this dataset.

3.2.3. Video Documentation of Craft Practitioners

The recording of a human expert was performed during silk weaving. More specifically, the expert was recorded as he went through the different stages of the loom and silk preparation, as well as the weaving process itself. For acquiring video documentation, in terms of equipment, we used three cameras for data acquisition: one digital camera (Nikon J1) with a 10–30 mm lens that captured single images of resolution 3872 × 2592, as well as full HD (1920 × 1080) video at 60 fps, and two GoPro cameras with an ultra-wide lens capable of 2.7 K (2704 × 1520) video footage at 70 fps. To ensure that the footage is free of motion blur, we captured the scene at a high frame rate, and additionally used tripods and a gimbal. Videos in high resolution at a high frame rate were captured from various viewpoints to ensure an unobstructed view of the expert that can be used to perform pose estimation of the human hand and body and extract the motion patterns of the expert. The latter type of data acquisition posed several challenges. Due to the physically tiring process of operating a heavy loom by a single expert, only a few repetitions of each task in the weaving process could be performed and recorded. A second challenge was the size and positioning of the looms. State-of-the-art 3D human-pose-estimation algorithms for visual data are typically trained using frontal full views of human bodies, where major identifying features of the body (face, arms, fingers) are visible and self-occlusions are limited. For instance, having two different frontal views of the body during the weaving process would have been ideal. In the case of the weaving room at Krefeld, all of the machines have several moving parts, are positioned and mounted on walls, and occlude the front side of the expert, making it impossible to capture frontal views of the body. We ended up recording the expert using back views of his body, from specific angles so as to be able to simultaneously capture his body and his feet, which play an active part in the operation of the looms pushing against pedals and levers. The capturing procedure resulted in a dataset of approximately 100 GB of pictures and videos where we recorded the artistic procedures of painting and selecting the tiling patterns, punching cards compatible with the Jacquard looms, the preparation (winding and unwinding) of silk, and the final weaving procedure.

3.2.4. MoCap of Craft Practitioners

MoCap of craft practitioners took place between 2–4 April 2019 at the HdS Museum, Krefeld, Germany. During that time, one expert was recorded performing the following tasks related to silk weaving: the creation of the punch cards, wrapping of the beam, preparation of the beam, and Jacquard weaving with looms of different sizes (small, medium, and large). These are illustrated in Figure 7.

3.2.5. MoCap of Narrators

MoCap of narrators is used to provide realistic animations for VHs acting as museum storytellers. To this end, the narration is captured as performed by a human, and the acquired motion is retargeted for the VH, as presented later on in this paper. To do so, the Rokoko equipment and software were used. For the narration moves to be more realistic, the stories during the recordings were also narrated, and the voice used was synchronously recorded. In this way, the synchronization of voice and movement in the narrations was a lot easier, and it also guaranteed a more natural narration.

3.3. Craft Representation

Data recorded using the aforementioned technical equipment and methodologies generated a large dataset to be post-processed. Post-processing in this context is considered the process of selecting the appropriate fragments from the raw data to be processed, and then feeding these fragments to specialized computer algorithms and software tools to generate higher-level results (e.g., motion files from video sources). The outcomes are then represented in the MOP.

3.3.1. Visual Tracking from Audiovisual Craft Documentation

For visual tracking, the OpenPose [69] library and the MocapNET [70] body-tracking software were used. In the Krefeld datasets, we observe that, when all limbs are visible, motions tend to translate well into the armature.

Failed frames mostly consist of several limbs being occluded and not visible in the camera observations. Invisible joints lead to missing 2D joint estimations, and missing 2D estimations, in turn, mean empty lines and columns in the NSDM matrices of MocapNET and not enough structural information for a clearly defined solution. This is particularly evident in datasets during the punching of cards, where despite the positioning of the cameras, feet often are occluded, going out of the field of view. This particular case could be remedied with specialized classes that handle joint estimation without feet. It should also be noted that both OpenPose and MocapNET, handle input as standalone frames, and that means that no temporal information is leveraged to mask possibly noisy input.

To further improve results despite the integration of newer, improved versions of the 2D and 3D pose-estimation software, motion interpolation could smooth 2D and 3D output. Finally, the curator could add missing points to reduce motion artifacts. Visual tracking examples are presented in Figure 8.

3.3.2. Visual Tracking from Audiovisual Archives

The audiovisual archives post-processed with visual tracking algorithms included the set of documentaries created by HdS showcasing the silk-weaving process to the public. The documentaries include interviews of members of HdS describing the whole process, as well as demonstrations of the preparation and operation of the looms. Given that the focus of these documentaries has been to provide broad knowledge regarding silk weaving and be an immersive experience to the viewer, recording experts’ weaving is only part of the storytelling. For instance, the recording of experts is interleaved with historical data about silk weaving, or only a small area of the expert’s body is visible during weaving to showcase the loom. To effectively handle these data, clips of the original videos that depict only the operation of the looms need to be selected and potentially enhanced with information from the accompanying audio regarding the semantics of the recorded motion.

To track the 3D motion of the expert in videos, we use a python/C++ implementation of tracking algorithms [71,72] that make use of a GPU for estimating the locations of the hand joints on the image, as well as the 3D skeletal pose of the hand. In our experiments, we have used an NVIDIA GeForce GTX 1080 GPU. We then import the estimated 3D skeletons into the open source modeling software Blender, and use the internal functionality to output the data in .bvh format. In Figure 9, we see representative clips of the weaving process as described in the documentaries.

3.3.3. MoCap Segmentation

After the correction and removal of noise from the MoCap data, the data was segmented by gestures. The recordings initially were performed as tasks, with one recording being a whole task, and afterward these recordings were segmented by gestures. One task could have only one gesture that is repeated several times, or might have more gestures that are repeated throughout the task. For example, Jacquard weaving consisted of the sequential repetition of three gestures. In general, the goal was to collect a minimum of three repetitions per gesture.

3.3.4. Motion Retargeting

The processing of MoCap input was initiated with the selection of the most accurate of the available sequences. To this end, each sequence was reviewed and a selection was made based on the accuracy and quality of the motion, avoiding, when possible, noises and sliding. In total, 16 files were selected. Figure 10 presents examples of the silk pilot animation files. The process is segmented into 6 sub-processes and, in total, 16 sequences were used.

The next step was completed on Autodesk MotionBuilder software [73], which is dedicated to animation and the direct integration of motion capture technologies. The process requires different steps: (1) creation of an “actor” in MotionBuilder with a skeleton definition corresponding to the BVH hierarchy, (2) transposition of the received animations (.bvh files) onto the actor, and (3) synchronization of the avatar with the actor by adjusting the two models so that the measurements match and the animations are correctly reproduced (retargeting).

3.4. Process Representation

Artifact creation is considered a process that is based on a schema. This process schema is conceptual and can be demonstrated or verbally described, i.e., as instructions. When a practitioner follows this archetype, we say that the executed process followed this schema. To refer to individual stages of creation, we say that process schemas and processes are comprised of step schemas and steps, respectively. Step schemas and steps are decomposed into smaller step schemas and steps, respectively. This decomposition leads to a hierarchy that starts from a coarse description and proceeds down to a finer analysis of its steps.

An activity diagram is a flow chart that models workflows and can depict sequential and concurrent activities. Activity diagrams and notations are borrowed from the UML specification [74]. In this work, they are used with a slightly different meaning: while in the UML, the basic actions of an activity diagram are computational actions performed by a machine, in the present context, the basic actions of an activity diagram are actions performed by humans or natural phenomena. Consequently, we use the following transition types, Transition, Fork, Merge, Join, Branch, and denote them as in the UML. The representation of the craft of silk weaving in the MOP is presented in Figure 11.

4. Craft Presentation and Preservation

4.1. Digital Preservation

Using the MOP, all knowledge elements are created through form-filling operations. Each type of element has a dedicated Webform, where metadata are edited. Furthermore, facilities to create links with other knowledge elements are provided. Links are provided in the form of URI for external resources, or in the form of semantic links for digital items curated in the MOP. Moreover, knowledge elements are linked to media objects of relevance. Examples of represented knowledge items are shown in Figure 12.

The digital assets hosted in the MOP repository are provided online in conventional and open formats. Each asset has a unique IRI to be directly integrated by third parties. Our knowledge is available on the Semantic Web via the MOP and with the SPARQL endpoint exposed. Furthermore, to ensure compatibility with online knowledge sources, definitions of terms are imported to the MOP by linking to terms from the Getty Arts and Architecture thesaurus [75] and the UNESCO thesaurus [76]. For further exploitation of the semantic knowledge encoded in the MOP, a Europeana Data Model (EDM) [77] export facility has been implemented allowing (a) the export of data in semantic compatible with the EDM format, and (b) the formulatation of SPARQL queries [78] on the MOP SPARQL endpoint to receive ED-formatted results.

4.2. Craft Documentation

The represented knowledge network is available through the WWW and the MOP [79] in hypertext format. Semantic links are implemented as hyperlinks that lead to the pages of cited entities. Contents are also organized and presented thematically, by class type. A keyword-based search is also provided. Documentation pages contain links to digital assets, textual presentations of metadata, and previews of the associated digital assets. For locations and events, specific UI modules are provided. For locations, embedded, dynamic maps are provided through OpenStreetMap [80]. Timeline and calendar views are available for events.

Process presentations are presented containing links to the recordings of the knowledge elements for the tools and materials, involving the participating practitioners, the date, the tools employed, and the location of the recording. If the process follows a process schema, a link to that schema and its preview are also provided. The hierarchy of process steps is presented using insets, each one presenting textual information and previews of the available digital assets. To present step organization, insets are dynamically unfolded at any depth in the process hierarchy, associated with image previews and embedded videos. Variations include images and textural descriptions.

4.3. Craft Demonstration

4.3.1. Presentation of Craft Actions

Figure 13 presents examples of the animated avatars created for silk weaving. Videos of the results can be also found online.

4.3.2. Implementation of Virtual Humans and Animation

The virtual human bodies and clothes are created to obtain one unified and optimized model, enhancing the visual impact of the characters with texture mapping and material editing. The 3D generation of the virtual bodies also has to take into consideration the total number of polygons used to create the meshes, to keep a balance between the 3D real-time simulation restrictions and the skin deformation accuracy of the models.

Avatars are created with a combination of different softwares: Adobe Fuse CC/Mixamo [81] is used for creating the body character, the clothes, the hair, and the rigging. The generated model is then imported into Autodesk 3DS max [82] for mesh geometry optimization. Manual methods using the editable poly tools are preferred since they allow for the keeping of the topology’s regularity, while the automatic methods generate a mess geometry which is not suitable for skin deformation nor regular UV texture map generation.

5. Exploiting the Representation

The multifaceted representation achieved from the proposed methodology is explored in a plethora of interactive presentations and demonstrations. These are categorized in this research work as follows: (a) applications that enhance the museum visiting experience, including the presentation of socio-historic context through narratives, (b) applications and concepts that are inspired by traditional crafts to create new products and services, and (c) innovative means for Web-based access to information and content for learning, education, and training.

5.1. Enhancing the Museum Visiting Experience

5.1.1. Museum Tour Guide—Narratives in the Museum

This is a dedicated museum tour guide application that presents narrations regarding the socio-historic context of the museum exhibitions while guiding visitors in the museum. The application is built on top of ten hot spots in the museum, each one connected to relevant narrations and audiovisual presentations that unveil the hidden treasures of the craft of textile manufacturing. From a technical perspective, the hot spots in the inside are linked to silk narratives authored in the MOP and other digitized museum assets. The application guides visitors through the ten hot spots in the museum and presents audio-visual content, based on the formulatedsilk narratives (see Figure 14). Narrations by VHs enhance the presentation of the narratives and contribute to the suspension of disbelief. In addition, scannable items and artifacts in the museum allow the user to access more information that is not included in the museum application per se. The museum tour guide is available in English and German.

5.1.2. Digital Information Visualization in the Museum

Two visualizations were considered for presenting digitization outcomes and visualizing the socio-historic context. The first visualization (Figure 15) regards the projection of digitization outcomes in an interactive form, while the second regards the provision of information on the socio-historic context of the museum through interactive timelines (see Figure 16).

5.2. A Handbag Inspired by Traditional Crafts

The idea of creating a contemporary accessory, and an AR app to augment it, was born by studying the historical patterns and transforming them into text-based narratives. This was in line with one of the most important goals of HdS, which is to raise awareness regarding the unique CH possessed by the museum and, at the same time, to connect its legacy with European history and tradition. The design of the handbag was a challenging task. By closely studying the HdS patterns, the following issues were encountered that needed a careful design approach: (a) the patterns are almost monochromatic, (b) the patterns are woven in bright yet intense color variations that require careful combinations, and (c) due to the intense iconographic elements of the patterns, the risk of creating a cluttered visual result is high.

Each motif has its history—a series of symbols that evolved in time and were artistically woven together to form traditional ecclesiastical fabrics. Our AR application provides three layers of pattern recognition. The first layer regards the bag, which has its own story, that being its creation. The second layer regards the multiple patterns which reveal the story of textile weaving in Krefeld. The third layer is the stories of the patterns, which have a story of their own (see Figure 17). The AR app augments the virtual space with a canvas to present those stories. Furthermore, as these stories relate to the history of Europe and the social and historic dimension of textile weaving, the bag is also a portal to a web of information provided through the MOP. To reach this information, a button entitled “view more info” navigates the user to the respective webpage of the MOP, from where the journey through European history may begin.

5.3. Web-Based Access to Immersive Visualizations

Web-based access is supported through a dedicated web page that summarizes all the key accomplishments of the proposed methodology for the use case of textile manufacturing. This page is structured using a modern long page approach initiated through a page index that guides visitors to the main results delivered online. These include rich audio-visual information supported by multimodal narratives available online. The web page can be accessed through the following link.

To address the needs of the widest possible user population, the web page has three variations: (a) for the general public, (b) for researchers and academics, and (c) for kids. In addition to the aforementioned web-based information, more details on the available web-based visualizations are presented in the following sections.

5.3.1. Historic Narratives

Access to historic narratives is provided through the MOP, where the text-based narratives formulated are semantically represented and connected to the represented socio-historic information. The narrative can be accessed by alternative means by studying (a) the narration text, (b) the key events presented in the narrative, (c) the main human participants, (d) the locations of events, and (e) the chronologic sequence of events.

5.3.2. Virtual Museum–Discover Ecclesiastical Garments and History

The virtual museum application exploits the digitization of historic silk ecclesiastical garments and the representation of silk fabulae in the MOP. Regarding the garments, the visualization displays the 3D reconstructions of these garments, and the user can move through the five garments and read the annotated information for each one of them. Furthermore, through interactive timelines, historical and social events related to the craft of silk in Krefeld are presented (see Figure 18). The presented content is exported from the MOP in JSON format and imported to the timeline. The JSON file contains all the necessary fabula and event metadata. This way, the content of the interactive timeline is automatically generated with minimal technical effort. Interaction with this visualization is initiated by selecting one of the available timelines. Upon selection of a timeline, the system displays various images that have been semantically linked to the events of the represented fabula. Each image links to information on the related event, while the order of appearance follows the chronological order of events in the fabula.

5.3.3. Craft Games

Two games specifically designed to explain both the design of a pattern for a Jacquard loom and how the punching card is created from the paper design were created previously (see Figure 19).

6. Evaluation of the Methodology—Lessons Learned

This paper presented the process for the representation and presentation of the TC of textile manufacturing as practiced by the community at the Haus der Seidenkultur (HdS), a former silk factory that was turned into a museum which still practices the traditional craft of Jacquard weaving. The proposed methodology has been applied in Mingei for 36 months, with the results presented as part of this paper. The multitude of dimensions explored while applying the methodology enhanced our understanding of working with TC communities, and provided insights about the applied methodology, as discussed in this section.

Craft understanding: During this phase of the methodology, we learned that TCs are deeply rooted in the social and historic context of the communities practicing them. Through history and archival research, many dimensions were discovered that, in this case, led to the formulation of rich social and historic narratives. Furthermore, the TC itself surprised us by revealing several hidden dimensions, spanning from the art elements of pattern creation and point paper design to the history of natural and synthetic fabric dyes. These largely affected both the work conducted in this phase and the consequent phases, multiplying the need for representation and data collection. Summarizing this step, we learned that TCs are usually “more than meets the eye”.

Data Collection: This aimed to support craft understanding by locating digitization subjects that would complement the social, historic, and craft knowledge acquired during craft understanding. At the same time, we learned that the discovery of a digitization subject may intrigue researchers to go back to craft understanding to acquire knowledge on the specific subject that was not covered during craft understanding. Thus, these two activities are strongly interconnected and should be executed with the close collaboration of the two teams. From a technical perspective, taking into account that several digitization technologies were employed, we learned that, in some cases, the specific requirements of a study could lead to the need for new digitization techniques, as in the case of acquiring ultra-high-definition textile scans of silk fabrics.

Craft representation and process representation: In these phases, it was important for the team to evaluate the capacity of the online platform to represent the vast amount of knowledge acquired. This posed two major challenges. The first challenge was the curation time needed to represent resources, and the second was the expressiveness of the representation needed to address complex knowledge elements such as processes. Regarding the first, we learned that a lot of effort is required from curators, and that training and help are needed to adapt to the new representation facilities. Regarding the second, the close collaboration of semantic experts, developers, and practitioners was required to conceptualize the representation, create the authoring facilities, and train on transforming ethnography to process representations. Overall, we learned through trial and error, which led to several redesign iterations of the online platform.

Craft presentation and preservation: There were two main challenges faced. The first was to create online presentations of the acquired representation so as to (a) allow researchers to build narrations for the acquired representation, (b) disseminate knowledge through the online platform, and (c) create online educational material and process representations. The second was to ensure the compatibility of the knowledge base with external sources, both in terms of linking to external sources and importing knowledge standards such as CIDOC and European EDM. The main lessons learned regarded the quality of our representation, which was capable of supporting several online presentations of the represented knowledge. Furthermore, taking into account that the system was based on existing knowledge standards, disseminating the represented knowledge was as simple as creating semantic associations of CIDOC-encoded metadata with other knowledge standards, and creating a SPARQL endpoint to deliver knowledge in various standardized formats.

Exploiting the representation: Moving to the more practical, for the end user’s exploitation of knowledge, the main lesson learned was that having a solid knowledge base and a rich representation that includes digital assets greatly enhanced the capacity to deliver results that could be experienced by end-users and visitors to the museum. In this, we learned that content makes a difference, as engaging narrations created through the web platform were disseminated in alternative means and modalities targeting a wide range of uses, including information, education, and entertainment. We learned that the separation of the representation from its usage allowed us to use a plethora of technical tools to create different forms of presentations, thus unleashing the creative powers of UX designers and developers.

Overall, in this work, we followed an exhaustive yet systematic method of representing the TC of textile manufacturing involving different scientific disciplines. The richness, depth, and genericity of the representation achieved through the proposed methodology are capable of supporting a multitude of presentations. This was achieved mainly by decoupling the implementation details of each presentation modality from the respective representation of information. Information represented in the form of narratives in the MOP can be used to author different presentations, each one targeting a different presentation modality. In this way, as demonstrated in this work. we were able to build a variety of visualizations targeting different presentation goals, including education, training, and information.

7. Future Work

Regarding future improvements of the presented methodology, several directions can be followed. Initially, we acknowledge that the process of representing the socio-historic context is time- and resource-demanding, as it involves the study of historic and archival resources, the selection of excerpts, and the identification of pieces of information that, when represented semantically, can produce a spatiotemporal representation. To this end, improvements of the methodology could include the integration of Artificial Intelligence for the semi-automatic detection of such instances for excerpts, to reduce the curatorial effort required for the formulation of the representation. This could be accompanied by facilities to support the transcription of such information in the MOP. On data collection, improvements could result in the need for less post-processing of the acquired documentation, which could be achieved through higher-end technologies such as a MoCap installation, instead of a MoCap suit. The selection of the equipment, in this case, was conscious, taking into consideration that the latter is an inexpensive solution that could allow for the replicability of our approach in more contexts, compared to the need to gain access to a specialized laboratory. In the same context, we expect that further improvements in the post-processing technologies in the future will further ease the burden implied by the proposed approach.