Next Article in Journal
Volcanic Stratigraphy, Petrology, Geochemistry and Precise U-Pb Zircon Geochronology of the Late Ediacaran Ouarzazate Group at the Oued Dar’a Caldera: Intracontinental Felsic Super-Eruptions in Association with Continental Flood Basalt Magmatism on the West African Craton (Saghro Massif, Anti-Atlas)
Previous Article in Journal
Mineral Characterization of Gold Ore Occurrences in the Khaptasynnakh Ore Zone, Anabar Shield, Far East Region, Russia
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Accessible Interface for Museum Geological Exhibitions: PETRA—A Gesture-Controlled Experience of Three-Dimensional Rocks and Minerals

by
Andrei Ionuţ Apopei
Department of Geology, Faculty of Geography and Geology, “Alexandru Ioan Cuza” University of Iaşi, 700505 Iaşi, Romania
Minerals 2025, 15(8), 775; https://doi.org/10.3390/min15080775
Submission received: 22 June 2025 / Revised: 20 July 2025 / Accepted: 23 July 2025 / Published: 24 July 2025
(This article belongs to the Special Issue 3D Technologies and Machine Learning in Mineral Sciences)

Abstract

The increasing integration of 3D technologies and machine learning is fundamentally reshaping mineral sciences and cultural heritage, establishing the foundation for an emerging “Mineralogy 4.0” framework. However, public engagement with digital 3D collections is often limited by complex or costly interfaces, such as VR/AR systems and traditional touchscreen kiosks, creating a clear need for more intuitive, accessible, and more engaging and inclusive solutions. This paper presents PETRA, an open-source, gesture-controlled system for exploring 3D rocks and minerals. Developed in the TouchDesigner environment, PETRA utilizes a standard webcam and the MediaPipe framework to translate natural hand movements into real-time manipulation of digital specimens, requiring no specialized hardware. The system provides a customizable, node-based framework for creating touchless, interactive exhibits. Successfully evaluated during a “Long Night of Museums” public event with 550 visitors, direct qualitative observations confirmed high user engagement, rapid instruction-free learnability across diverse age groups, and robust system stability in a continuous-use setting. As a practical case study, PETRA demonstrates that low-cost, webcam-based gesture control is a viable solution for creating accessible and immersive learning experiences. This work offers a significant contribution to the fields of digital mineralogy, human–machine interaction, and cultural heritage by providing a hygienic, scalable, and socially engaging method for interacting with geological collections. This research confirms that as digital archives grow, the development of human-centered interfaces is paramount in unlocking their full scientific and educational potential.

1. Introduction

The challenge of bringing vast geological collections to life in an era of digital interaction is reshaping the modern museum. For both cultural heritage and educational institutions, technology now drives a fundamental shift in how we preserve, communicate, and experience scientific materials [1,2,3,4,5,6]. This is especially true in the geosciences, where a clear trend towards more interactive and digitized practices in research and education is underway [7,8,9,10,11,12,13].
At the heart of this movement lies the three-dimensional (3D) digital model [2,14]. As noted in [11], these 3D data represent the fundamental core of any subsequent virtual reality (VR), augmented reality (AR), or other immersive project. Created by digitizing real-world specimens through methods like photogrammetry or micro-CT [11,15,16,17,18,19,20], these assets are invaluable. They offer richer, more interactive representations than 2D media and form the core data for virtual museum environments and engaging installations [1,5,21]. The creation of virtual rock collections for geology further underscores the importance of these data [8,12,22].
This digital shift aligns with the emerging concept of “Mineralogy 4.0”, a framework that sees digitalization and artificial intelligence (AI) as the drivers of the discipline’s next evolutionary stage [11]. This framework prioritizes the creation of immersive digital assets and the application of machine learning for advanced data analysis, all built upon open, FAIR-compliant (i.e., Findable, Accessible, Interoperable, and Reusable) platforms. This principle was recently demonstrated through the creation of the Atlas of 3D Rocks and Minerals, which is, to date, the largest curated open-access collection of its kind and includes a dedicated mobile application for broader accessibility [11]. The present study operates within this framework, investigating the practical implementation of an AI-driven interface to make such static digital collections dynamic and accessible. This work on user access and interaction complements other AI-driven advancements in the field, which are exemplified by the development of machine learning models for automated mineral identification [23] that rely on FAIR datasets like OpenMindat [9]. To support this vision, major initiatives like iSamples [24] are building the necessary infrastructure for natural science collections, focusing on robust metadata and permanent identifiers like DOIs or IGSNs [10,14,24,25,26].
Yet, bridging the gap between a digital object and a meaningful user experience presents significant hurdles. While immersive technologies like VR and AR are powerful, their deployment in public venues raises practical challenges related to high costs, user isolation, and hygiene [1,5,11,27]. Other popular installations, such as tangible augmented reality sandboxes [28,29], create highly attractive and haptic experiences but are designed for a different purpose, focusing on landscape topography rather than detailed object inspection. Alongside these, the interactive digital kiosk, though a fundamental tool for engagement [30], often relies on traditional touchscreens that can limit social interaction and lack the nuanced control needed for complex 3D models. A clear need has therefore emerged for an evolution of the museum kiosk toward more natural and compelling forms of touchless interaction.
A promising solution lies in Natural User Interfaces (NUIs), which reduce cognitive load by leveraging intuitive human behaviors, like gestures [31,32,33]. This approach is now highly feasible due to advances in Machine Learning (ML) that allow for the robust interpretation of visual data from a simple webcam [34]. Frameworks like MediaPipe are key enablers, providing complete pipelines for real-time hand skeleton detection that make the development of sophisticated yet accessible interactive systems possible [35].
The choice of MediaPipe for this project is contextualized by a dynamic landscape of computer vision tools. Foundational libraries, like OpenCV, provide extensive toolkits for core image processing [36], while frameworks like TensorFlow are crucial for training the underlying neural network models [37]. In parallel, dedicated hardware sensors have been pivotal in the development of gesture-based interaction. The Kinect sensor, for instance, has been employed for motion detection in museum learning [38], with its KinectFusion approach being influential for AR gestures [32,39]. Likewise, Leap Motion has served as another key sensor for high-fidelity hand tracking [40]. Furthermore, the final 3D environments that house these experiences are often built using powerful game engines like Unreal Engine [41] or web-based visualization libraries such as Three.js [42]. While all these tools are crucial contributors, MediaPipe was selected because it uniquely packages many of these concepts into a complete, pre-trained, and software-driven pipeline, making it an exceptionally accessible tool for building real-time interactive applications without reliance on specific hardware [35].
Addressing this need for accessible and engaging museum experiences, this paper introduces PETRA: an accessible interface for museum geological exhibitions. The name PETRA, meaning rock in both Greek and Latin, was chosen to reflect the project’s focus on geological heritage. The system utilizes 3D models of geological specimens as its core content and allows visitors to explore them through intuitive, webcam-based gesture control. To enhance the experience and reinforce learning, the system also incorporates an audio component, announcing the name of each specimen as it is selected. By adopting a natural interaction model, the system bypasses the need for costly or complex hardware like VR/AR headsets, thereby lowering the barrier to entry for institutions and enhancing visitor accessibility.
The PETRA system is developed within TouchDesigner by Derivative (https://derivative.ca/, accessed on 1 June 2025), a node-based visual programming environment designed for creating real-time, interactive multimedia systems [43,44]. The platform is recognized for its versatility in functioning as a connectivity hub, adept at processing diverse data streams from hardware like webcams and sensors to orchestrate complex user experiences [45]. Its application in academic and artistic projects, ranging from educational VJ (video jockeying) tools to AI-powered generative art installations, demonstrates its suitability for rapid prototyping and deployment of robust interactive systems [43,44]. The availability of a free non-commercial license further aligns with PETRA’s goal of accessibility.
This paper presents the design, implementation, and initial evaluation of the PETRA system. By leveraging AI-driven natural interaction through a widely available development platform, PETRA offers a novel contribution to the fields of digital mineralogy and cultural heritage. It provides a practical and highly engaging solution for enhancing visitor interaction with 3D geological collections in museums and educational settings. The following sections detail the system’s architecture, report on a case study of its use in a public exhibition, and discuss its implications for the future of immersive learning in the geosciences.

2. Materials and Methods

2.1. System Architecture and Hardware

The PETRA system consists of a data acquisition module (webcam), a central processing unit (a standard Windows or macOS computer), and a visual output module (a large display). A webcam captures the user’s movements (i.e., hand gestures), and this video stream is processed in real time by the PETRA application. The software then renders the manipulated 3D model on the display. The hardware is intentionally standard and widely available: any modern laptop or PC, a standard HD/FullHD webcam, and a flexible choice of display (e.g., a large TV or projector) are sufficient. For optimal performance, a system with a multi-core processor (e.g., Intel Core i5), 8 GB of RAM, and a dedicated graphics card (4 GB VRAM, OpenGL 4.3+) is recommended, along with a standard HD (720p) webcam for input. A conceptual diagram of this architecture is shown in Figure 1.

2.2. Digital Assets and Availability

The core content for the experience is a collection of ten 3D models of minerals and rocks, selected to showcase a diverse range of geological properties as detailed in [11]. The collection includes specimens such as pyrite and marcasite to demonstrate metallic luster, gypsum for its distinct crystal habit, and malachite for its vibrant color, among others, like obsidian, calcite, galena, scoria, granite, and sodalite.
All 3D assets are provided in the standard .obj file format. To ensure a realistic representation of each specimen’s surface properties, a Physically Based Rendering (PBR) material is used. This approach utilizes a set of high-resolution texture maps for each model, including base color, glossiness, metalness, ambient occlusion, and normal maps. The complete TouchDesigner project file and all 3D models are publicly available on GitHub (https://github.com/aapopei/PETRA, version 2025.6, accessed on 19 June 2025). The entire project is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0). This license requires users to give appropriate attribution (BY), to not use the work for commercial purposes (NC), and if they remix, transform, or build upon the material, they must distribute their contributions under the same license (SA). This “copyleft” approach was chosen to prohibit commercial exploitation, while simultaneously encouraging academic collaboration, experimentation, and the open sharing of any improvements made to the system, greatly benefiting educational institutions with limited resources.

2.3. Software Implementation

The PETRA application was developed entirely within TouchDesigner (non-commercial, build version 2023.12230), a node-based visual development environment. The entire application was built using standard, pre-built operators, reinforcing the project’s accessibility for non-programmers. The project workspace itself is organized into distinct, color-coded functional areas, including modules for camera input, 3D asset management, interaction logic, and final rendering.
The critical task of gesture recognition is handled by processing the webcam feed in real time. This is achieved using the MediaPipe TouchDesigner component (version 0.5.1), a third-party plugin that integrates Google’s MediaPipe framework directly into the node graph [46]. This component operates using a two-stage pipeline: a palm detector first localizes the hand, which is then fed to a hand landmark model that predicts the 21 keypoints of the skeleton [47]. This component provides robust, real-time skeletal tracking of the user’s hand, outputting positional data for the hand and its individual joints. These data are then used to drive the interaction logic.
This software-driven approach distinguishes PETRA from earlier gesture-based systems that rely on specialized hardware. While sensors like the Kinect use structured light or time-of-flight cameras to generate a depth map of the environment, and Leap Motion uses infrared stereo cameras to specifically model hands, MediaPipe achieves robust tracking through machine learning models applied to a standard 2D webcam feed. This reliance on software rather than specialized hardware is a core tenet of PETRA’s accessibility.

2.4. Interaction Design

The core of the PETRA experience is its intuitive control scheme, which relies on a small set of hand gestures mapped to corresponding actions. To facilitate immediate use, an on-screen graphic illustrates these primary interactions for the user (Figure 2).
Technically, the system defines these gestures by programmatically tracking the 3D coordinates of 21 specific hand landmarks, or keypoints, provided in real-time by the MediaPipe component (Figure 3). The two primary gestures are defined as follows: the rotational “hinge” gesture is calculated by tracking the angle of the vector formed by the index finger’s joints. The “pinch” gesture for 3D manipulation is defined by two simultaneous calculations: the manipulation mode is activated when the Euclidean distance between the thumb tip (keypoint 4) and index fingertip (keypoint 8) falls below a set threshold. This threshold was determined empirically through iterative testing to find a balance between responsive activation and preventing accidental triggers. Once active, this same distance is continuously mapped to the model’s scale (for zooming), while the 3D coordinates of the pinch’s midpoint are mapped directly to the model’s X, Y, and Z rotation.
A key feature of the PETRA system is that this gesture-to-action mapping is fully customizable. Due to the node-based nature of the TouchDesigner project, users can easily modify the interaction logic without traditional programming. For instance, the sensitivity thresholds for the gestures can be adjusted, actions can be remapped to different keypoints, or entirely new gestures can be implemented. This flexibility is a core design principle, intended to empower other institutions or researchers to adapt the system for their own specific exhibits or experimental questions.
The interaction is therefore governed by two primary user actions. For 3D specimen selection (Figure 2a), the user performs the rotational “hinge” gesture to advance to the next specimen in the collection. To enter the 3D manipulation mode (Figure 2b), the user makes and holds the “pinch” gesture. This single gesture enables simultaneous control over both rotation and zoom. Moving the pinched hand left, right, up, or down rotates the 3D model around its axes. At the same time, varying the distance between the pinched thumb and index finger smoothly zooms the model in or out.
Releasing the pinch gesture exits the manipulation mode. This simple, two-gesture system was designed to be quickly discoverable and easy to master for users of all ages and technical abilities.

3. Results

3.1. PETRA Implementation and Functionality

The PETRA system, as described in the Materials and Methods, was successfully implemented and deployed, with the complete project made available for public access (https://github.com/aapopei/PETRA, accessed on 19 June 2025). The application demonstrated high stability during extended use at a public exhibition, with no system crashes.
The system’s real-time performance varied based on the type of interaction. Manipulation of a loaded model, such as rotation and zoom, occurred instantaneously with no discernible latency. However, a one-time loading delay was observed during the initial model selection cycle. As each of the ten 3D models and its textures were loaded into the cache for the first time, a brief latency occurred upon switching. After this initial loading sequence was complete, all subsequent cycling between the models was fluid and immediate.
The integration of the MediaPipe component within the TouchDesigner environment provided robust and consistent hand-tracking from the webcam feed, allowing for effective mapping of user gestures. The final implemented system is presented in Figure 4, connecting the backend logic with the resulting interactive experience.

3.2. Case Study: “Long Night of Museums” Event

To evaluate the system’s real-world effectiveness and user reception, a case study was conducted during the annual “Long Night of Museums” event. The PETRA installation was set up at the “Grigore Cobălcescu” Museum of the Department of Geology, within the “Alexandru Ioan Cuza” University of Iaşi, Romania. PETRA was displayed on a 55-inch TV screen (TCL LED 55P655, TCL, Huizhou, China) with a vertical stand and was powered by a laptop (Lenovo Legion 5 Pro 16ACH6H, Lenovo Group Ltd., Quarry Bay, Hong Kong), using a FullHD webcam (Q95 3-in-1 webcam with 30 FPS, 95° FOV, PolaTab, Shenzhen, China) for input, which was connected via USB. The event was open to the public, and ticketing records confirm an attendance of 550 visitors. A significant portion of this diverse, all-ages audience engaged with the PETRA system (Figure 5). The findings reported here are based on direct, qualitative assessment of these interactions, as formal demographic data were not collected.

3.2.1. User Engagement and Reception

The installation generated a high level of user engagement, sparking curiosity among visitors and prompting discussions about the novel form of interaction. While some attendees initially expressed a brief moment of hesitation when faced with the unfamiliar interface, this “discomfort with the unknown” quickly transformed into an eagerness to discover, play, and embrace the experience. A common behavior observed was that users did not simply cycle through all ten minerals once; instead, they frequently looped through the collection multiple times, suggesting a deeper interest in playful exploration rather than simple task completion.

3.2.2. Discoverability and Ease of Use

The system proved to be highly intuitive and discoverable for a notably diverse audience that included a wide range of ages, from young children to seniors. The majority of visitors learned the controls through direct interaction (“learning by doing”), aided by the live visual feedback of the system. Without requiring verbal instruction, most users mastered the rotational and pinch-to-manipulate gestures within the first few seconds of use. In some instances where assistance was offered by student volunteers, it was often found to be unnecessary. The successful use of the system by a broad public audience confirms its accessibility and intuitive design.

3.2.3. Technical Performance and System Stability

Throughout the duration of the event, the PETRA system functioned without any glitches, errors, or system crashes, demonstrating high stability in a continuous-use public setting.

4. Discussion

The successful deployment of PETRA at a public event demonstrated the viability of webcam-based gesture control as an accessible and effective interface for virtual museum exhibits. Interpreting the case study results in the context of the project’s goals provides valuable insights into the design and impact of such systems.

4.1. Interpretation of Key Findings

The primary goal of PETRA was to create an experience that was both accessible and immersive. The “ease of use” observed during the “Long Night of Museums” event directly fulfilled the goal of accessibility. The rapid, instruction-free adoption of the controls by a diverse audience, from children to seniors, indicates that the system successfully lowered the technical barriers that can hinder interaction with digital exhibits. Similarly, the “high engagement” observed—characterized by prolonged interaction times and expressions of curiosity and play—fulfills the goal of creating an immersive experience. This aligns with established findings that successful museum kiosks adopt a playful and educational approach to foster higher levels of cognitive engagement and enjoyment [30].
The effectiveness of the controls can likely be attributed to their intuitive metaphorical mapping. The “pinch-and-rotate” gesture for 3D manipulation, in particular, maps metaphorically to how a person would physically pick up and handle a small object like a rock or crystal. This cognitive link between a physical action and a digital outcome makes the interaction feel natural and immediate, reducing the learning curve often associated with new technologies.

4.2. Contribution and Context

This work directly addresses several key challenges and opportunities, many of which were identified during the conceptualization of the “Mineralogy 4.0” framework [11]. PETRA presents a practical model for how AI (via MediaPipe’s hand tracking) and 3D technologies can be effectively integrated into a museum setting with low overhead and minimal hardware requirements. As a novel tool, PETRA’s real-world impact was demonstrated through the direct user engagement at our case study event, moving beyond theoretical application. Furthermore, this work confronts the obstacles of cost and hardware complexity that often hinder the widespread adoption of VR/AR, offering a more accessible and scalable alternative for institutions. This specific application of AI for user interaction complements its growing use in cultural heritage for other tasks, ranging from content recognition in digital archives [48] to the virtual restoration of artifacts [34].
Situating PETRA within the broader landscape of Human–Machine Interaction (HMI) highlights its contribution as a practical and timely case study in Natural User Interfaces (NUIs). The rapid learnability observed in the case study aligns with core NUI design principles, which aim for the interface to become “invisible” by leveraging skills users have acquired over a lifetime [49]. This approach positions PETRA as a compelling alternative to other HMI paradigms. Unlike immersive VR systems, which can be isolating, or Tangible User Interfaces (TUIs), which require physical replicas for interaction [40], PETRA’s vision-based, touchless method fosters a more open and social form of engagement. Specifically, PETRA can be viewed as an evolution of the traditional museum kiosk. While standard kiosks primarily facilitate individual “learner–content interaction” [30], the large-screen, public-facing nature of PETRA was observed to encourage collaborative use, offering a potential solution to this long-standing challenge in museum-based HMI [50]. This fosters a shared experience that aligns with established museum learning theories, such as Falk and Dierking’s Contextual Model of Learning, which emphasizes the importance of the sociocultural context in shaping a visitor’s understanding [51]. By successfully implementing a low-cost, hardware-agnostic solution, this work provides a valuable data point for the ongoing evolution of somatosensory and vision-based interaction.

4.3. Benefits and Limitations

The PETRA approach provides a number of clear advantages for institutional use. First and foremost, the system is highly accessible. By utilizing a standard webcam, it removes the need for expensive and complex hardware like VR headsets or specialized controllers, tackling a significant barrier that often prevents museums and schools from adopting new technologies [30]. This design significantly lowers both financial and logistical costs. Beyond its accessibility, the touchless nature of the interface makes it inherently hygienic, a critical factor for any public installation. Finally, the novelty of the gesture-based control itself serves as a powerful factor for engaging and retaining visitor attention.
The distinct advantages of the PETRA approach compared to other common interactive technologies are summarized in Table 1.
However, the current implementation has several limitations that provide clear avenues for future research.
  • Technical Limitations: The gesture recognition is dependent on adequate lighting and can be compromised in low-light environments. For reliable tracking, standard indoor ambient lighting is sufficient, but the system may struggle with strong backlighting or very dim conditions where the hand is not clearly visible to the camera. The system is also optimized for a single user; the presence of multiple hands can cause tracking errors, reflecting a known challenge in balancing individual and social experiences in museum interactives [50].
  • Interaction Limitations: The current gesture vocabulary is intentionally simple. It does not, for example, include a method for accessing the textual metadata about the specimens, which is a key informational function of traditional museum kiosks [30].
  • Evaluation Limitations: The findings from this case study are qualitative, based on direct observation of “in-the-wild” user interactions. While this provides valuable initial insights, a formal quantitative evaluation was not performed. A crucial direction for future work is to conduct controlled studies that not only use validated instruments like the System Usability Scale (SUS), but also benchmark PETRA’s technical performance (e.g., gesture accuracy rates, system latency) against other interactive systems. This would provide the rigorous data needed to explicitly validate the system’s effectiveness and potential technical advantages [1,30].

4.4. Broader Implications and Future Work

The success of this case study has broader implications for several fields. Within mineralogy education, PETRA offers a new paradigm for virtual classrooms, allowing students to remotely “handle” and inspect digital specimens in a way that is more interactive than simple on-screen models, aligning with calls for more interactive 3D models in geoscience teaching [11,22]. Its potential as a tool for remote or hybrid teaching scenarios, where it can serve as the basis for interactive online geology labs, is significant. Furthermore, this hands-on, exploratory approach complements other emerging AI-driven educational tools, such as deep learning systems for identifying minerals in microscopic images [23].
For museum exhibition design, it presents an affordable and engaging alternative to traditional interactive kiosks or more complex VR installations. More broadly, for digital humanities and accessibility, it contributes to the development of more inclusive digital experiences that do not rely on controllers that some users may find difficult to operate.
Furthermore, the versatility of the PETRA framework extends beyond the specific collection of rocks and minerals presented here. Its underlying architecture can be readily adapted for use in other geological disciplines and even entirely different scientific fields. For example, in crystallography education, the system could be used to load and manipulate the crystal forms, allowing students to intuitively grasp complex symmetries and spatial relationships in a way that static diagrams cannot. In other domains, such as biology or medicine, the same interface could allow for the hands-on exploration of complex protein structures, cellular organelles, or detailed anatomical models for student training and public outreach. In any field where the understanding of complex three-dimensional structures is crucial, this accessible, gesture-based approach offers a powerful new tool for teaching and engagement.
Future work will focus on addressing the current limitations and expanding the system’s capabilities. The immediate next steps include conducting formal user studies to gather quantitative data on usability and learning outcomes. The interaction model itself can be enhanced by expanding the gesture vocabulary to include actions like displaying metadata panels, potentially triggered by a simple, distinct gesture such as an open palm or a “victory” sign. Another promising direction is exploring full-body pose detection to lay the groundwork for more advanced and embodied interactions, such as creating immersive learning games or animated avatars for younger visitors. To enhance visual immersion, stereoscopic 3D rendering (e.g., anaglyph red-cyan) could be implemented, creating a powerful depth illusion that is still viewable with low-cost glasses and aligns with the project’s accessibility goals. Furthermore, exploring a two-handed interaction mode, in which one hand is dedicated to specimen selection and the other to manipulation, could resolve the multi-user tracking issue and provide a richer, more fluid control scheme. As a more advanced research direction, one could explore the use of machine learning to create personalized gesture profiles, allowing the system to adapt to an individual user’s unique range of motion or physical abilities, further enhancing its accessibility and inclusivity. Finally, future versions could integrate with online databases like Sketchfab using their API, allowing the system to dynamically load 3D models and their associated metadata from a vast, ever-growing collection.

5. Conclusions

This paper introduces PETRA, a novel gesture-controlled interface designed to enhance the accessibility and interactivity of 3D geological collections in museum and educational settings. Serving as a practical case study for the emerging “Mineralogy 4.0” paradigm, this work successfully demonstrated the integration of 3D technologies and machine learning to create an engaging user experience.
The implementation of PETRA using standard, low-cost hardware and an open-source, node-based development environment confirms its viability as an accessible solution for institutions. The findings from the “Long Night of Museums” case study further validated this approach, revealing high levels of public engagement, rapid learnability across diverse age groups, and robust technical stability.
By providing an intuitive, touchless, and socially engaging method for exploring digital specimens, PETRA offers a significant contribution to the fields of digital mineralogy, human–machine interaction, and cultural heritage. This research confirms that as digital collections continue to grow, the development of human-centered interfaces will be paramount in unlocking their full scientific and educational potential for the widest possible audience.

Funding

This research received no external funding.

Data Availability Statement

The complete dataset and project files generated and analyzed during this study are openly available in the PETRA GitHub repository at: https://github.com/aapopei/PETRA (accessed on 19 June 2025).

Acknowledgments

The author wishes to thank the “Grigore Cobălcescu” Museum and the Department of Geology at the “Alexandru Ioan Cuza” University of Iaşi for providing the venue and opportunity to conduct the case study during the “Long Night of Museums” event. Gratitude is also extended to the student volunteers who assisted during the public installation. The author gratefully acknowledges Torin Blankensmith for developing and maintaining the MediaPipe TouchDesigner plugin, which was a critical component for the implementation of the PETRA system.

Conflicts of Interest

The author declares no conflicts of interest.

References

  1. Arici, F.; Yildirim, P.; Caliklar, Ş.; Yilmaz, R.M. Research trends in the use of augmented reality in science education: Content and bibliometric mapping analysis. Comput. Educ. 2019, 142, 103647. [Google Scholar] [CrossRef]
  2. Champion, E.; Rahaman, H. 3D Digital Heritage Models as Sustainable Scholarly Resources. Sustainability 2019, 11, 2425. [Google Scholar] [CrossRef]
  3. Clini, P.; Nespeca, R.; Ferretti, U.; Galazzi, F.; Bernacchia, M. Inclusive Museum Engagement: Multisensory Storytelling of Cagli Warriors’ Journey and the Via Flamina Landscape Through Interactive Tactile Experiences and Digital Replicas. Heritage 2025, 8, 61. [Google Scholar] [CrossRef]
  4. Pescarin, S. Museums and Virtual Museums in Europe Reaching expectations. SCIRES-IT Sci. Res. Inf. Technol. 2014, 4, 131–140. [Google Scholar] [CrossRef]
  5. Wachowiak, M.J.; Karas, B.V. 3D scanning and replication for museum and cultural heritage applications. J. Am. Inst. Conserv. 2009, 48, 141–158. [Google Scholar] [CrossRef]
  6. Wali, A.; Collins, R.K. Decolonizing Museums: Toward a Paradigm Shift. Annu. Rev. Anthropol. 2023, 52, 329–345. [Google Scholar] [CrossRef]
  7. De Paor, D.G. Virtual Rocks. GSA Today 2016, 26, 4–11. [Google Scholar] [CrossRef]
  8. Janiszewski, M.; Uotinen, L.; Merkel, J.; Leveinen, J.; Rinne, M. Virtual Reality learning environments for rock engineering, geology and mining education. In Proceedings of the 54th U.S. Rock Mechanics/Geomechanics Symposium, Golden, CO, USA, 28 June–1 July 2020. [Google Scholar]
  9. Ma, X.; Ralph, J.; Zhang, J.; Que, X.; Prabhu, A.; Morrison, S.M.; Hazen, R.M.; Wyborn, L.; Lehnert, K. OpenMindat: Open and FAIR mineralogy data from the Mindat database. Geosci. Data J. 2024, 11, 94–104. [Google Scholar] [CrossRef]
  10. Prabhu, A.; Morrison, S.M.; Eleish, A.; Zhong, H.; Huang, F.; Golden, J.J.; Perry, S.N.; Hummer, D.R.; Ralph, J.; Runyon, S.E.; et al. Global earth mineral inventory: A data legacy. Geosci. Data J. 2021, 8, 74–89. [Google Scholar] [CrossRef]
  11. Apopei, A.I. Towards Mineralogy 4.0? Atlas of 3D Rocks and Minerals: Digitally Archiving Interactive and Immersive 3D Data of Rocks and Minerals. Minerals 2024, 14, 1196. [Google Scholar] [CrossRef]
  12. Apopei, A.I.; Buzgar, N.; Buzatu, A.; Maftei, A.E.; Apostoae, L. Digital 3d Models of Minerals and Rocks in a Nutshell: Enhancing Scientific, Learning, and Cultural Heritage Environments in Geosciences by Using Cross-Polarized Light Photogrammetry. Carpathian J. Earth Environ. Sci. 2021, 16, 237–249. [Google Scholar] [CrossRef]
  13. Cocal-Smith, V.; Hinchliffe, G.; Petterson, M.G. Digital Tools for the Promotion of Geological and Mining Heritage: Case Study from the Thames Goldfield, Aotearoa, New Zealand. Geosciences 2023, 13, 253. [Google Scholar] [CrossRef]
  14. Papadopoulos, C.; Gillikin Schoueri, K.; Schreibman, S. And Now What? Three-Dimensional Scholarship and Infrastructures in the Post-Sketchfab Era. Heritage 2025, 8, 99. [Google Scholar] [CrossRef]
  15. De Reu, J.; Plets, G.; Verhoeven, G.; De Smedt, P.; Bats, M.; Cherretté, B.; De Maeyer, W.; Deconynck, J.; Herremans, D.; Laloo, P.; et al. Towards a three-dimensional cost-effective registration of the archaeological heritage. J. Archaeol. Sci. 2013, 40, 1108–1121. [Google Scholar] [CrossRef]
  16. Fais, S.; Cuccuru, F.; Casula, G.; Bianchi, M.G.; Ligas, P. Characterization of rock samples by a high-resolution multi-technique non-invasive approach. Minerals 2019, 9, 664. [Google Scholar] [CrossRef]
  17. Mathys, A.; Semal, P.; Brecko, J.; Van den Spiegel, D. Improving 3D photogrammetry models through spectral imaging: Tooth enamel as a case study. PLoS ONE 2019, 14, e0220949. [Google Scholar] [CrossRef]
  18. Matsui, K.; Kimura, Y. Museum Exhibitions of Fossil Specimens Into Commercial Products: Unexpected Outflow of 3D Models due to Unwritten Image Policies. Front. Earth Sci. 2022, 10, 874736. [Google Scholar] [CrossRef]
  19. Sapirstein, P. A high-precision photogrammetric recording system for small artifacts. J. Cult. Herit. 2018, 31, 33–45. [Google Scholar] [CrossRef]
  20. Sutton, M.D.; Rahman, I.A.; Garwood, R.J. Techniques for Virtual Palaeontology; John Wiley & Sons: Hoboken, NJ, USA, 2014; pp. 1–200. [Google Scholar]
  21. Darda, K.M.; Estrada Gonzalez, V.; Christensen, A.P.; Bobrow, I.; Krimm, A.; Nasim, Z.; Cardillo, E.R.; Perthes, W.; Chatterjee, A. A comparison of art engagement in museums and through digital media. Sci. Rep. 2025, 15, 8972. [Google Scholar] [CrossRef]
  22. Andrews, G.D.M.; Labishak, G.D.; Brown, S.R.; Isom, S.L.; Pettus, H.D.; Byers, T. Teaching with Digital 3D Models of Minerals and Rocks. GSA Today 2020, 30, 42–43. [Google Scholar] [CrossRef]
  23. Zhang, J.; Gao, Q.; Luo, H.; Long, T. Mineral Identification Based on Deep Learning Using Image Luminance Equalization. Appl. Sci. 2022, 12, 7055. [Google Scholar] [CrossRef]
  24. Davies, N.; Deck, J.; Kansa, E.C.; Kansa, S.W.; Kunze, J.; Meyer, C.; Orrell, T.; Ramdeen, S.; Snyder, R.; Vieglais, D.; et al. Internet of Samples (iSamples): Toward an interdisciplinary cyberinfrastructure for material samples. Gigascience 2021, 10, giab028. [Google Scholar] [CrossRef]
  25. Davies, T.G.; Rahman, I.A.; Lautenschlager, S.; Cunningham, J.A.; Asher, R.J.; Barrett, P.M.; Bates, K.T.; Bengtson, S.; Benson, R.B.J.; Boyer, D.M.; et al. Open data and digital morphology. Proc. R. Soc. B Biol. Sci. 2017, 284, 20170194. [Google Scholar] [CrossRef] [PubMed]
  26. Hardisty, A.; Addink, W.; Glöckler, F.; Güntsch, A.; Islam, S.; Weiland, C. A choice of persistent identifier schemes for the Distributed System of Scientific Collections (DiSSCo). Res. Ideas Outcomes 2021, 7, e67379. [Google Scholar] [CrossRef]
  27. Rivero, O.; Dólera, A.; García-Bustos, M.; Eguilleor-Carmona, X.; Mateo-Pellitero, A.M.; Ruiz-López, J.F. Seeing is believing: An Augmented Reality application for Palaeolithic rock art. J. Cult. Herit. 2024, 69, 67–77. [Google Scholar] [CrossRef]
  28. Wellmann, F.; Virgo, S.; Escallon, D.; De La Varga, M.; Jüstel, A.; Wagner, F.M.; Kowalski, J.; Zhao, H.; Fehling, R.; Chen, Q. Open AR-Sandbox: A haptic interface for geoscience education and outreach. Geosphere 2022, 18, 732–749. [Google Scholar] [CrossRef]
  29. Reed, S.E.; Kreylos, O.; Hsi, S.; Kellogg, L.H.; Schladow, G.; Yikilmaz, M.B.; Segale, H.; Silverman, J.; Yalowitz, S.; Sato, E. Shaping Watersheds Exhibit: An Interactive, Augmented Reality Sandbox for Advancing Earth Science Education. In AGU Fall Meeting Abstracts; Department of Energy: Washington, DC, USA, 2014; Volume 2014. [Google Scholar]
  30. Barbieri, L.; Bruno, F.; Muzzupappa, M. Virtual museum system evaluation through user studies. J. Cult. Herit. 2017, 26, 101–108. [Google Scholar] [CrossRef]
  31. García-Gil, G.; del Carmen López-Armas, G.; de Jesús Navarro, J. Human-Machine Interaction: A Vision-Based Approach for Controlling a Robotic Hand Through Human Hand Movements. Technologies 2025, 13, 169. [Google Scholar] [CrossRef]
  32. Piumsomboon, T.; Clark, A.; Billinghurst, M.; Cockburn, A. User-Defined Gestures for Augmented Reality. In Proceedings of the 14th International Conference on Human-Computer Interaction (INTERACT), Cape Town, South Africa, 10–12 September 2013; pp. 282–299. [Google Scholar]
  33. Wobbrock, J.O.; Morris, M.R.; Wilson, A.D. User-defined gestures for surface computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, MA, USA, 4–9 April 2009; pp. 1083–1092. [Google Scholar]
  34. Pietroni, E.; Ferdani, D. Virtual Restoration and Virtual Reconstruction in Cultural Heritage: Terminology, Methodologies, Visual Representation Techniques and Cognitive Models. Information 2021, 12, 167. [Google Scholar] [CrossRef]
  35. Lugaresi, C.; Tang, J.; Nash, H.; McClanahan, C.; Uboweja, E.; Hays, M.; Zhang, F.; Chang, C.-L.; Yong, M.G.; Lee, J. Mediapipe: A framework for building perception pipelines. arXiv 2019, arXiv:1906.08172. [Google Scholar]
  36. Bradski, G. The OpenCV library. Dr. Dobb’s J. Softw. Tools 2000, 120, 122–125. [Google Scholar]
  37. Abadi, M.; Barham, P.; Chen, J.M.; Chen, Z.F.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; et al. TensorFlow: A system for large-scale machine learning. In Proceedings of the Osdi’16: 12th Usenix Symposium on Operating Systems Design and Implementation, Savannah, GA, USA, 2–4 November 2016; pp. 265–283. [Google Scholar] [CrossRef]
  38. Yoshida, R.; Tamaki, H.; Sakai, T.; Nakadai, T.; Ogitsu, T.; Takemura, H.; Mizoguchi, H.; Kamiyama, S.; Yamaguchi, E.; Inagaki, S.; et al. Novel Application of Kinect Sensor to Support Immersive Learning within Museum for Children. In Proceedings of the 2015 9th International Conference on Sensing Technology (ICST), Auckland, New Zealand, 8–10 December 2015; pp. 834–837. [Google Scholar] [CrossRef]
  39. Newcombe, R.A.; Izadi, S.; Hilliges, O.; Molyneaux, D.; Kim, D.; Davison, A.J.; Kohli, P.; Shotton, J.; Hodges, S.; Fitzgibbon, A. KinectFusion: Real-Time Dense Surface Mapping and Tracking. In Proceedings of the 2011 IEEE International Symposium on Mixed and Augmented Reality, Basel, Switzerland, 26–29 October 2011; pp. 127–136. [Google Scholar] [CrossRef]
  40. Palma, G.; Perry, S.; Cignoni, P. Augmented Virtuality Using Touch-Sensitive 3D-Printed Objects. Remote Sens. 2021, 13, 2186. [Google Scholar] [CrossRef]
  41. Berrezueta-Guzman, S.; Wagner, S. Immersive Multiplayer VR: Unreal Engine’s Strengths, Limitations, and Future Prospects. IEEE Access 2025, 13, 85597–85612. [Google Scholar] [CrossRef]
  42. Boutsi, A.M.; Ioannidis, C.; Soile, S. Interactive Online Visualization of Complex 3d Geometries. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2019, XLII-2/W9, 173–180. [Google Scholar] [CrossRef]
  43. Jerkku, I. Designing and Developing an Educational VJ Tool. Master’s Thesis, Aalto University, Espoo, Finland, 2024. [Google Scholar]
  44. Kuoppala, P. Real-Time Image Generation for Interactive Art: Developing an Artwork Platform with ComfyUI and TouchDesigner. Bachelor’s Thesis, Tampere University of Applied Sciences, Pirkanmaa, Finland, 2025. [Google Scholar]
  45. Ismiranti, A.S.; Kogia, A. Utilizing Digital Body Tracking Systems to Develop Sustainable and Ideal Workspace. KnE Social Sci. 2025, 10, 1–8. [Google Scholar] [CrossRef]
  46. Blankensmith, T. MediaPipe TouchDesigner Plugin. 2025. Available online: https://github.com/torinmb/mediapipe-touchdesigner (accessed on 19 June 2025).
  47. Zhang, F.; Bazarevsky, V.; Vakunov, A.; Tkachenka, A.; Sung, G.; Chang, C.-L.; Grundmann, M. MediaPipe Hands: On-device Real-time Hand Tracking. arXiv 2020, arXiv:2006.10214. [Google Scholar]
  48. Ioannakis, G.; Bampis, L.; Koutsoudis, A. Exploiting artificial intelligence for digitally enriched museum visits. J. Cult. Herit. 2020, 42, 171–180. [Google Scholar] [CrossRef]
  49. Kyriakou, P.; Hermon, S. Can I touch this? Using Natural Interaction in a Museum Augmented Reality System. Digit. Appl. Archaeol. Cult. Herit. 2019, 12, e00088. [Google Scholar] [CrossRef]
  50. Shehade, M.; Stylianou-Lambert, T. Virtual reality in museums: Exploring the experiences of museum professionals. Appl. Sci. 2020, 10, 4031. [Google Scholar] [CrossRef]
  51. Falk, J.H.; Dierking, L.D. The Museum Experience Revisited; Routledge: Abingdon, UK, 2016. [Google Scholar]
Figure 1. The interactive feedback loop and data flow of the PETRA system architecture. The process originates with the User performing a gesture, which is captured by the Webcam and sent as an input Live Video Stream to the Computer. The PETRA application processes this data, performing gesture recognition and 3D rendering. The resulting Rendered 3D Scene is sent as output to the Display, providing Visual Feedback to the user. This closed loop enables the intuitive, real-time manipulation of the 3D models.
Figure 1. The interactive feedback loop and data flow of the PETRA system architecture. The process originates with the User performing a gesture, which is captured by the Webcam and sent as an input Live Video Stream to the Computer. The PETRA application processes this data, performing gesture recognition and 3D rendering. The resulting Rendered 3D Scene is sent as output to the Display, providing Visual Feedback to the user. This closed loop enables the intuitive, real-time manipulation of the 3D models.
Minerals 15 00775 g001
Figure 2. The gesture control scheme for the PETRA system. (a) The model selection gesture, where the user pivots their index finger from a vertical to a horizontal orientation to advance to the next specimen. (b) The 3D manipulation gesture, which is activated with a “pinch” of the thumb and index finger. While this pinch gesture is active, hand movement controls the model’s rotation, and changing the distance between the fingers controls the zoom.
Figure 2. The gesture control scheme for the PETRA system. (a) The model selection gesture, where the user pivots their index finger from a vertical to a horizontal orientation to advance to the next specimen. (b) The 3D manipulation gesture, which is activated with a “pinch” of the thumb and index finger. While this pinch gesture is active, hand movement controls the model’s rotation, and changing the distance between the fingers controls the zoom.
Minerals 15 00775 g002
Figure 3. Hand landmark detection for gesture recognition in the PETRA system. (a) A screenshot from the application’s diagnostic view showing the real-time skeletal tracking of a user’s hand. (b) The official 21-keypoint hand landmark model from Google’s MediaPipe documentation (the hand tracking technology is detailed in [47]), which provides the anatomical basis for the tracking data. The gestures in PETRA are calculated from the spatial relationships between these numbered keypoints.
Figure 3. Hand landmark detection for gesture recognition in the PETRA system. (a) A screenshot from the application’s diagnostic view showing the real-time skeletal tracking of a user’s hand. (b) The official 21-keypoint hand landmark model from Google’s MediaPipe documentation (the hand tracking technology is detailed in [47]), which provides the anatomical basis for the tracking data. The gestures in PETRA are calculated from the spatial relationships between these numbered keypoints.
Minerals 15 00775 g003
Figure 4. Implementation and live interaction view of the PETRA system. (a) The complete project network within the TouchDesigner development environment, illustrating the node-based logic and the color-coded organization described in the methods, where (1) is the main web camera input node; (2) is the group nodes with scene setup and interaction logic; (3) is the group nodes with 3D models and audio playlist; (4) is the user interface setup; and (5) is the final output nodes. (b) A screenshot of the final application in portrait performance mode, showing a user’s hands performing a “pinch” gesture to manipulate a 3D model of pyrite in real-time. This figure demonstrates the direct link between the backend implementation (a) and the resulting interactive user experience (b).
Figure 4. Implementation and live interaction view of the PETRA system. (a) The complete project network within the TouchDesigner development environment, illustrating the node-based logic and the color-coded organization described in the methods, where (1) is the main web camera input node; (2) is the group nodes with scene setup and interaction logic; (3) is the group nodes with 3D models and audio playlist; (4) is the user interface setup; and (5) is the final output nodes. (b) A screenshot of the final application in portrait performance mode, showing a user’s hands performing a “pinch” gesture to manipulate a 3D model of pyrite in real-time. This figure demonstrates the direct link between the backend implementation (a) and the resulting interactive user experience (b).
Minerals 15 00775 g004
Figure 5. Visitors interacting with the PETRA installation during the “Long Night of Museums” event. The collage showcases the engagement of a diverse audience with the system. Key moments captured include collaborative exploration by family groups (top-right), individual interaction between users (top-left), and a young visitor using the “pinch” gesture to directly manipulate the 3D model of malachite (bottom-right).
Figure 5. Visitors interacting with the PETRA installation during the “Long Night of Museums” event. The collage showcases the engagement of a diverse audience with the system. Key moments captured include collaborative exploration by family groups (top-right), individual interaction between users (top-left), and a young visitor using the “pinch” gesture to directly manipulate the 3D model of malachite (bottom-right).
Minerals 15 00775 g005
Table 1. Comparison of interactive system approaches.
Table 1. Comparison of interactive system approaches.
FeaturePETRA (Webcam)Kinect/Leap MotionVR Headset
Hardware costVery LowLow to MediumMedium to High
Hardware dependencyany webcamspecific sensorspecific headset
Setup complexityVery LowMediumHigh
Hygiene (Touchless)ExcellentExcellentPoor (Shared headset)
User isolationLow (Social)Low (Social)High (Individual)
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Apopei, A.I. Accessible Interface for Museum Geological Exhibitions: PETRA—A Gesture-Controlled Experience of Three-Dimensional Rocks and Minerals. Minerals 2025, 15, 775. https://doi.org/10.3390/min15080775

AMA Style

Apopei AI. Accessible Interface for Museum Geological Exhibitions: PETRA—A Gesture-Controlled Experience of Three-Dimensional Rocks and Minerals. Minerals. 2025; 15(8):775. https://doi.org/10.3390/min15080775

Chicago/Turabian Style

Apopei, Andrei Ionuţ. 2025. "Accessible Interface for Museum Geological Exhibitions: PETRA—A Gesture-Controlled Experience of Three-Dimensional Rocks and Minerals" Minerals 15, no. 8: 775. https://doi.org/10.3390/min15080775

APA Style

Apopei, A. I. (2025). Accessible Interface for Museum Geological Exhibitions: PETRA—A Gesture-Controlled Experience of Three-Dimensional Rocks and Minerals. Minerals, 15(8), 775. https://doi.org/10.3390/min15080775

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop