Grasping in Shared Virtual Environments: Toward Realistic Human–Object Interaction Through Review-Based Modeling

Abstract

Virtual communication, involving the transmission of all human senses, is the next step in the development of telecommunications. Achieving this vision requires real-time data exchange with low latency, which in turn necessitates the implementation of the Tactile Internet (TI). TI will ensure the transmission of high-quality tactile data, especially when combined with audio and video signals, thus enabling more realistic interactions in virtual environments. In this context, advances in realism increasingly depend on the accurate simulation of the grasping process and hand–object interactions. To address this, in this paper, we methodically present the challenges of human–object interaction in virtual environments, together with a detailed review of the datasets used in grasping modeling and the integration of physics-based and machine learning approaches. Based on this review, we propose a multi-step framework that simulates grasping as a series of biomechanical, perceptual, and control processes. The proposed model aims to support realistic human interaction with virtual objects in immersive settings and to enable integration into applications such as remote manipulation, rehabilitation, and virtual learning.

1. Introduction

Grasping is one of the human motor actions that allows us to interact with and manipulate objects in our environment. Virtual communication is an important part of the developing modern world, combining various methods and algorithms, including intelligent systems for data analysis, visualization, and interaction. It is used as a substitute for physical presence, but not for real-time communication, creating new opportunities for interaction and presentation of information. In this regard, Tactile Internet (TI) provides real-time communication characterized by extremely low latency and high reliability [1]. This, in turn, allows not only the transmission of audio and video data but also sensor data, providing prerequisites for precise remote capture and manipulation of objects in virtual environments. The introduction of TI into augmented reality (AR), virtual reality (VR), and mixed reality (MR) would significantly improve the realism of interactions, overcoming important limitations that exist in the modeling of human–object interactions [2]. The main stages in the implementation of this type of communication range from multidimensional data processing to the creation of holographic environments and virtual applications.

For example, stroke rehabilitation applications that use haptic feedback to help patients re-think hand movements could benefit from the Tactile Internet. Similarly, simulations for training engineers, mechanics, and surgeons could mimic real-world tasks and provide learners with hands-on experience in virtual environments [3]. The Tactile Internet could provide realistic simulations of remote grasping and control without exposing the operator to danger in situations where physical access is limited or dangerous. This could be achieved while maintaining extremely accurate grasping by combining real-time control and haptic feedback. Simulations could include emergency, accident, or disaster response scenarios, where robots with tactile sensors could be programmed to perform delicate tasks such as moving broken objects or debris [4]. The constant tactile feedback provided by the Tactile Internet (TI) allows for real-time adaptation to different shapes and materials of objects. This enables the grasping models to dynamically adjust to the hardness, texture, and deformation of objects [5].

The potential of grasping, in the context of its application through the Tactile Internet in virtual, augmented, and mixed reality, is gaining even greater importance in areas such as mental health and medical education. For example, Space Exodus is a VR role-playing game for children with intellectual disabilities that has been shown to improve fine motor skills, attention, and engagement after six weeks of use [6]. Similarly, virtual reality simulations based on problem-based learning allow medical students to study remotely without losing the quality of education [7]. Furthermore, VR therapies are effective in reducing symptoms of anxiety, depression, and stress, suggesting that integrating grip modeling and haptic feedback could enhance their effect and therapeutic value [8].

Although the Tactile Internet is the next step in the development of telecommunications, its implementation and modeling in virtual environments must meet a number of requirements. The first condition is to achieve submillisecond latency for real-time feedback, especially in large-scale or geographically dispersed networks [9]. Then it is necessary to ensure the transmission of high-quality tactile data. This has a major impact on network capacity, especially when audio and video data are added [10]. Further, it is necessary to develop fusion algorithms to synchronize haptic feedback with other sensory modalities in dynamic environments. Data ethics and security, privacy, and accountability in failure scenarios are necessary for applications in telemedicine and human–robot collaboration [11]. Potential solutions to these limitations would be the development of 6G networks, the creation of operating protocols, and optimization. It should be noted that the creation of standardized protocols for tactile data transmission and synchronization is also necessary, which would facilitate the implementation of the TI in grasping modeling.

Motivation: Human–object interactions in virtual environments should become more natural and realistic with the development of VR, AR, and MR. However, these applications are currently limited by challenges related to accurately simulating the grasping process and hand–object interactions [12]. Nevertheless, limitations arise in scenarios where physical access is dangerous or limited, in particular when incorporating real-time feedback, dynamic adaptation to changing object attributes, and the need to accurately simulate human grasping [13]. Accurate real-time exchange of tactile and sensory data is possible thanks to the Tactile Internet. Implementing tactile and sensory infrastructure in grasping modeling requires high-quality datasets, accurate physics-based simulations, and building machine learning models to ensure realism and scalability.

Contributions:

This article makes the following important contributions:

1.

A seven-step grasping framework: We propose a multi-layered grasping model that combines biomechanical, perceptual, physical, and contextual processes. Our framework differs from others because it combines the layers of object geometry and physics into a single sequence, making human–object interactions more realistic and adaptive.

2.

Addressing technical issues: The proposed framework methodically addresses a number of important technical issues. We consider the latency and synchronization of multimodal feedback. We propose ways to address the computational complexity of real-time physics-based simulations. The model takes into account data limitations, such as data set bias and lack of diversity. We explore approaches to modeling deformable and composite objects that have not been extensively addressed in the majority of the current literature.

3.

Real-World Effects: Our conceptual framework supports real-world applications in areas where safe and realistic interactions are crucial, by connecting grasping modeling to the Tactile Internet and low-latency communication over 6G.

Structure: The article is organized into six main sections. Section 1 presents the motivation behind the study and our contributions. In Section 2, a process for modeling grasping in virtual environments is proposed. In Section 3, the role of physics-based modeling in grasping is presented. In Section 4, we present the more commonly used datasets in different studies and compare them. In Section 5, the effectiveness of grasping models is given, and potential limitations are suggested. In Section 6, future directions for grasping modeling research are suggested.

Figure 1 presents different examples of a realistic hand in a virtual environment, interacting with objects of different shapes and materials (grasping objects). Several positions of a hand grasping a green cube (a solid, geometrically simple object) are shown. The hand uses different grips, which is a consequence of a change in the positioning of the fingers and the dynamics of the interaction. Then, the hand is presented interacting with a blue object (a geometrically complex object), similar to a rabbit figurine. The type of material cannot be determined from the visual characteristics, but it can be assumed based on knowledge about the object itself. This creates uncertainty regarding its virtual characteristics (deformability, etc.). The third object with which the virtual hand interacts is a complex geometric structure containing intertwined circles and a central red part, which, in addition to the material, has no knowledge of what the object represents. In this case, a decision must be made about the interaction when choosing a grip—whether the object is likely to bend under pressure, whether precise gripping of small and thin parts is necessary, etc. Finally, an example with gripping a familiar object from everyday life (a jug) is presented, where different possibilities for gripping are possible—the edge of the jug, and in the second one, the handle is used. All these examples aim to highlight the importance of gripping depending on the shape and functionality of the object. In this section, we will describe the grasping process in seven steps. Together, these steps enable the construction of a complete, realistic grasping model that mimics human behavior in virtual environments. The next sections will expand on the data, physics, and evaluation mechanisms used to support each stage of this framework.

2. Modeling of the Grasping Process

In order to model grasping in a virtual environment, it is first necessary to consider the process in a real environment (see Figure 2).

Ideally, the hand is initially in a random position; of course, it has a certain range of motion and is at rest. We assume that there is a zero instant during which the hand does not move, even when one action is switched to another. If we assume that we have information about the object (shape, mass, volume, material, etc.) and know its position (located within the range of allowed coordinates for movement), then we first start by changing the position of the hand itself. Then we make a choice of grip, including the number of fingers.

There are different grasping strategies depending on the object geometry, material, and scene constraints (Figure 1). This is a decision that has become unconscious in humans over time and is influenced not only by the object itself but also by the environment in which it is located. After a choice has been made, for example, to use five fingers and the palm, they are in a position that determines a volume greater than the volume of the object itself. The hand approaches the object and grasps it while simultaneously applying force to perform an additional action: lifting, moving, pushing, etc. This subsequent action requires a change in the initial force in response to the real capabilities of the hand, i.e., even with initial knowledge of the object and the hypothetical forces that would need to be applied, it is possible for a correction to occur.

Modeling grasping in a virtual environment involves simulating the processes of perception, control, and adaptation that allow humans to interact with various objects. To adapt grasping to specific tasks, this involves building geometric and physical models, modeling the movement of one or both hands and the dynamics of the contact itself, as well as integrating contextual information. We split the modeling process into three main stages (see Figure 3):

• Stage 1: Perceptual Modeling (Steps 1–2)—representation of the hand and the object and identification of contact candidates;
• Stage 2: Physical and Dynamic Modeling (Steps 3–6)—simulation of the movement, applying physics, and refining the grip through feedback;
• Stage 3: Contextual Adaptation (Step 7)—adjustment of the grip according to the scene, the capabilities of the object, or the goals of the task.

In this section, we present a framework consisting of seven steps, which are summarized in Figure 3. The steps are sequential and aim to represent the construction of realistic grasping interactions.

2.1. Step 1: Geometric Representation of Hand and Object

The first step of the modeling process is a digital representation of the real world, where geometric and structural descriptions of the human hand and objects can be used. Realistic and physically plausible interactions can be produced using a variety of techniques, including a combination of geometric representations, sensor-based feedback, and simulation methods. To represent the human hand, typically, high-resolution 3D meshes, joint angle mapping, or parametric models such as MANO [15] or SMPL-X [16] are used. Other methods include fitting a Universal Hand Model (UHM) [17], which simulates accurate articulation and deformations of the hand using 3D joint coordinates, identification codes, and pose matching [17,18]. Depending on the computational needs of the application and the level of precision, objects can be modeled using Signed Distance Functions (SDFs), voxel meshes, triangle meshes, or point clouds. Geometric models can be derived from synthetic environments or photogrammetric reconstruction, as well as from real-world sensors such as RGB-D cameras, LiDAR, or tactile arrays, to facilitate contact prediction and simulation. To have the possibility for estimation of possible interaction zones and creating smooth surface representations of the hand and object, point clouds, 3D Gaussian representations, and implicit surface modeling can be used [19]. If we combine these representations, we will obtain spatial information about size, shape, and relative location in three dimensions, and can serve as input for the next step in the grasping process.

2.2. Step 2: Predicting Contact Points

It is possible to predict the contact areas during the interaction using geometric representations of the hand and the object. Predicting both the contact points and the associated forces required for stable interaction is the goal of contact estimation. This can be referred to as an optimal control problem [20,21]. Various methods are used for this purpose, including proximity sensors applied to the phalanges of the fingers or capsule triggers for collision detection. They detect the initial contact and calculate the distances between the hand and the object surface [22]. Other approaches are training-based and use annotated contact map datasets to predict interaction areas. The third type of techniques is optimization-based. Their goal is to refine hand postures by minimizing the elastic or potential energy in the contact fields [23]. The contact points can be dynamically adjusted during the grasping process using physics-based models that impose a balance of force and torque to maintain grip stability [24,25]. Accuracy becomes essential for the accuracy of the simulation as well as for the performance of the computer during virtual grasping, and these contact points are necessary to ensure stable, physically realistic, and efficient grips.

2.3. Step 3: Modeling Hand Motion Towards Grasp

Once the contact zones are predicted, the hand needs to be moved from its initial position to one that is more suitable for grasping. This movement must be consistent with the predicted contact zones, be able to occur in real time, and be anatomically correct. When creating trajectories for transitioning into a grasping position, joint constraints, motion continuity, and virtual environment constraints need to be taken into account. A common way to formalize this process is through physical interaction modeling, using simulation to predict how the hand and object will move. Motion paths, for example, are directed to the best contact areas, and 3D point clouds and Gaussian fields are used to estimate proximity [26,27].

Hand movements can be produced using inverse kinematics. In order to adhere to the principles of human biomechanics, anatomical constraints need to be introduced. Another option is to use movement models that have already been trained from real human movements or sensor-based control, which changes the approach in real time based on tactile and visual feedback. Sensor-based approaches change the way the movement works based on visual and touch feedback. This, in turn, improves the stability of the grip and the precision of the interaction. It is also possible to use tactile data analysis to understand how objects will react to movement, and from there, the system can predict the grip [28]. It is also possible to use Convolutional Neural Networks (CNNs) to make decisions about grip selection and movement type. Watkins et al. [29] also use them to represent the surfaces of objects that are hidden or only partially visible in the virtual scene. The goal is to be able to move the hand into the correct position to grasp an object in a way that makes sense physically and under the given circumstances.

2.4. Step 4: Finger Configuration and Initial Grasp

Let us assume that the hand is approaching the object, and during the movement towards it, no choice has been made with the finger configuration to establish a stable initial grip. In order to optimize the stability of the grip and the compatibility of the grasping itself, it is necessary to make a selection of the fingers to grasp and establish joint angles and initial force vectors. One possibility for this is to define the goal of the grasp, the number of fingers, the type of contact with the object, and its characteristics. This will also require sampling strategies and quality metrics to explore appropriate grasping postures. It is possible to use and train neural network architectures with reinforcement and Bayesian optimization to navigate in space with multidimensional finger configurations while taking into account the geometry of the object and the constraints of the virtual environment.

Haptic testing can be used to assess grip stability. It involves a robotic hand interacting with different points on an object. In the case of sensory uncertainty, it is used to determine the safest configurations using Bayesian optimization [30]. Modern systems increasingly integrate visual and tactile information to predict the outcome of a grasp, using local feedback signals and joint attention mechanisms. Reliable representations of the contact interface are created by fusing tactile sensor readings with 3D point cloud data, which allows for more precise and flexible grasping [31]. In addition, some models simulate complex human–object interactions, such as pushing, carrying, and lifting, using multimodal sensors, such as RGB-D. To faithfully represent physical contact and force distribution during manipulation, these simulations use accurate 3D meshes for the hand and the object [32].

2.5. Step 5: Applying Physics-Based Simulation

To verify that the grip is feasible, stable, and physically realistic, it is possible to implement a physics-based simulation. This will allow the system to validate that the proposed hand–object interaction is dynamically plausible under real-world constraints. Depending on the physical properties of the material, the simulation can include three elements. The first is rigid and soft body dynamics to model deformable and articulated components. Then, to perform a contact force and torque simulation to estimate the grip and sliding force. This, in turn, requires a balance check based on the torque balance and force distribution to estimate stability.

Some systems reduce the geometry of objects to primitives and use these abstractions to select the best gripping strategies based on various performance metrics obtained from human demonstrations [33]. To increase the accuracy of physical interaction reconstructions, others use kinematic modeling to estimate object attributes, contact points, and hand joint positions [34]. To obtain high-quality simulation and training data, multi-camera systems are often used to capture full 3D sequences of real-world grasping poses, covering the stages of approach, contact, grip closure, and stabilization [35]. Depending on the task, different simulation platforms can be used, such as Blender or TACTO. They can be used to simulate complex surface interactions, such as transparency or tactile effects [36,37]. Reliable dynamic modeling must be considered when managing interactions, collisions, and hiding of objects in more complex or cluttered virtual environments. To ensure reliability in these circumstances, various techniques have been used, including neural network architectures and volumetric reasoning [38,39].

2.6. Step 6: Grasp Refinement Using Feedback

Due to environmental variability, sensor noise, or uncertainties in object properties, initial grasps are often unstable. To overcome this, the system needs to have a possibility for monitoring and an adaptation mechanism that allows it to optimize the hand configuration in real time. The idea is to ensure stability and task performance in a manner comparable to human behavior in a real environment.

The fine-tuning process can be implemented through various types of feedback, such as haptic information, visual inspection, or trained models. In this context, deep learning approaches such as CNN, Variational Autoencoders (VAEs), and transformer architectures are used. They assist the system in predicting appropriate hand configurations, assessing the quality of the grasp, and adapting to changes in the posture or position of the object [40,41]. Generative models such as Conditional Variational Autoencoders (CVAEs) are used to generate initial grasp candidates, which are then refined using physical reasoning and feedback. Reinforcement learning, especially in tactile robotic grasping, allows the system to develop adaptive policies that maximize stability under uncertainty [37]. Training the model on large datasets of hand–object interactions, often recorded in controlled conditions, such as desktop scenes, also makes it much more realistic [35]. This kind of data supports refinement models that show how people act, such as two-handed and object-aware corrections [42]. Further improvements are achieved through hierarchical policy structures that separate high-level planning (e.g., task goal or object semantics) from low-level control (e.g., joint configurations and contact forces) [43]. These methods allow for task-specific improvements that generalize across a wide range of grasping scenarios.

The output of this step will be used as input to the final stage of the grasping process, which involves contextual adaptation of the grasp based on the task, scene, and semantic information about the object. In practice, this can be achieved through reinforcement learning with task-oriented reward functions or multimodal fusions that combine tactile and visual data. In the context of the Tactile Internet, for real-time application systems, feedback optimization is of particular importance. Ultra-low latency communication allows users to correct unstable grasps in real time in shared virtual or remote environments. This ensures realistic and immediate tactile responses, consistent with the goals of immersive VR, AR, and MR applications.

2.7. Step 7: Contextual Adaptation of the Grasp

Adapting the grasp according to the task goal, the context of the scene, and the semantic information about the object is the final step in the grasping process. For example, a cup may be held differently depending on whether it is placed on a surface filled by the user or handed to someone. The reasoning required for this semantic variation should not go beyond what is physically possible. Contextual reasoning modules contribute to better efficiency by aligning the selected grips with typical human behavior. In this way, the system not only performs the grip physically correctly, but also adapts it to the specific task and context in which it is located. This allows better connection between low-level control, such as positioning and grip strength, and higher-level understanding of the situation and goal. When this type of reasoning is integrated into the grasping process, the system responds more flexibly and makes its actions more meaningful, both functionally and in relation to the intention behind the interaction itself.

The movements of objects affect the overall stability of the scene and model the dependencies of the objects. By combining scene graphs and manipulation relationship graphs (MRGs), context modeling makes this flexibility possible [44]. Another option is to use task-specific constraints and language-based descriptions to help map grasps to desired actions or goals [45]. Depending on the functional roles and capabilities, the type of grasp is decided, depending on the purpose of the object. In addition to taking into account environmental constraints, context-aware approaches allow grasp planning in scenes that are dynamic or cluttered. Such systems reason about the broader implications of each grasp by combining information about the scene layout, object semantics, and current interactions between the hand and the object [46].

The final step in the grasping process is contextual adaptation, which tailors the grasp based on the task goal, the scene, and the semantic information of the object. In practical applications, this can be achieved through manipulation relationship graphs that define permissible interactions, or by using high-level language models to translate task instructions into specific grasp constraints. These strategies transform the conceptual stage into a functional module that effectively links high-level task semantics with low-level grasp planning.

When integrated with the Tactile Internet, semantic adaptation ensures that the system not only generates physically stable grasps, but also dynamically adjusts them to task-specific goals transmitted over multimodal channels in real time. This integration allows virtual and remote interactions to accurately reflect human intent, which is a priority for the next generation of immersive and teleoperation systems.

2.8. Guidelines for Implementing Tactile Internet in Grasping Frameworks

In order to model the grasping process in a virtual environment, it is necessary to take into account the technological requirements to achieve realism and responsiveness. Recent studies have shown that the choice of Tactile Internet solutions must be adapted to meet the requirements of ultra-low latency, reliability, and multimodal feedback [47,48]. Similar to human behavior, where unconscious choices are made about the type of grasp, reinforcement learning techniques have been used in the algorithmic design to enable the acquisition of adaptive grasping policies in a dynamic context [49].

In addition, generative models such as Conditional Variational Autoencoders and diffusion models can generate different grasping variants [49]. This creates a wealth of possibilities that need to be managed effectively. For this reason, data preprocessing is required, including techniques such as normalization, filtering, and the use of compact geometric representations, such as sign distance functions. These methods are essential for noise reduction and lower computational costs [49].

To improve the performance of Tactile Internet systems, it is recommended to use lightweight models (e.g., sieved or quantized Convolutional Neural Networks), which can be combined with additional algorithms. This ensures real-time operation without loss of responsiveness [48]. The balance between latency and computational load can be further achieved through optimization methods such as distributed learning, ADMM-based decomposition, or splitting the model between edge and cloud nodes. Particular attention is paid to the balance between latency, realism, and user experience, especially in medical training and rehabilitation applications [47].

3. Physics-Informed Modeling of Grasping

Besides being used as a simulation tool to validate grasping performance (as discussed in Section 2.5), physics is at the heart of the design, training, and evaluation of grasping models. Direct integration of physical laws, constraints, and material properties into the computational representation of hands, objects, and their interactions is known as physics-informed modeling. This section presents physics as embedded in grasping approaches from precomputed constraints and loss functions to real-time dynamic simulation and differentiable physical models. The goal is to improve grasping realism, physical plausibility, and generalization to complex scenarios, such as manipulating deformable objects or cluttered environments.

To summarize the main approaches to grasp modeling, a comparative analysis of different techniques used in the literature is presented in

Table 1. Comparative analysis of grasp modeling techniques.

Approach	Techniques	Applications
Geometric	Point clouds, 3D Gaussian, SDFs	Prediction of contact points and refinement of hand–object interactions [19,26,27]; applicability to precise positioning and shape adaptation; use of implicit surface modeling [19], contact by Gaussian approximation [27] and diffusion text-driven models [26].
Physical interaction modeling	Optimal control, force/torque equilibrium, elastic energy	Evaluation and stabilization of contact points and grip forces [20,22,24]; dynamic correction via capsular triggers [22]; realistic poses via elastic energy optimization [23]; fine-tuning with control strategies [20].
Tactile and sensor-based	Tactile data, haptic feedback, visual-tactile fusion	Tactile–visual grasp planning [28,29,30]; Bayesian optimization of secure configurations [30]; application to uncertain environments and transparent objects [28,36].
Simulation and kinematic	Rigid-body dynamics, differentiable simulation, 3D pose tracking	Simulation of contact dynamics and sequences from approach to stabilization of the grip [35,50,51]; extraction of detailed kinematic data through simulations and multi-camera configurations [35,50]; application in realistic animation and testing of robotic grippers.
Machine learning	CNNs, transformers, VAEs, RL	Predicting poses and grip quality through deep learning [37,40,52]; improving touch-based grips through reinforcement learning [37]; generative approaches to modeling stable configurations and refining through physical constraints [14,53].
Contextual	MRGs, contextual reasoning, language descriptions	Processing of task-specific constraints and scenes with high object density [44,46]; modeling dependencies between objects via manipulation relationship graphs [44]; contextual modules for task understanding and action prediction [45,46].
Multidimensional	Co-attention, RGB-D and tactile fusion, 3D fitting	Integration of multidimensional data for more accurate modeling [31,32]; combining RGB-D and tactile information in manipulation tasks [32]; co-attention mechanisms for combining visual and tactile signals in grasp prediction [31].
Human-centric learning	Human motion tracking, bimanual grasp, shape decomposition	Bimanual grip prediction [42]; structure extraction from human demonstrations for better grip quality [33]; motion tracking for adaptive learning [35].

. The table provides an overview of the methods used, their specific techniques, and main applications in the simulated environment.

One of the most challenging aspects of accurate and realistic hand–object interaction models is the incorporation of physics into the representation of hands, surfaces, and objects. Physics is incorporated into the methods in a number of ways, ranging from real-time simulation with actual machines to precomputed physical features. To create realistic hand–object interactions, several datasets incorporate physics during preprocessing. To ensure that hands do not cross objects in an unusual way and maintain consistency with physical expectations, for example, methods often use penetration loss and plane constraints [18,54]. By imposing constraints derived from physical principles, techniques such as SDF and loss functions that minimize interpenetration improve realism [55].

To simulate dynamic interactions, physics engines such as Bullet [56], PyBullet [57], Gazebo [58], and MuJoCo [59] are often used. They enable complex and flexible interactions during grasping tasks by simulating forces, torques, collisions, and deformations. The DART physics engine used in the Gazebo environment allows for the simulation of contact forces and object dynamics, making it suitable for modeling realistic robotic grasping [53,60]. Similar goals are achieved using PyBullet, which is used to evaluate grip stability and derive physically consistent configurations [61]. These tools are particularly effective in simulations involving soft materials or deformable objects, where real-time calculations are required [25].

Modeling dynamic interactions relies on fundamental physical principles, such as Newton’s laws, force balance, and collision detection. Calculations of contact forces, torques, and friction coefficients are used to improve grip. In other cases, biomechanical concepts, such as virtual springs or finite element methods (FEMs), are applied to simulate soft tissue deformation. These techniques can be applied to activities that require a high degree of precision. Physics is often incorporated into learning-based models as additional losses or constraints for direct learning. For example, realistic contact dynamics is simulated using repulsive and attractive forces in spring-mass systems [23]. Soft tissue deformation is addressed by differentiable, physics-based contact models, such as DiffContact [52]. The robustness of the trained models is improved by loss functions that penalize interpenetration in relation to physical constraints, such as forced closure [62,63].

When implementing physics in grasp simulation in a virtual environment, the physical properties are pre-computed, which saves computational effort at the expense of simulation flexibility. This approach is effective for real-time manipulation, but by robotic hands, which requires fast inference [64]. Dynamic physical simulation, which calculates forces and interactions in real time, is more precise and adaptive, but also requires more processing resources. The Isaac Gym simulator [63] checks if the grasping movements are correct.

The use of grasping simulations, such as those based on differentiable physical machines [65], requires a lot of computing power to accurately model contact forces, object deformations, and soft tissue interactions. While they provide computational efficiency, a drawback is the lack of adaptability to dynamic scenarios [42]. On the other hand, models that include real-time physics engines [66] need to combine accuracy and efficiency, which further slows down data processing. In addition, multi-view grasping methods [67] have additional computational costs due to the need to integrate multiple viewpoints and, consequently, process the sensor data themselves.

4. Types of Datasets and Their Focus

The way in which different elements and surfaces are represented in a virtual environment should be tied to the way in which grasping is represented as a process, and therefore consider the interaction between people and objects. This includes considering the type of material and its physical characteristics. For the simulation to be accurate, the user’s hand needs to be represented, and the movement during the grasping process must be realistic. The datasets used in grasping simulation models can be broadly divided into real data and synthetic data sets. Some visual examples are given in Figure 4. The comparison of nine datasets used in material synthesis and analysis is given in

Table 2. Comprehensive comparison of distinct datasets used in material synthesis and analysis.

Dataset	Type	Materials Represented	Data Representation	Preprocessing	Text-to-Motion
HO-3D [21]	Real	Ten objects (plastic, organic materials, metal, paper, etc.)	Three-dimensional hand/object poses; segmentation masks	Manual alignment of grasps and masks	No
ObMan [68]	Synthetic	Eight object categories from ShapeNet (bottles, bowls, cans, jars, knives, cellphones, cameras, remote controls)	MANO hand model; object meshes; point clouds	Contact map derivation; SDF computation	No
GRAB [70]	Real	Fifty-one objects (plastic, glass, organic materials, etc.)	SMPL-X models; 3D contact and motion data	Annotation filtering; ground-truth cleanup	Yes
DexYCB [35]	Real	YCB objects (wood, metal, plastic, cardboard, natural materials, etc.)	Six-dimensional hand/object poses; RGB-D images	Downsampling; normalization	No
OakInk [71]	Real	Ceramic, metal, etc.	Point clouds; labeled grasp vectors	Point segmentation	No
ContactPose [69]	Real	Glass, metal, plastic, etc.	Three-dimensional hand/object poses; high-resolution thermal-based contact maps; multi-view RGB-D images	Thermal data preprocessing; object surface extraction	No
MOW [55]	Real	Organic, plastic, metal, etc.	RGB images; 3D reconstructed hand– object models	Object segmentation; 3D mesh selection	No
HUMANISE [45]	Synthetic	Indoor objects; full-body interactions	Three-dimensional human motion sequences; point clouds	Alignment of motion and scenes; text description generation	Yes
Tactile Glove [28]	Real	Ceramic, plastic, metal	Tactile images; 3D positions; velocities	Tactile encoding via CNN; data embedding	No

. The Text-to-Motion column examines ways to perform or modify grasping movements using natural language instructions, such as “grab the jug by the handle.” These methods combine semantic descriptions with movement, allowing the grasping process to be controlled both semantically and geometrically.

4.1. Real-World Datasets

The first group, namely real data sets, is obtained by means of physical sensors, such as RGB-D cameras, LiDAR, or tactile sensors. This, in turn, includes, in addition to the useful signal, noise, and variability in the data, which requires preprocessing. BEHAVE is an example of such a dataset that records hand–object interactions using RGB-D cameras with multiple viewpoints. Contact points, object pose, and hand pose are included in the database [32]. Similarly, GRAB’s dataset uses motion capture (MoCap) to record hand–object interactions. The 3D reconstructions of hand positions and object interactions are the final output [72]. DexYCB’s dataset uses real-world RGB-D sequences and provides annotations of hand objects for comparative analysis [35]. The problem with collecting such datasets is the cost, time, and complexity of the process itself [28,73].

4.2. Synthetic Datasets

The second group of datasets, hereafter called synthetic datasets, is generated using 3D modeling software or simulation environments. This type of data provides significant advantages, including faster data generation, greater adaptability, and improved control over the characteristics of the dataset. For example, the ReplicaGrasp dataset simulates whole-body interactions between a person and an object in various 3D environments [74], while the ObMan dataset offers synthetic hand–object interaction data produced using the MANO model and ShapeNet objects [41]. Text descriptions are included in other datasets, such as TextGraspDiff, to guide the creation of synthetic grasp data using diffusion models [75]. The lack of real-world realism in these data sets is a disadvantage that can affect the generalization of the models trained with them [51,76].

4.3. Data Representation Approach

The way in which the data are represented is directly related to the way in which the grasping process is modeled and represented. This is related not only to the data structure or the quality of visualization, but also to the analysis and interpretation of the hand–object interactions and the grasping process. Commonly used data representations include point clouds, polygon meshes, voxels (see Figure 5) and also Signed Distance Functions (SDFs), and depth images.

Point clouds are used precisely because of their detailed spatial information, which makes them useful for capturing fine geometric features and modeling object surfaces [78]. Point clouds are represented as sets of 3D points in space. To illustrate hand–object interactions with accurate 3D spatial annotations, point clouds are used in datasets such as ObMan and ContactPose [41,52]. In real-time applications such as the REGRAD dataset, where the representation facilitates the creation of grip points and orientations, point clouds are also preferred due to their computational efficiency [44].

Polygon meshes also offer a more precise representation, capturing complex hand and object geometries, precisely because of their representation by vertices, edges, and faces. They are commonly used in data sets that require detailed modeling of hand–object interactions, such as the GRAB and DexYCB datasets, which use meshes to model objects at a fine-grained level [35,72]. DexYCB combines RGB-D images, meshes, and 6D object poses used for model development for grip prediction and posture estimation [35]. The Tactile Glove dataset [28] provides sensory feedback for dynamic grasping tasks by combining tactile data and point clouds. Meshes are particularly suitable for applications that demand realistic 3D reconstructions and interaction modeling. For example, the MOW dataset contains reconstructed hand and object meshes from real-world images [55].

Voxels represent 3D mesh model objects as structured volumetric data, allowing compatibility with Convolutional Neural Networks (CNNs) such as [78]. This structured representation is well-suited for tasks that involve the completion of the shape of the object or the prediction of grasp stability, as seen in IsaacGym-based simulation datasets that use voxel meshes to evaluate multi-finger grasps on deformable objects [51]. Another, also compatible with 2D CNNs, is depth mapping, which represents structured, pixel-based data that capture 3D spatial information. Some datasets, such as BEHAVE, use depth images to supplement RGB data to analyze and model hand–object interactions [32]. Depth images are also used in datasets such as HUMANISE, where they provide additional spatial context for human movement within 3D indoor scenes [45]. Although less sophisticated than point clouds or meshes, voxel-based meshes and depth representations are ideal for their simplicity and memory economy, as well as for fitting with deep neural networks.

Signed Distance Functions provide a continuous representation of three-dimensional shapes by encoding the distance between a point in space and the nearest surface of an object. In data sets such as ShapeNet and the Grasp database, objects are represented by SDF values to aid in the planning of collision-free interactions and predictive comprehension [50,79]. Geometric modeling of object surfaces in simulation environments is well-suited to SDFs.

4.4. Object Material Types

Materials found in databases range from soft and flexible, such as fabric, foam, or rubber, to rigid, such as metal and plastic. Accurate representation of interaction forces, deformation, and friction behavior is essential in simulations of realistic contact dynamics since their variations rely on the material properties.

Almost all datasets contain rigid materials, such as wood, plastic, and metal, which are among the easiest to image. For example, the YCB dataset includes objects made of plastic, metal, and other rigid materials, which allow for precise modeling of shapes with high structural stability and constant geometry [22,67]. Such objects are suitable for improving the accuracy, robustness, and reproducibility of requirement-related tasks.

On the other hand, when modeling interactions with deformable materials such as textiles, foams, or rubber, it is necessary to account for their elastic and plastic properties. For realistic simulation, datasets that contain physical characteristics such as Young’s modulus and loading behavior are used, which allow for an accurate representation of the material response upon contact. Some examples include virtual workspaces where precision during movement is required (e.g., virtual operating rooms). To track the interaction between deformable objects and a virtual hand, Grady et al. [52] have constructed the ContactPose dataset. It contains thermal data and labels related to soft materials, which are part of the deformable object family.

Glass and polished metal are examples of transparent and reflective materials that are somewhat more difficult to model. Datasets such as the Blender-based synthetic transparent object dataset are used to accurately reproduce the optical characteristics of these materials [36]. Algorithms that interpret complex visual features can be trained on real or synthetic data, but optical distortions can also occur.

In the real world, not all objects are made of a single type of material. Implementing these complex objects in the virtual world also requires complex feature extraction. For example, in the OBJECTFOLDER dataset [37], objects made of materials such as ceramic, wood, steel, and polycarbonate are added through visual, auditory, and tactile inputs. Similarly, the BEHAVE dataset represents objects made of fabric and plastic [32]. To simulate interactions with hard furniture and soft household objects, for example, datasets such as ReplicaGrasp include semantic information regarding the materials themselves [74]. For activities such as furniture assembly, packaging, or handling delicate products in virtual environments, this semantic information would enhance the realism of the simulation.

Datasets based on haptic feedback and interaction are already available. These represent material-specific characteristics, including compliance and coefficient of friction. Key features, such as surface texture, flexibility, and friction, are otherwise impossible to identify with visual information alone and are captured from measurements taken using tactile sensors. More exact simulation of grip stability is therefore possible in the future [30]. These datasets are particularly useful for models that aim to distinguish between slippery and stable surfaces. The goal is to improve grip planning and performance encountered in real-world grasping tasks. Methods based primarily on visual information often ignore important factors such as sliding resistance or deformation of the object during grasping. Due to the limitations of current simulation approaches, fragile or highly deformable objects are often absent from the databases. This makes it difficult to compare different models and realistically reproduce complex material properties.

5. Evaluation and Limitations

In the real world, two criteria, objective and subjective, can be distinguished to assess grasping. In the objective criteria, there are two possible outcomes: success and failure. Success is considered to be achieved when the object is in the hand and has been moved from its initial coordinates, with the process being controlled and following the movement and direction of the hand consciously. Failure can be considered to be the inability to grasp the object, improper feeding, dropping it, etc. In the robotic grasping of 3D deformable objects, Huang et al. [25] consider the following performance metrics: pickup success, stress, deformation, strain energy, linear and angular instability, and controllability of deformation. In

Table 3. Evaluation techniques for hand–object interaction.

Evaluation Category	Techniques	Advantages	Limitations
Quantitative metrics	MPJPE [26,80,81], Chamfer distance [17,80,82], F-Score [80,83], intersection volume [50,52], contact ratio [52,75], simulation displacement [18,78]	Objective and precise; widely used in assessing pose and geometry accuracy; ensures quantitative evaluation of joint-level accuracy (MPJPE) and evaluates shape fidelity (Chamfer distance)	Capturing nuances of human perception or complex interactions; does not always correlate with subjective quality; different object geometries comparison.
Physical realism	Penetration depth [18,84], contact consistency [52,75], stability in simulation [18,78,84]	Ensures grasps are physically possible. Penetration depth highlights unrealistic overlaps. Stability in simulation tests interaction feasibility in robotic settings.	Simulations do not fully capture real-world physics. Variability in object surface and friction coefficients may affect accuracy. Computationally intensive for complex models.
Qualitative assessments	Visual inspection [72,85], perceptual studies (e.g., Likert scales, user feedback) [61,70,78]	Captures nuances of human perception; identifies naturalness and aesthetic aspects missed by quantitative metrics. Effective for end-user-focused evaluations.	Subjective and often biased. Requires large-scale user studies for statistical significance. Difficult to replicate consistently.
Comparative analysis	Baseline methods (e.g., PointNetGPD [40], HOMan [86]), ablation studies [55,63]	Provides context by benchmarking in comparison to existing approaches. Highlights the impact of model components.	Dependent on the choice of baselines, not always captured interdependencies between components
Dataset-specific metrics	Diversity metrics [18,26], coverage of real-world interactions [35,36]	Ensures models handle a wide range of interaction scenarios, making them robust for real-world use. Diversity metrics capture variability across grasp styles and motions.	Dependent on dataset quality and diversity. Limited coverage may reduce generalizability. Metrics might overemphasize diversity while neglecting physical feasibility.
Real-world testing	Simulation-based validation [14,19], robotic experiments [30,87]	Tests the model in practical environments, validating its application readiness. Highlights failure modes not evident in simulations.	Expensive and resource-intensive. Real-world conditions like lighting, clutter, or material properties may introduce variability, making results less reproducible.

, we propose six formal evaluation categories for hand–object interaction in a virtual world.

The evaluation of grasping models in shared virtual environments enables realistic human–object interaction. In one of our previous studies [88], we provide guidance for selecting appropriate evaluation methods based on the application context. For comparison of large datasets, quantitative metrics are most appropriate. For physics-based simulations, physical realism tests can be applied. For VR/AR applications, where user perception and immersion are most important, qualitative evaluation is needed. To ensure a fair and balanced evaluation, we suggest that the selection be based on identifying the application scenario, selecting the main dimension to evaluate (e.g., geometric accuracy, physical stability, or user experience), and using at least two complementary metrics.

5.1. Evaluation

Real-world testing, benchmarking, and quantitative and qualitative evaluations are part of the evaluation of generated hand–object interactions. They are intended to assess the practical utility, variety, realism, and accuracy of the models. The idea of quantitative evaluation is to assess the accuracy and realism of the interactions generated using objective measurements. The accuracy of pose and shape reconstructions is often evaluated using metrics such as the mean joint position error (MPJPE) [26,80,81], Chamfer distance (CD) [17,80,82], and F-score [80,83]. While CD quantifies the similarity between the generated and ground-truth point clouds to assess geometric accuracy, MPJPE calculates the Euclidean distance between the predicted and actual joint positions. The assessment of physical plausibility in hand–object interactions is often performed using metrics that measure the degree of penetration between surfaces, such as penetration depth and intersection volume [18,84]. Additional information about the realism of the contact can be extracted from metrics such as contact ratio [52,75] and intersection volume (IV) [50,52], which assess the degree of overlap and precision of the established contact points. It is also possible to use the so-called simulation displacement (SD), which measures the displacement of the object during the simulation [18,78].

The capacity of a model to reproduce the diversity and unpredictability of real-world interactions is assessed using other criteria, such as diversity and coverage. Diversity metrics assess the variation in the movements produced, often examining rotational axes and angles [18,26]. Coverage metrics assess how well the generated movements capture the distribution of hand–object interactions in the real world [35,36]. Combining these criteria allows models to be generalized to increase robustness. In addition to classical quantitative metrics such as intersection over union (IoU) and scene interaction vector (SIV), which are used to assess the shape reconstruction and the accuracy of predicted contacts [29,40,62], indicators based on sensory output are also applied. For example, the frequency of successful capture, which serves to measure the efficiency and quality of the performed manipulation [89,90].

Qualitative methods are often added to quantitative assessments, which provide a more complete picture of the perceived behavior of the system. Experts or end users evaluate naturalness, visual plausibility, and overall sense of realism through visual inspection [72,85]. Additional information on subjective perception is obtained through surveys and Likert-type scales that detect the extent to which interactions are perceived as natural and functional [61,70,78].

Benchmarking, in particular the HOMan [86] and PointNetGPD [40] models, is used to objectively measure the performance of the models. HOMan uses a generative model to predict a human’s ability to grasp an object among many other objects from single RGB images to assess the realism and accuracy of grasping. On the other hand, PointNetGPD evaluates performance by generating spatial grasp predictions from sparse point clouds. The model evaluates the success rate, precision, and efficiency of grasp detection.

Ablation studies are used to assess the contribution of different components of a model [55,63]. The goal is to remove or modify components in the model to examine their impact on overall performance. For example, Cao et al. [55] conducted ablation studies and evaluated the effectiveness of using 2D images and 3D contacts together and separately. Xu et al. [63] used ablation studies to investigate how each part of their proposed approach contributes to the quality of the final grasp and the generalizability.

In addition to the above, tests are also performed in real-world conditions. For example, robotic experiments are being conducted [30,87], where models are evaluated in practical conditions. Performance is measured in the presence of sensor noise, changing object position, and environmental disturbances. For instance, GraspNet-1Billion [87] uses analytical computation to predict grip stability across a wide range of real-world objects, configurations, and locations. Siddiqui et al. [30] examine grip stability prediction using tactile feedback mixed with Bayesian optimization. They analyze the capacity of models to control uncertainty encountered during real-world robotic manipulations.

Simulation-based validation [14,19] creates controlled conditions for testing model performance without real-world limitations. Karunratanakul et al. [19] evaluated their Grasping Field model by comparing synthesized grasps with ground truth data in simulations, assessing the physical plausibility of the generated grasps for various invisible objects. Additional evaluation metrics are diversity [18,75], simulated displacement [78], and penetration depth [18,84]. These measures assess the diversity, stability, and physical plausibility of interactions. Furthermore, the accuracy and reliability of the methods in different contexts are characterized by surface estimation errors, depth estimation accuracy, action recognition speeds, and object position estimation metrics such as average distance (ADD) [21,53,79].

5.2. Limitations

5.2.1. Hardware Limitations

Simplified physical models or malfunctioning sensor data often cause unrealistic movements, resulting in contact points or forces that do not match real-world expectations. Approaches that use objects without distinctive features, such as highly reflective or untextured surfaces, cannot accurately represent interactions [64]. Errors in object recognition and pose estimation are more noticeable in cluttered or poorly lit virtual environments. Due to technological limitations and the difficulty of reproducing the sensation of touch in virtual worlds, the integration of tactile sensors remains a challenge.

The creation of unrealistic movements is one of the most common disadvantages that can compromise the apparent realism of interactions [91]. The scope of these methods is limited because they focus on single-handed grasping of static objects, typically ignoring dynamic interactions or those involving multiple hands [92]. In tasks that require coordinated actions, including grasping with two hands or interacting with moving objects, this can lead to confusion. For example, current models often assume stationary objects and predictable trajectories, making them unsuitable for tasks that involve grasping, passing, or manipulating objects with different movement patterns. The impossibility of models to adapt to small changes in object surfaces, including differences in texture or compliance, and therefore their inability to produce solid and secure grips, is caused by the ineffective integration of tactile feedback [52]. This often results in a grip that is either too weak, causing the object to drop, or too strong, which can cause the object to deform or break.

5.2.2. Data Limitations

Assuming that the data will be used for training of model, then the training data should be of good quality. Neural networks predict unsuccessfully when trained on inaccurate or biased data sets, which limits their application in the real world [93]. Over-fitted models that are unable to handle the diversity of the real world can be obtained due to a lack of data or tuning of model parameters. Current techniques have limited robustness and adaptability, as studies present difficulties in generalizing across different types and sizes of objects [94]. Variations in hand size, posture, and individual grasping approaches, which are often underrepresented in training data, make generalization very difficult.

5.2.3. Methodological Limitations

In addition, some approaches require a lot of computation, which limits their use in real time [60]. This limitation is an important element when methods require processing power. Both the use of neural networks in model building and simulations based on physics are computationally intensive. Touch dynamics and realistic simulation of the deformability of the hand and finger articulation should also be considered when refining the models. Especially when working with fragile or unusually shaped objects, inconsistencies between predicted and actual contact pressures can lead to unstable grasps or sliding [38].

6. Conclusions

In this article, we propose a conceptual framework for modeling human grasping in shared virtual environments, based on an extensive literature review. The method is based on creating realism, feedback, and a sequence of interactions. Future work will focus on implementation, validation, and application in immersive systems such as virtual learning and remote collaboration.

Future research should explore the integration of the proposed grasping framework with innovative technologies that impact next-generation interactive systems. The Tactile Internet and 6G communication networks will work together to create the ultra-reliable, low-latency channels needed for real-time grasping and haptic interaction on a global scale. Finally, embedding the grasping model in digital twin platforms would make a connection between simulated and real-world environments. This would allow humans to test, validate, and co-design grasping strategies over and over again before they are deployed in real-world or robotic systems. For example, it is possible to combine our conceptual framework for grasping in virtual environments with emerging energy-efficient robotic architectures such as neuromorphic computing and liquid flow batteries [95].