Three-Dimensional Shape Estimation of a Soft Finger Considering Contact States

Abstract

To achieve precise in-hand manipulation and feedback control using soft robotic fingers, it is essential to accurately measure their deformable structures. In particular, estimating the three-dimensional shape of a soft finger under contact conditions is a critical challenge, as the deformation state directly affects manipulation reliability. However, nonlinear deformations and occlusions arising from interactions with external objects make the estimation difficult. To address these issues, we propose a soft finger structure that integrates small magnets and magnetic sensors inside the body, enabling the acquisition of rich deformation information in both contact and non-contact states. The design provides a 15-dimensional time-series signal composed of motor angles, motor currents, and magnetic sensor outputs as inputs for shape estimation. Built on the sensing signals, we propose a mode-selection-based learning approach that outputs multiple candidate shapes and selects the correct one. The proposed network predicts the three-dimensional positions of four external markers attached to the finger, which serve as a proxy representation of the finger’s shape. The network is trained in a supervised manner using ground-truth marker positions measured by a motion capture system. The experimental results under both contact and non-contact conditions demonstrate that the proposed method achieves an average estimation error of approximately 4 mm, outperforming conventional one-shot regression models that output coordinates directly. The integration of magnetic sensing is demonstrated to be able to enable accurate recognition of contact states and significantly improve stability in shape estimation.

1. Introduction

Soft robotic fingers can conform to the geometry of grasped objects and enable stable manipulation of a wide variety of items [1]. Previously, several studies have investigated in-hand manipulation using soft fingers. These fingers rotate or translate objects within the grasp [2,3]. Such advanced manipulation requires real-time estimation of the soft finger’s 3D shape and fingertip positions, as the deformation state directly affects the success of manipulation. However, most existing systems do not estimate the soft finger shape explicitly. Instead, they rely on open-loop control, where the inherent compliance of the finger absorbs small errors during motion. While this approach can succeed in simple and predictable environments, it cannot adapt to unexpected contact or environmental changes. Without accurate state estimation, the system cannot detect or recover from failures, limiting the applicability of soft fingers in closed-loop manipulation.

Motivated by these limitations, recent research has increasingly shifted toward developing state estimation techniques for soft fingers. Several state-of-the-art studies have been focused on state estimation for soft fingers. Yet, despite the growing interest and efforts, accurate estimation remains highly challenging due to several reasons. First, the soft fingers have strong nonlinearity, hysteresis, and time-dependent behavior of soft materials. These difficulties are further exacerbated during manipulation, where contact forces and reactive deformation create highly complex and unpredictable shapes. Second, embedding sensors within the finger is also nontrivial; low-quality sensors can compromise the finger’s compliance, leading to a trade-off between sensing capability and softness. External vision-based approaches have been explored as an alternative, but they struggle to handle occlusion and physical contact, especially in cluttered or confined environments. These challenges have hindered the practical realization of soft finger state estimation and thus limited their application in closed-loop robotic manipulation.

In this study, we propose the design of a soft robotic finger embedded with small magnets and magnetic sensors, and develop a learning-based method for estimating its three-dimensional shape using these sensor signals together with motor angle and motor current measurements. The magnetic sensors detect local bending states through variations in magnetic flux density, while being embedded without compromising the flexibility of the soft structure. The proposed shape estimation method employs a two-stage architecture: an MDN (Mixture Density Network) that first generates multiple candidate shapes corresponding to contact and non-contact conditions based on motor angles, followed by a Selector head that determines the correct candidate using encoded time-series information from the magnetic sensors and motor currents via a GRU (Gated Recurrent Unit).

Figure 1 shows an overview of the hardware configuration and the proposed shape estimation system. The magnetic sensors embedded in the soft finger, together with the motor actuation signals, are used as inputs to the neural network, which then outputs the estimated three-dimensional shape of the finger. We conducted a series of experiments to evaluate the performance of the proposed system, and the results show that our method outperforms several baseline approaches. It achieves an average estimation error of ±4 mm and can reliably distinguish between contact and non-contact conditions.

The organization of this paper is as follows. Section 2 reviews related work on state estimation of soft fingers. Section 3 describes the design and sensor configuration of the proposed soft finger. Section 4 discusses the shape estimation approach, including data collection and network model selection. Section 5 presents the experimental evaluation and analysis. Finally, Section 6 concludes the paper and discusses future work.

2. Related Work

We review related work on soft finger state estimation from two perspectives: sensor selection/configuration and shape estimation methods.

2.1. Sensor Selection

Sensors used for estimating the shape of soft fingers can be broadly categorized into two approaches: vision-based methods using external cameras, and proprioceptive methods utilizing embedded sensors. In vision-based approaches, the shape of a soft finger is usually measured using monocular or multiple cameras, and its three-dimensional shape is reconstructed from the image data [4,5]. Such approaches have the advantages of not requiring any sensors to be embedded within the finger, thus preserving the structural flexibility. However, the vision-based approaches suffer from vulnerabilities to ambient lighting changes and constraints on the placement of grasped objects and surrounding equipment due to occlusion concerns.

To address these limitations, a widely explored alternative is to embed sensors within the finger structure. Common internal sensors include strain sensors, optical sensors (e.g., waveguide-based ones), Inertial Measurement Units (IMUs), and magnetic sensors [6,7]. Strain sensors are compact and can be integrated in large numbers. However, they are typically designed to detect deformation in a single direction and thus struggle to capture torsional or complex 3D deformations [8,9,10]. While recent work has explored soft, distributed strain sensors using materials like carbon nanotubes or liquid metals, their fabrication processes are complex, making reproducibility and stability difficult to achieve in practice [11,12]. Optical sensors offer flexibility by using elastomer or silicone-based substrates that do not hinder the finger’s motion. However, due to their widespread distribution, they have limited spatial resolution for detecting localized deformations [13,14,15]. IMUs can estimate relative pose changes via acceleration and angular velocity measurements, which is useful for shape reconstruction [16,17]. However, they are prone to integration drift and, being rigid components, present challenges for full integration into soft structures. In contrast to the above three sensor types, magnetic sensors are compact and lightweight despite being rigid components, making them suitable for embedding in soft structures in large numbers [18,19,20]. The embedded magnets and magnetic sensors allow the detection of local magnetic field changes while maintaining flexibility, enabling high-resolution estimation of 3D bending.

While existing studies using various types of sensors, including magnetic ones, have achieved promising results, most of them focus on estimating soft finger shapes in free space, with only limited work addressing deformations that occur during contact with external objects or under external forces. In contrast to these approaches, this study employs a magnetic sensing configuration that integrates both magnets and magnetic sensors within the soft finger, aiming to achieve contact-aware 3D shape estimation without relying on any external devices.

2.2. Pose Estimation Methods

Previous pose or finger shape estimation methods for soft fingers can also be broadly classified into two categories: (1) Analytical approaches based on mathematical models, and (2) data-driven approaches using machine learning.

Analytical approaches typically rely on mathematical models, such as the Finite Element Method (FEM), to estimate deformations with high accuracy by incorporating material properties and external contacts. For example, Liu et al. [11] and Martin-Barrio et al. [21] demonstrated that FEM-based models can capture complex nonlinear deformations of soft structures. These methods require minimal sensing and perform open-loop forward prediction of deformation states. However, they face significant challenges, such as the need for accurate identification of physical parameters in advance and the high computational cost associated with solving FEM models. These factors make FEM-based approaches difficult to implement in real time. In contrast to analytical methods, the data-driven approaches construct neural networks that learn the mapping between sensor data and soft finger shapes [22,23]. While such methods require a sufficient amount of training data, once trained, they can handle complex shape deformations, including nonlinearities and contact-induced changes. The methods are also efficient for real-time estimation.

Considering these trade-offs, we adopt a data-driven estimation approach to achieve both high-accuracy and real-time shape reconstruction of a soft finger. However, unlike most existing data-driven methods, which primarily address shape estimation in free space and do not explicitly handle deformation changes caused by contact, our method is designed to be contact-aware. Specifically, we propose a neural-network-based architecture that does not output a single coordinate estimate directly; instead, it first generates multiple candidate coordinate predictions and then selects the correct one based on the inferred contact condition. This design enables the model to account for contact-induced deformation and maintain estimation accuracy even during interactions with external objects.

3. Soft Finger with Embedded Magnetic Sensors

3.1. Basic Structure

The soft finger used in this study was designed to enable shape estimation by embedding magnetic sensors and magnets inside its body. An overview of the finger’s appearance is shown in Figure 2. The external shape of the soft finger is rectangular, with a total length of 230.4 mm, a width of 25.5 mm, and a height of 17 mm. The finger bends by winding multiple polyethylene tendons embedded inside the body using motors. Each tendon passes through through-holes in spacer plates fabricated with a 3D printer (PLA). When tension is applied along these paths, the tendons shorten along their respective routes, causing the finger to bend in the corresponding direction. Each spacer plate has a thickness of 3 mm, and the distance between adjacent plates is set to 23.9 mm. The three drive tendons on the ventral side are housed inside silicone tubes with an outer diameter of 2.5 mm and an inner diameter of 2 mm. These tubes pass through the through-holes in the spacer plates and serve to reduce friction between the tendons and the silicone body. On the dorsal side, two tendons are routed, but these are not used for actuation; instead, they are used to fix the soft finger to its base. By independently controlling the three ventral tendons, the soft finger can achieve actuation with three-dimensional degrees of freedom.

The main body of the finger is formed by casting liquid silicone rubber into a mold and curing it into shape. Inside the silicone, spacer plates are embedded at intervals of 23.9 mm to provide mounting positions for the magnetic sensors and to guide the tendon routing. Each spacer plate has a thickness of 3 mm and contains four through-holes for the tendons, as well as three dedicated slots for mounting magnetic sensors. The detailed arrangement of the plates are illustrated in Figure 2c. The structure of a single spacer plate is shown in Figure 2d. A single finger is supported by nine spacer plates. Magnetic sensors are installed on the 2nd, 5th, and 8th plates, which correspond to the finger base, middle, and fingertip positions, respectively, resulting in a total of 3 × 3 = 9 sensor chips. Each magnetic sensor is flanked by two cylindrical permanent magnets on its lateral sides. These magnets have a diameter and height of 2 mm and are positioned 5 mm away from the sensor surface. As the finger bends, the relative distance between the magnets and the sensors changes, leading to variations in the magnetic flux density and consequently in the sensor output. These placements of sensors and magnets are determined by the characteristics of the magnetic field distribution and the operational guidelines provided in the sensor datasheet. The enlarged view in Figure 2b illustrates the detailed arrangement of the magnets surrounding the sensors. We used two types of liquid silicone rubber (Mold Star^® 30 and Mold Star^® 15, Smooth-On, Inc., Macungie, PA, USA) with different elastic properties to make the main finger body. The Mold Star 30 liquid rubber has a Shore hardness of 30A and is used as the structural support for the entire finger. The dark blue parts of Figure 2 are made of it. In contrast, the Mold Star 15 liquid rubber is softer with a Shore hardness of 15A, and is used to form the regions embedding the magnets and magnetic sensors. These regions are highlighted in light green in Figure 2. By selectively combining the two silicone materials with different stiffness levels, the soft finger can provide sufficient global rigidity to maintain its overall shape, while at the same time preserve localized flexibility around the embedded sensing components.

3.2. Fabrication

The soft finger was fabricated using a custom mold printed using a 3D printer (PLA). The fabrication process is illustrated in Figure 3.

In the first step, we fabricated nine spacer plates using a 3D printer (PLA) and installed connector-equipped circuit boards on the 2nd, 5th, and 8th plates, each carrying three magnetic sensor chips. On each plate, the sensor chips were wired in parallel through the connector-mounted boards. Figure 3a shows the front and back views of a spacer plate with the sensor chips mounted. The interior of each plate is partially hollowed out to ensure that the silicone rubber forms a continuous structure throughout the finger after casting. After that, we inserted silicone tubes through the ventral-side holes of all nine spacer plates, and polyethylene tendons (tension cords) through both the tubes and the dorsal-side holes. The assembled structure was then placed into the casting mold. A positioning jig for the magnets and a partition plate were also installed in the mold to separate the two types of silicone materials. Figure 3b shows the mold with the installed spacer plates, positioning jig, and partition plate. Subsequently, we poured the softer silicone, Mold Star^® 15, into the sensor sections and filled half of them, as shown in Figure 3c. The silicone was degassed under vacuum to remove air bubbles and then cured. After curing, we removed the positioning jigs and inserted cylindrical magnets into the slots formed by the jigs, as illustrated in Figure 3d. Once the magnets were in place, we additionally poured Mold Star 15 into the sensing sections of the finger to encapsulate the magnets and complete the sensing regions. After these sections were fully degassed and cured, we removed the partition plates and filled the final cavity with Mold Star^® 30 to form the structural support of the finger, as shown in Figure 3e.

After all these steps, we obtained a composite structure that integrated two silicone materials with different hardness levels to achieve both localized flexibility around the sensing components and global rigidity for the overall finger.

3.3. Magnetic Sensing System

The soft finger used in this study is equipped with an EQ730L linear Hall sensor. This sensor is a uniaxial Hall-effect device that measures the component of magnetic flux density perpendicular to the sensor surface. The arrow in Figure 2a indicates this perpendicular direction. The sensor has a magnetic sensitivity of 130 mV/mT and outputs a voltage linearly proportional to the magnetic flux density. It operates at a supply voltage of 5 V, with the output voltage varying linearly between 0 and 5 V. Moreover, the sensor features a fast response time of 2 µs, enabling it to track dynamic deformations of the soft finger in real time. In addition to its electronic properties, the sensor is highly compact, with external dimensions of 4 mm (width) × 3 mm (height) × 1.4 mm (thickness), allowing it to be embedded into the spacer plates in a fitted configuration. The compactness also allows the sensor to be integrated into the soft finger without compromising flexibility. Another important feature of the sensor is its relatively stable temperature characteristics, as specified by the manufacturer. Under normal indoor operating conditions, the output variation due to temperature changes is limited, and because the soft finger does not contain active heat-generating components, the influence of temperature-induced drift is expected to be minimal.

In our configuration, as shown in Figure 3b,d, each Hall sensor is sandwiched by two cylindrical permanent magnets with identical magnetic orientation. The south poles of the magnets face the front side of the sensor, while the north poles face the back side. By combining magnetic sensors with permanent magnets in this way, the relative distance between them changes as the soft finger bends, leading to variations in the magnetic flux density along the sensor’s detection axis and consequently in the output voltage.

Mathematically, when using cylindrical magnets, the magnetic flux density $B\left(X\right)$ along the central axis can be approximated by the following equation:

$$B\left(X\right)={\displaystyle \frac{Br}{2}}\left({\displaystyle \frac{L+X}{\sqrt{R^2+{(L+X)}^2}}}-{\displaystyle \frac{X}{\sqrt{R^2+X^2}}}\right),$$

(1)

where B is the magnetic flux density (mT), L is the length of the magnet in the axial direction (mm), R is the radius of the magnet (mm), X is the distance from the magnet surface along the axis (mm), and $Br$ is the remanent magnetization (mT). In our design, we used N40 Neodymium magnets. The parameters of the magnet are R = 1, L = 2, and $Br$ = 1250 mT. Based on these parameters, the magnetic flux density $B\left(X\right)$ as a function of distance can be analytically calculated. By multiplying $B\left(X\right)$ by the magnetic sensitivity of the EQ730L sensor (130 mV/mT), we can obtain the theoretical relationship between the distance from the sensor to the magnet and the corresponding output voltage. Furthermore, this output voltage is measured by an micro-controller (Arduino Mega) and discretized into an integer value ranging from 0 to 1023 using a 10-bit Analog-to-Digital Converter (ADC) (The 10-bit resolution was fine enough for the signals, as the noise levels outweighed the quantization effects). Based on these relations and parameters, we can theoretically derive the theoretical curve representing the relationship between the sensor-to-magnet distance and the integer values obtained from the Arduino Mega. Figure 4 illustrates the theoretical curve. Assuming that the bending of the soft finger shortens the distance between the magnetic sensor and the magnet by approximately 2 mm, the curve implies that placing the initial distance at around 5 mm allows for a large output change without reaching the sensor’s upper limit. The sensors are arranged according to this consideration in practice and the virtual output range is approximately 2.5–5.0 V under the configuration.

Note that magnetic interference exists between the sensor–magnet pairs located on the same layer. However, eliminating this interaction was not the objective of our design. Instead, our intent is to obtain sufficiently rich spatial information for deformation discrimination. To this end, we deliberately use three sensor–magnet pairs arranged laterally to provide left–center–right magnetic responses. Any coupling between pairs is implicitly handled by a learning model, which maps the combined sensor outputs to the estimated finger deformation.

4. Learning to Estimate Finger Shapes

Based on the soft finger design and sensing system described in the previous section, we constructed a supervised learning model to estimate the finger’s shape from time-series sensor data. The model takes motor and sensor signals as input and outputs the estimated 3D positions of four markers placed along the finger.

4.1. Training Data Collection

Before detailing the model architecture, we first describe how the training data were collected, as the data format directly determines the model’s input representation. We collected three types of data: (1) the rotation angles of the motor driving the tendons, (2) the corresponding motor currents, and (3) the output voltages from the magnetic sensors. The actual data acquisition setup is shown in Figure 5. We used Dynamixel XM430-W350 motors (ROBOTIS, Seoul, Republic of Korea) to drive the tendons and could thus directly acquire their rotation angles and current values. The setup during finger actuation, as well as the directions in which each motor contributes to bending, is shown in Figure 6. The ground-truth shapes used for training were obtained through motion capture markers, as shown in Figure 7. Four reflective markers were attached along the side of the soft finger, and an additional marker was placed at the base. The three-dimensional positions of these markers were captured using a motion capture system consisting of nine cameras arranged around the finger. The tracked marker positions were used as the ground-truth representation of the finger shape. The measurement accuracy of the used motion capture system is approximately ±0.5 mm.

During data collection, motor terminating angles were randomly generated to actuate the finger and produce sensing information. Especially, to ensure that the finger motion covered the full operational range without damaging the soft finger, we randomly selected the motor terminating angles within the limits defined by the following equations:

$$\left\{\begin{array}{c}{\theta }_{\mathrm{left}},{\theta }_{\mathrm{right}},{\theta }_{\mathrm{center}}<{250}^{\circ }\hfill \\ {\theta }_{\mathrm{left}}+{\theta }_{\mathrm{right}}+{\theta }_{\mathrm{center}}>{250}^{\circ }\hfill \\ {\theta }_{\mathrm{left}}·{\theta }_{\mathrm{right}}=0\hfill \end{array}\right.$$

(2)

Here, ${\theta }_{\mathrm{left}}$, ${\theta }_{\mathrm{right}}$, ${\theta }_{\mathrm{center}}$ represent the terminating angles of the motors responsible for bending the finger leftward, rightward, and forward, respectively.

Once the terminating angles were determined, each motor was controlled to rotate toward its target angle, thereby actuating the soft finger and collecting data. The initial posture of the finger was defined as a critical state between deformation and non-deformation. It was set by starting from a state in which the tendons were fully loosened and then gradually pulling them, stopping at the point where the finger began to move. The motor angles were calibrated to be 0 that the initial posture for each collection routine. The bending motion toward the terminating angles was executed over a duration of 7 s. No periodic recalibration was applied during the execution.

An example of the actually collected data is shown in Figure 8. Note that the sampling rates were approximately 10 Hz for both the motor angles and currents, 10 Hz for the magnetic sensors, and 100 Hz for the motion capture system. Due to this difference in sampling frequencies, we attached timestamps to all data during collection and aligned the data across different frequencies. In particular, we resampled the high-frequency motion capture data by selecting the value closest in time to each low-frequency timestamp to re-align the mis-matched data.

4.2. Network Architecture

We adopted a two-stage neural network model to estimate the 3D shape of the soft finger. The overall architecture of the model is illustrated in Figure 9.

The first stage is an MDN that takes the motor rotation angles (${\theta }_1$ to ${\theta }_3$) as input. It outputs two Gaussian distributions, each representing a predicted 3D position of the four motion-capture markers attached to the fingertip (i.e., 12-dimensional vectors). One distribution is assumed to correspond to the non-contact state, and the other to the contact state. By using the negative log-likelihood as the loss function, the model avoids collapsing to a compromise between the two distributions and instead learns the prediction distinctly.

The second stage is a selection mechanism that takes as input the two predicted means (${\mu }_1$, ${\mu }_2$) from the MDN, along with a 12-dimensional signal consisting of magnetic sensor outputs ($m_1$ to $m_9$) and motor currents ($I_1$ to $I_3$) in time-series. The time-series signals consists of 16 consecutive frames and are fed into a GRU to extract temporal features (In trial experiments conducted during the development, we compared different sequence lengths (8, 16, and 32 frames). The results indicated no significant performance differences and the GRU model is not highly sensitive to length. The final choice of 16 frames was primarily guided by practical considerations of the current system setup. With a sensing and inference rate of approximately 20 Hz, a 16-frame sequence corresponds to roughly one second of temporal information). The GRU output is then concatenated with ${\mu }_1$ and ${\mu }_2$ and passed to a classifier that determines which prediction is more accurate. Cross-entropy is used as the loss function, and the ground-truth label is automatically assigned based on which of the two predicted means is closer to the true marker positions. To account for data imbalance and ambiguity in contact classification, the loss is weighted using the inverse of class frequencies. Furthermore, when the difference between ${\mu }_1$ and ${\mu }_2$ is small, the loss is down-weighted to emphasize clear distinctions between contact and non-contact states (At the early stage of motion, even when contact exists, the observed behavior is often very similar to the non-contact case. This leads the model to favor non-contact predictions, which artificially increases the overall accuracy due to class imbalance. To solve this problem, we intentionally down-weight the loss in the region where ${\mu }_1$ and ${\mu }_2$ are close, preventing the model from being overly penalized in such ambiguous states and reducing the bias toward non-contact responses).

The model was implemented in Python using PyTorch, and training was performed with the Adam optimizer. The dataset was randomly split into 80% for training and 20% for testing. Learning rate scheduling was handled by the ReduceLROnPlateau method, with an initial learning rate of 0.001, a reduction factor of 0.5, and a patience of 10 epochs. Early stopping was triggered if the test loss did not improve for 30 consecutive epochs. Input data were normalized separately for motor signals and sensor signals.

The estimation accuracy of the model can be evaluated by computing the Euclidean distance between the predicted and measured positions for each of the four markers, and then averaging the errors over all test samples. The evaluation metric is mathematically

$$E_j={\displaystyle \frac{1}{N}}\sum _{n=1}^N\left|{\widehat{\mathit{p}}}_{n,j}-{\mathit{p}}_{n,j}\right|,j=1,2,3,4,$$

(3)

where ${\widehat{\mathit{p}}}_{n,j}$ denotes the estimated position of marker j for the n-th sample, ${\mathit{p}}_{n,j}$ represents the ground-truth position obtained from the motion capture system, and N is the number of test samples.

5. Experiments and Analysis

To evaluate the effectiveness of the proposed method, we constructed datasets under two conditions: with and without contact between the soft finger and an object. The contact condition was reproduced by fixing a 3D-printed rectangular block in front of the soft finger. The block was placed at a position 20 mm forward and 135 mm upward from the fingertip of the soft finger, as shown in Figure 7. In the non-contact condition, experiments were conducted without the block. For each condition, we collected 1500 bending motions as training data and recorded an additional ten trials as test data. All data, including motor angles, motor currents, magnetic sensor outputs, and marker coordinates, were synchronized and acquired using the method described in the previous section.

5.1. Comparison by Data Modality

First, we analyzed the impact of different input data modalities on the shape estimation accuracy to understand which sensor information contributes most to the estimation. We used the same MDN model structure across all conditions and compared performance by varying the modalities of input data fed into the Selector. The model that incorporated motor current values and magnetic sensor outputs in addition to motor angles was treated as the base line (Full Modality). It was compared with the following two variants where each included only one of the additional modalities: (1) Current Only, which uses only motor current values together with motor angles; (2) Magnetic Sensor Only, which uses only magnetic sensor outputs together with motor angles. The configurations of these modality types are summarized in

Table 1. Specifications of different data modality types used for comparison experiments.

Modality Type	Input 1 Size	Input 2 Size	Angle	Current	Magnetic
Full Modality (Baseline)	3	12	◯	◯	◯
Current Only	3	3	◯	◯	×
Magnetic Sensor Only	3	9	◯	×	◯

^Note ◯ indicates that the data modality is included, whereas × indicates that it is excluded.

.

The comparison results using these varying configurations are summarized in

Table 2. Mean and deviation of estimation errors for contact and non-contact conditions.

		Marker1		Marker2		Marker3		Marker4
Modality Type	Contact State	Mean	Std.	Mean	Std.	Mean	Std.	Mean	Std.
Full Modality (Baseline)	Non-contact	0.61	0.66	1.33	2.11	2.19	3.14	3.07	4.45
	Contact	1.11	1.15	2.86	3.28	4.38	4.94	5.61	5.99
Current Only	Non-contact	2.17	3.26	6.09	9.86	9.78	15.54	12.68	19.34
	Contact	1.20	1.59	3.12	4.61	4.80	7.11	6.20	8.84
Magnetic Sensor Only	Non-contact	0.63	0.69	1.40	2.19	2.31	3.55	3.23	4.63
	Contact	0.94	0.91	2.48	2.64	3.85	4.00	4.89	4.58

for the mean errors and their standard deviations. We can see from the results that the model that relied solely on magnetic sensor data (Magnetic Sensor Only) slightly outperformed the Full Modality model. This is different from our expectation that the motor current would correlate with wire tension and provide useful information for contact detection. The reason was that even when the motor produced similar pulling forces, the measured current differed significantly between motor winding (pulling) and unwinding (releasing) motions, making it less informative for binary contact detection. In contrast to the motor current, the magnetic sensor directly captures local deformation of the soft finger and is more tightly coupled to physical contact events.

Meanwhile, we can also see that the Full Modality model had slightly better performance than the Magnetic Sensor Only one for the non-contact condition, and exhibited slightly larger errors for the contact condition. The Current Only model produced substantially larger errors in both contact and non-contact cases, indicating that motor current signals alone are insufficient for accurately inferring the deformation state of the soft finger. Overall, these results demonstrate that, within the proposed network structure, magnetic sensor data serves as the primary contributing modality. Even when used in isolation, it enables high-accuracy shape estimation.

The inference time measured over 6690 frames was 4.461 s, corresponding to 0.00067 s per data frame. Therefore, the estimator can meet real-time requirements for online servoing.

5.2. Comparison with Other Network Architectures

In this section, we investigate how differences in model architecture affect the accuracy of shape estimation. For comparison, we evaluated two alternative models: (1) a ResNet + GRU model, which is a time-series regression architecture commonly used for soft robot shape estimation [24], and (2) a Oneshot model, which is a simplified version of our proposed method with the mode-selection mechanism replaced by hidden states (The diffusion model method [25] was also investigated but were not included as baselines, as their performance degraded in our dataset with frequent contact/non-contact transitions, likely due to strong discontinuities and abrupt temporal distribution shifts). The detailed structure of the two models are illustrated in Figure 10 for comparison.

Here, the ResNet + GRU model serves as a baseline that directly regresses marker coordinates using a standard CNN + RNN structure. The Oneshot model shares the same layer structure as the proposed one but omits explicit selection, making it an ablation comparison target. A summary of the training settings and model architectures used for comparison are presented in

Table 3. Summary of each model used in the comparison experiments.

	MDN + Selector	Oneshot	ResNet + GRU
Rotation Input	1 frame	1 frame	16 frames
Magnetic Sensor Input	16 frames	16 frames	16 frames
Explicit Selection	Yes	No	No
Loss Function	MDN negative log-likelihood + cross_entropy	Gaussian negative log-likelihood	MSE
Purpose	Proposed	Ablation	Baseline

. Note that all candidates ignored current data since the current was found less significant in the first experiment.

The comparison results are shown in

Table 4. Comparison of mean and standard deviation of estimation errors under contact and non-contact conditions.

		Marker1		Marker2		Marker3		Marker4
Input Type	Contact State	Mean	Std.	Mean	Std.	Mean	Std.	Mean	Std.
MDN + Selector	Non-contact	0.61	0.66	1.33	2.11	2.19	3.14	3.07	4.45
	Contact	1.11	1.15	2.86	3.28	4.38	4.94	5.61	5.99
Oneshot	Non-contact	0.48	0.45	0.93	1.31	1.50	2.11	2.25	2.79
	Contact	1.61	1.34	4.24	4.19	6.45	6.63	8.21	8.50
Magnetic Sensor Only	Non-contact	0.61	0.59	1.76	1.69	2.99	2.73	4.33	3.55
	Contact	0.68	0.56	1.76	1.62	2.72	2.53	4.10	3.25

for the mean and standard deviation of errors. The Oneshot model exhibited explicit negative performance in the results. Meanwhile, the MDN + Selector and ResNet + GRU models are difficult to discern from these bare error values. Figure 11 further shows the ground-truth and predicted trajectories of the fingertip markers for each model. The blue and green lines indicate the ground-truth positions under contact and non-contact conditions, respectively, while the red and yellow lines show the corresponding predicted positions. From the results we can see that although the ResNet + GRU model achieved overall comparable error levels to the proposed one, its output trajectories exhibited high-frequency noise. This is likely because ResNet + GRU cannot model the multi-modality of the coordinate distribution, resulting in predictions that tend to converge toward intermediate positions between multiple plausible states. GRU has advantages in capturing temporal dependencies in the sensor signals, allowing the network to capture short-term dynamics rather than instantaneous measurements. To assess the practical contribution of the GRU, we conducted comparative experiments by removing the GRU module and using only the ResNet. The results show that there is little difference in average error. However, in terms of the top 1% error, the model with GRU achieves an improvement of approximately 5 mm. This observation indicates that the role of the GRU is not to explicitly select modes through interpretable attention or weights, but to provide temporal smoothing and robustness in challenging or ambiguous situations. We can also see from the results that the Oneshot model produced smoother outputs, but misclassifications were observed near the start and end of contact events, with the model mistakenly identifying non-contact states as contact. This implies that the Oneshot model implicitly attempted to perform mode selection based on contact state, but the lack of an explicit mechanism led to erroneous predictions. Overall, the proposed model maintained both smooth output trajectories and high estimation accuracy, demonstrating its robustness and effectiveness across both contact and non-contact conditions.

6. Conclusions and Future Work

In this study, we proposed a 3D shape estimation method that accounts for contact conditions by using a soft finger with internally integrated magnets and magnetic sensors. We constructed an architecture that estimates marker positions from sensor information by combining MDN and GRU, and employing a Selector head to determine the final results. The design was based on our observation that MDNs independently estimate single-contact-state shapes well, and contact and non-contact states are easily identified by time-lapse data. The experimental results demonstrated that the proposed method can estimate the fingertip position with an average error of approximately 4 mm, even when trained and tested on datasets containing both contact and non-contact conditions. In particular, it was confirmed that the presence or absence of magnetic sensors has a significant impact on estimation accuracy, highlighting their effectiveness in detecting contact states. Moreover, in terms of model structure, we found that generating multiple candidate coordinates and selecting the appropriate one has higher accuracy compared to directly outputting a single coordinate.

However, the proposed method is currently limited to predicting the presence or absence of contact between the soft finger and the environment. It does not explicitly distinguish between different contact conditions and forces. With the current sensor configuration and network architecture, representing and disentangling such fine-grained contact conditions remains challenging. In future work, we plan to jointly refine the sensor design and the network architecture to differentiate between richer contact representations.