Flex Sensor Compensator via Hammerstein–Wiener Modeling Approach for Improved Dynamic Goniometry and Constrained Control of a Bionic Hand

In this paper, a new control-centric approach is introduced to model the characteristics of flex sensors on a goniometric glove, which is designed to capture the user hand gesture that can be used to wirelessly control a bionic hand. The main technique employs the inverse dynamic model strategy along with a black-box identification for the compensator design, which is aimed to provide an approximate linear mapping between the raw sensor output and the dynamic finger goniometry. To smoothly recover the goniometry on the bionic hand’s side during the wireless transmission, the compensator is restructured into a Hammerstein–Wiener model, which consists of a linear dynamic system and two static nonlinearities. A series of real-time experiments involving several hand gestures have been conducted to analyze the performance of the proposed method. The associated temporal and spatial gesture data from both the glove and the bionic hand are recorded, and the performance is evaluated in terms of the integral of absolute error between the glove’s and the bionic hand’s dynamic goniometry. The proposed method is also compared with the raw sensor data, which has been preliminarily calibrated with the finger goniometry, and the Wiener model, which is based on the initial inverse dynamic design strategy. Experimental results with several trials for each gesture show that a great improvement is obtained via the Hammerstein–Wiener compensator approach where the resulting average errors are significantly smaller than the other two methods. This concludes that the proposed strategy can remarkably improve the dynamic goniometry of the glove, and thus provides a smooth human–robot collaboration with the bionic hand.


Introduction
Hand gesture recognition refers to the process of understanding the mathematical interpretation of the hand's movement by a computing device [1]. It is one of the popular research topics in the past few decades due to the rapid advancements in sensor and smart device technologies [2][3][4]. Its applications are not just limited to human-computer or human-machine interaction [5], but also include a diverse range of fields such as sign language recognition [6], clinical rehabilitations [7,8], human-robot collaborations [9,10], gaming and virtual reality control [11]. In a typical hand gesture recognition system, the initial stage is the data acquisition which can be performed either via vision-based [12] or non-vision-based [13] techniques. Both have their own advantages and disadvantages regarding their applications. A hybrid approach which combines the two methods has also been employed in some specific areas that require high speed and precision such as augmented reality technologies [14].
The vision-based technique, as the name suggests, uses cameras or optical sensors to capture hand gestures. A notable superiority of this technique is that it eliminates the need for using wearable of training and verification, and is useful to detect nonlinear relationships between variables, a large training data set is required and many free parameters need to be optimized in order to obtain a model with great accuracy [39]. In some cases where a high sample rate in data acquisition is used, or the sensor output is frequently perturbed by the Gaussian noise, Kalman filtering approach will be better suited, provided that the sensor bending angle has been well calibrated with the static goniometry [26]. Another recent technique via first principle modeling is proposed in [27] where the mathematical representation that relates the flex sensor's resistance and the bending angle is derived based on the understanding of the system's underlying physics. The advantage of this approach is the detailed insight into the behavior of the system and hence leads to a better prediction on the performance, whereas the disadvantage is the difficulties in determining the phenomenological parameters caused by internal and external disturbances.
It is also worth to note that even though similar types of sensors are used in a glove-based gestural system, a straightforward comparison between the methods that have been introduced over the years may not be appropriate due to different application-specific tasks and design requirements particularly those concerning the types and speed of gestures, sampling time, sensor locations, as well as data mapping [13]. In this work, a new control-centric approach is introduced to model the characteristic of flex sensors on a goniometric glove, which is designed to capture the user hand gesture that can be used to wirelessly control a bionic hand. The main technique employs the inverse dynamic model strategy along with a black-box identification for the compensator design, which is aimed to provide an approximate linear mapping between the raw sensor output and the dynamic finger goniometry. To smoothly recover the goniometry on the bionic hand's side during the wireless transmission, the linearity of the mapping needs to be improved. Hence, we propose a Hammerstein-Wiener model to enhance the structure of the compensator, which consists of a linear dynamic system and two static nonlinearities. The linear system is constructed by simplifying the dynamic model from the inverse dynamic design technique, while the static nonlinearities are introduced based on the constraints of the bionic hand, and to account for the uncertain behavior of the sensors as well as the unmodeled dynamics from the black-box identification method. A series of real-time experiments involving several hand gestures have been conducted to analyze the performance of the proposed method.
In the experiments, the goniometric speed for each finger is controlled at approximately 83 • /s for all gestures. The associated temporal and spatial data from both the glove and the bionic hand are recorded via an image processing technique, and the performance is evaluated in terms of the integral of absolute error between the glove's and the bionic hand's dynamic goniometry. The proposed method is also compared with the raw sensor data, which has been preliminarily calibrated with the finger goniometry, and the Wiener model, which is based on the initial inverse dynamic design strategy. Experimental results with several trials for each gesture show that the raw sensor data result in average errors between 515 • s and 1347 • s, whereas for the Wiener model, the average errors lie between 186 • s and 370 • s, which are well below the range from the raw data. A significant improvement is obtained via the Hammerstein-Wiener compensator where the resulting average errors are no greater than 102 • s. This concludes that the proposed strategy can remarkably improve the dynamic goniometry of the glove, and thus, provides a smooth human-robot collaboration with the bionic hand.
The remainder of the paper proceeds as follows: Section 2 discusses the background and statement of the problem concerning the nonlinear characteristics of the flex sensor from a preliminary analysis. The bionic hand description, the goniometric glove structure, the proposed compensator design method, and the experimental setup are explained in detail in Section 3. Section 4 presents the experimental results from several hand gesture tests, and the average error for each method. The results are finally concluded and discussed in Section 5, together with future work.

Background and Problem Statement
A flex sensor is basically a variable resistor that reacts to bends, i.e., it changes its resistance when the bending angle increases. The flex sensor considered in this work is of unidirectional type as shown in Figure 1. When in default position (i.e., flat/relaxed), the resistance measures around 25 kΩ, and may increase up to 125 kΩ as it bends towards 180 • . This is illustrated as in Figure 2 where θ and R 1 denote the bending angle and resistance, respectively.  The sensor can be configured to act as a voltage divider where the corresponding output, V out is simply: Theoretically, the value of R 1 (in kΩ) relates to the bending angle as follows: When the value of R 2 is fixed, we have the relation which implies a linear relationship between θ and V out . Nevertheless, when the sensor is attached to a moving finger, the exact value of θ will not be smoothly recovered due to the non-smooth finger's movement. Also, the position of the sensor with respect to the finger may additionally affect the resistance, leading to unpredictable behavior. A preliminary analysis has been conducted to investigate the correlation between the sensor output voltage and the bending angle when the sensor is tied on a cloth glove as shown in Figure 3. Results from four tests when R 2 = 35 kΩ and V in = 5 V have been recorded in Figure 4, which are also compared with the theoretical values from Equation (3). From the figure, the inconsistencies of the sensor output and the mismatch with the theoretical values reflect the existence of nonlinearities and uncertainties in the sensor model itself.    The focus of this research is to propose a compensator for the goniometric glove with the aforementioned flex sensors which can dynamically recover the gesture of each finger. To wirelessly control a bionic hand using the recovered gesture, the glove has also been preliminarily designed by taking into account the constraints of the bionic hand. The main strategy to achieve this objective is explained in detail in the next section.

Bionic Hand Description
Throughout this paper, the index i = 1, 2, 3, 4 and 5 will represent signals associated with the thumb, pointer, middle, ring, and pinky fingers, respectively. The bionic hand system used in this work is controlled by five dc motors where the input,β = [β 1 , β 2 , β 3 , β 4 , β 5 ] T , is supplied by the signals from an ATMega microcontroller (denoted as µC1). The overall closed-loop system can be illustrated as in Figure 5 where C(s) and H(s) denote the proportional-integral-based motor controller and the bionic hand, respectively. Each motor is assigned to control the flexion or extension of each finger from the metacarpophalangeal (MCP) joint, and the system is subject to possible bounded disturbances, d in . In this work, the effects of the disturbance are assumed only in terms of slow and slightly nonlinear movements due to deadzones or frictions, and do not lead to instability of the system. The bionic hand has also been designed to mimic the normal behavior of finger movements which can be mathematically described by where θ D i , θ P i and θ i are the goniometry measured at distal interphalangeal (DIP), proximal interphalangeal (PIP) and MCP joints respectively (see the corresponding counterparts in Figure 6). Equations (4) and (5) imply that the fingers' bending angles from DIP and PIP joints are often influenced by the movement from the MCP joints [26]. For i = 2, 3, 4 and 5, the goniometry share the same reference line as illustrated by the pointer finger in Figure 6, while for i = 1, the reference line is −90 • below that of those for i = 2, 3, 4, 5 (shown in Figure 7), and only MCP and interphalangeal (IP) joints exist.
The overall movement is controlled by the motors attached at the MCP joints, which automatically changes θ D i and θ P i when θ i is changed. Ideally,θ = [θ 1 , θ 2 , θ 3 , θ 4 , θ 5 ] T should follow the reference command,β, but it may not always be the case due to the presence of d in which can randomly enter the system at any time instance. Apart from that, a constraint, Ψ = diag(ψ 1 , ψ 2 , ψ 3 , ψ 4 , ψ 5 ), is imposed onθ to resemble the typical range of joint motions, which is described as where θ L and θ U denote the lower and upper bounds, respectively. For i = 1 (i.e., thumb), the movement is limited by θ L = 90 • and θ U = 170 • , whereas for i = 2, 3, 4, 5, the bounds are θ L = 28 • and θ U = 113 • .

Goniometric Glove with Compensators
In this research, the goniometric glove is made of cotton with a thickness of around 2 mm. Preliminary analyses with a grab and release movement have been conducted to investigate suitable positions of the sensors on the goniometric glove. Letρ r = [ρ r1 , ρ r2 , ρ r3 , ρ r4 , ρ r5 ] T be the angles measured at the MCP joints of the goniometric glove. An image processing technique in MATLAB is used to captureρ r . Figure 8 (left) shows suitable positions of the sensors with respect to the MCP and PIP joints which can register an acceptable and predictable goniometry for each finger. The outputs of the sensors are connected to analog pins of an ATMega microcontroller (denoted as µC2) with a sample rate of 10 Hz. We denote the raw sensor values read by µC2 asα = [α 1 , α 2 , α 3 , α 4 , α 5 ] T , where each α i ranges from 0 to 1023. The goniometric glove response in one of the tests is shown in Figure 9.
From Figure 9, it can be observed that when the sensors are positioned as depicted in Figure 8, the response does not deviate far from the fingers' goniometry. On the other hand, it also suggests that the bionic hand requires a good precompensator to minimize the error between the goniometry and the sensors' response.  To this end, we propose a dynamic compensator, P(s), which is constructed based on the inverse dynamic approach where the structure is designed using the inverse of the internal model that characterizes the behavior ofρ r andα. The black-box system identification technique is used to estimate the dynamics of the model where the input is fed from the value ofα while the output is the value fromρ r . The highest accuracy from several datasets from the black-box system identification is approximately 70%, and the model with the highest accuracy is given by a linear time-invariant (LTI) model, and D p =diag (0, 0, 0, 0, 0).
As the output of the bionic hand system is constrained, a static nonlinearity, is included in the compensator to ensure the input to the bionic hand system stays within the range. The nonlinearity is described as where σ wl = 90 • and σ wu = 170 • when i = 1, and σ wl = 29 • and σ wu = 112 • when i = 2, 3, 4, 5. The combination of P(s) and Φ w in series forms a Wiener-type compensator, which is illustrated as in Figure 10. The accuracy of 70% from the black-box system identification actually reflects the effects of nonlinearities in the model. To further enhance the tracking performance of the compensator, these effects need to be suppressed. Based on the variations of resistance in the preliminary analysis as shown in Figure 4, we propose a slight modification on the compensator where P is partitioned into two blocks as depicted in Figure 11, consisting of a simplified LTI model, P hw , and another static nonlinearity, The P hw is constructed based on the estimated dominant pole from P, which results in only first order linear model for each finger. The new configuration of the compensator is well-known as the Hammerstein-Wiener structure which, in general, is an LTI system in series with two static nonlinearities. The simplified dynamic block of the compensator, P hw ∼ (A ph , B ph , C ph , D ph ), is constructed as follows: and D ph =diag (0, 0, 0, 0, 0), whereas for the static nonlinearity, it can be described as where ε l = 10 and ε u = 20 when i = 1, 2, 3, 4, and ε l = 10 and ε u = 30 when i = 5.

Experimental Setup
For wireless communication between the goniometric glove and the bionic hand, an HC-12 serial communication module is connected to µC1 as a receiver, and another similar module is connected to µC2 and configured as a transmitter. Six different sets of gestures are considered for the experiments as follows: These are illustrated as in Figure 12. The first four gestures involve at most three hand movement transitions, while the last two gestures involve six movement transitions. The experimental setup is depicted in Figure 13 where the performance of the overall system is evaluated based on the temporal mismatch between the glove's and the bionic hand's goniometry which are registered through cameras connected to a PC desktop via USB cables. The values ofρ r andρ are extracted via image processing techniques in MATLAB.

Experimental Results and Performance Evaluations
In the experiments, the bionic hand response, i.e.,ρ, is compared when there is no compensator at all, and when Wiener and Hammerstein-Wiener compensators are implemented, which are denoted by "Raw", "W", and "HW" respectively in all figures. The response is also compared with the reference, ρ r , which is from the glove goniometry. The experiments conducted focus on temporal analysis, and the goniometric speed for each finger is controlled at approximately 83 • /s. A series of postures from the bionic hand are also recorded for a simple spatial analysis.
For the first experiment, i.e., Gesture 1, the hand gesture starts from the grab position, and all fingers slowly stretch at t ≈ 4 s, remain at this position between t ≈ 4.5 s and t ≈ 5.5 s, and finally return to the grab position at t ≥ 6 s. The values of ρ i for i = 1, 2, 3, 4 and 5 are plotted in Figure 14, and it can be clearly observed that without any compensator, the bionic hand fingers are slightly moving when there is no movement from the glove. The movement becomes worse for certain fingers as shown by the large fluctuations of ρ 2 , ρ 3 and ρ 5 between t = 4 s and t = 6 s. Also, ρ 1 shows an unexpected behavior after t = 6 s when the thumb is supposed to bend. These undesired responses can however be alleviated using both Compensators W and HW. It is also clear that the best response is obtained when the goniometric glove is controlled by Compensator HW, particularly during the "grab" instances. To show the error response, we define e i = ρ ri − ρ i which represents the mismatch between the glove's and the bionic hand's goniometry. The corresponding e i for Gesture 1 experiment is presented in Figure 15.
For Gesture 2 experiment, which is showing the number two sign, the values of ρ i for i = 1, 2, 3, 4 and 5 are plotted in Figure 16. The hand gesture starts when all fingers are vertically stretched, and the thumb, middle and pinky fingers slowly flex between t ≈ 2.6 s and t ≈ 4 s. In this experiment, ρ r2 and ρ r3 are not supposed to vary too much, but large fluctuations in ρ 2 and ρ 3 can be seen when there is no compensator applied. A quite similar behavior is also observed after t = 4 s for ρ 4 and ρ 5 , resulting in a large error. In this case, Compensator HW provides a significant improvement as compared to Compensator W due to the erratic readings as seen in ρ 1 , ρ 4 and ρ 5 before t = 4 s. The error can also be clearly seen from the plot of the corresponding e i in Figure 17.  Figure 14. Gesture 1: The mismatch between the glove's and bionic hand's goniometry is significantly reduced by using Compensators W and HW. Compensator HW is seen to provide a better response as compared to Compensator W, particularly before t = 4 s and after t = 6 s.   Figure 16. Gesture 2: The mismatch between the glove's and bionic hand's goniometry is significantly reduced by using Compensator HW, which also provides a better response as compared to Compensator W, particularly before t = 4 s. The bionic hand response for Gesture 3 experiment is shown in Figure 18 where the gesture starts when all fingers are vertically stretched. The pointer, middle and ring fingers start to flex at t ≈ 3.5 s, and the hand stays in the "call sign" gesture after t ≈ 4 s. The figure excludes ρ 1 as the thumb stays stationary in this gesture, and all responses from "Raw", "W" and "HW" are very close to ρ r1 . Similar to the behavior seen from Gesture 2 experiment, Compensator HW outperforms the rest as the fluctuations and the error are minimized as can be observed from ρ i (i = 2, 3, 4 and 5), as shown in Figure 19.  For Gesture 4, the response is shown in Figure 20 where the gesture starts when all fingers are vertically stretched. The thumb and pointer fingers start to flex at t ≈ 3.5 s, and the hand completely forms the "okay sign" after t ≈ 4 s. In this gesture, the middle, ring, and pinky fingers almost stay stationary and ρ 3 , ρ 4 , ρ 5 for "Raw", "W" and "HW" do not show significant deviations from ρ r . Hence, only ρ 1 and ρ 2 are highlighted in the left column of the figure, together with the corresponding error in the right column. The response shows very large fluctuations when the goniometric glove is not compensated, and the undesired behavior is significantly suppressed by using Compensators W and HW. The experiments with Gesture 5 and Gesture 6 are slightly different than the previous four gestures as they are designed to analyze the robustness of the proposed strategy when a rapid hand movement is involved. For the Gesture 5 experiment, the hand starts when all fingers are vertically stretched, and then one thumb bends towards the palm, followed by the rest after approximately 0.8 s. The transition proceeds with the pointer until pinky fingers stretch back, close, and stretch again within approximately 2 s. The experiment ends when the thumb is released to its initial position.
As can be observed from Figure 21, by using Compensator HW, the mismatch between the glove's and bionic hand's goniometry is drastically minimized as compared to that when the raw sensor data or Compensator W are implemented. The is also clearly seen in Figure 22 where the resulting error from Compensator HW implementation does not deviate too much from the zero value.  For the last gesture, which is Gesture 6, the hand starts when all fingers close, but the pointer is stretched away from, and the middle is slightly bent towards the palm. Then the pointer and the middle fingers exchange their positions, followed by all fingers close. The pointer until pinky fingers stretch back and close again within 1 second, and the transition ends when all fingers are released.
The responses are recorded in Figure 23, and a similar outcome can still be seen from this last experiment where the implementation of Compensator HW provides the least mismatch between the goniometric glove and the bionic hand. The corresponding error response is shown in Figure 24.  Some images taken from the camera during the performance evaluations are shown in Figure 25. Each of them illustrates the final position of each finger for each gesture (i.e., grab, number two, call, and okay signs).  As the closed-loop bionic hand system is susceptible to unknown disturbances, the experiments for Gesture 1 until Gesture 6 are repeated for three times to provide a better evaluation, and the performance is quantified in terms of the integral of absolute error, E ( • s), as follows: where t f denotes the final time of execution. The total error from each finger, which is calculated as for all trials and gestures are recorded in Table 1 when there is no compensation at all, and Table 2 when Compensators W and HW are applied. Table 1 shows average errors between 515 • s and 1347 • s for all gestures, which are much bigger than those from Table 2. Interestingly, Compensator HW shows average errors of less than 102 • s, while the average errors when Compensator W is applied vary between 186 • s and 370 • s. This signifies that Compensator HW can provide a major improvement over Compensator W in terms of the temporal mismatch between the goniometric glove's and the constrained bionic hand's movements.

Discussions and Conclusions
This paper has introduced a new control-centric approach to model the characteristic of flex sensors on a goniometric glove, which is designed to capture the user hand gesture that can be used to wirelessly control a bionic hand subject to some constraints. The main technique employs the inverse dynamic model strategy along with a black-box identification for the compensator design, which is aimed to provide an approximate linear mapping between the raw sensor output and the dynamic finger goniometry. To smoothly recover the goniometry on the bionic hand's side during the wireless transmission, the compensator is restructured into a Hammerstein-Wiener model, which consists of a linear dynamic system and two static nonlinearities. The linear system is constructed by simplifying the dynamic model from the inverse dynamic design technique, while the static nonlinearities are introduced based on the constraints of the bionic hand, and to account for the uncertain behavior of the sensors as well as the unmodeled dynamics from the black-box identification method. A series of real-time experiments involving several hand gestures have been conducted to analyze the performance of the proposed method. The experimental results with several trials for each gesture show that a great improvement is obtained via the Hammerstein-Wiener compensator approach where the resulting average errors are significantly smaller than the other two methods considered. This concludes that the proposed strategy can remarkably improve the dynamic goniometry of the glove, and thus, provides a smooth human-robot collaboration with the bionic hand.
While the experimental results show a great accuracy via the proposed method, this work only considers one degree-of-freedom movement from the MCP joint. For future work, the framework will be extended to include the overall 3D motion of the goniometric glove to further enhance the bionic hand control system. This however may require some modifications on the bionic hand's structure to allow more gestures from the glove to be recovered. The proposed method can also be combined with another technique such as artificial neural network to find the correlations between the hand palm and the fingers, as well as the correlation between the fingers itself.

Conflicts of Interest:
The authors declare no conflict of interest.

Notations and Abbreviations
The following notations and abbreviations are used in this manuscript: i The subscript i = 1, 2, 3, 4 and 5 on a symbol indicates the signal associated with the thumb, pointer, middle, ring, and pinky fingers, respectively.