Spatio-Temporal Human Grip Force Analysis via Sensor Arrays

This study describes a technique for measuring human grip forces exerted on a cylindrical object via a sensor array. Standardised resistor-based pressure sensor arrays for industrial and medical applications have been available for some time. We used a special 20 mm diameter grip rod that subjects could either move actively with their fingers in the horizontal direction or exert reactive forces against opposing forces generated in the rod by a linear motor. The sensor array film was attached to the rod by adhesive tape and covered approximately 45 cm2 of the rod surface. The sensor density was 4/cm2 with each sensor having a force resolution of 0.1 N. A scan across all sensors resulted in a corresponding frame containing force values at a frame repetition rate of 150/s. The force value of a given sensor was interpreted as a pixel value resulting in a false-colour image. Based on remote sensed image analysis an algorithm was developed to distinguish significant force-representing pixels from those affected by noise. This allowed tracking of the position of identified fingers in subsequent frames such that spatio-temporal grip force profiles for individual fingers could be derived. Moreover, the algorithm allowed simultaneous measurement of forces exerted without any constraints on the number of fingers or on the position of the fingers. The system is thus well suited for basic and clinical research in human physiology as well as for studies in psychophysics.


Introduction
The ability to grasp objects enables humans to perform a wide range of manipulative movements.
Skilled control of prehensile finger forces is an essential feature of tool use in daily life. In healthy subjects, grip force is scaled accurately to object properties so as to prevent a hand-held object from slipping [1][2][3]. Impaired hand function is a frequent observation in movement disorders. The characteristics of impaired finger force control include inefficient grip force scaling and imprecision of the temporal coupling between grip and load force profiles. Grip force analysis is a highly sensitive method for detecting even subtle impairments of finger force control or to document impaired dexterity and its rehabilitation in movement disorders [4,5].
The most common measurement system employs three or more force/torque transducers in a fixed arrangement [6,7,9,10,[14][15][16]. Although these systems measure force very accurately, their disadvantage for studies on patients with motor disorders is the fixed, and thus inconvenient, arrangement of finger positions as well as the reduced contact area. A different method of measuring grip forces has been presented by Pylatiuk and colleagues [13]. They attached sensors directly to the finger and palmar skin with double-sided adhesive tape, leading to altered skin sensitivity. Acute skin sensitivity, however, is required for the internal representation of the physical object properties [17][18][19][20]. Pylatiuk's system is suitable for studies on prehension of force grips but less appropriate for studies on grip forces of precision grips. An early publication using sensor arrays for grip force measurements [8] described a system specialised for quantifying grip activity during handwriting. In this system a forcemeasuring film was attached to a writing utensil. Due to its specialization this system had low spatial resolution and did not allow the separation of different finger grip forces over time.
The present study presents a technique employing a sensor array to measure grip forces exerted by the human hand on an object. The use of the sensor array enabled grip force measurements over a large contact area (up to 170 cm 2 ) with high spatial resolution (5.08 mm sensor distance), good force resolution (0.1 N) and reasonable temporal resolution (150 Hz). In addition, the forces exerted by any number of fingers could be measured simultaneously without any constraints on finger position. This is essential for studying patients with impaired dexterity. Our main research interest was the evaluation of grasping abilities of patients with cerebellar disease. For this reason, we created a new grasping task in which subjects had to pull an initially blocked and then unexpectedly released grip rod, which is a task comparable with picking a raspberry. Picking a raspberry requires perfectly adjusted, increasing grip forces to pull the raspberry off the bush without squashing it. This, however, was very difficult for cerebellar patients. The representative data presented here demonstrate the ability of the measurement system, and are not intended as a statement on grasping dexterities of patients with cerebellar disease.

Grip Rod and Force-Measuring Film
Grip rod, force-measuring film and algorithms have been described in detail elsewhere [11,12]. A brief but comprehensive description of the system follows. The grip rod comprised a cylindrical metal bar (20 mm diameter), a linear motor (type: STA2505, Copley Controls, Canton, MA, USA) that could move the grip rod up to 100 mm axially and horizontally, a linear potentiometer to measure the position (type: REM 13-200-K, Megatron Elektronik, Putzbrunn, Germany) and a force transducer to measure the force exerted along the rod (type: U9B, Hottinger Baldwin Messtechnik, Darmstadt, Germany). In the pull or push direction the maximal force was hardware-limited to 25 N. The motor was controlled by custom-written software using LabVIEW (v. 8.2, National Instruments, Austin, TX, USA). The static friction of the grip rod was 0.6 N and the dynamic friction 0.3 N. To produce a 6-N square pulse the motor reacted in less than 15 ms.
A standardised resistor-based pressure sensor array (type: 3000/HOT, Tekscan, MA, USA) was used as force-measuring film. It contained sensors in a rectangular order with a distance of 5.08 mm in each direction. The force range of each sensor was 0.89-13.3 N. Some 45 cm 2 of the rod surface was covered by the force-measuring film which was attached on one side to the rod with double-sided adhesive tape. The measuring film was bent from above around the rod so that the edges in axial direction were situated at the top (see Figure 1a). The used areas of the measuring film contained 180 sensors, 15 columns along the rod axis covering 75 mm of the grip rod length and 12 rows orthogonal to them covering 61 mm of the circumference. This allowed force measurements to be performed with extended fingers. Bending the measurement film around the rod elicited inhomogeneous noise with forces up to 1 N at a given single sensor (see Figure 1c). In the data image of were measured synchronously and stored as an image frame using F-Scan software (v. 5.24, Tekscan) on a desktop computer. Data were sampled at 150 frames/s.

Detection of Grip Forces and Correlation of Finger Positions with Forces Significantly Detected
Values recorded via the sensor array at any time t i are seen as pixels of a sensed image at t i and are presented as a false-colour image (e.g., Frame n and n+1 in Figure 1e). Based on this idea, significant grip forces [11] was achieved by a modification of Rogerson's algorithm for change detection in remotely sensed images [21]. Schema of the "pick the raspberry task" with determing the different corresponding phases.
The recording software is able to subtract a noise image from any recorded image. Nevertheless, measuring force values without any fingers in contact with the grip rod and applying the subtraction algorithm to the recorded values revealed a noticeable level of remnant noise in the data. This remaining noise level had to be taken into account when the grip rod was touched softly and consequently, the resulting signal-to-noise ratio was low. This consideration is taken into account in the formulation of Equation 1 and the remaining mean noise level was measured separately in a noise film of at least 300 frames ( Figure 1d). The algorithm was divided in three steps: (1) transformation of the image F i from a N(μ,σ)-distribution in a N(0,1)-distribution, (2) smoothing the data with a Gaussian kernel to detect sub regions of local spatial association, (3) detection of significant pixel with Bonferroni correction [21,22].
is the raw value of pixel p j ; = mean noise at pixel p j , measured separately across at least 300 frames of the noise film (see Figure   1d). n R i (p j ) = normalised residual of pixel p j , in frame F i . S. D. (R i ) = standard deviation of the residual of frame F i . 95% c.v. = 95% upper confidence value of the system noise level measured in the noise film. (

2) Smoothing each frame:
Each frame is smoothed with a Gaussian kernel K(p j ,p k ) to improve the signal to noise ratio.
The values z j of the smoothed normalised residual frame represent local statistics that have the purpose of detecting sub regions of local spatial association. The parameter σ (in pixel) is chosen to match the scale at which spatial association exists (Equations 3 and 4 from [21]).
with nR i (p j ) = normalised residual of pixel p j in Frame F i and K(p j ,p k ) as: where d(p j ,p k ) = distance between pixel p j and p k .
(3) Significant pixel detection. The third step was to detect the significance of force as a pixel in a frame, whereby the number of tests depended on the total number of pixels and the value of σ 2 of the Gaussian kernel (Equation 2). The larger the value of σ 2 , the greater was the spatial correlation between local tests. Rogerson provided an approximation (Equation 3) that allowed direct calculation of the observed maximum local statistics [21,22]: by using α = 0.05, σ = 0.5, A = 12 × 15 pixel = 180 resulted in M* = 3.425. Any value z j in the smoothed normalised residual frame exceeding M* is significant at the 5%-level of first error.

Pairwise Correlation of Single-Finger Forces between Frames
Position and force-values were assigned semi-automatically to individual fingers [11]. The algorithm assumes that the fingers were not crossed, that changes in finger position between frames were small, and that changes in finger position were continuous. In at least one frame significant data were assigned manually to the fingers. Starting from this position the algorithm assigned data to the fingers automatically up to the start/end of the complete sequence or the next starting position given by the user. The algorithm correlated pairwise the assigned position between neighbouring frames. This process was divided into three steps ( Figure 1e): 1. By using the "Flood-fill-algorithm", recognised coherent areas were marked (see http://en.wikipedia.org/wiki/Flood_fill). In this way significant pixels situated horizontally or vertically adjacent were combined (e.g., circle in Frame n of Figure 1e). The weighted centre of force was calculated for each marked area.
2. Centres of areas lying inside a given distance (intraframe distance: 1.645 pixel) were considered to be of the same origin and were combined. In this way combined areas contacted at their corners only (e.g., circle in Frame n+1 of Figure 1e).
3. Subsequently, the final weighted centres were compared and combined with the centres of the previous frame. The maximum distance between centres of the same finger was here 0.8 pixel (interframe distance). An example is given in Figure 1e. From the position of finger D1 (thumb) in Frame n followed the position of D1 in Frame n + 1.

Dynamic Torque Analysis
The term "torque" is used interchangeably in mechanics. In this study "torque" was used to designate a force moment resulting from normal finger forces which would tend to deviate the rod from the pull-direction and represent losses that subjects try unconsciously to minimise.
From the finger positions (see Figure 1a), a gripped rod slice element and its centre of mass (CoM) were defined. The rod slice element was the part of the rod between the remotest fingers (e.g., rod part between fingers D1 and D3 in Figure 1b). Levers were derived from the position of individual fingers and the centre of mass of the rod slice element. Individual torques were given by the vector product of finger force and the lever, defined by the distance from the position of the finger on the rod on which the force was exerted to the centre of mass of the rod slice element. Torques of the individual fingers were calculated at each time (Equation 4a). From these data a total torque function was calculated over time for each finger (Equation 4b). The torques described here were situated all orthogonally to the pull axis in the x/y-plane according to Figure 1b. Hence, the rod would deviate from the pull-axis.
However, given that the rod is blocked the torque produces a virtual deviation only.

"Pick the Raspberry" Task
The "pick the raspberry task" is a precision grip task that requires continuous adjustments of grip and pull forces [12]. Subjects were seated comfortably on a height-adjustable chair facing the rod. The upper arm was held almost vertical and the forearm was flexed at the elbow joint 90° in the sagittal plane. The forearm was in a neutral position between pronation and supination. Subjects were asked to pull the rod horizontally with a three-finger precision grip (see Figure 1a). Each trial was self-paced by the subjects and its start was detected when a pull force velocity threshold of 0.5 N/s was exceeded.
After a randomly variable interval (1-5 s) the rod was released (unpredictable rod release/URR). After rod release, the rod moved due to the subject's pull force. Subjects were asked to stop the pull movement of the rod with low grip force. The interval from recording start until the detected trial start was termed the grip phase, the following interval until the URR was termed the pull phase and the last interval was termed the pluck phase ( Figure 1f). If during pull-phase the velocity (i.e., the pull force rate) exceeded an upper limit (5 N/s) subjects were informed by a red LED in front of them. The intertrial-interval was in the range of 10 s to 15 s. Prior to recording, subjects were allowed at least 10 trials for practice. The time course of the corresponding and natural task and the requested increase of the pull force are shown in Figure 1f.

Simulation of the "Pick the Raspberry" Task
The reliability of the force-measuring film was evaluated by simulating the mean behaviour of a control subject during the "pick the raspberry" task. For this purpose a continuously increasing motordriven force (2.5-10 N) was applied to the sensor array over an area of approximately 300 mm 2 for 4.5.
After the maximal force value of 10 N the motor-driven force was released immediately.

Subjects and Patients
This study was performed with the permission of the ethics committee of the Ludwig in

Results and Discussion
This study presents a technique employing a sensor array for measuring grip forces exerted by the human hand on a cylindrical object. Force values of all sensors were interpreted as pixel values of a false-colour image. The detection of grip forces was based on a modified change detection algorithm for remote sensed image analysis which distinguished significant force-representing pixels from those affected by noise [11]. The position of identified fingers was tracked in subsequent frames to construct spatio-temporal grip force profiles of an individual finger. The algorithm allowed simultaneous measurement of forces exerted without any constraints on the number of fingers or on the position of the fingers. The system is thus well suited for studying grip forces exerted by both healthy subjects and patients with impaired dexterity.

Reliability test of the Force-Measuring Film
The "pick the raspberry" task was simulated to evaluate the reliability of the force sensors (see Section 2.6). The results from ten repetitions are shown in Figure 2a. The red lines show the motordriven forces; the green lines show the measured forces. The sensor array follows the motor-driven forces sufficiently precisely over the range of forces of interest, as can be seen by the averaged motordriven and measured forces (blue and yellow lines, respectively). After reaching the maximal force value of 10 N the motor-driven force was released immediately. This was followed by a vibration of the system due to technical constraints. Even this reaction was measured adequately by the sensor array. While the motor-driven forces were exposed to noise, the real error of the sensor array was determined by following equation: (5) with SEM(measured force), SEM(motor-driven force): standard error of the mean of the measured or motor-driven force, respectively.

Spatio-Temporal Profiles of Individual Finger Grip Force Development
The sensor array described here allows the measurement of human grip forces of any number of fingers exerted at any position of the grip rod. This arrangement allowed the development of individual grip forces and the change of positions of all gripping fingers to be studied. A three-finger grip is demonstrated as well as the area of sensor array used, limited by coloured edges (Figure 3a Patients with cerebellar degeneration showed a different picture. CBL1 showed a pattern similar to that of the healthy subject but had clearly higher grip forces and a larger finger aperture (see the distance of the fingers in the x/y-plane, Figure 3c). CBL2 exerted high grip forces over the total analysis time (Figure 3d). This latter patient stabilised the grip with the thumb and the index finger, as did CTRL, although he exerted a high grip force with the middle finger as well. Another patient (CBL3, Figure 3e,f) showed completely different behaviour depending on the hand used. With the left hand, the patient produced hypermetric grip forces and used the ring finger as well (magenta line in Figure 3e). In contrast, when using the right hand, the patient produced hypometric grip forces, just touching the rod and sliding along the rod with his fingers.
Consequently, measurement systems with fixed finger positions are suitable for studies with healthy subjects, but are of limited use for patients with motor disorders. For the latter group, measurement systems with enlarged contact area and reduced finger aperture are essential [4,5,11,12]. Although all subjects exhibit different strategies when gripping the rod and following the task instruction, all corresponding grip forces have to be recorded by the measuring system. The data presented in Figure 3 yield the potential of the sensor array technique, with the effectiveness of our system shown at the best in Figure 3e,f. When using the left hand CBL3 employed four fingers instead of three and when using the right hand the patient did not exert sufficient grip forces to prevent sliding along the rod. This kind of behaviour can be studied only via a sensor array-based system as introduced in this study. Grip force measuring systems with fixed finger positions are used frequently [6,7,9,10,[14][15][16] but are unable to detect losing a test object, as occurred with CBL3 (Figure 3e), or the employment of an additional finger (Figure 3e).

Dynamic Development of Torque
The individuality with which different subjects solve the task is illustrated in Figure 3. Although different pull forces were recorded they result from the same total grip forces. The development of grip forces during the trials illustrated in Figure 3b,c obtained from CTRL and CBL1 are compared now (Figure 4a,b). The total grip forces (black line) of both subjects reached approximately the same maximal value at URR. However, at URR CTRL produced 20 % more pull force than CBL1 (data not shown). This indicates that the grip of CTRL was more efficient than that of CBL1. Analysing the development of individual grip forces it can be seen that the CTRL had an equilibrated force relation over all fingers (thumb-blue line, index finger-green line, middle finger-red line) whereas the patient tended to employ the thumb as the main force-exerting finger (approx. 50% of total grip force at URR) and used the other fingers less effectively. Individual grip forces alone were not sufficient to explain the difference in economy of the two different grips. Due to the finger position the individual fingergrip forces produced not only pull force in horizontal direction but also torques, which would cause the grip rod to deviate from the pull axis. These torques are losses and are uneconomic in the biomechanical sense.   Figure 4e shows the extension of the projected torque vectors shown in the grey area. For a-f: colour convention as in Figure 3.
The economy of grip behaviour was calculated for all trials expressed by the total grip force, the pull force, and the total torque ( Figure 5). The averaged data show that the patient tended to exert higher grip forces than the control subject (Figures 5a,b) but produced less pull force (Figure 5c,d) with higher variation. At URR CBL1 produced 10.3 ± 5.2 N pull force whereas CTRL produced 12.0 ± 2.3 N. This inefficiency, expressed by the production of high torques, represents losses and thus, is uneconomic (Figure 5e,f). ± standard deviation (yellow area) (c, d) Averaged total pull force ± standard deviation.
(e, f) Averaged total torque ± standard deviation. All recordings are aligned to the unpredictable rod release.
Taken together, our system allows insights into both the temporal coupling of grip force and pull force and into the economy of grip behaviour due to different finger position in unrestraint conditions. This latter is an essential aspect in the understanding of impaired dexterity of patients with motor disorders.

Conclusions
Standardised resistor-based pressure sensor arrays for medical and industrial applications allow the position, the size of the contact area and forces exerted by individual fingers touching an object to be determined. The system described here is approved as medical device in several countries. The size of the force-measuring film can be adapted readily to different common objects and can be attached to surfaces by adhesive tape. This makes it a suitable, easy-to-use instrument. The enlarged contact area of our system allows measurements of forces exerted by any number of fingers simultaneously without any constraints on finger position with respect to time. Hence, it allows insights into the temporal coupling between grip force and pull force as well as into the economy of grip behaviour due to different finger position under unrestraint conditions. Analysis of gripping an object in a completely natural manner reveals differences in the efficiency of grips between healthy subjects and patients with motor disorders. This is an essential aspect in basic and clinical research for detecting and understanding impaired dexterity of patients.