Visualizing and Evaluating Finger Movement Using Combined Acceleration and Contact-Force Sensors: A Proof-of-Concept Study

The 10-s grip and release is a method to evaluate hand dexterity. Current evaluations only visually determine the presence or absence of a disability, but experienced physicians may also make other diagnoses. In this study, we investigated a method for evaluating hand movement function by acquiring and analyzing fingertip data during a 10-s grip and release using a wearable sensor that can measure triaxial acceleration and strain. The subjects were two healthy females. The analysis was performed on the x-, y-, and z-axis data, and absolute acceleration and contact force of all fingertips. We calculated the variability of the data, the number of grip and release, the frequency response, and each finger’s correlation. Experiments with some grip-and-release patterns have resulted in different characteristics for each. It was suggested that this could be expressed in radar charts to intuitively know the state of grip and release. Contact-force data of each finger were found to be useful for understanding the characteristics of grip and release and improving the accuracy of calculating the number of times to grip and release. Frequency analysis suggests that knowing the periodicity of grip and release can detect unnatural grip and release and tremor states. The correlations between the fingers allow us to consider the finger’s grip-and-release characteristics, considering the hand’s anatomy. By taking these factors into account, it is thought that the 10-s grip-and-release test could give us a new value by objectively assessing the motor functions of the hands other than the number of times of grip and release.


Introduction
With the development of medicine, life expectancy is increasing around the world [1]. In particular, Japan is called a super-aging society, and the aging rate is expected to increase in the future [2]. The difference between the average life expectancy and healthy life expectancy (the period time when a person can live without restrictions on daily life due to health problems) in Japan is 8.5 years for men and 10.2 years for women [3]. This difference is the length of time where some care is needed, and we need to close this gap by extending healthy life expectancy.
Nursing care's principal purpose is to assist the patient's motor function [4][5][6][7][8]. The Barthel index, a method for assessing activities of daily living (ADLs), consists of 10 items: feeding, bathing, grooming, dressing, bowels, bladder, toilet use, transfers, mobility, and stairs [9]. All the items are related to the dexterity in the hands. It has also been reported It was necessary to connect the connector and the calibration unit to a Personal Computer (PC) via USB and to run the HapLog software to use them.
The sensor attached to the finger was covered on the nail side. There were five different sizes of covers (11, 12, 14, 16 and 18 mm) to fit each nail. A triaxial accelerometer and a strain sensor were mounted on the cover. With the cover-up, the left and right accelerations were the x-axis, the front and back accelerations were the y-axis, and the top and bottom accelerations were the z-axis. The strain sensor detected strains by transmitting finger deformation to the cover. The sensor's sampling frequency was selectable from 1 to 1000 Hz, and the acceleration range was selectable from 2 to 8 G.
We connected up to three sensors to the bangle-shaped connector. The digital signals of measurement were displayed in real-time using HapLog software, and information on up to three sensors is saved in a single CSV file. The file includes the following information (Table 1): Table 1. Contents of the data output by HapLog.

Header information
Version information, timestamp, and memo Sensor information (for each sensor) Position of a sensor in connector, name, unit of target data, number of data, and the maximum value and minimum value Measurement data Time, finger contact force, absolute acceleration, raw contact-force data (με), x-axis acceleration, y-axis acceleration, z-axis acceleration, and mark The information includes the HapLog used, the timestamp at the start of the measurement, and memo fields. In the sensor information, the position in the connector, the name of the target data(finger contact force, absolute acceleration, raw contact-force data, x-axis acceleration, y-axis acceleration, z-axis acceleration, and the mark), the unit of the target data (N for finger contact force, G for absolute acceleration, x-axis acceleration, yaxis acceleration, and z-axis acceleration, με for raw contact-force data, and dots for marks), the number of target data, the maximum value, and the minimum value of the target data were displayed. The raw contact-force data pointed to the strain, and the finger contact force was the data converted from the raw contact-force data by the calibration unit. The following equation can express the x-axis acceleration ( ), y-axis acceleration ( ), and z-axis acceleration ( ) at the th time.

Header information
Version information, timestamp, and memo Sensor information (for each sensor) Position of a sensor in connector, name, unit of target data, number of data, and the maximum value and minimum value Measurement data Time, finger contact force, absolute acceleration, raw contact-force data (µε), x-axis acceleration, y-axis acceleration, z-axis acceleration, and mark The information includes the HapLog used, the timestamp at the start of the measurement, and memo fields. In the sensor information, the position in the connector, the name of the target data(finger contact force, absolute acceleration, raw contact-force data, x-axis acceleration, y-axis acceleration, z-axis acceleration, and the mark), the unit of the target data (N for finger contact force, G for absolute acceleration, x-axis acceleration, y-axis acceleration, and z-axis acceleration, µε for raw contact-force data, and dots for marks), the number of target data, the maximum value, and the minimum value of the target data were displayed. The raw contact-force data pointed to the strain, and the finger contact force was the data converted from the raw contact-force data by the calibration unit. The following equation can express the x-axis acceleration (x i ), y-axis acceleration (y i ), and z-axis acceleration (z i ) at the ith time.
The mark indicates that its button was clicked during measurement in the operation screen of the HapLog software. In the measurement data, we measure the time from zero (starting time), and the value of each target data at that time is recorded in the row of the time, and each column is a time series of the target data. Two decimal places show finger contact force, absolute acceleration, x-axis acceleration, y-axis acceleration, and z-axis acceleration.

Measurement
In this study, we used two HapLogs and two computers to measure five fingers: thumb, index, middle, ring, and little fingers. However, the use of an unlinked device may cause a gap in the measurement start time; therefore, we attached one of the sensors to Sensors 2021, 21,1918 5 of 17 the thumb of each HapLog and corrected the difference by preprocessing, as shown in Section 2.3 ( Figure 2). celeration.

Measurement
In this study, we used two HapLogs and two computers to measure five fingers: thumb, index, middle, ring, and little fingers. However, the use of an unlinked device may cause a gap in the measurement start time; therefore, we attached one of the sensors to the thumb of each HapLog and corrected the difference by preprocessing, as shown in Section 2.3 ( Figure 2). The subjects were two members of our laboratory. They were women in their 20 s and 40 s, and both were healthy. We explained the study and the experiment to the subjects and obtained their written consent to participate in the experiment.
The subjects wore the HapLog and gripped and released their hands under the following conditions. 1. Freely 2. As soon as possible 3. Slowly 4. Irregularly 5. Shivering 6. Without moving thumb as much as possible 7. Without moving the index finger as much as possible 8. Without moving the middle finger as much as possible 9. Without moving the ring finger as much as possible 10. Without moving little finger as much as possible The grip-and-release motions were performed by the left and right hands for 20 s each. The sampling frequency was 200 Hz and the acceleration range was set to 4 G. Condition 1, which was standard, and the subject did grip and release her hand in a natural state without any instruction. As the same method, the grip-and-release actions were performed as quickly as possible in condition 2. From condition 3 onward, we set up the gripand-release patterns for the difficulty for disease, symptoms, or other reasons. Video recording was also performed during the measurements. The subjects were two members of our laboratory. They were women in their 20 s and 40 s, and both were healthy. We explained the study and the experiment to the subjects and obtained their written consent to participate in the experiment.
The subjects wore the HapLog and gripped and released their hands under the following conditions. 1.
Without moving thumb as much as possible 7.
Without moving the index finger as much as possible 8.
Without moving the middle finger as much as possible 9.
Without moving the ring finger as much as possible 10. Without moving little finger as much as possible The grip-and-release motions were performed by the left and right hands for 20 s each. The sampling frequency was 200 Hz and the acceleration range was set to 4 G. Condition 1, which was standard, and the subject did grip and release her hand in a natural state without any instruction. As the same method, the grip-and-release actions were performed as quickly as possible in condition 2. From condition 3 onward, we set up the grip-and-release patterns for the difficulty for disease, symptoms, or other reasons. Video recording was also performed during the measurements.

Preprocessing
We corrected the gap in the measurement start time. Specifically, we obtained time series data of the absolute acceleration of the thumb from two sensors attached to the thumb and calculated multiple cross-correlation coefficients. We found the cross-correlation coefficient r for the number of data n by the following equation.
x is the thumb data of one HapLog and y is the thumb data of the other HapLog, with x and y being the mean of each. The numerator of the equation represents the covariance of the thumb data calculated from each HapLog. The denominator represents the standard Sensors 2021, 21,1918 6 of 17 deviation of the thumb data calculated from each HapLog. In this study, we shifted the 4 s time-series data of one HapLog by one point from 0 to 2 s, and calculated the crosscorrelation function with the time-series data of the other HapLog and found the maximum value. According to the above parameters, 400 cross-correlation coefficients with n = 800 data were calculated from the time-series data acquired at a sampling frequency of 200 Hz, and the point with the highest value was selected as the reference point for the start of the measurement. We obtained 10 s of data (2048 data points). A stable grip-and-release motion occurred later than 5 s after the newly established start point by correcting the misalignment. The data were obtained after 5 s from the start to get a stable grip-and-release motion not the grip-and-release motion of the start. The reason for the 10 s was to evaluate the 10-s grip and release. The number of data was set to 2048 because it needed to be a power of 2 for the frequency analysis in the 2.4 evaluation. We defined the data to be evaluated as absolute acceleration, raw contact-force data, x-axis acceleration, y-axis acceleration, and z-axis acceleration. The reason for excluding the finger contact force from the measurement data in the table was that the values were based on HapLog's algorithm using a calibration unit. We also did not operate the mark during the measurement, and for this reason, it was excluded.

Evaluation
The following items were obtained from the data and used in the evaluation.

•
Data variation • Number of times to grip and release • Frequency characteristics • Correlation of each finger In terms of data variability, we calculated the standard deviation of each finger and each item of time series data (absolute acceleration, x-axis acceleration, y-axis acceleration, z-axis acceleration, and raw contact-force data) for each subject and grip-and-release condition. We compared the standard deviations of conditions 1 to 10 by finger and item in the two groups of significant difference of the left and right hands. We compared the standard deviation of conditions 1 to 10 for each subject by finger and item for significant between-subject differences. To intuitively understand the characteristics of grip and release, we visualized the variability of each finger's data on each item in a radar chart. A study on communicating the value of healthcare using radar charts reported that radar charts, which incorporated data on patient outcomes and costs, allowed for comprehensive and collaborative discussions [54]. This study also aims to comprehensively capture the data of all fingers of one hand and extract their characteristics by using radar charts.
The number of grip and release was the same as in the 10-s grip and release. We compared the results with the number of grip and release calculated by visual inspection and evaluated the matching rate. The target data were each finger and item of time series data (absolute acceleration, x-axis acceleration, y-axis acceleration, z-axis acceleration, and raw contact-force data) and their summation over the same time series. However, before adding up, we found the maximum and minimum values for each finger's same item. We normalized to 0-1 based on those values. The combinations to be added were 2 25 (= 33, 554, 432). The number of grip and release was calculated by dividing the maximum and minimum values of the time series data by two. The number of times the data exceeded or fell below the threshold was divided by two (rounded up to the nearest whole number). The number of grip and release was counted, and the accuracy for all combinations was calculated.
We performed frequency analysis on each finger and item of time-series data (absolute acceleration, x-axis acceleration, y-axis acceleration, z-axis acceleration, and raw contactforce data) for each subject and grip-and-release condition. We used the fast Fourier transform for frequency analysis. To evaluate the periodicity of grip and release, we calculated the power of the most powerful frequency divided by the total power from the frequency analysis results.
In the correlation of each finger, we calculated the cross-correlation coefficients between each finger and each item of time-series data (absolute acceleration, x-axis acceleration, y-axis acceleration, z-axis acceleration, and raw contact-force data) for each finger, and grip-and-release condition. The cross-correlation coefficients were the same as (2) in Section 2.3. However, x was the time-series data of a certain item on a certain finger, y was the time series data of the same item on a finger different from x, and the means of each were x and y. The number of data was n = 2048 for 10 s.

Results
Based on the 2.4 evaluation, the experiment is as follows: in presenting the results, we label each item in each finger with the alphabetic combinations shown in Table 2 (e.g., absolute acceleration of the thumb, Ta).

Finger
T (thumb), I (index finger), M (middle finger), R (ring finger), and L (little finger) Item a (Absolute Acceleration), x (x-axis acceleration), y (y-axis acceleration), and z (z-axis acceleration) p (raw contact-force data) In addition, A is a female in her 20 s, and B is a female in her 40 s.

Data Variation
A paired t-test was used for the left and right significant difference test. For the between-subjects significant difference test, we used the Student t-test, the Welch t-test, and the Wilcoxon signed-rank test, as a result of the Shapiro-Wilk test and the F-test. The table below showed the significant differences (Table 3). The significant difference between the left and right sides was found in the x-axis acceleration of the ring finger, which was common to the subjects. There were more significantly different items for subject A than for subject B. Significant differences between subjects were common to the left and right sides: raw contact-force data of the thumb, raw z-axis acceleration, and contact-force data of the index finger, x-axis acceleration of the middle finger, and x-axis acceleration of the ring finger. There was a particularly large difference in the raw contact-force data. Figure 3 showed the variability of each finger's data for each item in subject A, represented by radar charts for each grip-and-release condition. Condition 1 is shown in blue, and conditions from 2 onward are shown in orange to overlap with condition 1. Similarly, the radar charts for subject B are shown in Figure 4.   The variation of absolute acceleration in condition 3 was small for all fingers in both subjects A and B when condition 1 was used as a reference. However, the raw contact-force data of the middle finger were large. In addition, the variability of the z-axis acceleration outside of the thumb tended to be large. In condition 4, there was no change in the variation of absolute acceleration. The variation of absolute acceleration of the finger was smaller than that of the first condition, and the variation of the z-axis acceleration tended to be smaller than that of the first condition.
In addition to the above, subject A tended to show a large variation in the x-axis acceleration in both the left and right sides in condition 1. In condition 2, the variation of the y-axis acceleration tended to be small for both sides, but the overall variation did not change much. In condition 5, the variation in the overall absolute acceleration was small, and the variation in the z-axis acceleration of the index, ring, and little fingers was large. Concerning subject B, there was a large variance in the raw contact-force data of condition 1 ring finger in the left hand. Moreover, the variation of the x-axis acceleration of the middle finger tended to be small. In condition 2, the variation of absolute acceleration was larger. In condition 5, contrary to A, there was a large variation in the overall absolute acceleration.
The relationship between the raw contact-force data and condition 1 was superimposed; therefore, it was not easy to understand the fingers' relationship. Figure 5 shows radar charts of the raw contact-force data.
Overall, the raw contact-force data for the middle and occasional ring finger were large. In addition, there was a tendency that the variation of the finger was small from condition 6 onwards. Table 4 shows the combinations with the highest matching rate without error in the number of grip and release and the single item with the highest matching rate. The right column shows the grip-and-release conditions for different numbers of times.

Number of Times to Grip and Release
The highest matching rate was 80.0%. Both combinations included the z-axis acceleration of the middle finger and the raw contact-force data of the little finger. The highest single matching rate was 75.0% of the z-axis acceleration of the middle finger. Conditions 5, 9, and 10 were often wrong. The left hand made more errors than the right hand. Figure 6 shows the results of the frequency analysis, where the highest frequency one divided by the total power is shown on a scale of red for the value close to 1 and blue for the value close to 0.

Frequency Characteristics
The mean was greater than 40% in conditions 1, 3 and 7. However, the mean was lower than 30% in conditions 4, 5, and 9.

Correlation of Each Finger
Conditions 1 to 5 were different, but the point at which all fingers moved simultaneously was the same. Therefore, after calculating the correlation coefficients for each item of data for each finger, the mean values were calculated for each subject. The value close to 1 is shown in red, and the value close to 0 is shown in blue on a scale in Figure 7.
In common with the subjects, the correlations were high for all combinations of the index and middle finger combined acceleration, index and middle finger y-axis acceleration, middle and ring finger y-axis acceleration, ring and little finger y-axis acceleration, and index finger to little finger z-axis acceleration. The combination with the highest mean of correlation coefficients was the z-axis acceleration of the middle finger and the z-axis acceleration of the ring finger for both subjects. The combination with the overall thumb item and raw contact-force data had a low correlation.
The mean of the correlation coefficients for each item from condition 6 onwards, when no specific finger is moved, is shown in Figure 8. Values close to 1 are shown as red and values close to 0 are shown as blue on a scale.
Compared with Figure 7, the relationship between the index, middle, and ring fingers tended to be stronger in condition 6; after 7, it tended to be weaker. Subject A had a particularly low correlation. There was also a high correlation between subject A's right hand in condition 10. Subject B correlated highly with the right hand in condition 9.

Data Variation
A total of 25 combinations were shown each finger and each item. From Table 3, regarding the significant differences between the left and right hands, A and B showed significant differences in 24% and 16% of the combinations, respectively, but the differences were not considered significant overall. Regarding the significant differences between subjects, significant differences were found in 32% of the combinations for the right hand and 36% for the left hand, but no significant differences were considered overall. However, the differences in raw contact-force data were particularly large between subjects. We need further validation of the data for healthy subjects.
In Figures 3-5, we found that the use of radar charts helps us to intuitively understand how the grip and release of radar charts occur. In particular, since absolute acceleration is an index of momentum, the variability of the whole of absolute acceleration was smaller in the case of condition 3. In addition, in the case of condition 6 and later, the variability of the absolute acceleration of the corresponding finger was small for both subjects. The z-axis acceleration was also considered to have a similar tendency because the grip and release of the fingers can be considered a vertical motion. Figure 5 shows that the same tendency was observed in the raw contact-force data because of the finger distortion in the gripping motion. In the raw contact-force data, the raw contact-force data of the middle and ring fingers were also larger, which was thought to be because these fingers were longer in hand and were more in contact with the palm when gripping.
In comparison with the actual grip and release of subject A in Figure 3, the variability of the x-axis acceleration (left-right) in condition 1 tended to be large in both the left and right hands and was considered because subject A grasps with her middle finger when she naturally grasps the hand. By using the radar charts, we found that the grip-and-release speed of condition 2, which assumed of a 10-s grip and release, was different from that of condition 1, but the motion itself did not change much from that of condition 1 (only the tempo of grip was changed). In condition 5, the radar charts did not capture the tremor, but it was found that she gripped and released her hands less and spread her hands more often. As for subject B in Figure 4, from the actual conditions and the variation of the x-axis acceleration, it was found that the middle finger gripped and released without any side-to-side motion. Moreover, unlike subject A, the grip and release of condition 2, which was simulated as 10-s grip and release, was more intense than in condition 1.
Thus, it is possible to grasp the trend from the radar charts without checking the grip and release of the radar. Medical professionals and the public were to be used to elucidate the abnormality of the hand and to check the progress of rehabilitation. However, it is not possible to obtain grip and release under different conditions for the same person, making it necessary to build a model for healthy people. Table 4 shows that the combinations with a high matching rate include the z-axis acceleration of the middle finger. Moreover, the matching rate increased when the raw contact-force data of the middle finger were combined with the raw contact-force data, not the z-axis of the middle finger alone, and since we considered the raw contact-force data in 4.1 as well, it was thought that obtaining information on strain is effective. Although the error was larger for certain conditions, such as conditions 5 and 9, the number of times the error was calculated using an abnormal grip estimated by combining it with other evaluations.

Frequency Characteristics
As in Section 4.1, there were 25 combinations for each finger and each item. The larger the power of the most potent frequency divided by the total power, the more periodicity is considered present. As shown in Figure 6, conditions 1, 3, and 7, which had higher values, were also cyclic in terms of actual grip and release. However, conditions 4, 5, and 9, where the mean value was below 30%, showed little periodicity in the grip-and-release movements. In the case of condition 4, the values were smaller because of the irregular grip-and-release instructions. In the case of condition 5, where the user was instructed to grip and release while trembling, the periodicity of the trembling was considered the noise, and values were smaller. Moreover, under condition 9, where the ring finger was instructed not to move, the high correlation between the middle and ring fingers affected the grip-and-release movements (Figure 6), which will be discussed in 4.4. Based on these results, it was considered possible to detect anomalies in the periodicity of grip and release by dividing the power of the most potent frequency divided by the total power.

Correlation of Each Finger
As in Section 4.1, there were 25 combinations for each finger and each item. Figure 7 shows that the high correlation from the index finger to the little finger was considered due to the connection of the tendons used to move the fingers. In particular, the middle and ring fingers were interlocked. The z-axis acceleration of the middle and ring fingers was also included in the combinations with the highest accuracy in Table 4. That is, the combination of the z-axis acceleration of the middle finger and the z-axis acceleration of the ring finger is considered to accentuate the grip-and-release features. The reason for the low correlation of the combinations of thumb data was due to the lack of motion during the grip-and-release motions and the fact that the motions are more independent than those of other fingers. Thus, we thought it is possible to consider the anatomical structure of the hand concerning to the correlation. After condition 7 in Figure 8, the correlation tended to be lower from the index finger to the little finger, and the right hand of subject A in condition 9 was particularly visually abnormal. For example, if a patient has this condition, regular measurement of his or her condition during rehabilitation would help visualize and quantify the progress of rehabilitation.
From the above, it was considered possible to understand the characteristics of grip and release and the state of the hand by combining each method. It was also thought that visualization on radar charts and scales would allow for an intuitive understanding of the features. However, since the subjects were healthy women and not all hand movements were covered, the generalization of the results is limited.

Limitations
The number of subjects is small, and the number of data is small. Our evaluation method's usefulness was suggested, and we will obtain more data, including the presence or absence of diseases. The number of N required for each disease will be determined by conducting a usefulness evaluation. Specifically, 10 cases of data for each disease should be accumulated, and the number of Ns should be determined by evaluating significant differences (including ROC curves). If it is difficult to distinguish from other diseases, it is necessary to accumulate the number of Ns over time.
The sensors used in this study proved to be useful in providing strain information. However, they were sometimes difficult to attach and challenging to operate during the experiments; therefore, it is necessary to investigate or develop a smaller, lighter, and easier attach the sensor. IN addition, the use of wireless systems is expected to be more widespread.

Conclusions
In this study, we used an accelerometer and a strain sensor to acquire data on the hand grip-and-release movements during a 10-s grip and release. Feature extraction of each finger, calculation of the number of times to grip and release the data, frequency analysis, and correlation between each finger were derived. Different characteristics were obtained depending on the grip-and-release patterns. It was suggested that the state of grip and release of the radar charts were intuitively known by expressing it in the radar charts. The raw contact-force data were found to be useful for knowing the characteristics of grip and release and for improving the accuracy of calculating the number of times of grip and release. Frequency analysis suggests that knowledge of the periodicity of grip and release can detect unnatural grip and release and tremor states. The correlations between the fingers allowed us to consider the grip-and-release characteristics of the fingers, considering the anatomy of the hand. By taking these factors into account, it was thought that the 10-s grip-and-release test could be given a new value by objectively assessing the motor functions of the hands, other than the number of times of grip and release.
However, this study was a proof-of-concept study, and the subjects were two healthy females. There are limitations in generalizing the results of this experiment and using them in clinical practice. In future studies, we will collect and analyze data from a larger number of subjects and diseases to establish the evaluation suggested in this study as useful. Institutional Review Board Statement: Ethical review and approval were waived for this study, due to REASON. This study is a preliminary study that also serves as a review of the conditions, therefore the study has not been applied for an ethics committee and is currently being applied for.