Method of Topological Skeletonization for Evaluation of Effectiveness of Medical Rehabilitation Based on Upper Limb Exoskeletons

Abstract

An important aspect of medical rehabilitation using exoskeletons is objective monitoring of the effectiveness of the exercise program. This control is most often manual and relies on the attention of a rehabilitation physician, but advanced rehabilitation systems also use computer vision technology. Topological skeletons generalize large areas of digital images, representing a virtual internal framework of the analyzed object. The patient and the exoskeleton are described either as a set of spatially disparate (but not explicitly related to either the patient or the exoskeleton) topological skeletons, or as branches of a single topological skeleton which does not allow for objective monitoring of joint displacements. A method to solve this problem for medical rehabilitation using an upper-limb exoskeleton is proposed. It includes the following stages: (I) identifying the exoskeleton, as well as upper and lower parts of the patient’s body; (II) independent construction of three topological skeletons (separately for the exoskeleton and for the upper and lower parts of the patient’s body); (III) their integration. This approach allows for accurate, real-time analysis of movements in the upper-limb joints and prompt notification to the rehabilitation physician of any significant deviations in the technique of performing prescribed exercises.

1. Introduction

Medical rehabilitation is an integral part of the modern healthcare system [1,2,3]. Broadly speaking, medical rehabilitation includes consultations with a psychologist, physical therapy, therapeutic exercise, mechanography and other necessary services (e.g., consultations with a speech therapist). The emphasis is on specialized medical rehabilitation systems using exoskeletons in the context of this study [4,5]. In particular, this study emphasizes more effective use of upper-limb exoskeletons for the full or partial restoration of musculoskeletal function impaired by disease or injury [6].

Upper-limb medical exoskeletons are highly promising but still relatively uncommon medical rehabilitation systems. Upper-limb exoskeletons distribute load (meaning overall intensity, distribution across muscle bundles, stretch range, etc.) to the relevant muscles and joints to promote proper recovery and coordination.

Upper-limb medical exoskeletons from a hardware perspective typically consist of a robust, relatively immobile frame and a set of movable guide and support elements [4,7,8,9]. The relative movement of these structural elements is most often organized according to electromechanical principles, but pneumatic and hydraulic systems also hold promise [10,11,12].

Data acquisition approaches are also largely straightforward [13,14,15]. The overall picture of the rehabilitation process is most often captured by one or more video cameras. The exoskeleton typically contains elements and modules for estimating distances, angles and local positioning. These could include, for example, optical and ultrasonic distance sensors, angle sensors (encoders), accelerometer modules, gyroscopes, magnetometers, etc. [6,11]. Additionally, pressure and force sensors applied by the patient could be used.

The methodology and software for medical rehabilitation systems using upper-limb exoskeletons are based on the cutting-edge technologies in automated systems, human–machine interfaces, computer vision, neural networks, as well as multidimensional processing and visualization of biomedical signals and data [15,16,17,18]. Construction of topological skeletons is a generally accepted approach for computationally efficient description of a region of interest in both static and dynamic conditions. For example, the processed image is represented as a steppe region of the same geometry when using the “GrassFire” mathematical transformation. Pixel intensities and their significance (information loss with depth is observed in some medical images) become analogous to the height of the vegetation cover and its flammability in the corresponding spatial regions. Simultaneous “ignition” of all the outer boundaries of the simulated steppe region leads to the formation of a set of fire fronts. The points of their collision represent the extinguishing points (the “quench” points). The topological skeleton is constructed as a set of fine lines based on the most significant “quench” points. Topological skeletons are insensitive to single signal spikes with this approach and the processed image is replaced by a relatively small set of “quench” points that generally describe it. The topology of each frame can be independently skeletonized in the situation of data stream digital processing (referring to frame-by-frame video signal processing). Comparing changes in topological skeletons over time has a huge practical value. It effectively characterizes motor activity [19,20].

At the same time, a number of medical and technical issues remain insufficiently studied, such as those related to the description of a set of regions of interest using relatively independent topological skeletons, the skeletonization of medical images with variable detail of different branches and adaptive pruning of some branches of topological skeleton, etc.

In general, short-term progress in the development of medical rehabilitation systems using upper-limb exoskeletons does not require the development of fundamentally new materials, actuators, sensors, computer systems, etc. [21,22,23,24,25,26]. Highly efficient methods for processing the obtained multimodal medical data are of particular interest and are the most important direction for the development of such systems [6,15]. An increase in the number of moving parts and degrees of freedom, the use of more accurate sensors (including thin-film), improving ergonomics, multiple therapeutic interventions (for example, combining physical exercise with physiotherapy in a single device) and integration with artificial intelligence technologies were highlighted as key long-term trends in the development of medical rehabilitation exoskeletons. The creation of highly efficient neural interfaces and the integration of stimulators with living tissue are key challenges in the field.

The aim of this study is to develop a computationally efficient method for the monitoring of significant deviations in the patient’s technique during a series of rehabilitation exercises in real time [6,23]. Key features of this method will be the identification and topological description of the spatial geometry of the upper-limb exoskeleton, as well as the patient’s upper and lower parts of body.

The key difference in the proposed approach is that it focuses not on collecting the maximum possible amount of potentially useful information, but on monitoring exercise technique. The exoskeleton guides the patient’s movements, but does not completely predetermine them. Redistributing the load between muscle groups and the individual fascicles within them is a typical situation associated with involuntary displacement, for example, of the elbow. Healthy physical exercise as well as self-massage of spasmodic muscles could be performed even at home. The purpose of medical rehabilitation is to correctly distribute load (in space, by time and level), which is possible only in accordance with the rehabilitation physician’s prescription. The proposed method includes assessing control (i.e., key, main, most important) distances and angles, comparing the obtained results with set limits, and subsequently (if necessary) promptly alerting medical staff to significant deviations in exercise technique. This is an appreciable difference from known approaches [1,7,8,9,10,11,12,13,14].

2. Materials and Methods

It is logical to divide the description of the proposed method into several relatively independent blocks: (I) localization of the medical rehabilitation exoskeleton as the region of interest; (II) construction of a topological skeleton for the aforementioned region of interest; (III) localization of the upper and lower parts of the patient’s body; (IV) construction of topological skeletons for these parts of the patient’s body; (V) data generalization, obtaining the resulting image and control distances and angles. Each of these relatively independent blocks corresponds to a module of the author’s software product [6,15].

It is important to note that the main version of the proposed method is focused on digital processing of two-dimensional (classical, 2D) images. This is the most convenient option both in terms of the hardware complexity of obtaining such images and the computational performance of their digital processing. The three-dimensional (3D) case mentioned below is only to highlight the full potential of the proposed approach.

The most common additive color scheme used for encoding colors in two-dimensional and three-dimensional images is the RGB scheme (which includes three channels: “RED”, “GREEN” and “BLUE”). It makes sense to focus on working with this type of image.

(I) Channels extracted from the processed images and a spatial region corresponding to the medical rehabilitation exoskeleton are preliminary identified. Further image processing is performed using a single, known channel of the additive color model “RGB” in the simplest case. For example, it is advisable to work with the “RED” channel if the medical rehabilitation exoskeleton has a bright red coating. A slightly more complex, but still compromise situation is when the software controlling the exoskeleton system in the medical rehabilitation department allows for the exoskeleton’s color to be associated with a specific patient. This approach is convenient for medical staff and the digital data processing becomes more efficient.

The next step is contrast correction of the processed images. This is achieved by exponentiation (ideally, using an adaptive gradient power transform) and allows for more accurate localization of the region of interest. The processed image is then binarized using a threshold value. This step simultaneously serves two functions: (a) further improves the accuracy of localization of the desired exoskeleton; (b) converts the processed image into a format suitable for morphological processing.

(II) The localized region of interest is generalized by a topological skeleton. It is constructed for each frame using an adapted version of the “quench” method. This method is based on the Bloom transform, but is suitable for constructing topological skeletons for objects with fuzzy boundaries. For each pixel or voxel, p, with a neighborhood $N_8^{\ast }(p)$, an outward normal n_q can be drawn at the point $q\in N_{nk}^{*}(p)$, and the following inequality holds:

$$FDT(q)-FDT(p)<{\displaystyle \frac{1}{2}}\left({\mu }_o(p)+{\mu }_o(q)\right)\left|p-q\right|,$$

(1)

Then, the above-mentioned pixel or voxel, p, inside the digital object, ο, corresponds to the “quench” point:

$$o=\left\{\left(p, {\mu }_o(p)\right)\left|p\in Z^3\wedge {\mu }_o:Z^3\to [0,1]\right.\right\},$$

(2)

where nk is a normalization coefficient equal to 8 in the two-dimensional case and 26 in the three-dimensional case; FDT is a distance transformation based on fuzzy logic (using K. Saha’s algorithm); Z is the set of integers forming the rectangular coordinate grid Z³ (in the two-dimensional case, the index of a power is correspondingly equal to two); ${\mu }_o$ determines the region of interest. The lower the level (in the range from 0 to 1) the higher the probability that a pixel or voxel belongs to the background. A high level corresponds to a high probability of belonging to an object and, therefore, suitability for constructing a topological skeleton. The factor |p − q| should be interpreted as the distance from p to q.

The distinction between significant and false “quench” points is based on the local significance factor, ξ, calculated as follows:

$$\xi (p)=1-\underset{q\in N_{26}^{\ast }}{{\displaystyle \max }}{\displaystyle \frac{f_{+}\left(FDT(q)-FDT(p)\right)}{\frac{1}{2}\left({\mu }_o(p)+{\mu }_o(q)\right)\left|p-q\right|}},$$

(3)

where f₊(x) is the calibration function, respectively, equal to

$$f_{+}(x)=\left\{\begin{array}{l}x,\mathrm{if} x>0\\ 0,\mathrm{otherwise} \end{array}\right..$$

(4)

Calibration function, f₊(x), normalizes the local density around the quench point so that significant ones receive a high ξ, while false quench points receive a low ξ value.

Local significance factor, ξ, varies in the range from 0 to 1 (fuzzy logic) and is necessary to generate a field of significance map:

$$M_p^x(i,j)=\max \left\{\xi (q_{i,j}^{+}),\xi (q_{i,j}),\xi (q_{i,j}^{-})\right\},$$

which is used as 8 (2D case) or 26 (3D case) digital masks to reduce the number of spurious quench pixels or voxels. A quench pixel or voxel is referred to as significant, if any of the average values is greater than a preset threshold.

Then, the significant points of the “quench” are combined into topological skeletons for each image. These skeletons are combinations of control points connected by thin lines. Each point in the topological skeleton is equidistant within its own high-level signal region. The resulting set of equidistant lines is further thickened using an additional morphological dilation operation.

The final version of the topological skeleton of the spatially localized exoskeleton is recoded to ensure its correspondence with the original image version [23]. The topological skeleton of the spatial region corresponding to the exoskeleton is superimposed on the full-size version of the original image and displayed to the end user (the rehabilitation physician).

(III) Localization of the patient’s upper and lower parts of body is performed based on the previously obtained result. Specifically, the known (localized) exoskeleton location region is expanded using morphological dilation to a level that virtually effectively divides the patient’s body into upper and lower parts. The expanded exoskeleton location region is then subtracted from the original image. All that remains is to localize the patient’s upper and lower parts of body, separating them from the background [15]. This is accomplished using operations completely analogous to those performed in Step I. The only exception is the segmentation and identification operation itself (assigning a label parameter).

(IV) The topological skeletons of the patient’s body parts are constructed similarly to the topological skeleton of the exoskeleton. Thus, point IV is almost completely analogous to point II.

(V) Calculating control distances is a routine but crucial task. Correct execution of many physical exercises requires multidimensional control of key joint positions. For example, the shoulder joint has six degrees of freedom. A medical rehabilitation skeleton can guide the patient’s movements, dose the load and balance it among muscle groups, but it obviously does not protect against displacement of key joints from the correct position (prescribed by the rehabilitation physician) to an overloaded position.

Therefore, control points are applied to at least two topological skeletons (the patient’s upper part of body and the exoskeleton). The points are grouped, and key joint displacements are assessed both statically and dynamically. The end user (the rehabilitation physician) has direct access [23] to this data and adjusts the spatial position of the patient’s limbs if his personal medical experience suggests such adjustments are appropriate. This significantly increases the effectiveness of the rehabilitation procedure, because it allows medical staff to quickly identify critical deviations in the implementation of the rehabilitation program.

3. Results and Discussion

The proposed method was implemented using the LabVIEW 2024 (version 24.1f0) software package. A study of this method’s operation should begin with a step-by-step evaluation of its effectiveness. Then, it is advisable to estimate the effectiveness of the proposed solution as a whole.

A typical example of rehabilitation exercises performed using a medical upper-limb exoskeleton is shown in Figure 1a. The patient is shown frontally. An inhomogeneous background around the patient was intentionally used to make the task more difficult. Shadows on the floor were also added [15].

Figure 1b shows the result of selecting the region of interest from the “RED” channel. The channel selection step initially had two options, because exoskeleton is colored yellow (yellow and its shades typically equal zero or a comparable number in the “BLUE” channel). The signal intensity in the “RED” channel was higher than in the “GREEN” channel, so the algorithm selected it as the source of information. Contrast correction was performed by raising to the third power. The threshold level used was 100 points. The localized exoskeleton is shown in Figure 1c. Then, the region of interest was subjected to two iterations of morphological removal of small objects at the edges of the field of view and morphological closing (a combination of sequential morphological dilation and erosion). The resulting topological skeleton for medical exoskeleton is based on fuzzy logic. It was omnidirectionally enhanced using morphological dilation and is shown in Figure 1d.

The result of combining the grayscale version of the original image and the final topological skeleton of the medical exoskeleton is shown in Figure 1e. The second branch of the same method allows subtracting the localized upper-limb exoskeleton from a selected channel of the original image (in this case, the “RED” channel) to obtain and subsequently segment (Figure 2a) the regions of the upper and lower limbs. Their topological skeletons are shown in Figure 2b. The result of superimposing the topological skeletons of the upper and lower parts of patient’s body onto the original grayscale image is shown in Figure 2c.

Figure 3 presents four additional examples of proposed method’s operation for a more detailed analysis of its effectiveness. The exoskeleton, patient and external environment are the same. The key difference is in the viewing angles. The above group of subfigures clearly demonstrate the stable operation of the proposed method in a set of situations that are as close as possible to real-life [6,23].

The calculations for all the examples considered above were performed on a control computer of the exoskeleton-based upper-limb rehabilitation system using proprietary software based on the 64-bit version of “LabVIEW 2024” graphical programming environment. Parallel computing on CUDA cores was not used. Main technical specifications of the control computer of the medical rehabilitation system is as follows: CPU Intel Core i7-9750H; SSD 256 GB; RAM 12 GB.

Figure 4 shows the result of topological skeletonization of the three above-used medical rehabilitation images using third-party software from “Google LLC”. “Media Pipe” (version 0.10.11) suite of libraries and tools quickly and accurately generate topological skeletons, but this approach is clearly not suitable for monitoring the effectiveness of a set of medical rehabilitation exercises. The patient’s body and the exoskeleton are at least partially described as a single unit in each of the three examples given. There are two topological skeletons each in Figure 4a,b, but the second is associated with the patient’s facial region. There are three topological skeletons in Figure 4c, but in addition to the primary skeleton (described by most of the body and the exoskeleton), the second and third ones describe the right hand and the facial region.

MediaPipe’s performance is 231–267 ms for processing a single frame. In video streaming mode, the performance ranges from 20 to 50 ms. The calculation according to our method takes place almost in real time (the calculation of the topological skeleton takes 14–32 ms, the full cycle of all operations, including the output of intermediate results and their display, takes up to 100 ms).

An example of estimating reference angles and distances using a virtual clamp with a measuring rake (a computer vision element) is shown in Figure 5. This figure demonstrates one of the most common training errors in sports and rehabilitation medicine: an incorrect fixation angle at one of the key joints. The left wrist joint in this position is clearly overloaded. Training with this position of the left wrist joint is dangerous to health.

The limitations of the proposed method’s performance are related to the quality of the images under study. In particular, spatial resolution, focusing, viewing angle, local (exoskeleton relative to the patient’s body) and global (patient and exoskeleton relative to the background) contrast are of particular importance. All of these limitations are surmountable, since medical systems are typically designed for use under pre-determined (well-controlled) conditions and undergo fine-tuning during commissioning.

The sensitivity of the method and software varies by settings for thresholding, morphological operations and topological skeletonization. The image thresholding limits are selected based on metrics associated with the high-level signal. Morphological operations are aimed at removing small and boundary objects using empirically determined masks. Topological skeletonization is performed using a field of significance map.

It should be noted that evaluation of most important angles and distances, promptly reporting significant deviations within each session and medical rehabilitation using upper-limb exoskeleton under the supervision of a rehabilitation physician (the person who will be promptly informed if necessary) are the target areas for the effective use of the proposed method. Increased range of motion, motor recovery and reducing fatigue are more likely to be trends over the course of a multi-week or even multi-month medical rehabilitation program.

4. Conclusions

A method for digitally describing the process of medical rehabilitation using an upper-limb exoskeleton [6] has been presented. A key feature of the proposed scientific and technical solution is the use of three topological skeletons. The first of these computationally efficiently describes the spatial position of the upper-limb medical exoskeleton. The second and third describe the patient’s upper and lower parts of body, respectively.

The proposed approach allows the description of each of the three regions of interest (the upper-limb medical exoskeleton and the patient’s upper and lower parts of body) using only a few dozen points [27,28,29]. Construction of the above-mentioned set of topological skeletons is possible in real time, if the correct settings for each of the performed steps (channel extraction, threshold, morphological processing, etc.) were used.

Most importantly, the proposed method not only allows for the computationally efficient description of the spatial geometry of rehabilitation procedures, but also for monitoring of individual aspects of their proper execution. The modern upper-limb medical exoskeleton allows for guidance of the patient’s movements. The load is dosed and distributed between muscle bundles [30,31,32,33]. However, the patient is not simply a spectator during the rehabilitation exercises. The exoskeleton only guides, but does not fully control the movements of the upper limbs [17,34]. The patient could subconsciously reduce the effectiveness of the exercise (by redistributing the load across more muscle bundles) by shifting the arm at the shoulder, elbow and wrist joints. This underloads the muscle fibers and tendons that require recovery. This makes the exercise easier to perform, but does not produce the desired rehabilitation effect. The second scenario is much worse: displacements in the aforementioned joints could lead to their overload. This makes physical exercise detrimental instead of restoring mutilated areas of the musculoskeletal system. Short-term joint overload should not cause problems, but systematic overload could lead to pain and inflammation. In general, therapeutic and preventative exercises are necessary not only to repair the initial mutilate, but also to prevent secondary negative effects. The proposed method helps avoid such situations. Key distances and angles are continuously monitored. The rehabilitation system’s software notifies the rehabilitation physician if a situation similar to the one shown in Figure 4 (left wrist joint overload) occurs. The rehabilitation physician independently assesses the situation and (if necessary) re-instructs the patient on the correct performance of the prescribed rehabilitation exercises.

Thus, the proposed method improves the effectiveness of medical rehabilitation using upper-limb exoskeletons by monitoring potential errors during exercise performance. Prompt notification of potential malposition of major joints during a series of rehabilitation exercises saves medical staff time. Furthermore, a critical response threshold (by time, malposition and load level) is provided. The rehabilitation physician could independently change the preset thresholds to avoid situations where the rehabilitation system becomes oversensitive to deviations in exercise performance [15,35].

The proposed scientific and technical solutions are associated with upper-limb medical exoskeletons, however with relatively minor modifications their range of application could be significantly expanded. They also have significant potential for application in related areas of medical rehabilitation [36,37], such as lower-limb exoskeletons. Adaptation of the proposed method for use in conjunction with a medical rehabilitation exoskeleton for the lower limbs is achieved by its obvious and significant simplification. There will no longer be a need to construct three topological skeletons (one for the upper-limb exoskeleton and one for each of the separately located upper and lower parts of patient body). Two (one for the lower limb exoskeleton and one for the merged central and upper parts of patient body) will be quite sufficient. The control angles and distances will be adapted.

The description of the results of preclinical trials and homologation according to established global and national standards for the production of medical equipment and clinical reporting are not the aim of this scientific publication. However, our future work will be aimed to make the method presented in this paper more adaptive and supplement it with artificial intelligence technologies. Auto setting mode will simplify the commissioning process. Artificial intelligence technologies will help indirectly assess the presence or absence of pain in the patient based on facial expressions. Monitoring changes in the patient’s posture and identifying excessive tension in the neck muscles during medical rehabilitation procedures is also practically useful for real clinical applications.