Video-Based Deep Learning Approach for 3D Human Movement Analysis in Institutional Hallways: A Smart Hallway

Abstract

New artificial intelligence- (AI) based marker-less motion capture models provide a basis for quantitative movement analysis within healthcare and eldercare institutions, increasing clinician access to quantitative movement data and improving decision making. This research modelled, simulated, designed, and implemented a novel marker-less AI motion-analysis approach for institutional hallways, a Smart Hallway. Computer simulations were used to develop a system configuration with four ceiling-mounted cameras. After implementing camera synchronization and calibration methods, OpenPose was used to generate body keypoints for each frame. OpenPose BODY25 generated 2D keypoints, and 3D keypoints were calculated and postprocessed to extract outcome measures. The system was validated by comparing ground-truth body-segment length measurements to calculated body-segment lengths and ground-truth foot events to foot events detected using the system. Body-segment length measurements were within 1.56 (SD = 2.77) cm and foot-event detection was within four frames (67 ms), with an absolute error of three frames (50 ms) from ground-truth foot event labels. This Smart Hallway delivers stride parameters, limb angles, and limb measurements to aid in clinical decision making, providing relevant information without user intervention for data extraction, thereby increasing access to high-quality gait analysis for healthcare and eldercare institutions.

1. Introduction

Motion analysis provides information and insights into the quality of movement for rehabilitation, performance analysis of professional athletes, and animation for video games or computer-generated imagery in movies. Of particular interest is improving the quality of information provided to healthcare professionals. Stride analysis and gait information are used in clinical decision making to optimally care for patients. Human motion analyses can aid in understanding rehabilitation progress [1], fall risk [1,2], progression of neurodegenerative diseases [3], and classifying gait patterns [4,5,6]. However, equipment, access, space, and human-resource requirements limit quantitative movement assessment within healthcare and eldercare environments. A Smart Hallway implementation could automatically record movement as a person walks through a hallway within an institution so that a therapist or physician can review walking parameters before their appointment. In an eldercare residence, movement data could be collected multiple times a day, thereby providing data to track changes in movement quality. Big data models could also be implemented to provide indicators for fall risk or dementia progression. The Smart Hallway design enables non-invasive data collection that would not interfere with existing hospital processes. These new capabilities rely on automated movement-data acquisition without human intervention.

A depth-camera approach for a Smart Hallway was created and validated by Gutta et al. [7]. Multiple Intel RealSense depth cameras were used to generate a point cloud of the person’s lower leg, processing the point-cloud-generated stride parameter outcome measures, which can be used for clinical decision making. Limitations of camera-person distance for accurate body digitization restricted the camera positioning to waist height along the hallway walls. This setup could lead to issues with camera obstruction and people hitting the cameras as they traverse the hallway in daily use.

A marker-less approach to human movement analysis could also utilize an array of RGB cameras, paired with an artificial intelligence model, to determine two-dimensional (2D) coordinates of a person’s major joints. The set of joint coordinates can then be used as point correspondences to create a three-dimensional (3D) skeleton reconstruction of the person. With these data, stride parameters and other biomechanical measurements can be extracted and used for clinical decision making [5,8]. Since data are extracted from video without requiring patient preparation or other interventions, this approach could eliminate many of the practical obstacles to implementing movement analysis in healthcare workflows.

Various marker-less motion-analysis systems have been reported. Labuguen et al. [9] implemented a four-camera system capturing 720p resolution and 30 frames per second (fps) to reconstruct 3D joint positions. OpenPose BODY25 was solely used to detect keypoints, and no tracking, filtering, or foot-event detection was implemented for full movement analysis. Results from this approach had a maximum average error of 7 cm on joint position. This system was also limited in its ability to track high-speed movements due to the low capture rate. Other methods implementing Microsoft Kinect^® sensors are limited in scalability due to each sensor requiring its own computer for processing [10]. Furthermore, depth-sensor approaches have significant noise when assessing foot-contact events and suffer from increasing depth-estimation errors at long ranges [7]. Due to the useable range of Kinect sensors, they typically must be kept close to the participant, as in the system implemented by Rodrigues et al. [10], where sensors were placed at waist level within two to three meters of the participant. This system utilized an array of Kinect sensors calibrated to each participant and synchronized at 35 fps. This method of calibration is too cumbersome to deploy for daily use and is therefore not adaptable to a Smart Hallway-type application. Methods that implement RGB camera arrays with pose-inference models have shown promising results for human motion analysis, such as the system implemented by Nakano et al. [11]. However, they did not implement proper synchronization methods or state-of-the-art calibration approaches needed to develop a useable system capable of producing semi-real-time results. Moreover, the Nakano et al. system does not automatically detect foot events and requires post-analysis of the data to segment strides, which is not useable for the Smart Hallway proposed in this paper. Overall, these existing marker-less motion-analysis approaches included some but not all the elements necessary for an automated Smart Hallway application.

Table 1. Comparison of the Smart Hallway to existing marker-less motion-capture systems.

System	Resolution (Pixels)	Framerate (fps)	Working Distance (m)	Calibration	Synchronization
Labuguen et al., 2020 [9]	1280 × 720	30	3	Calibrated to participant	Per-trial manual post-processing
Rodrigues et al., 2019 [10]	640 × 480	35	3	Calibrated to participant	Time-stamping algorithm
Nakano et al., 2020 [11]	1920 × 1080	120	4	Per-trial, in region of interest	Per-trial manual post-processing
Tamura et al., 2020 [12]	640 × 480	30	Not reported	None, only relative measures	Single camera
Stenum et al., 2021 [13]	960 × 540	25	3.3	Not reported	Automatic (hardware)
Albert et al., 2020 [14]	3840 × 540	30	3.5	None, single-depth sensor	Per-trial manual post-processing
Pasinetti et al., 2020 [15]	640 × 480	30	3	Performed once	Time-stamping algorithm
Smart Hallway (ours)	1440 × 1080	60–120	7.5	Performed once	Automatic (Hardware)

details other approaches to the non-invasive motion-analysis problem. System error is not reported, as no standardized measure is used across all studies.

The goal of this research is to design, develop, and evaluate a marker-less motion-analysis system that provides movement-outcome measures for institutional hallway settings. The system must be non-invasive in its inherent design and provide a modular approach that is optimized and can be deployed in any institutional hallway setting. The system should also improve on past marker-less systems by providing a sufficient capture rate, a robust synchronization and calibration approach, and a method to automatically detect foot events and return common outcome measures. By implementing in an institutional hallway, the system would be accessible and enable movement analysis to be integrated into daily schedules. The ultimate goal for a “Smart Hallway” is to accurately and non-invasively assess and report a person’s human-movement status in an institutional setting, with minimal or no human intervention.

2. Materials and Methods

Motion-capture systems require a variety of components to work optimally in tandem. This research includes computer simulations to determine optimal camera layout, temporospatial synchronization validation, calibration validation, and evaluation with various walking scenarios. Appendix A, Appendix B and Appendix C provide details of optimal camera layout, temporospatial synchronization validation, calibration validation, and validation methods used to design the Smart Hallway.

Based on preliminary research [16], the open-source OpenPose BODY25 model was used for all body keypoint inferences. The OpenPose model was trained on a combination of the COCO and MPII pose datasets. OpenPose BODY25 produced accurate keypoint results from preliminary testing on clinically relevant movements [16]. These 2-dimensional (2D) points combine to create a skeleton model of the person of interest (Figure 1).

2.1. System Design Requirements

The Smart Hallway’s goal is to provide a non-invasive approach to extract gait outcome measures without human intervention. To effectively incorporate the system into hospital processes, the system must not interfere with individuals moving through the hallway [17]. Thus, typical hospital hallway dimensions were considered (length × 2.4 m × 2.8 m) when determining the placement of system components. The system components (cameras, cables, high-performance computing unit) should be mountable on the ceiling or high enough from the ground to not interfere with carts or people passing through. To extract data for 3D reconstruction using triangulation, at least two cameras are needed [18,19,20]. Increasing the number of cameras improves 3D reconstruction accuracy; this is a function of the camera-view overlap and number of detections of the point of interest, which can be passed to the optimization method. Based on simulation results, four cameras were used [8,21]. Other factors relating to 3D reconstruction accuracy include the camera resolution and synchronization. For the most accurate keypoint placement from OpenPose BODY25, the target must be at least 300 pixels tall in the camera frame [16,22]. Camera resolution is also dependent on the target-capture volume and lens specifications.

Camera synchronization is paramount for accurate reconstruction; having all cameras capture images at the same time reduces 3D reconstruction error. For this level of synchronization, a hardware approach with a stable sync signal is desirable. The system framerate is dependent on the type of motion being analyzed. For normal walking, a framerate of 60 fps is sufficient to reconstruct movement and extract useful outcome measures [23,24,25].

An accurate calibration routine for the camera array is required to extract accurate 3D information from the marker-less keypoints. Each camera’s projection matrix relative to the world origin contains variables for lens distortion, intrinsic parameters, and extrinsic parameters. These parameters are normally determined using a patterned calibration object and techniques such as Zhang’s method with random sample consensus (RANSAC) and bundle adjustment for camera extrinsic parameters [26,27,28]. These parameters should be calibrated such that the reprojection error is less than 1.0 pixel; however, this is dependent on camera resolution.

2.2. System Design

3D simulations were performed to determine the volumetric coverage achievable with four and eight cameras, respectively, within a hallway scenario. The simulated hallway was modelled as 5 m × 2.4 m × 2.8 m based on measurements from a typical hospital hallway. Selected components were modelled using Blender’s (Blender Foundation, Blender 2.91) camera object, and several iterations were performed while varying camera pose and placement relative to the world origin. The various configurations were compared based on parameters relating to capture volume that each setup produced (i.e., total capture volume, ground-area coverage, and view overlap). An array of four FLIR BlackFly^® S USB3 (BFS-U3-16S2C-CS) machine-vision cameras with Fujinon 3 MP Varifocal Lenses (YV4.3X2.8SA-2) were selected based on simulations. Figure 2 shows the virtual Smart Hallway camera layout, providing a 5 m × 2.4 m × 2.8 m (29 m³) capture volume with four cameras in an arc layout. Appendix A provides a detailed explanation of the simulation methods used.

System components were selected based on geometric and data-transfer constraints. Geometric constraints were based on the institutional hallway simulations and the maximum cable lengths for each communication standard (USB, GiGE, etc.). Data-transfer constraints were based on the desired multi-camera system performance in terms of resolution (minimum 960 × 720), pixel format (minimum 8 bit colour depth), and frame capture rate (minimum 60 fps). The selected components that best addressed the Smart Hallway requirements are detailed in

Table 2. Selected components based on computer simulations, geometric and data-transfer constraints, and desired system accuracy.

Component	Name	Description	Data Handling
Cameras	FLIR BlackFly S USB3 (BFS-U3-16S2C-CS)	Resolution: 1440 × 1080 Frame rate: 1–226 fps	Output: 280 MB/s (USB) Cameras: 4
Data-Transfer Cables	USB-A with active extension cable	Length: 3 m + 5 m Format: 1 × USB3.0	Bandwidth: 625 MB/s (USB) Cables: 4
PCIe Card	StarTech (PEXUSB3S44V)	Format: 4 × USB3.0, 1 × PCIe x4 Gen 2.0	Bandwidth: 4 × 625 MB/s (USB), 4 GB/s (PCIe)
HPC Unit	NVIDIA Jetson AGX Xavier	Format: 1 × PCIe x8 Gen 4.0	Bandwidth: 16 GB/s (PCIe)
Data Storage	Samsung 970 EVO Plus (MZ-V7S500/AM)	Format: NVME M.2	Read/Write: 3.5 GB/s

.

A hardware synchronization cable was designed and created to ensure reliable image capture for the multi-camera system. A primary camera sends a sync signal at the beginning of exposure to the other cameras in the array, and the secondary cameras begin exposure once the sync signal has been received. This synchronization approach was validated by capturing 10,000 images and comparing the timestamps produced by each FLIR camera. The cameras remained synchronized within 5 μs of the primary camera when capturing at 60 fps (16,667 μs/image). This solution provides a repeatable synchronization method without the need for synchronization post data capture. Detailed cable design and validation methods are given in Appendix B for reproducibility.

Spatial synchronization for the multi-camera system was accomplished with a ChArUCo calibration pattern. Calibration was performed by capturing several views of the ChArUCo board and implementing OpenCV and Ceres libraries for robust camera parameter calculation. Final reprojection error from the distortion, intrinsic, and extrinsic parameters was less than 1.0 pixels. The multi-camera system calibration was tested by comparing system output to measured dimensions along the length of the capture volume (5 markers on the floor spaced at 1 m intervals). X-axis error was 1.7 (SD = 1.2) cm, Y-axis error was 2.4 (SD = 1.5) cm, and Z-axis error was 1.9 (SD = 1.4) cm. This solution allows for one-time system calibration that does not need to be performed prior to each data-collection session. The hardware pipeline is highlighted in Figure 3. Details of the calibration approach are given in Appendix C.

2.3. Signal Processing

Videos of participants were recorded and stored on the NVIDIA Jetson AGX’s solid-state drive. The videos were then passed to the OpenPose BODY25 model to perform inference and create a set of 2D keypoints, locating participant joint centres for every video frame. For every video, the 2D keypoint data contains confidence scores that describe the likelihood of correct marker location. Data from each video were preprocessed by removing points below 10% confidence and using a cubic spline to interpolate gaps in the dataset that are five frames (0.083 s) or less. The dataset was then filtered using a zero-phase low-pass 12 Hz Butterworth filter. Figure 4 shows an example of the 2D keypoint data after preprocessing.

2D keypoint data from each camera were passed to the triangulation pipeline, along with the intrinsic and extrinsic parameters. Point correspondences from the 2D keypoints were used in a non-linear optimization RANSAC triangulation method to determine an optimal set of 3D keypoints describing the body at each timestep in the video. For each trial’s 3D data, regions where 3D keypoint data reprojection error exceed two standard deviations from the mean were removed to reduce outlier effects.

Software was written in Python 3.7 to calculate body-segment lengths, stride parameters, and hip, knee, and ankle angles. 3D data were filtered using a zero-phase low-pass 5 Hz Butterworth filter, based on findings from other research involving OpenPose keypoint inferences and marker-based approaches [25,29].

Body-segment lengths were calculated using the Euclidean distance between limb endpoints. Body-segment lengths were measured at each timestep in the video. Measurements outside two standard deviations from the mean were identified as outliers and removed. For evaluation, body-segment length deltas were calculated as limb length subtracted from the measured limb length.

Ground-truth stride parameters were calculated from ground-truth foot events obtained by manually labelling foot offs and foot strikes in each video. Detected foot events were obtained by using the Zeni et al. algorithm [30]. The set of detected foot events was improved by implementing an algorithm to recover gait initiation and gait termination (initiation termination recovery, it recovery). Figure 5 shows an example of the detected foot events prior to the initiation and termination recovery algorithm.

Regions such as the one highlighted by the red ellipse in Figure 5 were recovered by searching the window between the stop region (red area) and the next detected foot event. The algorithm determined whether an initiation occurred by analyzing the linear fit of the curve in a calculated search region. The search region was assessed by detecting a potential foot event, using SciPy’s signal.find_peaks function, and fitting a line between the potential foot event and the next detected foot event [31]. Linear fits above an R-squared value of 0.85 were selected as gait initiations and added to the list of detected foot events. Foot events that were missed or not recovered by Zeni’s algorithm were backfilled using the algorithm proposed by Capela, Lemaire, and Baddour [32,33]. Algorithm 1 and Algorithm 2 describe the methods implemented by this research to detect foot events.

Algorithms 1 Method for detecting stops in a trial given an array of chest keypoint position per frame. Detect Stops.
0	Get first derivative of keypoints in chest keypoint array, AC, smooth the array with a low-pass filter
1	get net average velocity of keypoints in the array VN
2	set current stopped state to False
3	for index, point in AC
4	create a sliding window on AC of size K
5	get average velocity of the window VW
6	if VW less than VN * scalar and current state is not stopped
7	current stop append frame index
8	stopped state to True
9	else if VW greater than VN * scalar and current state is stopped
10	current stop append frame index → S
11	S appended to SN, current stop S set to empty list
12	stopped state to False
13	else pass
14	end
15	Return list of stops SN

Algorithm 2 Method for detecting foot strikes during gait initiation and gait termination. Detect Foot Strikes.
0	create an empty event displacement array E ← [empty]
1	for index, point in foot/heel keypoint array, AK
2	get heel keypoint displacement relative to the bottom chest keypoint
3	append the displacement data to E
4	smooth E with a low-pass filter
5	fine detect initial peaks in E as foot strikes FS
6	pass E, FS, and list of stop windows WL to initiation/termination recovery method
7	for W in WL
8	coarse detect peaks from E index 0 to start index of W
9	select the last detected peak as a potential gait termination strike PS
10	create window between the last detected strike in FS r and PS
11	if last detected strike is equal to potential gait termination
12	Return is termination False
13	else
14	check concavity of the displacement data inside the newly constructed window W
15	construct a line L from the start to the end of W
16	determine the linear fit between the data inside W and L
17	if linear fit is less than threshold
18	Return is termination False
19	Return is termination True → PS inserted in FS

Processing time for foot-event detection and stride-parameter measurement was calculated using Python 3′s built in nanosecond clock. Foot-event detection took, on average, 18 ms for a 1500 frame trial. Stride-parameter measurement took an average of 35 ms for calculating 30 parameters across a 1500 frame trial.

Stride parameters included stride length, stride time, stride speed, step length, step width, step time, cadence, stance time, swing time, stance swing ratio, and double support time. Results for comparison included mean ($\mu$) and standard deviation ($\sigma$) across all trials of the same walking condition for each participant.

Hip, knee, and ankle 3D angles for the left and right legs were calculated for each stride, defined by ground truth and detected foot events. Figure 6 shows how the vectors used in the angle calculations were defined. Hip angle was the angle between the torso vector and thigh vector, knee angle was the inner angle between the thigh and shank vector, and ankle angle was the inner angle between the shank and foot vector.

Figure 7 shows the final software pipeline of the Smart Hallway.

2.4. Validation

The Smart Hallway system was evaluated by testing two male participants (age: 28; height: 180 cm; weight: 64 kg and age: 25; height: 178 cm; weight: 90 kg). Each participant provided informed consent (University of Ottawa Ethics Board, H-01-21-5819). Data collection was completed in one testing session.

Prior to testing, each participant’s body-segment lengths were measured using an anthropometric tape. The segments matched OpenPose’s BODY25 model (Figure 1) and were measured by palpating the joint centres of interest for each measurement. Each participant completed five separate trials of five walking conditions (

Table 3. Testing protocol for the Smart Hallway validation.

Condition	Protocol
Walking straight	Start one meter outside the capture volume and walk straight. Once through the capture volume, turn around and walk back to the initial position. The turn occurs outside of the camera field of view.
Walking and turning	Start at the edge of the capture volume and walk towards a marker positioned 50 cm from the end of the capture volume. Turn around the marker and walk back to the initial position. The turn occurs within the camera field of view.
Walking in a curved path	Start at the edge of the capture volume and walk in a curved path around the capture volume. The test ends once the participant reaches their initial position. The participant performs each test in the same direction.
Walking with a cane	Follow the “walking straight” protocol while using a cane as a walking aid. Cane held in the same hand for all trials. Participants were instructed on how to properly use a cane
Walking with a walker	Follow the “walking straight” protocol while using a wheeled walker as a walking aid. Participants were instructed on how to properly use a walker.

). Participants were recorded with the four-camera array at 60 fps. A total of 50 videos were recorded, containing approximately 500 foot events.

3. Results

From

Table 4. Body-segment lengths for participant one: walking straight, walking turn, and walking curve test conditions (mean and standard deviation in brackets). Smart Hallway (SH) values were calculated using the 3D reconstructed data, and Delta is the difference between the Smart Hallway and ground-truth segment lengths.

Walking Condition	Walking Straight		Walking Turn		Walking Curve
Limb Segment (cm)	SH	Delta	SH	Delta	SH	Delta
Left Arm	29.60 (1.92)	0.10 (1.92)	28.49 (1.63)	−1.01 (1.63)	28.75 (1.33)	−0.75 (1.33)
Left Forearm	25.37 (2.41)	−1.13 (2.41)	25.75 (1.53)	−0.75 (1.53)	25.68 (1.15)	−0.82 (1.15)
Right Arm	29.07 (1.93)	0.07 (1.93)	28.74 (1.65)	−0.26 (1.65)	28.88 (1.24)	−0.12 (1.24)
Right Forearm	24.91 (2.76)	−2.09 (2.76)	25.24 (2.12)	−1.76 (2.12)	25.18 (1.51)	−1.82 (1.51)
Left Thigh	38.29 (2.51)	−3.21 (2.51)	38.45 (2.59)	−3.05 (2.59)	38.05 (2.26)	−3.45 (2.26)
Left Shank	39.01 (3.94)	−0.99 (3.94)	39.62 (4.11)	−0.38 (4.11)	39.38 (3.17)	−0.62 (3.17)
Right Thigh	37.88 (2.29)	−3.12 (2.29)	37.93 (1.93)	−3.07 (1.93)	37.30 (2.00)	−3.70 (2.00)
Right Shank	38.51 (3.15)	−1.49 (3.15)	39.17 (3.84)	−0.83 (3.84)	38.78 (3.13)	−1.22 (3.13)
Left Ankle to Heel	6.91 (2.51)	−0.09 (2.51)	7.50 (2.81)	0.50 (2.81)	6.76 (1.73)	−0.24 (1.73)
Left Ankle to Big Toe	17.16 (5.46)	−0.84 (5.46)	18.12 (4.40)	0.12 (4.40)	16.75 (2.88)	−1.25 (2.88)
Left Ankle to Small Toe	14.37 (4.49)	−2.63 (4.49)	15.29 (4.30)	−1.71 (4.30)	13.90 (2.60)	−3.10 (2.60)
Left Toe Width	7.40 (2.37)	−0.10 (2.37)	7.72 (2.96)	0.22 (2.96)	6.76 (1.86)	−0.74 (1.86)
Right Ankle to Heel	7.76 (2.90)	0.26 (2.90)	7.09 (2.68)	−0.41 (2.68)	6.87 (1.90)	−0.63 (1.90)
Right Ankle to Big Toe	15.63 (4.65)	−2.87 (4.65)	16.91 (5.09)	−1.59 (5.09)	16.21 (3.08)	−2.29 (3.08)
Right Ankle to Small Toe	12.87 (4.01)	−4.63 (4.01)	14.39 (3.22)	−3.11 (3.22)	13.96 (2.46)	−3.54 (2.46)
Right Toe Width	8.85 (3.51)	1.35 (3.51)	7.73 (2.40)	0.23 (2.40)	7.08 (2.62)	−0.42 (2.62)
Shoulder Width	35.33 (1.77)	−0.67 (1.77)	35.24 (1.35)	−0.76 (1.35)	34.62 (2.09)	−1.38 (2.09)
Hip Width	22.45 (1.33)	−1.05 (1.33)	22.25 (1.33)	−1.25 (1.33)	22.15 (1.70)	−1.35 (1.70)
Chest Height	53.91 (1.93)	−1.09 (1.93)	53.57 (1.45)	−1.43 (1.45)	54.27 (1.24)	−0.73 (1.24)

and

Table 5. Body-segment lengths for participant one: walking straight, cane, and walker test conditions (mean and standard deviation in brackets). Smart Hallway (SH) values were calculated using the 3D reconstructed data, and Delta is the difference between the Smart Hallway and ground-truth segment length.

Walking Condition	Walking Straight		Cane		Walker
Limb Segment (cm)	SH	Delta	SH	Delta	SH	Delta
Left Arm	29.60 (1.92)	0.10 (1.92)	29.36 (1.18)	−0.14 (1.18)	28.82 (1.64)	−0.68 (1.64)
Left Forearm	25.37 (2.41)	−1.13 (2.41)	25.62 (1.27)	−0.88 (1.27)	30.79 (7.50)	4.29 (7.50)
Right Arm	29.07 (1.93)	0.07 (1.93)	28.59 (1.66)	−0.41 (1.66)	28.61 (1.86)	−0.39 (1.86)
Right Forearm	24.91 (2.76)	−2.09 (2.76)	26.20 (3.92)	−0.80 (3.92)	30.57 (7.29)	3.57 (7.29)
Left Thigh	38.29 (2.51)	−3.21 (2.51)	38.19 (2.33)	−3.31 (2.33)	39.32 (2.90)	−2.18 (2.90)
Left Shank	39.01 (3.94)	−0.99 (3.94)	39.02 (2.75)	−0.98 (2.75)	40.29 (3.65)	0.29 (3.65)
Right Thigh	37.88 (2.29)	−3.12 (2.29)	38.20 (2.45)	−2.80 (2.45)	38.90 (3.19)	−2.10 (3.19)
Right Shank	38.51 (3.15)	−1.49 (3.15)	38.85 (3.04)	−1.15 (3.04)	40.47 (4.07)	0.47 (4.07)
Left Ankle to Heel	6.91 (2.51)	−0.09 (2.51)	7.17 (2.66)	0.17 (2.66)	6.87 (2.39)	−0.13 (2.39)
Left Ankle to Big Toe	17.16 (5.46)	−0.84 (5.46)	16.65 (5.02)	−1.35 (5.02)	14.89 (4.47)	−3.11 (4.47)
Left Ankle to Small Toe	14.37 (4.49)	−2.63 (4.49)	14.27 (3.82)	−2.73 (3.82)	12.34 (3.90)	−4.66 (3.90)
Left Toe Width	7.40 (2.37)	−0.10 (2.37)	7.29 (2.36)	−0.21 (2.36)	7.52 (2.41)	0.02 (2.41)
Right Ankle to Heel	7.76 (2.90)	0.26 (2.90)	6.51 (2.56)	−0.99 (2.56)	5.94 (1.96)	−1.56 (1.96)
Right Ankle to Big Toe	15.63 (4.65)	−2.87 (4.65)	15.10 (5.25)	−3.40 (5.25)	15.36 (4.07)	−3.14 (4.07)
Right Ankle to Small Toe	12.87 (4.01)	−4.63 (4.01)	12.39 (3.96)	−5.11 (3.96)	13.34 (3.75)	−4.16 (3.75)
Right Toe Width	8.85 (3.51)	1.35 (3.51)	7.05 (2.67)	−0.45 (2.67)	7.19 (2.02)	−0.31 (2.02)
Shoulder Width	35.33 (1.77)	−0.67 (1.77)	34.99 (1.26)	−1.01 (1.26)	35.33 (1.22)	−0.67 (1.22)
Hip Width	22.45 (1.33)	−1.05 (1.33)	22.21 (0.93)	−1.29 (0.93)	22.05 (0.96)	−1.45 (0.96)
Chest Height	53.91 (1.93)	−1.09 (1.93)	54.34 (1.26)	−0.66 (1.26)	54.45 (1.47)	−0.55 (1.47)

, mean differences between calculated and ground-truth values were small for the majority of body-segment lengths. In general, the calculated body-segment lengths were less than the ground-truth values. The average difference across all test conditions was 1.56 (SD = 2.77) cm. Since the delta results for participant one and participant two were similar, only the results for participant one were included in this manuscript. Results for participant two are located in Appendix D.

Table 6. Detected foot events frame offset from ground-truth values (mean and standard deviation in brackets) across both participants. The percentage of events detected using Zeni [30] and IT recovery algorithms in combination is shown alongside the percentage of events detected using all the proposed foot-event detection methods.

Condition	Offset (Frames, μ(σ))	Zeni and IT Recovery (%)	Zeni, IT Recovery, and Capela (%)
Walking Straight	4.38 (2.72)	89.2	98.2
Walking Turn	4.13 (2.60)	83.0	98.6
Walking Curve	3.99 (3.42)	84.3	97.4
Cane	3.01 (2.52)	85.5	97.7
Walker	5.25 (3.57)	88.9	98.7
Average	4.15 (2.97)	86.1	98.1

shows the mean absolute error of the foot-event detection algorithms compared to ground-truth values obtained from manual labelling. The error in ground-truth labelling was three frames, and the average error in detected events across all trials was four frames.

Table 7. Stride parameters for participant one from the Smart Hallway (SH) compared to ground-truth foot-event stride parameters for the walking-straight test (mean and standard deviation in brackets).

	Left			Right
Parameters (Units)	SH	Ground Truth	Delta	SH	Ground Truth	Delta
Stride length (m)	1.60 (0.26)	1.62 (0.26)	0.03 (0.07)	1.63 (0.21)	1.63 (0.19)	−0.01 (0.08)
Stride time (s)	1.18 (0.05)	1.19 (0.05)	0.00 (0.05)	1.19 (0.06)	1.19 (0.07)	0.00 (0.04)
Stride speed (m/s)	1.34 (0.22)	1.36 (0.21)	0.02 (0.03)	1.38 (0.16)	1.37 (0.14)	0.00 (0.06)
Step length (m)	0.50 (0.19)	0.38 (0.16)	−0.13 (0.06)	0.44 (0.19)	0.36 (0.16)	−0.08 (0.06)
Step width (m)	0.09 (0.03)	0.11 (0.02)	0.02 (0.01)	0.12 (0.04)	0.13 (0.03)	0.01 (0.02)
Step time (s)	0.63 (0.10)	0.61 (0.08)	−0.02 (0.06)	0.59 (0.06)	0.60 (0.08)	0.01 (0.05)
Cadence (steps/min)	98.13 (4.54)	99.73 (3.29)	1.60 (4.87)	102.41 (6.30)	101.45 (8.68)	−0.95 (4.60)
Stance time (s)	0.73 (0.04)	0.78 (0.04)	0.05 (0.05)	0.72 (0.06)	0.77 (0.05)	0.05 (0.07)
Swing time (s)	0.46 (0.10)	0.40 (0.04)	−0.06 (0.10)	0.50 (0.11)	0.43 (0.03)	−0.08 (0.10)
Stance swing ratio (NA)	1.61 (0.35)	1.94 (0.26)	0.33 (0.36)	1.44 (0.22)	1.82 (0.20)	0.38 (0.34)
Double support time (s)	0.13 (0.10)	0.18 (0.03)	0.05 (0.11)	0.13 (0.03)	0.18 (0.03)	0.05 (0.05)
Foot angle (°)	12.82 (4.86)	12.94 (4.70)	0.12 (1.04)	14.24 (6.14)	14.57 (7.30)	0.34 (2.33)

,

Table 8. Stride parameters for participant one from the Smart Hallway (SH) compared to ground-truth foot-event stride parameters for the walking-turn test (mean and standard deviation in brackets).

	Left			Right
Parameters (Units)	SH	Ground Truth	Delta	SH	Ground Truth	Delta
Stride length (m)	1.32 (0.36)	1.33 (0.39)	0.01 (0.13)	1.41 (0.35)	1.40 (0.37)	−0.01 (0.14)
Stride time (s)	1.18 (0.09)	1.19 (0.06)	0.00 (0.08)	1.24 (0.10)	1.21 (0.08)	−0.03 (0.09)
Stride speed (m/s)	1.12 (0.30)	1.11 (0.32)	−0.01 (0.10)	1.15 (0.35)	1.17 (0.35)	0.02 (0.06)
Step length (m)	0.41 (0.18)	0.33 (0.15)	−0.08 (0.07)	0.40 (0.17)	0.32 (0.14)	−0.08 (0.07)
Step width (m)	0.12 (0.05)	0.13 (0.04)	0.01 (0.02)	0.12 (0.05)	0.12 (0.04)	0.01 (0.01)
Step time (s)	0.61 (0.13)	0.59 (0.12)	−0.01 (0.08)	0.68 (0.25)	0.66 (0.22)	−0.02 (0.08)
Cadence (steps/min)	98.24 (8.45)	102.36 (12.35)	4.13 (7.25)	90.53 (10.52)	92.61 (10.40)	2.07 (2.60)
Stance time (s)	0.73 (0.09)	0.78 (0.06)	0.05 (0.07)	0.74 (0.10)	0.78 (0.08)	0.04 (0.07)
Swing time (s)	0.48 (0.07)	0.43 (0.05)	−0.06 (0.08)	0.50 (0.08)	0.43 (0.06)	−0.08 (0.09)
Stance swing ratio	1.57 (0.34)	1.81 (0.29)	0.25 (0.40)	1.47 (0.22)	1.79 (0.22)	0.32 (0.30)
Double support time (s)	0.13 (0.07)	0.17 (0.04)	0.04 (0.06)	0.13 (0.06)	0.17 (0.04)	0.04 (0.05)
Foot angle (°)	13.06 (4.92)	13.41 (4.93)	0.35 (0.91)	13.81 (6.71)	14.54 (7.37)	0.72 (1.52)

,

Table 9. Stride parameters for participant one from the Smart Hallway (SH) compared to ground-truth foot-event stride parameters for the walking-curve test (mean and standard deviation in brackets).

	Left			Right
Parameters (Units)	SH	Ground Truth	Delta	SH	Ground Truth	Delta
Stride length (m)	1.29 (0.33)	1.31 (0.24)	0.02 (0.16)	1.37 (0.27)	1.39 (0.25)	0.01 (0.12)
Stride time (s)	1.21 (0.13)	1.23 (0.05)	0.02 (0.10)	1.25 (0.09)	1.25 (0.06)	0.00 (0.06)
Stride speed (m/s)	1.05 (0.23)	1.06 (0.18)	0.01 (0.08)	1.11 (0.13)	1.12 (0.13)	0.00 (0.03)
Step length (m)	0.38 (0.18)	0.30 (0.13)	−0.08 (0.08)	0.41 (0.15)	0.32 (0.13)	−0.08 (0.06)
Step width (m)	0.29 (0.16)	0.24 (0.17)	−0.04 (0.09)	0.30 (0.18)	0.27 (0.15)	−0.04 (0.04)
Step time (s)	0.61 (0.16)	0.63 (0.10)	0.02 (0.10)	0.65 (0.12)	0.64 (0.12)	−0.01 (0.05)
Cadence (steps/min)	98.18 (7.68)	96.63 (6.01)	−1.55 (5.31)	93.06 (8.22)	93.80 (7.32)	0.75 (4.52)
Stance time (s)	0.76 (0.11)	0.83 (0.07)	0.07 (0.08)	0.78 (0.10)	0.84 (0.04)	0.06 (0.07)
Swing time (s)	0.45 (0.08)	0.40 (0.04)	−0.06 (0.07)	0.47 (0.07)	0.41 (0.04)	−0.06 (0.10)
Stance swing ratio	1.70 (0.29)	2.08 (0.20)	0.38 (0.32)	1.70 (0.28)	2.04 (0.24)	0.34 (0.27)
Double support time (s)	0.17 (0.07)	0.21 (0.03)	0.04 (0.08)	0.18 (0.06)	0.22 (0.03)	0.05 (0.06)
Foot angle (°)	32.09 (7.04)	32.59 (6.77)	0.50 (3.81)	36.83(11.21)	37.93 (11.49)	1.10 (2.41)

,

Table 10. Stride parameters for participant one from the Smart Hallway (SH) compared to ground-truth foot-event stride parameters for the Cane test (mean and standard deviation in brackets).

	Left			Right
Parameters (Units)	SH	Ground Truth	Delta	SH	Ground Truth	Delta
Stride length (m)	1.34 (0.18)	1.32 (0.16)	−0.01 (0.08)	1.31 (0.20)	1.31 (0.21)	0.00 (0.05)
Stride time (s)	1.90 (0.16)	1.88 (0.11)	−0.02 (0.10)	1.86 (0.13)	1.86 (0.13)	0.00 (0.06)
Stride speed (m/s)	0.71 (0.10)	0.71 (0.09)	0.00 (0.02)	0.73 (0.07)	0.73 (0.07)	0.00 (0.02)
Step length (m)	0.42 (0.15)	0.37 (0.14)	−0.05 (0.03)	0.43 (0.14)	0.41 (0.14)	−0.02 (0.03)
Step width (m)	0.13 (0.03)	0.14 (0.03)	0.01 (0.01)	0.14 (0.03)	0.14 (0.03)	0.00 (0.01)
Step time (s)	0.95 (0.18)	0.90 (0.19)	−0.05 (0.06)	0.96 (0.17)	1.00 (0.16)	0.04 (0.07)
Cadence (steps/min)	63.72 (6.00)	67.25 (7.59)	3.53 (2.38)	62.58 (4.62)	60.15 (3.65)	−2.43 (2.13)
Stance time (s)	1.21 (0.13)	1.31 (0.10)	0.09 (0.08)	1.15 (0.10)	1.18 (0.10)	0.03 (0.06)
Swing time (s)	0.66 (0.10)	0.56 (0.06)	−0.11 (0.08)	0.70 (0.08)	0.67 (0.09)	−0.04 (0.07)
Stance swing ratio	1.85 (0.37)	2.28 (0.34)	0.43 (0.26)	1.62 (0.22)	1.75 (0.31)	0.14 (0.24)
Double support time (s)	0.28 (0.10)	0.35 (0.06)	0.07 (0.07)	0.22 (0.06)	0.27 (0.07)	0.05 (0.06)
Foot angle (°)	11.86 (3.51)	12.11 (3.48)	0.25 (0.79)	12.44 (5.74)	12.55 (5.69)	0.12 (0.65)

and

Table 11. Stride parameters for participant one from the Smart Hallway (SH) compared to ground-truth foot-event stride parameters for the walker test (mean and standard deviation in brackets).

	Left			Right
Parameters (Units)	SH	Ground Truth	Delta	SH	Ground Truth	Delta
Stride length (m)	1.07 (0.15)	1.07 (0.14)	0.00 (0.07)	1.07 (0.16)	1.07 (0.15)	0.00 (0.07)
Stride time (s)	1.79 (0.15)	1.78 (0.15)	−0.01 (0.08)	1.77 (0.19)	1.77 (0.17)	−0.01 (0.11)
Stride speed (m/s)	0.60 (0.09)	0.60 (0.09)	0.00 (0.02)	0.62 (0.09)	0.62 (0.09)	0.00 (0.02)
Step length (m)	0.34 (0.14)	0.29 (0.12)	−0.05 (0.04)	0.33 (0.12)	0.29 (0.11)	−0.04 (0.04)
Step width (m)	0.09 (0.03)	0.11 (0.03)	0.01 (0.01)	0.12 (0.02)	0.13 (0.02)	0.01 (0.01)
Step time (s)	0.89 (0.12)	0.86 (0.13)	−0.03 (0.09)	0.90 (0.24)	0.91 (0.24)	0.01 (0.09)
Cadence (steps/min)	67.62 (5.24)	69.90 (5.72)	2.28 (2.53)	64.03 (7.74)	63.51 (8.44)	−0.52 (2.25)
Stance time (s)	1.14 (0.12)	1.29 (0.12)	0.14 (0.08)	1.15 (0.11)	1.25 (0.12)	0.11 (0.10)
Swing time (s)	0.64 (0.08)	0.50 (0.06)	−0.14 (0.07)	0.63 (0.13)	0.50 (0.09)	−0.12 (0.09)
Stance swing ratio	1.80 (0.25)	2.57 (0.33)	0.77 (0.37)	1.81 (0.20)	2.43 (0.28)	0.62 (0.35)
Double support time (s)	0.30 (0.20)	0.40 (0.18)	0.10 (0.08)	0.26 (0.07)	0.37 (0.06)	0.11 (0.08)
Foot angle (°)	12.89 (4.24)	13.10 (4.28)	0.21 (0.67)	13.05 (6.34)	13.38 (6.30)	0.33 (1.68)

, calculated stride parameters were comparable to ground-truth results.

Figure 8 shows the ensemble averaged leg angles during gait, from foot strike to foot off. The shape of the ankle, knee, and hip angle curves were similar to able-bodied joint angles from the literature [34].

4. Discussion

Based on the simulation, design, and evaluations from this research, marker-less human movement analysis is a viable option for outcome measurement of people moving within institutional hallways. The system configuration can lead to automated video capture and fully automated processing that enables outcome measurement with minimal or no human intervention. Unlike past research that has only tested the efficacy of marker-less human motion analysis, this work provides a fully implemented prototype that could be deployed in existing institutional environments and brings closer the adoption of marker-less motion analysis for use in practice.

Body-segment length measurements were within 1.56 (SD = 2.77) cm of ground-truth values. This is comparable to leg-length measurements used for clinical decision making, where the difference between Vicon limb lengths and X-Ray bone measurements was 0.98 (SD = 0.55) cm [35]. Stride parameters and joint angles were analyzed to determine the Smart Hallway’s capability for human motion analysis. Keypoint-based stride parameters were similar to ground-truth results. Across all test conditions, stride-parameter distances were 3.16 (SD = 3.26) cm from the ground truth. In all cases, the standard deviation of the delta was within the standard deviation of the calculated stride parameter. Stride-parameter times were 0.047 (SD = 0.037) s from the ground truth. The timing differences were very small and equivalent to the ground-truth foot-event timings. Stride-parameter velocities were 0.74 (SD = 0.75) cm/s from the ground truth. In particular, stride times for walking straight were 1.18 (SD = 0.05) s; this corresponds with findings from the literature, showing that the Smart Hallway’s standard deviation is within a similar range to existing marker-based gait datasets 1.02 (SD = 0.06) s [36].

Other studies analyzing physician ability for visual gait assessment concluded that raters had an average of 50% accuracy when compared to 3D marker data [37,38]. Even with such low accuracy, good clinical decision making is still possible, though some abnormal gait features are not detected during visual assessment [37,38,39]. Thus, the outcome measures calculated from the Smart Hallway can provide useful information for clinical decision making when compared to current visual assessment methods.

Stride parameters were affected by the walking aids; however, results were still similar to measures from the walking-straight condition. The ensemble average curves obtained from the leg-angle calculations showed similar shapes compared to leg angles from the literature [34].

Stride parameters that were only reliant on a single type of foot event (e.g., only left foot strikes) were highly accurate. Increased error and standard deviation in stride-parameter measurements were seen in parameters that relied on multiple types of foot events, including step length, step width, and stance-swing ratio. This is likely due to compounding errors by combining either contralateral foot events, foot strikes and foot offs, or foot events and 3D keypoint locations on the floor. Foot-event detection was within four frames (67 ms) of the ground-truth foot events. Stride parameters obtained using detected foot events were similar to stride parameters calculated using ground-truth foot events. More work is needed to improve foot-event detection accuracy when calculating stride parameters that rely on multiple data types, such as stance-swing ratio or contralateral step parameters.

Joint-angle standard deviations were greater when occlusions occurred or large variance existed in the Y depth coordinate between the keypoints of interest. Greater error in the global Y-axis is expected since this axis is related to scene depth and is sensitive to triangulation method accuracy. These occlusions and points of greater variance generally occurred at foot strike and foot off, where the leg is either at the maximum distance in front of the body or at the maximum distance behind the body, causing a greater variation in the Y-axis.

Smart Hallway accuracy was lower when keypoints were occluded by walking aids or the pose of the participant in the scene. Improvements can be made by increasing the number of cameras and implementing kinematic constraints on the BODY25 model to ensure that only realistic movements are produced in the 3D reconstruction. The current OpenPose BODY25 AI model does not account for physical and kinematic constraints such as consistent joint-to-joint segment lengths and range of motion of certain joints. Some recent approaches to keypoint-pose inference models, such as MotioNet, have included encodings for bone lengths and 3D joint rotation [40]. These models use a scaled estimation of the depth coordinate, which is learned through AI training processes that are also seen in Google ML Kit Pose Detection [41]. However, OpenPose BODY25 has better keypoint quality compared to these models.

The Smart Hallway produced viable results across all outcome measures, with low variance. For this prototype system, only two participants were recruited for testing; however, approximately 500 strides were analyzed in total. Improvements to the OpenPose BODY25 model or new, more advanced models should aid in more accurately detecting the feet and handling body-part occlusion. Currently, OpenPose processing is a system bottleneck, with results from a 10 s trial returned after approximately 120 s (NVIDIA Jetson AGX). Other processing stages, such as the triangulation step, could be further optimized to reduce data registration time. The current implementation is limited to handling one person in frame at a time; however, with upgrades to outcome-measurement software, groups of people could be processed since OpenPose BODY25 provides keypoints for all people in frame.

The Smart Hallway was successfully deployed in a manner that was non-invasive to the hallway environment. This implies that a system could be set up and gather data on multiple individuals that walk throughout the capture volume on a daily basis without obstructing existing institutional processes. The Smart Hallway system successfully captured data from individuals in a non-invasive manner that did not require markers or a data-collection device being affixed to the participants.

5. Conclusions

A Smart Hallway setup for marker-less 3D human motion analysis in institutional hallways was viable when using an array of four temporally and spatially synchronized cameras and OpenPose BODY25. Temporal synchronization was achieved for the multi-camera array, and spatial synchronization was achieved through a rigorous calibration procedure using RANSAC and bundle-adjustment techniques. 3D joint keypoints were successfully calculated from 50 videos (approximately 500 strides) that included straight walking, walking with turns, walking a curved path, using a walker, and using a cane. Body-segment lengths, foot events, and stride parameters from each condition were similar to manually identified and calculated ground-truth values. Ensemble averaged leg angles corresponded well with kinematic data from the literature [34]. The prototype system validated in this research allows for fully automated human motion analysis without the need for post-processing techniques for calibration, synchronization, or foot-event and stride-parameter analysis. This research helps to move human motion analysis from the lab to the point of patient contact by providing a full system design that is implementable in institutional settings but does not require extensive human resources for operation.

Future research could apply kinematic constraints to the reconstructed 3D results, such as consistent joint-to-joint segment lengths and constrained-joint range of motion in order to reconstruct only possible physical body positions. Furthermore, new training data or transfer learning, could be applied to make better inferences when movement aids are being used. Research into applying the Smart Hallway design to other areas of interest, such as gait-classification applications, could be performed to assess fall risk or neurodegenerative disease progression.