Skip to Content
You are currently on the new version of our website. Access the old version .
  • Proceeding Paper
  • Open Access

28 October 2026

9 Pages

An AI-Driven Multi-Camera Stereo Vision Framework for Real-Time 3D Human Movement Tracking, Biomechanical Analysis, and Sustainable Performance Optimization †

and
Department of Artificial Intelligence and HPC, Széchenyi István University, Egyetem tér 1, 9026 Győr, Hungary
*
Authors to whom correspondence should be addressed.
†
Presented at the Sustainable Mobility and Transportation Symposium 2026, Győr, Hungary, 8 October 2026.

Abstract

This project aims to introduce an AI-driven, multi-camera stereo vision system for real-time, high-precision 3D human motion tracking and analysis. The platform combines image and depth data from a network of stereo cameras into a single spatial reference coordinate system. It uses advanced human pose estimation, motion tracking, and data fusion for later biomechanical analysis to measure movement features like gait, posture, balance, and symmetry. The system supports objective assessment of human motion and helps identify abnormalities or functional limitations through continuous monitoring. Its main contribution is a robust, scalable framework for accurate 3D motion reconstruction for quantitative and objective movement analysis. This technology can be used in healthcare, rehabilitation, ergonomics, sports science, and research, providing reliable, non-invasive human motion data for better decision-making.

1. Introduction

The progress in the last decade in artificial intelligence, computer vision, and 3D sensing has greatly improved human motion analysis [1]. Accurately measuring movement is important in many areas, such as healthcare, rehabilitation, ergonomics, workplace safety, and sports science [2]. Traditional methods like manual observation, wearable sensors, and marker-based motion capture often have limits with scalability, comfort, environment, and cost [3].
Recent advances in stereo vision and deep learning for human pose estimation now allow real-time, markerless motion tracking in natural settings. Multi-camera stereo systems collect detailed spatial data, rebuild human motion in 3D, and work non-invasively without limiting movement [4]. Using networks of cameras also makes it possible to scale up and synchronize 3D tracking in large, complex spaces [5].
Even with these advances, there are still major challenges in getting reliable and accurate motion analysis. Multi-camera fusion needs careful camera calibration, synchronization, and alignment. Real-time use also has to deal with high computing needs, network delays, and problems like occlusions and changing environments [6].
Being able to monitor and assess human movement all the time is important for spotting functional problems early, tracking rehabilitation progress, checking workplace safety, and improving performance [7]. Objective, quantitative movement analysis gives valuable data that is hard to get with traditional observation [8].
This paper introduces a multi-camera stereo vision system using AI-based human pose estimation and movement tracking for biomechanical analysis. The system does AI-based pose estimation and tracking with distributed stereo cameras, and data fusion from multiple cameras. It is built to accurately analyze human motion in real-world settings, supporting uses like gait analysis, rehabilitation, ergonomics, healthcare, and research.

2. Materials and Methods

2.1. Traditional Movement Tracking Technologies

Human movement analysis has improved a lot recently, as more fields like healthcare, rehabilitation, ergonomics, and biomechanics need objective, quantitative motion assessment. Early methods used manual video annotation and expert observation, which gave some insight into movement but were slow, required a lot of work, and varied between observers [9].
Automated sensing technologies have led to the use of wearable systems like GNSS devices and IMUs to capture features such as speed, acceleration, movement, and activity level [10]. However, these systems have limitations; for example, GNSS does not work well indoors or when blocked, and IMUs can drift and need regular re-calibration to stay accurate. Wearable devices can also affect natural movement, cause discomfort, and need regular setup and maintenance [11].
Marker-based optical motion-capture systems give very precise biomechanical measurements using visible markers [12]. While they work well in labs, they are hard to use in real-world settings because they cost a lot, cover only small areas, and are sensitive to occlusions and environment. So, they are mostly used for lab studies and are not ideal for large-scale, real-world movement analysis.

2.2. Vision-Based Multi-Camera Tracking Systems

Progress in computer vision and AI has sped up the use of camera-based tracking for human movement analysis [13]. Vision-based methods are non-invasive and can track several people at once. Multi-camera systems are especially useful for rebuilding 3D spatial data from different angles. By using overlapping views, these systems can reduce occlusions and make motion tracking more reliable in busy, open environments [14].
Multi-camera fusion builds on stereo vision by combining data from several synchronized camera pairs into a single global coordinate system [15]. This solution allows for continuous movement tracking even in larger areas and makes the system more robust to occlusions. However, it needs accurate calibration, synchronized, tight timings, and good communication between the cameras [16].

2.3. Artificial Intelligence in Pose Estimation and Object Tracking

Deep learning models have made markerless human pose estimation much more accurate and reliable. Today’s neural networks can find skeletal keypoints directly from RGB images and work in real time on GPU hardware, making biomechanical analysis possible outside the lab [17].
AI-based body-tracking systems usually estimate the 2D positions of main joints like shoulders, elbows, hips, knees, and ankles to create a digital skeleton. Tracking over time helps keep identities consistent and reduces sudden jumps in motion. In sports, though, it is still hard to reliably identify players because of fast movements, physical contact, occlusions, and changing light [17,18].

2.4. Overview of the Project Framework

The project framework is a distributed, real-time platform that can rebuild and analyze human movement using multiple stereo cameras in a virtual environment. Its main goal is to provide accurate 3D motion tracking with low delay and the ability to scale up in large, open spaces. The system brings together stereo vision, AI-based pose estimation, multi-camera data fusion, and visualization in one processing pipeline.
The system uses several stereo-camera nodes placed around the monitored area. Every camera is connected to an edge device for local processing, and those edge devices to a central fusion server that combines all spatial data into a shared coordinate system. Each node captures images, estimates depth, and does basic motion tracking before sending the processed data over a fast network. The central server then synchronizes and merges these data streams to create a clear, real-time 3D view of human motion.
This modular design allows the system to be used in different environments and for various purposes. More cameras can be added easily, so the system can grow with the size and complexity of the space, without significant computational power loss. The distributed setup also makes processing more efficient and reliable by sharing the workload between edge devices and the central server.

2.5. Hardware Infrastructure

The hardware setup uses a network of stereo cameras and GPU-powered edge devices. Cameras are placed around the sports field to cover as much area as possible and reduce blind spots and occlusions. Good camera placement is especially important in team sports, where players move quickly and interact often, which can make tracking harder.
Each stereo camera is connected to a small processing device that can run real-time analysis locally. The current system uses NVIDIA Jetson Orin Nano edge compute units (NVIDIA Corporation, Santa Clara, CA, USA) with a built-in graphical processing unit because they offer good performance, power efficiency, and portability. These devices handle depth estimation, body tracking, and object detection on the incoming images, and then send the spatial data to the fusion server.
This distributed setup lowers the amount of bandwidth needed because it does not send full high-resolution videos to the central server. Instead, each edge device only sends the detected human body data, which makes the system more scalable and reduces network congestion and central processing demands.
Devices communicate over a fast local network, which allows quick synchronization and data transfer between camera nodes and the fusion server. The system leverages Ethernet connections and uses a simple publish–subscribe messaging system for real-time data sharing.

2.6. Camera Calibration and Spatial Alignment

The proposed calibration method computes the extrinsic parameters of every camera in a multi-camera system from a detected object represented in three-dimensional camera coordinates. A brightly colored ball was used for calibration, for which we created a custom detection model. For each camera, the detected object trajectory is temporally matched to that of a selected reference camera, yielding correspondence pairs (Pi, Gi). The calibration problem is formulated as the estimation of a rigid transformation (R, t) such that
Gi = RPi + t + ϵi,
where ϵi denotes the residual alignment error. The optimal parameters are obtained by minimizing the weighted least-squares objective.
min R , t ∑ i = 1 N w i ∥ G i − R P i + t ∥ 2 2 , subject to   R ∈ SO 3
Here, wi denotes a confidence-based weight derived from the object detector. To improve robustness, the initial estimate is computed with the weighted Kabsch algorithm and embedded in a RANSAC scheme, where the residual for each correspondence is defined as
r i = ∥ G i − R P i + t ∥ 2
Correspondences satisfying ri < τ are classified as inliers, where τ is a predefined threshold. The final extrinsic transformation is expressed in homogeneous coordinates as
T = R t 0 ⊤ 1 ,
which maps every 3D point from the local camera coordinate system to the reference camera coordinate system. This formulation enables automated calibration without manual landmark annotation while maintaining robustness through confidence weighting and outlier rejection.
As part of the calibration process, a body-based calibration technique was also developed and evaluated. This approach applied the same underlying computational principles as the object-based calibration method but utilized a larger number of reference points derived from human body keypoints.
The evaluation showed that the body-based approach performed less accurately than the object-based calibration method, primarily due to uncertainties in human body keypoint detection. Given its lower accuracy and higher computational and implementation complexity, the body-based calibration approach was ultimately discontinued.

2.7. AI-Based Player and Ball Tracking

For human body pose estimation, the system uses the pretrained YOLO26 pose model (Ultralytics Inc., Frederick, MD, USA, https://www.ultralytics.com/, accessed on 21 October 2026) with depth computation. Each client device first sets up stereo depth and position tracking. The model then estimates a full 3D skeleton with 34 joints, as well as 3D body position, velocity, confidence scores, and joint details. By using both visual features and stereo geometry, the system creates a true 3D skeleton in the camera’s coordinate system, not just a 2D pose.
The reconstruction of joint positions is based on stereo triangulation. Let f denote the focal length in pixels, B the stereo baseline, and d the disparity between corresponding joint projections in the left and right images. The depth of each joint is computed as
Z = fB d
Using the left image center (u, v) and the camera intrinsics (fx, fy, cx, cy), the 3D position of each joint in the camera coordinate system is obtained via back-projection:
X = ( u − c x ) Z f x , Y = − ( v − c y ) Z f y , Z = − depth
This formulation yields a metric 3D skeletal representation of the human body, where all joint positions are expressed in a unified spatial coordinate system, enabling downstream biomechanical analysis and motion evaluation.

2.8. Real-Time Data Processing Pipeline

After each camera finishes its own detection and depth computation, all human body detections are sent from the edge devices to a central server using a simple messaging system called ZeroMQ (https://zeromq.org/, accessed on 21 October 2026), which is a publish–subscribe-based lightweight messaging framework. The server uses the pre-made calibration file to convert each detection from the camera’s local coordinates to a shared world coordinate system. 3D joint positions are fully transformed, while directions like velocity are only rotated. This keeps all measurements physically consistent across the camera network.
The system fuses human body detection from each connected camera. It matches observations from different cameras by comparing how close they are, how well their velocities match, and how similar their skeletons look. The Hungarian algorithm is used to solve this matching problem. Once matched, the system averages the 3D joint positions, velocities, and body sizes to create a single, unified representation for each person. Stable identity tracking is kept over time by making sure sources are consistent and motion is smooth, allowing for clear tracking across the camera network.

3. Results

3.1. Calibration Results

The results show that the object-based calibration method is more accurate and reliable than the body-keypoint-based method. In all tested camera setups, the object-based approach has lower median inlier residuals, usually between 3.7 and 4.4 cm. The body-keypoint method has higher values, about 4.5–5.2 cm, over a 3-by-3-meter area, which means it is less accurate than the object-based method. The object-based method also has a higher inlier ratio, meaning more matches meet the calibration’s geometric rules. These results show that the object-based method gives more precise and dependable calibration.
Repeated tests confirm the calibration’s robustness. When calibration is run with different random seeds, the object-based method still gives lower residuals and higher inlier ratios than the body-keypoint method. Some camera setups show moderate changes, especially in harder cases, but the ranking of the methods stays the same. This means the object-based method’s advantage is stable, not just due to lucky starting points. Overall, the results show this approach gives a solid geometric base for later detection and fusion steps.

3.2. Detection and Fusion Results

The detection and fusion results show that the system creates stable human motion paths over time, and the measured runtimes focus on the fusion stage, as this is the part where the optimization of the fusion process is evaluated. Table 1 reports the per-camera pose detection metrics prior to fusion. Across all four camera views, the pose estimator achieved maximal detection confidence (100) and full keypoint completeness (100%), indicating no occlusion- or depth-related keypoint dropout during the evaluated sequences. Skeletal rigidity, measured as limb-length coefficient of variation, ranged from 0.072 to 0.123 across the four cameras (mean 0.101 ± 0.019), confirming anatomically plausible, near-constant bone lengths over time. Temporal stability, measured as motion-compensated keypoint jitter, ranged from 23.9 to 27.8 mm per frame (median across cameras 25.9 ± 1.4 mm), with 90th-percentile values between 109.0 and 120.9 mm, indicating sub-3 cm frame-to-frame pose noise once genuine body motion is removed. The low cross-camera variance in Table 1 further suggests consistent, viewpoint-independent pose estimation quality across the rig.
Table 1. Skeleton accuracy per camera before fusion.
Table 2 compares the pre-fusion (per-camera mean) and post-fusion metrics, showing the effect of combining the four camera views into a single fused skeleton. The fused skeleton retained full keypoint completeness (100%) and near-maximal detection confidence (99.1), confirming that fusion did not introduce missing-joint artifacts. Motion-compensated keypoint jitter dropped substantially, from a pre-fusion mean of 25.9 mm to 17.3 mm per frame (median), with the 90th-percentile jitter nearly halving from 113 mm to 59.1 mm—indicating that cross-camera fusion meaningfully improves temporal pose stability by averaging out independent per-camera noise. However, the limb-length coefficient of variation increased from 0.101 to 0.142, approaching the 0.15 consistency threshold, suggesting a modest trade-off: combining skeletons estimated from differing viewpoints and scales introduces small inconsistencies in bone length even as it suppresses frame-to-frame jitter.
Table 2. Pre- and after fusion evaluation of the system.
The reported values represent averages computed over 20 separate test recordings, providing a more robust estimate of typical system behavior rather than a single-sequence measurement. These metrics remain consistency proxies rather than ground-truth accuracy measures: they capture the stability of the estimated pose, not its correctness relative to true joint locations, so a systematically biased estimator could still score well. Future validation against a manually annotated or marker-based mocap reference would allow absolute accuracy (e.g., PCK or millimeter-level joint error) to be reported.
Figure 1 shows the final fused 3D human skeleton consisting of 38 keypoints generated by the proposed multi-camera system. The reconstructed keypoints provide a spatial representation of the subject’s body posture and movement if we consider the movement on a time scale, which can serve as the basis for subsequent rehabilitation-oriented movement analysis.
Figure 1. Output fused skeleton of the system. Grey dots mark the reconstructed 3D keypoints, red and blue segments distinguish the limbs of the two body sides, and the black segment represents the spine.
Performance results focus on the fusion stage, not the whole detection and tracking process. The average fusion cycle is fast, usually 2 to 3 ms, with some peaks at 10 to 12 ms. The system handles about 400–550 frames per second, which is much faster than real-time needs, and the maximum processing speed of 30 FPS on edge devices. This shows the fusion stage is efficient and does not slow down the system and it can detect and fuse the human body in real time, but it is locked to a maximum of 30 FPS due to the tested Jetson Orin Nano. With a more powerful edge device it could achieve higher computational speed.

4. Discussion

The results support the central claim that the proposed multi-camera stereo vision framework provides a viable foundation for accurate 3D human motion tracking without using any wearable sensor, so it does not affect the tracked person’s movement.
Calibration. The object-based method’s lower median inlier residuals (3.7–4.4 cm) and higher inlier ratios versus the body-keypoint baseline suggest that rigid-object correspondences are a more reliable calibration signal than human keypoints, which are themselves noisy estimates subject to the same detection uncertainty being calibrated against. Confidence-weighted RANSAC, which downweights unreliable correspondences rather than treating all observations equally, appears to directly improve repeatability.
Detection and fusion. Per-camera results already show high, uniform pose quality—full completeness (100%), near-maximal confidence (100), and consistent limb lengths (CV = 0.101 ± 0.019)—before fusion. This matters because it isolates the fusion improvement to genuine noise reduction rather than fixing per-camera detection failures: motion-compensated jitter fell from 25.9 ± 1.4 mm to 17.3 mm per frame after fusion (~33% reduction in median, ~50% at p90, from 109–121 mm to 59.1 mm), consistent with multi-view averaging suppressing uncorrelated per-camera noise.
This gain comes with a small cost: limb-length CV rose from 0.101 to 0.142, approaching the 0.15 threshold. This indicates fusion smooths high-frequency positional noise but can introduce low-frequency bias when skeletons from different viewpoints/scales are blended without a bone-length constraint—a natural target for future work (e.g., a rigid-body constraint in the fusion step).
Identity continuity (only five switches, tracks up to 7600 frames) further indicates that confidence-aware matching resolves cross-camera correspondence effectively, avoiding the fragmentation typical of naive tracking under viewpoint variation.
Computational efficiency. The fusion stage’s 2–3 ms per-cycle runtime (400–550 fps) shows these accuracy gains are not achieved at the cost of real-time performance, unlike many multi-view approaches that rely on expensive iterative optimization, leaving headroom to scale to more cameras or higher frame rates.
Limitations. All reported metrics are internal consistency proxies, not ground-truth accuracy—a systematically biased estimator could still score well. Validation against manual annotation or marker-based mocap (e.g., PCK, millimeter-level joint error) is needed to establish true accuracy. The framework also does not yet compute joint angles, forces, or other biomechanical measures, and the individual effects of lighting, occlusion, and camera placement were not systematically isolated in this evaluation. Extending fusion with rigid-skeleton constraints and layering biomechanical models onto the 3D joint output are natural next steps.

Author Contributions

Conceptualization, Z.K. and Á.K.; methodology, Z.K.; implementation, Z.K.; validation, Z.K.; conclusions, Z.K. and Á.K.; supervision, Á.K. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the National Research, Development and Innovation Fund of Hungary under the Proof of Concept 2025 programme, grant number POC2025_UNIINNO_008 (“Twintech” project).

Institutional Review Board Statement

Not applicable.

Data Availability Statement

No new data were created or analyzed in this study. Data sharing is not applicable to this article.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Abdelmohsen, A.M. Artificial Intelligence in Biomechanics: A Narrative Review of Current Applications in Diagnostic and Physical Rehabilitation. Physiother. Res. Int. 2025, 30, e70120. [Google Scholar] [CrossRef] [Scilit]
  2. Wade, L.; Needham, L.; McGuigan, P.; Bilzon, J. Applications and limitations of current markerless motion capture methods for clinical gait biomechanics. PeerJ 2022, 10, e12995. [Google Scholar] [CrossRef] [Scilit]
  3. Yang, L.; Amin, O.; Shihada, B. Intelligent wearable systems: Opportunities and challenges in health and sports. ACM Comput. Surv. 2024, 56, 1–42. [Google Scholar] [CrossRef] [Scilit]
  4. Evans, M.; Needham, L.; Wade, L.; Parsons, M.; Colyer, S.; McGuigan, P.; Bilzon, J.; Cosker, D. Synchronised video, motion capture and force plate dataset for validating markerless human movement analysis. Sci. Data 2024, 11, 1300. [Google Scholar] [CrossRef] [Scilit]
  5. Thomas, C.; Nolte, K.; Schmidt, M.; Jaitner, T. Comparison of Marker-Based and Markerless Motion Capture Systems for Measuring Throwing Kinematics. Biomechanics 2025, 5, 100. [Google Scholar] [CrossRef] [Scilit]
  6. Liu, K.; Chen, L.; Xie, L.; Yin, J.; Gan, S.; Yan, Y.; Yin, E. Auto calibration of multi-camera system for human pose estimation. IET Comput. Vis. 2022, 16, 607–618. [Google Scholar] [CrossRef] [Scilit]
  7. Egeonu, D.; Jia, B. A systematic literature review of computer vision-based biomechanical models for physical workload estimation. Ergonomics 2025, 68, 139–162. [Google Scholar] [CrossRef] [Scilit]
  8. Varcin, F.; Boocock, M.G. The accuracy, validity and reliability of Theia3D markerless motion capture for studying the biomechanics of human movement: A systematic review. Artif. Intell. Med. 2026, 173, 103332. [Google Scholar] [CrossRef] [Scilit]
  9. Picerno, P. 25 years of lower limb joint kinematics by using inertial and magnetic sensors: A review of methodological approaches. Gait Posture 2017, 51, 239–246. [Google Scholar] [CrossRef] [Scilit]
  10. Arlotti, J.S.; Carroll, W.O.; Afifi, Y.; Talegaonkar, P.; Albuquerque, L.; Ball, J.E.; Chander, H.; Petway, A. Benefits of IMU-based wearables in sports medicine: Narrative review. Int. J. Kinesiol. Sports Sci. 2022, 10, 36. [Google Scholar] [CrossRef] [Scilit]
  11. Seçkin, A.Ç.; Ateş, B.; Seçkin, M. Review on wearable technology in sports: Concepts, challenges and opportunities. Appl. Sci. 2023, 13, 10399. [Google Scholar] [CrossRef] [Scilit]
  12. Yang, C.; Wei, L.; Huang, X.; Tu, L.; Xu, Y.; Li, X.; Hu, Z. Comparison of lower limb kinematic and kinetic estimation during athlete jumping between markerless and marker-based motion capture systems. Sci. Rep. 2025, 15, 18552. [Google Scholar] [CrossRef] [Scilit]
  13. Scott, A.; Uchida, I.; Ding, N.; Umemoto, R.; Bunker, R.; Kobayashi, R. Teamtrack: A dataset for multi-sport multi-object tracking in full-pitch videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 17–18 June 2024; pp. 3357–3366. [Google Scholar]
  14. Rahimian, P.; Toka, L. Optical tracking in team sports: A survey on player and ball tracking methods in soccer and other team sports. J. Quant. Anal. Sports 2022, 18, 35–57. [Google Scholar] [CrossRef] [Scilit]
  15. Ezatzadeh, S.; Keyvanpour, M.R.; Shojaedini, S.V. A human fall detection framework based on multi-camera fusion. J. Exp. Theor. Artif. Intell. 2022, 34, 905–924. [Google Scholar] [CrossRef] [Scilit]
  16. Li, C.; Cai, C. A Calibration and Real-Time Object Matching Method for Heterogeneous Multi-Camera System. IEEE Trans. Instrum. Meas. 2023, 72, 1–12. [Google Scholar] [CrossRef] [Scilit]
  17. Avogaro, A.; Cunico, F.; Rosenhahn, B.; Setti, F. Markerless human pose estimation for biomedical applications: A survey. Front. Comput. Sci. 2023, 5, 1153160. [Google Scholar] [CrossRef] [Scilit]
  18. Ji, H.; Wang, L.; Zhang, Y.; Li, Z.; Wei, C. A Review of Human Pose Estimation Methods in Markerless Motion Capture. Comput. Aided. Des. Appl. 2024, 21, 392–423. [Google Scholar] [CrossRef] [Scilit]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.