Computer Vision-Based Safety Monitoring of Mobile Scaffolding Integrating Depth Sensors

Abstract

Mobile scaffolding is essential in construction but presents significant safety risks, particularly falls from height (FFH) due to improper use and insufficient monitoring. While prior research has identified hazards, it often lacks robust, actionable solutions, especially regarding the comprehensive analysis of worker behaviors and the spatial context. This study proposed a computer vision-based safety monitoring system that leverages depth cameras for accurate spatial assessments and incorporates temporal conditions to reduce false alarms. The proposed system extends object detection algorithms with mathematical logic derived from safety rules to classify four key unsafe conditions related to safety helmet use, guardrail and outrigger presence, and worker overcrowding on mobile scaffolds. A diverse dataset from multiple sources enhances the model’s applicability to real-world scenarios, while a status trigger module verifies worker behavior over a 3 s window, minimizing detection errors. The experimental results demonstrate high precision (0.95), recall (0.97), F1-score (0.96), and accuracy (0.95) for safe behaviors, with similarly strong metrics for unsafe behaviors. The qualitative analysis further confirms substantial improvements in worker position detection and safety compliance using 3D data over 2D approaches. These findings highlight the effectiveness of the proposed system in improving mobile scaffolding safety, addressing critical research gaps, and advancing construction industry safety standards.

1. Introduction

Maintaining a safe working environment at construction sites is a major challenge for construction practitioners worldwide. Despite ongoing efforts by health and safety organizations, construction continues to exhibit high fatality rates, with 9.6 deaths per 100,000 full-time workers—2.5 times higher than the average across all industries [1,2,3]. A significant portion of these fatalities results from falls from height (FFH), accounting for 34% of fall-related deaths [4,5,6]. However, more than 350 deaths have been caused by falls from low heights [7]. Among the contributing factors, mobile scaffolding poses particular risks due to its delicate structure and frequent misuse [8]. In South Korea, 488 scaffolding-related deaths occurred from 2017–2021, including 45 directly involving mobile scaffolds [9,10]. These incidents are commonly attributed to improper safety measures, insufficient equipment, and non-compliance with protocols [11]. Manual safety monitoring is labor-intensive and prone to error, necessitating the integration of automated systems for real-time oversight.

Traditional safety practices—such as toolbox talks, visual inspections, and PPE enforcement—are foundational, but limited by their subjective, time-consuming nature and inconsistency. Manual methods struggle to detect and respond promptly to dynamic site hazards. The rise of computer vision (CV) and deep learning offers promising alternatives for automated hazard detection, object recognition, and behavior analysis. Techniques such as CNNs, object detection, and instance segmentation are increasingly being applied in safety monitoring tasks [12,13]. Several studies have focused on detecting violations related to PPE, ladders, and worker positions [14,15,16], but rarely do they address mobile scaffolds or employ depth-sensing data. For example, existing systems detect workers and objects using bounding box collisions without a contextual awareness of the depth, leading to false positives when workers are merely near but not on the scaffold [17]. Moreover, most models operate on single-frame analysis and lack mechanisms to reassess or validate initial detections, resulting in misidentifications.

To address the critical gaps identified in current safety monitoring systems for mobile scaffolding, this study aims to develop a computer vision (CV)-based safety monitoring system with the following specific research objectives: (1) To develop an advanced detection system utilizing depth information to accurately assess the spatial context of workers in relation to the scaffold to reduce inaccurate detections. (2) To implement a status trigger module to minimize the false alarms. This dual-layer approach enables proactive safety management by monitoring a broad range of behaviors in real time. The proposed framework extends beyond specific safety checks to analyze a broad range of worker behaviors against established construction safety standards. This approach identifies unsafe conditions and informs risky behaviors before accidents occur, providing proactive safety management. Furthermore, this study contributes to the development of customized algorithms and specialized training datasets tailored to address the unique hazards associated with mobile scaffolding.

2. Related Work

Mobile scaffolding is extensively used at construction sites due to its portability and mobility. However, its complex design makes it one of the top three accident-prone elements in these environments. Despite its widespread use, research specifically addressing the safety of mobile scaffolding is limited. Several studies have explored methods for monitoring fixed scaffolding. For instance, Xu et al. proposed a three-dimensional (3D) local feature descriptor for detecting scaffolds in point clouds [18], and Cho et al. leveraged image and building information management (BIM) data for automatic progress monitoring [19]. Jung et al. [20] investigated failure detection by applying image processing to small-scale scaffolds and identifying temporary structure changes. However, the literature on compliance monitoring and detecting the unsafe use of mobile scaffolds is scarce, highlighting the need for further exploration in this crucial area.

Despite these advancements, significant gaps persist in current research, particularly regarding the integration of depth information and comprehensive safety rule checking. For instance, studies such as fall protection for lone workers focus on ladders using SSD [9] and mitigating falls by tracking objects using Mask R-CNN [14], but both lack depth information, limiting their accuracy and scope. Similarly, the detection of the non-usage of hardhats [21] and safety harnesses [22] employ Faster R-CNN, targeting persons and harnesses without incorporating depth data, which can lead to misclassifications in complex environments. Additionally, while motion capturing data for unsafe actions utilizes depth information with CNN + LSTM, it is restricted to ladders, failing to address other critical construction elements. Action classification based on video segments with Bayesian networks also falls short by not using depth information, which could enhance the precision of recognizing unsafe behaviors [16], as summarized in

Table 1. Summary of case conditions and behaviors.

Author	Objective	Safety Rules Checking	Targeted Objects	Depth Information	Technique	Model
[9]	Fall protection for lone worker	Fall	Ladder	No	Object detection	SSD
[14]	Mitigating falls by tracking objects	Fall	Person Steel support Concrete support	No	Tracking	Mask R-CNN
[21]	Detecting non-usage of hardhats from far-field surveillance videos	Hardhats	Person	No	Object detection	Faster R-CNN
[22]	Detection of safety harness	Falls from height	Person Harness	No	Object detection	Faster R-CNN
[23]	Motion capturing data for unsafe actions	No	Ladder	Yes	Pose estimation	CNN + LSTM
[16]	Action classification based on a set of video segments	No	Workers and equipment	No	Action Recognition	Bayesian networks
Proposed Study	Computer Vision-Based Safety Monitoring of Mobile Scaffolding Integrating Depth Camera	Fall from height	Mobile scaffold with components, guardrails, and outriggers	Yes	Object detection	YOLO

.

The development of an artificial intelligence (AI)-driven system to enhance the monitoring of mobile scaffolding and enforce compliance with construction safety rules arises from critical needs and gaps in the safety practices followed in the construction industry. Khan et al. highlighted this issue and proposed an approach to the real-time automatic monitoring of workers for identifying mobile scaffolds based on their physical properties [17]. For example, they demonstrated the potential of detecting the presence of outriggers along with personnel to effectively enforce safety rules. However, they covered only the basic safety conditions of mobile scaffolds, which are inadequate for ensuring overall safety during work. Moreover, they utilized Mask R-CNN, a two-stage detection process, for instance segmentation. This may lead to delays in real-time detection, posing a challenge for deployment at construction sites. Additionally, they employed a small dataset containing only 938 images, thereby raising concerns about the robustness of the object detection model in real-time monitoring. One of the key challenges in deploying such an object detection model at construction sites is minimizing misdetection and false positives to ensure the accurate identification of unsafe worker behaviors.

Our study bridges these gaps by integrating depth information with the YOLO model, specifically targeting mobile scaffolding and its components, such as guardrails and outriggers. This integration enables the precise detection of falls from height and other unsafe conditions, providing a more comprehensive and accurate safety monitoring solution. Additionally, our approach includes a status trigger module that evaluates behavior over a time window, significantly reducing false positives and missed detections. By leveraging advanced depth sensing technology and introducing the status trigger module, our approach ensures a step towards a reliable safety rule compliance and dynamic, real-time assessment of worker behavior in complex construction environments.

3. Methodology

This study adopts a deep learning-based computer vision (CV) framework to automatically detect unsafe worker behaviors associated with mobile scaffolding at construction sites. The methodology begins with dataset generation, followed by training a YOLO-based object detection model and applying rule-based classification aligned with OSHA regulations. Performance is evaluated using mean average precision (mAP), F1 score, confusion matrix, and ROC curve. The following subsections detail the dataset preparation, model training, and post-processing approach for regulatory compliance.

3.1. Analysis of OSHA and KOSHA Regulations for Mobile Scaffolding

In recent years, the number of accidents at construction sites caused by mobile scaffolding has risen to a concerning level [24]. In response, both OSHA and KOSHA have implemented strict regulations to prevent such accidents. The KOSHA regulations comprise 13 chapters that address various sectors. These regulations focus significantly on the construction industry, covering approximately 277 related sectors. The summary of OSHA and KOSHA rules related to mobile scaffolding is shown in

Table 2. Comparison of OSHA and KOSHA regulations for mobile scaffoldings.

Construction Entity	Construction Entity Status	OSHA	KOSHA
Mobile Scaffold	Outriggers Guardrail No more than two workers	OSHA 1926.452(w)(6)(iii) With outriggers OSHA 29 CFR 1926.451(g)(4)(i) With guardrail	Article 68 (1) Without outriggers Article 68 (3) Without guardrail [1] Article [C-28-2018]
Worker	Worker on scaffold	OSHA 1926.100 General Requirements With helmet Without helmet	Without helmet

.

According to Article 68 (1) and Article 68 (3) of KOSHA, scaffolding must be equipped with outriggers for stability and guardrails to ensure fall prevention must not be missing. Additionally, Article 68 (5) specifically addresses the weight limitations of mobile scaffolding, setting a maximum weight threshold of 240 kg. Moreover, the regulations stipulate that no more than two workers should be present at a time on a mobile scaffolding. Furthermore, as a general safety rule at construction sites, workers must wear helmets for personal protection. Therefore, workers should wear helmets while working on mobile scaffolding. This study evaluates safety behaviors by analyzing conditions related to mobile scaffolding use. Five typical scenarios are considered, involving helmet usage (workers with or without a helmet, denoted as $W$ or $W$′), the presence or absence of guardrails and outriggers ($M_g$, $M$, or $M^{\prime }$), and the number of workers. Mobile scaffolding with a missing guardrail is represented as $M_g$, with an outrigger as $M$, and without an outrigger as $M^{\prime }$. Unsafe behavior is denoted as $B^{\prime }$. Based on these conditions, each scenario is classified as either safe or unsafe, as summarized in

Table 3. Summary of case conditions and behaviors on mobile scaffoldings.

Case	Statement	Mathematical Logic	Behavior
1	A worker with/without a helmet on mobile scaffolding with a missing guardrail	$\left(W\cap M_g\right)\cup \left(W^{\prime }\cap M_g\right)\subseteq B^{\prime }$	Unsafe
2	A worker without a helmet on mobile scaffolding with an outrigger	$\left(W^{\prime }\cap M\right)\subseteq B^{\prime }$	Unsafe
3	A worker with/without a helmet on mobile scaffolding without outrigger	$\left(W^{\prime }\cap M^{\prime }\right)U\left(W\cap M’\right)\subseteq B^{\prime }$	Unsafe
4	More than two workers working on mobile scaffolding	$\left(\right|W\cup W^{\prime }|>2)\cap (M\cup M^{\prime })\subseteq B^{\prime }$	Unsafe
5	Fewer than or equal to two workers with helmets working on mobile scaffolding	|W| ≤ 2 ∩ M ⊆ B	Safe

.

3.2. Dataset Generation

To train the detection model, a comprehensive dataset of mobile scaffolds and construction workers was constructed using a combination of web scraping, real-world image capture, and video extraction. Videos recorded at the Construction Technology Innovation Laboratory (ConTI Lab) of Chung-Ang University, Seoul, South Korea, and a publicly available dataset from a previous study [17] were also incorporated to enhance diversity.

Frames were extracted from the collected videos using Adobe Premiere Pro 2023 [25] at a rate of 1 frame per second. In total, over 10,000 images were initially gathered, comprising web-scraped images, site captures, and frames from videos. To further increase diversity, additional images were captured in a controlled environment at ConTI Lab. Data preprocessing included duplicate removal using a similarity threshold of 60% [26], resulting in a final dataset of 4868 unique images. The images were subsequently annotated using Roboflow [27] and categorized into six classes: (1) missing guardrails (1860 instances), (2) mobile scaffolding without outriggers (1878 instances), (3) mobile scaffolding with outriggers (2234 instances), (4) workers with helmet (3333 instances), (5) workers without helmet (2829 instances), and (6) z-Tag (755 instances).

To ensure effective model training, the dataset was distributed into three subsets. Approximately 80% of the images were used to train the model, while 10% were used for validation. The remaining 10% was used to test the performance of the trained model. This distribution helped assess the model’s accuracy and generalizability on unseen data in the validation and testing stages. Data augmentation is a powerful method in deep learning that helps minimize the gap between the training and validation sets. By expanding the dataset with diverse variations, data augmentation creates a more comprehensive set of possible data points. This reduces the risk of overfitting and improves the model’s generalizability on unseen examples [28]. The abovementioned dataset was processed using the horizontal flipping, random hue adjustment (−25 to +25), random saturation adjustment (−25 to +25), random brightness adjustment (−25 to +25), and random exposure adjustment (−25 to +25) features of Roboflow [27]. These data augmentations expanded the dataset in terms of variations in object orientations, color tones, saturation levels, brightness levels, and exposure levels. This augmented dataset improved the model’s robustness, allowing it to handle different lighting conditions, and its generalizability to unseen examples. For evaluating classification performance, some random images from the test set were manually categorized into two classes: Safe (104 images) and Unsafe (158 images).

3.3. Object Detection Model

Object detection algorithms are broadly categorized into two types: two-stage and one-stage detectors. Two-stage methods, such as Region-based CNN (R-CNN) [29], Fast R-CNN [4], Faster R-CNN [30], and Mask R-CNN [31], first generate region proposals and then classify them. In contrast, one-stage methods, including the You Only Look Once (YOLO) series [32] and Single-Shot MultiBox Detector (SSD) [33], perform classification and localization simultaneously in a single pass through a convolutional neural network, eliminating the need for proposal generation.

One-stage detectors prioritize inference speed, making them suitable for real-time applications, although often at the cost of slightly reduced accuracy compared to two-stage approaches. Among one-stage detectors, the YOLO series has gained significant popularity due to its lightweight architecture, effective feature-fusion mechanisms, and strong detection performance. Notably, YOLOv5 and YOLOv7 have been widely adopted for their real-time capabilities and efficiency. However, YOLOv5 may face challenges in detecting small or densely packed objects, while YOLOv7’s performance can degrade under suboptimal training conditions or model configurations [34,35]. Subsequent advancements led to the development of YOLOv8 in 2023 [36], which integrated improvements from earlier versions and achieved approximately 1% higher detection accuracy compared to YOLOv5.

More recently, YOLOv12 [37] has been introduced by Ultralytics, offering further enhancements in both accuracy and robustness. YOLOv12 incorporates advanced feature extraction modules, improved anchor-free designs, and optimized backbone networks, enabling superior performance particularly in challenging detection scenarios such as construction sites [37]. Its efficient single-pass architecture has consistently demonstrated its effectiveness in detecting objects under challenging conditions, including variations in illumination and occlusions. Given the requirements of this study, YOLOv12 was selected as the object detection model [37]. Its efficient single-pass architecture, real-time processing capabilities, and robustness to variations in illumination, occlusions, and object scales make it an ideal choice for detecting workers and mobile scaffoldings at construction sites. Compared to earlier versions, YOLOv12 demonstrated improved accuracy and reliability, validating its effectiveness for the application scenarios considered in this research.

3.4. Post-Processing/OSHA Rules Compliance in 2D-Plane

The YOLOv12 model was trained to detect six construction safety-related classes. For helmet detection, the object detection model analyzes spatial and structural features in the head region of identified human figures. It distinguishes between “worker with helmet” and “worker without helmet” as separate classes by recognizing the presence or absence of helmet-like contours and textures. If these features are missing, the model classifies the instance as a “worker without helmet”, indicating a potential safety violation. Regarding guardrail detection, the model identifies geometric structures and contextual positioning characteristic of guardrails, particularly along the upper edges of mobile scaffolding. The training dataset includes two scaffold-related classes—mobile scaffold with outriggers and without outriggers—both of which are expected to include guardrails. Additionally, a separate class labeled “missing guardrails” was annotated for scaffolding lacking these safety components. This labeling strategy enables the model to distinguish compliant from non-compliant configurations, facilitating the accurate detection of guardrail absence as a critical hazard.

After object detection, post-processing is necessary to determine worker behaviors on mobile scaffoldings. Object detection provides the bounding box coordinates of the detected objects, but these coordinates are insufficient to assess whether a person is on mobile scaffolding or to determine the number of people on mobile scaffolding at any given time. To identify worker behavior on mobile scaffolding, a specific process must be followed to check the position of workers on the mobile scaffolding. Figure 1 presents the relative position of the person and the mobile scaffolding with a bounding box, while Figure 2 shows the possible cases of working scenarios.

If the detected object belongs to the “Mobile Scaffolding with Missing Guardrails” ($M_g$) subclass and there is a person ($P$) on the scaffolding, the behavior is considered unsafe. Similarly, if the mobile scaffolding belongs to the “Mobile Scaffolding without Outrigger” subclass and there is a person ($P$) on the scaffolding, the behavior is deemed unsafe. To determine whether a person is on a mobile scaffolding, we must examine the relationship between the bounding boxes of the mobile scaffolding and the person.

Figure 2 illustrates six different cases to discuss the relationship between the bounding box and mobile scaffolding. The x and y vertices originate from the top-left corner. The values of the x and y coordinates increase from top to bottom and left to right, respectively.

• Case A:

If left coordinate of a person $x_l^p$ is greater than that of mobile scaffolding $x_l^s$, right coordinate of the person $x_r^p$ is smaller than that of the mobile scaffolding $x_r^s$, and vertical center of the person ${\displaystyle \frac{y_t^p+y_b^p}{2}}$ is less than that of the mobile scaffolding ${\displaystyle \frac{y_t^s+y_b^s}{2}}$, the following condition is satisfied:

$$x_l^p>\mathrm{ }{x}_l^s,\mathrm{ }{x}_r^p<x_r^s\mathrm{ }and\mathrm{ }{\displaystyle \frac{y_t^p+y_b^p}{2}}<{\displaystyle \frac{y_t^s+y_b^s}{2}}$$

(1)

• Case B:

If the left coordinate of a person $x_l^p$ is greater than that of mobile scaffolding $x_l^s$, right coordinate of the person $x_r^p$ is smaller than that of the mobile scaffolding $x_r^s$, and vertical center of the person ${\displaystyle \frac{y_t^p+y_b^p}{2}}$ is higher than that of the mobile scaffolding ${\displaystyle \frac{y_t^s+y_b^s}{2}}$, the following condition is satisfied:

$$x_l^p>\mathrm{ }{x}_l^s,\mathrm{ }{x}_r^p<x_r^s\mathrm{ }and\mathrm{ }{\displaystyle \frac{y_t^p+y_b^p}{2}}>{\displaystyle \frac{y_t^s+y_b^s}{2}}$$

(2)

• Case C:

• The left coordinate of the first person $x_l^{p1}$ is greater than that of the mobile scaffolding $x_l^s,$ right coordinate of the first person $x_r^{p1}$ is smaller than that of the mobile scaffolding $x_r^s$, and y value of the person’s centroid ${\displaystyle \frac{y_t^{p1}+y_b^{p1}}{2}}$ is greater than that of the mobile scaffolding’s centroid ${\displaystyle \frac{y_t^s+y_b^s}{2}}$.
• The left coordinate of the second person $x_l^{p2}$ is smaller than that of the mobile scaffolding $x_l^s$, right coordinate of the second person $x_r^s$ is smaller than that of the mobile scaffolding $x_r^s$, and vertical center of the person ${\displaystyle \frac{y_t^{p2}+y_b^{p2}}{2}}$ is higher than that of the mobile scaffolding ${\displaystyle \frac{y_t^s+y_b^s}{2}}$.
• The left coordinate of the third person $x_l^{p3}$ is greater than that of the mobile scaffolding $x_l^s$, right coordinate of the third person $x_r^{p3}$ is greater than that of the mobile scaffolding $x_r^s$, and vertical center of the third person ${\displaystyle \frac{y_t^{p3}+y_b^{p3}}{2}}$ is higher than that of the mobile scaffolding ${\displaystyle \frac{y_t^s+y_b^s}{2}}.$ Then, the conditions for case C are satisfied.

  $$\begin{array}{cc}{x}_l^{p1}>\mathrm{ }{x}_l^s\mathrm{ }and\mathrm{ }{x}_r^{p1}& <\mathrm{ }{x}_r^s\mathrm{ }and\mathrm{ }{\displaystyle \frac{y_t^{p1}+y_b^{p1}}{2}}>\mathrm{ }{\displaystyle \frac{y_t^s+y_b^s}{2}},\mathrm{ }{x}_l^{p2}<\mathrm{ }{x}_l^s\mathrm{ }and\mathrm{ }{x}_r^{p2}\hfill \\ & <\mathrm{ }{x}_r^s\mathrm{ }and\mathrm{ }{\displaystyle \frac{y_t^{p2}+y_b^{p2}}{2}}>\mathrm{ }{\displaystyle \frac{y_t^s+y_b^s}{2}},\mathrm{ }{x}_l^{p3}>\mathrm{ }{x}_l^s\mathrm{ }and\mathrm{ }{x}_r^{p3}\hfill \\ & >\mathrm{ }{x}_r^s\mathrm{ }and\mathrm{ }{\displaystyle \frac{y_t^{p3}+y_b^{p3}}{2}}<\mathrm{ }{\displaystyle \frac{y_t^s+y_b^s}{2}}\hfill \end{array}$$

  (3)

• Case D:

• The left coordinate of a person $x_l^{p1}$ is greater than that of the mobile scaffolding $x_l^s$, right coordinate of the person $x_r^{p1}$ is smaller than that of the mobile scaffolding $x_r^s$, and vertical center of the person ${\displaystyle \frac{y_t^{p1}+y_b^{p1}}{2}}$ is higher than that of the mobile scaffolding ${\displaystyle \frac{y_t^s+y_b^s}{2}}$.
• The left coordinate of the person $x_l^{p2}$ is smaller than that of the mobile scaffolding $x_l^s$, right coordinate of the person $x_r^{p2}$ is smaller than that of the mobile scaffolding $x_r^s$, and vertical center of the person ${\displaystyle \frac{y_t^{p2}+y_b^{p2}}{2}}$ is higher than that of the mobile scaffolding ${\displaystyle \frac{y_t^s+y_b^s}{2}}$.
• The left coordinate of the person $x_l^{p3}$ is greater than that of the mobile scaffolding $x_l^s$, right coordinate of the person $x_r^{p3}$ is greater than that of the mobile scaffolding $x_r^s$, and vertical center of the person ${\displaystyle \frac{y_t^{p3}+y_b^{p3}}{2}}$ is greater than that of the mobile scaffolding ${\displaystyle \frac{y_t^s+y_b^s}{2}}$. Then, the following conditions for Case D are satisfied.

  $$\begin{array}{cc}{x}_l^{p1}>\mathrm{ }{x}_l^s\mathrm{ }and\mathrm{ }{x}_r^{p1}& <\mathrm{ }{x}_r^s\mathrm{ }and\mathrm{ }{\displaystyle \frac{y_t^{p1}+y_b^{p1}}{2}}>\mathrm{ }{\displaystyle \frac{y_t^s+y_b^s}{2}},\mathrm{ }{x}_l^{p2}<\mathrm{ }{x}_l^s\mathrm{ }and\mathrm{ }{x}_r^{p2}\hfill \\ & <\mathrm{ }{x}_r^s\mathrm{ }and\mathrm{ }{\displaystyle \frac{y_t^{p2}+y_b^{p2}}{2}}>\mathrm{ }{\displaystyle \frac{y_t^s+y_b^s}{2}},\mathrm{ }{x}_l^{p3}>\mathrm{ }{x}_l^s\mathrm{ }and\mathrm{ }{x}_r^{p3}\hfill \\ & >\mathrm{ }{x}_r^s\mathrm{ }and\mathrm{ }{\displaystyle \frac{y_t^{p3}+y_b^{p3}}{2}}>\mathrm{ }{\displaystyle \frac{y_t^s+y_b^s}{2}}\hfill \end{array}$$

  (4)

• Case E:

• The left coordinate of a person $x_l^{p1}$ is greater than that of the mobile scaffolding $x_l^s$ and the right coordinate of the person $x_r^{p1}$ is smaller than that of the mobile scaffolding $x_r^s$.
• The left coordinate of the person $x_l^{p2}$ is smaller than that of the mobile scaffolding $x_r^s$, right coordinate of the person $x_l^{p2}$ is smaller than that of the mobile scaffolding $x_r^{p2}$, and vertical center of the person ${\displaystyle \frac{y_t^{p2}+y_b^{p2}}{2}}$ is higher than that of the mobile scaffolding ${\displaystyle \frac{y_t^s+y_b^s}{2}}$.
• The left coordinate of the person $x_l^{p3}$ is greater than that of the mobile scaffolding $x_l^s$ and the right coordinate of the person $x_r^{p3}$ is greater than that of the mobile scaffolding $x_r^s$. Then, the following conditions for Case E are satisfied.

  $$\begin{gathered} x_l^{p1}>\mathrm{ }{x}_l^s\mathrm{ }and\mathrm{ }{x}_r^{p1}<\mathrm{ }{x}_r^s,\mathrm{ }{x}_l^{p2}<\mathrm{ }{x}_l^s\mathrm{ }and\mathrm{ }{x}_r^{p2}<\mathrm{ }{x}_r^s\mathrm{ }and\mathrm{ }{\displaystyle \frac{y_t^{p2}+y_b^{p2}}{2}} \\ >{\displaystyle \frac{y_t^s+y_b^s}{2}},\mathrm{ }{x}_l^{p3}>\mathrm{ }{x}_l^s\mathrm{ }and\mathrm{ }{x}_r^{p3}>\mathrm{ }{x}_r^s \end{gathered}$$

  (5)

• Case F:

• The horizontal center of a person ($x_c^{p1}={\displaystyle \frac{x_l^{p1}+x_r^{p1}}{2}}$) is higher than the left coordinate of the mobile scaffolding $x_r^s$ and it is smaller than the right coordinate of the mobile scaffolding $x_r^s$.
• The left coordinate of the person $x_l^{p2}$ is smaller than that of the mobile scaffolding $x_r^s,$ right coordinate of the person $x_l^{p2}$ is smaller than that of the mobile scaffolding $x_r^{p2}$, and vertical center of the person ${\displaystyle \frac{y_t^{p2}+y_b^{p2}}{2}}$ is higher than that of the mobile scaffolding ${\displaystyle \frac{y_t^s+y_b^s}{2}}$.
• The left coordinate of the person $x_l^{p3}$ is greater than that of the mobile scaffolding $x_l^s$ and the right coordinate of the person $x_r^{p3}$ is greater than the right coordinate of the mobile scaffolding $x_r^s.$ Then, the following conditions for Case F are satisfied.

  $$\begin{gathered} x_c^{p1}\ge \mathrm{ }{x}_l^s\mathrm{ }and\mathrm{ }{x}_c^{p1}\le \mathrm{ }{x}_r^s,\mathrm{ }{x}_l^{p2}<\mathrm{ }{x}_l^s\mathrm{ }and\mathrm{ }{x}_r^{p2}<\mathrm{ }{x}_r^s\mathrm{ }and\mathrm{ }{\displaystyle \frac{y_t^{p2}+y_b^{p2}}{2}} \\ >\mathrm{ }{\displaystyle \frac{y_t^s+y_b^s}{2}},\mathrm{ }{x}_l^{p3}>\mathrm{ }{x}_l^s\mathrm{ }and\mathrm{ }{x}_r^{p3}>\mathrm{ }{x}_r^s \end{gathered}$$

  (6)

3.5. Post-Processing to Check Worker Position in 3D-Plane

To check the worker’s accurate position, the depth information is considered to make a 3D-plane. Figure 3 illustrates the various positions of individuals on mobile scaffolding in three different scenarios, depicted in a three-dimensional space ($x,y,z$). In each scenario, M represents the mobile scaffolding, while $P1$ and $P2$ denote Person 1 and Person 2, respectively.

• Case A:

Person $P1$ is on the mobile scaffold $M$ while person $P2$ is standing behind the mobile scaffold at the same height level. The left coordinates of both persons $x_l^{p1}$ and $x_l^{p1}$ are greater than that of the mobile scaffold $x_l^m$. The right coordinates of the persons, $x_r^{p1}$ and $x_r^{p2}$ are smaller than that of the mobile scaffold $x_r^m$ Additionally, the vertical center of the persons, ${\displaystyle \frac{y_t^{p1}+y_b^{p1}}{2}}$ and ${\displaystyle \frac{y_t^{p2}+y_b^{p2}}{2}}$, is less than that of the mobile scaffold, ${\displaystyle \frac{y_t^m+y_b^m}{2}}$. To check the $z$ position, the rear position of the mobile scaffold $z_r^m$ is less than the front position of person $z_f^{p1}$ and the front position of the mobile scaffold $z_f^m$ is greater than the rear position of the first person $z_r^{p1}$. Since person $P2$ is behind the mobile scaffolding, the rear position of the mobile scaffold $z_r^m$ is greater than the front position of the second person $z_f^{p2}$ The following condition is satisfied:

$$\begin{array}{cc}{x}_l^{p1}>\mathrm{ }{x}_l^m,\mathrm{ }{x}_l^{p2}& <x_l^m\mathrm{ }and{x}_r^{p1}<\mathrm{ }{x}_r^m,\mathrm{ }{x}_r^{p2}<x_r^{m}and\mathrm{ }{\displaystyle \frac{y_t^{p1}+y_b^{p1}}{2}}\hfill \\ & <{\displaystyle \frac{y_t^m+y_b^m}{2}},\mathrm{ }{\displaystyle \frac{y_t^{p2}+y_b^{p2}}{2}}<{\displaystyle \frac{y_t^m+y_b^m}{2}}\mathrm{ },z_r^m<z_f^{p1},z_f^m\hfill \\ & <z_r^{p1},z_r^m<z_f^{p2},z_f^m<z_r^{p2}\hfill \end{array}$$

(7)

• Case B:

Person $P1$ is on the mobile scaffold $M$ while person $P2$ is standing in front the mobile scaffold on the ground level. Like Case A, the left coordinates of both persons $x_l^{p1}$ and $x_l^{p1}$ are greater than that of the mobile scaffold $x_l^m$. The right coordinates of the persons, $x_r^{p1}$ and $x_r^{p2}$, are smaller than that of the mobile scaffold $x_r^m$. Additionally, the vertical center of the persons, ${\displaystyle \frac{y_t^{p1}+y_b^{p1}}{2}}$, is less than that of the mobile scaffold, ${\displaystyle \frac{y_t^m+y_b^m}{2}}$, but ${\displaystyle \frac{y_t^{p2}+y_b^{p2}}{2}}$ is greater than that of the mobile scaffold, ${\displaystyle \frac{y_t^m+y_b^m}{2}}$. This means that worker is on ground level. The rear position of the mobile scaffold $z_r^m$ is less than the front position of person $P1$ $z_f^{p1}$, and the front position of the mobile scaffold $z_f^m$ is greater than the rear position of person $P1$ $z_r^{p1}$. Person $P2$ is in front of the mobile scaffolding and the front position of the mobile scaffold $z_f^m$ is less than the rear position of person $P2$ $z_r^{p2}$. The following condition is satisfied:

$$\begin{array}{cc}{x}_l^{p1}>\mathrm{ }{x}_l^m,\mathrm{ }{x}_l^{p2}& <x_l^m\mathrm{ }and{x}_r^{p1}<\mathrm{ }{x}_r^m,\mathrm{ }{x}_r^{p2}<x_r^{m}and\mathrm{ }{\displaystyle \frac{y_t^{p1}+y_b^{p1}}{2}}\hfill \\ & <{\displaystyle \frac{y_t^m+y_b^m}{2}},\mathrm{ }{\displaystyle \frac{y_t^{p2}+y_b^{p2}}{2}}>{\displaystyle \frac{y_t^m+y_b^m}{2}}\mathrm{ },z_r^m<z_f^{p1},z_f^m\hfill \\ & <z_r^{p1},z_f^m<z_f^{p2},z_f^m<z_r^{p2}\hfill \end{array}$$

(8)

• Case C:

Person $P1$ is on the mobile scaffold M while person $P2$ is standing in behind the mobile scaffold on the ground level. As in Case A and Case B, the left coordinates of both persons $x_l^{p1}$ and $x_l^{p1}$ are greater than that of the mobile scaffold $x_l^m$. The right coordinates of the persons, $x_r^{p1}$ and $x_r^{p2}$, are smaller than that of the mobile scaffold $x_r^m$. However, the vertical center of the persons, ${\displaystyle \frac{y_t^{p1}+y_b^{p1}}{2}}$, is less than that of the mobile scaffold ${\displaystyle \frac{y_t^m+y_b^m}{2}}$, but ${\displaystyle \frac{y_t^{p2}+y_b^{p2}}{2}}$ is greater than that of the mobile scaffold ${\displaystyle \frac{y_t^m+y_b^m}{2}}$. This indicates that person $P2$ is on ground level.

The rear position of the mobile scaffold $z_r^m$ is less than the front position of person $P1$ $z_f^{p1}$ and the front position of the mobile scaffold $z_f^m$ is greater than the rear position of person $P1$ $z_r^{p1}$. Person $P2$ is in behind the mobile scaffolding and the front position of the mobile scaffold $z_r^m$ is greater than the rear position of person $P2$ $z_f^{p2}$. The following condition is satisfied:

$$\begin{array}{cc}{x}_l^{p1}>\mathrm{ }{x}_l^m,\mathrm{ }{x}_l^{p2}& <x_l^m\mathrm{ }and{x}_r^{p1}<\mathrm{ }{x}_r^m,\mathrm{ }{x}_r^{p2}<x_r^{m}and\mathrm{ }{\displaystyle \frac{y_t^{p1}+y_b^{p1}}{2}}\hfill \\ & <{\displaystyle \frac{y_t^m+y_b^m}{2}},\mathrm{ }{\displaystyle \frac{y_t^{p2}+y_b^{p2}}{2}}<{\displaystyle \frac{y_t^m+y_b^m}{2}}\mathrm{ },z_r^m<z_f^{p1},z_f^m\hfill \\ & <z_r^{p1},z_r^m>z_f^{p2},z_f^m>z_r^{p2}\hfill \end{array}$$

(9)

3.6. Algorithms for Rule Compliance and Status Trigger Module

In the behavior analysis phase, the system takes a proactive approach to minimize detection errors and accurately identify unsafe worker behaviors on mobile scaffoldings. For instance, when the system detects a worker without a helmet on a mobile scaffold, it initiates a monitoring process for 3 s. During this time, the system continuously observes the worker’s actions. If the worker puts the helmet back on their head within the 3 s window, the system corrects the detection and classifies the behavior as safe, attributing the temporary removal to external factors such as hot weather. Similarly, in cases where multiple workers are detected on mobile scaffolding, the system tracks their presence over a period of 3 s. If the number of workers exceeds two within this observation window, it classifies the situation as genuinely unsafe as opposed to a transient occurrence.

This extended monitoring period prevents false positives and enhances the accuracy of identifying potential hazards related to the number of workers on a mobile scaffold. The flowchart in Figure 4 illustrates safe and unsafe behaviors, concluding in the initiation of the final status. The proposed method comprises two distinct stages: the initial stage involves the classification of safe and unsafe behaviors, and the subsequent stage entails monitoring status for 3 s and assigning the final status.

Appendix A shows the algorithms of both stages. Algorithm A1 describes the algorithm of detecting unsafe behaviors by using an image and the YOLO model. To incorporate third dimension, real sense camera is used that captures two images: an RGB image and a depth image. Both images are represented as two-dimensional arrays of the same size. To integrate depth information into the RGB image, the depth value from the corresponding index in the depth image is used, as shown in Figure 5. It initializes a list, infers the image using the model, checks for safe or unsafe behavior, and returns the result to Algorithm A2.

Algorithm A2 monitors the status received from Algorithm A1, checks for consistency over 3 s, and triggers the final status if the status remains unchanged for 3 s. By using this behavioral analysis approach, the system can make more informed decisions about worker safety, avoiding misclassifications and providing a comprehensive understanding of worker behavior on mobile scaffolding. This sophisticated analysis improves the overall effectiveness of the system in terms of monitoring mobile scaffolding and promoting a safer environment.

4. Experiment, Result, and Evaluation

To seamlessly integrate the third dimension and improve the accuracy of the detection system, an experiment was conducted utilizing the object detection algorithm YOLOv12. Through transfer learning, pre-training weights from the COCO dataset were employed and fine-tuned for the custom dataset. The model underwent training over 300 epochs, with an initial learning rate of 0.01, a batch size of 96, and an image size of 640. The training process utilized a computer equipped with an Intel Core i9 10th-generation CPU (3.30 GHz across 28 CPU cores), 128 GB of RAM, and running Ubuntu 22.04.3 LTS. Additionally, three NVIDIA RTX 3090 GPUs were employed during training.

The Intel RealSense Depth Camera D455 (Intel Corporation, Santa Clara, CA, USA), renowned for its exceptional precision, was selected for its high-definition depth-sensing capabilities. Offering a resolution of up to 1280 × 720 pixels and a frame rate of up to 90 frames per second, the D455’s extended range of 6 m is ideal for various applications, including robotics and augmented reality. Leveraging advanced stereo vision technology, the D455 employs a pair of infrared cameras to detect depth disparities, ensuring accurate depth mapping. Despite its ability to measure distances beyond 6 m, the accuracy diminishes with distance. Notably, the D455’s increased baseline of 95 mm enhances the depth accuracy, with a margin of error below 2% at a distance of 4 m. Its components, including an RGB camera, Left Imager, IR projector, and Right Imager, collectively contribute to its robust depth-sensing capabilities. Operating based on stereo vision technology akin to human eyes, the D455 compares disparities between images captured by its two infrared cameras to calculate precise depth maps. This process involves emitting infrared light patterns onto the scene, capturing images with depth disparities, and subsequently triangulating these disparities to calculate accurate distances to objects in the scene.

4.1. Evaluation Metrics

To evaluate the efficiency of the trained model and usefulness of the algorithm, the precision Equation (10), recall Equation (11), F1-score Equation (12), and accuracy (13) were determined. Precision measures the ratio of correctly predicted true positives, while recall or the true positive rate (TPR) indicate the accurate identification of true positives. The false positive rate (FPR) highlights instances in which the model inaccurately classifies the positive class. The average precision serves as a crucial evaluation metric, providing a consolidated numerical representation of the algorithm’s overall utility.

$$Precison\left(P\right)={\displaystyle \frac{TP}{\left(TP+FP\right)}}$$

(10)

$$Recall\left(R\right)={\displaystyle \frac{TP}{\left(TP+FN\right)}}$$

(11)

$$F1core=2{\displaystyle \frac{\left(P\times R\right)}{\left(P+R\right)}}$$

(12)

$$Accuracy={\displaystyle \frac{\left(TP+TN\right)}{\left(TP+TN+FP+FN\right)}}$$

(13)

In this context, true positive ($TP$), false negative ($FN$), false positive ($FP$), and true negative ($TN$) represent specific outcomes in the evaluation this context, and true positive ($TP$), false negative ($FN$), false positive ($FP$), and true negative ($TN$) represent specific outcomes in the evaluation process.

4.2. Results

Four well-known object detection models, YOLOv5, YOLOv8, YOLOv11, and YOLOv12, were trained on the curated dataset and their performances were evalauted on the validation and testing sets, as shown in

Table 4. Performance comparison of YOLO models on validation and test sets.

Dataset	Models	Precison	Recall	mAP-50	mAP50-95
Validation Set	Yolov5	0.959	0.965	0.969	0.741
	Yolov8	0.955	0.958	0.970	0.783
	Yolov11	0.966	0.952	0.979	0.785
	Yolov12	0.966	0.974	0.979	0.786
Test Set	Yolov5	0.948	0.938	0.957	0.703
	Yolov8	0.931	0.921	0.948	0.716
	Yolov11	0.940	0.918	0.948	0.721
	Yolov12	0.952	0.927	0.957	0.736

. YOLOv12 achieved the highest recall and mAP scores, indicating a better overall performance. While YOLOv5 had slightly higher precision, YOLOv12 showed the best balance across all metrics and was, therefore, selected for an evaluation performance of proposed mobile scaffolding safety and unsafe behavior classification.

The training results of the object detection YOLOv12 model are shown in Figure 6, which shows comprehensive results, including loss, precision, and recall. The accuracy values of the trained model for detecting instances of a “Missing Guardrail”, “Mobile Scaffold without outrigger”, “Mobile Scaffolding”, “Worker with helmet”, “Worker without helmet”, and “z-tag” were 99%, 99%, 99%, 96%, 88%, and 99% respectively. The overall mean average precision was 97% at a confidence level of 0.5 on the validation set, as illustrated in Figure 7. The recall score of the object detection model, along with its confidence scores, is shown in Figure 8. This information helps to select the confidence score to be used during the classification task.

For mobile scaffolding safety and worker behavior classification, the object detection model based on YOLOv12 was extended by incorporating the safety rules extracted from OSHA and KOSHA. The model was trained to detect four specific conditions: (1) workers with/without a helmet on a mobile scaffold with a missing guardrail, (2) workers without a helmet on a mobile scaffold with an outrigger, (3) workers with/without a helmet on a mobile scaffold with an outrigger, and (4) more than two workers on a single mobile scaffold. By detecting these unsafe conditions, managers can promptly implement corrective actions to prevent potential accidents. The results and findings of this study are discussed in two sections: the first section is the classification of two-dimensional (2D) detection to validate OSHA/KOSHA rules compliance, and then qualitatively validating the result of three-dimensional (3D) detection by comparing the results with 2D detection.

4.2.1. OSHA Rules Validation with Two-Dimensional Detection

Four different scenarios are tested to validate rule compliance. In Figure 9a, instances of safe behavior are depicted to validate all the scenarios that have been discussed earlier in the thesis. This figure effectively demonstrates that safety protocols were followed within each of these scenarios, emphasizing the correct utilization of safety equipment and adherence to established guidelines. Figure 9b depicts unsafe behaviors resulting from the absence of a guardrail on a mobile scaffold, in line with the specified condition. This figure highlights the potential hazards resulting from such situations, emphasizing the significance of guardrails in preventing accidents and promoting a safer work environment.

Figure 10a presents an example of unsafe behavior resulting from the presence of more than two workers on a single mobile scaffold, as outlined in the designated condition. The figure illustrates the potential risks associated with overcrowded scaffolding, underscoring the need to enforce guidelines that limit the number of workers per scaffold to ensure their safety.

Figure 10b portrays an instance of unsafe behavior owing to the absence of outriggers on a mobile scaffold, which is consistent with the specified condition. This figure depicts the potential instability and dangers posed by such circumstances, reinforcing the essential role of outriggers in maintaining scaffold stability and averting potential accidents. These figures not only visually enhance our understanding of various scenarios, but also contribute to a deeper comprehension of the importance of adhering to safety protocols and guidelines in the construction environment.

The model’s performance was assessed using a confusion matrix. Figure 11 shows a (2 × 2) confusion matrix, where two is the number of classes, which are safe behavior and unsafe behavior. In this matrix, the top-left cell represents the instances that were classified correctly (153 TPs). Conversely, the top-right cell denotes inaccurately assigned instances (5 FPs). The bottom-left cell represents the instances that were predicted incorrectly (8 FNs). Finally, the bottom-right cell represents the instances that were classified accurately (96 TN). This detailed breakdown offers valuable insights into the classification accuracy of the model and the areas that require further attention for model refinement.

Table 5. Performance indicators for the detection of safe and unsafe behaviors.

Actions	Precision	Recall	F1-Score	Accuracy
Safe Behavior	0.95	0.97	0.96	0.95
Unsafe Behavior	0.95	0.92	0.94

lists the key performance indicators of the proposed approach. These performance indicators highlight the system’s effectiveness in differentiating between safe and unsafe behaviors. In terms of detecting safe behaviors, the system achieved a precision of 0.95, indicating a high degree of accuracy in identifying true safe actions. Moreover, the recall value of 0.97 indicates the system’s ability to capture instances of safe behavior. This combination resulted in a robust F1-Score of 0.96, showcasing the system’s balanced performance in this regard.

Meanwhile, in terms of detecting unsafe behavior, the system achieved a perfect precision value of 0.95, implying that 95% of the instances classified as unsafe were accurate. However, the recall of 0.92 suggested that the system failed to capture a few instances of actual unsafe behavior. This discrepancy in recall led to an F1-Score of 0.94, indicating the need for improved sensitivity in identifying unsafe actions. The ROC-AUC curve for binary classification behavior is shown in Figure 12, which achieved 95% accuracy, demonstrating the reliability of the model in classifying unsafe behaviors. Overall, while the system achieved impressive accuracy in detecting safe behaviors, further refinement is needed to enhance its capability in identifying instances of unsafe behavior.

The F1-score was 95% on the test set, underscoring the robustness and reliability of the approach, even when applied to unseen data curated within the same environmental context as the training dataset. The efficacy of the proposed Algorithm A1 relied on the object detection model’s ability to generalize and accurately detect objects.

4.2.2. Three-Dimensional Detection and Comparison with Two-Dimensional Detection

In the evaluation of 3D detection, a comprehensive array of scenarios was examined to ensure thorough validation. These scenarios included assessing workers’ positions both behind and on mobile scaffolding with missing guardrails or outriggers, as well as scenarios involving more than two workers in varying positions. Additionally, the analysis considered workers without helmets, indicative of unsafe behavior, and instances of mobile scaffolding lacking guardrails or outriggers. Both setups—with and without the Intel RealSense camera—were utilized to provide a comparative perspective on spatial detection capabilities. While the core detection framework is based on a 2D object detection model (YOLOv12), depth information captured from the RealSense camera was integrated to introduce a third dimension (z-axis) to the data. The term “3D detection” is thus used to reflect this augmentation. The depth data were employed not to improve the detection accuracy directly, but rather to enhance spatial reasoning, specifically, to determine whether a detected worker is on the scaffold platform or behind it. This distinction is crucial for the safety assessment, but is not possible with standard 2D RGB cameras. Furthermore, a quantitative analysis—such as a confusion matrix—could not be performed for this evaluation, as it requires real-time ground truth verification, which was not feasible in the current offline setting.

For instance, Figure 13a portrays one worker positioned in front of the mobile scaffolding and another worker on the scaffolding, while Figure 13b depicts a worker positioned behind the scaffolding with another worker on it. Notably, in Figure 14a, the detection identifies a worker on mobile scaffolding lacking guardrails, categorizing the behavior as unsafe. Conversely, in Figure 14b, despite the absence of both guardrails and workers, the behavior is classified as safe. Therefore, if an unsafe scenario is detected but no workers are present, it will be classified as safe, as the 3D system can accurately determine the position of workers. In Figure 15 and Figure 16, the efficacy of 2D and 3D detection methods is compared. In Figure 15a, 2D detection erroneously identifies the behavior as unsafe due to the absence of a guardrail on the mobile scaffolding. However, Figure 15b demonstrates the precision of 3D detection, accurately classifying the behavior as safe. This correct classification is attributed to the fact that there is no worker interaction with the mobile scaffolding at the moment, despite the absence of a guardrail. Similarly, in Figure 16a,b, 2D detection inaccurately identifies the behavior as unsafe due to the lack of outriggers on the mobile scaffolding, whereas 3D detection precisely categorizes the behavior as safe.

Moreover, Figure 17 demonstrates the consistency of 3D detection in accurately classifying behavior, even in the absence of outriggers. Figure 17a illustrates a safe behavior with the worker positioned behind the mobile scaffolding, while Figure 17b showcases an unsafe behavior with the worker on the scaffolding.

Furthermore, when considering a scenario where two workers are on the mobile scaffolding and one is behind it, Figure 18a highlights the incorrect classification by 2D detection, whereas 3D detection accurately identifies the presence of two workers on the scaffolding and one behind it, providing a more precise assessment of the situation.

The comparative analysis between 3D and 2D detection methods in scaffolding safety assessments reveals substantial discrepancies in their efficacy and accuracy in identifying various safety hazards. Notably, while 2D detection often misclassifies unsafe scenarios, including workers positioned behind mobile scaffolding without guardrails, lacking outriggers, or incorrect counts of workers, 3D detection consistently delivers more precise results across these diverse conditions.

4.2.3. Status Trigger Module

A performance evaluation of the status trigger module (Algorithm A2) involved an analysis of a 15 s video captured within the controlled environment of ConTI Lab, featuring a spectrum of both safe and unsafe behaviors. Initially, our assessment focused on the post-processing module’s adherence to OSHA regulations, serving as a foundational benchmark for evaluating the subsequent system performance.

Subsequently, the status trigger module was deployed for evaluation, functioning within a narrow temporal window of 3 s to examine the evolving status. Upon the detection of unsafe behavior in an initial frame, typically occurring at the onset of an event, the module meticulously assessed subsequent frames to dynamically classify the behavior as safe or unsafe. For instance, as depicted in Figure 19a–c, capturing frames at time intervals t = 0, 3, and 6 s, respectively, the initial frame did not exhibit immediate status results. However, through continuous analysis over the subsequent three-second intervals, the system dynamically updated the status, ultimately flagging the behavior as unsafe in Figure 19b. Similarly, the status of the next three seconds was recorded and updated at the 6th second, as depicted in Figure 19c. This iterative process of real-time evaluation and status updates showcases the system’s adaptive capabilities in discerning evolving scenarios.

In contrast to conventional methods, the proposed system demonstrated a superior performance, effectively mitigating instances of misdetections and false positives (FPs). Despite encountering challenges such as misclassifications and FPs, the status trigger module adeptly managed these complexities, thereby averting erroneous status triggers. A well-optimized model facilitated the precise identification of unsafe behaviors, whereas inadequacies could lead to false positives (FPs) and false negatives (FNs), thereby affecting the overall accuracy of the image analysis algorithm. This resilience in accurately identifying and flagging unsafe behaviors underscores the robustness and reliability of the system in real-world applications. Furthermore, the system’s ability to dynamically adapt and refine its assessments in response to evolving scenarios highlights its effectiveness in ensuring workplace safety.

5. Limitations

Despite the significant advancements achieved in this study, several limitations require to be highlighted. The model’s mathematical framework enhances bounding box analysis, but may not encompass all safety scenarios, such as verifying the proper installation of mobile scaffolding, which necessitates additional contextual checks beyond visual detection. While the training dataset is extensive, comprising a diverse range of scenarios, its representation of real-world construction site variations may still be limited. This potential lack of comprehensive diversity could restrict the generalizability of the model across varied construction environments. Utilizing synthetic images could mitigate this limitation by augmenting the dataset with additional diverse scenarios. The operational capabilities of the Intel RealSense camera (D455), which effectively detects the depth within the range of 0.6 m to 6 m, pose constraints on its suitability for scenarios requiring detection beyond this distance. This limitation restricts the application of depth sensing technology in larger construction sites or environments with extensive spatial requirements. Furthermore, the algorithm developed for this study is optimized for real-time 3D detection. While advantageous for immediate hazard identification and response, its real-time processing constraint may limit its utility in environments necessitating post-event analysis or where continuous monitoring with delayed processing is preferred.

The strong performance seen in the confusion matrix (Figure 11) and evaluation metrics on the test set reflects this careful dataset design. However, it is important to note that if the model is deployed on entirely new construction sites with different environmental characteristics, equipment types, or camera configurations, the performance may degrade due to a domain shift. Real-world variability can introduce unseen visual patterns not covered in the training distribution. Therefore, ongoing model refinement with site-specific data or domain adaptation techniques would be necessary for reliable deployment across diverse construction environments. This highlights the importance of both dataset diversity and continuous evaluation in practical computer vision applications for safety monitoring.

These identified limitations highlight the need for further research and technological refinement to enhance the robustness and applicability of computer vision-based safety monitoring systems in complex construction settings.

6. Conclusions

This study presents a computer vision-based system for the real-time monitoring of unsafe behaviors on mobile scaffolds, targeting four critical safety violations: the absence of helmets, missing outriggers, a lack of guardrails, and overcrowding. A diverse dataset was compiled from construction sites and benchmark sources to train a YOLO-based object detection model integrated with a temporal status module. The results demonstrate the system’s accuracy and generalizability across various construction scenarios, enhanced by the use of depth information for spatial awareness. The inclusion of a status module improves real-time behavior classification, enabling the proactive enforcement of safety protocols. This dual-layered approach strengthens the system’s adaptability and positions it as a scalable solution for industrial safety monitoring.

Future research may focus on integrating more advanced hardware such as advanced depth cameras capable of providing extended range depth information, facilitating the more precise localization of construction workers and mitigating occlusion challenges. Additionally, the integration of blockchain technology holds promises for enhancing privacy and security, potentially incentivizing safety compliance through reward mechanisms aligned with adherence to safety protocols. Expanding the dataset with real-life examples from diverse construction sites remains a priority to enhance the system’s accuracy in detecting various construction-related hazards, further advancing the application of computer vision in construction safety practices. These advancements represent a significant step forward in leveraging computer vision technology to enhance safety practices in industries reliant on scaffolding and other complex structures. By enhancing the detection accuracy, operational efficiency, and safety compliance, our approach aims to mitigate risks, improve workplace conditions, and ultimately contribute to the overarching goal of ensuring safer working environments globally.