Development and Validation of a Computer Vision Dataset for Object Detection and Instance Segmentation in Earthwork Construction Sites

Featured Application

This dataset facilitates the development of AI-based safety monitoring systems for earthwork construction sites by enabling accurate object detection and segmentation of construction machinery and terrain.

Abstract

Construction sites report the highest rate of industrial accidents, prompting the active development of smart safety management systems based on deep learning-based computer vision technology. To support the digital transformation of construction sites, securing site-specific datasets is essential. In this study, raw data were collected from an actual earthwork site. Key construction equipment and terrain objects primarily operated at the site were identified, and 89,766 images were processed to build a site-specific training dataset. This dataset includes annotated bounding boxes for object detection and polygon masks for instance segmentation. The performance of the dataset was validated using representative models—YOLO v7 for object detection and Mask R-CNN for instance segmentation. Quantitative metrics and visual assessments confirmed the validity and practical applicability of the dataset. The dataset used in this study has been made publicly available for use by researchers in related fields. This dataset is expected to serve as a foundational resource for advancing object detection applications in construction safety.

1. Introduction

The construction industry is inherently high-risk due to its diverse risk factors [1,2]. Earthwork sites, in particular, are prone to serious accidents involving large machinery and heavy equipment. In response to these risks, the Serious Accidents Punishment Act, enacted in 2022 and amended in 2023, mandates the implementation of AI-based real-time safety management systems at construction sites employing five or more workers [3]. In addition, Seoul City has introduced institutional measures to enhance safety management, such as requiring CCTV installation at small-scale private construction sites with budgets under KRW 5 billion starting in 2024 [4]. Consequently, interest in intelligent safety systems utilizing CCTV footage to detect and respond to on-site hazardous situations in real time is increasing [5,6].

A core component of the key elements for real-time safety management is the ability to accurately identify objects, such as workers, equipment, and terrain, in visual data [7,8,9,10]. Advances in deep learning-based computer vision have enabled AI systems to detect human and equipment movement in construction site CCTV footage and issue timely warnings for potential hazards [11,12]. However, the effectiveness of such systems relies on the availability of high-quality training datasets that reflect diverse real-world site environments. Public datasets, such as ACID 2.0 [13], MOCS [14] and SODA [15], have supported research in this area. However, they primarily focus on general construction activities and lack detailed representations specific to earthwork environments.

To address these limitations, this study focused on a land development earthwork site as the target environment. Over a year, long-term image collection was conducted, capturing key objects, such as construction machinery and terrain elements. Bounding box and polygon annotations were applied to the collected data to support training for object detection and instance segmentation tasks. The reliability of the dataset was confirmed by evaluating its quality and format validity, ensuring suitability for training AI models using CCTV footage from large-scale construction sites.

Based on the Construction Safety Management Guidelines, the Enforcement Decree of the Construction Technology Management Act [16], and the Construction Information Classification System, a total of 15 hazardous object categories were defined, focusing on civil engineering contexts. These included various types of construction equipment and site-specific terrain features.

For optimal data acquisition, collection devices were installed at a housing development site in locations offering a panoramic field of view. Images were captured from eight different camera angles using image collection equipment. The labeling metadata adhered to the COCO format [17], a standard widely used in computer vision for object detection tasks. To assess the generalizability and performance of the dataset, experiments were conducted using the processed data with the representative object detection model YOLO v7 [18] and the instance segmentation model Mask R-CNN [19]. The final dataset comprised 89,766 images, divided into training, validation, and test subsets for comprehensive evaluation.

The resulting dataset can be widely applied to earthwork, road, and land development, and has been made publicly available on the Mendeley Data platform for use by researchers in related fields [20]. It holds significant potential as a public dataset for AI-based safety management systems, contributing to the protection of construction workers and supporting the reduction of time and costs in construction site safety management processes and accident mortality rates. This study also discusses the object selection process, field data collection methodology, and quantitative and qualitative evaluations of the dataset. Furthermore, it discusses the potential applicability of the dataset for developing general-purpose AI learning systems across various applications.

2. Establishment and Operation of a System to Secure Image Data at Earthwork Construction Sites

2.1. Site Selection for Effective Image Collection

The image collection system used in this study was selected for the Incheon land development site, considering the diversity of earthwork construction sites, suitability of equipment installation, and ease of acquiring high-quality images. Figure 1 shows the construction site status map, highlighting the positions of the installed CCTV cameras and the primary observation points (P1–P8). To capture a comprehensive view of the site, the image collection system was installed in a high-altitude area approximately 20 m above ground level, as shown in the left panel of Figure 1. Moreover, the panoramic view obtained from the installed equipment, shown in Figure 2, provides wide-angle footage with minimal occlusion—an essential advantage for effective image data collection.

2.2. Configuration and Observation Coverage of the PTZ-Based Image Acquisition System

In this study, a PTZ-based image acquisition system was implemented to capture high-resolution images across a large-scale earthwork construction site. The selected device, featuring up to 30× optical zoom, enabled detailed monitoring of distant targets and ensured stable performance under the challenging communication conditions typical of construction environments. The device was also compatible with various wireless protocols, including 3G, 4G, Bluetooth, and Wi-Fi.

To assess and visualize the operational coverage of the installed system, concentric circles with radii of 0.25 km, 0.5 km, 0.75 km, and 1.0 km were overlaid on the site map, as shown in Figure 1. These intervals were derived from empirical testing of the device’s 30× optical zoom, which confirmed effective image acquisition up to approximately 1.0 km under site conditions. This visualization was designed to illustrate the approximate monitoring range and clarify the spatial relationships between the camera, observation points, and major engineering zones.

The locations of key construction areas and observation targets were annotated to aid interpretation of spatial coverage and to provide contextual understanding of target distances. This setup supports the evaluation of detection performance relative to spatial variability within the site.

Detailed technical specifications of the deployed device are listed in

Table 1. Detailed specifications of the deployed CCTV camera.

Category	Specification
Manufacturer	KEDACOM
Image Sensor	1/2.8″ CMOS
Pixels	2.0 Megapixel
Focal Length	4.5~135 mm, 30× optical
Max. Aperture Ratio	F1.6 (WIDE)/F4.4 (TELE)
Pan/Tilt Range	360°/−15°~90°
Compression	H.265/H.264
Bitrate	64 kbps~16 Mbps
Wireless	3G/4G/Bluetooth/Wi-fi
Zoom	30× optical zoom

.

2.3. Operation Process of the Image Collection System

Based on the topographical characteristics of the site and the distribution of active construction equipment, eight observation areas were strategically selected. Observation points P1 to P4, located within the blue circular area in Figure 1, were selected because active earthwork operations such as filling and cutting were concentrated in this zone. Meanwhile, points P5 to P8, within the green circular area, were chosen due to frequent equipment movements and transport activities, making them suitable for capturing diverse operational scenes. These locations are marked with ‘x’ symbols in Figure 1. The image collection system was configured to automatically capture each observation point in sequence at regular frame intervals using its coordinate control function. Captured images were transmitted in real time via an LTE communication network and stored on a network-attached storage (NAS) system.

Figure 3 presents sample images from each observation point, with each point corresponding to specific construction zones within the site development project where key target objects were operated in real time.

3. Definition of Target Objects and AI Data Annotation Format at Earthwork Construction Sites

3.1. Selection of Objects at Earthwork Construction Sites

The hazardous objects to be detected at the earthwork construction site were identified based on the Guidelines for Performing Safety Management Tasks in Construction Projects and the Enforcement Decree of the Construction Machinery Management Act [21], with reference to the hazardous object profile and construction information classification system.

In conclusion, 15 hazardous object categories—consisting of 10 types of construction equipment and 5 types of terrain features—were identified and organized as presented in

Table 2. Classification of target objects.

Type of Work	Object Type	AI Application	Target Objects	Grouped Objects
Earthwork	Terrain	Instance Segmentation	Slope	Soil slope Rock slope
			Excavated slope
			Rock slope
			Cut slope
			Soil slope
			Soil mound	Soil mound
			Rock mound	Rock mound
	Construction equipment	Object Detection	Excavator	Excavator
			Backhoe
			Dump truck	Dump truck
				Loader
				Bulldozer
				Roller
Ground improvement and Reinforcement			Crawler drill	Crawler drill
Piling			Earth auger
			Pile driver	Pile driver
Construction of steel structure			Tower crane	Crane
Civil plumbing and drainage			Mobile crane
Etc	Etc		Car	Car
			Worker	Worker
			Work area	Work area

. To improve consistency and simplify annotation, similar equipment types were grouped under unified categories. For instance, excavators and backhoes were categorized as excavators, crawler drills and earth augers as crawler drills, and tower and mobile cranes as cranes. Similarly, various slope types were grouped into two categories: soil slope and rock slope.

Figure 4 shows schematic diagrams illustrating the labeled results for the target objects listed in Table 2, with only the relevant object parts extracted from the entire images. Panels (a) to (i) show construction equipment; panels (j) to (m) present terrain objects; and panels (n) and (o) present other objects.

3.2. AI Data Labeling for Earthwork Construction Sites

In computer vision, object recognition methods are generally divided into two categories: object detection and segmentation. Object detection involves marking the region around objects in an image with bounding boxes (rectangles), while segmentation identifies objects using polygons that outline their precise shapes. For AI training, the dataset must include the original images and their corresponding labels. In this study, the COCO Annotator [21], an open-source labeling tool, was used to simultaneously generate object detection and segmentation labels. The metadata was saved in the widely used COCO format [22].

Figure 5 provides an example of the metadata, where the metadata file name matches the image file name. The metadata includes the image file name (filename), width (width), height (height), class name of the objects, bounding box coordinates (x_min, y_min, x_max, y_max), and segmentation coordinates.

4. Deep Learning-Based Dataset Validation Experiment

4.1. Current Status of Earthwork Construction Site Learning Data

This study processed a total of 89,766 images for object detection, derived from an earthwork construction site. From these, 7600 images were selected to generate data for object segmentation.

Table 3. Number of images per viewpoint by data type.

	Viewpoint
	View 1	View 2	View 3	View 4	View 5	View 6	View 7	View 8
Number of images (Object detection)	10,305	5262	15,161	50,629	262	1955	517	5675
Total images	89,766
Number of images (Segmentation)	866	442	1270	3562	262	164	517	477
Total images	7600

presents the number of images at each of the eight observation points, categorized by the types of object detection and segmentation data. The data distribution indicates that the number of images at specific observation points (View1–View4) tends to be relatively skewed, as the images were primarily captured in areas where construction equipment was actively in use.

Figure 6 shows the current distribution of objects in the object detection and segmentation data by category. During the data collection period, the primary construction activities on site were earthworks and excavation. This led to the inclusion of a significant number of construction equipment specially designed for these tasks. Excluding regular vehicles, the highest inclusion rates were observed for excavators, dump trucks, and bulldozers, in that order.

Figure 7 illustrates the segmentation status of the object detection dataset, categorized by object type. Data analysis revealed that some objects had a small sample size, resulting in a data imbalance. The dataset was split into an 8:2 ratio, ensuring that all object categories were represented in the training and test datasets. Consequently, the training set contained 71,812 images, while the experimental set comprised 17,954 images.

Figure 8 presents the distribution of the object segmentation dataset by object category. Data analysis revealed that specific objects, such as rollers, loaders, and pile drivers, were not represented in the processed images, resulting in the inability to construct data for these items. Owing to these discrepancies in data availability across object categories, the entire dataset was split into training and test sets in an 8:2 ratio, ensuring that all major objects were represented in the training and evaluation phases. Consequently, the training dataset consisted of 6048 images, while the test dataset contained 1512 images.

4.2. Deep Learning Models

4.2.1. You Only Look Once (YOLO) Model for Object Detection Data Learning

Deep learning-based object detection methods are generally categorized into two approaches: one-stage and two-stage detectors.

One-stage detectors, such as YOLO [23,24], SSD [25], and RetinaNet [26], perform object localization and classification in a single forward pass of the network. By directly predicting object classes and bounding boxes from the input image without a separate region proposal step, these models achieve high inference speed, making them particularly suitable for real-time applications. Among them, the YOLO (You Only Look Once) series is notable for prioritizing processing speed by focusing on object areas defined by bounding boxes. While early versions of YOLO were limited in accuracy, recent variants, including YOLOv3 [24] and YOLOv7 [18], have achieved significant improvements in detection precision while maintaining real-time performance.

In contrast, two-stage detectors adopt a more sequential process. Models like R-CNN [27], Fast R-CNN [28], and Faster R-CNN [29] first generate region proposals that likely contain objects, and then perform classification and bounding box refinement on these candidate regions. This two-step process allows for more accurate localization and better incorporation of contextual information, which often results in superior detection accuracy compared to one-stage methods—albeit at the cost of slower inference speed.

This study adopted the YOLO v7 model owing to its ability to provide real-time object detection with high learning efficiency. Object detection accuracy was evaluated using the training dataset created for the model. This aligns with the study’s goal of enabling real-time analysis in dynamic environments such as construction sites.

Figure 9 illustrates the architecture of the YOLO object detection model, which processes an input image of size 448 × 448 × 3 through a deep convolutional neural network. The initial layers consist of a 7 × 7 convolution with 64 filters, followed by a 2 × 2 max pooling layer. This configuration initiates the feature extraction process by reducing the spatial resolution while capturing local patterns in the image. Subsequently, the network passes through a series of 3 × 3 and 1 × 1 convolutional layers. These layers serve to increase both the depth and abstraction of the extracted features while preserving essential spatial information. Each group of convolutional layers is followed by a max pooling operation, which progressively reduces the spatial dimensions and enables the network to focus on increasingly complex patterns.

During the final stages, the model employs three convolutional layers, each with 1024 filters, to capture high-level semantic representations with large receptive fields. These layers enhance the model’s ability to detect objects across various scales and contexts.

The resulting feature map is then flattened and passed through a fully connected layer comprising 4096 units. Finally, an output layer with 30 units encodes the object location (bounding box coordinates), objectness score, and class probability for each grid cell in the image. This architecture allows YOLO to perform real-time object detection by simultaneously predicting multiple objects within a single forward pass of the network.

The YOLOv7 model adopted in this study represents the most recent advancement in the YOLO object detection series, which is designed to strike an optimal balance between accuracy and computational efficiency. Compared to its predecessors, YOLOv7 introduces several architectural enhancements that significantly improve training speed and detection performance. These include the integration of E-ELAN (Extended Efficient Layer Aggregation Network) blocks, re-parameterized convolutional layers, and auxiliary heads. Such innovations contribute to better feature representation, increased training stability, and higher detection precision, all while maintaining a lightweight architecture suitable for real-time applications. The use of architectural re-parameterization also enables different configurations during training and inference, further optimizing the model’s performance under practical deployment conditions.

4.2.2. Mask R-CNN Model for Instance Segmentation Data Learning

Figure 10 illustrates the model architecture of Mask R-CNN [19], a representative two-stage deep learning framework designed for both object detection and instance segmentation. Built upon the Faster R-CNN architecture, Mask R-CNN introduces an additional parallel branch to produce pixel-level binary segmentation masks for each detected object, in addition to the conventional object classification and bounding box regression tasks.

In the first stage, a Region Proposal Network (RPN) extracts candidate regions of interest (RoIs) where objects are likely to exist by scanning convolutional feature maps generated by a backbone CNN (e.g., ResNet-FPN). These proposed RoIs are then refined in the second stage, where three parallel tasks are performed: (1) object classification, (2) bounding box refinement, and (3) mask prediction. The segmentation branch generates a fixed-size mask for each RoI independently, which is later resized and mapped back to the original image space.

A notable innovation in Mask R-CNN is the RoI Align technique, which replaces the previous RoI Pooling method. RoI Align preserves spatial alignment by using bilinear interpolation instead of quantization, significantly improving the accuracy of the generated masks—especially for small or overlapping objects.

This architectural design allows Mask R-CNN to precisely capture both the position and shape of objects at the pixel level. This is particularly beneficial in construction environments, where machinery and terrain often exhibit complex boundaries, occlusions, and variability in scale. Thus, Mask R-CNN enables reliable and high-resolution instance segmentation even under challenging visual conditions.

4.3. Dataset Standardization Through Deep Learning Training

4.3.1. Evaluation of Object Detection Result

To assess the performance of the object detection model, the mean average precision (mAP) was used as the primary evaluation metric. mAP is derived from two key components: precision and recall, as defined in Equations (1) and (2). Precision refers to the ratio of correctly predicted true instances to the total instances predicted as true by the model. Recall is the ratio of correctly predicted true instances to all actual true instances.

Ultimately, a precision–recall curve is plotted, with precision on the Y-axis and recall on the X-axis. The AP is calculated as the area under the curve [30,31,32].

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P})}}$$

(1)

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l} ={\displaystyle \frac{\mathrm{T}\mathrm{P}}{(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N})}}$$

(2)

Figure 11 presents the per-object mAP evaluation results for the object detection dataset. Overall, construction equipment classes such as dump trucks, rollers, and excavators achieved high detection performance, with mAP scores exceeding 80%. This can be attributed to their well-defined geometric structures, rigid shapes, and clear visual boundaries, which facilitate reliable feature extraction and classification by the detection model. In contrast, terrain and background-related objects—including rock slopes, soil mounds, and work areas—exhibited comparatively lower mAP scores. This performance disparity is likely caused by several factors: ambiguous object contours resulting from irregular and unstructured shapes, high intra-class variability, visual similarity between classes (e.g., rock versus soil), and imbalanced training data distributions. These results highlight that object distinctiveness, visual separability, and annotation quality substantially affect detection accuracy. To address these limitations, future efforts should focus on enhancing annotation consistency, applying class-specific data augmentation, and refining class definitions, particularly for terrain and environmental categories that pose greater challenges for detection models.

4.3.2. Evaluation of Instance Segmentation Result

For instance segmentation, the evaluation process is similar to that of object detection, with mAP based on bounding boxes being assessed. However, it also focuses on pixel-level accuracy to evaluate the precision of object boundary segmentation. Therefore, the evaluation metric used is mAP@IoU [33], which incorporates the intersection-over-union (IoU) threshold, as defined in Equation (3).

$$\mathrm{m}\mathrm{A}\mathrm{P}@\mathrm{I}\mathrm{o}\mathrm{U} ={\displaystyle \frac{1}{\mid \mathrm{C}\mid }}\sum _{\mathrm{c}=1}^{\mid \mathrm{C}\mid }{\displaystyle \frac{1}{\mid \mathrm{T}\mid }}\sum _{\mathrm{t}=1}^T{\mathrm{A}\mathrm{P}}_{\mathrm{c},\mathrm{t}}$$

(3)

AP is typically calculated across various IoU thresholds, with the average obtained to quantify overall segmentation performance. In this study, mAP@0.50 was used as the primary performance metric in accordance with the COCO evaluation standards.

Figure 12 compares the per-object mAP and mIoU performance for the instance segmentation dataset. Overall, the mAP, which reflects the accuracy of object localization, demonstrated relatively high values across most object categories. In contrast, the mIoU, which evaluates the pixel-level overlap between predicted and ground-truth masks, had a weaker performance. This indicates that while the model was generally successful in identifying the location and presence of target objects, it struggled to accurately delineate their boundaries.

These results suggest that segmentation errors are more prevalent in scenarios where object contours are ambiguous or where the distinction between objects and background is visually subtle. Such challenges are particularly common in terrain-related categories, where irregular shapes, overlapping textures, and inconsistent annotations can hinder precise segmentation. Furthermore, small objects or those with complex geometries may be partially segmented or misclassified, contributing to reduced mIoU scores.

It is also noteworthy that three object categories—roller, loader, and pile driver—were excluded from evaluation due to the absence of relevant instances in the dataset. This limitation points to an imbalance in the data distribution, which can lead to biased model training and insufficient generalization across underrepresented classes. Addressing these gaps through dataset augmentation and balanced sampling would be essential to improve the reliability and coverage of the segmentation model.

4.3.3. Visual Evaluation of Object Detection and Segmentation Data

Figure 13 visualizes the results of object detection and segmentation across 15 object classes defined in this study. Each object was detected using a model trained on original images captured at actual construction sites, with red boxes highlighting the corresponding objects.

Construction equipment objects, with their relatively clear shapes and distinct boundaries, were accurately detected and segmented, as confirmed by the object detection and segmentation results. By contrast, terrain-based objects, such as rock slopes, soil mounds, and work areas, exhibited unclear boundaries or similar backgrounds, leading to some omissions or errors in the segmented areas. This is consistent with the poorer performance observed in the evaluation metrics discussed earlier.

Overall, the visualization results are consistent with the numerical performance evaluation, demonstrating that the dataset constructed in this study is well-suited for training and validating object detection and segmentation models for construction sites.

5. Conclusions

This study aimed to create a CCTV-based image dataset to serve as foundational data for AI-driven construction safety management, with the goal of preventing industrial accidents and evaluating the feasibility and performance of object detection and segmentation models in construction-specific environments. To this end, images were collected over approximately one year from large-scale civil engineering land development sites, resulting in a COCO format-based dataset containing 15 classes of construction equipment and terrain objects.

To ensure reproducibility and provide a standardized benchmark, this study employed publicly available object detection and segmentation models (YOLOv7 and Mask R-CNN) using default parameters, without any custom modifications or hyperparameter tuning. YOLOv7 achieved a mean average precision (mAP) of 0.85 for object detection, and in particular, it excelled in categories with distinct shapes such as excavators and bulldozers. Mask R-CNN attained a mean average precision at IoU = 0.50 (mAP@0.50) of 0.78 and a mean intersection-over-union (mIoU) of 0.62 for instance segmentation, indicating reasonable performance despite the challenges of segmenting terrain objects with ambiguous boundaries.

The key innovations of this study include (1) long-term, real-site data collection from earthwork environments; (2) dual-format annotation supporting detection and segmentation tasks; (3) reproducible benchmark evaluation using open-source models with no tuning; and (4) open access publication of the dataset via Mendeley Data [1], facilitating broader adoption in both research and industry. These contributions are expected to accelerate the development of intelligent construction safety technologies and support ongoing digital transformation efforts in the construction sector.

One key limitation observed in this study was the imbalance in object category distribution, especially for segmentation tasks. In addition, although the dataset was constructed at a large scale, it was collected from a single earthwork construction site, which may limit the model’s generalization performance when applied to other types of construction environments.

Future research should expand the dataset to include diverse types of construction sites and validate the model across multiple project environments to enhance its generalizability. Additionally, efforts should be made to improve model robustness under variable conditions such as low-light environments and adverse weather, as well as to explore dynamic analysis tasks such as object tracking and behavior recognition.