Application of LMM-Derived Prompt-Based AIGC in Low-Altitude Drone-Based Concrete Crack Monitoring

Abstract

In recent years, large multimodal models (LMMs), such as ChatGPT 4o and DeepSeek R1—artificial intelligence systems capable of multimodal (e.g., image and text) human–computer interaction—have gained traction in industrial and civil engineering applications. Concurrently, insufficient real-world drone-view data (specifically close-distance, high-resolution imagery) for civil engineering scenarios has heightened the importance of artificially generated content (AIGC) or synthetic data as supplementary inputs. AIGC is typically produced via text-to-image generative models (e.g., Stable Diffusion, DALL-E) guided by user-defined prompts. This study leverages LMMs to interpret key parameters for drone-based image generation (e.g., color, texture, scene composition, photographic style) and applies prompt engineering to systematize these parameters. The resulting LMM-generated prompts were used to synthesize training data for a You Only Look Once version 8 segmentation model (YOLOv8-seg). To address the need for detailed crack-distribution mapping in low-altitude drone-based monitoring, the trained YOLOv8-seg model was evaluated on close-distance crack benchmark datasets. The experimental results confirm that LMM-prompted AIGC is a viable supplement for low-altitude drone crack monitoring, achieving >80% classification accuracy (images with/without cracks) at a confidence threshold of 0.5.

1. Introduction

A large multimodal model (LMM, an extensive LLM, large language model) is a type of artificial intelligence model (generative artificial intelligence) that can support humans in communicating with computers using natural language or other modalities (images, sounds). In recent years, LMMs (i.e., ChatGPT from OpenAI and Tencent-interfaced DeepSeek) [1,2] have been applied in different engineering fields (remote sensing and civil engineering) [3,4,5]. During the process of LMM-based application, the method or approach of “communicating with computers” is different from that of humans, requiring more detailed and specific keywords (prompts) to help the LMM to understand the conversation content. The approach of communicating with computers to obtain the desired output is termed “prompt engineering”.

Besides LMM, some image generation models (i.e., DALL-E and Stable Diffusion) have been applied in practical projects and have increased the possibility of generating the images (i.e., AI-generated content, AIGC or synthetic data) for data augmentation, even when researchers have not encountered such data before [6,7,8,9]. The similar features between real world and AIGC lead to the possible application, i.e., color, shape and texture. Nevertheless, sometimes the generated data have some implausible features, in that the features do not match the real-world standards and rules (i.e., hallucinations), and how to avoid generating these data is becoming one of the important themes for researchers in AI-related fields [10,11,12].

Considering the potential in civil engineering, the authors have explored applying this new technology in practical drone-based applications (i.e., waste detection from drone view) with the assistance of You Only Look Once (YOLO) [6,7,8,9]. Concurrently, more experiments are needed to apply AIGC effectively. Additionally, wider application scopes should be considered, and their feasibility must be demonstrated.

Until now, crack detection and segmentation have evolved significantly from manual inspections and traditional image processing techniques to advanced deep learning approaches utilizing both real-world and synthetic data. Manual methods are time-consuming and costly, while early automated techniques like edge detection and thresholding struggle with environmental variability such as shadows, dust, and low contrast. Deep learning models, particularly CNNs (e.g., U-Net), now dominate the field, offering high accuracy in pixel-level segmentation [13,14,15]. To address scarce real-world data, synthetic data generated via the CutMix technique [16,17] has been used to improve the generalization capability of CNN classifiers. Here, it is repurposed to generate a dataset that is a mix of a public labeled dataset and the background information of the target dataset. Key unresolved challenges include computational inefficiency, lack of dataset, dataset bias, and insufficient photorealism in synthetic data.

In previous studies [18,19,20,21,22,23], as shown in Figure 1a (upper), the drone-based crack detection process always requires close-distance on-site crack image data collection to train the YOLO model. Correspondingly, manual annotation requires engineers, researchers, and operators to go to the field to observe cracks and annotate them for correct model training. Figure 1a (upper) includes eight general steps, as follows:

• Step 1 (a, b): Engineers, researchers, and operators go to the field to observe and confirm the crack distribution situation (on-site close-distance, drone view).
• Step 2 (a, b): Collect crack image data (on-site close-distance, drone view).
• Step-3: Annotating the crack image data.
• Step 4: Train the YOLO model using paired crack images and annotations.
• Step-5: Saving the trained YOLO mode.
• Step-6: Transferring the drone-view data to infer using the trained YOLO model.
• Step 7: After inference, engineers can determine the location and size of cracks, and consider corresponding repairs.
• Step 8: Researchers can save crack distribution maps for further research.

The novel process in this work, as shown in the lower part of Figure 1a, includes two steps that differ from the previous work, as follows:

• Step 2 (a): Apply Stable Diffusion to generate images. During this process, systematic prompt application should be discussed, and reasonable prompts can reduce on-site work.
• Step-3: Using the trained model to annotate the generated crack image data.

Correspondingly, the authors aim to demonstrate the feasibility of the novel process in this research. As shown in Figure 1b, there are three sections as follows:

• Section 1: Train the ‘Visible Crack Dataset’ [24] (MIT License) using YOLOv8 as the annotation generator, and use low-altitude drone-view crack monitoring datasets, namely the Concrete Crack Image for Classification Dataset (CCI4CD) [25,26] (CC BY 4.0 License) and the Concrete Crack Segmentation Dataset (CCSD) [27] (CC BY-SA 4.0 License), for evaluation metrics.
• Section 2: Use the LMM (DeepSeek R1 with MIT license) to generate prompts for Stable Diffusion (v1.4 with CreativeML Open RAIL-M License) and collect the AIGC dataset (images and annotations).
• Section 3: Train the AIGC dataset and use CCI4CD and CCSD for evaluation metrics.

The authors apply the LMM to understand the range of the image generation (i.e., color, texture, scene, and photography style) and use prompt engineering to summarize and categorize them. With the assistance of LMM-generated prompts for AIGC, the synthetic data can be used as input for training the YOLOv8-seg model [28]. Based on the requirements of low-altitude crack monitoring, as a supplement, the authors tested the trained YOLOv8-seg models (i.e., Trained Model-1 and -2) on crack benchmark datasets (CCI4CD and CCSD).

2. Methods

2.1. YOLOv8 (Visible Cracks)

You Only Look Once (YOLO) is a one-stage model known for its fast inference speed and comparatively high detection accuracy. The YOLOv8 series comprises five scaled variants (nano-n, small-s, medium-m, large-l, and extra large-x), progressing from smallest to largest in size and complexity. It supports three task modes: detection, segmentation, and classification, applicable to training, validation, and inference. For crack segmentation (i.e., precise localization and background separation), the YOLOv8-seg model was selected. The Visible Crack Dataset (MIT License) [12] was used as the training data for YOLOv8-seg. As shown in

Table 1. Parameter setting (hardware).

Specification	Details
OS	Ubuntu 22.04.4 LTS
CUDA Version	12.4
NVIDIA-SMI	550.144.03
GPU	NVIDIA GeForce RTX 3090

and

Table 2. Parameter setting (YOLOv8x-seg, Visible_Cracks).

Specification	Details
Task	Segment
Model	yolov8x-seg
Early Stopping Patience	100
Batch size	16
Imgsz	640 pixels
Epochs	500
Optimizer	Auto
Lr0	0.01
Momentum	0.937

, the authors used the parameter setting as follows to train YOLOv8x-seg with the name of “Trained Model-1”. Figure 2 dictates the process of training and validating the model with the middle-process early stop. Trained Model-1 achieved 0.6 mAP50 (M) and 0.3 mAP50-95 (M) on the validation set, individually.

Figure 3 showed the label distribution statistics of the training process, which include instances numbers, central point location (x-y), and crack size (width-height). Figure 4 showed the samples of the batch images used for validation, which include different types of cracks, i.e., longitudinal, vertical, alligator. Among these crack-related types, alligators are the most difficult to identify or annotate using the instance segmentation annotation approach. Especially when the cracks are closed, the instance segmentation result can only extract the whole area without separating the individual cracks. After training the model, Trained Model-1 was saved as one best.pt file to be used as the annotation generator.

2.2. LMM

After training the annotation generator (YOLOv8-seg), Large Multimodal Models (LMMs) were employed to analyze image attributes such as color, texture, scene, and photography style. As shown in Figure 5, the authors used YuanBao [2] from Tencent as LMM in this research. Before the conversation with LMM, the user should prepare one image and corresponding question (prompt) about this image.

It is better to provide the image without any cracks, and it can also make the necessary information of the background available. The more detailed and theme-related the questions the users asked were, the much closer the answer was to what the user really needed. In this research, the author provided one image without any cracks selected from CCI4CD Negative Part. Correspondingly the prompt is “Firstly, please tell me what is this picture, from texture and color perspective?”.

During the process of answering the question, if the “deep think” option was chosen, the LMM should consider the purpose of the question in a deeper way, which can also lead the users to provide more detailed information. Incidentally, the LMM in “deep think” mode can ask and answer itself at the same time, and it will be positive for the user to adjust the answer. And the “deep think” mode can show the detailed process of thinking step by step; during the process, the user can obtain some inspirations and improve the quality of the question.

As shown in

Table 3. Prompts (texture, color and summary).

Specifications	Details
Object	Straight cracks perpendicular to road, 3–10 mm width, clean edges, no branching, spaced 5–10 m apart
Texture	Character: Rough and irregular. Surface Quality: It appears tactile and uneven, suggesting a dimensional surface rather than being flat and smooth. Likely Material: The description suggests it resembles coated, painted, or plastered wall. Detail: There are no visible patterns or specific features (like grooves or tiles) beyond the overall roughness and irregularity. Its relatively fine-grained roughness, not large pebbles or deep cracks. Visual Effect: Creates a sense of understated depth and physicality due to the light interacting with the uneven surface.
Color	Hue: Light gray. Tone: Uniform. Your description emphasizes “overall tone uniform, no obvious color difference or pattern.” Saturation: Implied to be low saturation (true gray, not blue-gray or green-gray), reinforcing the neutrality described. Variation: Lack of significant color variation, striations, fading, or stains contributes strongly to the described simplicity and neutrality.
Summary	The image depicts a surface with a rough, irregular texture resembling a coated or plastered wall. The texture provides subtle depth without prominent patterns. This surface is uniformly covered in a light gray color, lacking variations in hue, tone, or saturation, resulting in a clean, neutral, and minimalist appearance.

, after “deep think”, there are four main prompt-related sections generated that can be categorized as follows, i.e., object (straight cracks perpendicular to road), texture (character, surface quality, likely material, detail and visual effect), color (hue, tone, saturation and variation) and summary. The scene (a coated or plastered wall) and photography style (clean, neutral, and minimalist appearance) are included in the summary. Table 3 was derived from the prompts as follows: Please summarize and categorize the mentioned content using object, texture, color and summary sections in one table. The content in Table 3 displays the understanding of the LMM in Figure 5, but it depends on the model version, if the model is open/close source and what dataset was used for training.

If the LMM was changed, a similar output of the prompts could not be promised. So, confirming the main sections should be considered first, which could lead the model to at least provide the information in the specified scope. The user can adjust the content for easier understanding if there is something unusual (i.e., hallucinations, which are a common kind of error often occurring in large language models), and after the adjustments the prompts can be directly applied in the image generation.

Depending on the image generation, negative prompts can be considered as an approach to improving the image generation accuracy. If the image generation derived from the provided prompts already matches the image generation needs, negative prompts are not required.

2.3. Stable Diffusion

As shown in

Table 4. Parameter settings (txt2img).

Specifications	Details
Sampling steps	150
Width/Height	512/512 pixels
Batch count	100 (maximum, default = 1)
Batch size	8 (maximum, default = 1)

, the authors applied the parameters set in the Stable Diffusion WebUI txt2img function to generate the image. The Stable Diffusion checkpoint used for image generation is sd-v1-4.ckpt [fe4efff1e1]. As shown in Figure 6a, there is a generated crack image sample with vertical, horizontal and alligator cracks. The background of the cracks was correctly generated and derived from the prompts. Simultaneously, the crack distributions are random and the sizes of the crack are also not same, which increased the diversity for img2img image generation.

As shown in

Table 5. Parameter settings (img2img).

Specifications	Details
Resize mode	Just resize (latent upscale)
Sampling steps	150
Switch at	0.8
Sampling method	DDIM
Schedule type	DDIM
Width/Height	512/512 pixels
Batch count	100 (maximum, default = 1)
Batch size	8 (maximum, default = 1)
CFG Scale	7 (default)
Denoising strength	0.75 (default)

, the authors adjusted some parameters settings from Table 4, and as shown in Figure 6b, a grid image including 800 pieces was generated under the supervision of the prompts used in Table 3, with all the individual images saved. From the generated grid image in Figure 6b, the background and cracks were well generated.

2.4. YOLOv8 (ConcreteCrackImage4Classification, CCI4C)

Based on Trained Model-1 and the parameters settings shown in

Table 6. Parameters settings (annotation generations).

Specifications	Details
Imgsz	512 pixels
conf	0.01

, the authors inferred the images in Figure 6b and saved the labels. As shown in Figure 7, three samples were chosen for clear observation of the annotation (from left to right, original, with and without bounding box, individually).

Figure 8 dictated a general perspective of the annotation (mask mode) using a grid image (Figure 6b). The authors used the parameters setting shown in

Table 7. Parameter settings (YOLOv8l-seg, SD_Cracks).

Specification	Details
Task	Segment
Model	yolov8l-seg
Patience	100
Batch size	32
Imgsz	512 pixels
Epochs	500
Optimizer	Auto
Lr0	0.01
Momentum	0.937

, AIGC images, and annotations to train the model (i.e., Trained Model-2).

As shown in Figure 9, Trained Model-2 maintained mAP50 (m) and mAP50-95 (m), around 0.06 and 0.015. And as shown in Figure 10, the crack instances are mainly concentrated under 0.2 width and height, i.e., 512 pixels × 0.2 = 102.4 pixels. Compared with the instances in Trained Model-1, the instances in Trained Model-2 were much smaller. In other words, when faced with the task of segmenting small cracks in detail, Trained Model-2 has advantages compared with Trained Model-1.

3. Results

After collecting the trained models, the authors tried to use open-source datasets to evaluate them individually. The first open-source dataset is called “Concrete Crack Images for Classification” (CCI4CD) [11,12], which includes two groups, positive and negative (with and without cracks), and 20,000 images in each group, independently. Each image has a uniform size, 227 × 227 pixels. As shown in Figure 11, Figure 12, Figure 13 and Figure 14, the authors applied Trained Model-1 and -2 on positive and negative crack images using different confidence values (0.1, 0.25 and 0.5), individually. If any crack was detected in a positive crack image, it was classified as correct without considering the area on each crack. If any crack was detected in a negative crack image, it was classified as incorrect.

Noteworthy, as shown in Figure 12 and Figure 14, negative crack image 00361 was detected as positive using both Trained Model-1 and -2 under confidence values of 0.1, 0.25 and 0.5, individually. From a visual perspective, there are some parts that seem to be likely cracks with shallow depth in image 00361. In other words, the standard for classifying the cracks using Trained Model-1 and -2 is much stricter than that of the operators of CCI4CD.

Table 8. Accuracy results of detecting positive and negative cracks under individual confidence using Trained Model-1.

Confidence	Accuracy (%) (Positive, Cracks)	Accuracy (%) (Negative, No_Cracks)
0.1	99.9 (19,992/20,000)	95.6 (19,123/20,000)
0.25	99.8 (19,953/20,000)	97.5 (19,493/20,000)
0.5	97.7 (19,545/20,000)	98.9 (19,776/20,000)

and

Table 9. Accuracy results of detecting positive and negative cracks under individual confidence using Trained Model-2.

Confidence	Accuracy (%) (Positive, Cracks)	Accuracy (%) (Negative, No_Cracks)
0.1	99.9 (19,996/20,000)	88.0 (17,598/20,000)
0.25	99.5 (19,899/20,000)	93.5 (18,698/20,000)
0.5	82.2 (16,442/20,000)	97.7 (19,531/20,000)

show the accuracy results of classifying positive or negative images using Trained Model-1 and -2, separately. Trained Model-1 has an advantage on both positive and negative crack images, the accuracy is stable, and all the values are over 0.95. In contrast, although Trained Model-2 has some disadvantages in the imbalance of detections between positive and negative crack images, when the confidence value changes, the accuracy value still maintains over 0.8 for both positive and negative crack image classification.

The second open-source dataset is called “Concrete Crack Segmentation Dataset” [24], with CC BY-SA 4.0 license, and includes 458 images. Each image has a uniform size, 4032 × 3024 pixels. As shown in Figure 15, there were three chosen samples for comparison using Trained Model-1 and -2. As shown in Figure 16 and

Table 10. Recall values derived from Figure 15 using a 0.01 confidence value and imgsz 4032 pixels.

	Trained Model-1	Trained Model-2
17	0.899	0.834
19	0.915	0.976
30	0.810	0.946

, the Recall value is derived from the TP and FN in the confusion matrix, i.e., TP/(TP + FN). Both models can obtain a Recall value over 0.8, and in the case of No.19 and 30, Trained Model-2 has a higher Recall value than Trained Model-1.

4. Discussions

The authors applied LMMs in this research to understand specific images, and they confirmed the range of the image generation, which included but was not limited to object (straight cracks perpendicular to road), texture (character, surface quality, likely material, details and visual effect), color (hue, tone, saturation and variation), scene (a coated or plastered wall) and photography style (clean, neutral, and minimalist appearance). After summarizing and categorizing the mentioned prompts using prompt engineering [29,30,31,32], the extracted information can be used for image generation directly. And with the assistance of AIGC, Trained Model-2 showed its advantages compared with Trained Model-1 in some cases of low-altitude drone-view crack monitoring.

Noteworthy, real-world-data-derived Trained Model-1 has no significant advantages compared with the AIGC-derived Trained Model-2, i.e., AIGC-derived data have similar features to real-world-data-derived data. As shown in Figure 3 and Figure 10, Trained Model-1 has around 1400 instances, much fewer instances than in Trained Model-2 (over 70,000 instances), which is a potential reason why Trained Model-2 works as well as Trained Modle-1. To some degree, the number of instances increasing can overcome some disadvantages derived from the lack of diversity. Alternatively, if necessary, AIGC can not only increase the quantity of the instances for training the model but it can also expand the diversity of the model. If the dataset under a specific scene has not been collected before or if the quantity of instances under this new scene is limited, at this point, AIGC can assist in improving it on purpose. In particular, when the prompts are well designed and controlled by prompt engineering, it is much easier to obtain the necessary data rather than performing on-site data collection, which requires drone flying operation and control of the light conditions.

Although AIGC showed its advantages in increasing the instances’ quantity and diversity, it is difficult to promise that the quality of the prompt outputs can be maintained each time the LMM is used. Alternatively, more prompt-related experiments of other scopes should be in consideration for understanding the common features in prompts and LLMs/LMMs [33,34,35,36,37,38,39].

In this research, the benchmark datasets are basically derived from the concrete cracks; nonetheless, there are also other types of surface texture (i.e., asphalt pavement, white wall) that exist in practical crack monitoring applications. To overcome the feature gap between different surface textures, AIGC can assist by increasing the data diversity and quantity through further model training.

Supplementary Material was also provided as follow for clearer understanding: Based on manual annotation (True Label) and automatic annotation (Predicted) comparisons detailed in Table S1 and Figure S1, the high recall but low precision—attributed to a low confidence threshold (0.01) for mask generation—led to over-annotation and primarily affected model training. As shown in Table S2 and Figure S2, results from Trained Model-2 indicated minimal differences in close-distance segmentation/classification across confidence thresholds. The authors emphasized annotation quality over hyperparameter tuning, and through prompt ablation studies (Figures S3 and S4), found Object+Texture+Color prompts most effective. Finally, synthesis image diversity evaluation against real reference data (Figure S5 and Table S3) confirmed excellent perceptual diversity based on reported metrics.

5. Conclusions

Trained Model-2 obtained from the AIGC dataset produced results (i.e., classification accuracy over 0.9 under 0.25 confidence; segmentation, Recall value around 0.8 under 0.01 confidence) in “Concrete Crack for Classification” and “Concrete Crack Segmentation Dataset”, individually. This proposes that the approach explored in this work has practical applications in low-altitude drone-view crack monitoring. In conclusion, the approach described in this study has the potential to reduce the resources required for on-site data collection and annotation in low-altitude drone-view crack monitoring applications, making crack monitoring systems more deployable in a variety of locales.

6. Future Works

Originally, the method in this research was designed for crack monitoring, which can be observed from low-altitude drone view. After proving the feasibility of the prompt-engineering applied in generating high-resolution low-altitude drone-view crack images, researchers can apply this technology in edge AI mini-computers to produce datasets that directly include images and annotations (automatically labeled). And the application of this edge AI supports indoor operators in reducing the most time-consuming workload, i.e., on-site image collection and annotation labeling. In the near future, researchers can connect prompt engineering with some other novel technologies (i.e., LLM, Text-To-Speech, and Automatic-Speech-Recognition) in drone-platform Edge AI devices without any operation handling. All the data collection and dataset generation can be completed in the Edge AI device, and more close-distance and high-resolution imagery by real-world drone-view data are necessary for validation.

Based on the possible application issues mentioned, relative prompts should be accumulated and applied under more practical scenes [35,36], especially around drone-related applications.

Besides the mentioned points, there are also several bulleted points that warrant further research, as follows:

• Crack-related prompts;
• Texture-related prompts;
• Color-related prompts;
• Scene-related prompts;
• Photography style-related prompts;
• Scalability and automation of crack image generation;
• Performance of AIGC across different environments;
• Integration of AIGC with existing crack classification/detection/segmentation systems;
• Multiple crack classification/detection/segmentation algorithms for verification;
• Multiple study sites/locations/backgrounds for verification.