Annotation-Efficient and Domain-General Segmentation from Weak Labels: A Bounding Box-Guided Approach

Abstract

Manual pixel-level annotation remains a major bottleneck in deploying deep learning models for dense prediction and semantic segmentation tasks across domains. This challenge is especially pronounced in applications involving fine-scale structures, such as cracks in infrastructure or lesions in medical imaging, where annotations are time-consuming, expensive, and subject to inter-observer variability. To address these challenges, this work proposes a weakly supervised and annotation-efficient segmentation framework that integrates sparse bounding-box annotations with a limited subset of strong (pixel-level) labels to train robust segmentation models. The fundamental element of the framework is a lightweight Bounding Box Encoder that converts weak annotations into multi-scale attention maps. These maps guide a ConvNeXt-Base encoder, and a lightweight U-Net–style convolutional neural network (CNN) decoder—using nearest-neighbor upsampling and skip connections—reconstructs the final segmentation mask. This design enables the model to focus on semantically relevant regions without relying on full supervision, drastically reducing annotation cost while maintaining high accuracy. We validate our framework on two distinct domains, road crack detection and skin cancer segmentation, demonstrating that it achieves performance comparable to fully supervised segmentation models using only 10–20% of strong annotations. Given the ability of the proposed framework to generalize across varied visual contexts, it has strong potential as a general annotation-efficient segmentation tool for domains where strong labeling is costly or infeasible.

1. Introduction

Semantic segmentation is widely used in various computer vision applications, including biomedical analysis [1], autonomous driving [2], and infrastructure inspection [3,4]. It performs dense, pixel-level classification to localize target regions such as tumors, road signs, or surface cracks. However, obtaining strong performance with deep learning models in this domain typically requires substantial manually annotated data, particularly for fully supervised models such as U-Net [1] and DeepLab [5]. Producing pixel-level annotations is labor-intensive, time-consuming, and prone to inconsistencies—especially for small or ambiguous structures (e.g., pavement cracks [6,7] and skin lesions [4,8]).

To alleviate the annotation burden, researchers have studied weakly supervised segmentation methods based on coarse labels—such as image-level tags [9], scribbles [10], points [11], or bounding-boxes [12]—instead of full supervision. Among these, bounding-box annotations offer a good compromise between annotation effort and spatial localization. However, models trained only from boxes often produce imprecise boundaries or over-segmentation, because they lack fine-grained guidance [13].

Meanwhile, semi-supervised and hybrid methods seek to combine weak and strong supervision to enhance segmentation quality. While promising, many of these methods still handle the two label types separately and do not enforce alignment during training [14]. Moreover, most approaches are designed for a single domain and may not generalize well to visually different tasks, particularly when domain-specific features vary in scale, texture, or structure.

To address these limitations, we propose an annotation-efficient and domain-general segmentation framework that integrates sparse bounding box annotations with a small subset of strong (pixel-level) labels in a unified model. At the core of our approach is a lightweight Bounding Box Encoder, which transforms box-level labels into multi-scale spatial attention maps. These attention maps steer the encoder stages of a ConvNeXt–U-Net backbone [15], helping the network attend to semantically relevant regions, even with limited strong supervision.

Our framework is evaluated on two distinct and challenging domains: road crack detection and skin cancer segmentation. The results indicate that our method achieves segmentation performance on par with fully supervised models using only 10–20% of strong annotations, and it generalizes well across domains. These findings support bounding box-guided attention as a cost-effective and robust supervisory signal for training segmentation models with limited annotations. The contributions of this paper are:

• A novel segmentation framework is proposed that combines weak bounding box annotations with a small subset of strong pixel-level labels in an end-to-end training pipeline. The overall structure of the framework is shown in Figure 1.
• A lightweight Bounding Box Encoder is introduced to convert binary box masks into multi-scale spatial attention maps, which guide the ConvNeXt-U-Net encoder to focus on semantically relevant regions.
• The framework significantly reduces annotation costs, achieving performance comparable to fully supervised models while requiring only 10–20% of strong labels.
• Extensive experiments across two structurally distinct domains—road crack detection and skin lesion segmentation—demonstrate the method’s generalizability and effectiveness in real-world, annotation-limited settings.

This work extends our conference paper [16] with a fuller description of the Bounding Box Encoder and training objectives, expanded cross-domain experiments, comprehensive ablations, and an efficiency analysis.

The rest of this paper is organized as follows: Section 2 reviews related work. Section 3 details the proposed segmentation framework. Section 4 describes the datasets and training setup. Section 5 presents experimental results and analysis. Finally, Section 6 concludes the paper and discusses future directions.

2. Related Work

Semantic segmentation with limited supervision is increasingly important due to the high cost of dense, pixel-level annotations. We summarize recent advances in three relevant areas: (i) weakly supervised semantic segmentation with coarse annotations, (ii) methods guided by bounding-box supervision, and (iii) frameworks for annotation efficiency and domain generalization.

2.1. Weakly Supervised Semantic Segmentation

Weakly supervised semantic segmentation (WSSS) aims to reduce the need for dense pixel-level annotations by utilizing lower-cost forms of supervision, such as image-level tags, points, or scribbles. Early approaches used image-level labels to train image classifiers and derive class activation maps (CAMs), which were then refined using saliency cues or post-processing techniques such as conditional random fields (CRFs) [9,14]. While effective at coarse localization of objects, these methods often struggle to delineate fine-grained boundaries, particularly for thin or discontinuous structures.

To improve spatial fidelity, researchers have explored scribble-based [10] and point-based [11] supervision strategies. Although these methods provide localized cues at minimal annotation cost, they often require additional spatial priors or heuristic region-growing schemes to complete object shapes. Bounding boxes offer a practical compromise, striking a favorable balance between annotation effort and spatial specificity [12]. Given this, box-based approaches are examined in more detail in Section 2.2.

The advent of vision foundation models such as the Segment Anything Model (SAM) [17] and Grounding-DINO [18] has opened new avenues for WSSS. These models enable prompt-driven segmentation and localization across diverse tasks. For example, Ravanbakhsh et al. [19] used SAM and Grounding-DINO to automatically convert bounding boxes into accurate segmentation masks, showing the potential of large-scale pre-trained models to reduce annotation burden.

2.2. Bounding Box-Guided Segmentation

Bounding box-guided segmentation offers a practical compromise between annotation efficiency and spatial supervision. Early box-guided pipelines such as BoxSup [12] and the GrabCut-based refinement by Khoreva et al. [13] convert box-level annotations into pseudo-masks, which are subsequently used as surrogate ground truth during model training. However, these pipelines are typically multi-stage, rely on handcrafted heuristics, and can be sensitive to initialization quality.

To address these limitations, recent efforts have focused on integrating box-derived priors directly into learning objectives. Wang et al. [20] introduced a multiple-instance-learning (MIL)-based tightness constraint, encouraging predictions to remain within box regions and improving spatial consistency. Other studies, such as BBAM [21], use detector-driven attribution maps to highlight discriminative pixels within bounding boxes, thus guiding mask prediction without requiring manual refinement.

Bounding box guidance has also been explored for 3D instance segmentation. For instance, Deng et al. [22] proposed Sketchy-3DIS, which initializes segmentation from coarse 3D boxes and iteratively refines instance masks by enforcing geometric consistency.

Unlike the above approaches, the method proposed in this work does not rely on pseudo-mask generation, external prompting, or multi-step refinement. Instead, we introduce a lightweight Bounding Box Encoder that maps binary box masks into multi-scale spatial attention maps. These maps are aligned (via resizing) with the encoder’s feature stages and applied through element-wise modulation to influence the encoding process. This design enables the network to focus on semantically relevant regions under weak supervision while supporting end-to-end optimization without auxiliary modules or post-processing steps.

2.3. Feature Modulation in Segmentation

Feature modulation is widely used in computer vision to bias models toward informative regions. Attention modules such as SE-Net [23] and CBAM [24] use channel and/or spatial attention to improve feature selectivity. In segmentation tasks, spatial priors have been used to strengthen weak supervision, as seen in [25,26]. Feature conditioning approaches inject external cues (e.g., masks or priors) into the backbone to guide learning.

Our method follows this line of work by modulating encoder features using spatial attention maps generated from bounding-box masks. Unlike prior work, our design injects attention early within the encoder via element-wise modulation, enabling end-to-end guidance without additional refinement stages.

2.4. Annotation-Efficient and Domain-General Segmentation

A complementary direction is to reduce annotation costs by combining weak and strong labels through semi-supervised or hybrid training paradigms. Several methods have investigated confidence-weighted fusion or label propagation from a small subset of fully labeled images. A recent study by Kweon and Yoon [27] proposed WISH, a unified framework capable of training from image tags, points, scribbles, and bounding boxes by leveraging promptable foundation models like SAM. Although flexible, WISH relies on pre-trained vision–language models and is not optimized end-to-end for a specific segmentation target.

In addition to annotation efficiency, generalization across domains remains a critical challenge. Many segmentation frameworks are tailored for either medical imaging or natural scenes and struggle to transfer due to changes in scale, color distribution, and structural complexity [28]. Domain adaptation methods attempt to mitigate this through adversarial learning or feature alignment, but they typically require access to both source and target domain data during training.

In contrast, our approach demonstrates strong applicability across two structurally different domains—road crack detection and skin lesion segmentation—without modification to the architecture or supervision strategy. By directly incorporating weak bounding-box cues into the encoder and fusing them with a small set of strong labels, our framework achieves competitive performance under constrained annotation budgets and across diverse data distributions.

3. Methodology

This section outlines our proposed annotation-efficient segmentation framework, which combines a small number of strong pixel-level annotations with auxiliary spatial cues derived from bounding box masks. The architecture follows an encoder–decoder design based on ConvNeXt, enhanced by an attention mechanism that modulates feature encoding. Strong annotations are used exclusively for supervision, while bounding boxes serve as auxiliary spatial context. We describe the problem setup, model components, and training strategy in the following subsections. Figure 2 illustrates the overall architecture and data flow.

3.1. Problem Formulation

We address the task of semantic segmentation under constrained supervision, where full pixel-level annotations are available only for a subset of the training data. Each image in the dataset is accompanied by a binary bounding box mask that provides weak localization cues. Formally, the training set is defined as $\mathcal{D}={\left\{(I_i,B_i,M_i)\right\}}_{i=1}^N$, where:

• $I_i\in {\mathbb{R}}^{H\times W\times 3}$ denotes the input RGB image,
• $B_i\in {\{0,1\}}^{H\times W}$ is a binary mask derived from bounding box annotations, available for all images,
• $M_i\in {\{0,1\}}^{H\times W}$ is a pixel-level segmentation mask, provided only for a subset of the data.

The objective is to learn a segmentation function $f_{\theta }:{\mathbb{R}}^{H\times W\times 3}\to {[0,1]}^{H\times W}$, parameterized by $\theta$, that predicts dense segmentation masks ${\widehat{M}}_i$ for unseen inputs. During training, bounding box masks are used to generate spatial attention maps that guide the encoder to focus on semantically relevant regions.

3.2. Model Components

Our framework adopts an encoder–decoder architecture and introduces a lightweight mechanism to integrate spatial cues derived from bounding box masks. It comprises three main modules: (i) a ConvNeXt-Base encoder for hierarchical feature extraction, (ii) a lightweight Bounding Box Encoder for generating spatial attention maps, and (iii) a U-Net-style decoder for dense mask prediction. This design is task-agnostic and has been validated across two structurally different domains: road crack detection and skin lesion segmentation, demonstrating its robustness to varying image modalities, textures, and object scales.

Relation to prior work. This article is an extension of our preliminary conference version [16], providing a fuller methodological description (including the Bounding Box Encoder and training objectives), expanded experiments on two domains (RCFD and ISIC 2018) with stronger baselines, comprehensive ablations, and an efficiency analysis.

3.2.1. Encoder: ConvNeXt-Base

We employ ConvNeXt-Base [15] as the encoder backbone to extract multi-level semantic features from input images. Designed with insights from both convolutional networks and transformers, ConvNeXt achieves strong performance on dense prediction tasks while maintaining computational efficiency. Its architecture includes large kernel convolutions, inverted bottlenecks, and layer normalization, making it particularly well-suited for segmentation problems involving fine structures and complex textures.

Given an input image $I\in {\mathbb{R}}^{H\times W\times 3}$, the encoder processes it through four sequential stages. Each stage reduces spatial resolution and increases feature abstraction via convolutional downsampling and stacked ConvNeXt blocks:

$$f_{\left(l\right)}={\mathrm{ConvNeXtStage}}_l\left(f_{(l-1)}\right),l=1,2,3,4,$$

(1)

with $f_{\left(0\right)}=I$. Each output $f_{\left(l\right)}\in {\mathbb{R}}^{C_l\times H_l\times W_l}$ captures features at a distinct semantic and spatial scale.

These hierarchical features are subsequently modulated using bounding box-driven attention maps before being passed to the decoder.

3.2.2. Bounding Box Encoder and Modulation

To incorporate auxiliary spatial cues, we introduce a lightweight Bounding Box Encoder, inspired by [29], which transforms binary bounding box masks into attention maps aligned with the encoder’s feature hierarchy. For each input image, its corresponding bounding box mask $B\in {\{0,1\}}^{H\times W}$ is first resized to match the resolution of each encoder stage.

The encoder applies a single $3\times 3$ convolution followed by a sigmoid activation to produce a spatial attention map:

$$a_{\left(l\right)}=\sigma \left({\mathrm{Conv}}_{3\times 3}\left(B_{\left(l\right)}\right)\right),l=1,2,3,4,$$

(2)

where $B_{\left(l\right)}$ denotes the resized mask at stage l, and $a_{\left(l\right)}\in {[0,1]}^{H_l\times W_l}$ is the corresponding attention map.

Each attention map $a_{\left(l\right)}$ is then broadcast and multiplied element-wise with the output feature map $f_{\left(l\right)}$ from the ConvNeXt encoder:

$${f^{\prime }}_{\left(l\right)}=f_{\left(l\right)}\odot a_{\left(l\right)},$$

(3)

where ⊙ denotes element-wise multiplication. This modulation highlights semantically relevant regions indicated by the bounding boxes while suppressing irrelevant background noise.

The modulated features ${f^{\prime }}_{\left(l\right)}$ are subsequently passed to the corresponding decoder stages, enabling attention-guided segmentation without relying on full pixel-level labels for the entire dataset.

3.2.3. Decoder: U-Net-Style Reconstruction

The decoder follows a U-Net-style design that progressively upsamples and fuses features from the encoder to reconstruct the segmentation mask at full resolution. It operates on the four modulated encoder outputs $\{f_1^{\prime },f_2^{\prime },f_3^{\prime },f_4^{\prime }\}\in \left\{{\mathbb{R}}^{C_l\times H_l\times W_l}\right\}$, where $C_l\in \{128,256,512,1024\}$ and $(H_l,W_l)$ represent progressively lower resolutions.

Let the decoder function be defined as follows:

$$d_{\left(l\right)}={\mathrm{DecoderBlock}}_l(u_l,f_{(l-1)}^{\prime }),\mathrm{for}l=3,2,$$

(4)

where $u_l$ is the upsampled feature from the previous decoder block, and $f_{(l-1)}^{\prime }$ is the skip-connected modulated feature from the encoder and Bounding Box Encoder at the corresponding scale. Decoder block 4 is initialized as:

$$d_{\left(4\right)}={\mathrm{DecoderBlock}}_4(f_4^{\prime },f_3^{\prime }),$$

(5)

and subsequent blocks fuse the output of the previous decoder with the next encoder feature. The final decoder block (block 1) produces the low-level decoder output $d_{\left(1\right)}\in {\mathbb{R}}^{64\times H\times W}$.

Each ${\mathrm{DecoderBlock}}_l$ consists of:

• Nearest-neighbor upsampling to double the spatial resolution.
• Concatenation with the corresponding encoder feature $f_l$ (except for the last block, which receives no skip).
• Two sequential $3\times 3$ convolution layers, each followed by BatchNorm and ReLU activation.

The final segmentation prediction is obtained by applying a $3\times 3$ convolution followed by bilinear upsampling by scale factor equal to 2:

$$\widehat{M}=\mathrm{Upsample}\left({\mathrm{Conv}}_{3\times 3}\left(d_{\left(1\right)}\right)\right),\widehat{M}\in {[0,1]}^{H\times W}.$$

(6)

This architecture enables spatially accurate mask reconstruction without a bottleneck, maintaining fine details critical for thin or discontinuous structures.

3.3. Training Strategy

The proposed model is trained using a hybrid loss that combines strong supervision from a limited set of pixel-level annotations with an auxiliary consistency loss based on bounding box masks. Let N be the total number of training images, among which only a subset of size $N_s$ is associated with strong pixel-level labels.

1.

Binary Cross-Entropy Loss.

For each strongly labeled image $i\in \{1,\dots ,N_s\}$, let $M_i\in {\{0,1\}}^{H\times W}$ denote the ground truth binary segmentation mask, and ${\widehat{M}}_i\in {[0,1]}^{H\times W}$ the predicted probability map. The binary cross-entropy loss (BCE) is defined as:

$${\mathcal{L}}_{\mathrm{BCE}}=-{\displaystyle \frac{1}{N_s}}\sum _{i=1}^{N_s}\left[M_{i}log{\widehat{M}}_i+(1-M_i)log(1-{\widehat{M}}_i)\right],$$

(7)

where $M_i$ is the ground-truth and ${\widehat{M}}_i$ is the esitmated output.

2.

Box-Mask Consistency Loss.

To reduce false positive predictions outside regions marked by bounding boxes, we introduce a consistency loss applied to all N training images. Let $B_i\in {\{0,1\}}^{H\times W}$ be the binary bounding box mask for image i, where 1 indicates the interior of the box and 0 denotes the background. The consistency loss penalizes activations outside the bounding box as:

$${\mathcal{L}}_{\mathrm{box}}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[(1-B_i)·{\widehat{M}}_i\right],$$

(8)

which encourages the model to suppress crack predictions in areas not supported by bounding box evidence.

3.

Total Loss.

The final training objective combines both terms:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{BCE}}+\lambda ·{\mathcal{L}}_{\mathrm{box}}.$$

(9)

Here, $\lambda \in [0,1]$ weights the Box–Mask Consistency loss relative to the supervised segmentation loss. Setting $\lambda =0$ removes the box term; setting $\lambda$ close to 1 makes the box term dominant and degrades performance by over-constraining predictions to the boxes. We select $\lambda =0.1$ based on a small validation sweep (cf. [30]) and observe stable performance for $0.05\le \lambda \le 0.2$ (see Appendix A,

Table A1. Sensitivity of performance to the loss weight $\lambda$ on the test split (20% strong, 100% weak; threshold $\tau =0.5$). Bold highlighting values denote the highest results.

$\mathit{\lambda }$	RCFD Dataset		ISIC 2018 Dataset
	IoU (%)	F1-Score (%)	IoU (%)	F1-Score (%)
0.00	67.40	80.40	86.60	92.60
0.05	68.20	80.90	87.00	92.90
0.10	68.73	81.30	87.30	93.10
0.20	68.40	81.05	87.10	93.00
0.50	65.02	78.80	85.40	91.80
0.70	63.80	77.60	84.70	91.10
1.00	61.90	76.10	83.60	90.20

).

4.

Strong–Weak Supervision.

In our setting, a subset of images carries pixel-level masks (strong labels, $N_s$), which supervise the primary segmentation loss (BCE) on those samples. Bounding boxes are available for all images (N) and provide guidance through the auxiliary Box-Mask Consistency term. During training (mini-batch size $B=4$), the BCE loss is averaged only over samples with pixel-level masks (strong labels), whereas the Box-Mask Consistency loss is averaged over all samples in the batch. If a batch contains no strong samples, the update is computed solely from the Box-Mask Consistency loss scaled by $\lambda$.

4. Datasets and Experimental Setup

This section presents the datasets used to evaluate the proposed framework, along with the data augmentation strategies employed during training. It also outlines the implementation details of the training setup and describes the evaluation metrics used to assess segmentation performance across tasks.

4.1. Datasets

We evaluate the proposed framework on two structurally distinct datasets: the Road Crack Full-Scene Dataset [31,32] (RCFD) for road crack detection, and the ISIC 2018 [33] dataset for skin lesion segmentation. Both datasets contain bounding box annotations and strong pixel-level masks. In each case, bounding boxes are converted into binary mask format for use in the bounding box encoder. Figure 3 provides representative examples from both datasets, including the original images, strong pixel-level annotations, and weak bounding box masks. Dataset statistics are summarized in

Table 1. Details of the crack and skin lesion segmentation datasets used in this study. All images are resized to 512 × 512 pixels to ensure uniformity and facilitate robust model training and evaluation.

Dataset	Image Resolution	Training Set	Validation Set	Test Set
RCFD (in-house) [31,32]	512 × 512	973	170	285
ISIC 2018 [33]	512 × 512	2594	100	1000

.

• RCFD Dataset: The RCFD [31,32] dataset contains 1428 full-scene RGB images of roads with visible cracks, captured in China, India, the Czech Republic, and Japan using drones, motorcycles, and dashboard-mounted cameras. The images cover a wide range of lighting, weather, and surface conditions. The dataset is split into 973 training, 170 validation, and 285 test images. For weak supervision, bounding box annotations from the source dataset were converted into binary masks aligned with the image resolution.
• ISIC 2018 Dataset: The ISIC 2018 Skin Lesion Segmentation Challenge dataset [33] comprises dermoscopic images of skin lesions acquired under diverse imaging conditions. We use a total of 3694 images, split into 2594 for training, 100 for validation, and 1000 for testing. Similar to the RCFD dataset, bounding box annotations were generated from lesion outlines and converted into binary mask format for integration with the bounding box encoder.

4.2. Data Augmentation

To enhance model generalization and mitigate overfitting, we apply a set of augmentations provided by the MMSegmentation framework [34]. These augmentations increase data diversity and help simulate real-world variations in lighting, orientation, and contrast:

• Resizing: All images and corresponding masks are resized to a fixed resolution of $512\times 512$ pixels to ensure consistent input dimensions for the network.
• Random rotations and flips: Using RandomRotFlip, each image undergoes random rotation (${90}^{\circ }$, ${180}^{\circ }$, or ${270}^{\circ }$) and horizontal/vertical flipping with a combined probability of 0.5.
• Photometric distortions: Includes random brightness and contrast adjustments, HSV perturbations (saturation and hue), and conversion back to RGB, all applied with a probability of 0.5.

Figure 4 presents sample augmented images, illustrating how these transformations enhance the model’s robustness to variations in lighting and imaging conditions.

4.3. Implementation Details

All experiments were conducted using the MMSegmentation framework [34]. Models were trained on an NVIDIA RTX 3090 GPU (24 GB) using the AdamW optimizer [35] with an initial learning rate of $1\times {10}^{-4}$. A two-phase learning rate schedule was used: linear warm-up for the first 3000 iterations (starting at $1\times {10}^{-6}$), followed by a polynomial decay until iteration 20,000. Input images were resized to $512\times 512$ with a batch size of 4.

All models were trained for 20,000 iterations on each dataset. The same configuration was used for both crack and skin lesion segmentation tasks, ensuring fair comparison across domains.

4.4. Evaluation Metrics

We evaluate segmentation performance using five widely adopted pixel-wise measures, each capturing a different aspect of prediction quality. Let $M\in {\{0,1\}}^{H\times W}$ be the ground-truth mask and $\widehat{P}\in {[0,1]}^{H\times W}$ the predicted probability map. We binarize at threshold $\tau =0.5$ to obtain $\widehat{M}=1[\widehat{P}\ge \tau ]$ and then compute pixel-level counts of true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN) by comparing $\widehat{M}$ with M.

• Intersection over Union (IoU): Quantifies the overlap between predicted and ground truth masks:

  $$\mathrm{IoU}={\displaystyle \frac{TP}{TP+FP+FN}}$$

  (10)

  where $TP$, $FP$, and $FN$ represent true positives, false positives, and false negatives, respectively.
• Recall: The proportion of actual positive pixels correctly identified:

  $$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

  (11)
  High recall indicates that most of the true positive pixels are detected, though it does not penalize false positives.
• Precision: The proportion of predicted positive pixels that are correct:

  $$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

  (12)
  High precision means fewer false positives, but it may come at the cost of lower recall.
• F1-score: The harmonic mean of precision and recall:

  $$\mathrm{F}1={\displaystyle \frac{2·\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

  (13)
  This balances precision and recall into a single metric.
• Mean Intersection over Union (mIoU): The average IoU calculated over all C classes:

  $$\mathrm{mIoU}={\displaystyle \frac{1}{C}}\sum _{c=1}^C{\displaystyle \frac{T{P}_c}{T{P}_c+F{P}_c+F{N}_c}}$$

  (14)
  For binary segmentation ($C=2$), this is the mean IoU of the foreground and background classes.

Unless stated otherwise, we report micro-averaged results, i.e., pixel counts are accumulated across all images before computing the measures.

5. Experimental Results and Discussion

This section presents a comprehensive evaluation of the proposed framework from two perspectives: (i) an ablation study assessing the contribution of each architectural component and the impact of varying proportions of weak and strong annotations, and (ii) a comparative analysis against state-of-the-art segmentation methods. Experiments are conducted on both crack detection (RCFD dataset) and skin lesion segmentation (ISIC 2018) datasets, enabling performance assessment under different annotation budgets and across structurally distinct domains.

5.1. Ablation Study

We perform an ablation study to quantify the contribution of the proposed bounding box encoder and to evaluate the effect of varying proportions of weak (bounding box) and strong (pixel-level) annotations. The baseline model consists of a ConvNeXt-Base encoder and a U-Net-style decoder trained exclusively with strong annotations. Our full model augments this baseline with the bounding box encoder, which modulates encoder features using spatial attention maps derived from weak masks.

Table 2. Ablation study results on the RCFD dataset. Weak = bounding box annotations, Strong = pixel-level annotations.

Model	Weak	Strong	IoU (%)	Precision (%)	Recall (%)	F1-Score (%)	mIoU (%)
Baseline (Encoder + Decoder)	–	100%	63.45	80.92	74.62	77.64	80.97
+ Bbox Encoder (Ours)	90%	10%	52.44	72.14	65.76	68.8	75.18
	80%	20%	53.12	70.29	68.5	69.38	75.50
	50%	50%	57.61	75.95	70.47	73.11	77.90
	100%	10%	61.97	75.75	77.30	76.52	80.15
	100%	20%	69.29	80.24	83.54	81.86	83.99
	100%	40%	71.35	81.75	84.86	83.28	85.07
	100%	60%	73.76	85.52	84.29	84.90	86.35
	100%	80%	74.62	84.72	86.72	85.47	86.79
	100%	100%	76.58	84.48	89.12	86.74	87.81

and

Table 3. Ablation study results on the ISIC 2018 dataset. Weak = bounding box annotations, Strong = pixel-level annotations.

Model	Weak	Strong	IoU (%)	Precision (%)	Recall (%)	F1-Score (%)	mIoU (%)
Baseline (Encoder + Decoder)	–	100%	79.35	92.35	84.94	88.49	85.31
+ Bbox Encoder (Ours)	90%	10%	81.1	88.25	90.93	89.57	86.26
	80%	20%	82.47	90.98	89.82	90.39	87.39
	50%	50%	81.02	92.01	87.15	89.51	86.43
	100%	10%	87.41	92.35	94.23	93.28	90.92
	100%	20%	87.95	91.99	95.24	93.59	91.29
	100%	40%	88.12	93.33	94.05	93.69	91.46
	100%	60%	88.82	93.02	95.16	94.08	91.94
	100%	80%	89.1	92.61	95.92	94.23	92.12
	100%	100%	89.47	92.6	96.35	94.44	92.38

report the results on the RCFD and ISIC 2018 datasets, respectively. For both datasets, incorporating the bounding box encoder improves performance across all metrics, even when the proportion of strong labels is small. We further examine annotation coverage by varying the strong-label ratio $percentag{e}_{strong}\in \{10\%,20\%,40\%,100\%\}$ and the weak-label (bounding-box) ratio $percentag{e}_{weak}\in \{50\%,80\%,90\%,100\%\}$, and report the resulting performance to quantify the trade-off between annotation cost and accuracy.

On the RCFD dataset, using only 10% strong annotations and weak masks for all images achieves an IoU of 61.97%, which is 1.48% lower than the fully supervised baseline (63.45%), and an F1-score of 76.52%, which is just 1.12% lower than the baseline (77.64%). The mIoU difference is minimal at −0.82% (80.15% vs. 80.97%). With 20% strong annotations, performance improves significantly to an IoU of 69.29% (+5.84%), F1-score of 81.86% (+4.22%), and mIoU of 83.99% (+3.02%) compared to the baseline. Further increases in strong annotations lead to consistent improvements, reaching 87.81% mIoU with full supervision.

A similar trend is observed on ISIC 2018. With only 10% strong annotations and full weak masks, our method achieves an IoU of 87.41% (+8.06%), F1-score of 93.28% (+4.79%), and mIoU of 90.92% (+5.61%) compared to the fully supervised baseline. Using 20% strong annotations, the IoU rises to 87.95% (+8.60%), F1-score to 93.59% (+5.10%), and mIoU to 91.29% (+5.98%). The best performance of 92.38% mIoU is obtained with 100% strong and weak annotations.

Overall, these results confirm that: (i) the bounding box encoder consistently boosts segmentation accuracy across IoU, Precision, Recall, F1-score, and mIoU; (ii) the proposed framework can achieve near full-supervision or better performance with as little as 10–20% strong annotations, reducing annotation cost without sacrificing quality.

5.2. Comparative Experiments to the State of the Art

We compare the proposed framework with fully supervised segmentation methods and recent semi-/weakly supervised approaches that utilize both weak (bounding box) and strong (pixel-level) annotations. Table 4 and Table 5 summarize the results for the RCFD and ISIC 2018 datasets.

For methods whose original papers did not report results on RCFD or ISIC 2018 (e.g., EfficientCrackNet [36], Self-Correcting [29], DCR [37], Strong-Weak [38], Xiong et al. [39]), we re-trained the authors’ implementations—when available—or faithful re-implementations on our data splits using the unified configuration described in Section 4.3 and the evaluation protocol in Section 4.4.

RCFD dataset: Among fully supervised methods, CrackRefineNet [32] achieves the highest score with 65.41% IoU, 79.09% F1-score, and 81.97% mIoU. Our framework, with only 10% strong labels and full weak annotations, achieves 61.97% IoU, 76.52% F1-score, and 80.15% mIoU—just 3.44% IoU and 2.57% F1 below CrackRefineNet despite using only one-tenth of the strong annotations, while outperforming the best mixed-supervision competitor (Self-Correcting [29]) by +13.83% IoU, +11.53% F1, and +7.23% mIoU.

Table 4. Comparison with state-of-the-art methods on the RCFD dataset. Weak = bounding box annotations, Strong = pixel-level annotations. Bold highlighting values denote the highest results.

Model	Supervision Type	Weak	Strong	IoU (%)	Precision (%)	Recall (%)	F1-Score (%)	mIoU (%)
U-Net [1]	Full	–	100%	48.82	58.42	74.82	65.61	73.03
MobileNetv3 [40]		–	100%	51.69	71.57	65.04	68.15	74.78
SwinT [41]		–	100%	53.94	73.84	66.68	70.08	75.97
EfficientCrackNet [36]		–	100%	35.47	39.42	77.93	52.36	65.24
CrackMaster [42]		–	100%	63.53	79.37	76.09	77.7	81.0
CrackRefineNet [32]		–	100%	65.41	79.2	78.98	79.09	81.97
Self-Correcting [29]	Mixed (Box + Full)	100%	10%	48.14	69.65	60.92	64.99	72.92
Macro-Micro [43]		100%	10%	40.35	63.16	52.78	57.5	68.82
DCR [37]		100%	10%	40.62	64.88	52.07	57.77	68.98
Strong-Weak [38]		100%	10%	24.39	26.25	77.47	39.22	57.97
Xiong et al. [39]		100%	10%	16.99	18.35	69.66	29.05	52.53
Ours	Mixed (Box + Full)	100%	10%	61.97	75.75	77.30	76.52	80.15
Ours		100%	20%	69.29	80.24	83.54	81.86	83.99

Table 4. Comparison with state-of-the-art methods on the RCFD dataset. Weak = bounding box annotations, Strong = pixel-level annotations. Bold highlighting values denote the highest results.

Model	Supervision Type	Weak	Strong	IoU (%)	Precision (%)	Recall (%)	F1-Score (%)	mIoU (%)
U-Net [1]	Full	–	100%	48.82	58.42	74.82	65.61	73.03
MobileNetv3 [40]		–	100%	51.69	71.57	65.04	68.15	74.78
SwinT [41]		–	100%	53.94	73.84	66.68	70.08	75.97
EfficientCrackNet [36]		–	100%	35.47	39.42	77.93	52.36	65.24
CrackMaster [42]		–	100%	63.53	79.37	76.09	77.7	81.0
CrackRefineNet [32]		–	100%	65.41	79.2	78.98	79.09	81.97
Self-Correcting [29]	Mixed (Box + Full)	100%	10%	48.14	69.65	60.92	64.99	72.92
Macro-Micro [43]		100%	10%	40.35	63.16	52.78	57.5	68.82
DCR [37]		100%	10%	40.62	64.88	52.07	57.77	68.98
Strong-Weak [38]		100%	10%	24.39	26.25	77.47	39.22	57.97
Xiong et al. [39]		100%	10%	16.99	18.35	69.66	29.05	52.53
Ours	Mixed (Box + Full)	100%	10%	61.97	75.75	77.30	76.52	80.15
Ours		100%	20%	69.29	80.24	83.54	81.86	83.99

When increasing to 20% strong labels, the performance reaches 69.29% IoU, 81.86% F1-score, and 83.99% mIoU, surpassing CrackRefineNet by +3.88% IoU, +2.77% F1, and +2.02% mIoU, and significantly outperforming all other baselines.

Table 5. Comparison with state-of-the-art methods on the ISIC 2018 dataset. Weak = bounding box annotations, Strong = pixel-level annotations. Bold highlighting values denote the highest results.

Model	Supervision Type	Weak	Strong	IoU (%)	Precision (%)	Recall (%)	F1-Score (%)	mIoU (%)
U-Net [1]	Full	–	100%	58.39	77.41	70.38	73.73	69.81
MobileNetv3 [40]		–	100%	78.41	89.68	86.18	87.9	84.49
SwinT [41]		–	100%	78.31	90.9	84.97	87.84	84.50
EfficientCrackNet [36]		–	100%	71.4	80.42	86.43	83.31	78.81
CrackMaster [42]		–	100%	81.81	92.15	87.94	89.99	86.99
CrackRefineNet [32]		–	100%	83.11	89.97	91.6	90.78	87.77
Self-Correcting [29]	Mixed (Box+Full)	100%	10%	74.81	86.8	84.41	85.59	81.79
Macro-Micro [43]		100%	10%	71.98	85.15	82.31	83.7	79.71
DCR [37]		100%	10%	73.56	86.81	82.82	84.77	80.94
Strong-Weak [38]		100%	10%	66.28	79.47	79.97	79.72	75.25
UCMT (U-Net) [44]		100%	10%	69.81	84.75	81.60	83.33	80.67
DSBD [45]		100%	10%	71.42	84.71	87.54	86.31	78.05
Xiong et al. [39]		100%	10%	65.0	82.56	75.34	78.79	74.77
EGE-UDSMT [46]		100%	10%	74.62	89.65	86.36	88.65	81.63
Ours	Mixed (Box + Full)	100%	10%	87.41	92.35	94.23	93.28	90.92
Ours		100%	20%	87.95	91.99	95.24	93.59	91.29

Table 5. Comparison with state-of-the-art methods on the ISIC 2018 dataset. Weak = bounding box annotations, Strong = pixel-level annotations. Bold highlighting values denote the highest results.

Model	Supervision Type	Weak	Strong	IoU (%)	Precision (%)	Recall (%)	F1-Score (%)	mIoU (%)
U-Net [1]	Full	–	100%	58.39	77.41	70.38	73.73	69.81
MobileNetv3 [40]		–	100%	78.41	89.68	86.18	87.9	84.49
SwinT [41]		–	100%	78.31	90.9	84.97	87.84	84.50
EfficientCrackNet [36]		–	100%	71.4	80.42	86.43	83.31	78.81
CrackMaster [42]		–	100%	81.81	92.15	87.94	89.99	86.99
CrackRefineNet [32]		–	100%	83.11	89.97	91.6	90.78	87.77
Self-Correcting [29]	Mixed (Box+Full)	100%	10%	74.81	86.8	84.41	85.59	81.79
Macro-Micro [43]		100%	10%	71.98	85.15	82.31	83.7	79.71
DCR [37]		100%	10%	73.56	86.81	82.82	84.77	80.94
Strong-Weak [38]		100%	10%	66.28	79.47	79.97	79.72	75.25
UCMT (U-Net) [44]		100%	10%	69.81	84.75	81.60	83.33	80.67
DSBD [45]		100%	10%	71.42	84.71	87.54	86.31	78.05
Xiong et al. [39]		100%	10%	65.0	82.56	75.34	78.79	74.77
EGE-UDSMT [46]		100%	10%	74.62	89.65	86.36	88.65	81.63
Ours	Mixed (Box + Full)	100%	10%	87.41	92.35	94.23	93.28	90.92
Ours		100%	20%	87.95	91.99	95.24	93.59	91.29

ISIC 2018 dataset: In the fully supervised setting, CrackRefineNet [32] achieves the highest score with 83.11% IoU, 90.78% F1-score, and 87.77% mIoU.

Among mixed-supervision methods using 10% strong labels, Self-Correcting [29] achieves 74.81% IoU, 85.59% F1-score, and 81.79% mIoU, while EGE-UDSMT [46] achieves 74.62% IoU, 88.65% F1-score, and 81.63% mIoU.

Our framework, with 10% strong labels and full weak annotations, achieves 87.41% IoU, 93.28% F1-score, and 90.92% mIoU—outperforming Self-Correcting by +12.60% IoU, +7.69% F1, and +9.13% mIoU, and surpassing EGE-UDSMT by +12.79% IoU, +4.63% F1, and +9.29% mIoU. Compared to the best fully supervised model (CrackRefineNet), our method improves by +4.30% IoU, +2.50% F1, and +3.15% mIoU, despite using only one-tenth of the strong annotations.

When increasing to 20% strong labels, our method further boosts performance to 87.95% IoU, 93.59% F1-score, and 91.29% mIoU, achieving the best results across all metrics. The IoU is +4.84% higher than CrackRefineNet, and the mIoU is +3.52% higher, demonstrating that the proposed framework scales effectively with additional strong labels while maintaining a substantial lead over both mixed- and fully supervised baselines.

These results show that the proposed framework not only bridges the performance gap between weakly and fully supervised segmentation but also outperforms state-of-the-art fully supervised models in multiple settings while using as little as 10–20% of strong annotations. This confirms the effectiveness of the bounding box encoder in leveraging weak supervision to achieve competitive and even superior performance across different domains and metrics.

5.3. Qualitative Analysis of Results

To further illustrate the effectiveness of the proposed framework, Figure 5 presents qualitative comparisons on both the RCFD and ISIC 2018 datasets. Each column shows the input image, ground truth mask, and segmentation outputs from representative state-of-the-art methods alongside our approach with 10% and 20% strong annotations. Blue regions correspond to true positives, red regions indicate false positives, and green regions mark false negatives, with the best IoU scores highlighted in yellow.

On the RCFD dataset, our method produces more precise crack delineations with fewer spurious predictions compared to baselines such as Strong-Weak [38] and Xiong et al. [39], which tend to generate noisy or fragmented masks. Even under limited supervision (10% strong), our framework captures fine crack structures and suppresses background noise more effectively than fully supervised models such as U-Net [1] and MobileNetv3 [40]. Increasing the proportion of strong annotations to 20% further refines the predictions, yielding continuous and complete crack regions that closely match the ground truth.

On ISIC 2018, lesion boundaries obtained by our method align more faithfully with ground truth masks compared to competing approaches. For example, macro–micro [43] and DCR [37] often under-segment lesion areas, while Self-Correcting [29] and SwinT [41] occasionally introduce false positives along the lesion border. In contrast, our framework with only 10% strong labels already achieves compact and accurate delineations, and the 20% setting produces the sharpest lesion boundaries with minimal false positives and negatives.

These visual results reinforce the quantitative findings: the bounding box encoder effectively leverages weak supervision to guide feature learning, enabling the model to outperform both fully and weakly supervised baselines while reducing annotation requirements.

5.4. Efficiency Analysis

To complement the quantitative and qualitative results, we report inference efficiency and model size of different segmentation models in terms of trainable parameters, per-image inference time, and floating-point operations (FLOPs) under the implementation setup described in Section 4.3, as reported in

Table 6. Comparative analysis of model size and inference efficiency. Columns report trainable parameters (M), per-image inference time (s), and FLOPs per $512\times 512$ forward pass, measured under the hardware/software configuration in Section 4.3 (batch size 1; no test-time augmentation; times averaged after warm-up).

Model	Parameters (M)	Inference Time (s)	FLOPs (T)
U-Net [1]	31.04	0.0192	0.193
MobileNetv3 [40]	3.28	0.0105	0.008
SwinT [41]	58.94	0.0298	0.236
CrackMaster [42]	93.61	0.0192	0.087
EfficientCrackNet [36]	0.35	0.0532	0.045
CrackRefineNet [32]	99.66	0.0387	0.206
Self-Correcting [29]	48.77	0.0151	0.061
Xiong et al. [39]	44.68	0.0342	0.193
Ours	92.67	0.0201	0.092

.

Lightweight baselines such as MobileNetv3 [40] and EfficientCrackNet [36] demonstrate small model size (3.28 M and 0.35 M parameters, respectively) with very low compute (FLOPs). However, this efficiency comes at the cost of accuracy, as confirmed in previous evaluations. In contrast, fully supervised architectures such as SwinT [41] and CrackRefineNet [32] achieve higher accuracy but at a higher compute and model-size cost, requiring up to 99.66 M parameters and 0.206 T FLOPs, with longer inference times.

When considering semi-supervised frameworks, Self-Correcting network [29] achieves a relatively low compute footprint (48.77 M parameters and 0.061 T FLOPs) due to its streamlined design. However, this reduced compute budget comes with limitations in segmentation accuracy. Our proposed model, which integrates a ConvNeXt-Base backbone with a lightweight U-Net style CNN decoder and a dedicated bounding box encoder, introduces a higher parameter count (92.67 M) but keeps the computation tractable with only 0.092 T FLOPs. Importantly, the inference time (0.0201 s per image) remains close to that of U-Net (0.0192 s) and substantially faster than CrackRefineNet (0.0387 s), despite the stronger backbone.

This efficiency analysis highlights that the proposed framework strikes an effective balance between accuracy and efficiency in the semi-supervised setting. By leveraging bounding box guidance through the encoder, our model achieves superior segmentation performance while remaining efficient at inference, demonstrating its suitability for both research and real-world deployment.

5.5. Analysis of Failure Cases

Although the proposed framework achieves strong overall performance, certain limitations remain in challenging scenarios. Figure 6 highlights representative failure cases from the RCFD and ISIC 2018 datasets. The main sources of error can be grouped into three categories.

First, boundary ambiguity arises in regions where cracks or lesion edges are poorly defined. In such cases, the model may either under-segment the structure or include irrelevant background, leading to false positives or false negatives. Second, low-contrast or fine-scale details are sometimes missed, particularly when only a small fraction of strong annotations is available. This results in incomplete predictions, with thin or subtle structures not being fully captured. Finally, textured or noisy regions occasionally cause the model to over-segment, producing spurious detections that do not correspond to true target regions.

Despite these challenges, the overall alignment between predictions and ground truth remains high, even with as little as 10–20% strong supervision. These observations suggest that while the bounding box encoder effectively leverages weak annotations, future work could focus on improving boundary refinement and robustness to noise, for example through boundary-aware losses or post-processing modules.

6. Conclusions

In this work, we introduced a weakly supervised segmentation framework that combines a ConvNeXt-Base encoder with a lightweight bounding box encoder and a U-Net-style decoder. The key novelty lies in leveraging bounding box annotations to generate spatial attention maps that guide feature modulation, thereby reducing the reliance on dense pixel-level annotations.

Extensive experiments on two challenging datasets, RCFD and ISIC 2018, demonstrate that our method not only outperforms conventional fully supervised baselines but also achieves competitive or superior results compared to state-of-the-art semi- and weakly supervised approaches. Notably, with only 10–20% strong annotations and full weak supervision, the proposed framework surpasses the performance of fully supervised baselines, highlighting its annotation efficiency. For RCFD (Table 4), our model achieves IoU 69.29%, F1 81.86%, and mIoU 83.99% with 20% strong labels (vs. CrackRefineNet at 65.41% IoU, 79.09% F1, 81.97% mIoU using 100% strong). Even at 10% strong, ours attains 61.97% IoU, 76.52% F1, 80.15% mIoU, exceeding mixed baselines at 10% strong (e.g., Self-Correcting: 48.14% IoU, 64.99% F1, 72.92% mIoU). On ISIC 2018 (Table 5), our method reaches 87.41–87.95% IoU, 93.28–93.59% F1, and 90.92–91.29% mIoU with 10–20% strong labels, outperforming fully supervised CrackRefineNet (83.11% IoU, 90.78% F1, 87.77% mIoU) and other baselines.

Ablation studies further confirmed the contribution of the bounding box encoder and the effectiveness of integrating weak masks during training. Complexity analysis showed that the model achieves a favorable trade-off between accuracy and computational cost, maintaining efficiency while delivering high performance. Qualitative analyses also revealed the strengths and remaining challenges, particularly in handling boundary ambiguity and noisy textures. In terms of efficiency (Table 6), our model uses 92.67 M parameters with 0.092 T FLOPs and achieves 0.0201 s per-image inference—close to U-Net (0.0192 s) and faster than CrackRefineNet (0.0387 s) under the setup in Section 4.3.

Future work will focus on improving boundary refinement and generalizing the framework to other domains where annotation costs are high. We also aim to explore the integration of advanced loss functions and post-processing strategies to further mitigate errors in challenging cases. Overall, this study demonstrates that bounding box-guided feature modulation offers a practical and effective path toward annotation-efficient semantic segmentation.