iSAGE: A Human-in-the-Loop Framework for Remote Sensing Semantic Segmentation via Sparse Point Supervision

Highlights

What are the main findings?

• Automated selection methods that read a model’s outputs (uncertainty sampling, self-training, label propagation, and random selection) plateau well below dense supervision on ISPRS Vaihingen, even when an entropy oracle receives ground truth answers at up to 100 times the budget.
• An expert clicking confidently incorrect pixels, at most one per class per frame per iteration and with no auxiliary machinery, reaches 99.6% and 96.9% of the dense mIoU on ISPRS Vaihingen and BsB Aerial.

What are the implications of the main findings?

• Automated acquisition rules cannot find a model’s confident errors in its own outputs; a human inspecting the prediction overlay supplies exactly this missing signal, making expert clicks a sufficient replacement for the label-expansion machinery of prior pipelines.
• Segmentation models can be trained and maintained at a small fraction of the dense annotation effort (14 to 24 times fewer pointing actions), supported by the open-source iSAGE platform.

Abstract

Training semantic segmentation models is especially costly in remote sensing, since most problems require a new dataset as targets vary with resolution, sensors, and region. Approaches to reduce this cost are increasingly common, but most add machinery that expands few labeled pixels into a denser training signal using the model’s own predictions, where a confident prediction looks the same whether it is correct or incorrect. The hypothesis here is that these confident errors are the most valuable pixels to label, and that a human examining the image can identify them directly. We propose iSAGE (Iterative Sparse Annotation Guided by Expert), an open-source framework in which the annotator clicks confident errors in the prediction overlay, the clicks train the model under an error-weighted loss, and the updated model surfaces the next errors, with no automatic label expansion at any step. With at most one labeled pixel per class per frame per iteration, iSAGE recovers 96.9% of dense performance on BsB Aerial (74.79% mIoU from 0.040% of the pixels) and matches the dense baseline on ISPRS Vaihingen (76.65% vs. 76.93% from 0.011% of the pixels), surpassing published weakly supervised methods by nearly 4 mIoU points. Four automatic selection strategies run through the same loop plateau 7.3 to 14.4 points below iSAGE, and raising their budgets far higher does not close the gap; in a 35-method comparison, iSAGE is the only iterative human-in-the-loop framework without automatic label generation.

1. Introduction

How much data is needed to train computer vision models? Although the answer is task-, domain-, and target-dependent, the prevailing assumption is that extensive labeled data is a prerequisite. This belief has been reinforced by large-scale benchmark datasets such as Common Objects in Context (COCO) [1], Cityscapes [2], and ImageNet [3], which have fueled model-centric progress in deep learning through extensive experimentation and architectural innovation [4,5,6].

However, models in production demand more than benchmark-style labeling: (1) speed: Cityscapes reports 1.5 h of expert annotation per high-resolution urban scene [2], and MS-COCO took over a year and a half of collaborative effort [1]; (2) flexibility: dense datasets are hard to audit or extend, since adding a new class means revisiting every training image to insert the label; and (3) quality: dense annotation forces decisions on every pixel (including the ambiguous ones), introducing annotator biases (or errors) into the training signal. Remote sensing sharpens this problem because models trained on one sensor (multispectral, hyperspectral, or SAR), platform (satellites or UAVs), or geography rarely transfer to another, so nearly every problem requires its own dataset [7,8,9,10].

Cost-reduction strategies fall into three families, none of which removes the human from the loop. Foundation models (SAM [11], CLIPSeg [12]) segment from prompts but do not assign class labels in specialized taxonomies, and even when used as labeling shortcuts, they produce imprecise boundaries and miss small objects in medical imaging [13], fine-structure scenes [14], and remote sensing [15], limitations that the SAM2 paper [16] itself acknowledges. Few-shot methods [17,18] require meta-training on similar tasks, which is rare in specialized domains. The third family iterates on the target dataset by reading model predictions: weak supervision propagates sparse pixels through heuristics [19,20,21], active learning selects by uncertainty [22,23,24], pseudo-labeling reuses high-confidence predictions [25,26], and interactive frameworks combine human feedback with algorithmic label expansion [27,28,29].

These output-reading approaches differ in who produces the mask (human, heuristic, or model) but share the same input: the model’s predictive distribution. Acquisition rules based exclusively on the predictive distribution of a single model cannot distinguish a confidently incorrect prediction from a confidently correct one when their predictive vectors are identical, yet these pixels may carry corrective supervision that uncertainty-based selection does not prioritize. This information limitation motivates an external observer who can compare the prediction against the visual evidence in the image. Closing this gap requires both a conceptual move and the infrastructure to test it end-to-end. Yang et al. [29] came closest on the conceptual side by observing confident errors in a domain adaptation setting, but their response layered more machinery on top (staged acquisition, pseudo-label densification, consistency regularization, and domain-adaptation supervision) rather than testing whether the observation alone was enough. Concurrent work [30] reports the same confident-overlap phenomenon but addresses it through output-space correction, staying within the same input regime. On the infrastructural side, the experiment requires annotation UX, prediction overlays, record format, mask generation, training pipeline, and reproducibility tooling to live in one environment. Existing annotation tools (CVAT, Labelbox, and V7) stop at labeling, and existing training frameworks assume dense masks as input, so anyone running the experiment end-to-end would have to connect these stacks by hand at every iteration.

This paper proposes iSAGE (Iterative Sparse Annotation Guided by Expert), an integrated framework that fills both gaps, with the following contributions:

• A blind spot of output-reading acquisition. We show that four families of methods built on output-reading acquisition functions (uncertainty sampling, self-training, label propagation, and random selection) have a ceiling well below dense supervision, plateauing at 62.25 to 69.35 mIoU on ISPRS Vaihingen against 76.93.
• The human solution. We propose a human-in-the-loop framework that targets this blind spot directly, with no additional machinery and a restrictive budget of at most one labeled pixel per class per frame per iteration, reaching near-dense performance (99.6% and 96.9% of the dense mIoU on ISPRS Vaihingen and BsB Aerial). To the best of our knowledge, no prior work has run this experiment, and among the 35 methods surveyed, this is the only iterative human-in-the-loop configuration with no output-reading mechanism.
• iSAGE. The open-source platform couples inspection, annotation, record-keeping, and retraining in one environment, which no existing tool provides and without which the loop above cannot run. It extends to new domains through preprocessing and encoder choices, and every experiment in this paper was produced on it.

The paper is organized as follows. Section 2 reviews related work. Section 3 presents the iSAGE framework and its software platform. Section 4 describes the experimental protocols. Section 5 reports the results, including the controlled comparisons against output-reading baselines. Section 6 discusses implications and positions iSAGE in a 35-method comparison landscape. Section 7 concludes.

2. Related Work

iSAGE is positioned against three families of prior work: sparse and weak supervision, active learning, and interactive human-in-the-loop frameworks.

2.1. Sparse and Weakly Supervised Semantic Segmentation

Weak supervision strategies reduce labeling burden by replacing dense masks with less detailed annotations: image-level labels [31,32], bounding boxes [33], scribbles [34], class activation maps, and selected slices in volumetric data [35]. Sparse supervision is a stricter subset, providing only a few annotated pixels per image (typically below 1–5% [19,20,23,36]), and has been applied to medical imaging [37,38,39,40], remote sensing [20,41,42,43], and natural images [19,34].

How these sparse annotations reach a dense training signal divides the field into two strategies. Enhancement keeps the annotations sparse but compensates through architectural choices: Conditional Random Fields impose spatial coherence on predictions, either as post-processing or integrated modules [20,44,45,46], and custom losses weight the gradient contribution of labeled versus unlabeled pixels [36,37,47,48,49]. Label expansion generates pseudo-labels from sparse seeds through CRF-based refinement [50], GAN-based generation [51], student-teacher consistency [48], and, more recently, foundation models such as SAM [11] and CLIPSeg [12] that produce class-agnostic masks for downstream category assignment. Pseudo-labeling approaches typically mitigate the confirmation-bias risk through confidence thresholds, regularization, or ensemble validation [49,52,53]. The pattern persists in recent remote-sensing work: point-driven networks still densify their seeds through expansion modules [54,55], and SAM-adapted pipelines constrain the resulting pseudo-labels with uncertainty and consistency filters [56]. Whether through enhancement or expansion, the supervision signal that ultimately trains the model is generated algorithmically from the sparse seeds rather than supplied by the annotator directly.

2.2. Active Learning for Semantic Segmentation

Active learning reduces annotation effort by selecting which samples or regions a human annotates next, then refining the model after each iteration [57]. For semantic segmentation, acquisition functions score candidate samples along two main axes: uncertainty (entropy, margin, or model disagreement) and diversity/representativeness (coverage of the input distribution), often combined with task-specific signals such as loss prediction, multi-criterion scoring, simulation-before-annotation, or Bayesian decomposition of pixel-level uncertainty [58,59,60,61,62,63,64,65,66]. Recent work evaluates whether sparse annotations achieve comparable performance to dense supervision in this setting: Kellenberger et al. [23] showed that labeling less than 0.5% of a dataset could outperform fully supervised models in object detection when annotations are carefully selected; Li et al. [28] proposed HAL for few-shot segmentation with point-based annotations and iterative correction; Siddiqui et al. [67] introduced ViewAL for 3D scene annotation through viewpoint-based selection; and similar principles extend to point cloud segmentation [68].

A common assumption across uncertainty-based acquisition functions is that model uncertainty is a reliable proxy for informativeness. This assumption breaks down in two directions: a confident incorrect prediction registers no uncertainty, and a highly uncertain boundary pixel may contribute little new information about the semantic task. Mukhoti and Gal [69] found that the mapping between uncertainty and correctness in Cityscapes segmentation requires careful measurement rather than being self-evident; Gustafsson et al. [70] showed that deep ensembles improve over MC-dropout but still suffer calibration error under synthetic-to-real domain shift; and a recent benchmark of deep active learning concluded that many uncertainty-based methods barely outperform random selection [71]. These findings frame the limit of uncertainty-based acquisition. Confident errors register low uncertainty by definition, and so escape selection rules that query by uncertainty.

2.3. Interactive Human-in-the-Loop Frameworks for Semantic Segmentation

Human-in-the-loop (HITL) machine learning takes two distinct forms in computer vision [72,73,74]. Inference-time HITL refines individual predictions through user clicks or extreme points without altering the model [75,76,77]. Training-time HITL contributes to training supervision via labeled queries or coupled acquisition-and-correction loops, which is the focus of this section. iSAGE belongs to the latter. Inference-time methods and acquisition-only active learning lie outside its design space.

Several training-time HITL frameworks target semantic segmentation specifically. DIAL [27] combines uncertainty-driven sampling with region-based refinement and direct user feedback, using user input to compensate for cases where model uncertainty misses informative pixels. HAL [28] implements interaction loops for real-time correction of segmentation masks based on user clicks, using a dual-network mechanism that consolidates sparse clicks into dense masks for training. EasySeg [29] combines a See-First-Ask-Later (SFAL) acquisition strategy, which identifies obvious model errors before querying uncertain pixels, with an ISS-Net pseudo-label generator under a domain adaptation setting, reporting performance above its fully supervised baseline at a fraction of the annotation cost. Across these frameworks, sparse human input reaches a dense training signal through an algorithmic expansion step, whether patch retraining (DIAL), dual-network consolidation (HAL), or pseudo-label generation (EasySeg).

The inspection step itself, a human viewing the model’s predictions overlaid on the image and correcting visible errors, has precedents outside this family. In RootPainter [78] and Deep Interactive Learning [79], corrective annotations retrain the model directly, with no acquisition function or label expansion. In both, however, each correction is a densely painted region, so the annotator still delineates extents and boundaries at every correction. iSAGE keeps the inspection workflow but reduces each correction to a single labeled pixel, which removes the boundary decision from the annotation act.

More recent frameworks extend this pattern with foundation-model pseudo-labels. ALC [80] refines SAM-generated labels through human correction queries, and A2LC [81] automates the correction decision. In remote sensing, PointSAM adapts SAM to point supervision through self-training with prototype-based regularization [82]. Because these pipelines operate on SAM-generated regions, the training signal flows through boundaries the foundation model drew. These boundaries are imprecise on specialized domains such as medical imaging [13], fine-structure scenes [14], and remote sensing [15], and the SAM2 paper [16] acknowledges the issue itself. Making SAM usable at scale additionally requires engineering for coverage and cross-tile consistency [83]. iSAGE removes this dependency: clicks land only on pixels of visually unambiguous class, and the click itself never asks the annotator for a boundary decision.

3. iSAGE Framework

iSAGE is a framework for semantic segmentation in which sparse human clicks drive an iterative training loop. The annotator identifies pixels where the current model is confidently wrong at each iteration. The growing record of (coordinate, class) tuples, stored as a JSON file, is converted deterministically into training masks. The Error-Weighted Dice Loss (EWDL) then trains the next iteration with those clicks amplified relative to already correct predictions. No propagation, pseudo-labeling, or other algorithmic densification step sits between the click and the gradient. An integrated platform hosts inspection, annotation, retraining, and dataset maintenance as a single workflow. The following four subsections describe the click-based annotation, the iterative refinement loop, EWDL, and the platform. Figure 1 overviews the iteration cycle.

Figure 1. iSAGE workflow: Iterative annotation, training with EWDL, and user-decided stopping. Solid arrows follow the data through one iteration; the blue dashed box marks the human in the loop, who supplies the point annotations and the quality check; green and red boxes mark the start and the user-decided stop.

3.1. Sparse Annotations

Sparse annotation labels only a subset of pixels in each frame. The rest are treated as ignore during training. In iSAGE, each annotation corresponds to exactly one pixel. The user clicks on a location in the image, and the software records the coordinate $(x,y)$ along with the selected class label, with no region expansion, bounding box, or area selection. The resulting set of $(x,y,\mathrm{class})$ tuples constitutes the complete training supervision, and the dataset produced by iSAGE is the annotation file itself.

The choice of a single-pixel click as the basic annotation unit is deliberate. Scribbles and sparsely drawn polygons also reduce the total pixel count relative to dense masks, but points have a property that these alternatives do not share: per-decision auditability. Each point annotation is a single (coordinate, class) tuple, individually verifiable by returning to the pixel and independently correctable at the record level (remove one JSON entry). Scribbles bundle multiple pixel-level decisions into a single primitive, so errors in part of the scribble are not easily localized or corrected. Polygons reintroduce boundary decisions, where labels are intrinsically ambiguous, and annotator uncertainty becomes label noise; the training procedure cannot distinguish it from correct supervision. The point primitive therefore forces single-pixel granularity, which aligns with the workflow’s commitment to labeling only pixels where the class is visually unambiguous. Because each click contributes exactly one pixel regardless of object size, the annotator also balances class contribution independently of spatial extent, a second advantage that follows from the same primitive.

Formally, let I be an input image and $A:{\mathbb{R}}^2\to \{0,1,\dots ,C-1\}$ be an annotation function that assigns class labels to pixels, where C is the number of classes, and label 0 denotes the background. Supervision, in sparse annotations, only considers a limited set of pixels $S\subset {\mathbb{R}}^2$, and A is undefined outside S, thus:

$$A^{\prime }\left(x\right)=\left\{\begin{array}{cc}A\left(x\right),\hfill & \mathrm{if}x\in S,\hfill \\ -1,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(1)

Pixels with value $-1$ act as an ignore mask and contribute zero gradient. In the implementation, the ignore value is stored as C, the first integer outside the label range. This regime does not restrict what the network sees, only which pixels it is penalized on. The gradient from each labeled pixel updates all weights that influenced its prediction, so the network’s spatial inductive biases generalize the class signal to visually similar pixels, and shape and boundary information emerges from the learned representation rather than from explicit boundary annotation.

For binary segmentation ($C=2$), the labels reduce to $\{0,1\}$, with 1 for the foreground and 0 for the background. In the multiclass scenario, labels $\{1,\dots ,C-1\}$ correspond to the object classes.

3.2. Iterative Error-Driven Refinement

The iSAGE cycle improves the model by annotating regions where predictions are wrong. Error-focused supervision concentrates each new label on a region the current model gets wrong rather than on a randomly chosen pixel. Algorithm 1 describes this iterative process.

Algorithm 1 Iterative Error-Driven Refinement for Sparse Annotations

1: Input: Initial sparse annotations $S_0$, number of iterations N   2: Initialize: Train segmentation model $M_0$ on $S_0$   3: for $i=1$ to N do   4:    Generate segmentation predictions $P_i$ using model $M_{i-1}$   5:    Identify misclassified pixels (false positives and false negatives)   6:    Manually select new annotation points $S_i^{\prime }$ from misclassified regions   7:    Expand training set: $S_i\leftarrow S_{i-1}\cup S_i^{\prime }$   8:    Train updated model $M_i$ on $S_i$   9: end for 10: Output: Final trained model $M_N$

The initial set $S_0$ is collected on the raw images without a prediction overlay, since no model has been trained yet (in the same interface and under the same budget), choosing pixels for class clarity. From iteration 1 onward, the operator selects new points from the errors visible in the prediction overlay; the model is retrained on the expanded set, and the procedure repeats for N iterations.

Operationally, confidently wrong means a pixel where the class displayed in the argmax overlay differs from a class that is visually unambiguous in the imagery, and no confidence map, threshold, or uncertainty estimate takes part in the selection. The name refers to the model’s side. Such errors typically carry low output uncertainty, so output-reading rules do not prioritize them. A class omitted entirely from a frame stays annotatable, since the missed objects appear painted with another class’s color and are clicked with the true label.

The experiments in this paper deliberately adopt the smallest possible supervision budget, at most one pixel per class per frame per iteration, as an adversarial test of the workflow. In practice, iSAGE places no fixed limit on annotation density, and the annotator decides how many points each frame requires.

3.3. Error-Weighted Dice Loss

Standard loss functions treat all labeled pixels equally during training. Under sparse annotation, this can be problematic: the model receives limited supervision, and correct predictions (often the majority) dominate the gradient signal, while misclassified pixels receive insufficient attention. The proposed Error-Weighted Dice Loss (EWDL) addresses this imbalance by amplifying the contribution of incorrectly predicted pixels.

Formulation

Let $y_{gt}\left(x\right)\in \{0,1,\dots ,C-1\}$ denote the ground truth label at pixel x, and let $y_{pr}^{\left(c\right)}\left(x\right)\in [0,1]$ be the predicted probability for class c. The standard Dice loss for class c is

$${\mathcal{L}}_{\mathrm{Dice}}^{\left(c\right)}=1-{\displaystyle \frac{2{\sum }_x{y}_{gt}^{\left(c\right)}\left(x\right)·y_{pr}^{\left(c\right)}\left(x\right)+ϵ}{{\sum }_x{y}_{gt}^{\left(c\right)}\left(x\right)+{\sum }_x{y}_{pr}^{\left(c\right)}\left(x\right)+ϵ}},$$

(2)

where $y_{gt}^{\left(c\right)}\left(x\right)\in \{0,1\}$ is the one-hot ground truth for class c, and $ϵ$ ensures numerical stability.

EWDL introduces a per-pixel weight based on prediction correctness. For each labeled pixel $x\in S$, a binary indicator captures whether the predicted class matches the annotator-provided ground truth:

$$\mathrm{correct}\left(x\right)=\mathbb{1}\left[\arg \underset{c}{\max }{y}_{pr}^{\left(c\right)}\left(x\right)=y_{gt}\left(x\right)\right].$$

(3)

The weight assigned to each pixel depends on this correctness,

$$w\left(x\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\mathrm{correct}\left(x\right)=1,\hfill \\ \lambda \hfill & \mathrm{if}\mathrm{correct}\left(x\right)=0,\hfill \end{array}\right.$$

(4)

where $\lambda >1$ is the error penalty factor. When $\lambda =1$, EWDL reduces to standard Dice loss. The indicator is recomputed at every forward pass and treated as a constant with respect to the network parameters: the argmax is non-differentiable, and no gradient propagates through $w\left(x\right)$. Because the Dice numerator and denominator couple all pixels, the weight does not multiply each pixel’s gradient by exactly $\lambda$; it increases the relative influence of misclassified pixels in the class-wise ratio, shifting the optimization toward the model’s mistakes.

The weighted Dice loss for class c becomes:

$${\mathcal{L}}_{\mathrm{EWDL}}^{\left(c\right)}=1-{\displaystyle \frac{2{\sum }_{x}w\left(x\right)·y_{gt}^{\left(c\right)}\left(x\right)·y_{pr}^{\left(c\right)}\left(x\right)+ϵ}{{\sum }_{x}w\left(x\right)·\left(y_{gt}^{\left(c\right)}\left(x\right)+y_{pr}^{\left(c\right)}\left(x\right)\right)+ϵ}}.$$

(5)

For multiclass segmentation with C classes, the total loss averages across all classes:

$${\mathcal{L}}_{\mathrm{EWDL}}={\displaystyle \frac{1}{C}}{\displaystyle \sum _{c=0}^{C-1}}{\mathcal{L}}_{\mathrm{EWDL}}^{\left(c\right)}.$$

(6)

All summations over x above run over the labeled set S defined above. Unlabeled pixels contribute zero weight and zero gradient, consistent with the ignore mask behavior of standard sparse training. Three implementation details complete the specification. The sums pool the labeled pixels of all images in the batch, so the Dice ratio is computed per class over the batch rather than per image, and the class average includes the background class. The weights are rescaled to sum to the number of labeled pixels; the Dice ratio is homogeneous in w, so the rescaling changes nothing beyond the $ϵ$ smoothing and serves numerical stability ($ϵ=1$ throughout). A class with no labeled pixels in the batch contributes only through predicted probability mass assigned to it at the labeled pixels, and with $ϵ=1$, its term vanishes when that mass is zero. The binary tasks apply the same weighting to a single foreground Dice term.

3.4. Software Platform

The platform is the experimental instrument that makes the alternative to output-reading supervision testable end-to-end. Without a single environment coupling prediction overlay, click annotation, record persistence, and retraining, the iterative loop fragments into ad-hoc stitching across tools, and the experiment ceases to run at research scale. Figure 2 shows the annotation interface, with the model’s prediction overlaid on the image for click-based correction. The interface can also display a ground truth overlay for auditing when reference labels exist; the experiments reported here did not use it.

Figure 2. Annotation interface with the prediction overlay on training images. Center: the current frame with the prediction overlay and the click points; left: class selection and display settings; right: mini-map of the original image, image information, and the running count of annotation points.

The platform comprises four subsystems:

• Annotation interface: Hosts prediction overlay and click-driven annotation.
• Record storage: Persists every decision as JSON.
• Session layout: Packages each iteration as a reproducible snapshot.
• Training backend: Closes the loop with EWDL-supervised retraining.

The platform is domain-agnostic by construction: any dataset with standard image formats and enumerable classes runs through the pipeline, requiring only domain-specific preprocessing and encoder choice. The training backend itself is pluggable. The default implementation uses Segmentation Models PyTorch (version 0.4.1) [84], built on PyTorch [85], and the released platform drives the same workflow from a Jupyter notebook [86], a command-line interface, or the Python (version 3.11) API. Any trainer that reads iteration_N/masks/ and writes the corresponding model checkpoint and predictions can replace the default without touching the annotator or the record format.

The training loop does not read the JSON files directly. The annotation record is converted to dense per-image masks by a standalone mask generator, and the training dataloader consumes those masks. This separation has three consequences: (1) the JSON record is a self-contained audit trail of human decisions and can be versioned, diffed, shared, or replayed without running the training pipeline; (2) mask generation is reproducible: given the same JSON files and the same converter, two practitioners obtain identical training masks; (3) the dataset is portable: the JSON files plus the converter are sufficient to reconstruct the supervision signal, without requiring the annotation tool or its UI state. Together, these properties make the annotation record a complete, portable, auditable dataset rather than a sparse input to a densification pipeline.

4. Experiments

4.1. Experimental Setup

All experiments use the same U-Net with EfficientNet-B7 encoder. The fixed architecture isolates annotation-strategy effects from model variability, and since iSAGE’s predictions also guide annotation, architectural changes would shift point-selection trajectories across iterations. A cross-architecture validation (Section 4.3) retrains three additional configurations on the final annotation record, testing whether the sparse dataset produced by the loop replicates its result across backbones. The framework is architecture-agnostic by design: any model in the Segmentation Models PyTorch library [84] can replace it. Table 1 lists the hyperparameters. All training ran on a single NVIDIA GeForce RTX 4090. The iSAGE code and configuration files are released at https://github.com/osmarluiz/iSAGE (accessed on 27 August 2026) and archived at https://doi.org/10.5281/zenodo.20596185.

Table 1. Training configuration for iSAGE experiments.

Setting	Value
Architecture	U-Net [87]
Encoder	EfficientNet-B7 [88]
Optimizer	Adam (${\beta }_1=0.9$, ${\beta }_2=0.999$)
Learning rate	${10}^{-4}$
Batch size	10
Epochs per iteration	100
Checkpoint selection	Final epoch
Data augmentation	Horizontal and vertical flips

Performance is evaluated using IoU, F1-score, precision, and recall per class. iSAGE reports the final-epoch model, since the sparse-click setting presupposes no labeled data. Dense supervision baselines use the best-validation checkpoint per standard practice. On both datasets, this validation signal comes from the held-out evaluation partition itself (an asymmetry that can only favor the dense baselines), so the reported iSAGE-to-dense comparisons are conservative. Final-iteration and ablation values on the multiclass task, and the final-iteration iSAGE and dense values on Vaihingen, are means over five training seeds, and per-iteration trajectories come from the single interactive annotation session, whose clicks are collected once. Random-selection results on the multiclass task are means over five independently seeded chains sharing iSAGE’s iteration-0 bootstrap.

The compared frameworks report annotation budgets as pixel fractions, clicks, or region counts, and none measures annotation time [24,27,29,89,90,91], which conflates concentration, fatigue, and interface fluency. Measuring time credibly would also require timing the dense workflow on the same imagery under the same conditions, a controlled study in its own right. Annotation effort is therefore reported here as the fraction of labeled training pixels, the standard cost metric of this literature and one exactly reproducible from the annotation record.

The fraction measures supervision density rather than human work, so effort is additionally counted in pointing actions, the number of times an annotator marks a location. For iSAGE, this is the click count. For dense annotation, which delineates polygons, it is the number of vertices needed to reproduce the ground truth. Each training mask is binarized per class (clutter excluded), matching how an annotator outlines one class at a time, boundaries are extracted as outer contours plus holes, and each closed boundary is simplified with the Douglas–Peucker algorithm [92] at tolerances of 2 and 1 pixels, both stricter than the 3-pixel boundary erosion the ISPRS evaluation protocol applies to discount boundary uncertainty [7]. The vertex count is an idealized lower bound, assuming optimal placement and no corrections, and the exact count of the unsimplified boundaries is kept as a reference point.

4.2. Datasets

Two complementary remote sensing datasets serve distinct roles. BsB Aerial, curated by the authors, is the controlled laboratory for comparisons against dense supervision, random selection, and alternative losses that a public benchmark cannot support. ISPRS Vaihingen is the external benchmark for comparisons against published methods and for the four controlled output-reading baselines (oracle entropy, self-training pseudo-labels, CRF-based label propagation, uniform random). The two datasets vary across geography (Brazil vs. Germany), spectral composition (RGB vs. IRRG), and spatial resolution (0.24 m vs. 0.09 m).

4.2.1. BsB Aerial

BsB Aerial uses an aerial ortho image of Brasília, Brazil, at 0.24-m spatial resolution and three spectral bands (Red, Green, Blue), located at approximately 15°47′32″S latitude and 47°52′10″W longitude. The image was divided into non-overlapping 256 × 256 pixel frames, ensuring each annotated point appears in only one frame and preventing duplicated annotations. The study used 1250 patches: 1000 for training and 250 for held-out testing. No separate validation split enters the protocol. iSAGE presupposes that no labeled pool exists beyond the clicks, and the dense baselines’ checkpoint selection draws its validation signal from the 250 held-out patches. The ortho image and class definitions come from the authors’ earlier releases of the dataset [9,93,94]; the frame selection used in this study was drawn anew.

The four BsB Aerial classes are used as an experiment in class-type isolation. Each was selected to test iSAGE on a distinct combination of geometry, boundary, and texture:

• Small discrete objects (cars): Rigid geometry, sharp boundaries, low spatial frequency.
• Linear connected structures (roads): Elongated, with mixed boundary regimes and frequent occlusion.
• Large polygonal structures (buildings): Rectilinear, with sharp boundaries and rich interior texture.
• Amorphous smooth-boundary regions (permeable areas): Irregular, with mixed textures and high intra-class variance.

Each category carries a characteristic failure mode for methods relying on model outputs or dense ground truth. Figure 3 shows one example of each.

Figure 3. Examples of the four target classes: (A) buildings, (B) cars, (C) permeable areas, and (D) roads. Black squares outline the 256 × 256 pixel frames (0.24 m spatial resolution); scale bars indicate 10 m.

4.2.2. ISPRS Vaihingen

ISPRS Vaihingen [7] is a standard benchmark for urban semantic segmentation, with 33 aerial image tiles over Vaihingen, Germany, at 0.09 m spatial resolution and three spectral bands (near-infrared, red, green). Ground truth annotations cover six semantic classes: impervious surfaces, buildings, low vegetation, trees, cars, and clutter/background.

The test split follows the same 17-tile partition used by EasySeg [29], yielding 155 non-overlapping 512 × 512 test patches. Training uses 1000 non-overlapping 512 × 512 patches from the remaining 16 tiles, a 44% reduction relative to the 1784 patches used by EasySeg: this smaller training set is a deliberate constraint, not an advantage, kept consistent with iSAGE’s minimum-effort regime. The partition defines no validation tiles; the dense baseline’s checkpoint selection likewise draws its validation signal from patches of the test partition.

Training proceeds on five classes (impervious surfaces, buildings, low vegetation, trees, cars), excluding clutter. The exclusion follows from the class itself: clutter is a catch-all category whose contents differ between the training and test tiles of the EasySeg partition. Water bodies, for example, appear as clutter in the training tiles but not in the test tiles, so the class cannot be learned from the Vaihingen imagery alone, and every configuration evaluated here, sparse and dense alike, yielded zero IoU on it. EasySeg recovers clutter only because its Potsdam domain-adaptation signal supplies external dense supervision for the class, a mechanism iSAGE deliberately avoids. The exclusion applies identically to iSAGE and to the dense baseline trained under the same protocol, so it cannot favor iSAGE: the match-with-dense result is a ratio in which both terms share the five-class setting.

4.3. BsB Aerial Experiments

The BsB Aerial dataset was annotated by a single annotator, holding annotator identity constant across the controlled comparisons (EWDL vs. alternative losses, error-driven vs. random selection, sparse vs. dense supervision). External validation against an independent ground truth is provided by the Vaihingen experiments in Section 4.4. The protocol comprises seven experiments:

• Binary experiments. Four independent tasks, one per class, to isolate per-class convergence dynamics from class competition and observe how each visual category evolves across iterations. Each task runs for 5 iSAGE iterations after the initial seed.
• Multiclass experiment. Joint training over all four classes plus background, under the same 5-iteration protocol, to test whether multiclass supervision changes per-class behavior relative to the binary baselines.
• Dense supervision upper bound. Models trained with dense ground truth masks under the same architecture, providing the performance ceiling iSAGE is compared against.
• Random selection baseline. The iterative protocol with randomly selected sparse annotations in place of error-driven selection, controlling for whether iSAGE’s gain comes from sparsity per se or from error-targeting. This baseline is the without-expert-guidance control: it shares the pipeline, budget, loss, and training schedule, differing only in who selects the pixels.
• Alternative loss functions. EWDL compared against Binary Cross-Entropy (BCE), Focal, and Dice losses on the binary tasks and against Cross-Entropy (CE), Focal, and Dice on the multiclass task, on the iter-5 final annotation set, testing whether EWDL’s error-weighting actually contributes to iSAGE’s performance or a standard loss suffices.
• EWDL hyperparameter sensitivity. The error-penalty factor $\lambda$ varied over {1, 2, 5, 10, 20} on the iter-5 set, multi-seed, to probe the loss’s stability around the chosen $\lambda =5$.
• Cross-architecture validation. Four model configurations (the primary U-Net + EfficientNet-B7 plus U-Net + ResNet-101, DeepLabV3+ + ResNet-50, and SegFormer + MiT-B2), each paired with a matched dense-supervision baseline and trained under the identical five-seed protocol on the iteration-5 annotation set, covering three decoder families (encoder-decoder, atrous spatial pyramid, hierarchical transformer), to test whether the annotation record transfers across backbones.

4.4. ISPRS Vaihingen Experiments

The Vaihingen experiments comprise two groups: an external benchmarking block where iSAGE is run on Vaihingen and compared against published methods, and four output-reading baselines that re-run the iSAGE protocol with automated acquisition mechanisms in place of the human annotator.

4.4.1. External Benchmarking

• iSAGE on Vaihingen. The complete iSAGE pipeline (sparse seed annotations, error-driven refinement over 5 iSAGE iterations, EWDL training) is applied to Vaihingen under the same per-iteration budget used throughout (at most one labeled pixel per class per frame), accumulating to at most six labeled pixels per class per frame over the five iterations plus the seed. Performance is compared against published methods on EasySeg’s [29] 17-tile test partition.

4.4.2. Output-Reading Baselines

Four automated acquisition mechanisms are re-run under iSAGE’s protocol on Vaihingen, representing the dominant automated alternatives to human-in-the-loop supervision: uncertainty-based active learning [57,69], self-training with pseudo-labels [25,26], CRF-based label propagation [95], and uniform random sampling as a control. The goal is to test whether any of these can match iSAGE’s human-targeted signal. All four share iSAGE’s model, training schedule, and iter-0 seed annotations. Oracle entropy and uniform random keep the same per-iteration budget cap as iSAGE. The oracle selects per predicted class per frame, since an entropy rule can only rank pixels within classes the model predicts, while uniform random draws one pixel per ground truth class per frame, independently of the model’s outputs. Pseudo-labeling and CRF propagation have no per-iteration cap and expand the supervision set automatically across iterations. To rule out budget calibration and confidence-threshold calibration as the lever, the oracle is additionally evaluated at 10×, 50×, and 100× the per-class budget, and pseudo-labeling at confidence thresholds 0.90, 0.95, and 0.99.

• Oracle entropy [57,69]: At each iteration, the highest-entropy pixel per predicted class per frame is selected, with ground truth labels assigned at the selected coordinates. Tests whether the strongest possible uncertainty-driven acquisition, given ground truth answers at every query, can match iSAGE. The budget sweep at 10×, 50×, and 100× labels per class per frame reaches up to 0.95% of training pixels and tests whether budget alone closes the gap to iSAGE.
• Pseudo-labeling [25,26]: At each iteration, pixels whose predicted-class confidence exceeds a fixed threshold are converted to pseudo-labels and added to the supervision set. Tests whether self-confidence can expand sparse seeds into adequate supervision without human input. The threshold sweep at 0.90, 0.95, and 0.99 tests whether confidence calibration is the lever that closes the gap.
• CRF-based label propagation [95]: At each iteration, the model’s softmax outputs are refined by a DenseCRF with a Gaussian spatial pairwise term and a bilateral color-position pairwise term (pydensecrf defaults: Gaussian $sxy$ = 3, compatibility 3; bilateral $sxy$ = 80, $srgb$ = 13, compatibility 10; five mean-field iterations) before a confidence threshold matching the strongest pseudo-labeling configuration in the threshold sweep above is applied. Tests whether spatial smoothing of model outputs adds value on top of the best expansion baseline, isolating smoothing from threshold calibration.
• Uniform random: A control baseline that selects one pixel per ground truth class per frame, uniformly at random within that class’s region and independently of the model’s predictions. Tests whether any acquisition structure outperforms the simplest sampling.

5. Results

Results follow the structure of the experimental protocol (Section 4). The BsB Aerial subsection reports iSAGE performance on the binary and multiclass tasks, against the random and dense reference baselines, and across loss, $\lambda$, and architecture ablations. The ISPRS Vaihingen subsection compares iSAGE against published state-of-the-art methods on the shared 17-tile test partition and reports iteration trajectories for four output-reading baselines (oracle entropy, pseudo-labeling, CRF-based label propagation, and uniform random), with the structural interpretation reserved for Section 6.1.

5.1. BsB Aerial

On the four binary tasks, iSAGE reached final IoUs of 74.05 (car), 81.88 (road), 81.48 (building), and 88.17 (permeable area) at iteration 5 with annotation effort below 0.018% per task, recovering 95.0%, 97.4%, 95.1%, and 97.1% of the corresponding dense IoU (Table 2, Figure 4). Cars improved by +46.6 IoU between iterations 0 and 5 and still ended with the lowest absolute IoU (74.05 against a dense ceiling of 77.93). Permeable area improved by only 6.4 IoU and started highest.

Table 2. Binary segmentation results on BsB Aerial. Per-class IoU, precision, recall, F1-score, and annotation effort (%). Underlined values are the best per column within each block; the dense-supervision row is set in bold italics as the upper bound.

Setting	Car					Road					Building					Permeable Area
	IoU	Prec.	Recall	F1	Effort	IoU	Prec.	Recall	F1	Effort	IoU	Prec.	Recall	F1	Effort	IoU	Prec.	Recall	F1	Effort
iSAGE (EWDL)
Iter 0	27.44	30.69	72.20	43.07	0.0029	68.22	80.37	81.85	81.10	0.0030	54.98	60.52	85.71	70.95	0.0024	81.73	88.80	91.12	89.94	0.0030
Iter 1	39.12	44.04	77.79	56.24	0.0058	78.27	87.74	87.88	87.81	0.0059	67.99	73.26	90.43	80.95	0.0048	87.03	91.91	94.25	93.07	0.0059
Iter 2	61.09	73.32	78.55	75.85	0.0087	80.30	89.82	88.34	89.07	0.0086	76.70	82.40	91.73	86.81	0.0072	87.53	92.43	94.29	93.35	0.0085
Iter 3	71.38	84.15	82.46	83.30	0.0116	81.47	90.37	89.22	89.79	0.0110	80.26	87.50	90.65	89.05	0.0096	88.03	92.14	95.18	93.63	0.0106
Iter 4	73.12	86.44	82.59	84.47	0.0145	81.67	91.01	88.84	89.91	0.0128	80.66	86.43	92.36	89.30	0.0121	88.09	92.02	95.38	93.67	0.0122
Iter 5	74.05	86.02	84.18	85.09	0.0174	81.88	91.24	88.86	90.04	0.0140	81.48	88.62	91.01	89.80	0.0145	88.17	92.48	94.98	93.71	0.0130
Iterative Random Selection
Iter 0	38.52	39.92	91.64	55.62	0.0029	71.46	77.95	89.56	83.35	0.0030	68.91	73.93	91.03	81.59	0.0024	84.92	88.86	95.04	91.85	0.0030
Iter 1	43.03	44.26	93.93	60.17	0.0058	72.04	77.98	90.44	83.75	0.0059	70.67	75.56	91.61	82.82	0.0050	85.94	90.48	94.48	92.44	0.0060
Iter 2	43.51	44.61	94.60	60.63	0.0087	74.22	80.78	90.14	85.20	0.0089	71.80	76.50	92.13	83.59	0.0075	85.96	90.91	94.04	92.45	0.0091
Iter 3	45.12	46.13	95.37	62.19	0.0116	74.59	80.47	91.07	85.45	0.0118	72.96	77.83	92.10	84.37	0.0100	86.41	90.02	95.57	92.71	0.0121
Iter 4	44.84	45.76	95.68	61.92	0.0145	75.11	80.57	91.72	85.79	0.0148	73.29	78.63	91.52	84.59	0.0126	86.72	91.18	94.66	92.89	0.0151
Iter 5	48.47	49.85	94.61	65.30	0.0174	75.67	80.48	92.68	86.15	0.0177	73.82	78.91	91.95	84.94	0.0151	86.69	91.87	93.89	92.87	0.0181
Alternative Loss Functions (Final Iteration
BCE	74.49	87.25	83.58	85.38	0.0174	81.71	91.70	88.24	89.94	0.0140	81.53	89.23	90.43	89.83	0.0145	87.85	90.90	96.33	93.53	0.0130
Focal	74.01	86.37	83.80	85.07	0.0174	81.59	92.02	87.80	89.86	0.0140	80.98	88.31	90.70	89.49	0.0145	87.91	91.42	95.82	93.57	0.0130
Dice	74.38	86.40	84.25	85.31	0.0174	81.75	91.55	88.42	89.96	0.0140	81.37	88.83	90.65	89.73	0.0145	88.28	92.91	94.66	93.78	0.0130
Dense Supervision
Dense	77.93	86.30	88.94	87.60	100.0	84.07	89.95	92.79	91.35	100.0	85.71	92.08	92.53	92.30	100.0	90.80	93.52	96.90	95.18	100.0

Figure 4. Visual progression of segmentation results across iSAGE iterations. Column labels 1–6 follow the annotation tool’s 1-indexed sessions and correspond to iterations 0–5 in the text; IoU values printed on panels are per-image, not dataset means.

On the multiclass task, iSAGE reached 74.79% mIoU and 84.83% macro F1 at iteration 5 with 0.040% cumulative annotation effort across all five classes, recovering 96.9% of the dense ceiling (77.17% mIoU) (Table 3). Per-class IoUs at iteration 5 were 88.72 (permeable area), 81.89 (building), 82.36 (road), 70.72 (car), and 50.27 (background).

Table 3. Multiclass segmentation results on BsB Aerial. Macro metrics and per-class IoU (%). Underlined values are the best per column within each block. Iteration-5 iSAGE values and the loss-function panel are means over five training seeds (standard deviation at most 0.3 mIoU).

Setting	Macro Metrics (%)					IoU per Class (%)
	mIoU	mPrecision	mRecall	mF1	Effort	Background	Car	Road	Building	Perm. Area
iSAGE (EWDL)
Iter 0	58.38	71.63	77.08	72.97	0.0068	38.46	47.08	68.90	59.71	77.74
Iter 1	67.89	77.91	83.13	79.81	0.0135	45.64	56.30	78.18	73.08	86.23
Iter 2	72.05	82.34	83.41	82.86	0.0202	47.45	67.36	79.72	78.05	87.69
Iter 3	73.19	83.04	84.79	83.80	0.0270	47.89	68.03	81.37	80.52	88.13
Iter 4	74.14	83.41	85.64	84.36	0.0337	49.17	69.86	81.81	81.48	88.38
Iter 5	74.79	84.07	85.72	84.83	0.0404	50.27	70.72	82.36	81.89	88.72
Iterative Random Selection
Iter 0	56.87	69.67	75.17	71.31	0.0068	34.39	47.01	67.74	58.70	76.50
Iter 1	64.73	75.99	81.27	77.56	0.0136	41.65	53.89	74.51	71.68	81.95
Iter 2	66.67	77.49	82.35	79.04	0.0205	43.23	57.50	75.31	73.76	83.55
Iter 3	67.63	78.16	82.69	79.75	0.0273	43.74	59.21	76.06	75.07	84.08
Iter 4	68.41	78.77	83.25	80.34	0.0342	44.81	60.15	76.72	75.55	84.84
Iter 5	69.11	79.39	83.25	80.83	0.0410	45.02	61.73	77.08	76.33	85.36
Dense Supervision
Dense	77.17	87.13	86.13	86.45	100.0	53.63	73.09	83.76	84.71	90.70
Alternative Loss Functions (Final Iteration)
Dice $(\lambda =1)$	73.37	82.78	85.12	83.81	0.0404	48.15	68.66	81.63	80.28	88.15
Cross-Entropy	74.11	83.35	85.53	84.32	0.0404	48.94	69.52	82.29	81.38	88.43
Focal $(\gamma =2)$	74.66	83.85	85.82	84.74	0.0404	50.26	70.24	82.07	82.04	88.69

On the binary tasks, random selection reached final IoUs of 48.47 (car), 75.67 (road), 73.82 (building), and 86.69 (permeable area), with gaps to iSAGE of −25.6, −6.2, −7.7, and −1.5 IoU respectively (Table 2, Figure 5). Random selection saturated the budget cap on every frame, so its cumulative effort matched or exceeded iSAGE’s on every class.

Figure 5. Qualitative comparison of final-iteration segmentation results. Columns: Ground truth, random selection, dense supervision, BCE, Focal, Dice, and iSAGE. IoU values printed on panels are per-image, not dataset means.

On the multiclass task, random selection plateaued at 69.11% mIoU at iteration 5, 5.68 points below iSAGE at matched annotation effort (Table 3).

On the four binary tasks (Table 2, bottom panels), EWDL and the alternative losses land within 0.6 IoU of one another on every class at the final iteration.

On the multiclass task, evaluated on the iter-5 annotation set (Table 4), $\lambda \in \{2,5,10\}$ landed within 0.16 mIoU of one another (74.63, 74.79, 74.66), with per-$\lambda$ standard deviations of 0.19 to 0.33. $\lambda =1$ (standard Dice) reached 73.37, and $\lambda =20$ dropped to 72.18. Against alternative loss families under the same protocol (Table 3, bottom panel), EWDL ($\lambda =5$) at 74.79% led Cross-Entropy (74.11 ± 0.09) by 0.68 points and matched Focal ($\gamma =2$, 74.66 ± 0.20) within one standard deviation (EWDL 74.79 ± 0.27).

Table 4. EWDL hyperparameter sensitivity on BsB Aerial multiclass. Mean mIoU and per-class IoU (%) across 5 seeds under varying error-penalty $\lambda$ ($\lambda =1$ recovers standard Dice). Bold marks the default setting; underlined values are the best per column. Standard deviations range from 0.19 to 0.33 mIoU.

$\mathit{\lambda }$	mIoU	Car	Road	Building	Perm.	BG
1 (=standard Dice)	73.37	68.66	81.63	80.28	88.15	48.15
2	74.63	70.38	82.23	82.03	88.53	49.97
5 (default)	74.79	70.72	82.36	81.89	88.72	50.27
10	74.66	71.29	82.51	81.40	88.60	49.50
20	72.18	71.68	82.51	73.84	88.55	44.35

Across the four backbones trained on the same iter-5 annotation set (Table 5, the means over five seeds), iSAGE reached 74.79 (U-Net + Eff-B7), 72.32 (U-Net + R101), 70.69 (DLV3+ + R50), and 71.86 mIoU (SegFormer + MiT-B2), recovering 96.9, 94.4, 94.1, and 95.7% of each architecture’s matched dense ceiling. Table 5 also lists each architecture’s parameter count and multiply-accumulate cost. A 100-epoch iteration with the primary configuration took about 30 min on BsB Aerial and about one hour on ISPRS Vaihingen (Section 5.2), so the six trainings of a full protocol were completed within a working day.

Table 5. Cross-architecture validation on BsB Aerial multiclass with U-Net + EfficientNet-B7 (Eff-B7), U-Net + ResNet-101 (R101), DeepLabV3+ (DLV3+) + ResNet-50 (R50), and SegFormer + MiT-B2 backbones. mIoU and per-class IoU (%) for iSAGE (mean over five training seeds on the iteration-5 annotation set, identical training protocol across architectures; standard deviations at most 0.53 mIoU) and matched-architecture dense baselines. Params, trainable parameters in millions; GMACs, multiply-accumulate operations in billions for one 256 × 256 forward pass. The underlined row is the primary configuration used throughout the paper.

Architecture	Encoder	Params	GMACs	iSAGE (%)						Dense	iSAGE/
				mIoU	Car	Road	Build.	Perm.	BG	mIoU	Dense
U-Net	Eff-B7	67.1	3.2	74.79	70.72	82.36	81.89	88.72	50.27	77.17	96.9%
U-Net	R101	51.5	15.6	72.32	67.95	81.52	77.12	88.01	46.99	76.63	94.4%
DLV3+	R50	26.7	9.2	70.69	60.94	79.66	79.59	86.97	46.32	75.14	94.1%
SegFormer	MiT-B2	24.7	5.3	71.86	61.30	80.21	81.30	88.28	48.20	75.11	95.7%

5.2. ISPRS Vaihingen

The Vaihingen results are divided into the comparison against published methods on the shared benchmark and the four controlled output-reading baselines run under iSAGE’s protocol.

5.2.1. External Benchmarking

iSAGE reached 76.65% mIoU on the ISPRS Vaihingen five-class benchmark at iteration 5 (mean over five training seeds, ±0.12) with 29,052 labeled pixels (0.011% of training pixels), matching the dense baseline trained under iSAGE’s protocol (76.93 ± 0.27, five seeds) within 0.28 points (Table 6). Against published weakly supervised methods, iSAGE led EasySeg by 3.82 points (76.65 vs. 72.83), D2ADA by 4.90 (vs. 71.75), ILM-ASSL by 6.23 (vs. 70.42), and RIPU by 6.74 (vs. 69.91), while consuming about 40% of EasySeg’s labeled pixels and training on 44% fewer patches. Competitor numbers are as reported in EasySeg’s evaluation [29], with all methods evaluated on the same 17-tile test partition. The domain adaptation entries additionally train on dense source-domain supervision from Potsdam, which iSAGE does not use, so the comparison is conservative with respect to total supervision. The load-bearing internal control is the dense baseline trained under iSAGE’s protocol (76.93%), reported in the same table. The final-epoch versus best-validation asymmetry is verified to have negligible impact. The gap between iSAGE’s final-epoch and best-validation checkpoints on Vaihingen is within 0.03 mIoU under the learning-rate decay schedule used here.

Table 6. Comparison on ISPRS Vaihingen considering per-class IoU (%) and labeling cost. Weakly supervised competitor numbers from Yang et al. [29]. iSAGE and Fully Supervised values are means over five training seeds (standard deviations of at most 0.12 and 0.27 on mIoU, 0.34 and 0.50 per class). Bold marks the best value per column; iSAGE is this work.

Method	Cost	mIoU	Imperv.	Build.	Tree	Car	Low Veg.
Unsupervised Domain Adaptation (no target labels)
No Adaptation	0%	30.19	35.24	42.23	43.57	1.39	28.50
ADVENT [96]	0%	40.84	53.82	56.59	49.21	24.46	20.12
CLAN [97]	0%	46.40	63.38	61.78	55.03	28.98	22.83
Active Domain Adaptation
MADAv2 [98]	1%	50.62	65.73	59.94	54.26	21.57	51.58
RIPU [89]	0.015%	69.91	80.09	86.17	65.02	53.02	65.26
ILM-ASSL [91]	1%	70.42	81.03	87.28	64.36	57.13	62.32
D2ADA [90]	1%	71.75	80.71	87.62	66.69	57.58	66.17
EasySeg [29]	0.015%	72.83	81.62	88.40	69.16	57.90	67.25
Supervised Learning
Fully Supervised	100%	76.93	85.62	89.16	71.05	68.99	69.85
iSAGE	0.011%	76.65	83.93	88.23	71.55	70.02	69.51

The per-class breakdown at iteration 5 (Table 6) concentrates the iSAGE vs. published gap on cars: iSAGE reached 70.02% car IoU against 57.90 for EasySeg (the best published competitor on this class). Across the other four classes, iSAGE led EasySeg by 2.31 on impervious (83.93 vs. 81.62), 2.39 on trees (71.55 vs. 69.16), and 2.26 on low vegetation (69.51 vs. 67.25), and they essentially tied on buildings (88.23 vs. 88.40).

Table 7 counts annotation effort in pointing actions. Redrawing the dense ground truth as simplified polygons takes 412,812 vertices at the 2-pixel tolerance and 686,071 at 1 pixel, against 29,052 iSAGE clicks, so even an idealized dense annotator performs 14 to 24 times more pointing actions (156 times for exact reproduction with no tolerance). The advantage is largest on classes with long irregular boundaries (impervious surfaces, trees, low vegetation, 19 to 21 times) and smallest on rectilinear buildings (6.3) and small cars (5.6).

Table 7. Dense annotation cost in pointing actions on ISPRS Vaihingen. Vertices: Douglas–Peucker-simplified polygon vertices of the dense ground truth at tolerance $ϵ$ (pixels). Share: Class fraction of the ground truth pixels. Clicks: iSAGE labeled pixels at iteration 5. Ratio: Vertices at $ϵ=2$ per click.

Class	Share (%)	Polygons	Vertices ($\mathit{ϵ}$ = 1)	Vertices ($\mathit{ϵ}$ = 2)	Clicks	Ratio ($\mathit{ϵ}$ = 2)
Impervious	31.4	7302	209,550	123,011	5981	20.6×
Building	28.2	4560	72,521	37,618	5925	6.3×
Tree	21.6	7131	181,470	112,534	5932	19.0×
Car	1.5	5074	46,396	31,074	5574	5.6×
Low vegetation	17.3	8310	176,134	108,575	5640	19.3×
Total	100.0	32,377	686,071	412,812	29,052	14.2×

5.2.2. Output-Reading Baselines

All four baselines share iSAGE’s iter-0 seed annotations and model bootstrap, as well as the same training schedule and architecture. Oracle entropy and uniform random keep the same per-iteration budget cap as iSAGE, with the oracle selecting the highest-entropy pixel per predicted class per frame with the ground truth label assigned at the queried coordinate, and uniform random drawing one pixel per ground truth class per frame independently of the predictions, with the oracle additionally evaluated at 10×, 50×, and 100× that budget. Pseudo-labeling and CRF propagation expand the supervision set across iterations by adding every pixel whose predicted-class confidence exceeds the threshold, so by iteration 5, the training mask covers approximately 90% of the training pixels rather than 0.011%. Pseudo-labeling is evaluated at confidence thresholds 0.90, 0.95, and 0.99. The comparison is therefore between iSAGE with 29,052 human-targeted pixels and the expansion baselines with roughly 240 million auto-generated pseudo-labels.

Oracle entropy (Figure 6) plateaus at 66.38% mIoU by iteration 3 and stays there through iteration 5, 10.27 points below iSAGE. Raising the oracle budget to 10×, 50×, and 100× moves the iteration-5 plateau to 66.26%, 67.01%, and 67.85%, the last reaching 0.95% of training pixels and still 8.80 points below iSAGE. Uniform random tracks the 1× oracle closely at 66.60% at iteration 5 (gap of 0.22 points to the oracle), confirming that the model’s output distribution carries no acquisition signal beyond what uniform sampling extracts. Pseudo-labeling at confidence 0.95 rises monotonically to 69.35% at iteration 5, with per-iteration gains shrinking from +1.76 to +0.18 between consecutive rounds, plateauing 7.30 points below iSAGE even at the 240-million-pixel supervision scale. At thresholds 0.90 and 0.99, the iteration-5 plateau is 69.00% and 69.34%, a 0.35 pp spread across the three thresholds. CRF-based label propagation peaks at 68.78% at iteration 2 and then descends to 66.00%, 63.86%, and 62.25% at iterations 3, 4, and 5, ending 14.40 points below iSAGE and 6.53 points below its own peak (Table 8).

Figure 6. Per-iteration mIoU on ISPRS Vaihingen for iSAGE and the four output-reading baselines (oracle entropy, pseudo-labeling, DenseCRF label propagation, and uniform random) under identical protocol. The dense baseline trained under iSAGE’s protocol is shown for reference; the vertical axis starts at 60 mIoU. Each curve is a single training chain.

Table 8. Iteration-5 mIoU on ISPRS Vaihingen for iSAGE and the output-reading baselines under identical protocol. All baselines start from the same iter-0 seed model. “Pixels labeled” reports the cumulative percentage of training pixels carrying a supervision signal at iteration 5. iSAGE (bold) is the reference for the gap column.

Method	Pixels Labeled (%)	Iter-5 mIoU (%)	Gap to iSAGE (pp)
iSAGE	0.011	76.65	0.00
Uncertainty acquisition (oracle entropy budget sweep) [57,69]
Oracle entropy 1×	0.011	66.38	−10.27
Oracle entropy 10×	0.10	66.26	−10.39
Oracle entropy 50×	0.48	67.01	−9.64
Oracle entropy 100×	0.95	67.85	−8.80
Self-training pseudo-labels (confidence threshold sweep) [25,26]
Pseudo-labeling (0.90)	∼95	69.00	−7.65
Pseudo-labeling (0.95)	93.0	69.35	−7.30
Pseudo-labeling (0.99)	84.8	69.34	−7.31
CRF-based label propagation [95]
DenseCRF (0.95)	96.7	62.25	−14.40
Control
Uniform random	0.011	66.60	−10.05

6. Discussion

6.1. iSAGE in the Landscape and Why This Position Is Justified

This positioning proceeds in three steps: the output-reading limit, iSAGE’s place in the 35-method comparison, and the theoretical and empirical support for the limit.

6.1.1. The Output-Reading Limit

This paper argues that the six output-reading mechanisms current frameworks layer on top of human supervision do not address what they were assumed to address: as rules whose input is a single model’s predictive distribution, they cannot separate the pixels where the model is confidently wrong from those where it is confidently correct, because the two can be identical at the level these rules operate on. iSAGE is the methodological consequence of taking that limit seriously, a sparse human click delivered directly on the image with no machinery in between.

Three theoretical regimes lie outside this argument. Ensemble disagreement across independently trained models can, in principle, surface a subset of confident errors when ensemble members fail in uncorrelated ways. Out-of-distribution detection scores can flag confident errors that coincide with covariate shift. A supervised error-detector trained on image features with labeled examples of model errors is a function over outputs augmented with external error supervision. All three require resources the present setting does not assume (multiple parallel training budgets, an in-distribution prior, or labeled error examples respectively), and the empirical reach of each is limited: correlated ensemble failures persist under the calibration regimes studied by Gustafsson et al. [70], OOD scores degrade precisely when the wrong-class confidence is shape- or texture-driven rather than distribution-driven, and supervised error-detectors require the very signal whose acquisition iSAGE addresses. The information limitation therefore holds within the single-model, output-only regime that the 35-method comparison of Table 9 occupies and that this paper investigates, and the rest of the argument is framed accordingly.

Table 9. Conceptual comparison of segmentation frameworks across supervision mechanisms. Column abbreviations: Iter., iterative execution; HIL, human-in-the-loop feedback; Acq., automated acquisition function over the model’s predictive distribution; Prop., propagation heuristics; Pseudo, pseudo-labels; Consist., consistency regularization; DA, domain-adaptation source; FM, foundation-model labeling. A check mark indicates the mechanism is present. iSAGE (bold) is this work.

Method	Annotation	Iter.	HIL	Acq.	Prop.	Pseudo	Consist.	DA	FM
iSAGE	Sparse Clicks	✓	✓
Weak/sparse supervision (non-iterative)
Bearman et al. [19]	Sparse points
ScribbleSup [34]	Scribbles				✓
FESTA [20]	Scribbles				✓
Liu et al. [36]	One point				✓
PAMSNet [54]	Sparse points
Iterative weak/self-training/semi-supervised
Tree Energy [47]	Scribbles	✓			✓	✓
ScribFormer [99]	Scribbles	✓				✓	✓
PyMIC [48]	Points/Scribbles	✓				✓	✓
UniMatch V2 [100]	Unlabeled + few	✓				✓	✓
PENet [55]	Sparse points	✓			✓	✓			self
AESAM [56]	Sparse points	✓				✓	✓		self
Active learning/interactive (single-domain)
Desai and Ghose [51]	Points			✓		✓
DIAL [27]	Clicks	✓	✓	✓	✓
HAL-IA [28]	Sparse points	✓	✓	✓	✓
ViewAL [67]	3D points	✓		✓
S4AL [62]	Points	✓		✓		✓
ESA [65]	Clicks	✓	✓	✓	✓
Think Twice [64]	Points	✓		✓
Cold AL [31]	Points	✓		✓
Teng and Wang [33]	Points	✓		✓
Adapt. Superpixel AL [101]	Superpixels	✓		✓	✓
BalEntAcq [66]	Sparse pixels	✓		✓
ALC [80]	Clicks	✓	✓	✓		✓			self
A2LC [81]	Clicks	✓	✓	✓		✓			self
Active domain adaptation
RIPU [89]	Regions	✓		✓			✓	✓
D2ADA [90]	Regions	✓		✓				✓
ILM-ASSL [91]	Sparse points	✓		✓		✓		✓
EasySeg [29]	Sparse points	✓	✓	✓	✓	✓	✓	✓
Few-shot/zero-shot/foundation-model
PANet [102]	Support masks
RePRI [18]	Support masks
SAM [11]	Prompts/clicks								self
CLIPSeg [12]	Text/image								self
PointSAM [82]	Points/prompts	✓				✓			self
SEEM [103]	Prompts/text		✓						self

6.1.2. The Only Framework Without Auxiliary Machinery

Among the iterative human-in-the-loop frameworks in the 35-method comparison, iSAGE is the only one that adds no auxiliary machinery at all, and the experiments below show that this subtraction did not cost accuracy in the settings tested.

Table 9 compares the 35 surveyed methods, iSAGE included, across eight supervision mechanisms. The categorization is descriptive of mechanism types rather than a measurement of relative effectiveness. Within this survey, every iterative human-in-the-loop method incorporates at least one of the six auxiliary mechanisms (acquisition, propagation, pseudo-labels, consistency regularization, domain adaptation, or foundation-model labeling). iSAGE is the single exception.

The minimalism is informational rather than stylistic. Any selection rule defined on the model’s own predictive distribution cannot separate a confidently wrong pixel from a confidently correct one: at the level of that distribution, the two are identical by construction. The distinguishing information lies outside the model’s outputs, and supplying it is exactly the role of the human click. Each machinery column except domain adaptation is a rule operating on that distribution in some form. Acquisition functions surface uncertainty rather than error, propagation heuristics extrapolate from the same predictions, pseudo-labeling and consistency regularization train on the model’s own beliefs, and foundation-model labeling swaps the target model for a different model with the same blind spot, also inheriting that model’s boundary imprecision as a second source of noise. Domain adaptation is the one column outside this argument because it addresses cross-domain transfer rather than intra-domain confident errors.

iSAGE keeps only three components: the human click, the loss, and the platform, and the annotation record itself becomes the supervision artifact. A given record uniquely determines the training signal with no propagation hyperparameters, so reported performance reflects exactly what the annotator provided.

6.1.3. Theoretical and Empirical Support

Recent theoretical work argues that the aleatoric-epistemic dichotomy is insufficiently expressive and that popular information-theoretic measures over outputs are poor estimators of what they purport to quantify [104], which provides formal backing for this information limitation beyond the specific acquisition and propagation mechanisms tested here. Bayesian acquisition signals such as MC-dropout disagreement [69] and posterior-epistemic decompositions [66] remain within the single-model output-only regime the information limitation bounds. Deep ensembles [70] read more than one model and fall under the ensemble regime already delimited above.

The four output-reading mechanisms tested behave as the limitation predicts. Oracle-entropy acquisition, given ground truth answers on every query at iSAGE’s per-iteration budget (one pixel per predicted class per frame), plateaus at 66.38% mIoU and remains 10.27 points below iSAGE. Raising the oracle budget to 10×, 50×, and 100× labels per class per frame holds the plateau between 66.26% and 67.85%, the last reaching 0.95% of training pixels and still 8.80 points below iSAGE. Uniform random selection at the 1× budget tracks the oracle within 0.22 mIoU, confirming that no acquisition reading the model’s predictive distribution outperforms a baseline that does not. The reason is geometric: confident errors lie in the low-entropy region of the predictive distribution, precisely where uncertainty-based acquisition does not look. An entropy oracle, even handed the correct label at every query, never selects them, which is why it performs no better than uniform random. Self-training pseudo-labels rise monotonically to 69.35% by iteration 5, expanding the supervision set to 93% of the training pixels through self-confidence above 0.95, plateauing 7.30 points below iSAGE. Varying the pseudo-label confidence threshold between 0.90 and 0.99 spans the iteration-5 mIoU from 69.00% to 69.35%, a 0.35 pp range that places the information limitation beyond threshold calibration. CRF-based label propagation peaks at 68.78% by iteration 2 and then degenerates to 62.25% by iteration 5, ending below every other baseline as spatial smoothing of softmax outputs compounds confident errors across iterations. Across four mechanisms, three oracle budget scales, and three pseudo-label confidence thresholds, no acquisition or expansion strategy reading the model’s predictive distribution reaches iSAGE. The limit is not a budget question, a threshold question, or a smoothing question; it is a question of what information that distribution contains.

6.2. Findings

Four findings organize this subsection: the match with dense supervision, the per-class dynamics, the transfer of the annotation record across architectures, and the role of the loss.

6.2.1. Match-with-Dense Behavior

The relationship between iSAGE and dense supervision depends on the quality of the dense ground truth, and the two datasets expose opposite ends of that dependence. BsB Aerial was annotated under controlled conditions by a single annotator with consistent boundary conventions, and iSAGE recovers 96.9% of dense (74.79% vs. 77.17%, 0.040% labels). ISPRS Vaihingen carries known boundary inconsistencies across tiles, and iSAGE matches dense under an identical protocol within 0.28 points (76.65% vs. 76.93%). Dense supervision absorbs every inconsistency in the training labels and reproduces them at inference, while iSAGE labels only pixels where the class is visually unambiguous and is, therefore, less sensitive to boundary-label noise. The asymmetry is consistent with the framework’s design rather than a chance effect: iSAGE approaches dense when dense is clean and reaches parity when dense carries label noise. With two datasets at opposite ends of dense-label quality, this is a two-point observation rather than an established property. The reported numbers are also a lower bound: the one-pixel-per-class regime is the adversarial worst case, not a recommended budget, and practical deployments relax the cap. The match is additionally robust to the final-epoch versus best-validation reporting choice, with the gap between the two on iSAGE within 0.03 mIoU (Section 5.2). Recent point-supervised methods that expand their seeds algorithmically show the complementary pattern on the same benchmark. Under their own protocols and their own dense baselines, PENet reports 68.24 mIoU against a dense ceiling of 81.29 on Vaihingen, and AESAM reports 62.07 against 73.46, with recovery ratios of about 84% [55,56]. The double-digit gap these expansion pipelines leave to their dense ceilings is the gap iSAGE closes.

This compression has a simple explanation. In a converged model, most pixels are confidently correct, and a label on such a pixel contributes almost no gradient, since it only confirms what the model already predicts. The learning signal concentrates on the minority of pixels the model gets wrong, most of which lie at object boundaries or on small and high-variability classes. Dense supervision spends the overwhelming majority of its labels on the confident-correct majority, where they are redundant. iSAGE matches it not by extracting more from each label but by spending labels only where they carry information. The roughly 9000-fold reduction is therefore expected rather than surprising, as it reflects the fraction of dense supervision that was ever informative.

6.2.2. Per-Class Dynamics

The per-class pattern maps onto the stuff versus things distinction. iSAGE and random selection both recover most of dense performance on permeable areas (88.72% iSAGE, 85.36% random, and 90.70% dense, Table 3), but diverge sharply on cars (70.72% iSAGE, 61.73% random, and 73.09% dense). Targeted clicks reach the same neighborhood as dense supervision on cars (within 3 points), while random sampling stays 11 points behind. The binary experiments show the same pattern in isolation: permeable saturates from the seed, buildings and roads converge in a few iterations, and cars require the most late-iteration effort. The iter-0→iter-5 growth is +46.6 IoU on cars against +6.4 on permeable, with roads (+13.7) and buildings (+26.5) in between (Table 2, Figure 7). Two object properties explain the spread: a small spatial extent dilutes the per-click gradient signal, and high shape variability surfaces new confident errors across iterations. These dynamics suggest a heuristic for budget planning: deprioritize homogeneous classes once they reach target performance, and reserve the late-iteration budget for small high-variability classes where confident errors persist.

Figure 7. Per-class IoU across iSAGE iterations on the BsB Aerial binary tasks. Solid lines show iSAGE (EWDL), and dotted lines mark the dense-supervision ceiling for each class. Each trajectory is a single training chain.

6.2.3. Annotation-Record Transfer Across Architectures

The annotation record produced under one architecture transfers across backbones. Trained unchanged on the record from the U-Net + EfficientNet-B7 session, the four architectures recover between 94.1% and 96.9% of their own dense ceilings: U-Net with EfficientNet-B7 (96.9%) and with ResNet-101 (94.4%), DeepLabV3+ with ResNet-50 (94.1%), and SegFormer with MiT-B2 (95.7%) (Table 5). The recovery ratio stays above 94% across both convolutional and transformer encoders, indicating that the clicks capture scene structure rather than the idiosyncrasies of the backbone that selected them, and that the record transfers to whatever segmentation model a deployment already uses. This reach follows from the framework’s only architectural requirement, per-pixel class probabilities and an ignore mask, so any semantic segmentation model that produces per-pixel class scores can host the loop. The validated configurations span 24.7 to 67.1 million parameters and 3.2 to 15.6 GMACs per forward pass (Table 5), and the recovery band holds across that span.

6.2.4. The Loss Helps Slightly, the Strategy Drives the Result

The error-weighted loss gives a small benefit, but the real driver of performance is the error-driven strategy rather than the loss. Emphasizing hard pixels helps a little: trained from scratch on the same sparse annotation set, EWDL ($\lambda =5$, 74.79%) and Focal (74.66%) lead plain Dice ($\lambda =1$, 73.37%) by 1.3 to 1.4 points and match each other within one standard deviation, which is expected because Focal applies a similar hard-example emphasis (Table 3, bottom panel). Over-amplification hurts ($\lambda =20$ falls to 72.18%, Table 4). That margin is small next to the gap the strategy itself opens: under the same protocol, error-driven selection reaches 74.79%, while random selection at matched budget plateaus about 5.7 points lower (69.11%). EWDL’s value to iSAGE is therefore coherence rather than magnitude. The same indicator that defines where the annotator clicks ($\arg \max y_{pr}\left(x\right)\ne y_{gt}\left(x\right)$) defines where the loss amplifies the gradient, so every penalty pixel maps back to a specific click in the JSON record, and the optimization step inherits the auditability of the acquisition step.

6.3. Operational and Cognitive Properties of the Workflow

The properties discussed in this section are design consequences that follow from the framework’s architecture rather than measured outcomes.

6.3.1. Incremental Deployment and Maintenance

iSAGE’s minimalism enables four operational capabilities that dense-mask pipelines do not provide by default. Deploying the model to fresh imagery reduces to opening the platform on the current model and clicking visible errors, with the existing annotation record remaining valid. Adding a new class costs only clicks on instances of that class, with no revisit to prior data. Correcting an annotation error is a single JSON edit, locatable by image and iteration without redrawing any region. Any past iteration is reloadable from its session directory, so state can be rewound, re-evaluated, or compared against alternative methods using the same record. These capabilities share one root: the annotation record is the dataset, the platform reads it directly, and no compiled mask artifact sits between human decision and gradient. In production contexts where models need ongoing maintenance (new sensors, monitored areas expanding, drift, and evolving class priorities), dense-mask pipelines re-annotate from scratch at each increment, whereas iSAGE opens the platform on the current model and trains the next iteration at the cost of the clicks required. The cost scales with the change, not with the dataset.

6.3.2. An Auditable, Versionable Record

The annotation record itself is a practical artifact. It is a small, human-readable list of tuples rather than a set of mask files, so it can be versioned and inspected like source code. The difference between two iterations is a handful of added clicks, and every label traces back to a specific human decision rather than sitting anonymously in a dense mask, which makes the supervision auditable and easy to share or reproduce.

6.3.3. Directed Rather than Exhaustive Attention

The cognitive structure of the workflow mirrors this design. Dense polygon annotation requires sustained exhaustive attention: every pixel needs a label, including ambiguous boundary pixels where the annotator must decide without clear visual evidence. Fatigue degrades quality and speed, and the resulting errors are silent: a misplaced boundary is indistinguishable from a correct one in the final mask. iSAGE replaces exhaustive attention with directed attention: the annotator clicks only on pixels where visual evidence is unambiguous, skips the rest, and the training pass between iterations imposes a natural break. Deferral is free: a region the annotator cannot confidently classify at one iteration can wait for the next, where the model’s updated prediction either resolves the ambiguity or surfaces it as an explicit error. The model and annotator function as mutual guides across iterations. The loop is also self-balancing: each click counts equally regardless of object size, and correcting a false positive shifts the decision boundary so the next iteration surfaces previously hidden false negatives, rebalancing the error distribution across iterations without explicit intervention. Inspection cost also scales with the number of erroneous regions rather than with the number of pixels, since correct regions require no interaction, and each iteration surfaces fewer errors than the last. These are the cost dimensions that neither labeled-pixel fractions nor wall-clock time capture: a decision made under clear visual evidence costs less than a boundary decision made under ambiguity, independently of how long either takes. Table 7 quantifies the action side of this asymmetry, with dense annotation requiring 14 to 24 times more pointing actions than the click record, concentrated on the classes with the longest and most irregular boundaries.

6.3.4. A Model That Is Always Usable and Never Final

Because each iteration only adds clicks, the model is never final, and the practical stopping point is a visual judgment. The annotator watches the prediction overlay and the diminishing returns between rounds (on BsB Aerial multiclass, +9.51 mIoU from iteration 0 to 1 against +0.65 from iteration 4 to 5) and stops when the remaining errors are not worth correcting. The same loop makes recalibration cheap to check: running the current model on a new area, sensor, or acquisition surfaces its errors directly in the overlay, so confirming whether the model still holds, and adding a few clicks where it does not, costs only the clicks required.

6.4. Limitations and Future Work

iSAGE’s empirical validation is confined to the aerial remote sensing family, matching the prevailing convention in human-in-the-loop semantic segmentation, where each framework is validated within a single imaging-domain family (aerial RS [20,27,29], driving scenes [89,90,91], point clouds [67,68], or natural images [19,34]). Within that family, the evidence already spans two sensors, two resolutions, two geographies, seven class types, and four architectures. The two claims that organize this paper have different reach. The information limitation is a property of single-model output-only acquisition and does not depend on the datasets used here, whereas the effectiveness of the iSAGE framework is an empirical result established within this validated scope. The protocol carries no domain-specific priors: a click is an $(x,y,\mathrm{class})$ tuple, the JSON record is a flat list of such tuples, and EWDL operates on per-pixel correctness regardless of spectral or geometric properties. Extension to other sensors (SAR, multispectral, or hyperspectral), modalities (medical or industrial inspection), or tasks (change detection or instance segmentation) therefore requires preprocessing, encoder, or UI adjustments rather than methodological changes.

Several constraints frame the present design, each pointing to a clear evolution. The experiments use small patches (256 × 256 on BsB Aerial and 512 × 512 on Vaihingen), where some objects are hard to label because they do not fit fully within the field of view. Larger patches would help on two fronts: each object’s full extent becomes visible, making completeness easier to judge, and the annotation effort per total pixel drops further. Running iSAGE at larger patch sizes mainly requires a segmentation backbone that accommodates the larger input, a straightforward extension rather than a redesign. Multi-scale analysis and adaptive patch reshaping are complementary routes to the same end. Adaptive patch reshaping subdivides image and label patches and reassigns the pieces (by an information-theoretic similarity) to representative class characteristics, which improves the delineation of class boundaries on Sentinel-2 imagery [105]. Related strategies size the patches by local edge density so that boundary-rich regions receive finer patches and homogeneous regions coarser ones [106] or select the patch scale per geo-object in high-resolution remote sensing imagery [107]. These routes would let objects of widely different sizes be inspected and annotated at appropriate scales while preserving the boundaries between them. The annotation primitive is a single-pixel click, chosen for per-decision auditability. Scribbles, polygons, mixed-mode primitives, and exclusion clicks (“this pixel is not class c”) are additive extensions rather than replacements and map naturally onto complementary-label losses studied for noisy-label learning. EWDL was designed for sparse supervision and may penalize pixel-level errors disproportionately under dense annotation with imperfect masks, but this case lies outside the protocol’s premise. The workflow assumes an expert in the loop, which scopes its applicability to domains where class membership depends on conventions a non-expert cannot apply consistently, such as aerial imagery, medical segmentation, and industrial inspection.

The workflow also has boundary conditions. The loop depends on errors being visually adjudicable. Deferral lets an ambiguous region wait for the next iteration (Section 6.3), but a region that stays ambiguous across iterations never receives a click, so classes that cannot be resolved from the imagery alone gain little from error-driven selection. Gains also concentrate where confident errors are visible and localized. Diffuse classes benefit less, as background remains the weakest class under both supervision regimes on BsB Aerial (50.27 IoU for iSAGE against 53.63 dense), and the one-pixel-per-class budget was not tested on extremely imbalanced class distributions, where a rare class may surface few visible errors per iteration and need a larger per-class budget.

Three directions follow from these boundaries and from the infrastructure introduced here. Automated acquisition and human correction target different supervision signals rather than competing, since acquisition operates over predictive-distribution statistics, where confident errors are indistinguishable from confident correct predictions, while human correction supplies the externally derived signal that distinguishes them, so combining automated coverage with human-targeted confident-error correction is an open hybrid direction. The record-as-dataset design supports per-class behavior studies mapping class characteristics (geometry, texture, instance scale, and boundary type) to convergence behavior, turning the stuff versus things contrast observed here into a predictive model. The cost accounting invites the same treatment. Annotation effort is reported here in labeled-pixel fractions and pointing actions, and the wall-clock dimension calls for a study designed specifically to measure it, timing both the click-based and the dense workflow on the same imagery under a protocol that separates idle from active labeling time and controls for inter-annotator variability, per-class difficulty, and the changing cost across iterations. The platform’s session instrumentation supports such a study, and it is a natural follow-up.

7. Conclusions

This work addresses a gap in sparse-supervised semantic segmentation. Prior interactive frameworks combine human input with auxiliary machinery (pseudo-labels, propagation heuristics, uncertainty-based acquisition, consistency regularization, foundation-model labeling, and domain adaptation), and wherever these mechanisms select pixels by reading a single model’s predictive distribution, a confidently wrong pixel is indistinguishable from a confidently correct one. iSAGE is proposed to investigate whether such machinery is necessary, configuring a framework in which the specialist’s clicks on confident model errors constitute both the annotation record and the training supervision, the Error-Weighted Dice Loss amplifies the gradient at those labels, and an integrated software platform hosts inspection, annotation, and retraining as a continuous workflow.

Empirical validation on two aerial datasets shows that this minimalist configuration suffices to match dense fully supervised performance under adversarial labeling regimes. iSAGE recovered 96.9% of dense supervision on BsB Aerial (74.79% mIoU at 0.040% labeled pixels) and matched the dense baseline within 0.28 points on ISPRS Vaihingen (76.65% vs. 76.93%), exceeding the published weakly supervised methods evaluated on this benchmark. Four output-reading baselines plateaued below iSAGE under the same training protocol: oracle entropy with ground truth labels at every query at 66.38% mIoU (66.26 to 67.85% across a 1 to 100× budget sweep), uniform random sampling within 0.22 points of the oracle, self-training pseudo-labels at 69.00 to 69.35% across confidence thresholds 0.90 to 0.99, and CRF-based label propagation peaking at 68.78% before degenerating to 62.25% by iteration 5. These results are consistent with the gap being informational rather than a matter of budget, confidence threshold, or smoothing.

Within the 35-method comparison of Table 9, iSAGE is the only iterative human-in-the-loop approach operating without auxiliary machinery, supporting the claim that, in the settings tested, delivering the human signal directly at confident errors made the accumulated machinery of prior pipelines unnecessary. The open-source platform that enabled this validation is also a research instrument for follow-up work on cross-domain validation, annotation primitive comparisons, per-class convergence modeling, and inter-annotator dynamics within and beyond the validated remote sensing aerial scope.