Optimising Active Learning with a Decreasing-Budget-Based Strategy: A Medical Application Case Study

Abstract

The collection and annotation of data for supervised machine learning remain challenging and costly tasks, particularly in domains that demand expert knowledge. Depending on the application, labelling may require highly specialised professionals, significantly increasing the overall effort and expense. Active learning techniques offer a promising solution by reducing the number of annotations needed, thereby lowering costs without compromising model performance. This work proposes an active learning with a decreasing-budget-based strategy to reduce the effort required to annotate medical images. The strategy encourages data annotators to focus on initial iterations, optimise budget allocation, and ensure that the trained model achieves maximum performance with reduced effort in subsequent iterations. This strategy also improves the performance of deep learning models, which perform better with fewer images, reducing the specialists’ workload. This work also introduces three experiments that contribute to understanding the impact of the strategy in the annotation process.

1. Introduction

Today, the discipline of pathology is evolving towards digital pathology (DP) due to the emergence of whole slide image (WSI) technology and the increased computing power of computers through graphics processing units (GPU). According to [1], applications based on artificial intelligence (AI) algorithms and methods are transforming medical fields such as pathology. The results of these algorithms achieve better diagnostic accuracy and reproducibility by consistently handling a large amount of information in every case. In addition, diseases such as cancer are increasing in incidence in the population, requiring more pathologists to diagnose them. In this sense, AI can help to meet the growing demand for experienced pathologists.

In the field of AI, deep learning (DL) neural networks (NNs) have become the standard approach for many image-based tasks (e.g., classification, object detection, and segmentation). They enable the automation of processes such as tissue morphology analysis, cell and mitosis counting, biomarker quantification, and histopathological feature extraction, thereby helping pathologists reduce their workload, shorten report submission times, and standardise clinical practices. These benefits are evidenced by the introduction of clinical decision support systems (CDSS) in digital pathology (DP).

The development of an intelligent CDSS depends on the successful application of AI techniques and algorithms in WSI. This type of technology is expected to improve the performance of medical specialists in terms of precision, objectivity, and reproducibility [2]. However, using deep neural networks to solve computer vision problems involves learning thousands of parameters to identify complex patterns and achieve greater generalisation power when presenting new images to the models. However, to obtain a good response from the models, it is necessary to have, among other things, significant annotated images to successfully train these models and a proper data augmentation strategy [3].

Dataset annotation is very restrictive for training neural network architectures [4,5], considered one of the biggest problems in DP. This is due to the limited number of annotated images at the pixel level for segmentation tasks and/or at the patch level for classification tasks. In many cases, there are only global annotations, including diagnostics.

Active learning (AL) techniques focus on reducing this cost by building a small dataset with the most informative samples. This technique has been applied in many computer vision tasks (e.g., classification, semantic segmentation, pose estimation, and object detection) with successful results, both in 2D and 3D [6,7] environments.

Its application in classification (CL) and object detection (OD) tasks has received more attention in recent years. Several applications (e.g., digital pathology, autonomous driving) can benefit from this type of technology, where the cost of specialists is high and the size of the unlabelled dataset is also large. Many researchers focus on finding new sampling methods, selecting a constant number of images (constant budget) for each iteration. In this case, the workload of the annotators (specialists) remains similar for each iteration. Therefore, this work explores the implications of not using a constant number of images to annotate per iteration (i.e., a constant budget); and investigates the impact of adopting a variable annotation strategy across iterations, starting with a larger number of images in the initial rounds and gradually reducing this number in subsequent iterations.

This work proposes active learning with a decreasing-budget-based strategy and introduces three experiments that provide insights into its impact on the annotation process. The strategy enhances annotation efficiency by prioritising efforts in the initial iterations, optimising resource allocation, and ultimately boosting the performance of the trained model.

The rest of the work includes the following sections.

• Section 2 provides a short review of relevant previous work in the areas of classification, object detection methods, active learning sampling strategies, and their application in medical imaging;
• Section 3 presents the datasets used in the experiments, the decreasing-budget-based strategy, and the experiments focused on classification and object detection tasks;
• Section 4 and Section 5 present the results of the classification and object detection experiments, respectively, along with a discussion of these findings;
• Finally, Section 6 presents the conclusions of this work.

2. Related Work

In deep learning, classification consists of assigning a label or category to an input image based on its visual features. It involves training a deep neural network to learn the mapping between the input images and their corresponding labels. There are some popular DL architectures for classification applications. Two of them are MobileNetV3 [8] and InceptionV3 [9]. The first [8] focuses on efficiency and performance on mobile and resource-constrained devices, utilising depth-wise separable convolutions and attention mechanisms to balance accuracy with computational cost. The second [9] employs a multipath approach through inception modules, allowing it to capture features at various scales, and to incorporate techniques such as factorisation and auxiliary classifiers to enhance learning and convergence.

Otherwise, object detection is the process of identifying and localising objects within an input image. In the context of deep learning, object detection involves training a deep neural network to predict the bounding boxes and class labels of objects within an image. Today, several technologies such as You Only Look Once (YOLO) [10], Detection Transformers (DETR) [11], and others are racing to reach and maintain the state-of-the-art in the object detection task. Other well-established methods, such as Faster R-CNN [12], are used for comparison purposes between object detection architectures. Detection Transformers is an object detection architecture that relies on direct predictions, avoiding components that encode the prior knowledge of the task [11]. Its performance was measured using the MS-COCO dataset [13], outperforming other architectures. However, according to the authors, the model’s performance on small objects still requires further study [11]. The YOLO algorithm family, like YOLOv8 and its successors, is considered a cutting-edge, state-of-the-art model, supporting several artificial intelligence tasks such as classification, object detection, segmentation, pose estimation, and tracking [10]. The experiments carried out in this work used MobileNetV3 [8], InceptionV3 [9] and YOLOv8 [10] architectures as a basis, although any other model for classification and/or object detection can also be used.

2.1. Active Learning Sampling Strategies

Active learning strategies prioritise the selection of the most informative images for annotation, allowing neural network models to achieve performance similar to that of those trained on the entire dataset. By focusing on high-value samples, active learning improves efficiency, reducing annotation workload without compromising model accuracy.

The pool-based active learning scenario is well established in the active learning literature. The iterative process starts with a small training dataset ($Tr$), which is used to train a first model $M_0$ (iteration $i=0$). Later, the model can be used to score all images that belong to the pool (P) according to an acquisition function $f(M_i,P)$. A total of B top-scoring images are moved from P to $T_r$, allowing the execution of another training and selection iteration. The validation dataset ($V_a$) and the test dataset ($T_e$) are kept constant to decide if the model is ready to deploy or not.

Another feature of the pool-based scenario is that the pool P is fixed at the beginning of the iterations and only the images sampled are removed until the pool is empty. The stream-based scenario is another category that groups specific cases where the pool receives new images from time to time. This scenario includes the possibility of rejected instances being removed from the pool.

In [14], the authors present a review of active learning for object detection, which summarises the most commonly used strategies and how active learning can be mixed with other learning techniques. Moreover, the review also discusses the strengths, weaknesses, opportunities, and threats (SWOT) associated with AL methods for object detection.

2.2. Active Learning for Medical Applications

Medical applications are applying active learning to reduce annotation costs and monitor the assigned budget to obtain a useful deep learning model. This technique was successfully applied in classification [15,16] , object detection [17,18,19] and segmentation [16,20].

To address the problem of active learning for the Tumour-Infiltrating Lymphocytes classification task in digital pathology, the authors propose a hybrid acquisition function that highlights the diversity of tissue characteristics through the use of grouping and uncertainty through the model prediction output [15].

Segmentation was also addressed with the active learning approach, where a U-net architecture was trained in an iterative process. This neural network was applied to whole slide images of oral cavity cancer tumours stained with H&E. For each iteration, a team of pathologists scores each image on a scale of 0–5 for each tissue class. Only poorly classified images were used to retrain the U-net model [20].

In [16], active learning was employed to train classification and segmentation models to overcome dataset limitations and improve model performance. Experts manually annotated images exhibiting high uncertainty, whereas they assigned pseudo-labels to the low-uncertainty images.

In [21], a personalised AI platform was proposed for digital pathology. The platform enables pathologists to create manual annotations and employ an active learning strategy to rank and present images for validation. It also includes an application programming interface to provide an artificial intelligence service that assists pathologists in diagnosis. This platform was validated by a team of pathologists for the mitosis detection task.

3. Material and Methods

The main focus of this work is to evaluate the performance of a decreasing-budget-based strategy ($S_{DB}$), avoiding other topics such as the acquisition function, the user interface used to annotate, and the saved time in the annotation process. In this sense, the Section 3.1 introduces the proposed active learning with a decreasing-budget-based strategy and the Section 3.2 presents the datasets used in the experiments. Later, the Section 3.3 and Section 3.4 describe the experiments.

3.1. Proposed Active Learning with a Decreasing-Budget-Based Strategy

This work explores the impact of using a decreasing budget on each active learning iteration, starting with a high budget size, and continuously reducing this budget by iteration. Next, Algorithm 1 presents the steps to implement this strategy.

Algorithm 1 Active Learning with the Decreasing-Budget Strategy. The function $Validate\left(B_i\right)$ represents the manual validation process

1: Input: Unlabelled dataset U, initial budget $B_0$, initial iteration $i\leftarrow 0$   2: Output: Trained model $M_i$   3: Select and annotate initial set of images $D_0$ from U using budget $B_0$   4: while stopping criterion not met do   5:     Train model $M_i$ using dataset $D_i$   6:     Rank remaining unlabelled images in U using uncertainty-based sampling   7:     Select a subset $B_i$ from U, such that $B_{i+1}<B_i$   8:     Use model $M_i$ to generate pseudo-labels for $B_i$   9:     Specialists validate the pseudo-labelled images, such that $B_i^v=Validate\left(B_i\right)$  10:     $D_{i+1}\leftarrow D_i\cup B_i^v$  11:     $i\leftarrow i+1$  12: end while  13: return final model $M_i$

As mentioned above, the sampling strategy is based on the uncertainty of the model (U), specifically entropy, but it is not unique to this image score strategy. For classification problems, the uncertainty (see Equation (1)) depends on each class prediction confidence ($p_j$) with j denoting the class index and C denoting the number of classes.

$$U\left(I\right)=-\sum _{j=1}^C{p}_{j}log\left(p_j\right)$$

(1)

For object detection problems, the entropy depends on the class count (C) and the number of objects detected by class ($N_j$). First, it computes the uncertainty of all objects that belong to the same class j (see Equation (2)). Next, the uncertainty of the class is aggregated to compute the image uncertainty. This work uses the average as an aggregation method (see Equation (3)).

$$U_j=\sum _{i=1}^{N_j}-p_i^{j}log{p_i^j}_{j\in [1,\dots ,C]}$$

(2)

$$U\left(I\right)=\sum _{j=1}^C{U}_j/C$$

(3)

3.2. Dataset

To address the problem of mitosis detection, we rely on two public datasets (i.e., Mitos & Atypia [22] and TUPAC [23]). Also, additional images (iPath dataset) were collected and annotated by the Computer Graphics Centre (CCG), the Faculty of Medicine at the University of Coimbra (FMUC), and the BMD Software consortium in the context of the iPath project [3]. They were joined to form a single whole (see

Table 1. Mitosis detection dataset used in this work.

Dataset	Images Count
TUPAC	1288
Mitos & Atypia	1206
iPath project	1370
Total	3864

).

For the classification experiment, the three datasets in Table 1 were reviewed and adapted to create a new balanced dataset with 6214 images. (that is, 3107 mitosis images and 3107 non-mitosis images). Figure 1 shows 12 sample images, i.e., six for each class of the dataset. Initially, we randomly split the dataset into three parts (60% for training, 20% for validation, and 20% for testing) with the condition that they are balanced.

For the object detection experiments, the three datasets were reviewed and adapted too. In this sense, the images were cropped to $640\times 640$ pixels containing the mitosis. Later, the mitosis bounding box was adjusted to its minimum size.

3.3. Classification Experiments

To analyse the behaviour of the strategy $S_{DB}$, it is compared with three other strategies ($S_{B25}$, $S_{B50}$ and $S_{B100}$) where the budget remains constant on each active learning iteration. For each, the size of the dataset used to train the models increases, for each iteration, by B images where $B=25$, $B=50$ and $B=100$ according to the strategies $S_{B25}$, $S_{B50}$ and $S_{B100}$, respectively. The $S_{DB}$ strategy reduces the budget from $B=100$ to $B=25$ with a step of $-25$. The initial iteration $i=0$ for the four strategies starts with 20 images (see Figure 2). The validation and test datasets are the same for both architectures and remain constant for each iteration.

The experiment is repeated two times, observing the performance of two well-known deep learning architectures for classification tasks (i.e., MobileNetV3 [8] and InceptionV3 [9]) when they are trained iteratively. Both models have an input size of $256\times 256$ and $299\times 299$ pixels, respectively, and were trained using the Adam [24] and RMSprop [25] optimisers, respectively, with a constant learning rate of ${10}^{-5}$ during 100 epochs. For both, the same ImageNet [26] initialised weights are used at the start of every active learning iteration (transfer learning). The accuracy and F1-score metrics were used to evaluate the performance of trained models.

3.4. Object Detection Experiments

For object detection tasks, two experiments are included. In the first experiment, the YOLOv8Nano architecture [27] is trained iteratively, exploring its behaviour according to two variables: (1) warm-start learning or transfer learning, and (2) the use of a decreasing-budget or a constant budget. The second experiment explores: (1) the decreasing budget strategy versus the increasing-budget strategy, and (2) the performance of two YOLOv8 architectures (YOLOv8Nano or YOLOv8Medium).

For both experiments, each training goes up to epoch 100 using a batch size of 4 and the predefined YOLOv8 hyperparameters [10]. In addition, the images of Table 1 were randomly divided into five parts (five-fold cross-validation), repeating the active learning iterations five times ($R_1$, $R_2$, $R_3$, $R_4$ and $R_5$), retaining three parts for the pool dataset, one part for validation, and the other for testing according to

Table 2. Mitosis detection dataset used in this work.

Fold	Pool	Validation	Testing
$R_1$	$F_1$, $F_2$, $F_3$	$F_4$	$F_5$
$R_2$	$F_5$, $F_1$, $F_2$	$F_3$	$F_4$
$R_3$	$F_4$, $F_5$, $F_1$	$F_2$	$F_3$
$R_4$	$F_3$, $F_4$, $F_5$	$F_1$	$F_2$
$R_5$	$F_2$, $F_3$, $F_4$	$F_5$	$F_1$

. The model’s performance was assessed based on the F1-score. For each experiment, a different seed was used to divide the dataset into five folds, and 20 images were randomly selected from the pool dataset to train the model in the first iteration ($I_0$).

3.4.1. First Object Detection Experiment

In this experiment, warm-start learning means that the model $M_i$ is used to initialise the parameter values of the $M_{i+1}$, while transfer learning uses the same parameter values on each initialisation. The YOLOv8Nano model was used to initialise the model $M_0$ in the first iteration ($I_0$). The decreasing-budget strategy reduces the budget from $B=100$ to $B=60$ with a step of $-5$, and the constant strategy uses $B=100$ in each iteration (see Figure 3).

3.4.2. Second Object Detection Experiment

In this experiment, for each active learning iteration, transfer learning is used to initialise the model parameters, i.e., the YOLOv8Nano and YOLOv8Medium models were used to initialise the models $M_i$. The $S_{DB}$ strategy was evaluated in two different scenarios. The first scenario compares the decreasing-budget-based strategy with increasing the budget per iteration and keeping the budget constant by iteration. The second scenario evaluates the performance of two object detection architectures (YOLOv8Nano and YOLOv8Medium). Figure 4 presents the evolution of the training dataset size by iterations.

4. Classification Experiment: Results and Discussion

According to the Figure 2, the learning process with $S_{B25}$ used fewer images for training followed by the experiment with $S_{B50}$ and $S_{B100}$. The fourth experiment ($S_{DB}$) starts with $B=100$ and, in successive iterations, the budget is reduced by 25, i.e., 75, 50, and 25, and then remains constant until the last iteration.

Figure 5 shows the accuracy and evolution of the F1-score of the MobileNetV3 trained models in the test dataset. Similar behaviour is observed in trained models belonging to the InceptionV3 architecture (see Figure 6). The active learning experiment started at iteration $I_0$ for both models, using the same training dataset of 20 images. The models trained in the first iteration ($i=0$) have shown low accuracy as expected. Both models, in the last iteration ($i=6$), reached a higher accuracy and F1-score for $S_{B100}$. Appendix A 

Table A1. Accuracy by iteration considering the decreasing-budget strategy ($S_{DB}$ and three different strategies based on a constant budget size $S_{B25}$, $S_{B50}$, and $S_{B100}$.

Iteration	MobileNetV3				InceptionV3
	${\mathit{S}}_{\mathit{DB}}$	${\mathit{S}}_{\mathit{B}\mathbf{25}}$	${\mathit{S}}_{\mathit{B}\mathbf{50}}$	${\mathit{S}}_{\mathit{B}\mathbf{100}}$	${\mathit{S}}_{\mathit{DB}}$	${\mathit{S}}_{\mathit{B}\mathbf{25}}$	${\mathit{S}}_{\mathit{B}\mathbf{50}}$	${\mathit{S}}_{\mathit{B}\mathbf{100}}$
0	0.5927	0.5927	0.5927	0.5927	0.6753	0.6753	0.6753	0.6753
1	0.8189	0.7666	0.7911	0.8189	0.8204	0.7001	0.7498	0.8204
2	0.939	0.803	0.8684	0.9683	0.914	0.7328	0.8919	0.9361
3	0.9461	0.8506	0.8958	0.9841	0.9347	0.7786	0.9145	0.95
4	0.9462	0.8946	0.9304	0.9822	0.9371	0.842	0.928	0.9563
5	0.9774	0.9212	0.9462	0.9875	0.9419	0.8545	0.9304	0.9616
6	0.9808	0.9315	0.9505	0.9875	0.9467	0.9015	0.9361	0.9659

and

Table A2. F1-score by iteration considering the decreasing-budget strategy ($S_{DB}$ and three different strategies based on a constant budget size $S_{B25}$, $S_{B50}$, and $S_{B100}$.

Iteration	MobileNetV3				InceptionV3
	${\mathit{S}}_{\mathit{DB}}$	${\mathit{S}}_{\mathit{B}\mathbf{25}}$	${\mathit{S}}_{\mathit{B}\mathbf{50}}$	${\mathit{S}}_{\mathit{B}\mathbf{100}}$	${\mathit{S}}_{\mathit{DB}}$	${\mathit{S}}_{\mathit{B}\mathbf{25}}$	${\mathit{S}}_{\mathit{B}\mathbf{50}}$	${\mathit{S}}_{\mathit{B}\mathbf{100}}$
0	0.6056	0.6056	0.6056	0.6056	0.5831	0.5831	0.5831	0.5831
1	0.8239	0.7202	0.7912	0.8239	0.7932	0.7832	0.798	0.7932
2	0.9124	0.745	0.8913	0.9366	0.9406	0.81	0.8616	0.9681
3	0.9335	0.7765	0.9152	0.9495	0.9454	0.8378	0.903	0.9842
4	0.9372	0.854	0.9266	0.9557	0.9476	0.8382	0.928	0.9823
5	0.9408	0.8752	0.9312	0.9615	0.9775	0.87	0.9452	0.9875
6	0.9456	0.9025	0.9352	0.966	0.9808	0.8833	0.9518	0.9875

contains the data used to generate the graphs.

Also, Figure 7 presents the accumulated training time for each iteration, finding that the training times were similar in the initial iterations, increasing the difference at the end of the iterations.

These results show the relationship between model performance (accuracy and F1-score) and the amount of data used to train them. Experiments show that if deep learning models are trained using active learning techniques, they can achieve similar performance over several iterations using less annotated data.

The $S_{B25}$ configuration in terms of accuracy and F1-score never reaches the performance of the other configurations and is therefore excluded from the following analysis. MobileNetV3 and InceptionV3, trained in loop, using $S_{B100}$ show the best results compared to the rest of the settings, followed by the $S_{DB}$ configuration according to the proposed strategy. The configuration $S_{DB}$ performs better than $S_{B50}$ and $S_{B25}$.

The analysis of the MobileNetV3 and InceptionV3 architectures, in iteration $i=6$, where both models were trained with the same amount of images, shows that the configuration $S_{DB}$ performs better than $S_{B50}$ with the same annotation effort. Until now, $i=6$, the model trained according to the proposed strategy always performs better than $S_{B50}$. At iteration $i=6$, the model trained by the strategy with $S_{B100}$ uses 620 images for training, while the $S_{DB}$ strategy only used 320 images, approximately half of the images with a very similar result, in terms of both accuracy and F1-score.

MobileNetV3 architecture: F1-score of $S_{B100}$ equal to $0.9875$ versus F1-score of $S_{DB}$ equal to $0.9808$.

InceptionV3 architecture: F1-score of $S_{B100}$ equal to $0.966$ versus F1-score of $S_{DB}$ equal to $0.9456$.

From the previous analysis, it is inferred that using a high budget by iteration should not mean a good annotation strategy thinking in the reduction in the annotation effort. Meanwhile, it is observed in this experiment that starting with a high budget and reducing this value by iteration contributes to reducing the annotation effort of the oracle, which is the main goal of any active learning strategy. However, reducing the annotator’s effort is directly related to the increase in computational resources needed to reach a useful result. Furthermore, the results obtained with a strategy that reduces the budget per iteration ($S_{DB}$) showed promising performance results, as this approach consumes fewer computational resources than $S_{B100}$ and obtains similar results according to the accuracy and F1-score metrics.

5. Object Detection Experiments: Results and Discussion

5.1. First Object Detection Experiment

The figures in this section show the results of the five-fold cross-validation for object detection, represented as $R_1$, $R_2$, $R_3$, $R_4$, and $R_5$. The lines in the plots indicate the F1-score average, while the shaded area represents the range between the minimum and maximum F1-score values. Appendix B,

Table A3. F1-score by iteration for the first object detection experiment. WSL: Warm-start learning, TL: Transfer learning, CB: Constant budget, DB: Decreasing budget.

Iteration	WSL-CB			TL-CB			WSL-DB			TL-DB
	Mean	Min	Max	Mean	Min	Max	Mean	Min	Max	Mean	Min	Max
0	0.650	0.642	0.658	0.650	0.642	0.658	0.650	0.638	0.662	0.650	0.643	0.657
1	0.700	0.692	0.708	0.696	0.688	0.704	0.702	0.696	0.708	0.750	0.740	0.760
2	0.718	0.709	0.727	0.720	0.711	0.729	0.734	0.727	0.741	0.762	0.754	0.770
3	0.720	0.708	0.732	0.736	0.724	0.748	0.740	0.734	0.746	0.774	0.766	0.782
4	0.736	0.731	0.741	0.748	0.743	0.753	0.744	0.736	0.752	0.780	0.773	0.787
5	0.740	0.729	0.751	0.748	0.737	0.759	0.750	0.741	0.759	0.776	0.767	0.785
6	0.740	0.729	0.751	0.776	0.765	0.787	0.748	0.743	0.753	0.782	0.772	0.792
7	0.742	0.737	0.747	0.774	0.769	0.779	0.762	0.753	0.771	0.790	0.781	0.799
8	0.748	0.738	0.758	0.774	0.764	0.784	0.754	0.746	0.762	0.788	0.780	0.796
9	0.746	0.736	0.756	0.776	0.766	0.786	0.760	0.750	0.770	0.794	0.785	0.803

reports the complete data used to generate the graphs, while Table A3 includes a graphical representation of these results. For a more comprehensive analysis of the results, the experiments are examined according to the studied variables. To this end, this section presents simplified figures derived from the detailed results provided in Appendix B.

Two configurations are not directly comparable (see Figure 8) due to differences in their settings.

The first (Figure 8 left), the $S_{DB}$ strategy trained with warm-start learning, in the first iterations, performs better than the strategy with constant budget and the transfer learning option. This result is inverted at the end of the iterations, where the $S_{DB}$ strategy shows poor results when the models were trained using warm-start learning between iterations, i.e., transfer learning using a constant budget performs better. The second (Figure 8 right), a decreasing budget trained using transfer learning always performs better than a constant budget trained using warm-start learning.

After exploring the previous results, we compare each strategy, keeping constant the type of strategy associated with the number of images added to the training dataset (see Figure 9). Furthermore, the experiments demonstrate that, for this dataset, YOLOv8 performs better with transfer learning, regardless of the number of images added in the training dataset per iteration, and irrespective of whether a constant or decreasing budget is used.

Figure 10 compares the strategy $S_{DB}$ in two different scenarios: (1) warm-start and (2) transfer learning. In both cases, the strategy $S_{DB}$ achieves better results using a smaller number of annotated images, as shown in Figure 3.

From the data collected, it was observed that the transfer learning (initialising the model $M_i$ between iterations with the same values, i.e., $M_0$) represents a better alternative for initialising the model than warm-start learning (that is, initialising the $M_{i+1}$ model from the $M_i$ model). Mostly because the training datasets are small and warm-start learning may produces overfitting. Also, the $M_0$ used is trained on large datasets, making $M_0$ a good starting point for model training.

The first object detection experiments showed results similar to those obtained in the previous section. In this case, in iteration $i=9$ using the strategy $S_{DB}$ the dataset used to train the model is smaller by 200 images than the dataset formed using the strategy constant budget. However, it is observed that iterations using the $S_{DB}$ strategy outperform iterations using a constant budget.

5.2. Second Object Detection Experiment

Figure 11 presents the results of the second object detection experiment. The corresponding data used to generate the graphs are provided in Appendix C,

Table A5. F1-score by iteration for the second object detection experiment—YoloV8Medim.

Iteration	YoloV8Medium
	Increasing Budget			Constant Budget			Decreasing Budget
	Mean	Min	Max	Mean	Min	Max	Mean	Min	Max
0	0.458	0.414	0.505	0.458	0.414	0.505	0.458	0.414	0.505
1	0.601	0.567	0.622	0.626	0.615	0.642	0.644	0.636	0.658
2	0.654	0.634	0.676	0.660	0.658	0.663	0.691	0.686	0.697
3	0.686	0.680	0.693	0.681	0.669	0.689	0.699	0.691	0.707

.

The differences in F1-score observed between the first and second object detection experiments are due to two main factors. First, a different seed is used to split the dataset, and second, the number of images sampled by this iteration is smaller than in the first experiment.

Regardless of the network architecture with which the experiments were conducted, the proposed strategies behaved similarly. The decreasing-budget-based strategy performs better than the other two strategies as active learning iterations increase. The second best strategy was the constant budget strategy, and the worst performance was the increment budget strategy.

Furthermore, the first and second object detection experiments can be compared when the training dataset contains the same images. The first, in iteration 2 ($I_2$) and the second at iteration 4 ($I_4$), each contains 300 images in the training dataset. The first devotes the greatest effort in two iterations and the second distributes the effort over more iterations. This corroborates that most of the effort needs to be devoted to the creation of the training dataset in the initial iterations to achieve the best possible model performance.

6. Conclusions and Future Work

This work proposes an active learning with a decreasing-budget-based strategy to reduce specialists’ workload in the image annotation process. It presents three experiments that tackle different tasks: classification and object detection to understand the impact of the decreasing-budget-based strategy on the annotation process.

The proposed strategy considers that on each iteration of the process, the specialist reduces the number of images to annotate, i.e., the current iteration annotates fewer images than in the previous iteration. This strategy encourages data annotators to devote maximum effort to initial iterations, ensuring less effort by annotators in successive iterations. Therefore, budget management is more efficient, so the trained model performs at its best.

A weakness of this work is that it considers only one use case, specifically the identification of mitoses. Future work considers this approach to other applications, e.g., medical and industrial, focusing on the first iteration and looking for an unsupervised method to make a more diverse initial selection of the images that will form the training dataset.