An Adaptive Threshold for the Canny Edge with Weak Label

Abstract

Edge detection is critical in various computer vision applications such as object recognition, segmentation, and scene understanding. The Canny edge detector remains widely used in traditional methods due to its balance between accuracy and computational efficiency. However, optimal performance depends heavily on carefully selecting three threshold parameters. Prior studies have proposed reinforcement learning-based approaches to automatically determine these parameters. Still, they rely on a supervised edge evaluation network that requires manually annotated high-quality edge labels—a costly and time-consuming prerequisite. This paper proposes a novel approach that eliminates the need for manually labeled data by introducing a weakly supervised reward scheme. We automatically generate weak edge labels from gradient information and use them as pseudo ground truth to compute reward values during the reinforcement learning process. The proposed method leverages an Actor-Critic algorithm to learn adaptive thresholds for the Canny edge detector without explicit supervision. Experimental results demonstrate that our method achieves comparable or superior edge detection performance compared to previous supervised methods and generalizes effectively to unseen datasets.

1. Introduction

Image edge can be used for object detection, image segmentation, motion analysis, and 3D reconstruction [1,2,3,4]. Initially, filter-based and statistical analysis-based edge detection algorithms emerged [3,5,6]. After the emergence of deep learning, edge detection is also performed using CNN [7,8,9]. The Canny algorithm [5] remains widely used for detecting edges in images. It requires setting the values of three parameters related to the smoothing window size and two thresholds in the hysteresis process.

Our previous works [10,11] employed deep reinforcement learning to determine the three threshold values in the Canny algorithm. However, both methods relied on a supervised edge evaluation network, which required manually annotated edge labels. In this paper, we propose an algorithm that eliminates the need for an edge evaluation network to determine the values of three thresholds in the Canny algorithm, utilizing deep reinforcement learning. In our previous work [10], we used the Deep Q-Network (DQN) [12,13] to determine the values of two thresholds in the Canny algorithm. We used a fixed value for a parameter related to the smoothing window size. In our previous work [11], we proposed an algorithm that can automatically determine the three parameters in the Canny algorithm. We employed an actor-critic algorithm [14,15,16] with continuous action spaces. Both approaches [10,11] require training of the edge evaluation network. We should prepare the correct edge image for each input image for training, which requires a significant amount of time. It is well known that supervised learning performs well in environments similar to the one in which it was trained. However, it results in poor performance in environments that differ from the training environment. One way to adapt to a new environment is to retrain using an image of the corresponding domain. Additionally, this requires the preparation of ground-truth labels. This paper proposes an algorithm for automatically selecting optimal values for three thresholds in the Canny algorithm, eliminating the need for edge labels, as compared to our previous algorithms [10,11].

The proposed algorithm makes three primary contributions.

(1) We introduce a weakly supervised reinforcement learning framework that eliminates the need for manually annotated edge labels by automatically generating pseudo edge maps from gradient information.

(2) We design a quantitative reward function based on Baddeley’s distance metric, enabling the agent to learn adaptive threshold parameters directly from weak labels without relying on a separate edge evaluation network.

(3) We demonstrate through extensive experiments that the proposed method achieves competitive or superior performance to supervised actor–critic models and exhibits strong generalization capability to unseen domains.

Together, these contributions provide a scalable and label-efficient solution to adaptive edge detection, making it highly practical for real-world scenarios where manual annotation is infeasible.

2. Related Works

Edge detection has long been a subject of study, and algorithms can be broadly classified into filter-based, learning-based, and deep learning-based approaches.

2.1. Filter-Based Algorithms

The proper reflection of intensity, color, and texture changes is studied in filter-based algorithms [3,5]. In the early stage, hand-crafted features are manually designed for use in learning-based algorithms. Manually designed features with information theory are used in statistical edges [6], Pb [17], and gPb [18]. Also, various learning-based methods, including BEL [19], multi-scale [20], sketch tokens [21], and structured edges [22], use manually designed features. Dollar et al. [22] proposed an algorithm for detecting structured edges through joint learning, ground-truth clustering, and mapping image patches to clustered tokens. It demonstrated state-of-the-art performance on the BSDS500 dataset before the emergence of deep learning-based algorithms. These learning-based algorithms have the advantage of producing edge images in an automated way with structured information on images. However, their generality power is verified on several images compared to deep learning-based algorithms.

2.2. Deep Learning-Based Algorithms

Deep learning-based algorithms use features with convolutional neural networks (CNN) [7]. Bertasius et al. [8] utilize a CNN to extract features from candidate contour points. Holistically Nested Edge Detection (HED) is proposed by Xie et al. [9], which integrates the outputs from different intermediate layers through skip connections. Xu et al. [23] employ a gated conditional random field to identify multi-scale features within a hierarchical model. He et al. [24] propose a Bi-Directional Cascade Network (BDCN) structure to detect edges at different scales. The network is trained using corresponding edge labels at each scale. Recent algorithms for edge detection focus on accurately detecting object boundaries, which can be utilized for object detection, segmentation, and tracking.

This paper addresses the automatic selection of three threshold values in the Canny edge detection algorithm. Various algorithms have also been proposed. Lu et al. [25] proposed an algorithm that uses the histogram of the gradient image for the automatic selection of thresholds for the Canny algorithm. In Fang et al. [26], the Otsu method [27] is employed to determine a high threshold for the Canny algorithm, but they are unable to select a low threshold. Huo et al. [28] proposed an algorithm to determine high and low thresholds in the Canny algorithm, where a low threshold is determined with a probability model. In Lu et al. [29], the assumptions of minimal meaningful gradient and maximal meaningless gradient magnitude are used in the adaptive selection of two thresholds in the Canny algorithm. Yitzhaky and Peli [30] proposed a two-step approach to select the optimal edge parameters. First, they construct the Estimated Ground Truth (EGT) with different detection results. Finally, they determine the optimal parameter set using a Chi-square test.

Medina-Carnicer et al. [31] proposed an algorithm for the unsupervised determination of hysteresis thresholds by integrating the advantages and disadvantages of two thresholding algorithms. They find the best hysteresis thresholds among a set of candidates. In Mediana-Carnicer et al. [32], an algorithm is proposed for automatically selecting hysteresis thresholds in the Canny algorithm, which can be utilized as an unsupervised edge detector. Unsupervised methods have the advantage that they do not require a learning process. However, they only provide evaluation results for a limited number of images. To assess their performance objectively, they must be evaluated in diverse scenarios with numerous images, such as those used in deep learning.

Temporal decision-making problems can be solved with reinforcement learning [33]. An agent aims to learn a policy that maximizes accumulated reward from an environment in reinforcement learning. Following the emergence of deep learning, incorporating it into reinforcement learning has demonstrated human-level control [12,13,34]. Deep Q-Networks (DQN) [12,13] showed that human-level control is possible on Atari games. Deep reinforcement learning has achieved impressive successes on various tasks, including playing the board game Go [35,36,37], object localization [38], region proposal [39], and visual tracking [40].

Our previous works [10,11] proposed algorithms for automatically selecting three good threshold values in the Canny algorithm with deep reinforcement learning. They [10,11] use an edge evaluation network that requires a manually selected edge label to determine reward. This paper proposes an algorithm that eliminates the need for manually selected edge labels by utilizing automatically generated edge images as pseudo labels to determine the reward.

3. Proposed Method

3.1. Actor and Critic Network

We employ the same actor and critic networks as those described in our previous paper [11], which are illustrated in Figure 1 and Figure 2. These are included to help the reader understand and complete the paper.

In the A2C (Advantage Actor-Critic) [14,15,16], an actor network evaluates actions performed by an agent, and a critic network evaluates the accumulated return from the current state. There are two kinds of actor and critic network configurations in the A2C algorithm. The first method uses the same backbone and has two branch outputs for the actor and critic networks. The second method utilizes two distinct networks: the actor network and the critic network. Through experiments, we found that the first method, which uses the same backbone, gives inconsistent results, including divergence during training. Therefore, our previous work utilized separate actor and critic networks [11].

The A2C algorithm has the advantage of dealing with continuous actions compared to the DQN. The policy will have a continuous output if it follows a normal distribution.

$$\mathrm{\pi }\left(a_t|s_t;\theta \right)={\displaystyle \frac{1}{\sigma (s_t;\theta )\sqrt{2\pi }}}exp\left(-{\displaystyle \frac{{(a_t-\mu (s_t;\theta )}^2}{2{\sigma (s_t;\theta )}^2}}\right)$$

(1)

$\mu (s_t;\theta )$ and $\sigma (s_t;\theta )$ is the mean and standard deviation of the Gaussian distribution. $s_t$ and $a_t$ is state and action. $\theta$ represents the parameters of the policy network $\mathrm{\pi }$.

Figure 1 shows the structure of the proposed actor network. We use an original image as the input to the network. The actor network determines the mean and variance of the Gaussian distribution. The network output corresponds to the mean and variance of three actions. We select threshold values randomly from the Gaussian distribution.

We extract features of the original image using a pre-trained CNN of the ResNet50 structure. Then, they are used to yield the mean and standard deviation of the Gaussian distribution. Parameters related to the fully connected (FC) layers are trained after random initialization. We use the tanh activation function at the last layer to ensure the mean has a value between −1 and 1. We use the sigmoid function as the activation function in the last layer to provide a standard deviation that has a value between 0 and 1. The output of the actor network is continuous real values, and it is necessary to convert them into the range used in the Canny algorithm. For the high and low thresholds, we convert the actor network’s output into an integer ranging from 0 to 500. For the filter size of the smoothing window, we convert the output of the actor network into an integer value between 3 and 9.

Figure 2 shows the structure of the critic network used in [11]. The input of the critic network is composed of an original image and action values, while a typical critic network uses only the current state as input. We randomly select action values from the normal distribution provided by the actor network and then use them as input for the actor network. The output of the critic network corresponds to the state value function $\mathrm{V}\left(s_t\right)$. Since the critic network is a model that approximates the $\mathrm{V}\left(s_t\right)$ value according to each action, we use action values as the input of the critic network. Through this, we can guarantee that training will increase the probability of selecting proper actions from the normal distribution of the actor network. We do not use an activation function at the output layer in the critic network. It is based on the consideration that the critical network evaluates the accumulated return from the current state. In addition, we concatenate three action values with features from the pre-trained model in the middle of the fully connected layer to prevent them from being reflected in a small proportion.

3.2. Computing Reward Using Weak Labels

The authors’ previous methods [10,11] used a supervised learning-based method to calculate the reward value. In [10], edge images are used as inputs to the edge evaluation network. In [11], the original image and edge images are used as inputs to the edge evaluation network, and the reward value is determined based on the network’s output. To train the edge evaluation network, it is necessary to manually classify a large number of edge images. This training data preparation has the disadvantage of requiring a lot of time.

In this paper, we propose a method to overcome this disadvantage. Unlike previous studies, the proposed method does not use an edge evaluation network. The automatically generated edge images are considered weak labels. The reward value is determined by comparing the edge images with the weak labels and the agent’s output values. Although the weak labels have lower quality than manually selected edge images, they can be automatically generated, which gives them the advantage of being flexible in responding to new environments.

Figure 3 shows the automatic generation process of the proposed weak labels. Figure 3a shows the entire process. The gradient magnitude and direction are calculated for the input image using the change in brightness value for each pixel. In this process, the rate of change is calculated for each color channel, and the average value is used for each pixel. It is assumed that the gradient magnitude obtained in this way will show a Gaussian distribution in a sufficiently large number of data points. Based on this, the pixel with a value corresponding to 30% of the distribution of the gradient magnitude values is determined as the boundary point. This is implemented so that the pixel is defined as the boundary point if the difference between the average of the gradient magnitude and the pixel value exceeds 0.5 times the variance of the gradient magnitude. Comparing the boundary point and its surrounding values was applied to generate a more detailed boundary image.

As shown in Figure 3b, the gradient direction with eight directions was calculated based on the gradient theta. Based on this, the direction filter to be applied to each boundary point was determined, and the cases where the filtered value was zero or more were collected to generate a one-pixel edge image. In this study, the Sobel filter was used to obtain the gradient magnitude, and the gradient path was determined by applying arctan2, as shown in Figure 3b, and by dividing the eight directions into equal intervals.

The evaluation of edge images during the training process is performed as follows. First, check whether the high and low thresholds of the Canny edge among the three thresholds determined through the actor network are inverted. Threshold reversal refers to the case where the low threshold value exceeds the high threshold, resulting in incorrect hysteresis behavior. Such configurations are assigned the lowest reward during training.

If the two thresholds are not inverted, the reward value is determined by comparing the edge image generated with the three thresholds determined through the actor network with the automatically generated weak label edge image. Baddeley’s evaluation method [41] compares the two edges.

Baddeley’s measure [41] is used to quantify the performance of each method. It is defined as follows.

$$D_w^p\left(A,B\right)={\left[{\displaystyle \frac{1}{N}}{\sum }_{x\in X}{\left|w\left(d\left(x,A\right)\right)-w\left(d\left(x,B\right)\right)\right|}^p\right]}^{1/p}$$

(2)

$\mathrm{d}(\mathrm{x},\mathrm{A})$ denotes the shortest distance from $\mathrm{x}\in \mathrm{X}$ to $\mathrm{A}\subseteq \mathrm{X}$. $\mathrm{w}\left(\mathrm{t}\right)$ is a continuous function with concave and strictly increasing properties. $D_w^p\left(A,B\right)$ can have a value from 0 to 1. For similar images A and B, $D_w^p\left(A,B\right)$ gives a value close to 0. Baddeley used the transformation $\mathrm{w}\left(\mathrm{t}\right)=\mathrm{m}\mathrm{i}\mathrm{n}\left\{t,c\right\}$ for a fixed $\mathrm{c}>0$. We denote H and W as the width and height of an image. The values $\mathrm{c}=\sqrt{H^2+W^2}$ and $\mathrm{p}=2$ were used in the evaluation.

Table 1. Two types of methods to reflect Baddeley’s measure.

Method	Threshold	Section 1	Section 2	Section 3	Section 4
Absolute	Evaluation value	$D_w^p\le 0.001$	$0.001<D_w^p\le 0.015$	$0.015<D_w^p\le 0.025$	$D_w^p>0.025$
	Threshold reversal	no	no	no	yes
Relative	Evaluation value	${\displaystyle \frac{D_w^p-D_{mean}^{160}}{D_{std}^{160}}}\le -2$	$-2<{\displaystyle \frac{D_w^p-D_{mean}^{160}}{D_{std}^{160}}}\le -1$	$-1<{\displaystyle \frac{D_w^p-D_{mean}^{160}}{D_{std}^{160}}}\le 0$	${\displaystyle \frac{D_w^p-D_{mean}^{160}}{D_{std}^{160}}}>0$
	Threshold reversal	no	no	no	yes

shows two reward determination methods using the proposed Baddeley evaluation method. Determining rewards using the value obtained through the Baddeley evaluation involves two methods: absolute and relative evaluation.

In both methods, the evaluation is applied equally, based on the inversion of the Canny edge’s high and low threshold values. If the threshold value is not inverted, it is classified into three categories; if it is, it is classified into Section 4. The absolute evaluation method divides the evaluation of each image into four fixed ranges. It is divided into four sections based on the $D_w^p$ value, and the inversion of the two threshold values is also considered. The relative evaluation method selects sections based on the recent evaluation scores generated during training. The Gaussian distribution is calculated using the 160 most recent results to find the sections. $D_{mean}^{160}$ and $D_{std}^{160}$ represent the mean and standard deviation of 160. It is divided into four ranges, similar to the absolute evaluation method, with the top 3%, top 15%, and top 50% used as reference points.

Table 2. Reward value setting reflecting Baddeley’s measure and threshold reversal.

	Case 1	Case 2	Case 3	Case 4
Section 1	1.0	1.0	1.0	1.0
Section 2	0.5	0.5	0.5	0.5
Section 3	−0.5	−0.5	−0.5	−0.5
Section 4	−1.0	−5.0	−10.0	−50.0

shows the final reward value configuration. It consists of four cases, and the reward values from Section 1, Section 2 and Section 3 are the same. Section 4 consists of four cases, each corresponding to a distinct reward value. In the case of the reward value, since the four cases in Table 2 are used in each of the two methods in Table 1, a total of eight combinations are possible. The optimal reward value configuration is obtained through experiments.

4. Experimental Results

The training was conducted on a system equipped with an Intel Core i9-10900 processor and two NVIDIA GeForce RTX 3090 GPUs. During the training process, pseudo ground-truth data for each image were pre-generated and utilized. In this process, the input image size was 256(H) × 256(W) × 3(C), and the average processing time per image was approximately 10 ms.

Table 3. Hyperparameters and their values used in the training.

	Actor, Critic Network
Input image	256(H) × 256(W) × 3(C)
Pre-processing	Weakly label, image resize and norm ([0, 255] → [0., 1.]
optimizer	AdamW optim
Learning rate	0.001
Learning scheduler	No
Batch size (each GPU)	32
$\gamma$	0.99

presents the hyperparameters and their corresponding values used in the training.

4.1. Weak Label Creation Result

Figure 3 shows the generation process of weak labels used in the proposed training. It was generated by applying 0.5 as the threshold of z ($=\left|{\displaystyle \frac{G-G_{mean}}{G_{std}}}\right|$) applied to the normalized gradient array. For the image sized 256(H) × 256(W) × 3(C) used as input, it takes about 10 ms on average. The edge image generated by the proposed process, as shown in Figure 3, is presented in Figure 4.

4.2. Result of the Proposed Method

Because the proposed method utilizes weakly generated edge maps for reward computation, a direct comparison with supervised reward models [10,11] may introduce bias. Thus, we employ a model-agnostic evaluation metric to ensure fair and consistent performance comparison. Therefore, a universally comparable evaluation factor is needed. In general, the change in the brightness value of the image at the boundary of an object appears to be larger than in other areas. Therefore, if the model generates an edge image effectively, the average gradient magnitude of the selected edge points is significant. In this paper, the evaluation index was chosen to reflect this. It is determined by reflecting the degree of change in the brightness value of each pixel of the edge image generated by the model. The model performance is assessed using a gradient-weighted average edge strength (${EM}_{avg}$), which captures both the edge intensity and the proportion of detected edge pixels in the image.

$${EM}_{avg}={\displaystyle \frac{\sum _{i-0}^{i=M}\sum _{j=0}^{j=N}\left(M_{ij}{E}_{ij}\right)}{\sum _{i-0}^{i=M}\sum _{j=0}^{j=N}{M}_{ij}}}$$

(3)

$M_{ij}$ represents the gradient magnitude of point (i, j) on the image. $E_{ij}$ represents the edge degree of point (i, j) on the image and it represents whether the pixel is edge or not.

Figure 5 shows the training results for case 2 of fixed-value learning, which yielded the best results among the proposed models, and compares them with the existing A2C model [11]. Quantitatively, the weakly supervised method achieves ${EM}_{avg}$ = 0.979 (Case 2, Absolute Evaluation), which outperforms the best supervised baseline (${EM}_{avg}$ = 0.921). Additionally, the ratio of successful threshold configurations (i.e., no reversal) improved from 96.7% to 98.8%. Additionally, we can observe that more diverse output values are present in the results for each image. Detailed comparisons are summarized in

Table 4. Comparison results of the proposed algorithm with our previous algorithm [11].

Action Size	Train Type	Reward Method	Reward Style (Table 2)	Reward Average	${\mathit{E}\mathit{M}}_{\mathit{a}\mathit{v}\mathit{g}}$ (Equation (2))	Threshold Reversal
High threshold: [0, 500] Low threshold: [0, 500] Filters size: [3, 9]	Continuous A2C [11]	evaluation CNN model	Case 1	0.815	0.919	17,430/20,000 (87.1%)
			Case 2	0.832	0.916	18,981/20,000 (94.9%)
			Case 3	0.837	0.921	19,342/20,000 (96.7%)
			Case 4	0.794	0.917	19,485/20,000 (97.4%)
	Weakly A2C	weakly label absolute evaluation	Case 1	0.819	0.954	19,241/20,000 (96.2%)
			Case 2	0.850	0.979	19,767/20,000 (98.8%)
			Case 3	0.842	0.931	19,808/20,000 (99.0%)
			Case 4	0.782	0.935	19,143/20,000 (95.7%)
		weakly label relative evaluation	Case 1	0.823	0.933	19,066/20,000 (95.3%)
			Case 2	0.835	0.948	19,549/20,000 (97.7%)
			Case 3	0.826	0.924	19,482/20,000 (97.4%)
			Case 4	0.788	0.937	18,424/20,000 (92.1%)

.

Figure 6 shows the distribution of high and low threshold values determined by each method for 1000 randomly selected images. As shown in Figure 6, the model’s results, trained according to the method proposed in this study, exhibit diverse distributions similar to those of the model trained using the A2C method [11]. This result suggests that the weakly supervised model successfully learns image-dependent threshold values with a distribution comparable to the supervised A2C model, demonstrating the effectiveness of weak labels in guiding parameter selection.

4.3. Result on Unseen Images

The generalization ability was tested by applying it to a different dataset from the one used for training. To evaluate the generalization ability of the trained model and the potential of the model learning method to respond to additional data, it was applied to YTVOS [42] images obtained in a different environment from the BDD-100K [43] images used for training. Further training was performed using a new dataset of 1070 images, without modifying the environment configuration or the reward generation method, based on the results of learning each model with the existing dataset. As shown in

Table 5. Results in unseen images of the YTVOS dataset [42].

Action Size	Train Type	Additional Train	Reward Style (Table 1)	Reward Average	Magnitude Average (×5)	No Threshold Reversal Ratio (%)
High threshold: [0, 500] Low threshold: [0, 500] Filters size: [3, 9]	Continuous A2C [11]	No	Case 1	0.715	0.818	927/1070 (86.6%)
			Case 2	0.723	0.822	1013/1070 (94.7%)
			Case 3	0.746	0.834	1025/1070 (95.8%)
			Case 4	0.698	0.804	948/1070 (88.6%)
	Weakly label absolute evaluation		Case 1	0.729	0.836	1055/1070 (98.6%)
			Case 2	0.733	0.841	1067/1070 (99.7%)
			Case 3	0.730	0.832	1031/1070 (96.4%)
			Case 4	0.701	0.824	1004/1070 (93.8%)
		Yes (epoch = 50, half then pre-train)	Case 1	0.824	0.871	1061/1070 (99.2%)
			Case 2	0.831	0.878	1065/1070 (99.5%)
			Case 3	0.836	0.883	1068/1070 (99.8%)
			Case 4	0.830	0.879	1062/1070 (99.2%)

and Figure 7, on the YTVOS dataset, the proposed method achieved an ${EM}_{avg}$ of 0.878 after fine-tuning, compared to 0.822 for the supervised baseline. Furthermore, the no-threshold-reversal ratio reached 99.5%, a notable 4.8% improvement over the best prior model. This demonstrates superior generalization and adaptability to unseen domains without requiring manual annotations.

Overall, the proposed weakly supervised reinforcement learning method not only matches but, in several configurations, surpasses the supervised baseline in edge quality, parameter robustness, and cross-domain adaptability. The experimental results confirm that weak labels, when systematically generated and evaluated, can serve as an effective alternative to manual annotations.

5. Conclusions

The Canny edge algorithm requires selecting suitable values for three thresholds to produce a good edge image. This problem can be solved in various ways. This paper proposes an algorithm that utilizes automatically generated edge images as pseudo labels for computing rewards through deep reinforcement learning. Though automatically generated edge images are not optimal, we can obtain competitive results compared to the method that requires manually selected good edge images in training. In this study, we addressed the limitation of previous reinforcement learning-based Canny edge detection methods that require supervised training with manually annotated edge images. To overcome this, we introduced a weakly supervised learning approach that uses automatically generated edge maps as pseudo labels for reward computation. By doing so, we eliminated the need for a dedicated edge evaluation network, which significantly reduced the annotation cost and enhanced the model’s adaptability to new environments. The proposed method is based on an Actor-Critic reinforcement learning framework, where the agent learns to generate optimal threshold parameters by maximizing a reward derived from the similarity between the generated edge and a weak label. Using Baddeley’s metric, we designed both absolute and relative reward functions to ensure robustness, allowing the agent to learn from noisy but informative signals.

We demonstrated that the proposed algorithm outperforms prior supervised approaches in terms of accuracy and generalization ability through comprehensive experiments on diverse datasets, including those from unseen domains. Notably, the model maintains high edge detection quality without requiring any retraining of an evaluation network when deployed in new environments. Additional experiments showed that our approach supports continual learning by leveraging weak labels from new domains. In conclusion, our method offers a scalable and label-efficient solution to adaptive edge detection. It is especially valuable for real-world applications where access to clean, annotated data is limited or unavailable. Future work includes exploring more sophisticated weak label generation techniques and extending the approach to other vision tasks requiring adaptive parameter tuning.