Multilevel Features-Guided Network for Few-Shot Segmentation

Abstract

The purpose of few-shot semantic segmentation is to segment unseen classes with only a few labeled samples. However, most methods ignore the guidance of low-level features for segmentation, leading to unsatisfactory results. Therefore, we propose a multilevel features-guided network using convolutional neural network techniques, which fully utilizes features from each level. It includes two novel designs: (1) a similarity-guided feature reinforcement module (SRM), which uses features from different levels, it enables sufficient guidance from the support set to the query set, thus avoiding the situation that some feature information is ignored in deep network computation, (2) a method that bridges query features at each level to the decoder to guide the segmentation, making full use of local and edge information to improve model performance. We experiment on PASCAL-5${}^i$ and COCO-20${}^i$ datasets to demonstrate the effectiveness of the model, the results in 1-shot setting and 5-shot setting on PASCAL-5${}^i$ are 64.7% and 68.0%, which are 3.9% and 6.1% higher than the baseline model, respectively, and the results on the COCO-20${}^i$ are also improved.

1. Introduction

With the rapid development of deep neural networks [1,2,3,4,5,6,7], the research on semantic segmentation has made great progress [8,9,10,11]. These achievements can be largely attributed to the datasets with pixel-level manual annotation [12,13]. However, obtaining such annotated data is time-consuming and labor-intensive, and the performance of the model will drop significantly in the face of unseen classes. To alleviate the above problems and achieve considerable predicted segmentation results under the condition of giving only a few labeled samples, the few-shot semantic segmentation (FSS) method [14] was proposed. Many researchers have conducted a lot of work to improve the model’s performance, and few-shot semantic segmentation has become an active topic in the field of computer vision.

Few-shot semantic segmentation is similar to the few-shot learning approach used for image classification and recognition [15,16,17]. The network used for training is fed with two sets of images called support set and query set, the support set will contain the corresponding true segmentation mask and the query set needs to be given prediction values by the network. The role of the support set is to guide the prediction of the query set. In few-shot learning, the goal of few-shot image classification and recognition is to predict the class of the image or the location of the target class in the image (usually by labeling bounding boxes), both of which require image-level prediction. Unlike few-shot image classification and recognition tasks, few-shot image semantic segmentation requires pixel-level predictions, which is more complex and challenging. Early research [14,18] laid the basic model framework for FSS. A large number of follow-up studies [19,20,21,22] are based on these foundations to make innovative improvements. To achieve fine pixel-level prediction, the network needs to be trained on a relatively large dataset to obtain the weights, and the data in the test phase have classes that do not appear in the training phase. Since the network is more inclined to segment objects of known classes, it is not trivial to obtain fine segmentation predictions for unknown classes. Recent studies have introduced meta-classes [23], global and local contrastive learning [24], support and query cross-guide [25], and in the test phase, methods such as updating pixel classifiers for different unknown classes [26] or extracting latent features [27]. All the above methods can improve the performance of the model well. However, these methods are inadequate in utilizing the multilevel features obtained at the encoding stage, thus losing part of the detailed information and leading to insufficient segmentation prediction accuracy.

Based on the above problem, we propose a few-shot semantic segmentation network using multilevel features-guided segmentation, which makes full use of each level’s features from the support branch and query branch to obtain more detailed segmentation predictions. Specifically, we use a novel module to implement support features at each level to guide the query features at each level, and cross-connect the query features of each level to the decoder to guide the segmentation. The motivation of our method are (1) in the process of downsampling through the convolutional network, as the network goes deeper, the features are more abstract, and the information contained in the features is more global. Although the network can use these features to make segmentation predictions, some local and edge information will inevitably be lost, and support features will not provide sufficient guidance for query features, resulting in unsatisfactory results. Inspired by HSNet [28], we considered if the multilevel query features participate in the segmentation prediction process under the guidance of the support features, the different information contained in the features of different levels can be more fully utilized to improve the accuracy, and (2) for the abstracted high-level query features, the common method is to directly send it to the decoder to obtain the predicted segmentation result, but this will introduce uncertain factors due to upsampling interpolation, which will lead to some segmentation errors.

The semantic information contained in the features obtained in the encoding stage can guide segmentation, and a cross-connection approach can be used to guide the decoder upsampling process to reduce uncertainty and improve segmentation quality. The main contributions of this paper are as follows:

• We propose a similarity-guided feature reinforcement module (SRM) to enrich query features and enhance the coherence of support features to query feature guidance, so that more semantic information can be included when predicting query image segmentation results.
• Connecting the multilevel features of the encoder to the decoder provides more accurate segmentation guidance for the decoder upsampling, which effectively alleviates the accuracy loss caused by the depthwise convolution process, and a reasonable approach with limited use of cross-connect features also avoids the negative impact of non-critical information on segmentation.
• Compared to previous work, our proposed method has improved performance on the PASCAL-5${}^i$ dataset and also performs well on the COCO-20${}^i$ dataset.

2. Related Work

2.1. Semantic Segmentation

The purpose of image semantic segmentation is to understand the image at the pixel level and assign an object class to each pixel in the image. The introduction of the Fully Convolutional Network [10] marked the application of deep learning in image semantic segmentation. Many subsequent studies have improved the performance of the model and made many contributions in the field of image semantic segmentation. Such as Seg-Net [8], U-Net [9], PSPNet [11], deep-lab [29,30], DFANet [31], CRGNet [32], ENet [33]. Although remarkable achievements have been made in image semantic segmentation the current conventional semantic segmentation methods are not satisfactory in the face of unseen classes.

2.2. Few-Shot Learning

Few-shot learning refers to obtaining appropriate prediction results under the condition of giving only a few labeled samples. Few-shot learning was originally used for image classification and image recognition. According to different methods, it can be divided into metric-learning methods [17,34,35,36] or meta-learning based methods [16,37,38,39,40,41]. Graph neural network-based methods also have many applications in few-shot learning [15,42]. The goal of few-shot learning is to make image-level predictions, whereas the goal of few-shot semantic segmentation is to make pixel-level predictions, which is more challenging. Our work is closer to a meta-learning based approach, but also applies the idea of metric-learning.

2.3. Few-Shot Semantic Segmentation

OSLSM [14] applied the few-shot learning method to the field of semantic segmentation for the first time. Since then, many studies [19,43,44,45,46] have added different structures to the basic model to make few-shot segmentation methods more diverse. SG-one [47] proposed the use of global average pooling with mask to extract target features. CANet [48] focused on the mid-level features of the decoder to obtain the hidden local features of the target. FWB [49] enhanced the segmentation ability by activating foreground features. DAN [50] introduced a homogenized graph attention mechanism to activate more pixels on the object. BriNet [51] reduced intra-class variance by interacting support features and query features. SAGNN [52] designed a scale-aware graph neural network to explain the cross-scale structural relationship. PFENet [53] proposed a prior mask generation method and constructs a feature enrichment module, which significantly improves the model for few-shot segmentation. MANet [54] used mask aggregation network, simultaneously generate a fixed number of masks and their probabilities of being targets. DCP [55] leveraged effective masked average pooling operations to derive a series of support-induced proxies, with different agents conquering different challenges. Our method is constructed based on the framework of PFENet [53]. Different from PFENet [53], our method pays more attention to the different features contained in all levels of features and their relationships, especially the local and edge information contained in low-level features.

3. Material and Methods

3.1. Problem Definition

The goal of few-shot semantic segmentation is to obtain suitable segmentation results with a trained network environment, even for unseen classes with little image annotation information. In the few-shot semantic segmentation, the data of the training set $D_{train}$ and the test set $D_{test}$ are isolated from each other, that is, the classes of the training set $C_{train}$ and the classes of the test set $C_{test}$ are disjoint ($C_{train}\cap C_{test}=⌀$). The training set $D_{train}$ and the test set $D_{test}$ are divided into support set S and query set Q, respectively. For k-shot segmentation, $S=\{x_i^s,m_i^s{\}}_{i=1}^k$ means that it contains k support images, $x_i^s\in R^{3\times H\times W}$ is the RGB information of the support image, $m_i^s\in {(0,1)}^{H\times W}$ is the binary segmentation mask corresponding to the support image; $Q=\{x^q,m^q\}$ usually contains one query image, $x^q\in R^{3\times H\times W}$ is the RGB information of the query image, $m^q\in {(0,1)}^{H\times W}$ is the binary segmentation mask corresponding to the query image.

The main process of the few-shot semantic segmentation network is: first, the support-query image pair $(x^s,x^q)$ is sent to the encoder to extract the support feature $F_i^s$ and query feature $F^q$, then, combine the support feature $F_i^s$ with the corresponding mask $m_i^s$ to guide the query feature $F^q$ to obtain the predicted mask ${\widehat{m}}^q$, and finally, compare the predicted mask ${\widehat{m}}^q$ with the ground-truth mask $m^q$. After the network is trained on the training set $D_{train}$ for several rounds, the weights of the network are locked, and then the network is tested on the test set $D_{test}$ to judge the network performance.

3.2. The Proposed Model

We propose a few-shot semantic segmentation network, which uses multilevel features to guide segmentation prediction. By performing similarity-guided processing of features at different levels, local information of low-level features is retained while abstract information of high-level features is obtained. In the decoding process, query features at all levels are crossed over to the decoder, and limited use is made of query features when they guide the decoding process at all levels, thus reducing the adverse effects of non-critical information and making full use of edge and local information, thus enabling the network to better predict segmentation. The main frame structure of the network is shown in Figure 1.

First, the support-query image pair $\{x^s,x^q\}$ is processed by the backbone network to obtain the corresponding features of different levels $\{F_1^s,F_2^s,F_3^s\},\{F_1^q,F_2^q,F_3^q\}$:

$$\begin{array}{cc}\hfill F_1^s,F_2^s,F_3^s& ={\mathcal{F}}_b\left(x^s\right),\hfill \\ \hfill F_1^q,F_2^q,F_3^q& ={\mathcal{F}}_b\left(x^q\right),\hfill \end{array}$$

(1)

where ${\mathcal{F}}_b(•)$ represents the feature extraction of the image by the backbone network, and the encoder of the support branch and the query branch share the same weights. Then, in order to obtain the rich semantic information, the extracted support features of each level $\{F_1^s,F_2^s,F_3^s\}$ and their masks $m^s$, combined with the query features $\{F_1^q,F_2^q,F_3^q\}$, go through three independent similarity-guided feature reinforcement modules (SRM) to get the query feature $F_{SRM}$ containing rich information:

$$\begin{array}{cc}\hfill F_{SRM}^3& ={\mathcal{F}}_s^3(F_3^s,F_3^q,m^s),\hfill \\ \hfill F_{SRM}^2& ={\mathcal{F}}_s^2(F_2^s,F_2^q,m^s,F_{SRM}^3),\hfill \\ \hfill F_{SRM}& =F_{SRM}^1={\mathcal{F}}_s^1(F_1^s,F_1^q,m^s,F_{SRM}^2),\hfill \end{array}$$

(2)

where ${\mathcal{F}}_s^i{(•)}_{i=1}^3$ represents the calculation process of the i-th SRM, which we will describe in detail in Section 3.3. After that, $F_{SRM}$ is combined with the query feature, fed into the Atrous Spatial Convolution Pooling Pyramid (ASPP) to obtain $Y_q$. Finally, $Y_q$ are fed into the decoder under the guidance of $\{F_1^q,F_2^q,F_3^q\}$ to obtain the predicted segmentation mask ${\widehat{m}}^q$:

$$\begin{array}{cc}\hfill {\widehat{m}}^q& ={\mathcal{F}}_d({\mathcal{F}}_d({\mathcal{F}}_d(Y_q,F_3^q),F_2^q),F_1^q),\hfill \end{array}$$

(3)

where ${\mathcal{F}}_d(•)$ represents the calculation process of the decoder with limited use of cross-connect features guidance, which we will describe in detail in Section 3.4. The main process of network training is briefly shown in Algorithm 1. The number of iterations and pre-trained weights will be introduced in Section 4.2.

Algorithm 1 The main process of training multilevel features-guided network

Input: Support set $S=\{x^s,m^s\}$, Query set $Q=\{x^q,m^q\}$ 1: Initialize the network nodes (the backbone part is initialized with pre-trained weights) 2: for each iteration do 3:     Feed $x^s$ and $x^q$ to backbone to get $F_i^s$ and $F_i^q$. 4:     Feed $(F_i^s,m^s)$ and $F_i^q$ to the SRMs to get $F_{SRM}^i$. 5:     Feed $(F_2^s,m^s)$ and $F_{SRM}^1$ to the ASPP to get $Y_q$. 6:     Feed $Y_q$ to the decoder to get the prediction ${\widehat{m}}^q$. 7:     Compare ${\widehat{m}}^q$ with $m^q$, update network nodes except backbone. 8: end for

3.3. Similarity-Guided Feature Reinforcement Module (SRM)

As mentioned earlier, using only mid-level and high-level features when support features guide query features will lose some local and edge information, and adding low-level features can alleviate this problem. However, simply adding low-level features introduces redundant non-critical information, which is instead detrimental to the effective guidance of query features by support features. Therefore, we use the support features with mask information for guidance while processing the query features of the three levels in turn, and use the processed higher-level features to continue to guide the lower-level features, thus obtaining features with multiple levels of information. Based on this idea, we design Similarity-Guided Feature Reinforcement Module (SRM). The structure diagram of SRM is shown in Figure 2. First, the support features and their masks$\{F_i^s,m_i^s{\}}_{i=1}^3$ are fused to obtain the support features with mask information $\dot{F_i^s}$:

$$\begin{array}{c}\hfill {\dot{F}}_i^s=F_i^s\odot m_i^s,\end{array}$$

(4)

where ⊙ is the Hadamard product. Then, calculate the Similarity $\left(Sim\right)$ between the support-query feature pair ${\{{\dot{F}}_i^s,F_i^q\}}_{i=1}^3$ using the cosine similarity principle:

$$Sim=\frac{{\dot{F}}_i^s·F_i^q}{\parallel {\dot{F}}_i^s\parallel \parallel F_i^q\parallel },$$

(5)

Finally, combine the $Sim$ with the support-query feature pair ${\{{\dot{F}}_i^s,F{{}_i}^q\}}_{i=1}^3$ to get output feature of $F_{SRM}^i$:

$$F_{SRM}^i=F_i^q\odot ((Sim\otimes F_i^q)\oplus F_{SRM}^{higher}),$$

(6)

where $F_{SRM}^{higher}$ represents the features of the higher-level introduced in the first two SRMs, and the last SRM does not have this parameter, ⊗ represents the matrix multiplication, and ⊕ represents the concatenation operation along channel dimensions.

We designed this structure with branches following SENet [2] to enable the guidance of support features while processing query features. There are three independent SRMs in our model, corresponding to the low, medium, and high-level features of the support and query set, respectively. According to CANet [48], the use of mid-level features can yield satisfactory results, so how the different number of SRMs and the different ways of using features will affect the experimental results will be discussed in Section 5.2.

3.4. Multilevel Features-Guided Decoder

Following PFENet [53], the obtained features $F_{SRM}$ after processing in multiple SRM modules are fed into the Atrous Spatial Pyramid Pool (ASPP) together with the mid-level features $F_2^s$ of the query image to obtain new features $Y_q$:

$$\begin{array}{c}\hfill Y_q={\mathcal{F}}_a(F_{SRM}\oplus Pooling({\dot{F}}_2^s)),\end{array}$$

(7)

where ${\mathcal{F}}_a(•)$ represents the process of ASPP processing features, $F_{SRM}$ is calculated by Equation (2), and ${\dot{F}}_2^s$ is calculated by Equation (4), here we replace the FEM of PFENet [53] with ASPP to reduce the computational complexity.

The prediction segmentation can be obtained by decoding $Y_q$, but to take full advantage of the rich semantic information contained in each level of query features, we combine the multilevel query features $\{F_1^q,F_2^q,F_3^q\}$ with the new features $Y_q$ in the decoding process. In this case, more detailed segmentation can be achieved by using features wisely. However, if the features are directly concatenated with the new features $Y_q$, the decoding result becomes undesirable due to the introduction of too much non-critical information, so the useless information of the query features should be filtered out before combining them with the new features $Y_q$ in the decoder. Therefore, we make limited use of the introduction of query features, that is, we combine $Y_q$ with query features and then add them to $Y_q$, as shown in Figure 3. Specifically, the features of the query set ${\left(F_i^q\right)}_{i=1}^3$ are guided by $Y_q$ to get ${\left({\dot{F}}_i^q\right)}_{i=1}^3$, and then combined with $Y_q$ and update $Y_q$, as:

$$\begin{array}{c}{\dot{F}}_i^q=avgPooling\left(Y_q\right)\odot F_i^q,\hfill \\ Y_q+={\dot{F}}_i^q\hfill \end{array}$$

(8)

$Y_q$ passes through query set high-level features $F_3^q$, mid-level features $F_2^q$ and low-level features $F_1^q$ in turn to guide segmentation in the decoder. According to previous work [23,53], decoding without introducing multilevel feature guidance can yield the expected results, therefore, the effect of different numbers of feature-guided decoders on the results, and the effect of different ways of feature-guided decoders on the results, will be discussed in Section 5.2.

3.5. Loss Function

We use cross-entropy loss $\mathcal{L}=-\frac{1}{N}\sum [ylogp+(1-y)log(1-p)]$ as our loss function, where N is the total number of pixels, $y\in \{0,1\}$ is the pixel label (0 for background, 1 for foreground), and p is the probability that the prediction is positive. Following PFENet [53], an auxiliary loss function is also used to improve the performance of the model. The auxiliary loss ${\mathcal{L}}_{aux}$ is obtained by performing intermediate-level supervision training between the fuzzy segmentation prediction obtained by decoding $Y_q$ and the real binary segmentation mask of the query image. The final prediction result of the network produces the final loss ${\mathcal{L}}_{final}$, so the overall loss ${\mathcal{L}}_{all}$ is expressed as:

$${\mathcal{L}}_{all}=\lambda {\mathcal{L}}_{aux}+{\mathcal{L}}_{final},$$

(9)

where $\lambda$ represents the balance weight of the auxiliary loss, which we set to 1.0 in all experiments.

4. Implementation Details

4.1. Datasets

We evaluate the proposed model performance using two public datasets, PASCAL-5${}^i$ [14] and COCO-20${}^i$ [49], which are widely used in the field of few-shot semantic segmentation.

PASCAL-5${}^i$ was first used by Shaban et al. [14], derived from the PASCAL VOC [12] dataset, and enhanced with the SDS [56]. PASCAL VOC dataset contains 20 categories of images, which will be evenly divided into 4 subsets. The i subset is called PASCAL-5${}^i$, and each subset contains 5 categories of images.

COCO-20${}^i$ was first produced and used by Nguyen et al. [49], derived from MSCOCO [13]. In COCO-20${}^i$, a total of 80 categories of images are included. Similar to PASCAL-5${}^i$, these 80 categories will be divided into 4 subsets, and the i subset is called COCO-20${}^i$. Each subset contains 20 categories of images. Compared with PASCAL-5${}^i$, COCO-20${}^i$ has more categories and a larger number of pictures. Therefore, experiments on COCO-20${}^i$ are more challenging than those on PASCAL-5${}^i$.

Following OSLSM [14] and FWB [49], for four different subsets of PASCAL-5${}^i$ and COCO-20${}^i$, we use a cross-validation training method: three subsets are used as training sets to train the model, and the remaining subset is used as the test set to test the model, and 1000 support-query pairs are randomly selected from the test set for testing.

4.2. Experimental Setting

All experiments are conducted on PyTorch framework. We select ResNet-50 and ResNet-101 [1] as the backbone and use the weights pre-trained on ImageNet [57] for initialization. During the training process, the weights of ResNet remain unchanged, and the input images are all cropped to $473\times 473$. Following the previous work [23,53,58], the batch sizes of PASCAL-5${}^i$ and COCO-20${}^i$ are 4 and 8, respectively, both using SGD optimizer, the number of iterations (epochs) is 200 rounds and 50 rounds, respectively, the learning rate is 0.0025 and 0.005, and the poly strategy is used to adjust the learning rate, that is, $l{r}_{new}=l{r}_{base}\times {(1-\frac{curren{t}_{epoch}}{ma{x}_{epoch}})}^{power}$, where $power$ is equal to 0.9. All experiments are performed on an NVIDIA RTX 3090 GPU.

4.3. Evaluation Metrics

According to CANet [48] and BriNet [51], we adopt the mean intersection over union (mIoU) as our major evaluation metric because the foreground-background intersection over union (FB-IoU) cannot reflect the model capability well. However, we still add the FB-IoU results to the table for comparison. The mIoU is calculated by $mIoU=\frac{1}{C}{\sum }_{i=1}^{C}Io{U}_i$, where C is the number of each fold, and $IoU=\frac{TP}{TP+FP+FN}$, where $TP$, $FP$, and $FN$ represent the counts of true positives, false positives, and false negatives respectively.

5. Results

5.1. Comparison with Other Methods

We compared our experimental results with few-shot semantic segmentation methods from recent years.

As shown in

Table 1. Results and comparison of mIoU and FB-IoU on the four folds of PASCAL-5${}^i$. Bold numbers represent the best performance.

Backbone	Methods	1-Shot						5-Shot
		Fold-0	Fold-1	Fold-2	Fold-3	Mean	FB-IoU	Fold-0	Fold-1	Fold-2	Fold-3	Mean	FB-IoU
ResNet50	CANet${}_{2019}$ [48]	52.5	65.9	51.3	51.9	55.4	66.2	55.5	67.8	51.9	53.2	57.1	69.6
	BriNet${}_{2020}$ [51]	56.5	67.2	51.6	53.0	57.1	-	-	-	-	-	-	-
	PFENet${}_{2020}$ [53]	61.7	69.5	55.4	56.3	60.8	73.3	63.1	70.7	55.8	57.9	61.9	73.9
	CMN${}_{2021}$ [58]	64.3	70.0	57.4	59.4	62.8	72.3	65.8	70.4	57.6	60.8	63.7	72.8
	HSNet${}_{2021}$ [28]	64.3	70.7	60.3	60.5	64.0	76.7	70.3	73.2	67.4	67.1	69.5	80.6
	MANet${}_{2022}$ [54]	62.0	69.4	51.8	58.2	60.3	71.4	66.0	71.6	55.1	64.5	64.3	75.2
	DCP${}_{2022}$ [55]	63.8	70.5	61.2	55.7	62.8	75.6	67.2	73.2	66.4	64.5	67.8	79.7
	Ours	64.4	70.8	63.4	60.3	64.7	76.4	67.3	73.7	66.2	64.9	68.0	79.3
ResNet101	FWB${}_{2019}$ [49]	51.3	64.5	56.7	52.2	56.2	-	54.8	67.4	62.2	55.3	59.9	-
	PFENet${}_{2020}$ [53]	60.5	69.4	54.4	55.9	60.1	72.9	62.8	70.4	54.9	57.6	61.4	73.5
	DAN${}_{2020}$ [50]	54.7	68.6	57.8	51.6	58.2	71.9	57.9	69.0	60.1	54.9	60.5	72.3
	HSNet${}_{2021}$ [28]	67.3	72.3	62.0	63.1	66.2	77.6	71.8	74.4	67.0	68.3	70.4	80.6
	MANet${}_{2022}$ [54]	63.9	69.2	52.5	59.1	61.2	71.4	67.0	70.8	54.8	65.5	64.5	74.1
	Ours	66.1	72.8	64.9	62.0	66.5	76.8	67.6	74.5	67.2	65.4	68.7	79.6

, The results obtained by training and testing the model on the PASCAL-5${}^i$ dataset are presented, the data include the mIoU of each subset and the overall average mIoU and FB-IoU in 1-shot setting and 5-shot setting. It can be found that our method can achieve considerable results when the backbone networks are the same, especially in 1-shot setting with ResNet50 as the backbone, which is significantly better than other methods, there is 3.9% improvement compared to PFENet [53] and 0.7% improvement compared to HSNet [28].

As shown in

Table 2. Results and comparison of mIoU and FB-IoU on the four folds of COCO-20${}^i$. Bold numbers represent the best performance.

Backbone	Methods	1-Shot						5-Shot
		Fold-0	Fold-1	Fold-2	Fold-3	Mean	FB-IoU	Fold-0	Fold-1	Fold-2	Fold-3	Mean	FB-IoU
ResNet50	BriNet${}_{2020}$ [51]	32.9	36.2	37.4	30.9	34.4	-	-	-	-	-	-	-
	CMN${}_{2021}$ [58]	37.9	44.8	38.7	35.6	39.3	61.7	42.0	50.5	41.0	38.9	43.1	63.3
	HSNet${}_{2021}$ [28]	36.3	43.1	38.7	38.7	39.2	68.2	43.3	51.3	48.2	45.0	46.9	70.7
	MANet${}_{2022}$ [54]	33.9	40.6	35.7	35.2	36.4	-	41.9	49.1	43.2	42.7	44.2	-
	DCP${}_{2022}$ [55]	40.9	43.8	42.6	38.3	41.4	-	45.9	49.7	43.7	46.7	46.5	-
	Ours	40.8	45.5	41.1	39.1	41.6	65.2	46.1	52.3	46.2	44.3	47.2	69.1
ResNet101	FWB${}_{2019}$ [49]	17.0	18.0	21.0	28.9	21.2	-	19.1	21.5	23.9	30.1	23.7	-
	DAN${}_{2020}$ [50]	-	-	-	-	24.4	62.3	-	-	-	-	29.6	63.9
	PFENet${}_{2020}$ [53]	36.8	41.8	38.7	36.7	38.5	63.0	40.4	46.8	43.2	40.5	42.7	65.8
	HSNet${}_{2021}$ [28]	37.2	44.1	32.4	41.3	41.2	69.1	45.9	53.0	51.8	47.1	49.5	72.4
	Ours	41.0	45.6	40.6	39.6	41.7	65.6	46.5	53.1	45.6	43.2	47.1	69.1

, the results obtained by training and testing the model on the COCO-20${}^i$ dataset are presented, the data include the mIoU of each subset and the overall average mIoU and FB-IoU in 1-shot setting and 5-shot setting. It can be seen that our method has a slight performance improvement compared to some previous work and is competitive in 1-shot setting, there is 0.7% improvement compared to HSNet [28] with ResNet50 as the backbone, 3.7% improvement compared to PFENet [53] with ResNet101 as the backbone, and a 1.0% improvement compared to HSNet [28] with ResNet101 as the backbone.

However, our model does not perform well on the 5-shot setting, especially on COCO-20${}^i$ with ResNet101 as the backbone, and even shows degraded performance. One of the reasons may be the limitation of the hardware device used, which cannot support higher resolution images as input and can only crop the images to 473 × 473, and the number of training rounds is only 50, so the experimental results are not satisfactory.

The qualitative results are shown in Figure 4, where we use PFENet [53] as the baseline. From the figure, we can see that our method is more accurate in detail while segmenting the target effectively.

5.2. Ablation Study

To verify the effectiveness of the proposed method, we conducted experiments on Fold-0, a subset of PASCAL-5${}^i$, using ResNet-50 as the backbone, and compared the impact of our approach with different components on the experimental results.

5.2.1. Number of SRMs

As shown in

Table 3. Comparison of the results of ablation experiments at different numbers of SRMs on PASCAL-5${}^i$.

SRM			Mean-IoU
Low-Level	Mid-Level	High-Level
-	-	-	57.5
-	✓	✓	63.1
✓	-	✓	64.4
✓	✓	-	58.5
✓	-	-	58.2
-	-	✓	63.6
✓	✓	✓	65.7

, we tested the model performance with different numbers of SRMs and compared the results. It can be seen that if the high-level SRMs are used, the test results are close to that of using all levels of SRMs, and are significantly better than the result of removing the high-level SRMs. This may be due to the high-level features typically contain highly abstracted image features that play a critical role in computing prediction results. Using only a single low-level SRM, the query features can provide some guidance on the support features, but the effect is minimal. When high-level SRMs are added the model performance can be significantly improved and the performance results are better than those of high-level SRMs alone, which proves the relevance of using SRMs with multilevel features.

5.2.2. Number of Features Cross-Connect to the Decoder

As shown in

Table 4. Comparison of experimental results of ablation with different number of features cross-connect to decoders on PASCAL-5${}^i$.

Query Features			Mean-IoU
Low-Level	Mid-Level	High-Level
-	-	-	64.2
-	✓	✓	63.9
✓	✓	-	64.4
-	-	✓	62.8
✓	-	-	64.9
✓	✓	✓	65.7

, we tested and compared the results with different numbers of features cross-connect to the decoder. It can be seen that when using only high-level features cross-connect to the decoder for segmentation prediction, the performance is even worse than the case without cross-connect features. This may be due to the high-level features themselves are already global and abstract enough, cross-connecting to the decoder does not provide more segmentation information but worsens the segmentation result by introducing non-critical information. However, although the low-level features contain some interference from non-target information, the limited use of features in Section 3.4 makes it possible to effectively filter out too much useless information and provide more accurate segmentation guidance, and the performance is improved by using multiple levels of cross-connect features.

5.2.3. Different Methods of Using Multilevel Features to Guide Segmentation

As shown in

Table 5. Experimental results of ablation with different feature-guided methods on the decoder.

Feature-Guided Method	Mean-IoU
	Three-Level	Low-Level
-	64.2	64.2
Concatenate by channel	61.1	62.5
Element-wise product	63.6	63.3
Matmul product	64.4	63.1
limited use of features	65.7	64.9

, we tested and compared the results of using different features guide methods in the decoder. It can be seen that the performance of directly connecting features through channels is worse than that of the decoder without the guidance of cross-connect feature, and the performance of element multiplication and matrix multiplication is not significantly improved, whereas the limited use of features to guide segmentation in Section 3.4 can significantly improve the model performance. Because of the good performance of low-level features in Section 5.2.2, we add a set of experiments that only use low-level features to guide decoding for comparison. The experimental results are shown in the last column of Table 5. It can be seen that when only low-level features are used, if the features are not used in a limited manner, the results will also be worse than those with limited use of features.

6. Conclusions

We propose a multilevel features-guided network for few-shot semantic segmentation. In this network, we design a similarity-guided reinforcement module to better implement the guidance of support features to query features while processing query features at all levels. In the decoder part, we use the method of cross-connecting multilevel features of the query image to guide the segmentation prediction of the decoder and make limited use of the cross-connect features, so the segmentation prediction contains sufficient local and edge information. Our method is trained and tested on PASCAL-5${}^i$ and COCO-20${}^i$ datasets, and good results are obtained. Our method also has many shortcomings. During the guidance of support features to query features, only the foreground information is used, and the background information is not used, but reasonable processing of the background information can improve the generalization ability of the model. In addition, the network structure is not light enough, for example, the lower-level SRM is guided by the support features and the output features of the higher-level SRM, the redundant information of the two guiding features may lead to heavy computation. We hope to make improvements in our future work.