Deep Successive Convex Approximation for Image Super-Resolution

Abstract

Image super-resolution (SR), as one of the classic image processing issues, has attracted increasing attention from researchers. As a highly ill-conditioned, non-convex optimization issue, it is difficult for image SR to restore a high-resolution (HR) image from a given low-resolution (LR) instance. Recent researchers have tended to regard image SR as a regression task and to design an end-to-end convolutional neural network (CNN) to predict the pixels directly, which lacks inherent theoretical analysis and limits the effectiveness of the restoration. In this paper, we analyze image SR from an optimization perspective and develop a deep successive convex approximation network (SCANet) for generating HR images. Specifically, we divide non-convex optimization into several convex LASSO sub-problems and use CNN to adaptively learn the parameters. To boost network representation, we use residual feature aggregation (RFA) blocks and devise a spatial and channel attention (SACA) mechanism to improve the restoration capacity. The experimental results show that the proposed SCANet can restore HR images more effectively than other works. Specifically, SCANet achieves higher PSNR/SSIM results and generates more satisfying textures.

1. Introduction

Image super-resolution (SR) is one of the classic image processing issues [1]. Given a low-resolution (LR) image, the task of image SR is to generate a corresponding high-resolution (HR) image with higher subjective and objective performance [2]. Image SR has been widely considered in various applications, such as iris and ocular recognition [3] and other high-level computer vision tasks [4].

With the rapid development of deep learning, there are numerous convolutional neural networks (CNNs) that have been specifically designed to solve the image SR problem [5,6,7,8]. SRCNN [5] is the first CNN-based image SR method to generate satisfying results from the given instances. After SRCNN, deeper and wider networks have achieved good performance with elaborate block designs [7,8]. However, these works only regard image SR as a regression issue and use an end-to-end network to directly predict the pixels of HR images. This direct regression limits the precision of the restoration methods.

As a highly ill-conditioned problem, image SR can be regarded as an optimization issue [9,10,11,12,13]. From an optimization perspective, image SR aims to find a satisfying result which is in accordance with the LR representation after down-sampling and follows the prior HR image. As such, image SR can be regarded as an unconstrained optimization problem under the consideration of the fidelity term and the prior term. There are also works regarding image SR as an optimization issue [9,10,11,12,13]. Half-quadratic splitting (HQS) and the alternating direction method of multipliers (ADMM) are two representative strategies to iteratively find a feasible solution for non-convex tasks. Recently, works have considered HQS and ADMM for image SR, which have achieved good performances [9,10]. However, both HQS- and ADMM-based CNNs solve the non-convex problem without a specific prior HR image, which makes it hard to find a satisfying solution and limits the effectiveness of the network.

In this paper, we analyze image SR from a non-convex optimization perspective and solve the issue in the successive convex approximation (SCA) manner. Specifically, we divide the non-convex problem into several convex LASSO sub-problems and calculate the solution successively. From this point of view, the prior HR image is explicitly considered as the ${\ell }_1$-norm, which follows the sparsity of the natural images. We design a network, dubbed SCANet, to adaptively learn the parameters and conduct the optimization. To improve the network capacity, we use residual feature aggregation (RFA) blocks [14] and devise a spatial and channel attention (SACA) mechanism in SCANet. The experimental results show that the proposed SCANet achieves better performance than other image SR works, with higher PSNR/SSIM results. Furthermore, SCANet can generate more satisfying textures and details with better visual performances than other works.

Our contributions can be concluded as follows:

• We analyze image SR from a non-convex optimization perspective and propose an iterative solution by dividing the task into several convex LASSO sub-problems, which is more effective than other iteration solutions. Based on this analysis, we devise a SCANet for image super-resolution.
• We utilize residual feature aggregation (RFA) blocks and develop a spatial and channel attention (SACA) mechanism in SCANet, which can improve network representation and restoration capacity.
• The experimental results show that the proposed SCANet achieves better subjective and objective performance than other works, with higher PSNR/SSIM results.

2. Related Works

Recently, convolutional neural networks (CNNs) have found great success in image SR [15]. SRCNN [5] is the first CNN-based image SR method that utilizes three-layer architecture to generate HR details. After SRCNN, numerous well-designed networks have been proposed for effective image SR. FSRCNN [6] modified the SRCNN for faster speed and higher restoration precision with a thinner but deeper design. VDSR [7] used a very deep network design with a residual connection to achieve good restoration performance. EDSR [8] developed a deep residual network with residual blocks [16] for better information transmission and network representation. DRRN [17] and DRCN [18] considered the recursive design for image SR and utilized an iterative design for restoration. Recently, researchers have mainly focused on elaborate block designs for image SR. CARN [19] built a cascading network for efficient image SR. OISR [20] designed the block according to the ODE pattern and achieved good performance. RDN [21] combined the residual connection [16] and the dense connection [22] and developed a residual dense block for image SR. SRDenseNet [23] built a deep network with dense connections for better gradient transmission. RCAN [24] utilized a residual-in-residual design for effective gradient transmission and made the network deeper. SAN [25] developed a second-order attention mechanism for image SR and achieved state-of-the-art performance. However, these works only consider building a deeper and wider network with well-designed blocks for restoration, and neglect to consider the image SR problem from a mathematical perspective.

By considering the image SR issue from a mathematical perspective, it can be regarded as a highly ill-conditioned issue that can be solved using optimization methodologies. IRCNN [9] utilized a half-quadratic splitting (HQS) strategy to first separate the image SR into a closed-form solution and a denoiser. DPSR [10] considered a novel image SR degradation situation, and proposed a deep plug-and-play model to solve the problem. However, these works only aimed to solve the non-convex problem using the iterative mechanism, which lacks the explicit prior definition of HR images and makes it difficult to find a satisfying solution.

3. Methodology

In this section, we introduce the proposed SCANet in the following manner. First, we analyze image SR as an optimization issue and propose a solution with the help of successive convex approximation (SCA). Then, an end-to-end network is designed to find a solution that adaptively learns the parameters from the data pairs. Finally, we discuss the details of the proposed SCANet.

3.1. Successive Convex Approximation for Image Super-Resolution

Given a low-resolution (LR) image ${\mathbf{I}}^{LR}$, the task of image SR is to find a feasible high-resolution (HR) image ${\mathbf{I}}^{HR}$, satisfying

$${\mathbf{I}}^{HR}=arg\underset{{\mathbf{I}}^{HR}}{min}\frac{1}{2}\left|\right|{\mathbf{I}}^{LR}-\mathcal{D}\left({\mathbf{I}}^{HR}\right){\left|\right|}_{\ell }^2+\lambda \Phi \left({\mathbf{I}}^{HR}\right),$$

(1)

where $\mathcal{D}(·)$ is the degradation operation and $\Phi (·)$ is the prior image of the HR images. $\lambda$ is a scaling factor.

In general, Equation (1) is a non-convex issue, for which it is difficult to find a satisfying solution. To find a feasible result, we utilize the successive convex approximation (SCA) strategy to convert this non-convex problem into several convex tasks and find the solution in an iterative manner.

For a k-th iteration, we aim to solve the task

$${\mathbf{I}}_k^{HR}=arg\underset{{\mathbf{I}}^{HR}}{min}\frac{1}{2}\left|\right|{\mathbf{I}}^{LR}-\mathbf{H}{\mathbf{I}}_{k-1}^{HR}{\left|\right|}_2^2+{\lambda }_{k-1}\left|\right|{\mathbf{I}}_{k-1}^{HR}{\left|\right|}_1,$$

(2)

where $\mathbf{H}$ is the matrix representation of the degradation operation $\mathcal{D}(·)$. In Equation (2), we choose ${\ell }_1$ norm as the prior image, which considers the sparsity of the natural image. It can be seen that Equation (2) embodies the same situation as the LASSO problem. To solve this equation, we use the sub-gradient descent method and find a feasible descent direction for optimization. Then, the descent direction of the k-th iteration can be shown as

$$d_k=-({\mathbf{H}}^T\mathbf{H}{\mathbf{I}}_{k-1}^{HR}-\mathbf{H}{\mathbf{I}}^{LR}+{\lambda }_{k-1}\mathrm{sign}\left({\mathbf{I}}_{k-1}^{HR}\right)).$$

(3)

By using this sub-gradient descent direction, the optimization of the k-th step is described as

$${\mathbf{I}}_k^{HR}={\mathbf{I}}_{k-1}^{HR}+\gamma d_k,$$

(4)

where $\gamma$ is the step length and $d_k$ is the descent direction of the k-th step. The descent direction is a theoretical guidance for the optimization step that decreases the distance between $I_k^{HR}$ and the ideal $I^{HR}$. The optimization steps follow the original steps of successive convex optimization, which have been investigated in previous optimization studies [26,27,28].

3.2. Deep Successive Convex Approximation Network

Based on mathematical analysis of image SR, we develop the deep successive convex approximation network (SCANet) for image SR. The design of SCANet is shown in Figure 1. In this figure, $x_{lr}$ is the feature of the LR image, $x_i$ is the feature processed using the i-th LASSO block, and $x_{hr}$ is the restored feature of the HR image. We conduct SCA in the feature space by designing special LASSO blocks. First, one convolution processes the input LR image and generates the corresponding feature maps as

$$x_{lr}=Conv\left({\mathbf{I}}^{LR}\right).$$

(5)

Figure 1. Network design of SCANet. $x_{lr}$ is the feature of the LR image, $x_i$ is the feature processed using i-th LASSO blocks, and $x_{hr}$ is the restored feature of the HR image.

After that, K LASSO blocks are used to conduct the optimization. For the k-th step, this is

$$x_k=LASSO(x_{lr},x_{k-1}).$$

(6)

The LASSO blocks follow the operation of Equations (3) and (4).

After K LASSO blocks, a padding structure is used to restore the HR image feature with residual connections as

$$x_{hr}=Padding\left(x_K\right)+x_{lr},$$

(7)

where $Padding(·)$ is composed of two convolutional layers and a ReLU activation.

Finally, an upscale module restores the HR image from the feature maps as

$${\mathbf{I}}^{HR}=Upscale\left(x_{lr}\right).$$

(8)

The upscale module is composed of one convolutional layer and a sub-pixel convolution [29].

Specifically, we describe the design of the LASSO block, which is the core of SCANet. Figure 2 shows the block design of the LASSO block. The block follows the analysis of Equations (3) and (4). In this figure, we can see that the ${\mathbf{H}}^T\mathbf{H}$ in Equation (3) is simulated using four residual feature aggregation blocks (RFAB) and one convolution. The $\mathbf{H}$ is simulated using two convolutional layers and a ReLU activation. Then, the learned descent direction of the k-th iteration step is calculated as

$$d_k=Con{v}_{1\times 1}\left([{\mathbf{H}}^T\mathbf{H}{x}_k-\mathbf{H}{x}_{lr},x_k]\right),$$

(9)

where $[·]$ is the concatenate operation. The $1\times 1$ convolution is used to adaptively learn the scaling factor ${\lambda }_{k-1}$. Finally, one $1\times 1$ convolution is used to conduct Equation (4) by adaptively learning the step length as

$$x_{k+1}=Con{v}_{1\times 1}\left([d_k,x_k]\right).$$

(10)

Figure 2. Block design of the LASSO block.

Figure 3 shows the design of RFAB in the LASSO block. RFAB has been proven to be a good component for image SR in recent works [14,30] and has been widely considered in previous SR works. According to previous reports [14,30], RFAB can more effectively explore the image features than other well-designed blocks.

Figure 3. Block design of RFAB.

Different from residual blocks, RFAB aggregates the residual information from different blocks to jointly learn multi-scale information and restores the image information. Compared with traditional residual blocks, RFAB has an adaptive ability to describe the residual features. There are three processing modules with the same design in RFAB, which are composed of two convolutional blocks, one ReLU activation, and a spatial and channel attention (SACA) mechanism. Let $Pro{c}_{RFAB}(·)$ be the processing module and $x_{in}$ be the input of RFAB; then, the operation of the processing modules can be described as

$$\left\{\begin{array}{cc}{r}_1\hfill & =Pro{c}_{RFAB}\left(x_{in}\right),\hfill \\ x_{in}^1\hfill & =x_{in}+r_1,\hfill \\ r_2\hfill & =Pro{c}_{RFAB}\left(x_{in}^1\right),\hfill \\ x_{in}^2\hfill & =x_{in}+r_2,\hfill \\ r_3\hfill & =Pro{c}_{RFAB}\left(x_{in}^2\right).\hfill \end{array}\right.$$

(11)

Then, the aggregated residual information is calculated via one $1\times 1$ convolution as

$$r_{all}=Con{v}_{1\times 1}\left([r_1,r_2,r_3]\right).$$

(12)

The output of RFAB $x_{out}$ is calculated as

$$x_{out}=x_{in}+r_{all}.$$

(13)

SACA in RFAB is shown in Figure 4. SACA focuses on spatial attention and channel attention separately, and utilizes one convolution to combine dual attention information. To this end, the two types of attention can focus on different dimensions of the image and improve the restoration capacity. We calculate the spatial attention and the channel attention via two parallel pathways. In each path, there are two convolutional layers, one ReLU activation and one sigmoid activation, which follow the design of various attention mechanisms [24,25]. There is one global average pool in the channel attention mechanism. After the spatial and channel attentions, one $1\times 1$ convolution combines the two dual information sources, and a residual connection is introduced for efficient gradient transmission.

Figure 4. Block design of SACA.

3.3. Implementation Details

There are $K=4$ LASSO blocks in SCANet. All convolutional layers have $c=64$ channels, and the kernel sizes are set as $3\times 3$, except for the $1\times 1$ convolutions. The filter in the upscale module is set as $c_{up}=3\times s\times s$, where s is the scaling factor of the image SR.

4. Experiment

4.1. Settings

We train our network with the DIV2K [31] dataset. The DIV2K dataset contains 900 high-resolution images for training and validation, with a resolution up to 2K. In this work, we first choose 800 images for training, and the last 5 images for validation, which are the same settings used in recent works [21,24]. The testing datasets chosen are Set5 [32], Set14 [33], B100 [34], Urban100 [35], and Manga109 [36], which are commonly used in different SR works. The scaling factors of image SR chosen are $\times 2$, $\times 3$ and $\times 4$. The learning rate chosen is $lr={10}^{-4}$. The patch size chosen is $48\times 48$ for LR images, and the batch size is the same as the RDN [21]. We update our network using the Adam optimizer [37] for 1000 epochs, and halve the learning rate for every 200 epochs. The metrics of image SR are chosen as PSNR and SSIM.

4.2. Ablation Study

In SCANet, we develop RFAB and SACA for effective image SR. To investigate the effectiveness of these blocks, we compare the PSNR and the SSIM results against three different benchmarks: B100 [34], Urban100 [35], and Manga109 [36]. Table 1 shows the results of the different designs.

Table 1. Investigation of RFAB and SACA with scaling factor $\times 4$.

RFAB	SACA	B100 [34]	Urban100 [35]	Manga109 [36]
✗	✗	27.50/0.7338	25.76/0.7763	30.05/0.9027
✓	✗	27.58/0.7364	26.12/0.7876	30.45/0.9079
✓	✓	27.58/0.7365	26.17/0.7893	30.50/0.9083

In this table, we can see that RFAB brings a large improvement to PSNR and SSIM. By comparing the first and second lines, RFAB brings a 0.4 dB improvement to the Urban100 and Manga109 benchmarks. The model also benefits from SACA. In the second and third lines, SACA results in 0.05 db PSNR and 0.02 SSIM for the Urban100 and Manga109 benchmarks. From this point of view, RFAB and SACA are effective components for the proposed SCANet.

4.3. Results

We compare our SCANet with several CNN-based methods: SRCNN [5], FSRCNN [6], VDSR [7], DRCN [18], CNF [38], LapSRN [39], DRRN [17], BTSRN [40], MemNet [41], SelNet [42], CARN [19], RAN [43], DNCL [44], FilterNet [45], MRFN [46], and DEGREE [47]. The scaling factors chosen are $\times 2$, $\times 3$, and $\times 4$. The testing benchmarks chosen are Set5, Set14, B100, Urban100, and Manga109. The metrics chosen are PSNR and SSIM.

Table 2 shows the PSNR/SSIM comparisons against five testing benchmarks with three different scaling factors. In the table, we can see that our SCANet achieves competitive or better performances than those in other works. SCANet achieves a 0.2 dB improvement over Urban100 and a 0.3 dB improvement over Manga109, with a scaling factor of $\times 2$. Similarly, there is a 0.2 dB PSNR improvement over Set5 compared with CARN when the scaling factor is $\times 3$. When the scaling factor is $\times 4$, SCANet achieves the highest PSNR/SSIM results when compared with other works. From this point of view, SCANet is effective for image SR.

Table 2. Average PSNR/SSIM with degradation model BI $\times 2$, $\times 3$, and $\times 4$ against five benchmarks. (+) means the result is better than our method, and (-) means the result is worse than our method.

Scale	Model	Set5 [32]	Set14 [33]	B100 [34]	Urban100 [35]	Manga109 [36]
		PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM
$\times 2$	SRCNN [5]	36.66(-)/ 0.9542(-)	32.42(-)/ 0.9063(-)	31.36(-)/ 0.8879(-)	29.50(-)/ 0.8946(-)	35.74(-)/ 0.9661(-)
	FSRCNN [6]	37.00(-)/ 0.9558(-)	32.63(-)/ 0.9088(-)	31.53(-)/ 0.8920(-)	29.88(-)/ 0.9020(-)	36.67(-)/ 0.9694(-)
	VDSR [7]	37.53(-)/ 0.9587(-)	33.03(-)/ 0.9124(-)	31.90(-)/ 0.8960(-)	30.76(-)/ 0.9140(-)	37.22(-)/ 0.9729(-)
	DRCN [18]	37.63(-)/ 0.9588(-)	33.04(-)/ 0.9118(-)	31.85(-)/ 0.8942(-)	30.75(-)/ 0.9133(-)	37.63(-)/ 0.9723(-)
	CNF [38]	37.66(-)/ 0.9590(-)	33.38(-)/ 0.9136(-)	31.91(-)/ 0.8962(-)	-	-
	LapSRN [39]	37.52(-)/ 0.9590(-)	33.08(-)/ 0.9130(-)	31.80(-)/ 0.8950(-)	30.41(-)/ 0.9100(-)	37.27(-)/ 0.9740(-)
	DRRN [17]	37.74(-)/ 0.9591(-)	33.23(-)/ 0.9136(-)	32.05(-)/ 0.8973(-)	31.23(-)/ 0.9188(-)	37.92(-)/ 0.9760(-)
	BTSRN [40]	37.75(-)/-	33.20(-)/-	32.05(-)/-	31.63(-)/-	-
	MemNet [41]	37.78(-)/ 0.9597(-)	33.28(-)/ 0.9142(-)	32.08(-)/ 0.8978(-)	31.31(-)/ 0.9195(-)	37.72(-)/ 0.9740(-)
	SelNet [42]	37.89(-)/ 0.9598(-)	33.61(-)/ 0.9160(-)	32.08(-)/ 0.8984(-)	-	-
	CARN [19]	37.76(-)/ 0.9590(-)	33.52(-)/ 0.9166(-)	32.09(-)/ 0.8978(-)	31.92(-)/ 0.9256(-)	38.36(-)/ 0.9765(-)
	RAN [43]	37.58(-)/ 0.9592(-)	33.10(-)/ 0.9133(-)	31.92(-)/ 0.8963(-)	-	-
	DNCL [44]	37.65(-)/ 0.9599(-)	33.18(-)/ 0.9141(-)	31.97(-)/ 0.8971(-)	30.89(-)/ 0.9158(-)	-
	FilterNet [45]	37.86(-)/ 0.9610(-)	33.34(-)/ 0.9150(-)	32.09(-)/ 0.8990(-)	31.24(-)/ 0.9200(-)	-
	MRFN [46]	37.98(-)/ 0.9611(-)	33.41(-)/ 0.9159(-)	32.14(-)/ 0.8997(-)	31.45(-)/ 0.9221(-)	38.29(-)/ 0.9759(-)
	SeaNet-baseline [48]	37.99(-)/ 0.9607(-)	33.60(-)/ 0.9174(-)	32.18(-)/ 0.8995(-)	32.08(-)/ 0.9276(-)	38.48(-)/ 0.9768(-)
	DEGREE [47]	37.58(-)/ 0.9587(-)	33.06(-)/ 0.9123(-)	31.80(-)/ 0.8974(-)	-	-
	SCANet (Ours)	38.05/0.9607	33.54/0.9174	32.16/0.8995	32.10/0.9281	38.62/0.9770
$\times 3$	SRCNN [5]	32.75(-)/ 0.9090(-)	29.28(-)/ 0.8209(-)	28.41(-)/ 0.7863(-)	26.24(-)/ 0.7989(-)	30.59(-)/ 0.9107(-)
	FSRCNN [6]	33.16(-)/ 0.9140(-)	29.43(-)/ 0.8242(-)	28.53(-)/ 0.7910(-)	26.43(-)/ 0.8080(-)	30.98(-)/ 0.9212(-)
	VDSR [7]	33.66(-)/ 0.9213(-)	29.77(-)/ 0.8314(-)	28.82(-)/ 0.7976(-)	27.14(-)/ 0.8279(-)	32.01(-)/ 0.9310(-)
	DRCN [18]	33.82(-)/ 0.9226(-)	29.76(-)/ 0.8311(-)	28.80(-)/ 0.7963(-)	27.15(-)/ 0.8276(-)	32.31(-)/ 0.9328(-)
	CNF [38]	33.74(-)/ 0.9226(-)	29.90(-)/ 0.8322(-)	28.82(-)/ 0.7980(-)	-	-
	DRRN [18]	34.03(-)/ 0.9244(-)	29.96(-)/ 0.8349(-)	28.95(-)/ 0.8004(-)	27.53(-)/ 0.8378(-)	32.74(-)/ 0.9390(-)
	BTSRN [40]	34.03(-)/-	29.90(-)/-	28.97(-)/-	27.75(-)/-	-
	MemNet [41]	34.09(-)/ 0.9248(-)	30.00(-)/ 0.8350(-)	28.96(-)/ 0.8001(-)	27.56(-)/ 0.8376(-)	32.51(-)/ 0.9369(-)
	SelNet [42]	34.27(-)/ 0.9257(-)	30.30(-)/ 0.8399(-)	28.97(-)/ 0.8025(-)	-	-
	CARN [19]	34.29(-)/ 0.9255(-)	30.29(-)/ 0.8407(-)	29.06(-)/ 0.8034(-)	28.06(-)/ 0.8493(-)	33.50(-)/ 0.9440(-)
	IMDN [49]	34.36(-)/ 0.9270(-)	30.32(-)/ 0.8417(-)	29.09(-)/ 0.8046(-)	28.17(+)/ 0.8519(-)	33.61(+)/ 0.9445(+)
	RAN [43]	33.71(-)/ 0.9223(-)	29.84(-)/ 0.8326(-)	28.84(-)/ 0.7981(-)	-	-
	DNCL [44]	33.95(-)/ 0.9232(-)	29.93(-)/ 0.8340(-)	28.91(-)/ 0.7995(-)	27.27(-)/ 0.8326(-)	-
	FilterNet [45]	34.08(-)/ 0.9250(-)	30.03(-)/ 0.8370(-)	28.95(-)/ 0.8030(-)	27.55(-)/ 0.8380(-)	-
	MRFN [46]	34.21(-)/ 0.9267(-)	30.03(-)/ 0.8363(-)	28.99(-)/ 0.8029(-)	27.53(-)/ 0.8389(-)	32.82(-)/ 0.9396(-)
	SeaNet-baseline [48]	34.36(-)/ 0.9280(+)	30.34(+)/ 0.8428(+)	29.09(-)/ 0.8053(+)	28.17(+)/ 0.8527(+)	33.40(-)/ 0.9444(+)
	DEGREE [47]	33.76(-)/ 0.9211(-)	29.82(-)/ 0.8326(-)	28.74(-)/ 0.7950(-)	-	-
	SCANet (Ours)	34.42/0.9272	30.32/0.8421	29.09/0.8051	28.14/0.8524	33.55/0.9443
$\times 4$	SRCNN [5]	30.48(-)/ 0.8628(-)	27.49(-)/ 0.7503(-)	26.90(-)/ 0.7101(-)	24.52(-)/ 0.7221(-)	27.66(-)/ 0.8505(-)
	FSRCNN [6]	30.71(-)/ 0.8657(-)	27.59(-)/ 0.7535(-)	26.98(-)/ 0.7150(-)	24.62(-)/ 0.7280(-)	27.90(-)/ 0.8517(-)
	VDSR [7]	31.35(-)/ 0.8838(-)	28.01(-)/ 0.7674(-)	27.29(-)/ 0.7251(-)	25.18(-)/ 0.7524(-)	28.83(-)/ 0.8809(-)
	DRCN [18]	31.53(-)/ 0.8854(-)	28.02(-)/ 0.7670(-)	27.23(-)/ 0.7233(-)	25.14(-)/ 0.7510(-)	28.98(-)/ 0.8816(-)
	CNF [38]	31.55(-)/ 0.8856(-)	28.15(-)/ 0.7680(-)	27.32(-)/ 0.7253(-)	-	-
	LapSRN [39]	31.54(-)/ 0.8850(-)	28.19(-)/ 0.7720(-)	27.32(-)/ 0.7280(-)	25.21(-)/ 0.7560(-)	29.09(-)/ 0.8845(-)
	DRRN [18]	31.68(-)/ 0.8888(-)	28.21(-)/ 0.7720(-)	27.38(-)/ 0.7284(-)	25.44(-)/ 0.7638(-)	29.46(-)/ 0.8960(-)
	BTSRN [40]	31.85(-)/-	28.20(-)/-	27.47(-)/-	25.74(-)/-	-
	MemNet [41]	31.74(-)/ 0.8893(-)	28.26(-)/ 0.7723(-)	27.40(-)/ 0.7281(-)	25.50(-)/ 0.7630(-)	29.42(-)/ 0.8942(-)
	SelNet [42]	32.00(-)/ 0.8931(-)	28.49(-)/ 0.7783(-)	27.44(-)/ 0.7325(-)	-	-
	SRDenseNet [23]	32.02(-)/ 0.8934(-)	28.50(-)/ 0.7782(-)	27.53(-)/ 0.7337(-)	26.05(-)/ 0.7819(-)	-
	CARN [19]	32.13(-)/ 0.8937(-)	28.60(-)/ 0.7806(-)	27.58(-)/ 0.7349(-)	26.07(-)/ 0.7837(-)	30.47(-)/ 0.9084(+)
	IMDN [49]	32.21(+)/ 0.8948(+)	28.58(-)/ 0.7811(-)	27.56(-)/ 0.7353(-)	26.04(-)/ 0.7838(-)	30.45(-)/ 0.9075(-)
	RAN [43]	31.43(-)/ 0.8847(-)	28.09(-)/ 0.7691(-)	27.31(-)/ 0.7260(-)	-	-
	DNCL [44]	31.66(-)/ 0.8871(-)	28.23(-)/ 0.7717(-)	27.39(-)/ 0.7282(-)	25.36(-)/ 0.7606(-)	-
	FilterNet [45]	31.74(-)/ 0.8900(-)	28.27(-)/ 0.7730(-)	27.39(-)/ 0.7290(-)	25.53(-)/ 0.7680(-)	-
	MRFN [46]	31.90(-)/ 0.8916(-)	28.31(-)/ 0.7746(-)	27.43(-)/ 0.7309(-)	25.46(-)/ 0.7654(-)	29.57(-)/ 0.8962(-)
	SeaNet-baseline [48]	32.18(-)/ 0.8948(-)	28.61(-)/ 0.7822(+)	27.57(-)/ 0.7359(-)	26.05(-)/ 0.7896(+)	30.44(-)/ 0.9088(+)
	DEGREE [47]	31.47(-)/ 0.8837(-)	28.10(-)/ 0.7669(-)	27.20(-)/ 0.7216(-)	-	-
	SCANet (Ours)	32.20/0.8947	28.61/0.7821	27.58/0.7365	26.17/0.7893	30.50/0.9083

Furthermore, we compare the visualization quality with VDSR [7]. Figure 5 shows the restored images of the Urban100 benchmark with a scaling factor ×4. In the figure, we can see that the proposed SCANet can restore the circles and lines more clearly than other works. Furthermore, the results of SCANet are the closest to the HR images. As such, the PSNR results of SCANet are better than other works. From this point of view, SCANet can effectively restore LR images with more satisfying results.

Figure 5. Visual comparisons against the Urban100 benchmark.

We also investigate the subjective qualities of different methods. To this end, we use the no-reference indicators and measure the visual quality of different methods. The indicators are chosen as NIQE and BRISQUE. Table 3 shows the subjective comparisons. It should be noted that the lower NIQE and BRISQUE values mean a better visual performance. In the figure, it can be seen that SCANet achieves the best performance with the lowest indicator scores. From this point of view, SCANet obtains better visual quality than other works.

Table 3. Subjective quality comparisons with different no-reference image quality indicators.

Indicators	SRCNN	VDSR	SCANet
NIQE ↓	7.17	5.85	5.48
BRISQUE ↓	58.87	41.64	39.51

To demonstrate the significance of our method, we conduct the two-sample t-test on the NIQE results of the three methods. Table 4 shows the results. The figure “0” in the table means there is no significant difference between the two methods; “1” means the upper method outperforms the left method; and “−1 ” means the left method outperforms the upper one. We can see from the table that SCANet performs significantly better than other works, with good visual quality.

Table 4. Subjective quality comparisons with different no-reference image quality indicators.

Methods	SRCNN	VDSR	SCANet
SRCNN	0	1	1
VDSR	−1	0	1
SCANet	−1	−1	0

4.4. Discussion

In this work, we compare our method with others, with a scaling factor of $\times 2$, $\times 3$, and $\times 4$. There are works with much larger scaling factors, such as 16, 32, or even 64 [50,51]. However, these methods only focus on some specific kinds of images, such as facial SR [50]. These methods only work in the specific situation, but cannot be generalized to different applications. Our work is a general SR method that can be directly utilized in various scenarios. The most commonly compared scaling factors are 2, 3, and 4 for general SR works, such as SRCNN [5], LapSRN [39], and CARN [19].

There are also cases of failure for SCANet. SCANet regards image super-resolution as an optimization task, which does not specifically focus on edge restoration. From this point of view, works aiming at structural restoration, such as SeaNet [48], achieve higher SSIM scores on the Urban100 [35] and Manga109 [36] datasets, which contain plentiful edges and lines. In further work, we will consider special edges and structural priors in the optimization step and thus improve restoration performance.

Different from vanilla optimization methods, the deep learning-based optimization method has a fixed number of iteration steps. Convergence is guaranteed by the convex optimization algorithm, and the network adaptively learns the parameters during the training step. Successive convex approximation has proved to be convergent in the previous literature [26,27,28]. The LASSO sub-problem also has a converged solution [52]. From this point of view, SCANet can be converged during iteration steps.

In general, this network provides an effective restoration method for image SR and achieves a better performance than other related works. Furthermore, this network provides a new perspective on combining deep learning and the optimization problem, which can be further investigated for different computer vision tasks.

5. Conclusions

Currently, researchers usually regard image SR as a regression task and use an end-to-end convolutional neural network (CNN) to predict the pixels directly, which lacks inherent theoretical analysis and limits the effectiveness of restoration. In this paper, we propose a deep successive convex approximation network for image super-resolution (SR), dubbed SCANet. In SCANet, the SR task was investigated from an optimization perspective, which was converted into LASSO sub-problems and solved using network blocks. The network blocks were established with residual feature aggregation blocks (RFAB) and the spatial and channel attention (SACA) mechanism for better performance. The experimental results show that the proposed SCANet achieved a better objective performance than other works and restored more satisfying results with higher subjective quality.