Accelerating the Super-Resolution Convolutional Neural Network

Chao Dong, Chen Change Loy, Xiaoou Tang

Department of Information Engineering, The Chinese University of Hong Kong

dongchao@sensetime.com, {ccloy, xtang}@ie.cuhk.edu.hk

 

Abstract

As a successful deep model applied in image super-resolution (SR), the Super-Resolution Convolutional Neural Network (SRCNN) has demonstrated superior performance to the previous hand-crafted models either in speed and restoration quality. However, the high computational cost still hinders it from practical usage that demands real-time performance (24 fps). In this paper, we aim at accelerating the current SRCNN, and propose a compact hourglass-shape CNN structure for faster and better SR. We re-design the SRCNN structure mainly in three aspects. First, we introduce a deconvolution layer at the end of the network, then the mapping is learned directly from the original low-resolution image (without interpolation) to the high-resolution one. Second, we reformulate the mapping layer by shrinking the input feature dimension before mapping and expanding back afterwards. Third, we adopt smaller filter sizes but more mapping layers. The proposed model achieves a speed up of more than 40 times with even superior restoration quality. Further, we present the parameter settings that can achieve real-time performance on a generic CPU while still maintaining good performance. A corresponding transfer strategy is also proposed for fast training and testing across different upscaling factors.

 

From SRCNN to FSRCNN

1

This figure shows the network structures of the SRCNN and FSRCNN. The proposed FSRCNN is different from SRCNN mainly in three aspects. First, FSRCNN adopts the original low-resolution image as input without bicubic interpolation. A deconvolution layer is introduced at the end of the network to perform upsampling. Second, The non-linear mapping step in SRCNN is replaced by three steps in FSRCNN, namely the shrinking, mapping, and expanding step. Third, FSRCNN adopts smaller filter sizes and a deeper network structure. These improvements provide FSRCNN with better performance but lower computational cost than SRCNN.

 

Results

1

The proposed FSRCNN networks could achieve better super-resolution quality than existing methods, and are tens of times faster. Especially, the FSRCNN-s can be implemented in real-time (>24 fps) on a generic CPU. The chart is based on Set14 results summarized in the following table.

 

The results of PSNR (dB) and test time (sec on CPU) on three test datasets. We present the best results reported in the corresponding paper. The proposed FSCNN and FSRCNN-s are trained on both 91-image and General-100 dataset. More comparisons with other methods on PSNR, SSIM and IFC can be found in the supplementary file.

test

dataset

scaling

factor

Bicubic

PSNR / Time

SRCNN [1]

PSNR / Time

SRCNN-Ex [2]

PSNR / Time

SCN [3]

PSNR / Time

FSRCNN-S

PSNR / Time

FSRCNN

PSNR / Time

Set5

Set14

B200

2

2

3

33.66 / -

30.23 / -

29.70 / -

36.34 / 0.18

32.18 / 0.39

31.38 / 0.23

36.66 / 1.3

32.45 / 2.8

31.63 / 1.7

36.93 / 0.94

33.10 / 1.7

31.63 / 1.1

36.58 / 0.024

32.28 / 0.061

31.48 / 0.033

37.00 / 0.068

32.63 / 0.160

31.80 / 0.098

Set5

Set14

B200

3

3

3

30.39 / -

27.54 / -

27.26 / -

32.39 / 0.18

29.00 / 0.39

28.28 / 0.23

32.75 / 1.3

29.30 / 2.8

28.48 / 1.7

33.10 / 1.8

29.41 / 3.6

28.54 / 2.4

32.61 / 0.010

29.13 / 0.023

28.32 / 0.013

33.16 / 0.027

29.43 / 0.061

28.60 / 0.035

Set5

Set14

B200

4

4

4

28.42 / -

26.00 / -

25.97 / -

30.09 / 0.18

27.20 / 0.39

26.73 / 0.23

30.49 / 1.3

27.50 / 2.8

26.92 / 1.7

30.86 / 1.2

27.64 / 2.3

27.02 / 1.4

30.11 / 0.0052

27.19 / 0.0099

26.75 / 0.0072

30.71 / 0.015

27.59 / 0.029

26.98 / 0.019

 

[1] Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: ECCV. (2014) 184–199.

[2] Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. TPAMI 38(2) (2015) 295–307.

[3] Wang, Z., Liu, D., Yang, J., Han, W., Huang, T.: Deeply improved sparse coding for image super-resolution. ICCV (2015) 370–378.

Code and Dataset

Citation

Chao Dong, Chen Change Loy, Xiaoou Tang. Accelerating the Super-Resolution Convolutional Neural Network, in Proceedings of European Conference on Computer Vision (ECCV), 2016