Content-guided deep residual network for single image super-resolution

Optik - International Journal for Light and Electron Optics 202 (2020) 163678

Contents lists available at ScienceDirect

Optik journal homepage: www.elsevier.com/locate/ijleo

Original research article

Content-guided deep residual network for single image superresolution

T

Liangliang Chen, Qiqi Kou, Deqiang Cheng*, Jie Yao School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, 221116, China

A R T IC LE I N F O

ABS TRA CT

Keywords: Super-Resolution Residual network Content-guided Loss function Curvature

We propose a content-guided deep residual network for single image super-resolution. Firstly, the relationship between the sparsity of images and the difficulty of reconstruction through a convolution network is studied. On this basis, a guided residual block is built, which can compensate for part of the high-frequency information and improve the convergence of the network. On the other hand, local loss and curvature loss values are integrated into the loss function, enhancing the network's ability to perceive the advanced texture information of the image. Finally, the optimized loss function is used to the network composed of several guided residual blocks. Experimental results demonstrate that the network achieves superior results over some existing deep residual networks in PSNR and SSIM evaluation.

1. Introduction Image super-resolution reconstruction refers to the technology of recovering high-resolution images from low-resolution images or image sequences. At present, image super-resolution is usually divided into two categories: multiple image super-resolution and single image super-resolution (SISR), the later has been widely used in medical treatment, coal mining, satellite remote sensing and other fields because of its flexibility and simplicity [1–5]. The problem of SISR is to construct the high-resolution image with more high-frequency information from a single low-resolution image, that is, to solve the lost high-frequency information according to prior knowledge. The earlier interpolation method reconstructs the signal based on the theory of signal sampling, which is characterized by fast speed but blurred image edge [6–8].In the reconstruction method, based on the assumption that the LR image is one obtained from a HR image through a series of image transformation processes, the model of inverse transformation is established to transform the problem into finding the optimal estimation of the real image by inputting LR image [9–13]. The complex condition of image degradation different from the ideal one of the model leads to the limited robustness of the method. In recent years, learning-based methods have become a hot topic in SISR research. Sparse coding [14–16], neighborhood embedding [17–19] and example-based methods [20–22] are used to obtain mapping priors between LR and HR images by training datasets. In particular, the outstanding performance of deep learning marks SISR reaching a new level [23–29], such as SRCNN [23], VDSR [25] and ESPCN [26]. Recently, Xu et al. [30] proposed the DFFNet based on DenseNet, in which the hierarchical information is regularly fused to obtain appreciable HR images. Similarly, Zhu et al. [31] utilized the residual channel-wise attention block to adjust the contribution of hierarchical features. Inspired by SRResNet [27] and guided filters [32,33], we propose a content-guided residual network (CGRNet) for SISR. Firstly,

⁎

Corresponding author. E-mail address: [email protected] (D. Cheng).

https://doi.org/10.1016/j.ijleo.2019.163678 Received 12 October 2019; Received in revised form 24 October 2019; Accepted 28 October 2019 0030-4026/ © 2019 Elsevier GmbH. All rights reserved.

Optik - International Journal for Light and Electron Optics 202 (2020) 163678

L. Chen, et al.

Fig. 1. The structure of the GRBlock.

we propose the assumption that the sparser the image is, the better it can be reconstructed by a deep convolution network, and design experiments to verify it. Then, we construct the guided residual block (GRBlock), incorporating the estimated difference as a guide map into the residual network unit in a specific way. Considering that the current loss function mostly adopts the L2 loss limitedly perceiving the features of the high-level texture content [27,28], the curvature feature that can better describe the texture [34–37] is added and combined with the local loss function of CGRNet to obtain a new high-level content-aware loss function. In the paper, our contribution is mainly divided into three parts. Firstly, we propose and verify the qualitative relationship between image sparsity and the convolutional network for SISR. On this basis, the residual block with the guided map is conducted to accelerate convergence and enhance the ability of the network. Secondly, the loss function is optimized by introducing curvature features and local losses to improve the network's perception of texture details. Lastly, the CGRNet is constructed after considering the optimized loss function and the GRBlock. 2. Materials and methods In this section, we describe the guided residual block, the optimized loss function (multi-feature loss function of perceived content) and the final content-guided residual network. 2.1. Guided residual block As shown in Fig. 1, Guided Residual Block (GRBlock) is devised based on the original residual block by referring to the guided filter. The f_dif obtained from the LR image through the estimated difference layer and the extended feature layer is entered into the guidance layer along with the output f_res of the original residual block to obtain the final output of the GRBlock. We assume that the sparser the image is, the easier it is to reconstruct through the convolution network. The sparsity [38] of the image is calculated is:

sparsity (X ) =

n − ( ∑ |x i|)/ ∑ x i2 (1)

n −1

where n represents the dimension and xi is the i-th data of the matrix X. Table 1 presents the mean sparsity of the datasets (Set5 and Set14) and their difference images between HR and LR images. The difference images are such in Fig. 2. According to the contents of Set5 and Set14 in Table 1 and Table 2, we find that the overall PSNR results are consistent with our hypothesis, and the existence of some images different from the hypothesis does not affect the overall result, so we choose the difference image that could be better reconstructed rather than the complete image as the guide image. As can be seen from Fig. 1, the estimated difference layer is used to predict the difference between the LR and the HR image, which mainly consists of three blocks composed of two convolution layers and an activation layer. Table 1 Sparsity of the datasets and the difference images. Scale

2 (LR/Res)

3 (LR/Res)

4 (LR/Res)

Set5 Set14

0.1109/0.4120 0.0763/0.4074

0.1086/0.4261 0.0741/0.4098

0.1064/0.3896 0.0719/0.3969

2

Optik - International Journal for Light and Electron Optics 202 (2020) 163678

L. Chen, et al.

Fig. 2. Difference images between HR and LR (butterfly, woman, scale 4).

Table 2 PSNR and SSIM obtained by different networks on different datasets. Methods

scale

Set5

Set14

B100

Urban100

DIV2K

Bicubic A+ SRCNN VDSR EDSR(base) RDN DDBPN CGRNet Bicubic A+ SRCNN VDSR EDSR(base) RDN DDBPN CGRNet Bicubic A+ SRCNN VDSR EDSR(base) RDN DDBPN CGRNet

2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4

33.66/0.9299 36.54/0.9544 36.66/0.9542 37.53/0.9587 38.00/0.9590 38.24/0.9614 38.09/0.960 38.68/0.9600 30.39/0.8682 32.58/0.9088 32.75/0.9090 33.66/0.9213 34.45/0.9263 34.71/0.9296 -/34.77/0.9290 28.42/0.8104 30.28/0.8603 30.48/0.8628 31.35/0.8838 32.17/0.8924 32.47/0.8990 32.47/0.898 32.53/0.9053

30.24/0.8688 32.28/0.9056 32.42/0.9063 33.03/0.9124 33.55/0.9189 34.01/0.9212 33.85/0.919 34.02/0.9457 27.55/0.7742 29.13/0.8188 29.28/0.8209 29.77/0.8314 30.38/0.8449 30.57/0.8468 -/30.59/0.8665 26.00/0.7027 27.32/0.7491 27.49/0.7503 28.01/0.7674 28.61/0.7868 28.81/0.7871 28.82/0.786 29.21/0.7901

29.56/0.8431 31.21/0.8863 31.36/0.8879 31.90/0.8960 32.16/0.8924 32.34/0.9017 32.27/0.900 32.74/0.9077 24.46/0.7349 28.29/0.7835 28.41/0.7863 28.82/0.7976 29.12/0.8047 29.26/0.8093 -/29.85/0.8203 23.14/0.6577 26.82/0.7087 26.90/0.7101 27.29/0.7251 27.59/0.7319 27.72/0.7419 27.72/0.740 28.00/0.7501

26.88/0.8403 29.20/0.8938 29.50/0.8946 30.76/0.9140 31.89/0.8960 32.89/0.9353 33.02/32.32/0.9144 24.46/0.7349 26.03/0.7903 26.24/0.7989 27.14/0.8278 28.13/0.8640 28.80/0.8653 -/28.78/0.8799 23.14/0.6577 24.32/0.7183 24.52/0.7221 25.18/0.7524 26.01/0.7983 26.61/0.8028 27.08/26.71/0.8202

31.01/0.9393 32.89/0.9570 33.05/0.9581 33.66/0.9625 34.59/0.9669 -/-/35.31/0.9700 28.22/0.8906 29.50/0.9116 29.64/0.9138 30.09/0.9208 30.95/0.9296 -/-/31.43/0.9379 26.66/0.8521 27.70/0.8739 27.78/0.8753 28.17/0.8841 28.97/0.8963 -/-/29.57/0.9217

The extended feature layer shares the feature convolution layer of the residual network (shown in Fig.1), so as to achieve the same decomposition of the estimated difference as the residual block. The guidance layer is used to generate more efficient feature channels. The feature f_tot recombined by f_dif and f_res is input into a number of convolution layers and activation layers to obtain the guidance features.

f _tot = F (f _dif , f _res ) = [(f _difi , f _resi )i = 1,2, ⋯ , n − 1, n] f _dif = [f _dif1 , f _dif2 , ..., f _difn ] f _res = [f _res1, f _res2, ..., f _resn]

(2)

2.2. Multi-feature loss function of perceived content The loss function of current deep networks mostly adopts mean square deviation. Although the mean square deviation can evaluate the difference between SR and HR image, it only stays on the perception of pixel level, while the texture details between SR image and HR image cannot be well perceived. Therefore, we propose the loss function LossSR on image content features, namely multi-feature loss function of perceived content. The LossSR includes the content loss function Losscontent (acting on the global network) and the local optimization loss function LossLocal (acting on the GRBlock), calculated by (3)

LossSR = Losscontent + LossLocal

If the gradient of a two-dimensional image is taken as the third dimension, it can be found that the texture information of the image is the gradient change information. The curvature can better represent the gradient change of the image [34,37], so the curvature features of the image are extracted and the Euclidean distance of curvature features is calculated as the loss function lossgrad to describe the gradient information. According to the calculation formula of curvature in [39], it can be simplified to: 3

Optik - International Journal for Light and Electron Optics 202 (2020) 163678

L. Chen, et al.

curv (I ) = div (∇I /|∇I|) =

Iyy Ix2 − 2Ixy Ix Iy + Ixx I y2 (Ix2 + I y2)3/2

(4)

where Ix and Iy are the first derivative of the image I in horizontal and vertical directions respectively, Ixx and Iyy are the second derivative in the vertical and horizontal directions. Then the lossgrad is calculated by:

lossgrad = D (curv (HR), curv (SR))

(5)

where D(⋅) represents the Euclidean distance, HR is the high-resolution image, and SR is the reconstructed image. Meanwhile, the lossres focuses on describing the difference of curvature features between the reconstructed residual information and the real one, calculated by: (6)

lossres = D (cur (Ct (Res )), cur (HR − LR))

where Res represents the reconstructed residual information, LR is the low-resolution image after interpolation, and Ch (⋅) means convolution operation of the convolution layer Conv_h. The Losscontent is calculated as follows:

Losscontent = (lossmse + lossgrad + lossres )/3

(7)

where lossmse is the loss value of the SR calculated by the L1 loss function. On the other hand, the local loss LossLocal describes the loss of the output of each residual block which is used to optimize each residual block, so as to achieve the optimization of the overall network. The LossLocal is obtained as follows: n

LossLocal = ∑i = 1 lossi lossi = D (cur (PREi ), cur (HR)) PREi = Ct (Feati + Ch (LR))

(8)

where n represents the number of residual blocks, Feati is the output of the i-th residual block in the network, Ct (⋅) means convolution operation of the convolution layer Conv_t, as shown in Fig. 3. Finally, the loss function LossSR of the whole network is optimized to better perceive the texture details of the image. 2.3. Content-Guided residual network In the previous section, GRBlock and multi-feature loss function of perceived content are introduced. To integrate them and leverage their strengths, a complete network is built, named the content-guided enhancement residuals network (CGRNet). The structure is shown in Fig. 3. The network replaces the original residual block with the GRBlock, keeping the same parameter settings as SRResNet, such as the GRBlocks B = 16, the filters F = 64. Differently, our network uses the proposed loss function (introduced in Section 2.2) for training instead of the L2 loss. Otherwise, the PixShuffle layer is removed because the input of the network is the image after interpolation. 3. Experimental results 3.1. Experiment settings and datasets For training, we use the RGB input patches of size 96 × 96 from the LR images which are obtained by performing Bicubic interpolation on training images of DIV2K dataset [40]. ADAM is chosen as the optimizer, its setting β1 = 0.9, β2 = 0.999, ε = 10−8 and initial learning rate 0.1. The experimental platform is the framework of Pytorch 1.0 on the device with some important hardware:

LossSR loss1 loss2

loss3 loss4

SR

Conv_t

Add

GRBlock

Losscontent

GRBlock

GRBlock

GRBlock

Conv_h

LR

Conv_t Add

Fig. 3. The whole structure of the proposed network. 4

Optik - International Journal for Light and Electron Optics 202 (2020) 163678

L. Chen, et al.

Fig. 4. PSNR and SSIM of baby.png reconstructed by different models (scale = 4). (a)Bicubic /30.45/0.8442 (b)A+/31.84/0.9477 (c)SRCNN/ 33.13/0.9501 (d)VDSR/33.42/ 0.9500 (e) EDSR/ 33.74/0.9528 (f)RDN/33.84/0.9536 (g)DDBPN/33.82/0.9534 and (h)CGRNet/33.86/0.9537.

the GPU is GTX1080Ti, the CPU is i7-7800X with 32GB RAM. In order to verify the effectiveness of the proposed method, we perform the model CGRNet on Set5 [41], Set14 [15], B100 [42], Urban100 [43] and DIV2K public datasets and compare it with the methods of Bicubic, A+ [44], SRCNN [23], VDSR [25] and EDSR (baseline) [28]. The up-sampling scale is 2, 3 and 4, respectively. Furthermore, RDN [45] and DDBPN [46], two models that show excellent competitiveness in the models based on dense connections, are also included in the comparison with our work. 3.2. Comparison with other methods The CGRNet is introduced in Section 2.3. PSNR and SSIM are calculated for the reconstructed images, as shown in Table 2. In particular, we test the baseline structure of the EDSR network with 16 residual blocks, 64 filters and L1 loss function. In order to compare with other methods, the results are only derived from the calculation of the Y channel in YCbCr where a certain number of boundary pixels (equal to the corresponding scale factor) are ignored. As can be seen from Table 2, our network obtains the optimal results of PSNR and SSIM on most datasets compared with other methods. In addition, the clearer texture and the enhanced edge of the contour indicate that the quality of the reconstructed images is also improved in subjective vision, as shown in Figs. 4–7. However, since the estimated difference is introduced into the generated feature maps of the network, the ambiguity information inconsistent with the HR image is added to the reconstructed image, which leads to the result that the improvement of PSNR and SSIM is not particularly obvious. 3.3. Analysis of guided residual block In order to further compare the effect with the traditional residual blocks, the guided residual block is respectively integrated into VDSR, SRResNet [27] and EDSR(baseline) to obtain new networks denoted as VDSR + GRBlock, SRResNet + GRBlock and EDSR + GRBlock. The test is performed on the Urban100 dataset with the scale factor of 4. The parameters of these networks remain unchanged to facilitate the comparison of the improved PSNR of reconstructed images. Their results are presented in Table 3. According to Table 3, it can improve the result by about 0.3 dB that the traditional residual block is replaced with the GRBlock in these networks, which objectively proves that the GRBlock can enhance the capacity of the residual network for SISR. In addition, considering that EDSR (baseline) is a more optimized model than VDSR and SRResNet, and the GRBlock is an enhanced version of the traditional residual block in EDSR(baseline), we separately compare the loss and accuracy curves between EDSR(baseline)+GRBlock and EDSR (baseline). Fig. 8 shows the loss and accuracy value of EDSR (baseline) with and without the GRBlock in the training. It can be seen that the former network has significantly better convergence and higher accuracy. Combined with the results in Table 3, the GRBlock can be interpreted as expressing features better than traditional residual.

Fig. 5. PSNR and SSIM of head.png reconstructed by different models (scale = 4). (a)Bicubic/ 30.28/0.8154 (b)A+/31.19/0.8789 (c)SRCNN/ 32.44/0.9148 (d) VDSR/ 32.73/0.8848 (e) EDSR/ 32.97/0.8895 (f)RDN/33.00/0.8911 (g)DDBPN/33.02/0.8911 and (h)CGRNet/33.04/0.8926. 5

Optik - International Journal for Light and Electron Optics 202 (2020) 163678

L. Chen, et al.

Fig. 6. PSNR and SSIM of monarch.png reconstructed by different models (scale = 4). (a)Bicubic /26.14/0.9199 (b)A+/28.11/0.9686 (c)SRCNN/ 30.22/0.9745 (d) VDSR/ 31.60/0.9800 (e) EDSR /32.44/0.9834 (f)RDN/33.33/0.9859 (g)DDBPN/33.54/0.9856 and (h) CGRNet/33.61/0.9865.

Fig. 7. PSNR and SSIM of img098.png reconstructed by different models (scale = 4). (a)Bicubic/ 18.40/0.5257 (b)A+/18.79/0.5534 (c)SRCNN/ 20.09/0.5689 (d) VDSR/20.29/0.5646 (e) EDSR/ 20.54/0.5777 (f)RDN/20.89/0.5824 (g)DDBPN/20.80/0.5825 and (h)CGRNet/20.89/0.5833.

Table 3 PSNR and SSIM for the residual networks with and without GRBlock. Method

VDSR

SRResNet

EDSR(baseline)

Orignal +GRBlock

25.18/0.7524 25.51/0.7674

26.07/0.7839 26.25/0.7941

26.01/0.7983 26.49/0.8043

3.4. Analysis of loss function Different loss functions also have a significant impact on the ability of the network. In order to analyze the impact of loss functions and the optimization of the proposed function on the network, L2, L1, Losslocal + L1, and Losslocal+Losscontent are adopted as the loss function in VDSR, SRResNet and EDSR (baseline) respectively, other parameters remain unchanged. The test is performed on the Urban100 dataset with the scale factor of 4, their results shown in Table 4. Table 4 shows the ascending order of these loss functions: L2, L1, Losslocal+L1 and Losslocal+Losscontent, which indicates that the choice of loss function has an effect on the capacity of the network. Among those loss functions, Losslocal+Losscontent contributes the most to the performance of the network, which means that increasing the attention of learning the missing texture information can improve the reconstruction ability of the network. 3.5. Discussion The merits and limitations of our work are discussed. The competitiveness of our work is reflected in objective evaluation and model complexity. In terms of objective evaluation, according to the data shown in Table 2, PSNR and SSIM results are superior to some popular models which includes some ones based on the residual network and dense connection, such as VDSR [25], EDSR (baseline) [28], RDN [45] and DDBPN [46]. In addition, our work can still achieve competitive results when compared with the latest model DFFNet [30] and SUSR [31]. The PSNR and SSIM results for DFFNet, SUSR and our model CGRNet in the SISR task are shown in Table 5, which intuitively show that our model works best of the three. In terms of model complexity, the number of parameters and computation time are taken into account. In Table 6 are the details of the different models working on Set5 dataset with an up-sampling factor of 4, including the number of model parameters, calculation time in CPU mode and PSNR. Although the number of parameters in our model is more than other models based on residual network, it is far less than the number of parameters in RDN and DDBPN. In terms of runtime, our model is superior to other ones except EDSR (baseline). Some limitations also exist in our model. On the one hand, compared with the residual network, our model improves the superresolution reconstruction capability, but also increases the model parameters. On the other hand, new noise is introduced in the process of difference image guided fusion in the model, which greatly limits the improvement of image quality. 4. Conclusion In this paper, we propose the content-guided deep residual network for SISR. The guided residual block is devised to make full use of the estimated guidance graph to supplement the texture details that are not present in the LR image and reduce the convergence 6

Optik - International Journal for Light and Electron Optics 202 (2020) 163678

L. Chen, et al.

Fig. 8. The curves of the networks with and without GRBlock on DIV2K. Left is the loss curve, and right is the accuracy curve. The scale factors of the results from top to bottom are equal to 2, 3, and 4, respectively. Table 4 PSNR and SSIM for the residual networks with different loss functions. Function

L2

L1

Losslocal+L1

Losslocal+losscontent

VDSR SRResNet EDSR(baseline)

25.18/0.7524 26.07/0.7839 26.59/0.7968

25.19/0.7532 26.09/0.7857 26.01/0.7983

25.19/0.7532 26.12/0.7901 26.08/0.8097

25.30/0.7737 26.21/0.8021 26.20/0.8188

Table 5 The comparison of two new networks and the proposed network (PSNR/SSIM). Models

Scale

Set5

Set14

B100

Urban100

DFFNet SUSR CGRNet DFFNet SUSR CGRNet DFFNet SUSR CGRNet

2 2 2 3 3 3 4 4 4

38.13/0.961 37.83/0.959 38.68/0.960 34.58/0.927 34.25/0.926 34.77/0.929 32.44/0.895 32.05/0.893 32.53/0.905

33.62/0.918 33.62/0.917 34.02/0.946 30.32/0.841 30.28/0.841 30.59/0.867 28.65/0.781 28.57/0.781 29.21/0.790

32.29/0.900 32.15/0.899 32.74/0.908 29.21/0.806 29.04/0.805 29.85/0.820 27.76/0.738 27.55/0.735 28.00/0.750

32.32/0.930 32.19/0.929 32.32/0.914 28.25/0.855 28.05/0.850 28.78/0.880 26.20/0.789 26.01/0.782 26.71/0.820

7

Optik - International Journal for Light and Electron Optics 202 (2020) 163678

L. Chen, et al.

Table 6 Parameters, Time and PSNR of different networks working on Set5. Model

SRCNN

SRResNet

VDSR

EDSR(base)

RDN

DDBPN

CGRNet

Param Time(s) PSNR(dB)

57K 31.60 28.42

1543K 11.12 30.48

665K 15.22 31.35

1515K 4.18 32.17

22271K 29.59 32.47

10430K 91.11 32.47

3800K 8.07 32.53

loss of the network. Content loss and local loss based on image curvature features are used to enhance the loss function of the network. Experiments show that the proposed network can generate competitive results compared with other state-of-the-art residual networks. Our future work will continue in the direction that further optimization of the guided network and the content-based loss function can improve the ability of deep networks for SISR. Funding National Natural Science Foundation of China (51774281); National Key Research and Development Project of China (2018YFC0808302). Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29]

K. Hayat, Multimedia super-resolution via deep learning: a survey, Digit. Signal Process. 81 (2018) 198–217. Z. Huang, Y. Zhang, Q. Li, T. Zhang, N. Sang, Spatially adaptive denoising for X-ray angiogram image, Biomed. Signal Process. Control 40 (2018) 131–139. C. Kwan, Remote sensing performance enhancement in hyperspectral images, Sensors 18 (2018) 3598. Z. Huang, Y. Zhang, Q. Li, T. Zhang, N. Sang, H. Hong, Progressive dual-domain filter for enhancing and denoising optical remote-sensing images, IEEE Geosci. Remote Sens. Lett. 15 (2018) 759–763. Z. Huang, L. Chen, Y. Zhang, Z. Yu, H. Fang, T. Zhang, Robust contact-point detection from pantograph-catenary infrared images by employing horizontalvertical enhancement operator, Infrared Phys. Technol. 101 (2019) 146–155. Xin Li, M.T. Orchard, New edge-directed interpolation, IEEE Trans. Image Process. 10 (2001) 1521–1527. C.E. Duchon, Lanczos filtering in one and Two dimensions, J. appl. meteor 18 (1979) 1016–1022. R. Keys, Cubic convolution interpolation for digital image processing, IEEE Trans. Acoust. Speech Signal Process. 29 (1981) 1153–1160. A.M. Tekalp, M.K. Ozkan, M.I. Sezan, High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration, Speech, and Signal Processing, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, 3 1992, pp. 169–172. M. Irani, S. Peleg, Improving resolution by image registration, Cvgip Graph. Models Image Process. 53 (1991) 231–239. M. Elad, A. Feuer, Super-resolution restoration of an image sequence: adaptive filtering approach, IEEE Trans. Image Process. 8 (1999) 387–395. Z. Huang, L. Huang, Q. Li, T. Zhang, N. Sang, Framelet regularization for uneven intensity correction of color images with illumination and reflectance estimation, Neurocomputing 314 (2018) 154–168. Z. Huang, H. Fang, Q. Li, Z. Li, T. Zhang, N. Sang, Y. Li, Optical remote sensing image enhancement with weak structure preservation via spatially adaptive gamma correction, Infrared Phys. Technol. 94 (2018) 38–47. Jianchao Yang, J. Wright, T. Huang, Yi Ma, Image super-resolution as sparse representation of raw image patches, IEEE Conference on Computer Vision and Pattern Recognition, 2008 2008, pp. 1–8. R. Zeyde, M. Elad, M. Protter, On single image scale-up using sparse-representations, Proceedings of the 7th International Conference on Curves and Surfaces, Springer Verlag, 2012, pp. 711–730. Z. Huang, Q. Li, T. Zhang, N. Sang, H. Hong, Iterative weighted sparse representation for X-ray cardiovascular angiogram image denoising over learned dictionary, IET Image Process. 12 (2018) 254–261. K. Su, Q. Tian, Q. Xue, N. Sebe, J. Ma, Neighborhood issue in single-frame image super-resolution, IEEE International Conference on Multimedia and Expo, 2005 2005, pp. 1122–1125. J. Jiang, X. Ma, C. Chen, T. Lu, Z. Wang, J. Ma, Single image super-Resolution via locally regularized anchored neighborhood regression and nonlocal means, IEEE Trans. Multimed. 19 (2017) 15–26. D. Mishra, B. Majhi, P.K. Sa, R. Dash, Development of robust neighbor embedding based super-resolution scheme, Neurocomputing 202 (2016) 49–66. W.T. Freeman, T.R. Jones, E.C. Pasztor, Example-based super-resolution, IEEE Comput. Graph. Appl. 22 (2002) 56–65. C. Kim, K. Choi, J.B. Ra, Example-based super-Resolution via structure analysis of patches, IEEE Signal Process. Lett. 20 (2013) 407–410. D. Trinh, M. Luong, F. Dibos, J. Rocchisani, C. Pham, T.Q. Nguyen, Novel example-based method for super-Resolution and denoising of medical images, IEEE Trans. Image Process. 23 (2014) 1882–1895. C. Dong, C.C. Loy, K. He, X. Tang, Image super-Resolution using deep convolutional networks, IEEE Trans. Pattern Anal. Mach. Intell. 38 (2016) 295–307. C. Dong, C.C. Loy, X. Tang, Accelerating the super-Resolution convolutional neural network, in computer vision – ECCV, in: B. Leibe, J. Matas, N. Sebe, M. Welling (Eds.), Springer International Publishing, 2016 2016, pp. 391–407. J. Kim, J.K. Lee, K.M. Lee, Accurate image super-Resolution using very deep convolutional networks, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 2016, pp. 1646–1654. W. Shi, J. Caballero, F. Huszár, J. Totz, A.P. Aitken, R. Bishop, D. Rueckert, Z. Wang, Real-time single image and video super-Resolution using an efficient subpixel convolutional neural network, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 2016, pp. 1874–1883. C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, W. Shi, Photo-realistic single image super-Resolution using a generative adversarial network, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017 2017, pp. 105–114. B. Lim, S. Son, H. Kim, S. Nah, K.M. Lee, Enhanced deep residual networks for single image super-Resolution, IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 2017, pp. 1132–1140. Z. Huang, Y. Zhang, Q. Li, Z. Li, T. Zhang, N. Sang, S. Xiong, Unidirectional variation and deep CNN denoiser priors for simultaneously destriping and denoising optical remote sensing images, Int. J. Remote Sens. 40 (2019) 5737–5748.

8

Optik - International Journal for Light and Electron Optics 202 (2020) 163678

L. Chen, et al.

[30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46]

W. Xu, R. Chen, B. Huang, X. Zhang, C. Liu, Single image super-Resolution based on global dense feature fusion convolutional network, Sensors 19 (2019) 316. L. Zhu, S. Zhan, H. Zhang, Stacked U-shape networks with channel-wise attention for image super-resolution, Neurocomputing 345 (2019) 58–66. K. He, J. Sun, X. Tang, Guided image filtering, IEEE Trans. Pattern Anal. Mach. Intell. 35 (2013) 1397–1409. K. He, J. Sun, Fast guided filter, Comput. Sci. (2015). Q. Kou, D. Cheng, L. Chen, K. Zhao, A multiresolution gray-scale and rotation invariant descriptor for texture classification, IEEE Access 6 (2018) 30691–30701. J.J. Koenderink, A.J. van Doorn, Surface shape and curvature scales, Image Vis. Comput. 10 (1992) 557–564. K. Hanbay, N. Alpaslan, M.F. Talu, D. Hanbay, Principal curvatures based rotation invariant algorithms for efficient texture classification, Neurocomputing 199 (2016) 77–89. Q. Kou, D. Cheng, H. Zhuang, R. Gao, Cross-complementary local binary pattern for robust texture classification, IEEE Signal Process. Lett. 26 (2019) 129–133. P.O. Hoyer, Non-negative matrix factorization with sparseness constraints, J Mach Learn Res 5 (2004) 1457–1469. R. Malladi, J.A. Sethian, Image processing: flows under Min/Max curvature and mean curvature, Graph. Models Image Process. 58 (1996) 127–141. E. Agustsson, R. Timofte, NTIRE 2017 challenge on single image super-Resolution: dataset and study, IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 2017, pp. 1122–1131. M. Bevilacqua, A. Roumy, C. Guillemot, M.-L. Alberi Morel, Low-complexity single-image super-Resolution based on nonnegative neighbor embedding, British Machine Vision Conference (BMVC), 135 2012, pp. 1–10. D. Martin, C. Fowlkes, D. Tal, J. Malik, A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics, Proceedings Eighth IEEE International Conference on Computer Vision(ICCV), 2 2001, pp. 416–423. J. Huang, A. Singh, N. Ahuja, Single image super-resolution from transformed self-exemplars, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015 2015, pp. 5197–5206. R. Timofte, V. De Smet, L. Van Gool, A+: adjusted anchored neighborhood regression for fast super-Resolution, in: D. Cremers, I. Reid, H. Saito, M.-H. Yang (Eds.), Computer Vision – ACCV, 2015 Springer International Publishing, 2014, pp. 111–126. Y. Zhang, Y. Tian, Y. Kong, B. Zhong, Y. Fu, Residual dense network for image super-Resolution, in arXiv:1802.08797 (2018). M. Haris, G. Shakhnarovich, N. Ukita, Deep Back-projection networks for super-Resolution, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018 2018, pp. 1664–1673.

9