An adaptive super-resolution method based on regional pixel information and ringing artifacts suppression

Optik 125 (2014) 5962–5968

Contents lists available at ScienceDirect

Optik journal homepage: www.elsevier.de/ijleo

An adaptive super-resolution method based on regional pixel information and ringing artifacts suppression Xin Yang a,b,∗ , Tianshu Liu b , Dake Zhou a,b a b

College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, Jiangsu 210016, China Department of Mechanical and Aeronautical Engineering, Western Michigan University, Kalamazoo, MI 49008, USA

a r t i c l e

i n f o

Article history: Received 21 October 2013 Accepted 31 May 2014 Keywords: Super-resolution Image enhancement Regularization Ringing artifacts

a b s t r a c t Multi-frame image super-resolution (SR) aims to utilize information from a set of low-resolution (LR) images to compose a high-resolution (HR) one. In this paper, a novel multi-frame image super-resolution algorithm is proposed based on regional pixel information and ringing artifacts suppression. Firstly, a new regularization term which adopts Regional Adaptive Weight Coefficients (RAWC) is produced to keep edges and flat regions. After detailed analysis, an iterative process is given for image reconstruction. Then an adaptive term according to the local variance of iterative correction image is designed to evaluate the ringing artifacts. Finally, the original iteration is updated by adding the restraint term for better visual effects and lower noise of reconstructive HR image. Thorough experimental results show the proposed algorithm is effective for SR reconstruction and ringing artifacts suppression. © 2014 Elsevier GmbH. All rights reserved.

1. Introduction High-resolution (HR) image is always desirable in almost all situations, but it is difficult to obtain because of expensive cost on hardware devices. So super resolution (SR) [1], which adopts mathematical algorithm to obtain HR with low cost, has been one of the most research hot spot in image processing and computer vision. The primary concept of SR is to improve image resolution by utilizing the redundancy information between the low resolution (LR) images, as well as the prior information of the original image and high frequency information which has been lost during image collecting process. In view of its effectiveness and practicability, the SR technique has been extensively studied and used in several areas such as medical imaging, remote sensing, video surveillance, video delivery, image processing and computer vision. In the past decades, many SR algorithms have been put forward. Common typical algorithms of them are as follows: The frequency domain approach, which starts early and implements simply, reconstructs an HR image by removing the aliasing which exists in LR images. This approach was derived by Tsai and Huang [2] firstly. They described the aliasing relationship of LR images and desired HR image in DFT (discrete Fourier transform)

∗ Corresponding author at: College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, Jiangsu 210016, China. E-mail address: [email protected] (X. Yang). http://dx.doi.org/10.1016/j.ijleo.2014.07.039 0030-4026/© 2014 Elsevier GmbH. All rights reserved.

and CFT (continuous Fourier transform), according to the relative motion characteristics and aliasing characteristics between LR images. In this post-study survey, spatial blur and observation noise aroused the attention from Kim [3], who provided a weighted least squares formulation and iterative algorithm to reconstruct HR image. The approach improves the reconstruction effect but instability because of the fuzzy kernel function. After that, an adaptively fuzzy kernel function method was provided by Kim and Su [4], which is based on a assumption that all LR images have the same blur and noise characteristics. Theoretical simplicity and low computational complexity are major advantages of the frequency domain approach, but the limitation on observation model and scanty priori knowledge restricts the development of frequency domain approach severely. As a widely used and researched method, MAP SR algorithm provides a flexible way to model priori knowledge. The concept of MAP is to achieve the maximum posteriori probability of HR image under LR images. MAP was firstly introduced into single image SR algorithm, which based on Markova prior model, by Schultz and Stevenson [5]. After that, MAP was spread to video SR field by Schultz [6]. To solve the ill-posed problem effectively, the Tikhonov regularization method was adopted by Nguyen [7]. Furthermore, an adaptively regularization parameters method was proposed in [8]. Robustness in iterative convergence process and flexibility in applying priori knowledge are the major advantage of MAP method. But on the other hand, it is hard to implement and has a heavily cost in calculation. MAP also causes blurring of the edges of features.

X. Yang et al. / Optik 125 (2014) 5962–5968

The basic idea for POCS algorithm is that various closed convex sets are projected onto the solution space. Convex sets could be denoted by data reliability, energy boundedness and positive definitiveness. This method was provided by Stark and Oskoui [9] firstly. In their research, the blur factor of sensors was considered. In the following, better reconstruction image was obtained by considering the sensor noise, motion blur factor and space varying blur in the paper from patti [10]. POCS algorithm is widely applied because of its powerful ability in utilizing the spatial domain observation model and applying the prior knowledge. However, the reconstruction results depending on the initial estimate seriously. Besides that, the convergence speed and stability for iteration have yet to be improved. Iterative back projection approach (IBP) [11]: In this kind of algorithm, HR image is estimated by projecting the error projection matrix onto the initial estimated value with the iterative process. The matrix is obtained by calculating the difference between simulated LR images and real LR images. Nevertheless, the IBP algorithm suffers from ringing artifacts around the edge of image because of the accumulation of isotropic reconstruction errors and noises. Besides that, the lack of control of the error projection matrix exacerbates the problem. To enforce the visual effect of the reconstructed image and reduce ringing artifacts, researchers have proposed various improved algorithms. For instance, in paper [12], Song et al. employ a revised term so as to penalize the superabundance high-frequency components, while the high-frequency information is gained though the second-order differentia. However, this algorithm does not take into account the effects of error projection matrix. Many other methods also have the problem of ringing artifacts. So how to suppress ringing artifacts is one of the key points in SR technology. In literature [13], high order interpolation was drawn into modified convex sets to restrain ringing artifacts. The point spread function (PSF) was improved based on edge-characteristic in literature [14]. Our research objective is to suppress ringing artifacts effectively and improve the performance of reconstructive image. So in this paper, a new algorithm based on regional pixel information and ringing artifacts suppression is proposed to address multi-frame image SR reconstruction problem. A new regularization term using Regional Adaptive Weight Coefficients (RAWC) is presented to keep edges and flat regions. This new regularization term can be seen as the great improvement of the BTV regularizer. Then an adaptive analysis is designed based on local variance variation of correction image. At last, a restraint term is added into the iterative reconstruction process for suppress the ringing artifacts. The rest of the paper is organized as follows. The imaging model of SR is described in the next section. Section 3 proposes an image SR method using a novel adaptive fidelity term and regularization term. In Section 4, detailed experimental results are presented and discussed. Finally, we conclude this paper.

2. Observation model Let the underlying HR image be denoted in the vector form by X = [x1 , x2 , . . ., xN ]T , where N is the HR image size and N = L1 N1 × L2 N2 , Letting L1 and L2 denote the down-sampling factors in the horizontal and vertical directions, respectively, each observed LR image having the size N1 × N2 . Thus, the LR image can be represented as Yk = [yk,1 , yk,2 , . . ., yk,M ]T , where M = N1 × N2 and k = 1,2,. . .,P with P being the number of the LR images. As one important step for super resolution, the model of imaging degradation should be constructed at first. It is assumed that k LR images Yk are obtained from the HR image X via acquisition process, which is the set of motion transform, Gaussian blur, sub-sampling

5963

by pixel and additive Gaussian noise. The model can be written as Eq. (1) Yk = DBk Fk X + nk

1≤k≤P

(1)

where Fk is motion matrix for modeling the motion degradation process of the kth LR image, which can be calculated by image registration algorithm, Bk is a N × N blurring matrix, usually adoption with low pass filter operator, D is a M × N sub-sampling matrix, and nk represents the M × 1 zero mean Gaussian noise vector. 3. SR reconstruction based on regional pixel information 3.1. Proposed model The observation model defined in Eq. (1) describes the direct LR image acquisition process by an imaging degradation system. According to Eq. (1), the corresponding HR image from observed LR images can be estimated, and this process is termed as the superresolution. However, the operators Bk , related to the point spread functions, are derived from the discretization of compact operator, so SR process is ill-posed. Thus, even small changes in LR images can result in large perturbations in the final solution and there exist an infinite number of solutions when (1) is solved directly. Therefore, regularization technique is necessarily applied in SR to well pose this problem. A specific regularization (X) is always imposed on the observation model. The regularization (X) can incorporate prior knowledge of the desirable HR solution, e.g., degree of smoothness. So, additional constraints that favor well-behaved behaved solutions can be enforced by specific regularization to remove artifacts from final result. Accordingly, SR process can be converted to a generalized minimization cost function [15] i.e., min J(X),

J(X) =

P 

(DBk Fk X − Yk ) + (X),

(2)

k=1

where  is the Lagrangian constant coefficient, and  is the distance between the observation and an estimation. According to Eq. (2), following formula is obtained as: Xˆ = arg min

 P 



(DBk Fk X − Yk ) + (X)

,

(3)

k=1

where Xˆ is the unknown high-resolution image to be estimated.  the Lagrangian constant coefficient. Let F(X) =

P 

(DBk Fk X − Yk ). F(X) is defined as the fidelity

k=1

term, which measures the closeness of an estimated HR image to the captured LR images. The term (X), called the regularization term, is utilized to regularize the problem and to achieve a stable solution to the problem. So Eq. (3) could be written as Xˆ = arg min{F(X) + (X)},

(4)

Usually, the fidelity term F(X) used in Eq. (4) is defined by the Lp norm of the residual, i.e., F(X) = ||DBk Fk X − Yk ||LL

(L = 1 or 2)

(5)

According to [15], L1 norm leads to a more robust result in error estimation but L2 norm results in better SR resolution. In this paper, the L2 norm is adopted. 3.2. Adaptive bilateral edge-preserving regularization term The main role of regularization term is to solve the ill-posed problem, control the perturbation of the solution and guarantee a

5964

X. Yang et al. / Optik 125 (2014) 5962–5968

stable HR estimation. Farsiu et al. [15] proposed a novel regularization term using bilateral total variation model by combining the total variation and the bilateral filter, BTV (X) =

w w  





˛|m|+|l| X − Sxl Sym X  ,

Window

2T+1

(6)

l=−w m=0

where l + m ≥ 0, Sxl and Sym are shift matrices to present l and m pixels shift in horizontal and vertical directions, respectively, and a ˛ (0 < ˛ < 1) is the weighting coefficient. The weighting coefficient ˛ affects the estimated HR image significantly. Small ˛ sharpens edges while amplifies noises in the estimated HR image. Large ˛ helps to suppress noise while smoothes the estimated HR image. It is explicit that ˛ is very important to the estimation of HR image. Small ˛ on edges and large ˛ on smooth regions are expected in the HR image estimation so that both the edge preservation and noise suppression could be achieved simultaneously. Consequently, ˛ should have adaptivity. To achieve this purpose, a novel regularization term which termed as locally adaptive bilateral edge-preserving regularization is proposed in this paper. Let X  (l, m) = Sxl Sym X. The regularization term based on L2 norm is defined as follow: BTV (X) =

w w  



2

˛|m|+|l| X − X  (l, m) ,

(7)

l=−w m=0

T

Fig. 1. The diagram of how to get Regional Adaptive Weighed Coefficients ˛i . (a) Adding a square window into the HR image; (b) detail of the window.

However, xi (l,m) may be just a noise point. The single pixel could not indicate the situation of the whole area around ith pixel xi (l,m). To solve this problem, a calculation method using area information of the HR image is adopted. As shown in Fig. 1(a), a square window is put into the HR image. The center pixel of this window is xi (l,m). The side of the square window is 2T + 1. The detail of the window is shown in Fig. 1(b). From Fig. 1(b), there are total (2T + 1)2 pixels in the window. It is reasonable that more effective HR image information could be given from so many pixels around pixel xi (l,m) in the window than from a single pixel xi (l,m). Taking the average of all pixels in the window, we have:

Suppose: X − X  (l, m) = [x1 (l, m), x2 (l, m), . . ., xr1 m×r2 n (l, m)],

(8)

where xi (l,m) is the pixel in the difference image between X and x (l,m). 0 ≤ xi (l,m) ≤ 225. If xi (l,m) too large, it is indicated that the area around ith pixel of the reconstructive HR image exist huge change. In other words, there are edges around ith pixel in the image. So some measures should be taken. The edges should be sharpened and the reconstructive SR image would be improved. Conversely, the area around ith pixel of the reconstructive HR image would be smooth if xi (l,m) is too small. So the key point has changed to how to suppress the noise. In summary, a method using adaptive bilateral total variation regularization term ABTV (X) is proposed according to the characteristics of different LR images. The ABTV is expressed as follow: ABTV (X) =

w w   l=−w m=0

r



1 m×r2 n



2 |m|+|l| (˛i xi (l, m))

,

(2T + 1)2

xi+n1 +n2 L2 N2 (l, m).

(11)

n1 =−T n2 =−T

So Eq. (10) is updated to

˛i = 1 −

xi (l, m) 256

=1−

T T  

1 256(2T + 1)2

xi+n1 +n2 L2 N2 (l, m)

n1 =−T n2 =−T

(12) ˛i is defined as Regional Adaptive Weight Coefficients (RAWC) which related to the regional pixels around xi (l,m). Not only does it take into account the state of pixel xi (l,m), but also consider the information of the surrounding pixels. So it could achieve better result. 3.4. Implementation

3.3. Regional Adaptive Weighed Coefficients ˛i ˛i could directly affect the HR image reconstruction. So how to choose ˛i is very important to proposed method. The basic strategy of choosing ˛i is given in Table 1. From Table 1, it can be concluded that ˛i is directly related to the size of pixel xi (l,m), accordingly a simple calculation approach of ˛i is obtained as xi (l, m) . 256

T T  

1

(9)

i=1

where ˛i is defined as Regional Adaptive Weighed Coefficients (RAWC). According to the difference of each pixel in the reconstruction HR image, ˛i can be adjusted automatically. In this way, self-adaptability of the algorithm could be effectively improved. As a result, the quality of the reconstructive HR images would be enhanced.

˛i = 1 −

xi (l, m) =

Image SR reconstruction has transform to the problem of how to solve the Eqs. (4)–(9). So this subsection describes the solving method of minimization model as follow: The model differs from the others, mainly, by the utilization of the proposed adaptive norm of the regularization terms. The functional J(X) to be minimized is defined by p w w      DBk Fk X − Yk 2 +  J(X) =



1 m×r2 n

2

l=−w m=0

k=1

 2 |m|+|l| (˛i xi (l, m))

i=1

(13) Define ˛ as Regional Adaptive Weighed Matrix and ˛ = |m|+|l| |m|+|l| |m|+|l| , ˛2 , . . ., ˛r1 m×r2 n ) diag (˛1 Then Eq. (13) can be changed to

(10)

According to Eq. (10), it can be found that 1/256 ≤ ˛i ≤ 1 and when xi (l,m) is large, ˛i would be small. It meets the basic requirement.

r

J(X) =

p w w        DBk Fk X − Yk 2 +  ˛(X − Sxl Sym X)2 2

k=1

l=−w m=0

(14)

X. Yang et al. / Optik 125 (2014) 5962–5968

5965

Table 1 The effect of ˛i in HR image reconstruction. Pixel xi (l,m)

Characteristic in HR image

Probability of occurrence

How to choose ˛i

The effect of the HR image

Large Small

There are edges on the area around ith pixel There is smooth on the area around ith pixel

Small Large

Choose small ˛i Choose large ˛i

Sharpen edges Suppress noise

There is no closed form solution for Eq. (14) and the steepest descent is adopted in this paper to find a solution of Eq. (14). Then the minimized solution of Eq. (14) can be calculated from X (n+1) = X (n) + ˇ(n) R(n) where R(n) =



(15)

FkT BkT DT (DBk Fk X (n) − Yk )

k

+

w w  

T

(16)

X(n) is the HR image of nth iteration, ˇ(n) is the learning rate of nth iteration. ˛(n) is the ˛ of nth iteration. The multiplication between matrices F, B, D, S with their transposes and the image vector can be implemented by shift, shift-back, convolution, down-sampling and up-sampling operations on the image directly to speed up the SR reconstruction process and to save both the time and space costs for computation [15,16]. Too large or too small ˇ(n) would cause both bad influences on the convergence. If ˇ(n) is too small, the convergence rate will be very slow. Conversely, if ˇ(n) is too large, the algorithm will be instable. In this paper, ∂J(X(n+1) = X(n) − ˇ(n) R(n) )/∂X = 0. Then the learning rate can be computed as T

(R(n) ) R(n)

(17)

Q (n)

where

  2  1  ˛(I − Sxl Sym )R(n) 2 c2k DBk Fk R(n)  +  2 2 2 p

Q (n) =

k=1

w

As expected, the correction image E(n) should be convergent with iteration. The convergence is not only reflected in the variation on E(n) , but also in local variance variation on E(n) , both of them are calculated between two iterations. The local variance is proportional to the regional complexity, that is to say, the value of local variance is small in relatively smooth region. Consequently, the changing process of correction image can be revealed by calculating its local variance matrix  2(n) effectively. The definition of E

T

(I − Sxl Sym ) (˛(n) ) ˛(n) (I − Sxl Sym )X (n) ,

l=−w m=0

ˇ(n) ≈

4.2. Restraint term of ringing artifacts

w

(18)

l=−w m=0

4. Ringing artifacts suppression 4.1. Suppression model According Eq. (15), the HR image can be updated using iterative process. Eq. (15) can be converted as follow: X (n+1) = X (n) + E (n)

(19)

where E(n) = ˇ(n) R(n) E(n) is considered as correction image which constantly correct HR images with the iterative processes of SR reconstruction. In original algorithm, reconstructed images have distinct ringing artifacts along with the edge. During the simulated degradation process, dim (n) ringing artifacts in X(n) are passed to Yk , and then passed to E(n) without any correction and restraint. Even worse, the ringing artifacts become more obvious during iterative processes. Based on the analysis, a restraint term J(E(n) ), which is used to correct and restrain the correction image E(n) , is introduced to suppress the ringing artifacts in this paper. Then the final SR reconstruction formula of proposed algorithm can be obtained by adding the restraint term into formula (19), as shown in Eq. (20). X (n+1) = X (n) + E (n) − J(E (n) ) where J(E(n) ) is restraint term,  is restraint coefficient.

(20)

local mean matrix mE(n) and local variance matrix m2(n) is obtained E from formulas (21) and (22). mE(n) (s, t) =

E2(n) (s, t) =

1 ω2

1 ω2



s+(ω−1)/2



t+(ω−1)/2

E (n) (i, j)

(21)

i=s−(ω−1)/2j=t−(ω−1)/2



s+(ω−1)/2



t+(ω−1)/2

[E (n) (i, j) − mE(n) (i, j)]

2

(22)

i=s−(ω−1)/2j=t−(ω−1)/2

where ω is the size of sliding window, mE(n) (s, t) and  2(n) (s, t) repE

resent the elements of mE(n) and  2(n) respectively. E As already mentioned above, an expected correction image should be convergent. However, the correction image E(n) , which brings the ringing artifacts due to the accumulation of noise, will diverge with iterative process. So an adaptively method, which based on calculating the mean variation and local variance variation on correction image, is proposed to reduce the ringing artifact. Both of the two kinds of variation are denoted respectively by mE(n) and  2(n) , as shown in Eqs. (23)–(24). E



 



mE(n) = mE(n)  − mE(n−1) 

(23)

E2(n)

(24)

    =   2(n)  −   2(n−1)  E

E

Furthermore, a threshold parameter is introduced into the adaptively analysis. If the variation is larger than threshold value, the corresponding correction will be considered to be divergent, that is to say, the correction includes the ringing artifact. Based on the above analysis, the restraint term J(E(n) ) is given as follow: J(E (n) ) =

1 [S(mE(n) , (n) ) ∧ S(mE(n) , (n) )](mE(n) + E2(n) ) 2 (25)

where S is an operator which compares the two kinds of variation, rˆepresent logical “AND” operation. The definition of S is given in Eq. (26).



S(a, b) =

1,

a>b

0,

a≤b

(26)

4.3. Adaptive threshold It is easy to known that more efficient ringing artifacts suppression can be achieved with more small threshold . So an adaptive parameter (n) instead of a fixed value is adopted in the reconstructive process to improve the effect of ringing artifacts suppression.

(n) can regulate automatically with the reconstruction result in

5966

X. Yang et al. / Optik 125 (2014) 5962–5968

Table 2 The overall procedure of proposed algorithm. Get Fk according to image registration algorithm. Initialize X(0) using common interpolation method. Get Regional Adaptive Weighed Coefficients ˛i based on Eq. (12). Get ˇ(n) R(n) according to Eqs. (16)–(18). Compute the restraint term J(E(n) ) and restraint coefficient  using Eqs. (21)–(28). Get  the HR image  X(n) ofnth iteration based on Eq. (20).

If X (i+1) − X (i)  / X (i)  < 10−6 , stop.

Fig. 2. Test images (a) “Lena”, (b) “Boat”, (c) “Peppers”,. (d) “Barbara”.

Repeat from Step. 3.

 2 each iteration. It is easy to know that the value of X (n) − X (n−1)  gets smaller along with the iteration process and the ringing artifacts become more obvious due to the accumulation of noise. Serious ringing artifacts should correspond to a smaller threshold parameter as mentioned above. So (n) should be reduced with the iteration process. On the other hand, it would achieve better ringing artifacts suppression if threshold parameter varies in the range of −2 to 0 according to a large number of experiments. Considering two opinions above, the self-adapting threshold parameter (n) is given in Eq. (27).

(IE) for the convergence study and the peak signal-to-noise ratio (PSNR) for the reconstructed HR image. IE is defined as IE(X (n) ) =

  X (n) − X 2  2 X 

PSNR is given as PSNR(X

(n)



) = 10 log10

(29)

2552 × L1 N1 × L2 N2



  X (n) − X 2

(30)

⎧  2 −1 ⎪ −0.2, − log  X (n) − X (n−1)  < −0.2 ⎪ ⎪ ⎪ ⎨  

 2 −1 2 −1

(n) = , −0.2 ≤ − lg  X (n) − X (n−1)  ≤0 − log  X (n) − X (n−1)  ⎪ ⎪   ⎪ −1  ⎪ 2 ⎩ 0, − log X (n) − X (n−1)  >0 4.4. Adaptive restraint coefficient Restraint coefficient  in Eq. (20) is an adaptive parameter which can affect the performance of the algorithm directly. When the calculation in reconstructive process is convergent, we believe that ringing artifact gets smaller. Accordingly,  should get smaller. It is assumed that the reconstruction is convergent if residual

p    DBk Fk X (n) − Yk 2 of nth frame is smaller than the 2 k=1

mean of the residual between (n − 1)th to (n − Ts + 1)th frame. Considering restraint coefficient  is related to the residual, we have

=

⎧ 0, ⎪ ⎨ ⎪ ⎩

h 1/Ts

if J(E (n) ) = 0 if J(E (n) ) = / 0 and h < 1/Ts , if

J(E (n) )

(28)

= / 0 and h ≥ 1/Ts

 p  DBk Fk X (n) −Yk 2 k=1 2 where h = n−T +1 p   , Ts is suppression threshs DBk Fk X (i) −Yk 2 i=n

k=1

2

old. Commonly, we let Ts = 3 for acceptable computational cost.

5. Overall procedure of proposed algorithm According to the above discussion, the overall procedure of proposed algorithm is given in Table 2.

6. Experiment In order to demonstrate the performance of the algorithm, a number of experiments are presented here. To evaluate the performance, the following well-known metrics is used: iteration errors

(27)

where X(n) is the estimator of X after ith iteration. Generally, good algorithm is reflected by low NMSE and high PSNR. Nevertheless, the best performance measure remains human inspection of the reconstructed HR images. In this section, we conducted various experiments to study the effect of the initial choices of motion parameters on the convergence of the proposed method. We conducted the experiments using the “Lena” and “Boat” images, shown in Fig. 2. To generate N = 5 LR images for each HR image, the HR images were rotated by different randomly chosen angles , shifted by randomly chosen subpixel translations (sx , sy ), and blurred by 2 × 2 uniform kernel before sub-sampling by a decimation factor of  = 2. The images were further degraded by AWGN. The motion parameters of different generated LR image are shown in Table 3. In our experiments, the algorithm would be terminated if the convergence criterion is satisfied:   following  X (i+1) − X (i)  / X (i)  < 10−4 . 6.1. Convergence study We plotted the IE of the reconstructed HR images for our algorithm against the number of iteration in Fig. 3. From the figures, it is clear that the iteration error converges to the true value and the convergence speed of our method is faster than that of the BTV algorithm.

Table 3 Motion parameters of different generated LR image. Regularization parameters

degrees Sx pixels Sy pixels

nth LR image 1th

2th

3th

4th

5th

6th

0 0 0

−9.2 0.51 0.82

0.9 0.33 0.14

−3.3 0.37 0.71

2.0 0.71 0.34

−4.1 0.26 0.19

X. Yang et al. / Optik 125 (2014) 5962–5968

5967

Proposed Algorithm BTV

0.004

Iteration Error

0.003

0.002

0.001

0.000 0

2

4

6

8

10

12

14

16

18

20

22

Iteration Number (a)

Proposed Algorithm BTV

0.004

Iteration Error

0.003

Fig. 4. Results obtained by applying fidelity term L2 and various regularization terms (a) degraded image, (b) L2 + RTik , (c) L2 + RBTV , (d) L2 + RLABTV , (e) proposed method.

0.002

0.001

0.000 0

2

4

6

8

10

12

14

16

18

20

22

Iteration Number (b)

Fig. 3. IE of the reconstructive HR image (a) “Lena” image, (b) “Boat” image. Table 4 PSNR (dB) results obtained by applying fidelity term L2 and various regularization terms. Image

“Lena”

“Boat”

“Peppers”

“Barbara”

Tik BTV LABTV Proposed

27.951 28.241 28.957 29.412

28.112 28.325 28.509 29.987

28.328 28.547 28.885 29.947

28.564 28.638 29.017 30.012

6.2. Effect of the proposed method Secondly, the proposed method is compared with the algorithms using the Tik, BTV and LABTV [17] regularization terms to evaluate the performance. From the experimental results shown in Table 4 and Fig. 4, it can be found that the effect of proposed method is better than others. The result of proposed method is generally more visually appealing without destroying fine details in the image. As shown in Table 5 and Fig. 5, the comparison reveals that the ringing artifacts are suppressed effectively using our method and the visual effect gets more delicate. At the same time, the

Fig. 5. The details obtained by applying different methods. (a) Original image, (b) L2 + RBTV , (c) L2 + RLABTV , (d) proposed method ( = 0.2), (e) proposed method (adaptive ).

quantitative measure gets a positive improvement. From 5, the effect of reconstruction using fixed restraint coefficient is better than using adaptive restraint coefficient. But the gap is not obvious. This can also be seen from Fig. 5.

Table 5 PSNR and NMSE results obtained by applying different methods. L2 + BTV

PSNR NMSE PSNR NMSE

“Lena” 28.241 0.0589 “Peppers” 28.547 0.0384

L2 + LABTV

Proposed method ( = 0.2)

Proposed method (adaptive )

28.957 0.0485

29.265 0.0474

29.412 0.0395

28.885 0.0353

29.546 0.0259

29.947 0.0247

L2 + BTV

“Boats” 28.325 0.0461 “Barbara” 28.638 0.0468

L2 + LABTV

Proposed method ( = 0.2)

Proposed method (adaptive )

28.509 0.0267

29.628 0.0211

29.987 0.0195

29.017 0.0426

29.856 0.0401

30.012 0.0392

5968

X. Yang et al. / Optik 125 (2014) 5962–5968

method has a preferable visual effect and satisfied ringing artifacts suppression. References

Fig. 6. SR reconstruction for real-life images (a) sample of the scaled-up LR images (b) BTV algorithm (c) LABTV algorithm (d) proposed algorithm.

6.3. Experiments on real-life images The real-life experiment was conducted by capturing four “Campus” images using a web camera with relative translations and rotations. A LR image is shown in Fig. 4(a). Next the BTV method, LABTV method [17] and the proposed method were run to perform simultaneous image registration and SR image reconstruction. The reconstructed HR images using the three methods are given in Fig. 6(b)–(d), respectively. Comparison reveals that our approach is superior in handling real-life image SR, as it is able to suppress ringing artifacts accurately, leading to superior HR image reconstruction. 7. Conclusion In order to improve reconstructive effect, this paper proposes a novel algorithm based on regional pixel information and ringing artifacts suppression. Firstly, we presented a new regularization term using Regional Adaptive Weight Coefficients (RAWC) to keep edges and flat regions. This new regularization term can be seen as the great improvement of the BTV regularizer. Then an adaptive analysis is designed based on local variance variation of correction image. At last, a restraint term is added into the iterative reconstruction process for suppress the ringing artifacts. The performance of the proposed algorithm is evaluated by testing various LR images and experiment results indicate that improved

[1] S.C. Park, M.K. Park, M.G. Kang, Super-resolution image reconstruction – a technical overview, IEEE Signal Process. Magazine 20 (3) (2003) 21–36. [2] R.Y. Tsai, T.S. Huang, Multiple frame image restoration and registration, Proc. Adv. Comput. Vision Image Process. (1984) 317–339. [3] S.P. Kim, N.K. Bose, H.M. Valenzuela, Recursive reconstruction of high resolution image from noisy under sampled multi-frames, IEEE Trans. Acoustics Speech Signal Process. 38 (6) (1990) 1013–1027. [4] S.P. Kim, W.Y. Su, Recursive high-resolution reconstruction of blurred multiframe images, IEEE Trans. Image Process. 2 (4) (1993) 534–539. [5] R.R. Schulz, R.L. Stevenson, A Bayesian approach to image expansion for improved definition, IEEE Trans. Image Process. 3 (3) (1994) 233–242. [6] R.R. Schultz, R.L. Stevenson, Extraction of high-resolution frames from video sequences, IEEE Trans. Image Process. 5 (June (6)) (1996) 996–1011. [7] N. Nguyen, P. Milanfar, G. Golub, Efficient generalized cross-validation with applications to parametric image restoration and resolution enhancement, IEEE Trans. Image Process. 10 (September (9)) (2001) 1299–1308. [8] M.G. Kang, A.K. Katesaggelos, General choice of the regularization functional in regularized image restoration, IEEE Trans. Image Process. 4 (5) (1995) 594–602. [9] H. Stark, P. Oskoui, High resolution image recovery from image-plane arrays, using convex projections, J. Opt. Soc. Am. A 6 (11) (1989) 1715–1726. [10] A.J. Patti, M.I. Sezan, A.M. Tekalp, Super-resolution video reconstruction with arbitrary sampling lattices and nonzero aperture time, IEEE Trans. Image Process. 6 (8) (1997) 1064–1076. [11] M. Irani, S. Peleg, Improving resolution by image registration, CVGIP: Graph Models Image Process. 53 (1991) 231–239. [12] H. Song, X. He, W. Chen, Y. Sun, An improved iterative back-projection algorithm for video super resolution reconstruction, in: Photonics and Optoelectronic (SOPO), 2010 Symposium, 2010, pp. 1–4. [13] A.J. Patti, Y. Altunbasak, Artifact reduction for set theoretic super resolution image reconstruction with edge adaptive constraints and higher-order interpolants, IEEE Trans. Image Process. 10 (1) (2001) 179–186. [14] C. Xiao, J. Duan, J. Yu, POCS super-resolution reconstruction from image sequences, J. Beijing Univ. Technol. 35 (1) (2009) 108–113. [15] S. Farsiu, M.D. Robinson, M. Elad, P. Milanfar, Fast and robust multi-frame super resolution, IEEE Trans. Image Process. 13 (10) (2004) 1327–1344. [16] A. Zomet, S. Peleg, Efficient super-resolution and application to mosaics, in: Proceedings of IEEE 15th International Conference on Pattern Recognition, vol. 1, Barcelona, Spain, 2000, pp. 579–583. [17] X. Li, Y. Hu, X. Gao, D. Tao, B. Ning, A multi-frame image super-resolution method, Signal Process. 90 (2010) 405–414.