- Email: [email protected]
Blind image deblurring using elastic-net based rank prior
Computer Vision and Image Understanding 168 (2018) 157–171
Contents lists available at ScienceDirect
Computer Vision and Image Understanding journal homepage: www.elsevier.com/locate/cviu
Blind image deblurring using elastic-net based rank prior a
b
⁎,a
Hongyan Wang , Jinshan Pan , Zhixun Su , Songxin Liang a b
T
a
Dalian University of Technology, China Nanjing University of Science and Technology, China
A R T I C L E I N F O
A B S T R A C T
Keywords: Image deblurring Non-local self-similarity Kernel estimation
In this paper, we propose a new image prior for blind image deblurring. The proposed prior exploits similar patches of an image and it is based on an elastic-net regularization of singular values. We quantitatively verify that it favors clear images over blurred images. This property is able to facilitate the kernel estimation in the conventional maximum a posterior (MAP) framework. Based on this prior, we develop an efficient optimization method to solve the proposed model. The proposed method does not require any complex filtering strategies to select salient edges which are critical to the state-of-the-art deblurring algorithms. We also extend the prior to deal with non-uniform image deblurring problem. Quantitative and qualitative experimental evaluations demonstrate that the proposed algorithm performs favorably against the state-of-the-art deblurring methods.
1. Introduction Blind image deblurring has been witnessed significant advances in the vision and graphics community with the last decade as it involves many challenges in problem formulation and optimization. The goal of blind image deblurring is to recover a clear image and a blur kernel from a blurred input. The blur process is usually modeled by convolution when the blur is uniform:
B = I ⊗ k + n,
(1)
where B denotes the observed blurred image, I denotes the latent clear image, k denotes the blur kernel (a.k.a., point spread function (PSF)), n denotes the additive noise, and ⊗ denotes convolution operator. This problem is highly ill-posed because only a blurred image B is known and many different pairs of I and k give rise to the same B, such as blurred images and delta blur kernels. To make this problem tractable, most of deblurring methods make assumptions on blur kernels and latent images. The assumptions usually play critical roles in deblurring methods. The sparsity of image gradients (Fergus et al., 2006; Levin et al., 2007; Shan et al., 2008) is one kind of image prior in image deblurring. Levin et al. (2009) analyze that the blind deblurring methods based on the sparsity of image gradient prior (e.g., hyper-Laplacian prior (Levin et al., 2007)) tend to favor blurred images over clear images, especially for those algorithms formulated within the MAP framework.
⁎
To overcome this limitation, some new image priors that favor clear images over blurred images have been proposed, such as normalized sparsity prior (Krishnan et al., 2011), internal patch recurrence (Michaeli and Irani, 2014), dark channel prior (Pan et al., 2016). Another kind of deblurring methods (Cho and Lee, 2009; Xu and Jia, 2010) use additional heuristic edge selection to estimate blur kernels to overcome aforementioned limitations as sharp edges usually work well in MAP based methods according to the analysis of Levin et al. (2009). However, these methods usually fail when sharp edges are not available. Different from these existing methods, we propose a novel image prior by exploiting similar patches of an image. We develop an elasticnet regularization of singular values computed from similar patches of an image to guide kernel estimation. The contributions of this work are as follows:
• We propose a novel image prior for blind image deblurring. The • • •
prior is based on an elastic-net regularization of singular values computed from similar patches of an image. We develop an efficient numerical optimization method to solve the proposed model, which is able to converge well in practice. We analyze that the proposed prior favors clear images over blurred images, which is able to facilitate kernel estimation. We extend our algorithm to deal with the non-uniform deblurring problem.
Corresponding author. E-mail address: [email protected] (Z. Su).
https://doi.org/10.1016/j.cviu.2017.11.015 Received 15 December 2016; Received in revised form 24 November 2017; Accepted 27 November 2017 Available online 05 December 2017 1077-3142/ © 2017 Elsevier Inc. All rights reserved.
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 1. The statistical results of the proposed prior on 54 pairs of test images. The proposed prior favors clear images over blurred images.
2. Related work
robustness of the algorithms. Kim et al. (2015) propose an elastic-net regularization of singular values for a number of low-rank matrix approximation problems. In this work, we propose an elastic-net based image prior for blind image deblurring. The proposed prior is built over weighted nuclear norm, giving a less shrinkage for large singular values to help preserving major structures, together with a convex F-norm to achieve a stable and robust solution. Most importantly, the proposed prior favors clear images over blurred images, which is able to facilitate blur kernel estimation.
In this section we discuss the most relevant algorithms and put this work in the proper context. As blind image deblurring is an ill-posed problem, it requires additional information to constrain the solution space. Fergus et al. (2006) use a mixture of Gaussians to learn image gradient prior via variational Bayesian inference. Shan et al. (2008) develop a certain parametric model to approximate the heavy-tailed natural image prior, where the deblurring processing is performed in a MAP framework. Comprehensive analysis by Levin et al. (2009) shows that variational Bayesian based deblurring methods (e.g., Fergus et al., 2006) are able to remove trivial solutions in comparison to other approaches with naive MAP formulations. To overcome the limitations of naive MAP formulations, numerous deblurring methods based on MAP formulations have been developed with different likelihood functions and image priors (Krishnan et al., 2011; Levin et al., 2011; Michaeli and Irani, 2014; Pan et al., 2016; Shan et al., 2008; Xu et al., 2013). As pointed by Levin et al. (2009), sharp edges usually work well in MAP framework, some edge selection based methods have been proposed (Cho and Lee, 2009; Xu and Jia, 2010). These methods introduce an additional step in the conventional MAP framework and achieve better results as evidenced by Xu and Jia (2010). However, these edge selection methods are based on the assumption that strong edges exist in the latent images and involve some heuristic image filters, e.g., bilateral filter (Cho and Lee, 2009) and shock filter (Cho and Lee, 2009; Money and Kang, 2008; Xu and Jia, 2010). These methods are likely to fail when sharp edges are not available. Instead of selecting sharp edges from blurred images, recent exemplar-based methods (Hacohen et al., 2013; Pan et al., 2014; Sun et al., 2013) exploit information contained in both a blurred input and example images from an exemplar dataset. However, querying a large external dataset is computationally expensive. Non-local self-similarity prior has been widely used in image denoising problem (Buades et al., 2005; Dong et al., 2013; Gu et al., 2014; Xu et al., 2015). Dong et al. (2013) establish the relationship between non-local self-similarity prior and low-rank prior in image denoising. As low-rank prior is usually modeled by standard nuclear norm (Cai et al., 2010; Dong et al., 2013) which may shrink all the singular values fairly, Gu et al. (2014) propose a weighted nuclear norm in image denoising. Based on the low-rankness of similar patches from both image intensity and gradient maps, Ren et al. (2016) introduce an enhanced low-rank prior for image deblurring and employ a weighted nuclear norm minimization method for optimization. Elastic-net regularization is a successful method in statistical modeling (e.g., the matrix recovery), which facilitates the stability and
3. Proposed method In this section, we first give an overview of the deblurring model in Section 3.1, and then present the proposed prior in Section 3.2. Statistical property of the proposed prior is discussed in Section 3.3. 3.1. Deblurring model Our deblurring method is based on conventional MAP framework, which is usually modeled as
min I ⊗ k − B I ,k
2 2
+ αφ (I ) + γψ (k ),
(2)
where the first term is the data term, φ(I) and ψ(k) are regularized terms formulated from image prior and kernel prior, respectively. α and γ are weights. For kernel regularization ψ(k), we follow the existing deblurring methods (Cho and Lee, 2009; Pan et al., 2014) and adopt ψ (k ) = k 22 . As the regularization φ(I) plays a critical role in image deblurring, we propose an effective prior based on similar patches of an image which is modeled by an elastic-net regularization of singular values to define φ(I) for blur kernel estimation. 3.2. Proposed prior The image prior proposed in this paper is based on non-local selfsimilarity, which is based on the observation that natural images usually contain recurrent patterns. Naturally, the deblurred image should also satisfy this property. We consider the similar patches of an image and develop an elastic-net regularization of singular values computed from similar patches. Specifically, the prior constrained on restored image I is defined as
φ (I ) = λ1 ∑ Ii i
158
+ ω, *
λ2 2
∑ i
Ii
2 F,
(3)
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
where Ii denotes the matrix stacked by those patches which are similar to ith patch in image I, ω denotes a series of different positive weights, λ1 and λ2 are parameters. By the Singular Value Decomposition (SVD) of Ii, (3) has an equivalent form which can be written as
φ (I ) = λ1 ∑ ∑ ωj σj (Ii ) + i
j
λ2 2
∑∑ i
σj (Ii ) 2 , (4)
j
where σj(Ii) denotes the jth singular value of Ii. The proposed prior is important for kernel estimation for the following reasons. The weighted nuclear norm term in (3) is a low-rank constraint. According to Gu et al. (2014), by assigning smaller weights to larger singular values and larger weights to smaller singular values, the major data components can be preserved. In our algorithm, this term plays an important role on preserving salient structures. The second term is the F-norm penalty term. By enforcing this strong convex constraint, a shrinkage from λ2 will be imposed on singular values to prevent the instability of the algorithm according to Kim et al. (2015). This may take proper correction on suboptimal solution during alternating minimization, furthermore, a possible poor solution can be avoided. 3.3. Analysis on the proposed prior
Algorithm 1. Solving intermediate image I.
An important reason why we use this prior for kernel estimation is that we find that the proposed prior favors clear images over blurred ones. To verify this property, we synthesize 54 blurred images from 9 natural images with 6 kernels, and compute the value of the proposed prior by (4) for each image. We show the comparisons of prior values between clear images and blurred images in a logarithmic scale in Fig. 1. It is clear that the blurred images have higher energy values under the proposed prior than those of clear ones, which indicates that the proposed prior favors clear images over blurred images. This property ensures that the proposed method performs well under the conventional MAP framework. 4. Optimization
min I ⊗ k − B I
2 2
+ αφ (I ),
2 2
+ γψ (k ).
Input: Kernel k. Initialize I from the previous iteration. for l = 1 → L do solve J by minimizing (10). solve I by minimizing (9). end for
In order to obtain the solutions of I and k from the blurred image B based on the image prior proposed in Section 3.2, we alternatively solve (5)
and
min I ⊗ k − B k
(6)
In the following, we present the optimization details about these two problems. 4.1. Intermediate image estimation As φ(I) in (3) involves a hybrid norm, it is difficult to obtain the solutions directly. To solve I, we use half-quadratic splitting technique proposed by Geman and Reynolds (1992) and Geman and Yang (1995). By introducing an auxiliary variable J, the minimization problem (5) can be rewritten as 2
min I ⊗ k − B I ,J 2
⎛ + α ⎜λ1 ∑ Ji i ⎝
+ ω, *
λ2 2
2
∑ i
Ji
⎞ 2 ⎟+β I − J 2 , F⎠
(7)
159
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 2. Deblurring process in a coarse-to-fine manner: intermediate estimated images and kernels at each scale level.
where β is a positive weight penalizing the similarity of I and J. For (7), we alternatively solve the following two tractable subproblems:
minαλ1 ∑ Ji J
i
+ ω, *
αλ2 2
where F (·) and F−1 (·) denote the FFT and inverse FFT, and F (·) denotes the complex conjugate of F (·) . The main process for solving I is summarized in Algorithm 1.
2
∑
+ β I − J 22 ,
Ji
i
4.2. Kernel estimation (8)
F
Given a fixed I, we can estimate the blur kernel k by (6). We note that kernel estimation methods based on gradients have been shown to be more accurate according to Cho and Lee (2009). Thus, we estimate k in gradient domain by
and
min I ⊗ k − B I
2 2
+ β I − J 22 .
(9)
For a fixed I, the minimization problem (8) can be equivalently expressed as
min ∑ J
i
2αλ1 Ji αλ2 + 2β
+ Ji − ω, *
2β Ii αλ2 + 2β
min ∇I ⊗ k − ∇B k
2
(10)
F
k = F−1 ⎛⎜ ⎝
We note that (10) is a weighted nuclear norm minimization problem (Gu et al., 2014). Thus, we can obtain the solution of (10) by
Ji = Ui Sω (Σi) ViT , 2β
where Ui, Σi and Vi are derived from the SVD of matrix αλ + 2β Ii , Sω(Σi) 2 denotes the result of weighted thresholding operating over the diagonal singular value matrix Σi, whose jth diagonal element is (12)
The weight vector ω is chosen inversely proportional to σ(Ji), and its jth element is defined by
max(σ j2 (Ii ) − 2mαλ1/(αλ2 + 2β ), 0) ,
(14)
min I ⊗ k − B 22 + μ ∇I 0.8 . I
where σj(Ii) is the jth singular value of Then a low rank matrix Ji corresponding to ith patch in I can be obtained. Once J and k are both known, model (9) is a least square problem. Thus, we can get its closed-form solution through fast Fourier transforms (FFTs) according to (Shan et al., 2008; Xu and Jia, 2010)
F (k )F (B ) + βF (J ) ⎞ ⎟, F (k )F (k ) + β ⎠
(17)
Once the blur kernel k is determined, the latent image can be estimated by a number of non-blind deconvolution methods. In this paper, we employ a hyper-Laplacian prior L0.8 by Levin et al. (2007) to recover the latent image, which can be formulated as
2β I. αλ2 + 2β i
I = F−1 ⎛⎜ ⎝
F (∂x I )F (∂x B ) + F (∂y I )F (∂y B ) ⎞ ⎟, F (∂x I )2 + F (∂y I )2 + γ ⎠
4.3. Final latent image estimation
(13)
where c is a positive constant, ϵ is a small number to avoid a zero denominator, and σj(Ji) is the jth singular value of Ji. Since Ji is unavailable in (13), we initialize it according to Gu et al. (2014) by
σ j(0) (Ji ) =
(16)
where ∂x and ∂y denote the horizontal and vertical differential operators, respectively. After obtaining blur kernel k, we set its negative elements to be 0, and normalize k so that k satisfies that the sum of its elements is 1. Similar to most of state-of-the-art deblurring methods, we adopt a coarse-to-fine manner using an image pyramid to help kernel estimation (Fergus et al., 2006; Krishnan et al., 2011; Levin et al., 2011; Shan et al., 2008), as shown in Fig. 2. The main steps for the proposed kernel estimation algorithm are shown in Algorithm 2.
(11)
ωj = c m /(σj (Ji ) + ϵ),
+ γ k 22 .
As (16) is a least square problem, we can get its closed-form solution by
.
(Sω (Σi)) jj = (Σi) jj − ωj .
2 2
(18)
We use the iterative reweighted least square (IRLS) method to solve (18). 5. Convergence property and limitations In this section, we discuss the convergence property and limitations of the proposed method.
(15)
160
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
5.1. Convergence property As the proposed objective function is highly non-convex, a natural question is whether our optimization method converges (to a good local minimum). We quantitatively evaluate the proposed method on the benchmark dataset by Levin et al. (2009). We compute the average kernel similarity metric proposed by Hu and Yang (2012) of 32 blurred images from 4 images with 8 kernels. This metric measures the similarity between estimated kernels and the ground-truth kernel via computing maximum response of normalized cross-correlation with some possible shift. Fig. 3 shows that the proposed method converges after less than 20 iterations, in terms of the average kernel similarity values (Hu and Yang, 2012). 5.2. Limitations Although the proposed method is able to help estimate blur kernels, it has limitations. If an image contains rich textures and these rich textures are located in most regions or nearly whole image, non-local self-similarity does not hold for this case. This is mainly because that there are not enough good similar patches to form a low-rank matrix due to the variety of rich textures. As a result, the proposed method is likely to fail. Fig. 4 shows a failure example. 6. Experimental results and evaluations
Algorithm 2. Kernel estimation algorithm.
In this section, we first show more insights about each term of the proposed prior. Then we examine the proposed method on both synthetic and real-world blurred images and compare it to state-of-the-art image deblurring methods. The proposed algorithm is implemented in MATLAB on a computer with an Intel Xeon E5630 CPU and 12GB RAM. The kernel estimation process takes about 12 min for a 255 × 255 image with a 29 × 29 kernel without code optimization. Parameters Setting. In all experiments, we empirically set μ = 0.002, α = 1, γ = 2, λ2 = 0.05, β = 0.025, and T = 5. The parameter λ1 is set according to blur kernel size and weighted vector parameter c are set according to Gu et al. (2014).
Input: Blurred image B and kernel size k1 , k2 . Initialize I with B. for t = 1 → T do solve for k using (17). solve for I using Algorithm 1. end for
6.1. Evaluations of the proposed prior The expression in (4) indicates that the prior relies on nonzero singular values of matrix Ii. Since each Ii is stacked by similar patches, a natural constraint is its low-rankness. In our algorithm, this constraint is modeled by the weighted nuclear norm term. It is helpful to preserve major structures and further facilitates kernel estimation. When λ1 = 0, the subproblem (10) for solving each Ji would not be a weighted nuclear norm minimization any more. Without any thresholding effect on singular values, details and tiny edges which are damaged by blur would not be removed. As they are harmful for kernel estimation, a poor kernel may be generated. On the other hand, compared with the intuitional low-rank constraint term in the prior, the role of the F-norm term seems ambiguously. In fact, the F-norm term is very critical. We find that it helps to favor clear images over blurred images. To verify we also compute the energy values of only F-norm term for clear images and blurred images respectively. The average energy value of blurred images is nearly 10 times larger than that of clear images. To better understand the role of each term, we show an example with some intermediate kernel results from kernel evolutionary process of the case of λ1 = 0, the case of λ2 = 0, and proposed prior in Fig. 5. As can be seen in Fig. 5(a)–(c), at first several coarse scales, both the case of λ1 = 0 and the case of λ2 = 0 fail to generate acceptable intermediate kernels. In comparison, the proposed prior is favorable to generate
161
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
6.3. Quantitative evaluations on the dataset by Levin et al. (2009) We evaluate the proposed algorithm on the benchmark dataset by Levin et al. (2009). This dataset contains 32 blurred images of size 255 × 255 from 4 ground-truth images with 8 blur kernels. The error ratio metric proposed by Levin et al. (2009) is commonly used to evaluate the restored results, which is defined as
ζ=
2 2 Ig 22
Ie − Ig Ik −
,
(19)
where Ie is the recovered image by the estimated blur kernel, Ik is the recovered image by the ground-truth blur kernel, and Ig is the groundtruth clear image. For fair comparisons, we use the provided code of other state-of-theart methods to generate the blur kernels. The final deblurring results are all generated by the non-blind deconvolution method (Levin et al., 2007). We compare our method with state-of-the-art methods including Cho and Lee (2009), Krishnan et al. (2011), Levin et al. (2011), Xu and Jia (2010), Xu et al. (2013), Sun et al. (2013), Michaeli and Irani (2014), and Pan et al. (2014). To better measure the quality of a recovered blur kernel k, we use both error ratios measure and average kernel similarity measure for evaluation. We show comparisons of the cumulative error ratios and average kernel similarity in Fig. 7. The graph in Fig. 7(a) shows that the proposed method and another patch-based method (Sun et al., 2013) outperform other approaches. The rightmost column in Fig. 7(b) shows that the average kernel similarity value of our method is higher than those of Sun et al. (2013) and other state-of-art methods over the entire dataset. We also test our method using other examples as shown in Fig. 8. The number in red on each image is its PSNR. Compared to other deblurring methods, our method generates images with highest PSNR.
Fig. 3. Convergence property of the proposed algorithm.
pleasing intermediate kernels. We also show some kernels at the finest scale in Fig. 5(d)–(f). Similarly, only kernels estimated by the proposed method become much cleaner and more accurate. Additionally, visual comparisons of final deblurred images (Fig. 5(g)) also indicate the effectiveness of the proposed prior.
6.2. Comparisons with commonly used priors (Krishnan et al., 2011; Michaeli and Irani, 2014; Perrone and Favaro, 2014; Sun et al., 2013) To further evaluate the effectiveness of the proposed prior, we compare the proposed prior with other priors (e.g., normalized sparsity prior (Krishnan et al., 2011), Total Variation (Perrone and Favaro, 2014), and patch-based priors (Michaeli and Irani, 2014; Sun et al., 2013)) commonly used in image deblurring. We choose 14 blurred images synthesized from the dataset (Sun et al., 2013) for test and use PSNR to evaluate the quality of each restored images. Quantitative comparisons on PSNR are shown in Fig. 6. As can be seen in Fig. 6, our method has the highest PSNR for almost all tested images. Accordingly, the average PSNR value of our method is higher than both commonly used priors (Krishnan et al., 2011; Michaeli and Irani, 2014; Perrone and Favaro, 2014; Sun et al., 2013) and the case of λ2 = 0 . These comparisons shown in Fig. 6 demonstrates the effectiveness of the proposed method.
6.4. Evaluations on real images We evaluate the proposed method on the benchmark dataset (Köhler et al., 2012) and compare it with state-of-the-art deblurring methods (Cho and Lee, 2009; Krishnan et al., 2011; Levin et al., 2011; Shan et al., 2008; Xu et al., 2013). Figs. 9–11 are three examples with close-ups from Köhler et al. (2012). As can be seen in the close-ups, our deblurred result is clearer than others as theirs contain some unpleasing effects. As the proposed method can be categorized to patch-based method and it follows similar half-quadratic splitting strategy as Xu et al. (2013) and Pan et al. (2014), we further evaluate our method by comparing it with two patch-based methods (Michaeli and Irani, 2014; Sun et al., Fig. 4. One failure example. The blurred image contains rich details which do not hold for the proposed prior due to the variety of rich textures.
162
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 5. Comparisons of kernel evolutionary process and the final deblurred results of the case of λ1 = 0, the case of λ2 = 0 and the proposed prior. Kernels on the columns from (a) to (f) are generated after scale 4 (the coarsest scale), scale 3, scale 2, 1st iteration at scale 1 (the finest scale), 3rd iteration at scale 1, and 6th iteration at scale 1, respectively. Final deblurred results are shown on the column (g). Both the case of λ1 = 0 and the case of λ2 = 0 fail to generate pleasing intermediate kernels and final deblurred results. In comparison, the intermediate kernels estimated by the proposed prior are much cleaner and more accurate. Our final deblurred result is also visually clear.
Fig. 6. Comparisons with the commonly used priors (Krishnan et al., 2011; Michaeli and Irani, 2014; Perrone and Favaro, 2014; Sun et al., 2013) in blind image deblurring. The average PSNR values of all the images are shown on the rightmost column.
Fig. 7. Quantitative evaluation on the dataset by Levin et al. (2009). Comparisons on these two quality metrics indicate that the proposed algorithm generates better results than other methods.
163
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 8. Two examples synthesized from the ground-truth dataset by Sun et al. (2013). Results shown in comparison are deblurred by Shan et al. (2008), Krishnan et al. (2011), Levin et al. (2011), Xu and Jia (2010), Xu et al. (2013), Sun et al. (2013), Michaeli and Irani (2014), and Pan et al. (2014) respectively.
164
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 9. Deblurred results with close-ups of an example from natural image dataset (Köhler et al., 2012).
2010; Xu et al., 2013) still contain severe blur effects as shown in Fig. 12(b)–(i). It is worth mentioning that the deblurred result by Sun et al. (2013) in Fig. 12(j) is acceptable visually. In comparison, our result shown in Fig. 12(l) is slightly better than their result shown in
2013) and methods (Pan et al., 2014; Xu et al., 2013). Fig. 12 shows one real blurred example. Note that all the recovered images by deblurring methods (Cho and Lee, 2009; Krishnan et al., 2011; Levin et al., 2011; Michaeli and Irani, 2014; Pan et al., 2014; Shan et al., 2008; Xu and Jia,
165
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 10. Another example from natural image dataset (Köhler et al., 2012).
Fig. 13(k), which further indicates the effectiveness of the proposed prior.
Fig. 12(j). Another real blurred example is shown in Fig. 13. As can be seen, there are several artifacts in most of the results deblurred by other stateof-the-art methods shown in Fig. 13(b)–(j). Ours shown in Fig. 13(l) is clearer and has fewer artifacts. We note that the method without using F-norm term does not perform well as shown in Fig. 12(k) and
6.5. Non-uniform image deblurring The proposed elastic-net based rank prior can be extended for non-
166
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 11. Another example from natural image dataset (Köhler et al., 2012).
uniform image deblurring. Our non-uniform deblurring algorithm is based on the geometric model of camera motion (Whyte et al., 2012). The discretized non-uniform blur model is
B=
∑ wt Ht I + n, t
image, n denotes possible additive noise, Ht is a matrix corresponding to the tth camera pose, wt is the non-negative weight which satisfies ∑t wt = 1. According to Whyte et al. (2012), when either the blur kernel or the latent sharp image is known, model (20) can be rewritten as
(20)
B = KI + n = Aw + n,
where B denotes the observed blurred image, I denotes the latent clear
167
(21)
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 12. Comparisons on a real blurred image. The recovered image by state-of-the-art deblurring methods contain significant blur effects. Ours is visually more pleasing.
where either K or A is known coefficient matrix in corresponding subproblem optimization. Our non-uniform deblurring process is achieved by alternatively optimizing:
min KI − B I
2 2
and
min Aw − B w
2 2
+ w 22 ,
(23) λ
+ αφ (I ),
where φ (I ) = λ1 ∑i Ii ω, * + 22 ∑i Ii 2F . Similar to the uniform case, we introduce an auxiliary variable J and
(22)
168
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 13. Another real blurred example compared with stateof-the-art methods. The proposed method generates much clearer image.
rewrite (22) as
min KI − B 22 +α (λ1 ∑ Ji I
i
+ ω, *
λ2 2
can be solved by alternatively computing I and J. Note that given a fixed I, the optimization of J is same to (8). We search for similar patches in the entire image since non-local self-similarity is a global property of natural images. Even though the selected similar patches may contain different blur effects in initial phase, they can be weakened in J.
2
∑ i
) + β I − J 22 .
Ji F
(24)
By using half-quadratic splitting technique, minimization problem (24)
169
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 14. Comparisons on a non-uniform example with the estimated kernel. The recovered image by the proposed prior is visually comparable.
7. Conclusion
Considering that blurs in different regions are not uniform any more, we chop B and J into small regions for solving I. In nth window, the intermediate sharp image I(n) can be restored in the same way as (15). After alternative optimization of I and J, kernel estimation process is conducted on the output sharp image I. We use the same optimization method proposed by Xu et al. (2013) for solving (23). We show some deblurred results compared with non-uniform deblurring method (Gupta et al., 2010; Pan et al., 2014; Whyte et al., 2010; Xu et al., 2013) in Figs. 14 and 15. As can be seen in Figs. 14 and 15, our method generates comparable results.
In this paper, a new image prior is proposed for blind image deblurring. The prior is based on an elastic-net regularization of singular values, which exploits the similarity of recurrent patches in an image. We show that the proposed prior favors clear images over blurred images, which helps kernel estimation within conventional MAP framework. We also develop an efficient algorithm to solve the proposed model and extend the method to deal with non-uniform case. Both quantitative and qualitative experiments show that the proposed method performs favorably against the state-of-the-art deblurring methods.
170
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 15. Another non-uniform example.
Acknowledgements
27–40. Krishnan, D., Tay, T., Fergus, R., 2011. Blind deconvolution using a normalized sparsity measure. CVPR. pp. 233–240. Levin, A., Fergus, R., Durand, F., Freeman, W.T., 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. 26 (3), 70. Levin, A., Weiss, Y., Durand, F., Freeman, W.T., 2009. Understanding and evaluating blind deconvolution algorithms. CVPR. pp. 1964–1971. Levin, A., Weiss, Y., Durand, F., Freeman, W.T., 2011. Efficient marginal likelihood optimization in blind deconvolution. CVPR. pp. 2657–2664. Michaeli, T., Irani, M., 2014. Blind deblurring using internal patch recurrence. ECCV. pp. 783–798. Money, J.H., Kang, S.H., 2008. Total variation minimizing blind deconvolution with shock filter reference. Image Vis. Comput. 26 (2), 302–314. Pan, J., Hu, Z., Su, Z., Yang, M.-H., 2014. Deblurring text images via l0-regularized intensity and gradient prior. CVPR. pp. 2901–2908. Pan, J., Sun, D., Pfister, H., Yang, M.-H., 2016. Blind image deblurring using dark channel prior. CVPR. Perrone, D., Favaro, P., 2014. Total variation blind deconvolution: the devil is in the details. CVPR. pp. 2909–2916. Ren, W., Cao, X., Pan, J., Guo, X., Zuo, W., Yang, M., 2016. Image deblurring via enhanced lowrank prior. IEEE Trans. Image Process. 25 (7), 3426–3437. Shan, Q., Jia, J., Agarwala, A., 2008. High-quality motion deblurring from a single image. ACM Trans. Graph. 27 (3), 73. Sun, L., Cho, S., Wang, J., Hays, J., 2013. Edge-based blur kernel estimation using patch priors. ICCP. pp. 1–8. Whyte, O., Sivic, J., Zisserman, A., Ponce, J., 2010. Non-uniform deblurring for shaken images. CVPR. pp. 491–498. Whyte, O., Sivic, J., Zisserman, A., Ponce, J., 2012. Non-uniform deblurring for shaken images. IJCV 98 (2), 168–186. Xu, J., Zhang, L., Zuo, W., Zhang, D., Feng, X., 2015. Patch group based non-local self-similarity prior learning for image denoising. ICCV. pp. 244–252. Xu, L., Jia, J., 2010. Two-phase kernel estimation for robust motion deblurring. ECCV. pp. 157–170. Xu, L., Zheng, S., Jia, J., 2013. Unnatural l0 sparse representation for natural image deblurring. CVPR. pp. 1107–1114.
This work has been partially supported by National Natural Science Foundation of China (No. 61572099, 51379033, and 51522902) and National Science and Technology Major Project (No. ZX20140419 and 2014ZX04001011). References Buades, A., Coll, B., Morel, J.-M., 2005. A non-local algorithm for image denoising. CVPR. 2. pp. 60–65. Cai, J.-F., Candès, E.J., Shen, Z., 2010. A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20 (4), 1956–1982. Cho, S., Lee, S., 2009. Fast motion deblurring. ACM Trans Graph 28 (5), 145. Dong, W., Shi, G., Li, X., 2013. Non-local image restoration with bilateral variance estimation: a low-rank approach. IEEE Trans. Image Process. 22 (2), 700–711. Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T., Freeman, W.T., 2006. Removing camera shake from a single photograph. ACM Trans. Graph. 25 (3), 787–794. Geman, D., Reynolds, G., 1992. Constrained restoration and the recovery of discontinuities. IEEE Trans. Pattern Anal. Mach. Intell. (3), 367–383. Geman, D., Yang, C., 1995. Nonlinear image recovery with half-quadratic regularization. IEEE Trans. Image Process. 4 (7), 932–946. Gu, S., Zhang, L., Zuo, W., Feng, X., 2014. Weighted nuclear norm minimization with application to image denoising. CVPR. pp. 2862–2869. Gupta, A., Joshi, N., Zitnick, C.L., Cohen, M., Curless, B., 2010. Single image deblurring using motion density functions. ECCV. pp. 171–184. Hacohen, Y., Shechtman, E., Lischinski, D., 2013. Deblurring by example using dense correspondence. ICCV. pp. 2384–2391. Hu, Z., Yang, M.-H., 2012. Good regions to deblur. ECCV. pp. 59–72. Kim, E., Lee, M., Oh, S., 2015. Elastic-net regularization of singular values for robust subspace learning. CVPR. pp. 915–923. Köhler, R., Hirsch, M., Mohler, B.J., Schölkopf, B., Harmeling, S., 2012. Recording and playback of camera shake: benchmarking blind deconvolution with a real-world database. ECCV. pp.
171
Contents lists available at ScienceDirect
Computer Vision and Image Understanding journal homepage: www.elsevier.com/locate/cviu
Blind image deblurring using elastic-net based rank prior a
b
⁎,a
Hongyan Wang , Jinshan Pan , Zhixun Su , Songxin Liang a b
T
a
Dalian University of Technology, China Nanjing University of Science and Technology, China
A R T I C L E I N F O
A B S T R A C T
Keywords: Image deblurring Non-local self-similarity Kernel estimation
In this paper, we propose a new image prior for blind image deblurring. The proposed prior exploits similar patches of an image and it is based on an elastic-net regularization of singular values. We quantitatively verify that it favors clear images over blurred images. This property is able to facilitate the kernel estimation in the conventional maximum a posterior (MAP) framework. Based on this prior, we develop an efficient optimization method to solve the proposed model. The proposed method does not require any complex filtering strategies to select salient edges which are critical to the state-of-the-art deblurring algorithms. We also extend the prior to deal with non-uniform image deblurring problem. Quantitative and qualitative experimental evaluations demonstrate that the proposed algorithm performs favorably against the state-of-the-art deblurring methods.
1. Introduction Blind image deblurring has been witnessed significant advances in the vision and graphics community with the last decade as it involves many challenges in problem formulation and optimization. The goal of blind image deblurring is to recover a clear image and a blur kernel from a blurred input. The blur process is usually modeled by convolution when the blur is uniform:
B = I ⊗ k + n,
(1)
where B denotes the observed blurred image, I denotes the latent clear image, k denotes the blur kernel (a.k.a., point spread function (PSF)), n denotes the additive noise, and ⊗ denotes convolution operator. This problem is highly ill-posed because only a blurred image B is known and many different pairs of I and k give rise to the same B, such as blurred images and delta blur kernels. To make this problem tractable, most of deblurring methods make assumptions on blur kernels and latent images. The assumptions usually play critical roles in deblurring methods. The sparsity of image gradients (Fergus et al., 2006; Levin et al., 2007; Shan et al., 2008) is one kind of image prior in image deblurring. Levin et al. (2009) analyze that the blind deblurring methods based on the sparsity of image gradient prior (e.g., hyper-Laplacian prior (Levin et al., 2007)) tend to favor blurred images over clear images, especially for those algorithms formulated within the MAP framework.
⁎
To overcome this limitation, some new image priors that favor clear images over blurred images have been proposed, such as normalized sparsity prior (Krishnan et al., 2011), internal patch recurrence (Michaeli and Irani, 2014), dark channel prior (Pan et al., 2016). Another kind of deblurring methods (Cho and Lee, 2009; Xu and Jia, 2010) use additional heuristic edge selection to estimate blur kernels to overcome aforementioned limitations as sharp edges usually work well in MAP based methods according to the analysis of Levin et al. (2009). However, these methods usually fail when sharp edges are not available. Different from these existing methods, we propose a novel image prior by exploiting similar patches of an image. We develop an elasticnet regularization of singular values computed from similar patches of an image to guide kernel estimation. The contributions of this work are as follows:
• We propose a novel image prior for blind image deblurring. The • • •
prior is based on an elastic-net regularization of singular values computed from similar patches of an image. We develop an efficient numerical optimization method to solve the proposed model, which is able to converge well in practice. We analyze that the proposed prior favors clear images over blurred images, which is able to facilitate kernel estimation. We extend our algorithm to deal with the non-uniform deblurring problem.
Corresponding author. E-mail address: [email protected] (Z. Su).
https://doi.org/10.1016/j.cviu.2017.11.015 Received 15 December 2016; Received in revised form 24 November 2017; Accepted 27 November 2017 Available online 05 December 2017 1077-3142/ © 2017 Elsevier Inc. All rights reserved.
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 1. The statistical results of the proposed prior on 54 pairs of test images. The proposed prior favors clear images over blurred images.
2. Related work
robustness of the algorithms. Kim et al. (2015) propose an elastic-net regularization of singular values for a number of low-rank matrix approximation problems. In this work, we propose an elastic-net based image prior for blind image deblurring. The proposed prior is built over weighted nuclear norm, giving a less shrinkage for large singular values to help preserving major structures, together with a convex F-norm to achieve a stable and robust solution. Most importantly, the proposed prior favors clear images over blurred images, which is able to facilitate blur kernel estimation.
In this section we discuss the most relevant algorithms and put this work in the proper context. As blind image deblurring is an ill-posed problem, it requires additional information to constrain the solution space. Fergus et al. (2006) use a mixture of Gaussians to learn image gradient prior via variational Bayesian inference. Shan et al. (2008) develop a certain parametric model to approximate the heavy-tailed natural image prior, where the deblurring processing is performed in a MAP framework. Comprehensive analysis by Levin et al. (2009) shows that variational Bayesian based deblurring methods (e.g., Fergus et al., 2006) are able to remove trivial solutions in comparison to other approaches with naive MAP formulations. To overcome the limitations of naive MAP formulations, numerous deblurring methods based on MAP formulations have been developed with different likelihood functions and image priors (Krishnan et al., 2011; Levin et al., 2011; Michaeli and Irani, 2014; Pan et al., 2016; Shan et al., 2008; Xu et al., 2013). As pointed by Levin et al. (2009), sharp edges usually work well in MAP framework, some edge selection based methods have been proposed (Cho and Lee, 2009; Xu and Jia, 2010). These methods introduce an additional step in the conventional MAP framework and achieve better results as evidenced by Xu and Jia (2010). However, these edge selection methods are based on the assumption that strong edges exist in the latent images and involve some heuristic image filters, e.g., bilateral filter (Cho and Lee, 2009) and shock filter (Cho and Lee, 2009; Money and Kang, 2008; Xu and Jia, 2010). These methods are likely to fail when sharp edges are not available. Instead of selecting sharp edges from blurred images, recent exemplar-based methods (Hacohen et al., 2013; Pan et al., 2014; Sun et al., 2013) exploit information contained in both a blurred input and example images from an exemplar dataset. However, querying a large external dataset is computationally expensive. Non-local self-similarity prior has been widely used in image denoising problem (Buades et al., 2005; Dong et al., 2013; Gu et al., 2014; Xu et al., 2015). Dong et al. (2013) establish the relationship between non-local self-similarity prior and low-rank prior in image denoising. As low-rank prior is usually modeled by standard nuclear norm (Cai et al., 2010; Dong et al., 2013) which may shrink all the singular values fairly, Gu et al. (2014) propose a weighted nuclear norm in image denoising. Based on the low-rankness of similar patches from both image intensity and gradient maps, Ren et al. (2016) introduce an enhanced low-rank prior for image deblurring and employ a weighted nuclear norm minimization method for optimization. Elastic-net regularization is a successful method in statistical modeling (e.g., the matrix recovery), which facilitates the stability and
3. Proposed method In this section, we first give an overview of the deblurring model in Section 3.1, and then present the proposed prior in Section 3.2. Statistical property of the proposed prior is discussed in Section 3.3. 3.1. Deblurring model Our deblurring method is based on conventional MAP framework, which is usually modeled as
min I ⊗ k − B I ,k
2 2
+ αφ (I ) + γψ (k ),
(2)
where the first term is the data term, φ(I) and ψ(k) are regularized terms formulated from image prior and kernel prior, respectively. α and γ are weights. For kernel regularization ψ(k), we follow the existing deblurring methods (Cho and Lee, 2009; Pan et al., 2014) and adopt ψ (k ) = k 22 . As the regularization φ(I) plays a critical role in image deblurring, we propose an effective prior based on similar patches of an image which is modeled by an elastic-net regularization of singular values to define φ(I) for blur kernel estimation. 3.2. Proposed prior The image prior proposed in this paper is based on non-local selfsimilarity, which is based on the observation that natural images usually contain recurrent patterns. Naturally, the deblurred image should also satisfy this property. We consider the similar patches of an image and develop an elastic-net regularization of singular values computed from similar patches. Specifically, the prior constrained on restored image I is defined as
φ (I ) = λ1 ∑ Ii i
158
+ ω, *
λ2 2
∑ i
Ii
2 F,
(3)
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
where Ii denotes the matrix stacked by those patches which are similar to ith patch in image I, ω denotes a series of different positive weights, λ1 and λ2 are parameters. By the Singular Value Decomposition (SVD) of Ii, (3) has an equivalent form which can be written as
φ (I ) = λ1 ∑ ∑ ωj σj (Ii ) + i
j
λ2 2
∑∑ i
σj (Ii ) 2 , (4)
j
where σj(Ii) denotes the jth singular value of Ii. The proposed prior is important for kernel estimation for the following reasons. The weighted nuclear norm term in (3) is a low-rank constraint. According to Gu et al. (2014), by assigning smaller weights to larger singular values and larger weights to smaller singular values, the major data components can be preserved. In our algorithm, this term plays an important role on preserving salient structures. The second term is the F-norm penalty term. By enforcing this strong convex constraint, a shrinkage from λ2 will be imposed on singular values to prevent the instability of the algorithm according to Kim et al. (2015). This may take proper correction on suboptimal solution during alternating minimization, furthermore, a possible poor solution can be avoided. 3.3. Analysis on the proposed prior
Algorithm 1. Solving intermediate image I.
An important reason why we use this prior for kernel estimation is that we find that the proposed prior favors clear images over blurred ones. To verify this property, we synthesize 54 blurred images from 9 natural images with 6 kernels, and compute the value of the proposed prior by (4) for each image. We show the comparisons of prior values between clear images and blurred images in a logarithmic scale in Fig. 1. It is clear that the blurred images have higher energy values under the proposed prior than those of clear ones, which indicates that the proposed prior favors clear images over blurred images. This property ensures that the proposed method performs well under the conventional MAP framework. 4. Optimization
min I ⊗ k − B I
2 2
+ αφ (I ),
2 2
+ γψ (k ).
Input: Kernel k. Initialize I from the previous iteration. for l = 1 → L do solve J by minimizing (10). solve I by minimizing (9). end for
In order to obtain the solutions of I and k from the blurred image B based on the image prior proposed in Section 3.2, we alternatively solve (5)
and
min I ⊗ k − B k
(6)
In the following, we present the optimization details about these two problems. 4.1. Intermediate image estimation As φ(I) in (3) involves a hybrid norm, it is difficult to obtain the solutions directly. To solve I, we use half-quadratic splitting technique proposed by Geman and Reynolds (1992) and Geman and Yang (1995). By introducing an auxiliary variable J, the minimization problem (5) can be rewritten as 2
min I ⊗ k − B I ,J 2
⎛ + α ⎜λ1 ∑ Ji i ⎝
+ ω, *
λ2 2
2
∑ i
Ji
⎞ 2 ⎟+β I − J 2 , F⎠
(7)
159
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 2. Deblurring process in a coarse-to-fine manner: intermediate estimated images and kernels at each scale level.
where β is a positive weight penalizing the similarity of I and J. For (7), we alternatively solve the following two tractable subproblems:
minαλ1 ∑ Ji J
i
+ ω, *
αλ2 2
where F (·) and F−1 (·) denote the FFT and inverse FFT, and F (·) denotes the complex conjugate of F (·) . The main process for solving I is summarized in Algorithm 1.
2
∑
+ β I − J 22 ,
Ji
i
4.2. Kernel estimation (8)
F
Given a fixed I, we can estimate the blur kernel k by (6). We note that kernel estimation methods based on gradients have been shown to be more accurate according to Cho and Lee (2009). Thus, we estimate k in gradient domain by
and
min I ⊗ k − B I
2 2
+ β I − J 22 .
(9)
For a fixed I, the minimization problem (8) can be equivalently expressed as
min ∑ J
i
2αλ1 Ji αλ2 + 2β
+ Ji − ω, *
2β Ii αλ2 + 2β
min ∇I ⊗ k − ∇B k
2
(10)
F
k = F−1 ⎛⎜ ⎝
We note that (10) is a weighted nuclear norm minimization problem (Gu et al., 2014). Thus, we can obtain the solution of (10) by
Ji = Ui Sω (Σi) ViT , 2β
where Ui, Σi and Vi are derived from the SVD of matrix αλ + 2β Ii , Sω(Σi) 2 denotes the result of weighted thresholding operating over the diagonal singular value matrix Σi, whose jth diagonal element is (12)
The weight vector ω is chosen inversely proportional to σ(Ji), and its jth element is defined by
max(σ j2 (Ii ) − 2mαλ1/(αλ2 + 2β ), 0) ,
(14)
min I ⊗ k − B 22 + μ ∇I 0.8 . I
where σj(Ii) is the jth singular value of Then a low rank matrix Ji corresponding to ith patch in I can be obtained. Once J and k are both known, model (9) is a least square problem. Thus, we can get its closed-form solution through fast Fourier transforms (FFTs) according to (Shan et al., 2008; Xu and Jia, 2010)
F (k )F (B ) + βF (J ) ⎞ ⎟, F (k )F (k ) + β ⎠
(17)
Once the blur kernel k is determined, the latent image can be estimated by a number of non-blind deconvolution methods. In this paper, we employ a hyper-Laplacian prior L0.8 by Levin et al. (2007) to recover the latent image, which can be formulated as
2β I. αλ2 + 2β i
I = F−1 ⎛⎜ ⎝
F (∂x I )F (∂x B ) + F (∂y I )F (∂y B ) ⎞ ⎟, F (∂x I )2 + F (∂y I )2 + γ ⎠
4.3. Final latent image estimation
(13)
where c is a positive constant, ϵ is a small number to avoid a zero denominator, and σj(Ji) is the jth singular value of Ji. Since Ji is unavailable in (13), we initialize it according to Gu et al. (2014) by
σ j(0) (Ji ) =
(16)
where ∂x and ∂y denote the horizontal and vertical differential operators, respectively. After obtaining blur kernel k, we set its negative elements to be 0, and normalize k so that k satisfies that the sum of its elements is 1. Similar to most of state-of-the-art deblurring methods, we adopt a coarse-to-fine manner using an image pyramid to help kernel estimation (Fergus et al., 2006; Krishnan et al., 2011; Levin et al., 2011; Shan et al., 2008), as shown in Fig. 2. The main steps for the proposed kernel estimation algorithm are shown in Algorithm 2.
(11)
ωj = c m /(σj (Ji ) + ϵ),
+ γ k 22 .
As (16) is a least square problem, we can get its closed-form solution by
.
(Sω (Σi)) jj = (Σi) jj − ωj .
2 2
(18)
We use the iterative reweighted least square (IRLS) method to solve (18). 5. Convergence property and limitations In this section, we discuss the convergence property and limitations of the proposed method.
(15)
160
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
5.1. Convergence property As the proposed objective function is highly non-convex, a natural question is whether our optimization method converges (to a good local minimum). We quantitatively evaluate the proposed method on the benchmark dataset by Levin et al. (2009). We compute the average kernel similarity metric proposed by Hu and Yang (2012) of 32 blurred images from 4 images with 8 kernels. This metric measures the similarity between estimated kernels and the ground-truth kernel via computing maximum response of normalized cross-correlation with some possible shift. Fig. 3 shows that the proposed method converges after less than 20 iterations, in terms of the average kernel similarity values (Hu and Yang, 2012). 5.2. Limitations Although the proposed method is able to help estimate blur kernels, it has limitations. If an image contains rich textures and these rich textures are located in most regions or nearly whole image, non-local self-similarity does not hold for this case. This is mainly because that there are not enough good similar patches to form a low-rank matrix due to the variety of rich textures. As a result, the proposed method is likely to fail. Fig. 4 shows a failure example. 6. Experimental results and evaluations
Algorithm 2. Kernel estimation algorithm.
In this section, we first show more insights about each term of the proposed prior. Then we examine the proposed method on both synthetic and real-world blurred images and compare it to state-of-the-art image deblurring methods. The proposed algorithm is implemented in MATLAB on a computer with an Intel Xeon E5630 CPU and 12GB RAM. The kernel estimation process takes about 12 min for a 255 × 255 image with a 29 × 29 kernel without code optimization. Parameters Setting. In all experiments, we empirically set μ = 0.002, α = 1, γ = 2, λ2 = 0.05, β = 0.025, and T = 5. The parameter λ1 is set according to blur kernel size and weighted vector parameter c are set according to Gu et al. (2014).
Input: Blurred image B and kernel size k1 , k2 . Initialize I with B. for t = 1 → T do solve for k using (17). solve for I using Algorithm 1. end for
6.1. Evaluations of the proposed prior The expression in (4) indicates that the prior relies on nonzero singular values of matrix Ii. Since each Ii is stacked by similar patches, a natural constraint is its low-rankness. In our algorithm, this constraint is modeled by the weighted nuclear norm term. It is helpful to preserve major structures and further facilitates kernel estimation. When λ1 = 0, the subproblem (10) for solving each Ji would not be a weighted nuclear norm minimization any more. Without any thresholding effect on singular values, details and tiny edges which are damaged by blur would not be removed. As they are harmful for kernel estimation, a poor kernel may be generated. On the other hand, compared with the intuitional low-rank constraint term in the prior, the role of the F-norm term seems ambiguously. In fact, the F-norm term is very critical. We find that it helps to favor clear images over blurred images. To verify we also compute the energy values of only F-norm term for clear images and blurred images respectively. The average energy value of blurred images is nearly 10 times larger than that of clear images. To better understand the role of each term, we show an example with some intermediate kernel results from kernel evolutionary process of the case of λ1 = 0, the case of λ2 = 0, and proposed prior in Fig. 5. As can be seen in Fig. 5(a)–(c), at first several coarse scales, both the case of λ1 = 0 and the case of λ2 = 0 fail to generate acceptable intermediate kernels. In comparison, the proposed prior is favorable to generate
161
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
6.3. Quantitative evaluations on the dataset by Levin et al. (2009) We evaluate the proposed algorithm on the benchmark dataset by Levin et al. (2009). This dataset contains 32 blurred images of size 255 × 255 from 4 ground-truth images with 8 blur kernels. The error ratio metric proposed by Levin et al. (2009) is commonly used to evaluate the restored results, which is defined as
ζ=
2 2 Ig 22
Ie − Ig Ik −
,
(19)
where Ie is the recovered image by the estimated blur kernel, Ik is the recovered image by the ground-truth blur kernel, and Ig is the groundtruth clear image. For fair comparisons, we use the provided code of other state-of-theart methods to generate the blur kernels. The final deblurring results are all generated by the non-blind deconvolution method (Levin et al., 2007). We compare our method with state-of-the-art methods including Cho and Lee (2009), Krishnan et al. (2011), Levin et al. (2011), Xu and Jia (2010), Xu et al. (2013), Sun et al. (2013), Michaeli and Irani (2014), and Pan et al. (2014). To better measure the quality of a recovered blur kernel k, we use both error ratios measure and average kernel similarity measure for evaluation. We show comparisons of the cumulative error ratios and average kernel similarity in Fig. 7. The graph in Fig. 7(a) shows that the proposed method and another patch-based method (Sun et al., 2013) outperform other approaches. The rightmost column in Fig. 7(b) shows that the average kernel similarity value of our method is higher than those of Sun et al. (2013) and other state-of-art methods over the entire dataset. We also test our method using other examples as shown in Fig. 8. The number in red on each image is its PSNR. Compared to other deblurring methods, our method generates images with highest PSNR.
Fig. 3. Convergence property of the proposed algorithm.
pleasing intermediate kernels. We also show some kernels at the finest scale in Fig. 5(d)–(f). Similarly, only kernels estimated by the proposed method become much cleaner and more accurate. Additionally, visual comparisons of final deblurred images (Fig. 5(g)) also indicate the effectiveness of the proposed prior.
6.2. Comparisons with commonly used priors (Krishnan et al., 2011; Michaeli and Irani, 2014; Perrone and Favaro, 2014; Sun et al., 2013) To further evaluate the effectiveness of the proposed prior, we compare the proposed prior with other priors (e.g., normalized sparsity prior (Krishnan et al., 2011), Total Variation (Perrone and Favaro, 2014), and patch-based priors (Michaeli and Irani, 2014; Sun et al., 2013)) commonly used in image deblurring. We choose 14 blurred images synthesized from the dataset (Sun et al., 2013) for test and use PSNR to evaluate the quality of each restored images. Quantitative comparisons on PSNR are shown in Fig. 6. As can be seen in Fig. 6, our method has the highest PSNR for almost all tested images. Accordingly, the average PSNR value of our method is higher than both commonly used priors (Krishnan et al., 2011; Michaeli and Irani, 2014; Perrone and Favaro, 2014; Sun et al., 2013) and the case of λ2 = 0 . These comparisons shown in Fig. 6 demonstrates the effectiveness of the proposed method.
6.4. Evaluations on real images We evaluate the proposed method on the benchmark dataset (Köhler et al., 2012) and compare it with state-of-the-art deblurring methods (Cho and Lee, 2009; Krishnan et al., 2011; Levin et al., 2011; Shan et al., 2008; Xu et al., 2013). Figs. 9–11 are three examples with close-ups from Köhler et al. (2012). As can be seen in the close-ups, our deblurred result is clearer than others as theirs contain some unpleasing effects. As the proposed method can be categorized to patch-based method and it follows similar half-quadratic splitting strategy as Xu et al. (2013) and Pan et al. (2014), we further evaluate our method by comparing it with two patch-based methods (Michaeli and Irani, 2014; Sun et al., Fig. 4. One failure example. The blurred image contains rich details which do not hold for the proposed prior due to the variety of rich textures.
162
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 5. Comparisons of kernel evolutionary process and the final deblurred results of the case of λ1 = 0, the case of λ2 = 0 and the proposed prior. Kernels on the columns from (a) to (f) are generated after scale 4 (the coarsest scale), scale 3, scale 2, 1st iteration at scale 1 (the finest scale), 3rd iteration at scale 1, and 6th iteration at scale 1, respectively. Final deblurred results are shown on the column (g). Both the case of λ1 = 0 and the case of λ2 = 0 fail to generate pleasing intermediate kernels and final deblurred results. In comparison, the intermediate kernels estimated by the proposed prior are much cleaner and more accurate. Our final deblurred result is also visually clear.
Fig. 6. Comparisons with the commonly used priors (Krishnan et al., 2011; Michaeli and Irani, 2014; Perrone and Favaro, 2014; Sun et al., 2013) in blind image deblurring. The average PSNR values of all the images are shown on the rightmost column.
Fig. 7. Quantitative evaluation on the dataset by Levin et al. (2009). Comparisons on these two quality metrics indicate that the proposed algorithm generates better results than other methods.
163
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 8. Two examples synthesized from the ground-truth dataset by Sun et al. (2013). Results shown in comparison are deblurred by Shan et al. (2008), Krishnan et al. (2011), Levin et al. (2011), Xu and Jia (2010), Xu et al. (2013), Sun et al. (2013), Michaeli and Irani (2014), and Pan et al. (2014) respectively.
164
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 9. Deblurred results with close-ups of an example from natural image dataset (Köhler et al., 2012).
2010; Xu et al., 2013) still contain severe blur effects as shown in Fig. 12(b)–(i). It is worth mentioning that the deblurred result by Sun et al. (2013) in Fig. 12(j) is acceptable visually. In comparison, our result shown in Fig. 12(l) is slightly better than their result shown in
2013) and methods (Pan et al., 2014; Xu et al., 2013). Fig. 12 shows one real blurred example. Note that all the recovered images by deblurring methods (Cho and Lee, 2009; Krishnan et al., 2011; Levin et al., 2011; Michaeli and Irani, 2014; Pan et al., 2014; Shan et al., 2008; Xu and Jia,
165
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 10. Another example from natural image dataset (Köhler et al., 2012).
Fig. 13(k), which further indicates the effectiveness of the proposed prior.
Fig. 12(j). Another real blurred example is shown in Fig. 13. As can be seen, there are several artifacts in most of the results deblurred by other stateof-the-art methods shown in Fig. 13(b)–(j). Ours shown in Fig. 13(l) is clearer and has fewer artifacts. We note that the method without using F-norm term does not perform well as shown in Fig. 12(k) and
6.5. Non-uniform image deblurring The proposed elastic-net based rank prior can be extended for non-
166
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 11. Another example from natural image dataset (Köhler et al., 2012).
uniform image deblurring. Our non-uniform deblurring algorithm is based on the geometric model of camera motion (Whyte et al., 2012). The discretized non-uniform blur model is
B=
∑ wt Ht I + n, t
image, n denotes possible additive noise, Ht is a matrix corresponding to the tth camera pose, wt is the non-negative weight which satisfies ∑t wt = 1. According to Whyte et al. (2012), when either the blur kernel or the latent sharp image is known, model (20) can be rewritten as
(20)
B = KI + n = Aw + n,
where B denotes the observed blurred image, I denotes the latent clear
167
(21)
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 12. Comparisons on a real blurred image. The recovered image by state-of-the-art deblurring methods contain significant blur effects. Ours is visually more pleasing.
where either K or A is known coefficient matrix in corresponding subproblem optimization. Our non-uniform deblurring process is achieved by alternatively optimizing:
min KI − B I
2 2
and
min Aw − B w
2 2
+ w 22 ,
(23) λ
+ αφ (I ),
where φ (I ) = λ1 ∑i Ii ω, * + 22 ∑i Ii 2F . Similar to the uniform case, we introduce an auxiliary variable J and
(22)
168
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 13. Another real blurred example compared with stateof-the-art methods. The proposed method generates much clearer image.
rewrite (22) as
min KI − B 22 +α (λ1 ∑ Ji I
i
+ ω, *
λ2 2
can be solved by alternatively computing I and J. Note that given a fixed I, the optimization of J is same to (8). We search for similar patches in the entire image since non-local self-similarity is a global property of natural images. Even though the selected similar patches may contain different blur effects in initial phase, they can be weakened in J.
2
∑ i
) + β I − J 22 .
Ji F
(24)
By using half-quadratic splitting technique, minimization problem (24)
169
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 14. Comparisons on a non-uniform example with the estimated kernel. The recovered image by the proposed prior is visually comparable.
7. Conclusion
Considering that blurs in different regions are not uniform any more, we chop B and J into small regions for solving I. In nth window, the intermediate sharp image I(n) can be restored in the same way as (15). After alternative optimization of I and J, kernel estimation process is conducted on the output sharp image I. We use the same optimization method proposed by Xu et al. (2013) for solving (23). We show some deblurred results compared with non-uniform deblurring method (Gupta et al., 2010; Pan et al., 2014; Whyte et al., 2010; Xu et al., 2013) in Figs. 14 and 15. As can be seen in Figs. 14 and 15, our method generates comparable results.
In this paper, a new image prior is proposed for blind image deblurring. The prior is based on an elastic-net regularization of singular values, which exploits the similarity of recurrent patches in an image. We show that the proposed prior favors clear images over blurred images, which helps kernel estimation within conventional MAP framework. We also develop an efficient algorithm to solve the proposed model and extend the method to deal with non-uniform case. Both quantitative and qualitative experiments show that the proposed method performs favorably against the state-of-the-art deblurring methods.
170
Computer Vision and Image Understanding 168 (2018) 157–171
H. Wang et al.
Fig. 15. Another non-uniform example.
Acknowledgements
27–40. Krishnan, D., Tay, T., Fergus, R., 2011. Blind deconvolution using a normalized sparsity measure. CVPR. pp. 233–240. Levin, A., Fergus, R., Durand, F., Freeman, W.T., 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. 26 (3), 70. Levin, A., Weiss, Y., Durand, F., Freeman, W.T., 2009. Understanding and evaluating blind deconvolution algorithms. CVPR. pp. 1964–1971. Levin, A., Weiss, Y., Durand, F., Freeman, W.T., 2011. Efficient marginal likelihood optimization in blind deconvolution. CVPR. pp. 2657–2664. Michaeli, T., Irani, M., 2014. Blind deblurring using internal patch recurrence. ECCV. pp. 783–798. Money, J.H., Kang, S.H., 2008. Total variation minimizing blind deconvolution with shock filter reference. Image Vis. Comput. 26 (2), 302–314. Pan, J., Hu, Z., Su, Z., Yang, M.-H., 2014. Deblurring text images via l0-regularized intensity and gradient prior. CVPR. pp. 2901–2908. Pan, J., Sun, D., Pfister, H., Yang, M.-H., 2016. Blind image deblurring using dark channel prior. CVPR. Perrone, D., Favaro, P., 2014. Total variation blind deconvolution: the devil is in the details. CVPR. pp. 2909–2916. Ren, W., Cao, X., Pan, J., Guo, X., Zuo, W., Yang, M., 2016. Image deblurring via enhanced lowrank prior. IEEE Trans. Image Process. 25 (7), 3426–3437. Shan, Q., Jia, J., Agarwala, A., 2008. High-quality motion deblurring from a single image. ACM Trans. Graph. 27 (3), 73. Sun, L., Cho, S., Wang, J., Hays, J., 2013. Edge-based blur kernel estimation using patch priors. ICCP. pp. 1–8. Whyte, O., Sivic, J., Zisserman, A., Ponce, J., 2010. Non-uniform deblurring for shaken images. CVPR. pp. 491–498. Whyte, O., Sivic, J., Zisserman, A., Ponce, J., 2012. Non-uniform deblurring for shaken images. IJCV 98 (2), 168–186. Xu, J., Zhang, L., Zuo, W., Zhang, D., Feng, X., 2015. Patch group based non-local self-similarity prior learning for image denoising. ICCV. pp. 244–252. Xu, L., Jia, J., 2010. Two-phase kernel estimation for robust motion deblurring. ECCV. pp. 157–170. Xu, L., Zheng, S., Jia, J., 2013. Unnatural l0 sparse representation for natural image deblurring. CVPR. pp. 1107–1114.
This work has been partially supported by National Natural Science Foundation of China (No. 61572099, 51379033, and 51522902) and National Science and Technology Major Project (No. ZX20140419 and 2014ZX04001011). References Buades, A., Coll, B., Morel, J.-M., 2005. A non-local algorithm for image denoising. CVPR. 2. pp. 60–65. Cai, J.-F., Candès, E.J., Shen, Z., 2010. A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20 (4), 1956–1982. Cho, S., Lee, S., 2009. Fast motion deblurring. ACM Trans Graph 28 (5), 145. Dong, W., Shi, G., Li, X., 2013. Non-local image restoration with bilateral variance estimation: a low-rank approach. IEEE Trans. Image Process. 22 (2), 700–711. Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T., Freeman, W.T., 2006. Removing camera shake from a single photograph. ACM Trans. Graph. 25 (3), 787–794. Geman, D., Reynolds, G., 1992. Constrained restoration and the recovery of discontinuities. IEEE Trans. Pattern Anal. Mach. Intell. (3), 367–383. Geman, D., Yang, C., 1995. Nonlinear image recovery with half-quadratic regularization. IEEE Trans. Image Process. 4 (7), 932–946. Gu, S., Zhang, L., Zuo, W., Feng, X., 2014. Weighted nuclear norm minimization with application to image denoising. CVPR. pp. 2862–2869. Gupta, A., Joshi, N., Zitnick, C.L., Cohen, M., Curless, B., 2010. Single image deblurring using motion density functions. ECCV. pp. 171–184. Hacohen, Y., Shechtman, E., Lischinski, D., 2013. Deblurring by example using dense correspondence. ICCV. pp. 2384–2391. Hu, Z., Yang, M.-H., 2012. Good regions to deblur. ECCV. pp. 59–72. Kim, E., Lee, M., Oh, S., 2015. Elastic-net regularization of singular values for robust subspace learning. CVPR. pp. 915–923. Köhler, R., Hirsch, M., Mohler, B.J., Schölkopf, B., Harmeling, S., 2012. Recording and playback of camera shake: benchmarking blind deconvolution with a real-world database. ECCV. pp.
171